Increase_intel_Buffersize

How to increase sizes of buffer on SecurePlatform/Gaia for Intel NIC and Broadcom NIC 

Solution

Table of Contents:

• Background
• Explanation
• Instructions for Gaia OS
• Instructions for SecurePlatform OS
• Related solutions
• Questions and Answers

Background

Increasing the size of receive and transmit ring buffers allocated by the NIC driver, allows the NIC driver to queue more incoming/outgoing packets, and thus decreases packet drops - especially in scenarios with high CPU utilization.

Explanation

Size if ring buffer is the number of slots/entries/descriptors in the ring buffer. A ring buffer size of 2048 could hold 2048 frames; assuming all frames were of maximal size (MTU 1500 + 22 for header) that would be a total of 3.1 MB when set at 2048.

Instructions for Gaia OS

To check the current and the maximum ring buffer sizes, run the following command:
HostName> show interface NAME_of_PHYSICAL_INTERFACE rx-ringsize
HostName> show interface NAME_of_PHYSICAL_INTERFACE tx-ringsize
Output example:

HostName> show interface eth0 rx-ringsize
Receive buffer ring size:256
Maximum receive buffer ring size:4096

In this example, 4096 is the maximum size of RX ring buffers for this particular driver, and the current ring buffer size is 256.

Increase the size of the ring buffer to 1024 descriptors.
If needed, keep increasing the size to 2048 descriptors, and to the maximum 4096 descriptors.

• To increase the size of the RX ring buffer on-the-fly, run this command:
  HostName> set interface NAME_of_PHYSICAL_INTERFACE rx-ringsize NUMBER_OF_DESCRIPTORS
  Example:
  HostName> set interface eth0 rx-ringsize 1024
• To increase the size of the TX ring buffer on-the-fly, run this command:
  HostName> set interface NAME_of_PHYSICAL_INTERFACE tx-ringsize NUMBER_OF_DESCRIPTORS
  Example:
  HostName> set interface eth0 tx-ringsize 1024
• To make the change permanent, save the configuration:
  HostName> save config

Instructions for SecurePlatform OS

To get the list of all interfaces, run the following command (and ignore the VLAN/Bond/Warp interfaces):
[Expert@HostName]# ifconfig -a | grep -E "HWaddr|inet"
To check the current and the maximum ring buffer sizes, run the following command:
[Expert@HostName]# ethtool -g NAME_of_PHYSICAL_INTERFACE
Output example:

[Expert@HostName]# ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:  4096
RX Mini: 0
RX Jumbo: 0
TX:  4096
Current hardware settings:
RX:   256
RX Mini: 0
RX Jumbo: 0
TX:   256

In this example, 4096 is the maximum size of RX and TX ring buffers for this particular driver, and the current ring buffer size is 256.

Increase the size of the ring buffer to 1024 descriptors.
If needed, keep increasing the size to 2048 descriptors, and to the maximum 4096 descriptors.

• To increase the size of the RX ring buffer on-the-fly, run this command:
  [Expert@HostName]# ethtool -G NAME_of_PHYSICAL_INTERFACE rx NUMBER_OF_DESCRIPTORS
  Example:
  [Expert@HostName]# ethtool -G eth0 rx 1024
• To increase the size of the TX ring buffer on-the-fly, run this command:
  [Expert@HostName]# ethtool -G NAME_of_PHYSICAL_INTERFACE tx NUMBER_OF_DESCRIPTORS
  Example:
  [Expert@HostName]# ethtool -G eth0 tx 1024
• To increase the size of the RX ring buffer and the TX ring buffer on-the-fly, run this command:
  [Expert@HostName]# ethtool -G NAME_of_PHYSICAL_INTERFACE rx NUMBER_OF_DESCRIPTORS tx NUMBER_OF_DESCRIPTORS
  Example:
  [Expert@HostName]# ethtool -G eth0 rx 1024 tx 1024
• To make the change permanent, add the relevant command (for each involved inteface) in the /etc/init.d/network start-up script (right above the last line "exit 0"), and reboot the Security Gateway after making changes.

  ethtool -G NAME_of_PHYSICAL_INTERFACE rx NUMBER_OF_DESCRIPTORS tx NUMBER_OF_DESCRIPTORS
  
  exit 0
  

  Another way to make the change permanent is to add the relevant parameters for each involved driver in the /etc/modprobe.conf file and reboot the Security Gateway (for relevant parameters consult the driver vendor's web site).
  

  Example for e1000 driver:
  options e1000 RxDescriptors=1024 TxDescriptors=1024

Related solutions

• sk40120 - What is the Ring Descriptor Size and under what circumstances should it be increased on IPSO OS.
• sk90062 - TCP Segmentation Offload (TSO) is reenabled after reboot on 10GB interface.

Questions and Answers

Click Here to Show Entire List

• What are the side effects of increasing the buffer? Or what are the downsides of implementing the buffer increase?
  
• Can an over-sized buffer cause issues?
  
• Does the limit of 4096 exist for 10Gb and 1Gb?
  
• Should a user go ahead and apply this change to healthy environment as counter-measure for future issues?
  
• How can I see the buffer utilization statistics?