Introduction
CoreXL is a performance-enhancing technology for Security Gateways on platforms with multiple CPU cores. CoreXL enhances Security Gateway performance by enabling the processing CPU cores to concurrently perform multiple tasks.
On a Security Gateway with CoreXL enabled, the Firewall kernel is replicated multiple times. Each replicated copy, or Firewall instance, runs on one processing CPU core. These Firewall instances handle traffic concurrently, and each Firewall instance is a complete and independent Firewall inspection kernel. When CoreXL is enabled, all the Firewall kernel instances in the Security Gateway process traffic through the same interfaces and apply the same security policy.
The CoreXL software architecture includes the Secure Network Distributor (SND). The SND is responsible for:
- Processing incoming traffic from the network interfaces
- Securely accelerating authorized packets (if SecureXL is running)
- Distributing non-accelerated packets or Medium Path packets among CoreXL FW kernel instances - this functionality is also referred to as dispatcher
Traffic received on network interface cards (NICs) is directed to a processing core running the SND.
The dispatcher is executed when a packet should be forwarded to a CoreXL FW instance (in Slow path and Medium path - see sk98737 for details) and is in charge of selecting the CoreXL FW instance that will inspects the packet.
In R77.20 and lower versions, traffic distribution between CoreXL FW instances is statically based on Source IP addresses, Destination IP addresses, and the IP 'Protocol' type. Therefore, there are possible scenarios where one or more CoreXL FW instances would handle more connections, or perform more processing on the packets forwarded to them, than the other CoreXL FW instances.
This may lead to a situation, where the load is not balanced across the CPU cores, on which the CoreXL FW instances are running.
To help mitigate the above issue, CoreXL Dynamic Dispatcher feature was introduced in R77.30 Security Gateway.
CoreXL Dynamic Dispatcher
Rather than statically assigning new connections to a CoreXL FW instance based on packet's IP addresses and IP protocol (static hash function), the new dynamic assignment mechanism is based on the utilization of CPU cores, on which the CoreXL FW instances are running.
The dynamic decision is made for first packets of connections, by assigning each of the CoreXL FW instances a rank, and selecting the CoreXL FW instance with the lowestrank.
The rank for each CoreXL FW instance is calculated according to its CPU utilization.
The higher the CPU utilization, the higher the CoreXL FW instance's rank is, hence this CoreXL FW instance is less likely to be selected by the CoreXL SND.
The CoreXL Dynamic Dispatcher allows for better load distribution and helps mitigate connectivity issues during traffic "peaks", as connections opened at a high rate that would have been assigned to the same CoreXL FW instance by a static decision, will now be distributed to several CoreXL FW instances.
When CoreXL Dynamic Dispatcher is enabled, the dynamic decision is always made (even when there is no significant load).
Configuration on Security Gateway R80.10 and above
CoreXL Dynamic Dispatcher is enabled by default. Meaning, CoreXL dynamically assigns new connections to a CoreXL FW instance based on the utilization of CPU cores.
Instructions:
- To check the current mode on Security Gateway:
[Expert@HostName]# fw ctl multik dynamic_dispatching get_mode
Example output:
[Expert@R80.10:0]# fw ctl multik dynamic_dispatching get_mode Current mode is On [Expert@R80.10:0]#
- To Enable the CoreXL Dynamic Dispatcher on Security Gateway:
Note: In cluster environment, this procedure must be performed on all members of the cluster. Since a reboot is required, it is recommended to follow the Installation and Upgrade Guide - Chapter 'Upgrading ClusterXL Deployments' - either "Minimal Effort Upgrade" procedure, or "Zero Downtime Upgrade" procedure. - Run in Expert mode:
[Expert@HostName]# fw ctl multik dynamic_dispatching on
Example output:
[Expert@R80.10:0]# fw ctl multik dynamic_dispatching on New mode is: On Please reboot the system [Expert@R80.10:0]# - Reboot (in cluster, this might cause fail-over).
- Run in Expert mode:
- To Disable the CoreXL Dynamic Dispatcher on Security Gateway:
- Run in Expert mode:
[Expert@HostName]# fw ctl multik dynamic_dispatching off
Example output:
[Expert@R80.10:0]# fw ctl multik dynamic_dispatching off New mode is: Off Please reboot the system [Expert@R80.10:0]# - Reboot (in cluster, this might cause fail-over).
- Run in Expert mode:
Monitoring CoreXL load distribution
This section describes how administrator can monitor either the current CoreXL load distribution to decide whether to enable CoreXL Dynamic Dispatcher, or the CoreXL Dynamic Dispatcher in action (after it was enabled):
Note: To monitor the CoreXL Dynamic Dispatcher in action, it must already be enabled with fw ctl multik set_mode 9 command and Security Gateway must already be rebooted.
The examples provided below were taken from the machine without VPN blade or VoIP traffic and with disabled CoreXL Dynamic Dispatcher. These examples show an excellent reason to enable CoreXL Dynamic Dispatcher to mitigate the load imbalance between CoreXL FW instances:
- Check on which CPU cores the CoreXL FW instances are running with the fw ctl affinity -l -r command.
Example (CoreXL FW instances are running on CPU 1, CPU 2, and CPU 3):
[Expert@HostName]# fw ctl affinity -l -r CPU 0: eth0 eth1 eth2 CPU 1: fw_2 CPU 2: fw_1 CPU 3: fw_0 All: cpca status_proxy in.geod fwm cpstat_monitor fwd mpdaemon cpsead cpd cprid [Expert@HostName]# - Check the current CPU utilization by each CoreXL FW instance with the top command.
Note: If the output does not show all CPU cores (if 3rd line shows "Cpu(s):"), then press "1" and then "Shift+W".
Example (Load on CPU 3 is 24% by SoftIRQ; CoreXL FW instance 0 is consuming 18%; other CoreXL FW instances (fw_worker threads) are idle):
Tasks: 118 total, 3 running, 115 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0%us, 0.0%sy, 0.0%ni, 94.0%id, 0.0%wa, 1.2%hi, 4.8%si, 0.0%st Cpu1 : 0.0%us, 1.2%sy, 0.0%ni, 98.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu2 : 40.2%us, 9.2%sy, 0.0%ni, 49.4%id, 0.0%wa, 0.0%hi, 1.1%si, 0.0%st Cpu3 : 1.3%us, 1.3%sy, 0.0%ni, 73.3%id, 0.0%wa, 0.0%hi, 24.0%si, 0.0%st Mem: 4078484k total, 4021144k used, 57340k free, 241380k buffers Swap: 3140696k total, 64k used, 3140632k free, 414744k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 15964 admin 18 0 1644 436 368 R 41 0.0 0:00.42 cp_logrotate 4129 admin 15 0 0 0 0 R 18 0.0 8:23.67 fw_worker_0 15954 admin 15 0 2176 1112 840 R 2 0.0 0:00.09 top 14 admin 10 -5 0 0 0 S 0 0.0 47:01.09 events/0 4995 admin 15 0 207m 66m 24m S 1 1.7 79:04.08 cpd 1 admin 15 0 2044 724 624 S 0 0.0 0:00.43 init 2 admin RT -5 0 0 0 S 0 0.0 3:08.83 migration/0 3 admin 15 0 0 0 0 S 0 0.0 0:02.15 ksoftirqd/0 4 admin RT -5 0 0 0 S 0 0.0 0:00.05 watchdog/0 5 admin RT -5 0 0 0 S 0 0.0 2:46.42 migration/1 6 admin 15 0 0 0 0 S 0 0.0 0:00.01 ksoftirqd/1 7 admin RT -5 0 0 0 S 0 0.0 0:00.36 watchdog/1 8 admin RT -5 0 0 0 S 0 0.0 2:36.56 migration/2 9 admin 17 0 0 0 0 S 0 0.0 0:00.24 ksoftirqd/2 - Check the distribution of connections across all CoreXL FW instances with the fw ctl multik stat command.
Example (refer to the "Peak" column - CoreXL FW instance 0 processed almost all connections, while other CoreXL FW instances were idle):
[Expert@HostName]# fw ctl multik stat ID | Active | CPU | Connections | Peak ---------------------------------------------- 0 | Yes | 3 | 1 | 21623 1 | Yes | 2 | 0 | 6 2 | Yes | 1 | 1 | 13 [Expert@HostName]#
Limitations
The CoreXL Dynamic Dispatcher is not supported in the following scenarios:
- Security Gateway is configured in VSX mode (not supported on any of the VSs, including VS0)
- SAM acceleration card is installed on the appliance
- Carrier Grade NAT (CGN) is configured
- Security Gateway is configured in Monitor Mode (per sk101670)
- 6in4 tunnel (SIT interface) is configured
- Some lines in the $FWDIR/boot/modules/fwkern.conf file are commented out (refer to sk106309).
The following types of traffic are not load-balanced by the CoreXL Dynamic Dispatcher (this traffic will always be handled by the same CoreXL FW instance):
- VoIP
- VPN encrypted packets
Additional known limitations:
- sk108894 - Difficulties in connecting to untrusted sites when both HTTPS Inspection and CoreXL Dynamic Dispatcher are enabled
- sk108856 - R77.30 cluster member might go Down after disabling CoreXL Dynamic Dispatcher only on one member
FAQ
- When should I enable CoreXL Dynamic Dispatcher?
Note: The thresholds given in this answer are approximate and should be suitable for majority of environments.
Administrator should enable CoreXL Dynamic Dispatcher if:
- A CoreXL FW instance consumes its CPU core at 85% (and above) even for 1 second.
- Other CoreXL FW instances consume their CPU cores at 75% (and below) - i.e., the difference in CPU consumption between overloaded and normally loaded CoreXL FW instances is 10% and more.
- When will Security Gateway benefit from enabling CoreXL Dynamic Dispatcher?
- When CPU load is not properly balanced, Dynamic Dispatcher will distribute the load equally between the CPU cores.
- Dynamic Dispatcher will mostly benefit on machines with large number of CPU cores (multi core machines). On machine with small number of CPU cores, the static hash distribution of traffic is good enough, and the benefit of enabling Dynamic Dispatcher is low.
Additional common scenarios, in which enabling CoreXL Dynamic Dispatcher will improve Security Gateway performance, are cases when traffic is dropped/delayed due CPU load on Security Gateway.
For example, during traffic peaks, during policy installation.
Refer to "Monitoring CoreXL load distribution" section. - Can CoreXL assign some connections statically and some connections dynamically?
By default, CoreXL assigns all connections statically.
If you fully enable the CoreXL Dynamic Dispatcher, then CoreXL assigns all connections dynamically.
In addition, refer to "Limitations" section. - Does affinity of interfaces change when enabling CoreXL Dynamic Dispatcher?
CoreXL Dynamic Dispatcher does not affect affinity of interfaces.
For information about affinity of interfaces when CoreXL is enabled, refer to sk98737 - ATRG: CoreXL - section "(4) Architecture".
