Wednesday, October 5, 2016

gratuitous_arp




Edit the file $FWDIR/boot/modules/fwkern.conf, adding the line:
fwha_use_arp_packet_queue=1
Note: If the file does not exist, create it.


~expert mode~
cd $FWDIR/boot/modules
vi fwkern.conf
fwha_gratuitous_arp_timeout = 10
esc
:w
:q


How to validate the Gratuitous Arp 

[Expert@myvpn-fwa:0]# fw ctl get int fwha_gratuitous_arp_timeout
fwha_gratuitous_arp_timeout = 600
[Expert@myvpn-fwa:0]#



How to enable ClusterXL Virtual MAC (VMAC) mode
Solution
Table of Contents:
  1. Overview
  2. VMAC support
  3. Configuring VMAC mode
    • In SmartDashboard (preferred method)
    • On Cluster Members
  4. VMAC value
    • Versions R76 and above
    • Versions prior to R76
  5. Verifying VMAC mode
  6. Related Solutions

(1) Overview

Upon fail-over in ClusterXL that is configured in High Availability New mode / Load Sharing Unicast mode (Note: any VSX cluster works in High Availability mode), a new Active member / new Pivot member will send a series of Gratuitous ARP Requests (G-ARP packets) for its Virtual IP addresses with the Physical MAC address of the new Active member / new Pivot member on each cluster subnet (and then the new Active member / new Pivot member sends such G-ARP packet every 'N' seconds - the timeout is controlled by the value (in HTUs) of the global kernel parameter 'fwha_gratuitous_arp_timeout').
Reports from the field show that some switches do not integrate the information from G-ARP packets fast enough into their ARP cache table. This leads to problems, where the switches still send the traffic to the physical MAC address of the old Active member / Pivot member that failed. As a result, there is a traffic outage on the network until the switches update their ARP cache table.
In some other cases, when there is a large number of Static NAT entries on a ClusterXL member, upon failover, the ClusterXL mechanism sends many G-ARP packets. Reports from the field show that in some cases, the number of G-ARP packets is more than different switches can handle, and some traffic is lost. In some other cases, different network components (VoIP phones) categorically ignore the G-ARP packets.
This Solution introduces a new variation of the High Availability New mode / Load Sharing Unicast mode for the cluster product: Cluster Virtual MAC (VMAC), which is available starting in R71.
When you configure a cluster to use VMAC with ClusterXL High Availability New mode, or Load Sharing Unicast mode, all cluster members associate the same Virtual MAC address with Virtual IP address.
Traffic originating from Active / Pivot cluster member is sent out in the following way:
  • Layer 2 Source (MAC) address - physical MAC address of corresponding member's interface.
  • Layer 3 Source (IP) address - Virtual IP address that was configured for corresponding cluster interfaces.
Traffic originating from hosts on the network should be sent out in the following way:
  • Layer 2 Source (MAC) address - physical MAC address of corresponding host's interface.
  • Layer 2 Destination (MAC) address - Virtual MAC address created for corresponding cluster interfaces.
  • Layer 3 Source (IP) address - IP address of corresponding host's interface.
  • Layer 3 Destination (IP) address - Virtual IP address that was configured for corresponding cluster interfaces.
As a result, the potential traffic outage during fail-over is minimized, and the use of G-ARP packets for NATed IP addresses becomes unnecessary.
In case switches/routers on the network still do not integrate the information from G-ARP packets fast enough into their ARP cache table, configure static MAC entry to associate Virtual MAC address and ports, which are connected to cluster members (refer to any ClusterXL Administration Guide (R71, R75, R75.20, R75.40, R75.40VS, R76, R77) - chapter 'High Availability and Load Sharing in ClusterXL' - 'Hardware Requirements, Compatibility and Cisco Example' - Table 4-2, Table 4-3, Table 4-4, Table 4-5).
Use of Virtual MAC is different from the use of Shared MAC in HA Legacy mode. VMAC that is advertised by the cluster members (via Gratuitous ARP Requests) keeps the real MAC address of each member and adds another Virtual MAC address on top of it. In HA Legacy mode, the Shared MAC address that is advertised replaces the real MAC address.
It is necessary to keep the real MAC address of each member, because connectivity to the member itself is required (with the real IP address of the member).

(2) VMAC support

Operating Systems The VMAC mode is supported only on SecurePlatform OS and on Gaia OS.
ClusterXL The VMAC mode is supported in ClusterXL only in High Availability New mode, and in Load Sharing Unicast mode.
VSX cluster The VMAC mode is supported in VSX cluster - both in High Availability (default mode), and in Virtual System Load Sharing (VSLS) mode (Note: any VSX cluster works in High Availability mode).
VMAC configuration method VMAC mode can be configured:
  • In SmartDashboard (via checkbox in 'ClusterXL and VRRP' section) - only in versions R76 and above, on Gaia OS (not including VSX)
  • On Cluster Members (via CLI) - in versions R71 and above; in R75.40VS/R76 and above in VSX mode
Virtual MAC address The 'ifconfig' command does not show the VMAC addresses (use 'cphaprob -a if' command instead).

(3) Configuring VMAC mode

Configure the VMAC mode in one of the following two ways:

  • In SmartDashboard (preferred method)

    Note: This method is available only for versions R76 and above, on Gaia OS (not including VSX).
    1. SmartDashboard - cluster object - 'Properties' - 'ClusterXL and VRRP' section:

      • under 'Select the cluster mode and configuration' - select 'High Availability' - select 'ClusterXL'
      • under 'Advanced Settings' - check the box 'Use Virtual MAC' - click 'OK'

    2. Save the changes: 'File' menu - 'Save'.
    3. Install the security policy onto this cluster object.

  • On Cluster Members

    • To enable VMAC mode, set the value of global kernel parameter 'fwha_vmac_global_param_enabled' to 1 (default value is 0).
    • To disable VMAC mode, set the value of global kernel parameter 'fwha_vmac_global_param_enabled' to 0.
    • To set the VMAC mode value on-the-fly,
      run this command on all cluster members:
      [Expert@HostName]# fw ctl set int fwha_vmac_global_param_enabled VALUE
    • To set the VMAC mode value permanently:
      refer to sk26202 (Changing the kernel global parameters on all platforms)
    Note:
    In order to get the current value of global kernel parameter 'fwha_vmac_global_param_enabled', run the this command on a cluster member:
    [Expert@HostName]# fw ctl get int fwha_vmac_global_param_enabled

(4) VMAC value

  • Versions R76 and above

    All cluster interfaces are configured with the same VMAC address using the following format (having different VMAC addresses for each cluster interface is not necessary - typically a modern switch uses a different CAM table for each VLAN, so it is possible to have the same MAC address on the same switch in different VLANs; in addition, having the same VMAC for each cluster interface makes it easier to understand, which cluster is sending the packet without a need to know each interface address):
    Total 48 bits (from left-to-right):

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
    0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
    00 1C 7F 0000000 VSID value of 'fwha_mac_magic'

  • Versions prior to R76

    Specific range of VMAC addresses was assigned for VMAC mode:
    00:1C:7F:00:00:00 -> 00:1C:7F:1F:FF:FF
    Total 48 bits (from left-to-right):
    • Left 24 bits - 00:1C:7F - unique constant value.
    • Next 3 bits - are 000.
    • Next 13 bits - sequence number - interfaces on different members that belong to the same VIP, share the same sequence number.
    • Next 8 bits - value of the global kernel parameter 'fwha_mac_magic' (default hex value is 'FE') - needed to prevent collisions between different clusters on the same segment - refer to sk25977 (Connecting multiple clusters to the same network segment (same VLAN, same switch).

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
    0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 1 1 1 1 1 1 0 0 0 ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
    00 1C 7F 000 sequence number value of 'fwha_mac_magic'

(5) Verifying VMAC mode

  1. Run the 'cphaprob -a if' command on all cluster members:

    Notes:
    • On SecurePlatform OS, this command has to be run in Expert shell.
    • On Gaia OS, this command can be run in either shell (Clish, or Bash).
    • Each cluster interface will display its VMAC address, if VMAC mode is configured.


    • Example for versions prior to R76: 

      [Expert@HostName]# cphaprob -a if

Required interfaces: 3
Required secured interfaces: 1

eth0       UP                    non sync(non secured), multicast
eth2       UP                    non sync(non secured), multicast
eth1       UP                    sync(secured), multicast

Virtual cluster interfaces: 2

eth0            192.168.204.20        VMAC address: 00:1C:7F:00:D0:FE
eth2            20.20.20.20        VMAC address: 00:1C:7F:00:D1:FE

    • Example for versions R76 and above (non-VSX): 

      [Expert@HostName]# cphaprob -a if

Required interfaces: 3
Required secured interfaces: 1

eth0       UP                    non sync(non secured), multicast
eth2       UP                    non sync(non secured), multicast
eth1       UP                    sync(secured), multicast

Virtual cluster interfaces: 2

eth0            192.168.204.20        VMAC address: 00:1C:7F:00:00:FE
eth2            20.20.20.20        VMAC address: 00:1C:7F:00:00:FE

  2. Run the following debug commands on all cluster members (in Expert shell):

    [Expert@HostName]# fw ctl zdebug -m cluster + conf > /var/log/data.txt &
    [Expert@HostName]# cphaconf debug_data
    [Expert@HostName]# fg
    press CTRL+C

    The /var/log/data.txt file contains useful cluster information.

    The 'VMAC mode' section in that file should look similar to the examples below (refer to 'VMAC value' section above):


    • Example for versions prior to R76: 
      ..............................
;[cpu_0];[fw_0];----   VMAC mode:   ----
;
;[cpu_0];[fw_0];VMAC: vmac mode is enabled;
;[cpu_0];[fw_0];VMAC: the vmac of each interface:;
;[cpu_0];[fw_0];Interface: 0) eth0, vmac: 00:1C:7F:00:D0:FE;
;[cpu_0];[fw_0];Interface: 1) eth2, vmac: 00:1C:7F:00:D1:FE;
;[cpu_0];[fw_0];VMAC: priomisc mode interfaces (by the VMAC mechanism) are:;
;[cpu_0];[fw_0];Interface: 0) eth0, vmac_index=0xd0;
;[cpu_0];[fw_0];Interface: 1) eth2, vmac_index=0xd1;
;[cpu_0];[fw_0];------------------------
..............................

    • Example for versions R76 and above (non-VSX): 
      ..............................
;[cpu_0];[fw_0];----   VMAC mode:   ----
;
;[cpu_0];[fw_0];VMAC: vmac mode is enabled;
;[cpu_0];[fw_0];VMAC: the vmac of each interface:;
;[cpu_0];[fw_0];Interface: 0) eth0, vmac: 00:1C:7F:00:00:FE;
;[cpu_0];[fw_0];Interface: 1) eth2, vmac: 00:1C:7F:00:00:FE;
;[cpu_0];[fw_0];------------------------
..............................
    The output shows 'vmac mode is enabled', and shows that a VMAC address is assigned to each interface.
  3. Make sure that the ARP tables of the hosts connected to these cluster members now contain the VMAC value.
    Use the relevant command for the Operating System on the hosts (e.g., Linux : # arp -an ; Windows: # arp -a)
  4. Capture traffic that arrived at the ClusterXL gateway and make sure it is destined (check the 'Destination MAC Address' field) to the cluster VMAC according to the incoming VIP.
Posted by at