Tuesday, March 26, 2019

Upgrade R80.10 to R80.20


Upgrading R80.10 to R80.20 is very easy and straight forward but there are some things to look out for ... Make sure you check the arp entry cache, it can full up and cause your dynamic routing such as OSPF to fail.  The default ARP configuration settings are different (lower) on the new R80.20 code the firewalls than in R80.10.

One key thing to note .. if SSL inspection is enable, upgrade from R80.10 to R80.20 .. The difference between base R80.10 and base R80.20 is that R80.10 failed open if the probe bypass mechanism failed while R80.20 fail-closes which causes a much more obvious impact to traffic.

This command will set probe bypass back to fail open on the fly. To fix the issue:
[Gateway@Expert]# fw ctl set int bypass_on_enhanced_ssl_inspection 1

In order to make the change permanent and survive reboots, run this command:

[Gateway@Expert]# echo "bypass_on_enhanced_ssl_inspection=1" >> $FWDIR/boot/modules/fwkern.conf

[Gateway@Expert]#  more /opt/CPsuite-R80.20/fw1/boot/modules/fwkern.conf
enhanced_ssl_inspection=1
bypass_on_enhanced_ssl_inspection=1

Traffic fails because the gateway has probe bypass enabled and the problematic sites require the SNI extension in our Client Hello packets. This has been a known limitation for a very long time.


There should be no impact to traffic functionally but it is technically an impact to security that naturally comes with a fail-open setting.

This should fix your issues but if not then the backup plan would be to discuss disabling probe bypass and making sure the required bypass rules based on IPs are in place.




The OLD default value for the gc_threshx were 256, 512 and 1024. Since R80.x Gaia, the default values are been increased to 512, 2048 and 4096. So in this case, the issue was Gaia OS didn't load the pre-configured value and instead it is using the defaults. 


The configuration were changed between SecurePlatform OS and Gaia OS. The way to keep the configuration persistence is to modify the value through the Gaia portal, because the Gaia Portal configuration overrides the configuration from /etc/sysctl.conf.

Based on additional information this can be used as an action plan:

1. Regain access to the Gaia portal, starting from the "Standby" member, and then repeat the following steps on "Active" member.

[Expert@HostName:0]# cpstop

(If the ARP is still effecting the connections, please disable the WIFI interface, and wait for 1 minute for ARP table to be cleaned)

You can disable the wifi interface by using the command in expert mode:

[Expert@HostName:0]# ifconfig ethx down

Or in clish mode:

set interface ethx state off

2. After stop the Check Point services, you should be able to access the Gaia Portal.

3. Once you login, move to "Network Management" -> "ARP" -> "ARP Table Settings" -> change "Maximum Entries" to 8192 -> click on "Apply".
(Please note, changes from here are surviving reboot).

4. Verify the values are been changed as follows:

[Expert@HostName:0]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh1
1024
[Expert@HostName:0]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh2
4096
[Expert@HostName:0]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh3
8192

[Expert@HostName:0]# dbget dbget ip:arp:cache_size

5. Once values are been confirmed, reboot the device.

[Expert@HostName:0]# reboot

6. Please run the command in step 4 again after reboot to verify the values are staying there. Then proceed with the same change on "Active" member.

[Please note, the mean purpose is to configure a value that suitable for you network. Keep an eye on /var/log/messages file. After a correct value was been set, you should not see the message "kernel: neighbour table overflow" again. If the value for 8192 didn't stop the message, you need to increase the value again to 2 x 8192 = 16384]

For any unwanted circumstance, the workaround to keep this value configured is to reset the value on fly (doesn't survive from reboot):

[Expert@HostName:0]# dbset ip:arp:cache_size 8192
(to confirm the change run:)
[Expert@HostName:0]# for FILE in $(ls -1 /proc/sys/net/ipv4/neigh/default/gc_thresh*) ; do echo "$FILE" contains: ; echo $(cat $FILE) ; done

*If the system kernel unable to load the correct value by completing above procedures, I would consider a problem with the system kernel after upgrade. I recommend reverting back to R80.10 if issue is persisting to keep you business running.




Active Member in Cluster active firewall
Arp Entry  is ***
Number of Routes  1006   


[Expert@myfw-fwa]# arp -an | wc -l
**
[Expert@myfw-fwa]# netstat -rn | wc -l
888

[Expert@myfw-fwa]



Check the threshold - min, soft max , hard max
[Expert@myfw-fwa]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh1
1024
[Expert@myfw-fwa] cat /proc/sys/net/ipv4/neigh/default/gc_thresh2
4096
[Expert@myfw-fwa]# cat /proc/sys/net/ipv4/neigh/default/gc_thresh3
8192
[Expert@myfw-fwa]