Troubleshooting - High CPU - Memory

 

1. Contents of spike detective folder /var/log/spike_detective/*
2. Screenshot of top -H
3. CPinfo file as per sk92739 - In order to extract the file please run the following command:
• # cpinfo -d -D -z -o /var/log/<gwname>.cpinfo
4. HCP file as per sk171436 - "HealthCheck Point (HCP) Release Updates" - In order to extract the file please run the following command:
5. # hcp -r all --include-wts yes

Reset - Multi-queue, dynamic balancing, and flow director to default values

 

 Please, perform the steps in this order:

1. Set the Multiqueue to auto first:
• # mq_mng --set-mode auto
2. Enable Dynamic balancing:
• # dynamic_balancing -o enable
• No need to change the coreXL instances in cpconfig since dynamic balancing will take care of that after its enabled.
3. Enable Flow Director:
• # ethtool -K eth1-01 ntuple on
4. Reboot member and repeat on the other firewall.

Firewall migrate export

fw tab -t userc_key -s

Is the Management Environment an MDS or SMS( Smart Center)?
This may need to be performed at the end business and close all sessions. 

If exporting a MDS. Take a mds backup
1. Login to Manager via ssh
2. Enter expert mode
3. mds_backup -b -d /var/tmp/mds_backup_SEPT062022
4. Also provide the MDS Management IP

If exporting a Smart Center:
How to Export the existing SmartCenter
1. Login to Manager via ssh
2. Enter expert mode
3. [Expert@MGMT]# cd $FWDIR/bin/upgrade_tools
4. [Expert@MGMT]# yes | nohup ./migrate export /<Full Path to export to>/<Name of Exported File>
5. Also provide the SMS Management IP

[Expert@bos0105fwm01:0]# cd $FWDIR/bin/upgrade_tools

[Expert@bos0105fwm01:0]# pwd

/opt/CPsuite-R81.20/fw1/bin/upgrade_tools

[Expert@bos0105fwm01:0]# 

yes | nohup ./migrate export /var/log/bos0105fwm01_Export-dk11042024

yes | nohup ./migrate export /var/log/M-TST-CPMGMT01_Export-dk11042024

Files uploaded to 

W:\sysadmin\Network\Firewall_Backups\Checkpoint_Firewall_Manager_Migrate_Export

# cpstop
# cd $FWDIR/bin/upgrade_tools/
# ./migrate export /var/log/migrate-exports/myfwm01-DATE

Transfer off box 



Database 
How do you Data Duplexing
indexing 
purging  

OSPF Route Map and Redistribution

Each protocol with Routemap support allows configuration of Import Routemaps and Export Routemaps.

A protocol's Import Routemaps govern, which routes will be imported into the routing table from that protocol.

A protocol's Export Routemaps govern, which routes from other routing protocols will be sent out with that protocol.

Within a routemap, the same procedure applies when checking individual routemap IDs: the IDs are checked from lowest to highest until a match is found. Therefore, more specific match conditions should have lower IDs or should be present in lower preference routemaps so that they are checked first, instead of being ignored in favor of less specific match conditions.

Configuring IPv4 OSPFv2 Router ID

• Do not use the IP address 0.0.0.0 as the Router ID.

• In a cluster, you must configure the Router ID to one of the Cluster Virtual IP addresses.

  In a Cluster, you must configure all the Cluster Members in the same way.

Configuring IPv4 OSPFv2 in Gaia Portal

• In a Cluster, you must configure all the Cluster Members in the same way.

• Start the OSPF configuration from Router ID (see Configuring IPv4 OSPFv2 Router ID).

• Gaia Clish does not have commands for route filtering and redistribution. You must configure inbound routing policies and redistribution of routes through the Gaia Portal.

  You can configure route maps and route aggregation using Gaia Clish commands. Route map configuration done through the Gaia Clish takes precedence over route filtering and redistribution configured in the Gaia Portal. For example, if OSPF uses route maps for inbound filtering, anything configured in the Gaia Portal for inbound route filters for OSPF is ignored. You can still use the Gaia Portal to configure route redistribution into OSPF.

set interface eth1-01 comments "Inside OSPF (Area 4 - Regular)" 
set interface eth1-01 state on 
set interface eth1-01 auto-negotiation on 
set interface eth1-01 mtu 1500 
set interface eth1-01 ipv4-address 100.14.255.29 mask-length 29 

set router-id 100.14.255.28

OSPF  

set ospf instance default area backbone on
set ospf instance default area 0.0.0.4 on
set ospf instance default interface eth1-01 area 0.0.0.4 on
set ospf instance default interface eth1-01 priority 1

set ospf instance default interface eth1-01 authtype cryptographic key 1 algorithm md5 secret already_scrambled_Bm4JO9gDBWc=_00000000000000000000000000000000000000000000000000

set ospf instance default export-routemap static-to-ospf preference 4 on

set inbound-route-filter ospf2 instance default accept-all-ipv4
set inbound-route-filter rip accept-all-ipv4

ROUTE MAP

set routemap static-to-ospf id 4 on
set routemap static-to-ospf id 4 allow
set routemap static-to-ospf id 4 match network 10.114.32.0/19 all
set routemap static-to-ospf id 4 match protocol static

## The following items are listed under their respective command sets
## (e.g. "set bgp") and are displayed here for informational purposes:
# set ospf instance default export-routemap static-to-ospf preference 4 on

Implicit Rule

In every Security policy, the 1st layer must have its implicit "cleanup rule" set to "drop" and for the 2nd ordered layer the "implicit cleanup" rule must be set to "accept". 

These are the defaults when creating policies & layers, Every layer has the "implicit cleanup rule" in its properties. 

Implied rules you can modify implied_rules.def-file on the management-server. 

Implied rules are "attached" during install policy, to the relevant context. The implied rules that are selected to appear "first", are added to the first ordered layer in the policy. 

The implied rules that are selected to appear "before last" or "last", are added to all the layers.

tcpdump -

Capture any traffic -  on Interface eth1-01

Expert@MY-VPN0]# tcpdump -enni eth1-01 -s 0 -w traffic.pcap

tcpdump: listening on eth1-01, link-type EN10MB (Ethernet), capture size 262144 bytes

Firewall Build & Validation

 Validate Firewall Functionality

Build Out Quality Control

☐ Hostname

☐ DNS

☐ NTP

☐ Login Banner

☐ TACACS - Login

☐ Domain name

☐ Domain Prefix

☐ show asset all

Software and Jumbo hotfix

☐ fw ver

☐ cpinfo -y all

☐ fwunload local

☐ fw stat

VALIDATES

☐ Version         fw ver

☐ License         cplic print      cat $CPDIR/conf/cp.license

☐ Routes          netstat -nr | wc -l

☐ Arp             fw ctl arp

☐ Connections     fw tab -t connections -s

Before Establish SIC on Management ensure that your gateway CDP process is running cpwd_admin list

.The Gateway must  have CPD running in E Stat .. to validate it, you can run cpwd_admin list  if it is a T stat, you will not be able to establish SIC

[Expert@MY-VPN-FW01:0]# cpwd_admin list

APP        PID    STAT  #START  START_TIME             MON  COMMAND             
FWK_FORKER 73879  E     1       [21:37:50] 14/4/2023   N    fwk_forker          
FWK_WD     73888  E     1       [21:37:50] 14/4/2023   N    fwk_wd -i 43 -i6 0  
CPVIEWD    74765  E     1       [21:38:08] 14/4/2023   N    cpviewd             
CPVIEWS    74782  E     1       [21:38:08] 14/4/2023   N    cpview_services     
CVIEWAPIS  74787  E     1       [21:38:08] 14/4/2023   N    cpview_api_service  
SXL_STATD  74792  E     1       [21:38:08] 14/4/2023   N    sxl_statd           
CPD        74804  E     1       [21:38:08] 14/4/2023   Y    cpd                 
MPDAEMON   74816  E     1       [21:38:08] 14/4/2023   N    mpdaemon /opt/CPshrd-R81.10/log/mpdaemon.elg /opt/CPshrd-R81.10/conf/mpdaemon.conf
TP_CONF_SERVICE 230716 E     1       [00:24:39] 15/4/2023   N    tp_conf_service --conf=tp_conf.json --log=error
CXLD       75062  E     1       [21:38:10] 14/4/2023   N    cxld -d             
CI_CLEANUP 75078  E     1       [21:38:10] 14/4/2023   N    avi_del_tmp_files   
CIHS       75081  E     1       [21:38:10] 14/4/2023   N    ci_http_server -j -f /opt/CPsuite-R81.10/fw1/conf/cihs.conf
FWD        75105  E     1       [21:38:10] 14/4/2023   N    fwd                 
SPIKE_DETECTIVE 75120  E     1       [21:38:10] 14/4/2023   N    spike_detective     
DSDAEMON   158764 E     1       [01:32:16] 15/4/2023   Y    dsd                 
DASERVICE  100901 E     1       [21:39:35] 14/4/2023   N    DAService_script    
AUTOUPDATER 100918 E     1       [21:39:35] 14/4/2023   N    AutoUpdaterService.sh
CPHAMCSET  124212 E     1       [21:43:27] 14/4/2023   N    cphamcset -d        
WSDNSD     40975  E     1       [00:47:51] 15/4/2023   Y    wsdnsd              
RAD        125442 E     1       [21:43:30] 14/4/2023   N    rad                 
RTMD       125479 E     1       [21:43:31] 14/4/2023   N    rtmd                
LPD        15444  E     1       [04:34:46] 15/4/2023   N    lpd                 
[Expert@MY-VPN-FW01:0]# 

dmidecode

dmiparse

/bin/log_start limit 0 2097152 10

3rd Party Monitoring

☐ Solarwinds

☐ TACACS authentication

☐ WebUI Login

☐ Serial Console Raritan

☐ Access via SSH/HTTPS

Interface  (Subnet Mask /speed/duplex)

☐ Ifconfig
☐ ifconfig -a
☐ Show configuration interface
☐ netstat -i
☐ cat /proc/net/bonding/bond0
☐ Kernel Version [Expert@MyFW:0]# cat /proc/version
Linux version 3.10.0-957.21.3cpx86_64 (builder@8700486_0_Docker) (gcc version 4.9.2 (GCC) ) #1 SMP Mon Feb 20 16:46:42 IST 2023
[Expert@MyFW:0]# 

Cluster XL (High Availability)

☐ cpstop

☐ cpstart

☐ cphastop

☐ cphastart

☐ clusterXL_admin up/down

☐ cphaprob –a if

☐ cphaprob list

☐ cphaprob stat

☐ cpstat ha -f all

☐ cphaprob syncstat

☐ cphaprob list

☐ cpwd_admin list

ClusterXL Functioning 

☐ cphaprob stat

☐ cphaprob –a if

☐ cphaprob list

☐ cpwd_admin list

cpconfig

Route

route -n

netstat -nr | wc -l

netstat -i

netstat -nr | grep -v D

netstat -rn |grep eth1 | awk -F' ' ' {print $1, $2, $3}' | sort >routes.txt

netstat -rn |grep eth1 | awk -F' ' ' {print $1, $2, $3}' | sort | wc -l

Unload Local Firewall Policy     fwunload local

Identity Awareness

pep show user all

pep show user all | wc -l

pep show

pep show user

pep show user query shyam

pep show user query usr d

pdp show

pdp show connections

pdp connections

pdp connections pep

Performance

top

ps auxwww

fw tab -t connections -s

fw ctl pstat

fwaccel stats  Usage: fwaccel on|off | ver|stat |conns| dbg <...>

fwaccel conns

fwaccel conns -s

fw ctl multik stat

fw ctl affinity -l -a -v  check CPU core to NIC Mapping can be change in)

fw ctl multik dynamic_dispatching get_mode

fw ctl multik dynamic_dispatching on

fw ctl multik get_mode

9. Validate Sync is ESTABLISHED netstat -an | grep 2010

10. Validate Logs are Flowing to Logger and observe it for any errors

11. Test Cluster-XL HA Sync Failover by Rebooting the Primary Firewall and validate if traffic goes to Secondary without interrupting the Ping    Test to

12. Reboot Secondary and make sure it came back into the cluster. Firewall overall health Checks

13. validate firewall is not dropping any packets fw ctl zdebug + drop

14. uptime

15. fw ver

df

fw ctl pstat

cat /etc/sysconfig/ntp

netstat –i

ethtool –i eth0  (please enter all active interfaces – this will let us know what version of NIC driver is running) vmstat 1 10 free ps auxwww

vmstat 1 10

free

Troubleshooting

if you build a new gateway and wants to test connectivity across the interfaces and the ping test does not work, do the following:

[Expert@myfw-fwa:#  fw unloadlocal

[Expert@myfw-fwa:#  cpstop

[Expert@myfw-fwa:#  sbin/sysctl -w net.ipv4.ip_forward=1

fw monitor | grep 100.210.7.250

fw ctl zdebug + drop > text.drops

fw ctl zdebug + drop | grep 200.105.57.69

tcpdump -ni eth8 src 172.60.25.132

tcpdump -i eth1 port 1089 and dst 215.118.184.254

netstat -rn |grep 204.105

What happening to the traffic.

From expert mode on the Active Firewall:

fwaccel off Turn off SecureXL, if enabled)

df -h   (Check for sufficient diskspace capture and debug_

fw monitor -o /var/log/fwmon.cap   (In one session: Run the capture.)

fw ctl zdebug drop > /var/log/drop.txt  (In another session: Run the kernel debug for drops.)

tcpdump -nnei any -w /var/log/tcp.cap (In a third session: Run a tcpdump capture.)

Re-create the problem.

Control-C   (End the fw monitor, tcpdump and the kernel debug with the following:)

fwaccel on  (Turn on SecureXL, if you disabled it)

----------------------------------------------

Checkpoint Special Config Files

----------------------------------------------

1. fwkern.conf - $FWDIR/boot/modules/fwkern.conf Magic Mac

2. local.arp   - $FWDIR/conf/local.arp   GAiA manual ARP

3. sdconf.rec  -  /var/ace  RAS authentication

4. rc.local    -  /etc/rc.d/rc.local

5. netconf.C      (/etc/sysconfig) Network interfaces/Routes

6. external.if    (/etc/sysconfig)

7. ifcfg-eth1      (/etc/sysconfig/network-scripts/)

----------------------------------------------

checkpoint scripts:

----------------------------------------------

1. checkup.sh

2. cpsizeme

3. nohup mpstat -P ALL 1 86400 > mpstat.out &

Checkpoint Health Checks -Commands

uptime

ver

fw ver

cpinfo -y all

cplic print

expert

df -h

reboot

shutdown

fwunload local

 

Firewall Performance

top

ps auxwww

fw tab -t connections -s

fw ctl pstat

fwaccel stats

tcpdump -i eth0 src 100.25.240.57 and dst 26.23.64.82

Verification:

cat /etc/sysconfig/ntp

vmstat 1 10

cat /proc/meminfo

cpstat os -f cpu

cpstat os -f memory

Interface Configurations

netstat -i

ifconig -a

netstat -rn

ethtool –i eth0

ethtool -S eth1-01

ethtool -S eth1-02

cpview

fw tab -t userc_key -s

fw tab -t userc_users -s

fw tab -t om_assigned_ips -s  (verify # of Seed license)

Performance -cpconfig utility enable/disable Checkpoint SecureXL

fwaccel stats  (Usage: fwaccel on | off | ver | stat | conns | dbg <...> | help

fwaccel conns

fwaccel conns -s

fw ctl multik stat

fw ctl affinity -l -a -v

gaia> fwaccel conns -s  (checks SecureXL Connections)

gaia> fw ctl get int fwx_max_conns (checks maximum connectins 0 means it set to auto/unlimited

[Expert@fw:0]# mq_mng --show
Total 48 cores. Available for MQ 7 cores
i/f             driver          driver mode     state           mode (queues)   cores          
                                                                actual/avail                   
------------------------------------------------------------------------------------------------
Mgmt            igb             Kernel          Up              Auto (7/7)      0,24,1,25,2,26,3
Sync            igb             Kernel          Up              Auto (7/7)      0,24,1,25,2,26,3
eth1-01         i40e            Kernel          Up              Auto (7/7)      0,24,1,25,2,26,3
eth1-04         i40e            Kernel          Up              Auto (7/7)      0,24,1,25,2,26,3
 
[Expert@myfww#

[Expert@myfww## fw ctl affinity -l -r
CPU 0:
CPU 1:
CPU 2:
CPU 3:
CPU 4:  fw_38 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 5:  fw_36 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 6:  fw_34 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 7:  fw_32 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 8:  fw_30 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 9:  fw_28 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 10: fw_26 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 11: fw_24 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 12: fw_22 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 13: fw_20 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 14: fw_18 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 15: fw_16 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 16: fw_14 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 17: fw_12 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 18: fw_10 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 19: fw_8 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 20: fw_6 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 21: fw_4 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 22: fw_2 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 23: fw_0 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 24:
CPU 25:
CPU 26:
CPU 27: fw_39 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 28: fw_37 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 29: fw_35 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 30: fw_33 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 31: fw_31 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 32: fw_29 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 33: fw_27 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 34: fw_25 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 35: fw_23 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 36: fw_21 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 37: fw_19 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 38: fw_17 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 39: fw_15 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 40: fw_13 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 41: fw_11 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 42: fw_9 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 43: fw_7 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 44: fw_5 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 45: fw_3 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 46: fw_1 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 47: cprid lpd mpdaemon fwd in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
All:
Interface Mgmt: has multi queue enabled
Interface Sync: has multi queue enabled
Interface eth1-01: has multi queue enabled
Interface eth1-04: has multi queue enabled

[[Expert@myfww#

[Expert@my-fw:0]# dynamic_balancing -o disable
Disabling Dynamic Balancing, please wait for the operation to complete
Successfully disabled Dynamic Balancing
 
Dynamic Balancing made changes that require a reboot, please reboot your machine in order for the changes to take effect

Expert@my-fw:0]# cpconfig
This program will let you re-configure
your Check Point products configuration.
 
 
Configuration Options:
----------------------
(1)  Licenses and contracts
(2)  SNMP Extension
(3)  PKCS#11 Token
(4)  Random Pool
(5)  Secure Internal Communication
(6)  Disable cluster membership for this gateway
(7)  Enable Check Point Per Virtual System State
(8)  Enable Check Point ClusterXL for Bridge Active/Standby
(9)  Hyper-Threading
(10) Check Point CoreXL
(11) Automatic start of Check Point Products
 
(12) Exit
 
Enter your choice (1-12) :10
 
 
 
Configuring Check Point CoreXL...
=================================
 
 
CoreXL is currently enabled with 43 IPv4 firewall instances.
 
(1) Change the number of firewall instances
(2) Disable Check Point CoreXL
(3) Change firewall mode
 
(4) Exit
Enter your choice (1-4) : 1
 
This machine has 48 CPUs.
 
Note: All cluster members must have the same number of firewall instances
enabled.
 
How many IPv4 firewall instances would you like to enable (2 to 48) [43] ? 40
 
CoreXL was enabled successfully with 40 firewall instances.
Important: This change will take effect after reboot.

 

 

[Expert@myfwe-int02:0]# fw ctl multik stat  (connection to Core Distribution)

ID | Active  | CPU    | Connections | Peak

----------------------------------------------

 0 | Yes     | 11     |         178 |      303

 1 | Yes     | 10     |         203 |      380

 2 | Yes     | 9      |         168 |      262

 3 | Yes     | 8      |         179 |      188

 4 | Yes     | 7      |         149 |      278

 5 | Yes     | 6      |         113 |      194

 6 | Yes     | 5      |         128 |      221

 7 | Yes     | 4      |         282 |      387

 8 | Yes     | 3      |         186 |      292

 9 | Yes     | 2      |         296 |      439

[Expert@myfwe-int02:0]#

[Expert@myfwe-int02:0]# fw ctl affinity -l -a -v    (check CPU core to NIC Mapping -can be change in $FWDIR/conf/fwaffinity/conf)

Interface eth1-05 (irq 218): CPU 1

Interface Sync (irq 124): CPU 1

Interface eth1-01 (irq 107): CPU 1

Interface eth1-02 (irq 123): CPU 0

Interface eth3-01 (irq 171): CPU 0

Kernel fw_0: CPU 11

Kernel fw_1: CPU 10

Kernel fw_2: CPU 9

Kernel fw_3: CPU 8

Kernel fw_4: CPU 7

Kernel fw_5: CPU 6

Kernel fw_6: CPU 5

Kernel fw_7: CPU 4

Kernel fw_8: CPU 3

Kernel fw_9: CPU 2

Daemon in.geod: CPU all

Daemon vpnd: CPU all

Daemon in.acapd: CPU all

Daemon in.asessiond: CPU all

Daemon in.msd: CPU all

Daemon rad: CPU all

Daemon rtmd: CPU all

Daemon fwd: CPU all

Daemon mpdaemon: CPU all

Daemon usrchkd: CPU all

Daemon cpd: CPU all

Daemon cprid: CPU all

[Expert@myfwe-int02:0]#

[Expert@myfwe-int02:0]# fwaccel stat

Accelerator Status : on

Accept Templates   : disabled by Firewall

                     disabled from rule #35

Drop Templates     : disabled

NAT Templates      : disabled by user

Accelerator Features : Accounting, NAT, Cryptography, Routing,

                       HasClock, Templates, Synchronous, IdleDetection,

                       Sequencing, TcpStateDetect, AutoExpire,

                       DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,

                       WireMode, DropTemplates, NatTemplates,

                       Streaming, MultiFW, AntiSpoofing, ViolationStats,

                       Nac, AsychronicNotif, ERDOS, McastRoutingV2

Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,

                        3DES, DES, CAST, CAST-40, AES-128, AES-256,

                        ESP, LinkSelection, DynamicVPN, NatTraversal,

                        EncRouting, AES-XCBC, SHA256

[Expert@hinfwe-int02:0]#

[Expert@myfwe-int02:0]# fwaccel conns  |grep  26.31.83.28 | more

Source          SPort Destination     DPort PR Flags       C2S i/f S2C i/f Inst Identity

--------------- ----- --------------- ----- -- ----------- ------- ------- ---- --------

 26.31.83.28    53   74.94.152.161  1580 17 F..A...S... 7/8     8/7      7        0

   66.189.0.104 21318  26.31.83.28    53 17 ...A...S... 7/8     8/7      7        0

 

Usage: fwaccel on | off | ver | stat | cfg <...> | conns | dbg <...> | help

Options:

on-turns acceleration on

off-turns acceleration off

ver-show acceleration/FW version

stat-show acceleration status

cfg-configure acceleration parameters

stats-print the acceleration statistics

conns-print the accelerator's connection table

templates-print the accelerator's templates table

dbg-set debug flags

help-this help messages

diag-policy diagnostics

----------------------------------------------

Troubleshooting

----------------------------------------------

fw monitor | grep 10.210.7.250

fw ctl zdebug + drop > text.drops

fw ctl zdebug + drop | grep 204.105.57.69

tcpdump -ni eth8 src 172.30.25.132

tcpdump -i eth1 port 1089 and dst 216.118.184.254

netstat -rn |grep 204.105

RE: Traffic failing between internet  Clusters

Run a packet capture and a kernel debug on the firewall so I can get a packet-level look at what is happening to the traffic.

From expert mode on the Active Firewall:

1. # fwaccel off  (Turn off SecureXL, if enabled)

2. # df -h (Check your disk space to make sure you have sufficient space to run a capture and debug_

3. # fw monitor -o /var/log/fwmon.cap   (In one session: Run the capture.)

4. # fw ctl zdebug drop > /var/log/drop.txt  (In another session: Run the kernel debug for drops.)

5. # tcpdump -nnei any -w /var/log/tcp.cap (In a third session: Run a tcpdump capture.)

6. Re-create the problem.

7. Control-C   (End the fw monitor, tcpdump and the kernel debug with the following:)

8. # fwaccel on  (Turn on SecureXL, if you disabled it)

----------------------------------------------

log files

----------------------------------------------

$FWDIR/log/vpnd.elg

/var/log/messages

dmesg

/var/log/routed.log

----------------------------------------------

How to Set Specific

----------------------------------------------

clusterXL_admin down/up

ethtool -s eth1  autoneg on

--------------------------------------

/etc/resolv.conf    # What's the name resolution config -- sometimes performance is adversely influenced by bad DNS settings

/etc/ntpd.conf      # Time config

/etc/ntp.conf

/etc/modprobe.conf  # Any NIC or kernel tweaks?

/etc/sysctl.conf    # Any kernel tweaks?

/etc/ssh/sshd_config # Any hacks to sshd?

/etc/issue           # console banner file

/etc/issue.net       # network banner file

/etc/motd            # message of the day file

/etc/grub.conf       # Grub config -- important to see vmalloc

/etc/gated.ami       # gated config file

/etc/gated_xl.ami    # gated config file

/etc/rc.d/rc.local   # local RC files -- any changes here (such as kernel tweaks)?

$FWDIR/boot/boot.conf              # Firewall boot params

$FWDIR/boot/modules/fwkern.conf    # Any firewall kernel tweaks?

$PPKDIR/boot/modules/simkern.conf  # Any SIM tweaks?

$FWDIR/conf/discntd.if             # ClusterXL Disconnected Interfaces

$FWDIR/conf/local.arp              # SPLAT / GAiA manual ARP

$FWDIR/conf/vsaffinity_exception.conf      # Relevant to R75.40VS and later Virtual systems only

$MDSDIR/conf/external.if                   # Relevant to P1 / MDSM only

ARPING

[myinet-fwa]# fw ctl arp

 (26.18.190.123) at 00-1c-7f-3f-6c-fd

 (26.18.190.100) at 00-1c-7f-3f-6c-fd

[myinet-fwa]# arping -I eth3-04 26.18.190.88 (check for Arping on specific interface for an IP)

ARPING 216.118.190.88 from 26.18.190.7 eth3-04

Sent 7 probes (7 broadcast(s))

Received 0 response(s)

[myinet-fwa]# arping -I eth3-04 26.18.190.87

ARPING 26.18.190.87 from 26.18.190.7 eth3-04

Sent 9 probes (9 broadcast(s))

Received 0 response(s)

[myinet-fwa]# arping -I eth3-04 26.18.190.89

ARPING 26.18.190.89 from 26.18.190.7 eth3-04

Sent 14 probes (14 broadcast(s))

Received 0 response(s)

[myinet-fwa]

[Expert@mydev-fwa]# cphaprob stat

Cluster Mode:   New High Availability (Active Up)

Number     Unique Address  Assigned Load   State

1 (local)  192.168.42.1    100%            Active

2          192.168.42.2    0%              Standby

[Expert@mydev-fwa]#

[Expert@mydev-fwa]# cphaprob -a if

Required interfaces: 6

Required secured interfaces: 1

eth0       UP                    non sync(non secured), multicast

eth1       UP                    non sync(non secured), multicast

eth2       UP                    non sync(non secured), multicast

eth3       UP                    non sync(non secured), multicast

eth4       UP                    non sync(non secured), multicast

eth5       UP                    sync(secured), multicast

Virtual cluster interfaces: 5

eth0            172.30.25.54

eth1            10.125.240.4

eth2            10.125.242.4

eth3            10.125.244.4

eth4            10.125.246.4

[Expert@mydev-fwa]#

[Expert@mydev-fwa]# cphaprob list

Built-in Devices:

Device Name: Interface Active Check

Current state: OK

Registered Devices:

Device Name: Synchronization

Registration number: 0

Timeout: none

Current state: OK

Time since last report: 388872 sec

Device Name: Filter

Registration number: 1

Timeout: none

Current state: OK

Time since last report: 388866 sec

Device Name: cphad

Registration number: 2

Timeout: 2 sec

Current state: OK

Time since last report: 0.9 sec

Device Name: fwd

Registration number: 3

Timeout: 2 sec

Current state: OK

Time since last report: 0.9 sec

Expert@mydev-fwa]#

[Expert@mydev-fwa]# cpwd_admin list

cpwd_admin:

APP        PID    STAT  #START  START_TIME             COMMAND              MON

CPD        3449   E     1       [20:24:21] 7/6/2013    cpd                  Y

CI_CLEANUP 3534   E     1       [20:24:35] 7/6/2013    avi_del_tmp_files    N

CIHS       3546   E     1       [20:24:35] 7/6/2013    ci_http_server -j -f /opt/CPsuite-R71/fw1/conf/cihs.conf N

FWD        3548   E     1       [20:24:36] 7/6/2013    fwd                  N

RTMD       4051   E     1       [20:24:59] 7/6/2013    rtmd                 N

[Expert@mydev-fwa]#

cphaprob list. There reported issues with fib with problem state. Next ran cpwd_admin list and noticed that FIBMGR and DROUTER restarted 2x's.

[Expert@myvpn-fwb]# clusterXL_admin down

Setting member to administratively down state ...

Member current state is Down

[Expert@myvpn-fwb]# cphaprob stat

Cluster Mode:   New High Availability (Active Up)

Number     Unique Address  Assigned Load   State

1          192.168.25.241  100%            Active

2 (local)  192.168.25.242  0%              Down

[Expert@myvpn-fwb]# clusterXL_admin up

Setting member to normal operation ...

Member current state is Standby

[Expert@myvpn-fwb]#