Wednesday, December 6, 2023

Troubleshooting - High CPU - Memory

 

  1. Contents of spike detective folder /var/log/spike_detective/*
  2. Screenshot of top -H
  3. CPinfo file as per sk92739 - In order to extract the file please run the following command:
    • # cpinfo -d -D -z -o /var/log/<gwname>.cpinfo
  4. HCP file as per sk171436 - "HealthCheck Point (HCP) Release Updates" - In order to extract the file please run the following command:
  5. # hcp -r all --include-wts yes

Wednesday, November 8, 2023

Reset - Multi-queue, dynamic balancing, and flow director to default values

 

 Please, perform the steps in this order:

  1. Set the Multiqueue to auto first:
    • # mq_mng --set-mode auto
  2. Enable Dynamic balancing:
    • # dynamic_balancing -o enable
    • No need to change the coreXL instances in cpconfig since dynamic balancing will take care of that after its enabled.
  3. Enable Flow Director:
    • # ethtool -K eth1-01 ntuple on
  4. Reboot member and repeat on the other firewall.

Monday, November 6, 2023

Firewall migrate export


fw tab -t userc_key -s

Is the Management Environment an MDS or SMS( Smart Center)?
This may need to be performed at the end business and close all sessions. 

If exporting a MDS. Take a mds backup
1. Login to Manager via ssh
2. Enter expert mode
3. mds_backup -b -d /var/tmp/mds_backup_SEPT062022
4. Also provide the MDS Management IP

If exporting a Smart Center:
How to Export the existing SmartCenter
1. Login to Manager via ssh
2. Enter expert mode
3. [Expert@MGMT]# cd $FWDIR/bin/upgrade_tools
4. [Expert@MGMT]# yes | nohup ./migrate export /<Full Path to export to>/<Name of Exported File>
5. Also provide the SMS Management IP





# cpstop
# cd $FWDIR/bin/upgrade_tools/
# ./migrate export /var/log/migrate-exports/myfwm01-DATE

Transfer off box 



Database 
How do you Data Duplexing
indexing 
purging  


Monday, October 16, 2023

OSPF Route Map and Redistribution

Each protocol with Routemap support allows configuration of Import Routemaps and Export Routemaps.

A protocol's Import Routemaps govern, which routes will be imported into the routing table from that protocol.

A protocol's Export Routemaps govern, which routes from other routing protocols will be sent out with that protocol.

Within a routemap, the same procedure applies when checking individual routemap IDs: the IDs are checked from lowest to highest until a match is found. Therefore, more specific match conditions should have lower IDs or should be present in lower preference routemaps so that they are checked first, instead of being ignored in favor of less specific match conditions.

Configuring IPv4 OSPFv2 Router ID

  • Do not use the IP address 0.0.0.0 as the Router ID.

  • In a cluster, you must configure the Router ID to one of the Cluster Virtual IP addresses.

    In a Cluster, you must configure all the Cluster Members in the same way.


Configuring IPv4 OSPFv2 in Gaia Portal

  • In a ClusterClosed, you must configure all the Cluster Members in the same way.

  • Start the OSPF configuration from Router ID (see Configuring IPv4 OSPFv2 Router ID).

  • Gaia ClishClosed does not have commands for route filtering and redistribution. You must configure inbound routing policies and redistribution of routes through the Gaia PortalClosed.

    You can configure route maps and route aggregation using GaiaClosed Clish commands. Route map configuration done through the Gaia Clish takes precedence over route filtering and redistribution configured in the Gaia Portal. For example, if OSPF uses route maps for inbound filtering, anything configured in the Gaia Portal for inbound route filters for OSPF is ignored. You can still use the Gaia Portal to configure route redistribution into OSPF.


set interface eth1-01 comments "Inside OSPF (Area 4 - Regular)" 
set interface eth1-01 state on 
set interface eth1-01 auto-negotiation on 
set interface eth1-01 mtu 1500 
set interface eth1-01 ipv4-address 100.14.255.29 mask-length 29 

set router-id 100.14.255.28


OSPF  

set ospf instance default area backbone on
set ospf instance default area 0.0.0.4 on
set ospf instance default interface eth1-01 area 0.0.0.4 on
set ospf instance default interface eth1-01 priority 1

set ospf instance default interface eth1-01 authtype cryptographic key 1 algorithm md5 secret already_scrambled_Bm4JO9gDBWc=_00000000000000000000000000000000000000000000000000


set ospf instance default export-routemap static-to-ospf preference 4 on


set inbound-route-filter ospf2 instance default accept-all-ipv4
set inbound-route-filter rip accept-all-ipv4


ROUTE MAP

set routemap static-to-ospf id 4 on
set routemap static-to-ospf id 4 allow
set routemap static-to-ospf id 4 match network 10.114.32.0/19 all
set routemap static-to-ospf id 4 match protocol static

## The following items are listed under their respective command sets
## (e.g. "set bgp") and are displayed here for informational purposes:
# set ospf instance default export-routemap static-to-ospf preference 4 on


Sunday, October 15, 2023

Implicit Rule

In every Security policy, the 1st layer must have its implicit "cleanup rule" set to "drop" and for the 2nd ordered layer the "implicit cleanup" rule must be set to "accept". 

These are the defaults when creating policies & layers, Every layer has the "implicit cleanup rule" in its properties. 

Implied rules you can modify implied_rules.def-file on the management-server. 

Implied rules are "attached" during install policy, to the relevant context. The implied rules that are selected to appear "first", are added to the first ordered layer in the policy. 

The implied rules that are selected to appear "before last" or "last", are added to all the layers.

Friday, September 15, 2023

tcpdump -

Capture any traffic -  on Interface eth1-01

Expert@MY-VPN0]# tcpdump -enni eth1-01 -s 0 -w traffic.pcap

tcpdump: listening on eth1-01, link-type EN10MB (Ethernet), capture size 262144 bytes


Thursday, August 31, 2023

Firewall Build & Validation

 Validate Firewall Functionality


Build Out Quality Control
☐ Hostname
 DNS
 NTP
 Login Banner
 TACACS - Login
 Domain name
 Domain Prefix
 show asset all

Software and Jumbo hotfix
 fw ver
 cpinfo -y all
☐ fwunload local
☐ fw stat

VALIDATES
 Version         fw ver
 License         cplic print      cat $CPDIR/conf/cp.license
 Routes          netstat -nr | wc -l
 Arp             fw ctl arp
 Connections     fw tab -t connections -s

Before Establish SIC on Management ensure that your gateway CDP process is running cpwd_admin list

.The Gateway must  have CPD running in E Stat .. to validate it, you can run cpwd_admin list  if it is a T stat, you will not be able to establish SIC

[Expert@MY-VPN-FW01:0]# cpwd_admin list

APP        PID    STAT  #START  START_TIME             MON  COMMAND             
FWK_FORKER 73879  E     1       [21:37:50] 14/4/2023   N    fwk_forker          
FWK_WD     73888  E     1       [21:37:50] 14/4/2023   N    fwk_wd -i 43 -i6 0  
CPVIEWD    74765  E     1       [21:38:08] 14/4/2023   N    cpviewd             
CPVIEWS    74782  E     1       [21:38:08] 14/4/2023   N    cpview_services     
CVIEWAPIS  74787  E     1       [21:38:08] 14/4/2023   N    cpview_api_service  
SXL_STATD  74792  E     1       [21:38:08] 14/4/2023   N    sxl_statd           
CPD        74804  E     1       [21:38:08] 14/4/2023   Y    cpd                 
MPDAEMON   74816  E     1       [21:38:08] 14/4/2023   N    mpdaemon /opt/CPshrd-R81.10/log/mpdaemon.elg /opt/CPshrd-R81.10/conf/mpdaemon.conf
TP_CONF_SERVICE 230716 E     1       [00:24:39] 15/4/2023   N    tp_conf_service --conf=tp_conf.json --log=error
CXLD       75062  E     1       [21:38:10] 14/4/2023   N    cxld -d             
CI_CLEANUP 75078  E     1       [21:38:10] 14/4/2023   N    avi_del_tmp_files   
CIHS       75081  E     1       [21:38:10] 14/4/2023   N    ci_http_server -j -f /opt/CPsuite-R81.10/fw1/conf/cihs.conf
FWD        75105  E     1       [21:38:10] 14/4/2023   N    fwd                 
SPIKE_DETECTIVE 75120  E     1       [21:38:10] 14/4/2023   N    spike_detective     
DSDAEMON   158764 E     1       [01:32:16] 15/4/2023   Y    dsd                 
DASERVICE  100901 E     1       [21:39:35] 14/4/2023   N    DAService_script    
AUTOUPDATER 100918 E     1       [21:39:35] 14/4/2023   N    AutoUpdaterService.sh
CPHAMCSET  124212 E     1       [21:43:27] 14/4/2023   N    cphamcset -d        
WSDNSD     40975  E     1       [00:47:51] 15/4/2023   Y    wsdnsd              
RAD        125442 E     1       [21:43:30] 14/4/2023   N    rad                 
RTMD       125479 E     1       [21:43:31] 14/4/2023   N    rtmd                
LPD        15444  E     1       [04:34:46] 15/4/2023   N    lpd                 
[Expert@MY-VPN-FW01:0]# 


dmidecode
dmiparse
/bin/log_start limit 0 2097152 10



3rd Party Monitoring
 Solarwinds
 TACACS authentication
 WebUI Login
 Serial Console Raritan
 Access via SSH/HTTPS

Interface  (Subnet Mask /speed/duplex)
 Ifconfig
 ifconfig -a
 Show configuration interface
 netstat -i
 cat /proc/net/bonding/bond0
 Kernel Version [Expert@MyFW:0]# cat /proc/version
Linux version 3.10.0-957.21.3cpx86_64 (builder@8700486_0_Docker) (gcc version 4.9.2 (GCC) ) #1 SMP Mon Feb 20 16:46:42 IST 2023
[Expert@MyFW:0]# 

Cluster XL (High Availability)
 cpstop
 cpstart
 cphastop
 cphastart
 clusterXL_admin up/down
 cphaprob –a if
 cphaprob list
 cphaprob stat
 cpstat ha -f all
 cphaprob syncstat
 cphaprob list
 cpwd_admin list

ClusterXL Functioning 
 cphaprob stat
 cphaprob –a if
 cphaprob list
 cpwd_admin list

cpconfig

Route
route -n
netstat -nr | wc -l
netstat -i
netstat -nr | grep -v D
netstat -rn |grep eth1 | awk -F' ' ' {print $1, $2, $3}' | sort >routes.txt
netstat -rn |grep eth1 | awk -F' ' ' {print $1, $2, $3}' | sort | wc -l
Unload Local Firewall Policy     fwunload local




Identity Awareness
pep show user all
pep show user all | wc -l
pep show
pep show user
pep show user query shyam
pep show user query usr d

pdp show
pdp show connections
pdp connections
pdp connections pep


Performance
top
ps auxwww
fw tab -t connections -s
fw ctl pstat

fwaccel stats  Usage: fwaccel on|off | ver|stat |conns| dbg <...>
fwaccel conns
fwaccel conns -s

fw ctl multik stat
fw ctl affinity -l -a -v  check CPU core to NIC Mapping can be change in)

fw ctl multik dynamic_dispatching get_mode
fw ctl multik dynamic_dispatching on
fw ctl multik get_mode


9. Validate Sync is ESTABLISHED netstat -an | grep 2010
10. Validate Logs are Flowing to Logger and observe it for any errors

11. Test Cluster-XL HA Sync Failover by Rebooting the Primary Firewall and validate if traffic goes to Secondary without interrupting the Ping    Test to
12. Reboot Secondary and make sure it came back into the cluster. Firewall overall health Checks
13. validate firewall is not dropping any packets fw ctl zdebug + drop
14. uptime
15. fw ver


df
fw ctl pstat
cat /etc/sysconfig/ntp
netstat –i
ethtool –i eth0  (please enter all active interfaces – this will let us know what version of NIC driver is running) vmstat 1 10 free ps auxwww
vmstat 1 10
free


Troubleshooting

if you build a new gateway and wants to test connectivity across the interfaces and the ping test does not work, do the following:

[Expert@myfw-fwa:#  fw unloadlocal
[Expert@myfw-fwa:#  cpstop
[Expert@myfw-fwa:#  sbin/sysctl -w net.ipv4.ip_forward=1


fw monitor | grep 100.210.7.250
fw ctl zdebug + drop > text.drops
fw ctl zdebug + drop | grep 200.105.57.69
tcpdump -ni eth8 src 172.60.25.132
tcpdump -i eth1 port 1089 and dst 215.118.184.254
netstat -rn |grep 204.105


What happening to the traffic.
From expert mode on the Active Firewall:

fwaccel off Turn off SecureXL, if enabled)

df -h   (Check for sufficient diskspace capture and debug_

fw monitor -o /var/log/fwmon.cap   (In one session: Run the capture.)

fw ctl zdebug drop > /var/log/drop.txt  (In another session: Run the kernel debug for drops.)

tcpdump -nnei any -w /var/log/tcp.cap (In a third session: Run a tcpdump capture.)

Re-create the problem.
Control-C   (End the fw monitor, tcpdump and the kernel debug with the following:)

fwaccel on  (Turn on SecureXL, if you disabled it)


----------------------------------------------
Checkpoint Special Config Files
----------------------------------------------
1. fwkern.conf - $FWDIR/boot/modules/fwkern.conf Magic Mac
2. local.arp   - $FWDIR/conf/local.arp   GAiA manual ARP
3. sdconf.rec  -  /var/ace  RAS authentication
4. rc.local    -  /etc/rc.d/rc.local
5. netconf.C      (/etc/sysconfig) Network interfaces/Routes
6. external.if    (/etc/sysconfig)
7. ifcfg-eth1      (/etc/sysconfig/network-scripts/)
----------------------------------------------
checkpoint scripts:
----------------------------------------------
1. checkup.sh
2. cpsizeme
3. nohup mpstat -P ALL 1 86400 > mpstat.out &


Checkpoint Health Checks -Commands
uptime
ver
fw ver
cpinfo -y all
cplic print
expert
df -h
reboot
shutdown
fwunload local

 
Firewall Performance
top
ps auxwww
fw tab -t connections -s
fw ctl pstat
fwaccel stats
tcpdump -i eth0 src 100.25.240.57 and dst 26.23.64.82


Verification:

cat /etc/sysconfig/ntp
vmstat 1 10
cat /proc/meminfo
cpstat os -f cpu
cpstat os -f memory

Interface Configurations
netstat -i
ifconig -a
netstat -rn
ethtool –i eth0
ethtool -S eth1-01
ethtool -S eth1-02


cpview
fw tab -t userc_key -s
fw tab -t userc_users -s
fw tab -t om_assigned_ips -s  (verify # of Seed license)


Performance -cpconfig utility enable/disable Checkpoint SecureXL

fwaccel stats  (Usage: fwaccel on | off | ver | stat | conns | dbg <...> | help
fwaccel conns
fwaccel conns -s
fw ctl multik stat
fw ctl affinity -l -a -v
gaia> fwaccel conns -s  (checks SecureXL Connections)
gaia> fw ctl get int fwx_max_conns (checks maximum connectins 0 means it set to auto/unlimited

[Expert@fw:0]# mq_mng --show
Total 48 cores. Available for MQ 7 cores
i/f             driver          driver mode     state           mode (queues)   cores          
                                                                actual/avail                   
------------------------------------------------------------------------------------------------
Mgmt            igb             Kernel          Up              Auto (7/7)      0,24,1,25,2,26,3
Sync            igb             Kernel          Up              Auto (7/7)      0,24,1,25,2,26,3
eth1-01         i40e            Kernel          Up              Auto (7/7)      0,24,1,25,2,26,3
eth1-04         i40e            Kernel          Up              Auto (7/7)      0,24,1,25,2,26,3
 
[Expert@myfww#


[Expert@myfww## fw ctl affinity -l -r
CPU 0:
CPU 1:
CPU 2:
CPU 3:
CPU 4:  fw_38 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 5:  fw_36 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 6:  fw_34 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 7:  fw_32 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 8:  fw_30 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 9:  fw_28 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 10: fw_26 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 11: fw_24 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 12: fw_22 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 13: fw_20 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 14: fw_18 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 15: fw_16 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 16: fw_14 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 17: fw_12 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 18: fw_10 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 19: fw_8 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 20: fw_6 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 21: fw_4 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 22: fw_2 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 23: fw_0 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 24:
CPU 25:
CPU 26:
CPU 27: fw_39 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 28: fw_37 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 29: fw_35 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 30: fw_33 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 31: fw_31 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 32: fw_29 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 33: fw_27 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 34: fw_25 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 35: fw_23 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 36: fw_21 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 37: fw_19 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 38: fw_17 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 39: fw_15 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 40: fw_13 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 41: fw_11 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 42: fw_9 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 43: fw_7 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 44: fw_5 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 45: fw_3 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 46: fw_1 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 47: cprid lpd mpdaemon fwd in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
All:
Interface Mgmt: has multi queue enabled
Interface Sync: has multi queue enabled
Interface eth1-01: has multi queue enabled
Interface eth1-04: has multi queue enabled

[[Expert@myfww#


[Expert@my-fw:0]# dynamic_balancing -o disable
Disabling Dynamic Balancing, please wait for the operation to complete
Successfully disabled Dynamic Balancing
 
Dynamic Balancing made changes that require a reboot, please reboot your machine in order for the changes to take effect

Expert@my-fw:0]# cpconfig
This program will let you re-configure
your Check Point products configuration.
 
 
Configuration Options:
----------------------
(1)  Licenses and contracts
(2)  SNMP Extension
(3)  PKCS#11 Token
(4)  Random Pool
(5)  Secure Internal Communication
(6)  Disable cluster membership for this gateway
(7)  Enable Check Point Per Virtual System State
(8)  Enable Check Point ClusterXL for Bridge Active/Standby
(9)  Hyper-Threading
(10) Check Point CoreXL
(11) Automatic start of Check Point Products
 
(12) Exit
 
Enter your choice (1-12) :10
 
 
 
Configuring Check Point CoreXL...
=================================
 
 
CoreXL is currently enabled with 43 IPv4 firewall instances.
 
(1) Change the number of firewall instances
(2) Disable Check Point CoreXL
(3) Change firewall mode
 
(4) Exit
Enter your choice (1-4) : 1
 
This machine has 48 CPUs.
 
Note: All cluster members must have the same number of firewall instances
enabled.
 
How many IPv4 firewall instances would you like to enable (2 to 48) [43] ? 40
 
CoreXL was enabled successfully with 40 firewall instances.
Important: This change will take effect after reboot.

 

 


[Expert@myfwe-int02:0]# fw ctl multik stat  (connection to Core Distribution)
ID | Active  | CPU    | Connections | Peak
----------------------------------------------
 0 | Yes     | 11     |         178 |      303
 1 | Yes     | 10     |         203 |      380
 2 | Yes     | 9      |         168 |      262
 3 | Yes     | 8      |         179 |      188
 4 | Yes     | 7      |         149 |      278
 5 | Yes     | 6      |         113 |      194
 6 | Yes     | 5      |         128 |      221
 7 | Yes     | 4      |         282 |      387
 8 | Yes     | 3      |         186 |      292
 9 | Yes     | 2      |         296 |      439
[Expert@myfwe-int02:0]#


[Expert@myfwe-int02:0]# fw ctl affinity -l -a -v    (check CPU core to NIC Mapping -can be change in $FWDIR/conf/fwaffinity/conf)
Interface eth1-05 (irq 218): CPU 1
Interface Sync (irq 124): CPU 1
Interface eth1-01 (irq 107): CPU 1
Interface eth1-02 (irq 123): CPU 0
Interface eth3-01 (irq 171): CPU 0
Kernel fw_0: CPU 11
Kernel fw_1: CPU 10
Kernel fw_2: CPU 9
Kernel fw_3: CPU 8
Kernel fw_4: CPU 7
Kernel fw_5: CPU 6
Kernel fw_6: CPU 5
Kernel fw_7: CPU 4
Kernel fw_8: CPU 3
Kernel fw_9: CPU 2
Daemon in.geod: CPU all
Daemon vpnd: CPU all
Daemon in.acapd: CPU all
Daemon in.asessiond: CPU all
Daemon in.msd: CPU all
Daemon rad: CPU all
Daemon rtmd: CPU all
Daemon fwd: CPU all
Daemon mpdaemon: CPU all
Daemon usrchkd: CPU all
Daemon cpd: CPU all
Daemon cprid: CPU all
[Expert@myfwe-int02:0]#

[Expert@myfwe-int02:0]# fwaccel stat
Accelerator Status : on
Accept Templates   : disabled by Firewall
                     disabled from rule #35
Drop Templates     : disabled
NAT Templates      : disabled by user

Accelerator Features : Accounting, NAT, Cryptography, Routing,
                       HasClock, Templates, Synchronous, IdleDetection,
                       Sequencing, TcpStateDetect, AutoExpire,
                       DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
                       WireMode, DropTemplates, NatTemplates,
                       Streaming, MultiFW, AntiSpoofing, ViolationStats,
                       Nac, AsychronicNotif, ERDOS, McastRoutingV2
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
                        3DES, DES, CAST, CAST-40, AES-128, AES-256,
                        ESP, LinkSelection, DynamicVPN, NatTraversal,
                        EncRouting, AES-XCBC, SHA256
[Expert@hinfwe-int02:0]#



[Expert@myfwe-int02:0]# fwaccel conns  |grep  26.31.83.28 | more
Source          SPort Destination     DPort PR Flags       C2S i/f S2C i/f Inst Identity
--------------- ----- --------------- ----- -- ----------- ------- ------- ---- --------
 26.31.83.28    53   74.94.152.161  1580 17 F..A...S... 7/8     8/7      7        0
   66.189.0.104 21318  26.31.83.28    53 17 ...A...S... 7/8     8/7      7        0
 

Usage: fwaccel on | off | ver | stat | cfg <...> | conns | dbg <...> | help
Options:
on-turns acceleration on
off-turns acceleration off
ver-show acceleration/FW version
stat-show acceleration status
cfg-configure acceleration parameters
stats-print the acceleration statistics
conns-print the accelerator's connection table
templates-print the accelerator's templates table
dbg-set debug flags
help-this help messages
diag-policy diagnostics


----------------------------------------------
Troubleshooting
----------------------------------------------
fw monitor | grep 10.210.7.250
fw ctl zdebug + drop > text.drops
fw ctl zdebug + drop | grep 204.105.57.69
tcpdump -ni eth8 src 172.30.25.132
tcpdump -i eth1 port 1089 and dst 216.118.184.254
netstat -rn |grep 204.105

RE: Traffic failing between internet  Clusters
Run a packet capture and a kernel debug on the firewall so I can get a packet-level look at what is happening to the traffic.
From expert mode on the Active Firewall:
1. # fwaccel off  (Turn off SecureXL, if enabled)
2. # df -h (Check your disk space to make sure you have sufficient space to run a capture and debug_
3. # fw monitor -o /var/log/fwmon.cap   (In one session: Run the capture.)
4. # fw ctl zdebug drop > /var/log/drop.txt  (In another session: Run the kernel debug for drops.)
5. # tcpdump -nnei any -w /var/log/tcp.cap (In a third session: Run a tcpdump capture.)
6. Re-create the problem.
7. Control-C   (End the fw monitor, tcpdump and the kernel debug with the following:)
8. # fwaccel on  (Turn on SecureXL, if you disabled it)


----------------------------------------------
log files
----------------------------------------------
$FWDIR/log/vpnd.elg
/var/log/messages
dmesg
/var/log/routed.log


----------------------------------------------
How to Set Specific
----------------------------------------------
clusterXL_admin down/up
ethtool -s eth1  autoneg on


--------------------------------------
/etc/resolv.conf    # What's the name resolution config -- sometimes performance is adversely influenced by bad DNS settings
/etc/ntpd.conf      # Time config
/etc/ntp.conf
/etc/modprobe.conf  # Any NIC or kernel tweaks?
/etc/sysctl.conf    # Any kernel tweaks?
/etc/ssh/sshd_config # Any hacks to sshd?
/etc/issue           # console banner file
/etc/issue.net       # network banner file
/etc/motd            # message of the day file
/etc/grub.conf       # Grub config -- important to see vmalloc
/etc/gated.ami       # gated config file
/etc/gated_xl.ami    # gated config file
/etc/rc.d/rc.local   # local RC files -- any changes here (such as kernel tweaks)?
$FWDIR/boot/boot.conf              # Firewall boot params
$FWDIR/boot/modules/fwkern.conf    # Any firewall kernel tweaks?
$PPKDIR/boot/modules/simkern.conf  # Any SIM tweaks?
$FWDIR/conf/discntd.if             # ClusterXL Disconnected Interfaces
$FWDIR/conf/local.arp              # SPLAT / GAiA manual ARP
$FWDIR/conf/vsaffinity_exception.conf      # Relevant to R75.40VS and later Virtual systems only
$MDSDIR/conf/external.if                   # Relevant to P1 / MDSM only



ARPING

[myinet-fwa]# fw ctl arp
 (26.18.190.123) at 00-1c-7f-3f-6c-fd
 (26.18.190.100) at 00-1c-7f-3f-6c-fd


[myinet-fwa]# arping -I eth3-04 26.18.190.88 (check for Arping on specific interface for an IP)
ARPING 216.118.190.88 from 26.18.190.7 eth3-04
Sent 7 probes (7 broadcast(s))
Received 0 response(s)

[myinet-fwa]# arping -I eth3-04 26.18.190.87
ARPING 26.18.190.87 from 26.18.190.7 eth3-04
Sent 9 probes (9 broadcast(s))
Received 0 response(s)

[myinet-fwa]# arping -I eth3-04 26.18.190.89
ARPING 26.18.190.89 from 26.18.190.7 eth3-04
Sent 14 probes (14 broadcast(s))
Received 0 response(s)
[myinet-fwa]



[Expert@mydev-fwa]# cphaprob stat

Cluster Mode:   New High Availability (Active Up)

Number     Unique Address  Assigned Load   State

1 (local)  192.168.42.1    100%            Active
2          192.168.42.2    0%              Standby

[Expert@mydev-fwa]#


[Expert@mydev-fwa]# cphaprob -a if

Required interfaces: 6
Required secured interfaces: 1

eth0       UP                    non sync(non secured), multicast
eth1       UP                    non sync(non secured), multicast
eth2       UP                    non sync(non secured), multicast
eth3       UP                    non sync(non secured), multicast
eth4       UP                    non sync(non secured), multicast
eth5       UP                    sync(secured), multicast

Virtual cluster interfaces: 5

eth0            172.30.25.54
eth1            10.125.240.4
eth2            10.125.242.4
eth3            10.125.244.4
eth4            10.125.246.4

[Expert@mydev-fwa]#


[Expert@mydev-fwa]# cphaprob list

Built-in Devices:

Device Name: Interface Active Check
Current state: OK

Registered Devices:

Device Name: Synchronization
Registration number: 0
Timeout: none
Current state: OK
Time since last report: 388872 sec

Device Name: Filter
Registration number: 1
Timeout: none
Current state: OK
Time since last report: 388866 sec

Device Name: cphad
Registration number: 2
Timeout: 2 sec
Current state: OK
Time since last report: 0.9 sec

Device Name: fwd
Registration number: 3
Timeout: 2 sec
Current state: OK
Time since last report: 0.9 sec

Expert@mydev-fwa]#

[Expert@mydev-fwa]# cpwd_admin list
cpwd_admin:
APP        PID    STAT  #START  START_TIME             COMMAND              MON
CPD        3449   E     1       [20:24:21] 7/6/2013    cpd                  Y
CI_CLEANUP 3534   E     1       [20:24:35] 7/6/2013    avi_del_tmp_files    N
CIHS       3546   E     1       [20:24:35] 7/6/2013    ci_http_server -j -f /opt/CPsuite-R71/fw1/conf/cihs.conf N
FWD        3548   E     1       [20:24:36] 7/6/2013    fwd                  N
RTMD       4051   E     1       [20:24:59] 7/6/2013    rtmd                 N
[Expert@mydev-fwa]#

cphaprob list. There reported issues with fib with problem state. Next ran cpwd_admin list and noticed that FIBMGR and DROUTER restarted 2x's.


[Expert@myvpn-fwb]# clusterXL_admin down
Setting member to administratively down state ...
Member current state is Down
[Expert@myvpn-fwb]# cphaprob stat

Cluster Mode:   New High Availability (Active Up)


Number     Unique Address  Assigned Load   State

1          192.168.25.241  100%            Active
2 (local)  192.168.25.242  0%              Down

[Expert@myvpn-fwb]# clusterXL_admin up
Setting member to normal operation ...
Member current state is Standby
[Expert@myvpn-fwb]#