Friday, November 11, 2022

Validate - FW build

Validate Firewall Functionality

Build Out Quality Control
☐ Hostname
 DNS
 NTP
 Login Banner
 TACACS - Login
 Domain name
 Domain Prefix
 show asset all

Software and Jumbo hotfix
 fw ver
 cpinfo -y all
☐ fwunload local
☐ fw stat

VALIDATES
 Version         fw ver
 License         cplic print      cat $CPDIR/conf/cp.license
 Routes          netstat -nr | wc -l
 Arp             fw ctl arp
 Connections     fw tab -t connections -s

Before Establish SIC on Management ensure that your gateway CDP process is running cpwd_admin list

.The Gateway must  have CPD running in E Stat .. to validate it, you can run cpwd_admin list  if it is a T stat, you will not be able to establish SIC

[Expert@MY-VPN-FW01:0]# cpwd_admin list

APP        PID    STAT  #START  START_TIME             MON  COMMAND             
FWK_FORKER 73879  E     1       [21:37:50] 14/4/2023   N    fwk_forker          
FWK_WD     73888  E     1       [21:37:50] 14/4/2023   N    fwk_wd -i 43 -i6 0  
CPVIEWD    74765  E     1       [21:38:08] 14/4/2023   N    cpviewd             
CPVIEWS    74782  E     1       [21:38:08] 14/4/2023   N    cpview_services     
CVIEWAPIS  74787  E     1       [21:38:08] 14/4/2023   N    cpview_api_service  
SXL_STATD  74792  E     1       [21:38:08] 14/4/2023   N    sxl_statd           
CPD        74804  E     1       [21:38:08] 14/4/2023   Y    cpd                 
MPDAEMON   74816  E     1       [21:38:08] 14/4/2023   N    mpdaemon /opt/CPshrd-R81.10/log/mpdaemon.elg /opt/CPshrd-R81.10/conf/mpdaemon.conf
TP_CONF_SERVICE 230716 E     1       [00:24:39] 15/4/2023   N    tp_conf_service --conf=tp_conf.json --log=error
CXLD       75062  E     1       [21:38:10] 14/4/2023   N    cxld -d             
CI_CLEANUP 75078  E     1       [21:38:10] 14/4/2023   N    avi_del_tmp_files   
CIHS       75081  E     1       [21:38:10] 14/4/2023   N    ci_http_server -j -f /opt/CPsuite-R81.10/fw1/conf/cihs.conf
FWD        75105  E     1       [21:38:10] 14/4/2023   N    fwd                 
SPIKE_DETECTIVE 75120  E     1       [21:38:10] 14/4/2023   N    spike_detective     
DSDAEMON   158764 E     1       [01:32:16] 15/4/2023   Y    dsd                 
DASERVICE  100901 E     1       [21:39:35] 14/4/2023   N    DAService_script    
AUTOUPDATER 100918 E     1       [21:39:35] 14/4/2023   N    AutoUpdaterService.sh
CPHAMCSET  124212 E     1       [21:43:27] 14/4/2023   N    cphamcset -d        
WSDNSD     40975  E     1       [00:47:51] 15/4/2023   Y    wsdnsd              
RAD        125442 E     1       [21:43:30] 14/4/2023   N    rad                 
RTMD       125479 E     1       [21:43:31] 14/4/2023   N    rtmd                
LPD        15444  E     1       [04:34:46] 15/4/2023   N    lpd                 
[Expert@MY-VPN-FW01:0]# 


dmidecode
dmiparse
/bin/log_start limit 0 2097152 10



3rd Party Monitoring
 Solarwinds
 TACACS authentication
 WebUI Login
 Serial Console Raritan
 Access via SSH/HTTPS

Interface  (Subnet Mask /speed/duplex)
 Ifconfig
 ifconfig -a
 Show configuration interface
 netstat -i
 cat /proc/net/bonding/bond0
 Kernel Version [Expert@MyFW:0]# cat /proc/version
Linux version 3.10.0-957.21.3cpx86_64 (builder@8700486_0_Docker) (gcc version 4.9.2 (GCC) ) #1 SMP Mon Feb 20 16:46:42 IST 2023
[Expert@MyFW:0]# 

Cluster XL (High Availability)
 cpstop
 cpstart
 cphastop
 cphastart
 clusterXL_admin up/down
 cphaprob –a if
 cphaprob list
 cphaprob stat
 cpstat ha -f all
 cphaprob syncstat
 cphaprob list
 cpwd_admin list

ClusterXL Functioning 
 cphaprob stat
 cphaprob –a if
 cphaprob list
 cpwd_admin list

cpconfig

Route
route -n
netstat -nr | wc -l
netstat -i
netstat -nr | grep -v D
netstat -rn |grep eth1 | awk -F' ' ' {print $1, $2, $3}' | sort >routes.txt
netstat -rn |grep eth1 | awk -F' ' ' {print $1, $2, $3}' | sort | wc -l
Unload Local Firewall Policy     fwunload local




Identity Awareness
pep show user all
pep show user all | wc -l
pep show
pep show user
pep show user query shyam
pep show user query usr d

pdp show
pdp show connections
pdp connections
pdp connections pep


Performance
top
ps auxwww
fw tab -t connections -s
fw ctl pstat

fwaccel stats  Usage: fwaccel on|off | ver|stat |conns| dbg <...>
fwaccel conns
fwaccel conns -s

fw ctl multik stat
fw ctl affinity -l -a -v  check CPU core to NIC Mapping can be change in)

fw ctl multik dynamic_dispatching get_mode
fw ctl multik dynamic_dispatching on
fw ctl multik get_mode


9. Validate Sync is ESTABLISHED netstat -an | grep 2010
10. Validate Logs are Flowing to Logger and observe it for any errors

11. Test Cluster-XL HA Sync Failover by Rebooting the Primary Firewall and validate if traffic goes to Secondary without interrupting the Ping    Test to
12. Reboot Secondary and make sure it came back into the cluster. Firewall overall health Checks
13. validate firewall is not dropping any packets fw ctl zdebug + drop
14. uptime
15. fw ver


df
fw ctl pstat
cat /etc/sysconfig/ntp
netstat –i
ethtool –i eth0  (please enter all active interfaces – this will let us know what version of NIC driver is running) vmstat 1 10 free ps auxwww
vmstat 1 10
free


Troubleshooting

if you build a new gateway and wants to test connectivity across the interfaces and the ping test does not work, do the following:

[Expert@myfw-fwa:#  fw unloadlocal
[Expert@myfw-fwa:#  cpstop
[Expert@myfw-fwa:#  sbin/sysctl -w net.ipv4.ip_forward=1


fw monitor | grep 100.210.7.250
fw ctl zdebug + drop > text.drops
fw ctl zdebug + drop | grep 200.105.57.69
tcpdump -ni eth8 src 172.60.25.132
tcpdump -i eth1 port 1089 and dst 215.118.184.254
netstat -rn |grep 204.105


What happening to the traffic.
From expert mode on the Active Firewall:

fwaccel off Turn off SecureXL, if enabled)

df -h   (Check for sufficient diskspace capture and debug_

fw monitor -o /var/log/fwmon.cap   (In one session: Run the capture.)

fw ctl zdebug drop > /var/log/drop.txt  (In another session: Run the kernel debug for drops.)

tcpdump -nnei any -w /var/log/tcp.cap (In a third session: Run a tcpdump capture.)

Re-create the problem.
Control-C   (End the fw monitor, tcpdump and the kernel debug with the following:)

fwaccel on  (Turn on SecureXL, if you disabled it)


----------------------------------------------
Checkpoint Special Config Files
----------------------------------------------
1. fwkern.conf - $FWDIR/boot/modules/fwkern.conf Magic Mac
2. local.arp   - $FWDIR/conf/local.arp   GAiA manual ARP
3. sdconf.rec  -  /var/ace  RAS authentication
4. rc.local    -  /etc/rc.d/rc.local
5. netconf.C      (/etc/sysconfig) Network interfaces/Routes
6. external.if    (/etc/sysconfig)
7. ifcfg-eth1      (/etc/sysconfig/network-scripts/)
----------------------------------------------
checkpoint scripts:
----------------------------------------------
1. checkup.sh
2. cpsizeme
3. nohup mpstat -P ALL 1 86400 > mpstat.out &


Checkpoint Health Checks -Commands
uptime
ver
fw ver
cpinfo -y all
cplic print
expert
df -h
reboot
shutdown
fwunload local

 
Firewall Performance
top
ps auxwww
fw tab -t connections -s
fw ctl pstat
fwaccel stats
tcpdump -i eth0 src 100.25.240.57 and dst 26.23.64.82


Verification:

cat /etc/sysconfig/ntp
vmstat 1 10
cat /proc/meminfo
cpstat os -f cpu
cpstat os -f memory

Interface Configurations
netstat -i
ifconig -a
netstat -rn
ethtool –i eth0
ethtool -S eth1-01
ethtool -S eth1-02


cpview
fw tab -t userc_key -s
fw tab -t userc_users -s
fw tab -t om_assigned_ips -s  (verify # of Seed license)


Performance -cpconfig utility enable/disable Checkpoint SecureXL

fwaccel stats  (Usage: fwaccel on | off | ver | stat | conns | dbg <...> | help
fwaccel conns
fwaccel conns -s
fw ctl multik stat
fw ctl affinity -l -a -v
gaia> fwaccel conns -s  (checks SecureXL Connections)
gaia> fw ctl get int fwx_max_conns (checks maximum connectins 0 means it set to auto/unlimited

[Expert@fw:0]# mq_mng --show
Total 48 cores. Available for MQ 7 cores
i/f             driver          driver mode     state           mode (queues)   cores          
                                                                actual/avail                   
------------------------------------------------------------------------------------------------
Mgmt            igb             Kernel          Up              Auto (7/7)      0,24,1,25,2,26,3
Sync            igb             Kernel          Up              Auto (7/7)      0,24,1,25,2,26,3
eth1-01         i40e            Kernel          Up              Auto (7/7)      0,24,1,25,2,26,3
eth1-04         i40e            Kernel          Up              Auto (7/7)      0,24,1,25,2,26,3
 
[Expert@myfww#


[Expert@myfww## fw ctl affinity -l -r
CPU 0:
CPU 1:
CPU 2:
CPU 3:
CPU 4:  fw_38 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 5:  fw_36 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 6:  fw_34 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 7:  fw_32 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 8:  fw_30 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 9:  fw_28 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 10: fw_26 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 11: fw_24 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 12: fw_22 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 13: fw_20 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 14: fw_18 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 15: fw_16 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 16: fw_14 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 17: fw_12 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 18: fw_10 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 19: fw_8 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 20: fw_6 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 21: fw_4 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 22: fw_2 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 23: fw_0 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 24:
CPU 25:
CPU 26:
CPU 27: fw_39 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 28: fw_37 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 29: fw_35 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 30: fw_33 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 31: fw_31 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 32: fw_29 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 33: fw_27 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 34: fw_25 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 35: fw_23 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 36: fw_21 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 37: fw_19 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 38: fw_17 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 39: fw_15 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 40: fw_13 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 41: fw_11 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 42: fw_9 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 43: fw_7 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 44: fw_5 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 45: fw_3 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 46: fw_1 (active)
        cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 47: cprid lpd mpdaemon fwd in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
All:
Interface Mgmt: has multi queue enabled
Interface Sync: has multi queue enabled
Interface eth1-01: has multi queue enabled
Interface eth1-04: has multi queue enabled

[[Expert@myfww#


[Expert@my-fw:0]# dynamic_balancing -o disable
Disabling Dynamic Balancing, please wait for the operation to complete
Successfully disabled Dynamic Balancing
 
Dynamic Balancing made changes that require a reboot, please reboot your machine in order for the changes to take effect

Expert@my-fw:0]# cpconfig
This program will let you re-configure
your Check Point products configuration.
 
 
Configuration Options:
----------------------
(1)  Licenses and contracts
(2)  SNMP Extension
(3)  PKCS#11 Token
(4)  Random Pool
(5)  Secure Internal Communication
(6)  Disable cluster membership for this gateway
(7)  Enable Check Point Per Virtual System State
(8)  Enable Check Point ClusterXL for Bridge Active/Standby
(9)  Hyper-Threading
(10) Check Point CoreXL
(11) Automatic start of Check Point Products
 
(12) Exit
 
Enter your choice (1-12) :10
 
 
 
Configuring Check Point CoreXL...
=================================
 
 
CoreXL is currently enabled with 43 IPv4 firewall instances.
 
(1) Change the number of firewall instances
(2) Disable Check Point CoreXL
(3) Change firewall mode
 
(4) Exit
Enter your choice (1-4) : 1
 
This machine has 48 CPUs.
 
Note: All cluster members must have the same number of firewall instances
enabled.
 
How many IPv4 firewall instances would you like to enable (2 to 48) [43] ? 40
 
CoreXL was enabled successfully with 40 firewall instances.
Important: This change will take effect after reboot.

 

 


[Expert@myfwe-int02:0]# fw ctl multik stat  (connection to Core Distribution)
ID | Active  | CPU    | Connections | Peak
----------------------------------------------
 0 | Yes     | 11     |         178 |      303
 1 | Yes     | 10     |         203 |      380
 2 | Yes     | 9      |         168 |      262
 3 | Yes     | 8      |         179 |      188
 4 | Yes     | 7      |         149 |      278
 5 | Yes     | 6      |         113 |      194
 6 | Yes     | 5      |         128 |      221
 7 | Yes     | 4      |         282 |      387
 8 | Yes     | 3      |         186 |      292
 9 | Yes     | 2      |         296 |      439
[Expert@myfwe-int02:0]#


[Expert@myfwe-int02:0]# fw ctl affinity -l -a -v    (check CPU core to NIC Mapping -can be change in $FWDIR/conf/fwaffinity/conf)
Interface eth1-05 (irq 218): CPU 1
Interface Sync (irq 124): CPU 1
Interface eth1-01 (irq 107): CPU 1
Interface eth1-02 (irq 123): CPU 0
Interface eth3-01 (irq 171): CPU 0
Kernel fw_0: CPU 11
Kernel fw_1: CPU 10
Kernel fw_2: CPU 9
Kernel fw_3: CPU 8
Kernel fw_4: CPU 7
Kernel fw_5: CPU 6
Kernel fw_6: CPU 5
Kernel fw_7: CPU 4
Kernel fw_8: CPU 3
Kernel fw_9: CPU 2
Daemon in.geod: CPU all
Daemon vpnd: CPU all
Daemon in.acapd: CPU all
Daemon in.asessiond: CPU all
Daemon in.msd: CPU all
Daemon rad: CPU all
Daemon rtmd: CPU all
Daemon fwd: CPU all
Daemon mpdaemon: CPU all
Daemon usrchkd: CPU all
Daemon cpd: CPU all
Daemon cprid: CPU all
[Expert@myfwe-int02:0]#

[Expert@myfwe-int02:0]# fwaccel stat
Accelerator Status : on
Accept Templates   : disabled by Firewall
                     disabled from rule #35
Drop Templates     : disabled
NAT Templates      : disabled by user

Accelerator Features : Accounting, NAT, Cryptography, Routing,
                       HasClock, Templates, Synchronous, IdleDetection,
                       Sequencing, TcpStateDetect, AutoExpire,
                       DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
                       WireMode, DropTemplates, NatTemplates,
                       Streaming, MultiFW, AntiSpoofing, ViolationStats,
                       Nac, AsychronicNotif, ERDOS, McastRoutingV2
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
                        3DES, DES, CAST, CAST-40, AES-128, AES-256,
                        ESP, LinkSelection, DynamicVPN, NatTraversal,
                        EncRouting, AES-XCBC, SHA256
[Expert@hinfwe-int02:0]#



[Expert@myfwe-int02:0]# fwaccel conns  |grep  26.31.83.28 | more
Source          SPort Destination     DPort PR Flags       C2S i/f S2C i/f Inst Identity
--------------- ----- --------------- ----- -- ----------- ------- ------- ---- --------
 26.31.83.28    53   74.94.152.161  1580 17 F..A...S... 7/8     8/7      7        0
   66.189.0.104 21318  26.31.83.28    53 17 ...A...S... 7/8     8/7      7        0
 

Usage: fwaccel on | off | ver | stat | cfg <...> | conns | dbg <...> | help
Options:
on-turns acceleration on
off-turns acceleration off
ver-show acceleration/FW version
stat-show acceleration status
cfg-configure acceleration parameters
stats-print the acceleration statistics
conns-print the accelerator's connection table
templates-print the accelerator's templates table
dbg-set debug flags
help-this help messages
diag-policy diagnostics


----------------------------------------------
Troubleshooting
----------------------------------------------
fw monitor | grep 10.210.7.250
fw ctl zdebug + drop > text.drops
fw ctl zdebug + drop | grep 204.105.57.69
tcpdump -ni eth8 src 172.30.25.132
tcpdump -i eth1 port 1089 and dst 216.118.184.254
netstat -rn |grep 204.105

RE: Traffic failing between internet  Clusters
Run a packet capture and a kernel debug on the firewall so I can get a packet-level look at what is happening to the traffic.
From expert mode on the Active Firewall:
1. # fwaccel off  (Turn off SecureXL, if enabled)
2. # df -h (Check your disk space to make sure you have sufficient space to run a capture and debug_
3. # fw monitor -o /var/log/fwmon.cap   (In one session: Run the capture.)
4. # fw ctl zdebug drop > /var/log/drop.txt  (In another session: Run the kernel debug for drops.)
5. # tcpdump -nnei any -w /var/log/tcp.cap (In a third session: Run a tcpdump capture.)
6. Re-create the problem.
7. Control-C   (End the fw monitor, tcpdump and the kernel debug with the following:)
8. # fwaccel on  (Turn on SecureXL, if you disabled it)


----------------------------------------------
log files
----------------------------------------------
$FWDIR/log/vpnd.elg
/var/log/messages
dmesg
/var/log/routed.log


----------------------------------------------
How to Set Specific
----------------------------------------------
clusterXL_admin down/up
ethtool -s eth1  autoneg on


--------------------------------------
/etc/resolv.conf    # What's the name resolution config -- sometimes performance is adversely influenced by bad DNS settings
/etc/ntpd.conf      # Time config
/etc/ntp.conf
/etc/modprobe.conf  # Any NIC or kernel tweaks?
/etc/sysctl.conf    # Any kernel tweaks?
/etc/ssh/sshd_config # Any hacks to sshd?
/etc/issue           # console banner file
/etc/issue.net       # network banner file
/etc/motd            # message of the day file
/etc/grub.conf       # Grub config -- important to see vmalloc
/etc/gated.ami       # gated config file
/etc/gated_xl.ami    # gated config file
/etc/rc.d/rc.local   # local RC files -- any changes here (such as kernel tweaks)?
$FWDIR/boot/boot.conf              # Firewall boot params
$FWDIR/boot/modules/fwkern.conf    # Any firewall kernel tweaks?
$PPKDIR/boot/modules/simkern.conf  # Any SIM tweaks?
$FWDIR/conf/discntd.if             # ClusterXL Disconnected Interfaces
$FWDIR/conf/local.arp              # SPLAT / GAiA manual ARP
$FWDIR/conf/vsaffinity_exception.conf      # Relevant to R75.40VS and later Virtual systems only
$MDSDIR/conf/external.if                   # Relevant to P1 / MDSM only



ARPING

[myinet-fwa]# fw ctl arp
 (26.18.190.123) at 00-1c-7f-3f-6c-fd
 (26.18.190.100) at 00-1c-7f-3f-6c-fd


[myinet-fwa]# arping -I eth3-04 26.18.190.88 (check for Arping on specific interface for an IP)
ARPING 216.118.190.88 from 26.18.190.7 eth3-04
Sent 7 probes (7 broadcast(s))
Received 0 response(s)

[myinet-fwa]# arping -I eth3-04 26.18.190.87
ARPING 26.18.190.87 from 26.18.190.7 eth3-04
Sent 9 probes (9 broadcast(s))
Received 0 response(s)

[myinet-fwa]# arping -I eth3-04 26.18.190.89
ARPING 26.18.190.89 from 26.18.190.7 eth3-04
Sent 14 probes (14 broadcast(s))
Received 0 response(s)
[myinet-fwa]



[Expert@mydev-fwa]# cphaprob stat

Cluster Mode:   New High Availability (Active Up)

Number     Unique Address  Assigned Load   State

1 (local)  192.168.42.1    100%            Active
2          192.168.42.2    0%              Standby

[Expert@mydev-fwa]#


[Expert@mydev-fwa]# cphaprob -a if

Required interfaces: 6
Required secured interfaces: 1

eth0       UP                    non sync(non secured), multicast
eth1       UP                    non sync(non secured), multicast
eth2       UP                    non sync(non secured), multicast
eth3       UP                    non sync(non secured), multicast
eth4       UP                    non sync(non secured), multicast
eth5       UP                    sync(secured), multicast

Virtual cluster interfaces: 5

eth0            172.30.25.54
eth1            10.125.240.4
eth2            10.125.242.4
eth3            10.125.244.4
eth4            10.125.246.4

[Expert@mydev-fwa]#


[Expert@mydev-fwa]# cphaprob list

Built-in Devices:

Device Name: Interface Active Check
Current state: OK

Registered Devices:

Device Name: Synchronization
Registration number: 0
Timeout: none
Current state: OK
Time since last report: 388872 sec

Device Name: Filter
Registration number: 1
Timeout: none
Current state: OK
Time since last report: 388866 sec

Device Name: cphad
Registration number: 2
Timeout: 2 sec
Current state: OK
Time since last report: 0.9 sec

Device Name: fwd
Registration number: 3
Timeout: 2 sec
Current state: OK
Time since last report: 0.9 sec

Expert@mydev-fwa]#

[Expert@mydev-fwa]# cpwd_admin list
cpwd_admin:
APP        PID    STAT  #START  START_TIME             COMMAND              MON
CPD        3449   E     1       [20:24:21] 7/6/2013    cpd                  Y
CI_CLEANUP 3534   E     1       [20:24:35] 7/6/2013    avi_del_tmp_files    N
CIHS       3546   E     1       [20:24:35] 7/6/2013    ci_http_server -j -f /opt/CPsuite-R71/fw1/conf/cihs.conf N
FWD        3548   E     1       [20:24:36] 7/6/2013    fwd                  N
RTMD       4051   E     1       [20:24:59] 7/6/2013    rtmd                 N
[Expert@mydev-fwa]#

cphaprob list. There reported issues with fib with problem state. Next ran cpwd_admin list and noticed that FIBMGR and DROUTER restarted 2x's.


[Expert@myvpn-fwb]# clusterXL_admin down
Setting member to administratively down state ...
Member current state is Down
[Expert@myvpn-fwb]# cphaprob stat

Cluster Mode:   New High Availability (Active Up)


Number     Unique Address  Assigned Load   State

1          192.168.25.241  100%            Active
2 (local)  192.168.25.242  0%              Down

[Expert@myvpn-fwb]# clusterXL_admin up
Setting member to normal operation ...
Member current state is Standby
[Expert@myvpn-fwb]#