Validate Firewall Functionality
Build Out Quality Control
☐ Hostname
☐ DNS
☐ NTP
☐ Login Banner
☐ TACACS - Login
☐ Domain name
☐ Domain Prefix
☐ show asset all
Software and Jumbo hotfix
☐ fw ver
☐ cpinfo -y all
☐ fwunload local
☐ fw stat
VALIDATES
☐ Version fw ver
☐ License cplic print cat $CPDIR/conf/cp.license
☐ Routes netstat -nr | wc -l
☐ Arp fw ctl arp
☐ Connections fw tab -t connections -s
Before Establish SIC on Management ensure that your gateway CDP process is running cpwd_admin list
.The Gateway must have CPD running in E Stat .. to validate it, you can run cpwd_admin list if it is a T stat, you will not be able to establish SIC
[Expert@MY-VPN-FW01:0]# cpwd_admin list
APP PID STAT #START START_TIME MON COMMAND
FWK_FORKER 73879 E 1 [21:37:50] 14/4/2023 N fwk_forker
FWK_WD 73888 E 1 [21:37:50] 14/4/2023 N fwk_wd -i 43 -i6 0
CPVIEWD 74765 E 1 [21:38:08] 14/4/2023 N cpviewd
CPVIEWS 74782 E 1 [21:38:08] 14/4/2023 N cpview_services
CVIEWAPIS 74787 E 1 [21:38:08] 14/4/2023 N cpview_api_service
SXL_STATD 74792 E 1 [21:38:08] 14/4/2023 N sxl_statd
CPD 74804 E 1 [21:38:08] 14/4/2023 Y cpd
MPDAEMON 74816 E 1 [21:38:08] 14/4/2023 N mpdaemon /opt/CPshrd-R81.10/log/mpdaemon.elg /opt/CPshrd-R81.10/conf/mpdaemon.conf
TP_CONF_SERVICE 230716 E 1 [00:24:39] 15/4/2023 N tp_conf_service --conf=tp_conf.json --log=error
CXLD 75062 E 1 [21:38:10] 14/4/2023 N cxld -d
CI_CLEANUP 75078 E 1 [21:38:10] 14/4/2023 N avi_del_tmp_files
CIHS 75081 E 1 [21:38:10] 14/4/2023 N ci_http_server -j -f /opt/CPsuite-R81.10/fw1/conf/cihs.conf
FWD 75105 E 1 [21:38:10] 14/4/2023 N fwd
SPIKE_DETECTIVE 75120 E 1 [21:38:10] 14/4/2023 N spike_detective
DSDAEMON 158764 E 1 [01:32:16] 15/4/2023 Y dsd
DASERVICE 100901 E 1 [21:39:35] 14/4/2023 N DAService_script
AUTOUPDATER 100918 E 1 [21:39:35] 14/4/2023 N AutoUpdaterService.sh
CPHAMCSET 124212 E 1 [21:43:27] 14/4/2023 N cphamcset -d
WSDNSD 40975 E 1 [00:47:51] 15/4/2023 Y wsdnsd
RAD 125442 E 1 [21:43:30] 14/4/2023 N rad
RTMD 125479 E 1 [21:43:31] 14/4/2023 N rtmd
LPD 15444 E 1 [04:34:46] 15/4/2023 N lpd
[Expert@MY-VPN-FW01:0]#
dmidecode
dmiparse
/bin/log_start limit 0 2097152 10
3rd Party Monitoring
☐ Solarwinds
☐ TACACS authentication
☐ WebUI Login
☐ Serial Console Raritan
☐ Access via SSH/HTTPS
Interface (Subnet Mask /speed/duplex)
☐ Ifconfig
☐ ifconfig -a
☐ Show configuration interface
☐ netstat -i
☐ cat /proc/net/bonding/bond0
☐ Kernel Version [Expert@MyFW:0]# cat /proc/version
Linux version 3.10.0-957.21.3cpx86_64 (builder@8700486_0_Docker) (gcc version 4.9.2 (GCC) ) #1 SMP Mon Feb 20 16:46:42 IST 2023
[Expert@MyFW:0]#
Cluster XL (High Availability)
☐ cpstop
☐ cpstart
☐ cphastop
☐ cphastart
☐ clusterXL_admin up/down
☐ cphaprob –a if
☐ cphaprob list
☐ cphaprob stat
☐ cpstat ha -f all
☐ cphaprob syncstat
☐ cphaprob list
☐ cpwd_admin list
ClusterXL Functioning
☐ cphaprob stat
☐ cphaprob –a if
☐ cphaprob list
☐ cpwd_admin list
cpconfig
Route
route -n
netstat -nr | wc -l
netstat -i
netstat -nr | grep -v D
netstat -rn |grep eth1 | awk -F' ' ' {print $1, $2, $3}' | sort >routes.txt
netstat -rn |grep eth1 | awk -F' ' ' {print $1, $2, $3}' | sort | wc -l
Unload Local Firewall Policy fwunload local
Identity Awareness
pep show user all
pep show user all | wc -l
pep show
pep show user
pep show user query shyam
pep show user query usr d
pdp show
pdp show connections
pdp connections
pdp connections pep
Performance
top
ps auxwww
fw tab -t connections -s
fw ctl pstat
fwaccel stats Usage: fwaccel on|off | ver|stat |conns| dbg <...>
fwaccel conns
fwaccel conns -s
fw ctl multik stat
fw ctl affinity -l -a -v check CPU core to NIC Mapping can be change in)
fw ctl multik dynamic_dispatching get_mode
fw ctl multik dynamic_dispatching on
fw ctl multik get_mode
9. Validate Sync is ESTABLISHED netstat -an | grep 2010
10. Validate Logs are Flowing to Logger and observe it for any errors
11. Test Cluster-XL HA Sync Failover by Rebooting the Primary Firewall and validate if traffic goes to Secondary without interrupting the Ping Test to
12. Reboot Secondary and make sure it came back into the cluster. Firewall overall health Checks
13. validate firewall is not dropping any packets fw ctl zdebug + drop
14. uptime
15. fw ver
df
fw ctl pstat
cat /etc/sysconfig/ntp
netstat –i
ethtool –i eth0 (please enter all active interfaces – this will let us know what version of NIC driver is running) vmstat 1 10 free ps auxwww
vmstat 1 10
free
Troubleshooting
if you build a new gateway and wants to test connectivity across the interfaces and the ping test does not work, do the following:
[Expert@myfw-fwa:# fw unloadlocal
[Expert@myfw-fwa:# cpstop
[Expert@myfw-fwa:# sbin/sysctl -w net.ipv4.ip_forward=1
fw monitor | grep 100.210.7.250
fw ctl zdebug + drop > text.drops
fw ctl zdebug + drop | grep 200.105.57.69
tcpdump -ni eth8 src 172.60.25.132
tcpdump -i eth1 port 1089 and dst 215.118.184.254
netstat -rn |grep 204.105
What happening to the traffic.
From expert mode on the Active Firewall:
fwaccel off Turn off SecureXL, if enabled)
df -h (Check for sufficient diskspace capture and debug_
fw monitor -o /var/log/fwmon.cap (In one session: Run the capture.)
fw ctl zdebug drop > /var/log/drop.txt (In another session: Run the kernel debug for drops.)
tcpdump -nnei any -w /var/log/tcp.cap (In a third session: Run a tcpdump capture.)
Re-create the problem.
Control-C (End the fw monitor, tcpdump and the kernel debug with the following:)
fwaccel on (Turn on SecureXL, if you disabled it)
----------------------------------------------
Checkpoint Special Config Files
----------------------------------------------
1. fwkern.conf - $FWDIR/boot/modules/fwkern.conf Magic Mac
2. local.arp - $FWDIR/conf/local.arp GAiA manual ARP
3. sdconf.rec - /var/ace RAS authentication
4. rc.local - /etc/rc.d/rc.local
5. netconf.C (/etc/sysconfig) Network interfaces/Routes
6. external.if (/etc/sysconfig)
7. ifcfg-eth1 (/etc/sysconfig/network-scripts/)
----------------------------------------------
checkpoint scripts:
----------------------------------------------
1. checkup.sh
2. cpsizeme
3. nohup mpstat -P ALL 1 86400 > mpstat.out &
Checkpoint Health Checks -Commands
uptime
ver
fw ver
cpinfo -y all
cplic print
expert
df -h
reboot
shutdown
fwunload local
Firewall Performance
top
ps auxwww
fw tab -t connections -s
fw ctl pstat
fwaccel stats
tcpdump -i eth0 src 100.25.240.57 and dst 26.23.64.82
Verification:
cat /etc/sysconfig/ntp
vmstat 1 10
cat /proc/meminfo
cpstat os -f cpu
cpstat os -f memory
Interface Configurations
netstat -i
ifconig -a
netstat -rn
ethtool –i eth0
ethtool -S eth1-01
ethtool -S eth1-02
cpview
fw tab -t userc_key -s
fw tab -t userc_users -s
fw tab -t om_assigned_ips -s (verify # of Seed license)
Performance -cpconfig utility enable/disable Checkpoint SecureXL
fwaccel stats (Usage: fwaccel on | off | ver | stat | conns | dbg <...> | help
fwaccel conns
fwaccel conns -s
fw ctl multik stat
fw ctl affinity -l -a -v
gaia> fwaccel conns -s (checks SecureXL Connections)
gaia> fw ctl get int fwx_max_conns (checks maximum connectins 0 means it set to auto/unlimited
[Expert@fw:0]# mq_mng --show
Total 48 cores. Available for MQ 7 cores
i/f driver driver mode state mode (queues) cores
actual/avail
------------------------------------------------------------------------------------------------
Mgmt igb Kernel Up Auto (7/7) 0,24,1,25,2,26,3
Sync igb Kernel Up Auto (7/7) 0,24,1,25,2,26,3
eth1-01 i40e Kernel Up Auto (7/7) 0,24,1,25,2,26,3
eth1-04 i40e Kernel Up Auto (7/7) 0,24,1,25,2,26,3
[Expert@myfww#
[Expert@myfww## fw ctl affinity -l -r
CPU 0:
CPU 1:
CPU 2:
CPU 3:
CPU 4: fw_38 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 5: fw_36 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 6: fw_34 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 7: fw_32 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 8: fw_30 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 9: fw_28 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 10: fw_26 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 11: fw_24 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 12: fw_22 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 13: fw_20 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 14: fw_18 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 15: fw_16 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 16: fw_14 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 17: fw_12 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 18: fw_10 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 19: fw_8 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 20: fw_6 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 21: fw_4 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 22: fw_2 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 23: fw_0 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 24:
CPU 25:
CPU 26:
CPU 27: fw_39 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 28: fw_37 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 29: fw_35 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 30: fw_33 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 31: fw_31 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 32: fw_29 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 33: fw_27 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 34: fw_25 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 35: fw_23 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 36: fw_21 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 37: fw_19 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 38: fw_17 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 39: fw_15 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 40: fw_13 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 41: fw_11 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 42: fw_9 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 43: fw_7 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 44: fw_5 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 45: fw_3 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 46: fw_1 (active)
cprid lpd mpdaemon in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
CPU 47: cprid lpd mpdaemon fwd in.geod in.msd in.acapd wsdnsd rad rtmd pepd in.asessiond usrchkd pdpd iked vpnd core_uploader scanengine_b cprid cpd
All:
Interface Mgmt: has multi queue enabled
Interface Sync: has multi queue enabled
Interface eth1-01: has multi queue enabled
Interface eth1-04: has multi queue enabled
[[Expert@myfww#
[Expert@my-fw:0]# dynamic_balancing -o disable
Disabling Dynamic Balancing, please wait for the operation to complete
Successfully disabled Dynamic Balancing
Dynamic Balancing made changes that require a reboot, please reboot your machine in order for the changes to take effect
Expert@my-fw:0]# cpconfig
This program will let you re-configure
your Check Point products configuration.
Configuration Options:
----------------------
(1) Licenses and contracts
(2) SNMP Extension
(3) PKCS#11 Token
(4) Random Pool
(5) Secure Internal Communication
(6) Disable cluster membership for this gateway
(7) Enable Check Point Per Virtual System State
(8) Enable Check Point ClusterXL for Bridge Active/Standby
(9) Hyper-Threading
(10) Check Point CoreXL
(11) Automatic start of Check Point Products
(12) Exit
Enter your choice (1-12) :10
Configuring Check Point CoreXL...
=================================
CoreXL is currently enabled with 43 IPv4 firewall instances.
(1) Change the number of firewall instances
(2) Disable Check Point CoreXL
(3) Change firewall mode
(4) Exit
Enter your choice (1-4) : 1
This machine has 48 CPUs.
Note: All cluster members must have the same number of firewall instances
enabled.
How many IPv4 firewall instances would you like to enable (2 to 48) [43] ? 40
CoreXL was enabled successfully with 40 firewall instances.
Important: This change will take effect after reboot.
[Expert@myfwe-int02:0]# fw ctl multik stat (connection to Core Distribution)
ID | Active | CPU | Connections | Peak
----------------------------------------------
0 | Yes | 11 | 178 | 303
1 | Yes | 10 | 203 | 380
2 | Yes | 9 | 168 | 262
3 | Yes | 8 | 179 | 188
4 | Yes | 7 | 149 | 278
5 | Yes | 6 | 113 | 194
6 | Yes | 5 | 128 | 221
7 | Yes | 4 | 282 | 387
8 | Yes | 3 | 186 | 292
9 | Yes | 2 | 296 | 439
[Expert@myfwe-int02:0]#
[Expert@myfwe-int02:0]# fw ctl affinity -l -a -v (check CPU core to NIC Mapping -can be change in $FWDIR/conf/fwaffinity/conf)
Interface eth1-05 (irq 218): CPU 1
Interface Sync (irq 124): CPU 1
Interface eth1-01 (irq 107): CPU 1
Interface eth1-02 (irq 123): CPU 0
Interface eth3-01 (irq 171): CPU 0
Kernel fw_0: CPU 11
Kernel fw_1: CPU 10
Kernel fw_2: CPU 9
Kernel fw_3: CPU 8
Kernel fw_4: CPU 7
Kernel fw_5: CPU 6
Kernel fw_6: CPU 5
Kernel fw_7: CPU 4
Kernel fw_8: CPU 3
Kernel fw_9: CPU 2
Daemon in.geod: CPU all
Daemon vpnd: CPU all
Daemon in.acapd: CPU all
Daemon in.asessiond: CPU all
Daemon in.msd: CPU all
Daemon rad: CPU all
Daemon rtmd: CPU all
Daemon fwd: CPU all
Daemon mpdaemon: CPU all
Daemon usrchkd: CPU all
Daemon cpd: CPU all
Daemon cprid: CPU all
[Expert@myfwe-int02:0]#
[Expert@myfwe-int02:0]# fwaccel stat
Accelerator Status : on
Accept Templates : disabled by Firewall
disabled from rule #35
Drop Templates : disabled
NAT Templates : disabled by user
Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, ViolationStats,
Nac, AsychronicNotif, ERDOS, McastRoutingV2
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256
[Expert@hinfwe-int02:0]#
[Expert@myfwe-int02:0]# fwaccel conns |grep 26.31.83.28 | more
Source SPort Destination DPort PR Flags C2S i/f S2C i/f Inst Identity
--------------- ----- --------------- ----- -- ----------- ------- ------- ---- --------
26.31.83.28 53 74.94.152.161 1580 17 F..A...S... 7/8 8/7 7 0
66.189.0.104 21318 26.31.83.28 53 17 ...A...S... 7/8 8/7 7 0
Usage: fwaccel on | off | ver | stat | cfg <...> | conns | dbg <...> | help
Options:
on-turns acceleration on
off-turns acceleration off
ver-show acceleration/FW version
stat-show acceleration status
cfg-configure acceleration parameters
stats-print the acceleration statistics
conns-print the accelerator's connection table
templates-print the accelerator's templates table
dbg-set debug flags
help-this help messages
diag-policy diagnostics
----------------------------------------------
Troubleshooting
----------------------------------------------
fw monitor | grep 10.210.7.250
fw ctl zdebug + drop > text.drops
fw ctl zdebug + drop | grep 204.105.57.69
tcpdump -ni eth8 src 172.30.25.132
tcpdump -i eth1 port 1089 and dst 216.118.184.254
netstat -rn |grep 204.105
RE: Traffic failing between internet Clusters
Run a packet capture and a kernel debug on the firewall so I can get a packet-level look at what is happening to the traffic.
From expert mode on the Active Firewall:
1. # fwaccel off (Turn off SecureXL, if enabled)
2. # df -h (Check your disk space to make sure you have sufficient space to run a capture and debug_
3. # fw monitor -o /var/log/fwmon.cap (In one session: Run the capture.)
4. # fw ctl zdebug drop > /var/log/drop.txt (In another session: Run the kernel debug for drops.)
5. # tcpdump -nnei any -w /var/log/tcp.cap (In a third session: Run a tcpdump capture.)
6. Re-create the problem.
7. Control-C (End the fw monitor, tcpdump and the kernel debug with the following:)
8. # fwaccel on (Turn on SecureXL, if you disabled it)
----------------------------------------------
log files
----------------------------------------------
$FWDIR/log/vpnd.elg
/var/log/messages
dmesg
/var/log/routed.log
----------------------------------------------
How to Set Specific
----------------------------------------------
clusterXL_admin down/up
ethtool -s eth1 autoneg on
--------------------------------------
/etc/resolv.conf # What's the name resolution config -- sometimes performance is adversely influenced by bad DNS settings
/etc/ntpd.conf # Time config
/etc/ntp.conf
/etc/modprobe.conf # Any NIC or kernel tweaks?
/etc/sysctl.conf # Any kernel tweaks?
/etc/ssh/sshd_config # Any hacks to sshd?
/etc/issue # console banner file
/etc/issue.net # network banner file
/etc/motd # message of the day file
/etc/grub.conf # Grub config -- important to see vmalloc
/etc/gated.ami # gated config file
/etc/gated_xl.ami # gated config file
/etc/rc.d/rc.local # local RC files -- any changes here (such as kernel tweaks)?
$FWDIR/boot/boot.conf # Firewall boot params
$FWDIR/boot/modules/fwkern.conf # Any firewall kernel tweaks?
$PPKDIR/boot/modules/simkern.conf # Any SIM tweaks?
$FWDIR/conf/discntd.if # ClusterXL Disconnected Interfaces
$FWDIR/conf/local.arp # SPLAT / GAiA manual ARP
$FWDIR/conf/vsaffinity_exception.conf # Relevant to R75.40VS and later Virtual systems only
$MDSDIR/conf/external.if # Relevant to P1 / MDSM only
ARPING
[myinet-fwa]# fw ctl arp
(26.18.190.123) at 00-1c-7f-3f-6c-fd
(26.18.190.100) at 00-1c-7f-3f-6c-fd
[myinet-fwa]# arping -I eth3-04 26.18.190.88 (check for Arping on specific interface for an IP)
ARPING 216.118.190.88 from 26.18.190.7 eth3-04
Sent 7 probes (7 broadcast(s))
Received 0 response(s)
[myinet-fwa]# arping -I eth3-04 26.18.190.87
ARPING 26.18.190.87 from 26.18.190.7 eth3-04
Sent 9 probes (9 broadcast(s))
Received 0 response(s)
[myinet-fwa]# arping -I eth3-04 26.18.190.89
ARPING 26.18.190.89 from 26.18.190.7 eth3-04
Sent 14 probes (14 broadcast(s))
Received 0 response(s)
[myinet-fwa]
[Expert@mydev-fwa]# cphaprob stat
Cluster Mode: New High Availability (Active Up)
Number Unique Address Assigned Load State
1 (local) 192.168.42.1 100% Active
2 192.168.42.2 0% Standby
[Expert@mydev-fwa]#
[Expert@mydev-fwa]# cphaprob -a if
Required interfaces: 6
Required secured interfaces: 1
eth0 UP non sync(non secured), multicast
eth1 UP non sync(non secured), multicast
eth2 UP non sync(non secured), multicast
eth3 UP non sync(non secured), multicast
eth4 UP non sync(non secured), multicast
eth5 UP sync(secured), multicast
Virtual cluster interfaces: 5
eth0 172.30.25.54
eth1 10.125.240.4
eth2 10.125.242.4
eth3 10.125.244.4
eth4 10.125.246.4
[Expert@mydev-fwa]#
[Expert@mydev-fwa]# cphaprob list
Built-in Devices:
Device Name: Interface Active Check
Current state: OK
Registered Devices:
Device Name: Synchronization
Registration number: 0
Timeout: none
Current state: OK
Time since last report: 388872 sec
Device Name: Filter
Registration number: 1
Timeout: none
Current state: OK
Time since last report: 388866 sec
Device Name: cphad
Registration number: 2
Timeout: 2 sec
Current state: OK
Time since last report: 0.9 sec
Device Name: fwd
Registration number: 3
Timeout: 2 sec
Current state: OK
Time since last report: 0.9 sec
Expert@mydev-fwa]#
[Expert@mydev-fwa]# cpwd_admin list
cpwd_admin:
APP PID STAT #START START_TIME COMMAND MON
CPD 3449 E 1 [20:24:21] 7/6/2013 cpd Y
CI_CLEANUP 3534 E 1 [20:24:35] 7/6/2013 avi_del_tmp_files N
CIHS 3546 E 1 [20:24:35] 7/6/2013 ci_http_server -j -f /opt/CPsuite-R71/fw1/conf/cihs.conf N
FWD 3548 E 1 [20:24:36] 7/6/2013 fwd N
RTMD 4051 E 1 [20:24:59] 7/6/2013 rtmd N
[Expert@mydev-fwa]#
cphaprob list. There reported issues with fib with problem state. Next ran cpwd_admin list and noticed that FIBMGR and DROUTER restarted 2x's.
[Expert@myvpn-fwb]# clusterXL_admin down
Setting member to administratively down state ...
Member current state is Down
[Expert@myvpn-fwb]# cphaprob stat
Cluster Mode: New High Availability (Active Up)
Number Unique Address Assigned Load State
1 192.168.25.241 100% Active
2 (local) 192.168.25.242 0% Down
[Expert@myvpn-fwb]# clusterXL_admin up
Setting member to normal operation ...
Member current state is Standby
[Expert@myvpn-fwb]#
