/var/log/crash
/var/log/dump/usermode/
1. Was this connections working before? Yes /No
2. Is the application up and running
- validated on the destination server?
- netstat a | grep listening port
- do a local telnet to listening port
- can the destination server ping its default gateway
2. When was the connection last worked? validate date or via SmartView Tracker
3. What is the Source and Destination of Connection?
4. What changed in the environment?
5. Is traffic flowing thru the firewalls? SmartView Tracker
6. Validate if a policy was updated on the firewalls fw stat or cpstat fw
7. Do some basic layer 2 and 3 traffic flow
- traceroute or tracert to destination ip from source (determine the path of traffic flow)
- ping the destination address (validate if you are seeing traffic in tracker)
- telnet to destination address port the application is listening on
8. Are you routing the traffic to the correct interface of the firewalls?
- netstat -rn | grep destination IP address
9. Is the traffic hitting the firewall interface
- tcpdump -i eth0 port 1089 and src 11.11.11.11
10. is the traffic leaving the firewall
- tcpdump -i eth1 port 1089 and dst 10.10.10.10
Please run a packet capture and a kernel debug on the firewall so I can get a packet-level look at what is happening to the traffic.
From expert mode on the Active Firewall:
1. Turn off SecureXL, if enabled.
# fwaccel off
2. Check your disk space to make sure you have sufficient space to run a capture and debug.
# df -h
3. In one session: Run the capture.
# fw monitor -o /var/log/fwmon.cap
4. In another session: Run the kernel debug for drops.
# fw ctl zdebug drop > /var/log/drop.txt
5. In a third session: Run a tcpdump capture.
# tcpdump -nnei any -w /var/log/tcp.cap
6. Re-create the problem.
7. End the fw monitor, tcpdump and the kernel debug with the following:
Control-C
8. Turn on SecureXL, if you disabled it.
# fwaccel on
9. Upload the packet captures and zdebug drop output using the Check Point Uploader Utility.
# cp_uploader -s 5-1100876571 -u youremail@yourdomain.com /var/log/drop.txt /var/log/tcp.cap /var/log/fwmon.cap
10. Please provide me with the Source IP, Destination IP, External IP of Active Gateway, and if Gateway is set up in a cluster, the VIP of the cluster as well so I can thoroughly review the debugs.
Checkpoint Command Line
[Expert@myfw-fwa]# tcpdump -i eth0 port 1089 and src 205.105.57.69
[Expert@myfw-fwa]# fw ctl zdebug + drop | grep 204.105.57.69
[Expert@myfw-fwa]# fw monitor | grep 10.210.7.250
[Expert@myfw-fwa]# fw monitor -ci 10 | grep 172.30.25.132
[Expert@myfw-fwa]# tcpdump -ni eth8 src 172.30.25.132
[Expert@myfw-fwa]# netstat -rn | wc -l
[Expert@myfw-fwa]# netstat -rn | grep -v D
[Expert@myfw-fwa]# ip route show match 167.211.210
fw verdf -h
cphaprob –a if
cphaprob list
cphaprob stat
fw ctl pstat
fw tab –t connections –s
cat /etc/sysconfig/ntp
netstat –i
netstat -rn | grep -v D
ethtool –i eth0
vmstat 1 10
free
ps auxw
netstat -rn
uptime
ifconig -a
[Expert@myfwl-int01:0]# cpstat os -f routing | grep 216.231.64
| 216.231.64.0| 255.255.255.0| 10.15.249.2|eth3-01 |
| 216.231.64.0| 255.255.255.0| 10.15.249.3|eth3-01 |
[Expert@myfwl-int01:0]#
[Expert@myfw-fwa]# ip route show match 167.211.210.
167.211.210.0/24 via 10.25.0.102 dev eth3
default via 216.231.83.11 dev eth1
[Expert@myfw-fwa]# fw ctl zdebug + drop | grep 204.105.57.69
fw_log_drop: Packet proto=6 204.105.57.69:53771 -> 10.25.227.13:1089 dropped by fw_handle_first_packet Reason: Rulebase drop - rule 157
fw_log_drop: Packet proto=6 204.105.57.69:53771 -> 10.25.227.13:1089 dropped by fw_handle_first_packet Reason: Rulebase drop - rule 157
fw_log_drop: Packet proto=6 204.105.57.69:53751 -> 10.25.227.13:1089 dropped by fw_handle_first_packet Reason: Rulebase drop - rule 157
fw_log_drop: Packet proto=6 204.105.57.69:53771 -> 10.25.227.13:1089 dropped by fw_handle_first_packet Reason: Rulebase drop - rule 157
/bin/cpfw_start: line 6: 10924 Broken pipe $FWDIR/bin/fw "$@"
[Expert@myfw-fwa]# tcpdump -i eth0 port 1089 and src 205.105.57.69
tcpdump: listening on eth0
12:34:11.617913 205.105.57.69.52433 > 10.25.227.13.1089: S 608260088:608260088(0) win 49640 <mss 1460,nop,nop,sackOK> (DF)
12:34:14.981560 204.105.57.69.52433 > 10.25.227.13.1089: S 608260088:608260088(0) win 49640 <mss 1460,nop,nop,sackOK> (DF)
12:34:21.731098 205.105.57.69.52433 > 10.25.227.13.1089: S 608260088:608260088(0) win 49640 <mss 1460,nop,nop,sackOK> (DF)
12:34:24.648919 205.105.57.69.52889 > 10.25.227.13.1089: S 670970286:670970286(0) win 49640 <mss 1460,nop,nop,sackOK> (DF)
12:34:24.899705 205.105.57.69.52433 > 10.25.227.13.1089: R 608260089:608260089(0) win 0 (DF)
12:34:28.010299 205.105.57.69.52889 > 10.25.227.13.1089: S 670970286:670970286(0) win 49640 <mss 1460,nop,nop,sackOK> (DF)
[Expert@lansrv-fwa]# fw monitor | grep 10.210.7.250
monitor: getting filter (from command line)
monitor: compiling
monitorfilter:
Compiled OK.
monitor: loading
monitor: monitoring (control-C to stop)
eth10:i[142]: 10.210.7.250 -> 10.25.210.252 (TCP) len=142 id=35118
eth10:I[142]: 10.210.7.250 -> 10.25.210.252 (TCP) len=142 id=35118
eth8:o[142]: 10.210.7.250 -> 10.25.210.252 (TCP) len=142 id=35118
eth8:O[142]: 10.210.7.250 -> 10.25.210.252 (TCP) len=142 id=35118
eth8:i[93]: 10.210.0.203 -> 10.210.7.250 (TCP) len=93 id=47693
eth8:I[93]: 10.210.0.203 -> 10.210.7.250 (TCP) len=93 id=47693
eth10:o[93]: 10.210.0.203 -> 10.210.7.250 (TCP) len=93 id=47693
eth10:O[93]: 10.210.0.203 -> 10.210.7.250 (TCP) len=93 id=47693
eth10:i[40]: 10.210.7.250 -> 10.210.0.203 (TCP) len=40 id=35991
eth10:I[40]: 10.210.7.250 -> 10.210.0.203 (TCP) len=40 id=35991
[Expert@myfw-fwa]# fw monitor -ci 10 | grep 172.30.25.132
monitor: getting filter (from command line)
monitor: compiling
monitorfilter:
Compiled OK.
monitor: loading
monitor: monitoring (control-C to stop)
monitor: unloading
[fw_4] Exp2-1:o[124]: 172.30.25.132 -> 10.210.52.13 (TCP) len=124 id=1320
[fw_4] Exp2-1:O[124]: 172.30.25.132 -> 10.210.52.13 (TCP) len=124 id=1320
[fw_4] Exp2-1:i[40]: 10.210.52.13 -> 172.30.25.132 (TCP) len=40 id=21012
[fw_4] Exp2-1:I[40]: 10.210.52.13 -> 172.30.25.132 (TCP) len=40 id=21012
[Expert@hinoff-fwa.bcbsma.com]#
[Expert@myfw-fwa]# netstat -rn | wc -l
716
[Expert@myfw-fwa]# netstat -rn | grep -v D
Kernel IP routing table
135.89.101.234 172.30.25.116 255.255.255.255 UGH 0 0 0 eth2
10.25.1.254 172.30.25.113 255.255.255.255 UGH 0 0 0 eth2
10.25.1.227 172.30.25.116 255.255.255.255 UGH 0 0 0 eth2
10.25.1.226 172.30.25.113 255.255.255.255 UGH 0 0 0 eth2
[Expert@myfw-fwa]# tcpdump -ni eth8 src 172.30.25.132
tcpdump: listening on eth8
12:29:27.906458 172.30.25.132 > 10.210.7.250: icmp: echo request (DF)
12:29:28.906268 172.30.25.132 > 10.210.7.250: icmp: echo request (DF)
