Dhansham - Engineer's Notebook Checkpoint Firewalls Gaia: July 2016

fw mon

The following lists the order in which the rules/policies/ACLs (whatever you may call it) are enforced in Check Point R75:-

I was just about to put some FW Monitor templates on my blog for quick reference when I need to troubleshoot some issues in Check Point but I thought it would be a nice thing to explain this first (for myself too, as I keep forgetting this stuff :D).

When traffic flows through a Check Point Security Gateway (look here if you want to know about the architecture) it has to cross a series of inspection points. This post tries to explain what those inspection points are and how to troubleshoot traffic flows based on the inspection points. The next post will show how to use the FW Monitor.

Inspection Points (in the order they are passed):

1st	i – Pre-inbound	rule/inspection is applied at this point
2nd	I – Post-inbound	after being permitted by the rule/inspections
3rd	o – Pre-outbound	route look up has been performed to determine the egress interface
4th	O – Post-outbound	after egressing the correct firewall interface based on route lookup

The above image shows how a firewall kernel protects the Check Point OS. The start and end depict the start and end of the originating traffic flow. The return traffic too goes through the same inspection points. The below diagram shows how a complete two-way traffic is treated by the inspection points.

For a complete two-way communicating traffic you should see 8 logs (also depends if you do want to see 8 or less)

So looking at the above diagram you see that a complete traffic flow of a two-way communication goes through 8 inspection points;

Originating traffic from A to B = 4 logs of iIoO
and the return traffic from B to A = 4 logs of iIoO

I you don’t want to see what is going on at the inspection points ‘I’ and ‘o’ then you only grep ‘i’ and ‘O’ and get 4 logs in total for a complete two-way traffic;

Originating traffic from A to B = 2 logs of iO
and the return traffic from B to A = 2 logs of iO

Sometimes you won’t see 8 logs for a complete two-way communicating traffic and will only see 4 logs

If you are grepping iIoO in your captures and you still don’t see a ‘I’ or ‘o’, but you do see a ‘i’ and an ‘O’, then most likely SecureXL is fast switching/accelerating the traffic because the initial packets of that flow were already allowed.

You can disable SecureXL while setting captures if you want to inspect them at that granularity (i.e ‘I’ and ‘o’). In my experience, disabling SecureXL hasn’t been a problem* and I haven’t seen any performance impact as such.

Right before the capture, turn off SecureXL on the gateway:

fwaccel off

Immediately after the capture, turn on SecureXL on the gateway:

fwaccel on

To turn on fw accel on multiple VSXs at once:

fwaccel on -a

*Do it at your own risk, I do not take any responsibility for anything going wrong with this as it does affect the live traffic immediately. Read the site disclaimer. :)

Troubleshooting the inspection points

Troubleshooting ‘i’ – When you don’t see any traffic on the ‘i’ inspection point:-

Traffic isn’t hitting your firewall. You won’t get back any captures and the console will just be a black blank screen (or any other background color you use on your terminal :D).

Troubleshooting ‘I’ – When you see traffic hitting the ‘i’ inspection point but don’t see it hitting ‘I’:-

The firewall is blocking that ingress traffic either by a rule, improper NAT, IP spoofing, Application Inspection or anything else that you can think of.

Troubleshooting ‘o’ – When you see an ‘i’ and ‘I’ but don’t see an ‘o’:-

I’m not very sure on this but in one case that I observed, there was a route missing for the destination in the routing table, so the route look up failed to determine an egress interface and I didn’t see the traffic hitting ‘o’ and it was stuck at ‘I’.

Troubleshooting ‘O’ – When you see an ‘i’, ‘I’ and ‘o’ but not an ‘O’:-

I too don’t know, please tell me what that would be and I’ll update it here. :) I think it might be some kind of inspection or something. But I’m quite sure it will be a very rare case.

FW Monitor

FW Monitor is the tool that can be used to see your traffic flowing through different inspection points. FW Monitor cannot give you the information what SmartviewTracker/Monitor can, because it is a wire capture. It just tells you what is on the wires and does not look into the rule base to see which policy allowed the traffic and was it NAT’d or not. So the Smartview Tracker and Monitor serves a slightly different purpose than the FW Monitor.

I’m writing a separate post on how to use the FW Monitor at it’s best and in different ways so you can be better at troubleshooting traffic flow issues in Check Point.

___________________________________________________________________________________________________________________________________________________________________________

If you see any mistakes in the post or have any suggestions to add to this post, please feel free to comment. Thank you!

fw_monitor

So to start off with, lets see what the different fields are in the above picture of a sample log from the console;

Virtual Firewall System	This shows the VSX from which the log was captured from. Depending on your hardware/software platform you may or may not see this field.
Firewall interface	This is the interface where the traffic was seen and captured.
Inspection Point	This can be either (i), (I), (o) or (O) based on what you are grepping when you setup the fw monitor. Read more about inspection points here.
Packet Length	This is the size of the frame as seen “on the wire”
Source IP	The source
Destination IP	The destination
Header/Payload	The type of header on that packet. Can either be TCP/UDP or something else in similar lines.
Packet ID	This is a unique number assigned to a packet until it is changed/altered in its course. Quite important when it comes to reading the fw monitor logs on your the console. (I also want to say this is locally significant on the firewall, but not very sure about it.)
Source Port	The source port
Destination Port	The destination port
TCP Flags	One of the most important fields in my opinion. These are the TCP flags you have always know about. 6 bits are reserved for the TCP flags denoted by 6 dots (……) in the logs. Each dot is a TCP flag in this order FSRPA* = FIN-SYN-RST-PUSH-ACK-*** and I don’t know what the last one is, yet. So, you will usually see something like this for a SYN-ACK = .S..A. \| RST = ..R.. \| PUSH ACK = …PA. \| Hope I made this simple enough for you to be able to interpret the TCP flags.
Sequence Number	This is the sequence number that is the usual thing for TCP because of which it gets its name for being a reliable protocol.
Acknowledge Sequence Number	This is again a part of its reliable behavior where the other end host responds to denote that it has received its previous packet.

Now that we know which field means what, let’s see how all of this fit together; Below is a complete set of logs for a two-way communication between two hosts (from the time it originates and gets back a reply from the destination). I omitted the (I) and (o) to minimize the amount of logs here so we’re only inspecting (i) pre-inbound and (O) post-outbound. We’ll take our time to analyze the logs and see what each set of packet ID means.

Logs on the console:

[vs_4][fw_0] wrp256:i[48]: 10.1.1.1 -> 20.2.2.2 (TCP) len=48 id=27361
TCP: 59255 -> 443 .S.... seq=6cc07657 ack=00000000
[vs_4][fw_0] bond31.69:O[48]: 10.1.1.1 -> 20.2.2.2 (TCP) len=48 id=27361
TCP: 59255 -> 443 .S.... seq=6cc07657 ack=00000000
[vs_4][fw_0] bond31.69:i[48]: 20.2.2.2 -> 10.1.1.1 (TCP) len=48 id=44853
TCP: 443 -> 59255 .S..A. seq=37a73c7d ack=6cc07658
[vs_4][fw_0] wrp256:O[48]: 20.2.2.2 -> 10.1.1.1 (TCP) len=48 id=44853
TCP: 443 -> 59255 .S..A. seq=37a73c7d ack=6cc07658
[vs_4][fw_0] wrp256:i[40]: 10.1.1.1 -> 20.2.2.2 (TCP) len=40 id=27386
TCP: 59255 -> 443 ....A. seq=6cc07658 ack=37a73c7e
[vs_4][fw_0] bond31.69:O[40]: 10.1.1.1 -> 20.2.2.2 (TCP) len=40 id=27386
TCP: 59255 -> 443 ....A. seq=6cc07658 ack=37a73c7e
[vs_4][fw_0] wrp256:i[158]: 10.1.1.1 -> 20.2.2.2 (TCP) len=158 id=27396
TCP: 59255 -> 443 ...PA. seq=6cc07658 ack=37a73c7e
[vs_4][fw_0] bond31.69:O[158]: 10.1.1.1 -> 20.2.2.2 (TCP) len=158 id=27396
TCP: 59255 -> 443 ...PA. seq=6cc07658 ack=37a73c7e
[vs_4][fw_0] bond31.69:i[87]: 20.2.2.2 -> 10.1.1.1 (TCP) len=87 id=45365
TCP: 443 -> 59255 ...PA. seq=37a73c7e ack=6cc076ce
[vs_4][fw_0] bond31.69:i[633]: 20.2.2.2 -> 10.1.1.1 (TCP) len=633 id=45621
TCP: 443 -> 59255 ...PA. seq=37a73cad ack=6cc076ce
[vs_4][fw_0] wrp256:O[87]: 20.2.2.2 -> 10.1.1.1 (TCP) len=87 id=45365
TCP: 443 -> 59255 ...PA. seq=37a73c7e ack=6cc076ce
[vs_4][fw_0] wrp256:O[633]: 20.2.2.2 -> 10.1.1.1 (TCP) len=633 id=45621
TCP: 443 -> 59255 ...PA. seq=37a73cad ack=6cc076ce
[vs_4][fw_0] wrp256:i[40]: 10.1.1.1 -> 20.2.2.2 (TCP) len=40 id=27478
TCP: 59255 -> 443 ....A. seq=6cc07a18 ack=37a7436b
[vs_4][fw_0] bond31.69:O[40]: 10.1.1.1 -> 20.2.2.2 (TCP) len=40 id=27478
TCP: 59255 -> 443 ....A. seq=6cc07a18 ack=37a7436b
[vs_4][fw_0] wrp256:i[40]: 10.1.1.1 -> 20.2.2.2 (TCP) len=40 id=27479
TCP: 59255 -> 443 F...A. seq=6cc07a18 ack=37a7436b
[vs_4][fw_0] bond31.69:O[40]: 10.1.1.1 -> 20.2.2.2 (TCP) len=40 id=27479
TCP: 59255 -> 443 F...A. seq=6cc07a18 ack=37a7436b
[vs_4][fw_0] bond31.69:i[40]: 20.2.2.2 -> 10.1.1.1 (TCP) len=40 id=11794
TCP: 443 -> 59255 ....A. seq=37a7436b ack=6cc07a19
[vs_4][fw_0] wrp256:O[40]: 20.2.2.2 -> 10.1.1.1 (TCP) len=40 id=11794
TCP: 443 -> 59255 ....A. seq=37a7436b ack=6cc07a19

Each packet ID is shown at different inspection points depending on how the fw monitor is set. In our example we are looking for only (i) pre-inbound and (O) post-outbound so we see two logs with the same packet ID. If we were looking for all 4 inspection points (iIoO) we would’ve seen 4 such logs with the same packet ID. Makes sense? Okay, let’s proceed…

Starting with the first packet ID 27361:

[vs_4][fw_0] wrp256:i[48]: 10.1.1.1 -> 20.2.2.2 (TCP) len=48 id=27361
TCP: 59255 -> 443 .S.... seq=6cc07657 ack=00000000
[vs_4][fw_0] bond31.69:O[48]: 10.1.1.1 -> 20.2.2.2 (TCP) len=48 id=27361
TCP: 59255 -> 443 .S.... seq=6cc07657 ack=00000000

The first log with packet ID 27361 was seen on the virtual firewall 4 on ingress interface wrp256 on the (i) pre-inbound inspection point. Requestor 10.1.1.1 has sent a SYN (.S….) to 20.2.2.2 to setup an HTTPS (443) connection. In the second log with the same packet ID we see the exact same information but the inspection point has changed here to (O) and the egress interface to bond31.69. This tells us the SYN packet was successfully passed through the firewall towards the destination IP. Keep an eye on the ‘seq’ and ‘ack’ numbers from now on.

Next packet ID 44853;

[vs_4][fw_0] bond31.69:i[48]: 20.2.2.2 -> 10.1.1.1 (TCP) len=48 id=44853T
CP: 443 -> 59255 .S..A. seq=37a73c7d ack=6cc07658[
vs_4][fw_0] wrp256:O[48]: 20.2.2.2 -> 10.1.1.1 (TCP) len=48 id=44853
TCP: 443 -> 59255 .S..A. seq=37a73c7d ack=6cc07658

This is the SYN-ACK (.S..A.) that responder 20.2.2.2 is sending back to the requestor 10.1.1.1. Notice how the ingress/egress interfaces are swapped now, since this is the exact opposite traffic flow, obviously. Now carefully compare the seq and ack number with that of the first packet ID 27361. Packet 27361 did not have an ack number since it was the first packet that was being sent but it did have a seq number 6cc07657 to which 44853 replied back incrementing it by ‘1’ in its ack – 6cc07658. It has also sent a seq number of 37a73c7d in the SYN-ACK, so in the next packet that will be an ACK from the requestor, we should see the ack number of 37a73c7e.

Next packet ID 27386;

[vs_4][fw_0] wrp256:i[40]: 10.1.1.1 -> 20.2.2.2 (TCP) len=40 id=27386
TCP: 59255 -> 443 ....A. seq=6cc07658 ack=37a73c7e
[vs_4][fw_0] bond31.69:O[40]: 10.1.1.1 -> 20.2.2.2 (TCP) len=40 id=27386
TCP: 59255 -> 443 ....A. seq=6cc07658 ack=37a73c7e

Look at the ack number 37a73c7e, it has incremented by 1 to the seq number of the previous packet. This keeps on going back and forth until this session is torn down. So if you don’t see proper seq and ack numbers in sync, it should tell you that you are missing some packets (that would result in seeing many re-transmissions and out-of-order logs in wireshark). So this is an ACK (….A.) response by the initiator (10.1.1.1) which now completes the TCP three-way handshake.

Next packet ID is 27396;

[vs_4][fw_0] wrp256:i[158]: 10.1.1.1 -> 20.2.2.2 (TCP) len=158 id=27396
TCP: 59255 -> 443 ...PA. seq=6cc07658 ack=37a73c7e
[vs_4][fw_0] bond31.69:O[158]: 10.1.1.1 -> 20.2.2.2 (TCP) len=158 id=27396
TCP: 59255 -> 443 ...PA. seq=6cc07658 ack=37a73c7e

Immediately after the three-way handshake, the requester sends its HTTPS request to the responder that we see in the PUSH-ACK messages (…PA.). You will also notice the packet length has changed since it now contains a good amount of payload (len=158).

Next packet IDs are 45365 and 45621;

[vs_4][fw_0] bond31.69:i[87]: 20.2.2.2 -> 10.1.1.1 (TCP) len=87 id=45365
TCP: 443 -> 59255 ...PA. seq=37a73c7e ack=6cc076ce
[vs_4][fw_0] bond31.69:i[633]: 20.2.2.2 -> 10.1.1.1 (TCP) len=633 id=45621
TCP: 443 -> 59255 ...PA. seq=37a73cad ack=6cc076ce
[vs_4][fw_0] wrp256:O[87]: 20.2.2.2 -> 10.1.1.1 (TCP) len=87 id=45365
TCP: 443 -> 59255 ...PA. seq=37a73c7e ack=6cc076ce
[vs_4][fw_0] wrp256:O[633]: 20.2.2.2 -> 10.1.1.1 (TCP) len=633 id=45621
TCP: 443 -> 59255 ...PA. seq=37a73cad ack=6cc076ce

The responder has replied back as we see the PUSH-ACKs going back to the requester and the packet length has a significant value.

Next packet ID is 27478;

[vs_4][fw_0] wrp256:i[40]: 10.1.1.1 -> 20.2.2.2 (TCP) len=40 id=27478
TCP: 59255 -> 443 ....A. seq=6cc07a18 ack=37a7436b
[vs_4][fw_0] bond31.69:O[40]: 10.1.1.1 -> 20.2.2.2 (TCP) len=40 id=27478
TCP: 59255 -> 443 ....A. seq=6cc07a18 ack=37a7436b

The requester 10.1.1.1 has replied back with an ACK to the HTTPS response it received from the responder in the earlier packets that were seen.

Next packet ID is 27479;

[vs_4][fw_0] wrp256:i[40]: 10.1.1.1 -> 20.2.2.2 (TCP) len=40 id=27479
TCP: 59255 -> 443 F...A. seq=6cc07a18 ack=37a7436b
[vs_4][fw_0] bond31.69:O[40]: 10.1.1.1 -> 20.2.2.2 (TCP) len=40 id=27479
TCP: 59255 -> 443 F...A. seq=6cc07a18 ack=37a7436b

The requester then sends a FIN-ACK telling the responder I’m done with my requests and I’ll close this session from my end.

Next packet ID is 11794;

[vs_4][fw_0] bond31.69:i[40]: 20.2.2.2 -> 10.1.1.1 (TCP) len=40 id=11794
TCP: 443 -> 59255 ....A. seq=37a7436b ack=6cc07a19
[vs_4][fw_0] wrp256:O[40]: 20.2.2.2 -> 10.1.1.1 (TCP) len=40 id=11794
TCP: 443 -> 59255 ....A. seq=37a7436b ack=6cc07a19

The responder acknowledges the closing of the session too and the connection comes to an end. Pheww.

Throughout all this traffic going back and forth, you want to keep an eye on the TCP flags as that is where you will see the most problems. Also when you want to troubleshoot packet loss, seq/ack numbers are your best friend and you want to track them very diligently.

Tuesday, July 5, 2016

Firewall Debugging

In certain specific scenarios, the debugging commands included in this document may need to be supplemented by more advanced debugging procedures.
Advanced procedures should be executed in conjunction with the Check Point Escalation engineers.
Debugging should only be performed when the described issue can be captured.

FireWall Common debugging

Kernel debugging

Usage

% fw ctl debug -buf [buffer size]

% fw ctl debug [-x] [-m <module>] [+|-] <options | all | 0>

% fw ctl kdebug –f > <output file>

To disable the Kernel debugging, execute:

% fw ctl debug –buf 0

% fw ctl debug x

Common Syntax

% fw ctl debug –buf 12288

% fw ctl debug –m fw conn drop ld packet if

% fw ctl kdebug –f > <output file>

The ld option may cause high CPU usage. It is advised to use it for short session debugging

only.

To execute the kernel you can also use fw ctl zdebug to allocate the buffer (where the

buffer can only be 1024).

% fw ctl zdebug

% fw ctl kdebug -f > <output file>

User Mode Processes debugging

Usage

% fw debug <process name> <on/off> TDERROR_ALL_ALL=<value 1-5>

CPD is treated differently from the other User Mode processes and will be executed

differently,

Debugging CPD

CPD is a high in the hierarchichal chain and helps to execute many services, such as Secure

Internal Communcation (SIC), Licensing and status report.

For CPD debug, execute: % cpd_admin debug on TDERROR_ALL_ALL=5

The debug file is located under $CPDIR/log/cpd.elg

To stop the CPD debug, execute: % cpd_admin debug off TDERROR_ALL_ALL=1

Debugging FWM

The FWM process is responsible for the execution of the database activities of the

SmartCenter server. It is; therefore, responsible for Policy installation, Management High

Availability (HA) Synchronization, saving the Policy, Database Read/Write action, Log

Display, etc.

For FWM debug, execute:

% fw debug fwm on TDERROR_ALL_ALL=5

% fw debug fwm on OPSEC_DEBUG_LEVEL=9

The debug file is located under $FWDIR/log/fwm.elg

To stop the FWM debug, execute:

% fw debug fwm off TDERROR_ALL_ALL=1

% fw debug fwm off OPSEC_DEBUG_LEVEL=1

Debugging FWD

The FWD process is responsible for logging. It is executed in relation to logging, Security

Servers and communication with OPSEC applications.

For FWD debug, execute: % fw debug fwd debug on TDERROR_ALL_ALL=5

The debug file is located under $FWDIR/log/fwd.elg

To stop the FWD debug, execute: % fw debug fwd off TDERROR_ALL_ALL=1

FireWall Monitor Network Capturing

The FireWall Monitor is responsible for packet flow analysis.

To execute: % fw monitor –e “accept;” –o <output file>

ClusterXL debugging

For ClusterXL debugging for Clustering, Synchronization, High Availability, Fail-over,

execute:

% cphaprob state

% cphaprob -ia list

% cphaprob -a if

% fw ctl pstat

Kernel debug for packet filter analysis

% fw ctl debug –buf 12288

% fw ctl debug –m fw conn drop packet if sync

% fw ctl debug –m cluster all

% fw ctl kdebug –f > <output file>

Disable SecureXL for capturing proper logs:

If you have SecureXL enabled, it fast switches the packets and won’t show you all the detailed logs that you would like to see in your captures. You can disable SecureXL temporarily, if you want to inspect packets at that granularity (i.e ‘I’ and ‘o’). In my experience, disabling SecureXL hasn’t been a problem* and I haven’t seen any performance impact as such.

Right before the capture, turn off SecureXL on the gateway:

fwaccel off

Immediately after the capture, turn on SecureXL on the gateway:

fwaccel on

To turn on/off SecureXL on multiple VSXs at once:

fwaccel on -a fwaccel off -a

Ensure that you re-enable after you are done capturing the logs.
Be aware that SecureXL does affect how the traffic passes through the firewall and only disable it if you are 100% sure of what you’re doing. Do it at your own risk, I do not take any responsibility for anything going wrong with this as it does affect the live traffic immediately. Read the blog’s disclaimer. :)

Display all 8 inspection points (iIoO) for a complete two-way traffic on the console:

For specific flow:
fw monitor -e 'accept (src=10.1.1.1 and dst=20.2.2.2) or (src=20.2.2.2 and dst=10.1.1.1);' -m iIoO
OR
For specific IP:
fw monitor -e 'accept (src=10.1.1.1 or dst=10.1.1.1);' -m iIoO

Display 4 inspection points (iO) for a complete two-way traffic on the console:

fw monitor -e 'accept (src=10.1.1.1 and dst=20.2.2.2) or (src=20.2.2.2 and dst=10.1.1.1);' -m iO

Note that we are only grepping iO, that means you are not looking for what is going on at the post-inbound (I) and pre-outbound (o) inspection points and just trying to get fewer output to see if it actually leaves the egress interface or not.
If you suspect something missing, then run -m iIoO and see at which inspection point it’s getting stuck and use the previous post for troubleshooting tips.

Saving fw monitor logs to a .pcap file to analyse in wireshark:

fw monitor -e 'accept (src=10.1.1.1 and dst=20.2.2.2) or (src=20.2.2.2 and dst=10.1.1.1);' -m iIoO -o wireshark.pcap

By default, the capture file is saved to the /home/admin directory.
Use WinSCP to access the Security Gateway and copy the file to your local drive to analyze it in Wireshark.

Saving a TCP dump in a .pcap file:

tcpdump -w capture.pcap -i eth-s1p2c0 host 10.1.1.1 and host 20.2.2.2
tcpdump -nni any host 10.1.1.1 -w capture.pcap
tcpdump -nni any host 10.1.1.1 and host 20.2.2.2 -w capture.pcap

Replace the interface name and capture name as required.

Switches:

Ctrl+C	This is the break sequence
-e	Custom expression. Sets the filter for fw monitor.
-o	Writes the raw packet data output to a file.
-ci -co	Captures only the number of inbound (ci) and outbound (co) packets. This is useful if you have many packets going between a source and destination and the break sequence can take time to stop those captures.-vSpecifies the virtual device where the fw monitor needs to run

Chapter 2 Chain Module

Check Point kernel is the bridge between the hardware and the OS.
The inspection operations in the kernel are divided into modules, and the modules are divided into chains.
The number of chains on every Security Gateway is different. It depends on which blades/features are enabled on the Security Gateway.
Each direction has its own modules and order of inspection.
Handlers (INSPECT code) decide which modules will inspect the packet.
The kernel consist of inbound (ingress) and outbound (egress) Chain Modules.
It inspects packets in both directions.
Each packet passes through a list of Inbound and Outbound chain modules which may modify pass or drop packets.
They are responsible for decryption, encryption, rule matching, and policy verification.
Chain Positions is an absolute number and never change
Each chain module is independent and redundant
They execute their own unique inspection.
Inspection is performed on virtually fragmented packets.
In (VPN) firewall kernel, each kernel chain is associated with a key. The key specify the type of traffic applicable to this chain.

fw monitor inserts its own modules in this module chain and is capturing packets there. By default this is not the first and last position in the chain.

To see all active chains in the Security Gateway, run:

fw ctl chain

Security Gateway

User Mode (CPD/FWD/WPND/FWSSD/CPWD)
OS IP Stack
Firewall-1 Kernel (Inbound/Outbound Chain)
NIC Hardware Layer NIC

Use "fw ctl chain" command to see this list of Chain Modules

Module Location in the chain (Serial Number)
Chain Position
Function Pointer
Mode in which the chain applies:
1. - Stateful mode
2. - Wired mode
3. - All Packets
fff - All packets (identical to 3)

[Expert@bostestint-fwa:0]# fw ctl chain
in chain (11):
0: -7f800000 (ffffffff8882b820) (ffffffff) IP Options Strip (in) (ipopt_strip)
1: - 1fffff8 (ffffffff8882d050) (00000001) Stateless verifications (in) (asm)
2: - 1fffff7 (ffffffff8886eb00) (00000001) fw multik misc proto forwarding
3: - 1000000 (ffffffff88911d10) (00000003) SecureXL conn sync (secxl_sync)
4: 0 (ffffffff887ceee0) (00000001) fw VM inbound (fw)
5: 10 (ffffffff887e3690) (00000001) fw accounting inbound (acct)
6: 10000000 (ffffffff88910330) (00000003) SecureXL inbound (secxl)
7: 7f600000 (ffffffff88820bf0) (00000001) fw SCV inbound (scv)
8: 7f730000 (ffffffff88a35fb0) (00000001) passive streaming (in) (pass_str)
9: 7f750000 (ffffffff88c53260) (00000001) TCP streaming (in) (cpas)
10: 7f800000 (ffffffff8882bbe0) (ffffffff) IP Options Restore (in) (ipopt_res)
out chain (10):
0: -7f800000 (ffffffff8882b820) (ffffffff) IP Options Strip (out) (ipopt_strip)
1: - 1fffff0 (ffffffff88c534a0) (00000001) TCP streaming (out) (cpas)
2: - 1ffff50 (ffffffff88a35fb0) (00000001) passive streaming (out) (pass_str)
3: - 1f00000 (ffffffff8882d050) (00000001) Stateless verifications (out) (asm)
4: - 1ff (ffffffff88e6eb80) (00000001) NAC Packet Outbound (nac_tag)
5: 0 (ffffffff887ceee0) (00000001) fw VM outbound (fw)
6: 10000000 (ffffffff88910330) (00000003) SecureXL outbound (secxl)
7: 7f000000 (ffffffff887e3690) (00000001) fw accounting outbound (acct)
8: 7f700000 (ffffffff88c53690) (00000001) TCP streaming post VM (cpas)
9: 7f800000 (ffffffff8882bbe0) (ffffffff) IP Options Restore (out) (ipopt_res)
[Expert@bostestint-fwa:0]#

Chain Insertion Points

FireWall-1 kernel module between Layer 2 NIC driver and Layer 3 IP stack

fw monitor uses its own kernel module to capture packets

it capture packets in 4 positions

Capture position fw monitor mask value

pre-inbound i (lowercase i)

post-inbound I (uppercase i)

pre-outbound o (lowercase o)

post-outbound O (uppercase o)

[Expert@mytestint-fwa:0]# fw ctl chain

in chain (11):

0: -7f800000 (ffffffff8882b820) (ffffffff) IP Options Strip (in) (ipopt_strip)

1: - 1fffff8 (ffffffff8882d050) (00000001) Stateless verifications (in) (asm)

2: - 1fffff7 (ffffffff8886eb00) (00000001) fw multik misc proto forwarding

3: - 1000000 (ffffffff88911d10) (00000003) SecureXL conn sync (secxl_sync)

4: 0 (ffffffff887ceee0) (00000001) fw VM inbound (fw)

5: 10 (ffffffff887e3690) (00000001) fw accounting inbound (acct)

6: 10000000 (ffffffff88910330) (00000003) SecureXL inbound (secxl)

7: 7f600000 (ffffffff88820bf0) (00000001) fw SCV inbound (scv)

8: 7f730000 (ffffffff88a35fb0) (00000001) passive streaming (in) (pass_str)

9: 7f750000 (ffffffff88c53260) (00000001) TCP streaming (in) (cpas)

10: 7f800000 (ffffffff8882bbe0) (ffffffff) IP Options Restore (in) (ipopt_res)

out chain (10):

0: -7f800000 (ffffffff8882b820) (ffffffff) IP Options Strip (out) (ipopt_strip)

1: - 1fffff0 (ffffffff88c534a0) (00000001) TCP streaming (out) (cpas)

2: - 1ffff50 (ffffffff88a35fb0) (00000001) passive streaming (out) (pass_str)

3: - 1f00000 (ffffffff8882d050) (00000001) Stateless verifications (out) (asm)

4: - 1ff (ffffffff88e6eb80) (00000001) NAC Packet Outbound (nac_tag)

5: 0 (ffffffff887ceee0) (00000001) fw VM outbound (fw)

6: 10000000 (ffffffff88910330) (00000003) SecureXL outbound (secxl)

7: 7f000000 (ffffffff887e3690) (00000001) fw accounting outbound (acct)

8: 7f700000 (ffffffff88c53690) (00000001) TCP streaming post VM (cpas)

9: 7f800000 (ffffffff8882bbe0) (ffffffff) IP Options Restore (out) (ipopt_res)

[Expert@mytestint-fwa:0]#

[Expert@mytestint-fwa:0]# fw ctl chain

in chain (13):

0: -7f800000 (ffffffff8882b820) (ffffffff) IP Options Strip (in) (ipopt_strip)

1: -70000000 (ffffffff88804d70) (ffffffff) fwmonitor (i/f side)

2: - 1fffff8 (ffffffff8882d050) (00000001) Stateless verifications (in) (asm)

3: - 1fffff7 (ffffffff8886eb00) (00000001) fw multik misc proto forwarding

4: - 1000000 (ffffffff88911d10) (00000003) SecureXL conn sync (secxl_sync)

5: 0 (ffffffff887ceee0) (00000001) fw VM inbound (fw)

6: 10 (ffffffff887e3690) (00000001) fw accounting inbound (acct)

7: 10000000 (ffffffff88910330) (00000003) SecureXL inbound (secxl)

8: 70000000 (ffffffff88804d70) (ffffffff) fwmonitor (IP side)

9: 7f600000 (ffffffff88820bf0) (00000001) fw SCV inbound (scv)

10: 7f730000 (ffffffff88a35fb0) (00000001) passive streaming (in) (pass_str)

11: 7f750000 (ffffffff88c53260) (00000001) TCP streaming (in) (cpas)

12: 7f800000 (ffffffff8882bbe0) (ffffffff) IP Options Restore (in) (ipopt_res)

out chain (12):

0: -7f800000 (ffffffff8882b820) (ffffffff) IP Options Strip (out) (ipopt_strip)

1: -70000000 (ffffffff88804d70) (ffffffff) fwmonitor (i/f side)

2: - 1fffff0 (ffffffff88c534a0) (00000001) TCP streaming (out) (cpas)

3: - 1ffff50 (ffffffff88a35fb0) (00000001) passive streaming (out) (pass_str)

4: - 1f00000 (ffffffff8882d050) (00000001) Stateless verifications (out) (asm)

5: - 1ff (ffffffff88e6eb80) (00000001) NAC Packet Outbound (nac_tag)

6: 0 (ffffffff887ceee0) (00000001) fw VM outbound (fw)

7: 10000000 (ffffffff88910330) (00000003) SecureXL outbound (secxl)

8: 70000000 (ffffffff88804d70) (ffffffff) fwmonitor (IP side)

9: 7f000000 (ffffffff887e3690) (00000001) fw accounting outbound (acct)

10: 7f700000 (ffffffff88c53690) (00000001) TCP streaming post VM (cpas)

11: 7f800000 (ffffffff8882bbe0) (ffffffff) IP Options Restore (out) (ipopt_res)

[Expert@mytestint-fwa:0]#

Insert fw monitor into the chain to report on all packets. It may be inserted into the chain at the following four positions.

inbound minus 0x70000000

Inbound 0x70000000

outbound minus 0x70000000

Outbound 0x70000000

To put the monitor in a certain place use the -p parameter and the number of item that will be after fw monitor.

To put the monitor in the end of the chain, use -pI 999 or -pO 999. This -p parameter has the following syntax:

fw monitor - p [i|I|o|O] absolute pos| relative pos| [+ |-} alias]

fw ctl chain with FW Monitor inserted

fw monior captures packets as they enter or leave the firewall kernel and when the packet enters and leaves the inbound and outbound chain.

With fw monitor running once fw monitor is executed, a specified INSPECT filter is compiled an loaded into kernel. Any parameters following "accept" in the fw monitor command will be displayed by the fw monitor. The same filter is applied to all interface in all direction.

fw monitor output uses specific expressions to explain the location of the packet as it moves through the firewall. There are four inspection points as a packet passes through the kernel (or Virtual Machine)

i Before the virtual machine, in the inbound direction (pre-inbound)

I After virtual machine in the inbound direction (post -inbound)

o Before the virtual machine, in the outbound direction (pre-outbound)

O After virtual machine in the outbound direction (post -outbound)

In client to server scenario,

'i' represent the packet as it left the client.

'I' represent the packet already checked against the tables and rule base.

in case of static-nat, the destination address will be changed.

'o' means the packet is before the oubound kernel (same as 'I') and

'O' means the packet is in the outbound kernel chain. as it appears at the webserver.

in the case of hide-nat the source IP address will be different.

The general syntax:

fw monitor -e "accept <expression>;"

[Expert@bostestint-fwa:0]# fw ctl chain

in chain (13):

0: -7f800000 (ffffffff8882b820) (ffffffff) IP Options Strip (in) (ipopt_strip)

1: -70000000 (ffffffff88804d70) (ffffffff) fwmonitor (i/f side)

2: - 1fffff8 (ffffffff8882d050) (00000001) Stateless verifications (in) (asm)

3: - 1fffff7 (ffffffff8886eb00) (00000001) fw multik misc proto forwarding

4: - 1000000 (ffffffff88911d10) (00000003) SecureXL conn sync (secxl_sync)

5: 0 (ffffffff887ceee0) (00000001) fw VM inbound (fw)

6: 10 (ffffffff887e3690) (00000001) fw accounting inbound (acct)

7: 10000000 (ffffffff88910330) (00000003) SecureXL inbound (secxl)

8: 70000000 (ffffffff88804d70) (ffffffff) fwmonitor (IP side)

9: 7f600000 (ffffffff88820bf0) (00000001) fw SCV inbound (scv)

10: 7f730000 (ffffffff88a35fb0) (00000001) passive streaming (in) (pass_str)

11: 7f750000 (ffffffff88c53260) (00000001) TCP streaming (in) (cpas)

12: 7f800000 (ffffffff8882bbe0) (ffffffff) IP Options Restore (in) (ipopt_res)

out chain (12):

0: -7f800000 (ffffffff8882b820) (ffffffff) IP Options Strip (out) (ipopt_strip)

1: -70000000 (ffffffff88804d70) (ffffffff) fwmonitor (i/f side)

2: - 1fffff0 (ffffffff88c534a0) (00000001) TCP streaming (out) (cpas)

3: - 1ffff50 (ffffffff88a35fb0) (00000001) passive streaming (out) (pass_str)

4: - 1f00000 (ffffffff8882d050) (00000001) Stateless verifications (out) (asm)

5: - 1ff (ffffffff88e6eb80) (00000001) NAC Packet Outbound (nac_tag)

6: 0 (ffffffff887ceee0) (00000001) fw VM outbound (fw)

7: 10000000 (ffffffff88910330) (00000003) SecureXL outbound (secxl)

8: 70000000 (ffffffff88804d70) (ffffffff) fwmonitor (IP side)

9: 7f000000 (ffffffff887e3690) (00000001) fw accounting outbound (acct)

10: 7f700000 (ffffffff88c53690) (00000001) TCP streaming post VM (cpas)

11: 7f800000 (ffffffff8882bbe0) (ffffffff) IP Options Restore (out) (ipopt_res)

[Expert@bostestint-fwa:0]#

[Expert@myfwl-int01:0]# fw ctl chain

in chain (12):

0: -7f800000 (ffffffff8882e8e0) (ffffffff) IP Options Strip (in) (ipopt_strip)

1: - 1fffff8 (ffffffff88830110) (00000001) Stateless verifications (in) (asm)

2: - 1fffff7 (ffffffff88871ce0) (00000001) fw multik misc proto forwarding

3: - 1000000 (ffffffff88914f00) (00000003) SecureXL conn sync (secxl_sync)

4: 0 (ffffffff887d1f00) (00000001) fw VM inbound (fw)

5: 10 (ffffffff887e66d0) (00000001) fw accounting inbound (acct)

6: 10000000 (ffffffff88913520) (00000003) SecureXL inbound (secxl)

7: 7f600000 (ffffffff88823cb0) (00000001) fw SCV inbound (scv)

8: 7f730000 (ffffffff88a39b10) (00000001) passive streaming (in) (pass_str)

9: 7f750000 (ffffffff88c574f0) (00000001) TCP streaming (in) (cpas)

10: 7f800000 (ffffffff8882eca0) (ffffffff) IP Options Restore (in) (ipopt_res)

11: 7fb00000 (ffffffff890210c0) (00000001) HA Forwarding (ha_for)

out chain (10):

0: -7f800000 (ffffffff8882e8e0) (ffffffff) IP Options Strip (out) (ipopt_strip)

1: - 1fffff0 (ffffffff88c57730) (00000001) TCP streaming (out) (cpas)

2: - 1ffff50 (ffffffff88a39b10) (00000001) passive streaming (out) (pass_str)

3: - 1f00000 (ffffffff88830110) (00000001) Stateless verifications (out) (asm)

4: - 1ff (ffffffff88e76ae0) (00000001) NAC Packet Outbound (nac_tag)

5: 0 (ffffffff887d1f00) (00000001) fw VM outbound (fw)

6: 10000000 (ffffffff88913520) (00000003) SecureXL outbound (secxl)

7: 7f000000 (ffffffff887e66d0) (00000001) fw accounting outbound (acct)

8: 7f700000 (ffffffff88c57920) (00000001) TCP streaming post VM (cpas)

9: 7f800000 (ffffffff8882eca0) (ffffffff) IP Options Restore (out) (ipopt_res)

[Expert@myfwl-int01:0]#

Stateful Features

Streaming based application (web security etc)
Sequence verification and translation
Hide NAT (when server to client packets returning to firewall might not match the rule base, need to add explicit entry to connection table)
Logging, Accounting, monitoring, etc
Client and server identication
Data connections.

fw tab -s
fw tab -t
[Expert@mytestint-fwa:0]# fw tab -t connections -s
HOST NAME ID #VALS #PEAK #SLINKS
localhost connections 8158 47 1223 49
[Expert@mytestint-fwa:0]#

[Expert@myvpn-fwb:0]# fw ctl chain

in chain (20):

0: -7ffffff0 (ffffffff8ab199f0) (00000001) tcpt inbound (tcp_tun)

1: -7f800000 (ffffffff88c17f30) (ffffffff) IP Options Strip (in) (ipopt_strip)

2: -7d000000 (ffffffff8ab22f90) (00000003) vpn multik forward in

3: - 2000000 (ffffffff8ab06690) (00000003) vpn decrypt (vpn)

4: - 1fffff8 (ffffffff8ab121a0) (00000001) l2tp inbound (l2tp)

5: - 1fffff6 (ffffffff88c19760) (00000001) Stateless verifications (in) (asm)

6: - 1fffff5 (ffffffff88c51a60) (00000001) fw multik misc proto forwarding

7: - 1fffff2 (ffffffff8ab30a60) (00000003) vpn tagging inbound (tagging)

8: - 1fffff0 (ffffffff8ab05290) (00000003) vpn decrypt verify (vpn_ver)

9: - 1000000 (ffffffff88c9c440) (00000003) SecureXL conn sync (secxl_sync)

10: 0 (ffffffff88bc0650) (00000001) fw VM inbound (fw)

11: 1 (ffffffff88c3ad10) (00000002) wire VM inbound (wire_vm)

12: 10 (ffffffff88bd4400) (00000001) fw accounting inbound (acct)

13: 2000000 (ffffffff8ab04900) (00000003) vpn policy inbound (vpn_pol)

14: 10000000 (ffffffff88c9acd0) (00000003) SecureXL inbound (secxl)

15: 7f600000 (ffffffff88c0d3f0) (00000001) fw SCV inbound (scv)

16: 7f730000 (ffffffff88db3050) (00000001) passive streaming (in) (pass_str)

17: 7f750000 (ffffffff88fbfbd0) (00000001) TCP streaming (in) (cpas)

18: 7f800000 (ffffffff88c182f0) (ffffffff) IP Options Restore (in) (ipopt_res)

19: 7fb00000 (ffffffff8936dca0) (00000001) HA Forwarding (ha_for)

out chain (18):

0: -7f800000 (ffffffff88c17f30) (ffffffff) IP Options Strip (out) (ipopt_strip)

1: -78000000 (ffffffff8ab22f70) (00000003) vpn multik forward out

2: - 1ffffff (ffffffff8ab04140) (00000003) vpn nat outbound (vpn_nat)

3: - 1fffff0 (ffffffff88fbfa50) (00000001) TCP streaming (out) (cpas)

4: - 1ffff50 (ffffffff88db3050) (00000001) passive streaming (out) (pass_str)

5: - 1ff0000 (ffffffff8ab30a60) (00000003) vpn tagging outbound (tagging)

6: - 1f00000 (ffffffff88c19760) (00000001) Stateless verifications (out) (asm)

7: - 1ff (ffffffff891d42e0) (00000001) NAC Packet Outbound (nac_tag)

8: 0 (ffffffff88bc0650) (00000001) fw VM outbound (fw)

9: 1 (ffffffff88c3ad10) (00000002) wire VM outbound (wire_vm)

10: 2000000 (ffffffff8ab04150) (00000003) vpn policy outbound (vpn_pol)

11: 10000000 (ffffffff88c9acd0) (00000003) SecureXL outbound (secxl)

12: 1ffffff0 (ffffffff8ab12ee0) (00000001) l2tp outbound (l2tp)

13: 20000000 (ffffffff8ab07120) (00000003) vpn encrypt (vpn)

14: 60000000 (ffffffff8ab1a130) (00000001) tcpt outbound (tcp_tun)

15: 7f000000 (ffffffff88bd4400) (00000001) fw accounting outbound (acct)

16: 7f700000 (ffffffff88fc2070) (00000001) TCP streaming post VM (cpas)

17: 7f800000 (ffffffff88c182f0) (ffffffff) IP Options Restore (out) (ipopt_res)

[Expert@myvpn-fwb:0]#

Inbound
-------
NIC
Wireside Acct/Virtual Reass IP Options Strip (in) (ipopt_strip)
VPN Dec
VPN verify
VM/NAT
Accounting
VPN Policy
FG Policy
IQ Engine
RTM/E2E
TCP/IP

Outbound
-------
TCP/IP
Virtual Reass/Wireside Acct
VM/NAT
VPN Policy
FG Policy
VPN Enc
IQ Engine
Accounting
RTM/E2E
NIC

Relative Positions

To Insert fw monitor after TCP streaming (out) for the outbound chain, execute the following command
fw monitor -po -0x1ffffe0

[Expert@mytestint-fwa:0]# fw ctl chain
in chain (11):
0: -7f800000 (ffffffff8882b820) (ffffffff) IP Options Strip (in) (ipopt_strip)
1: - 1fffff8 (ffffffff8882d050) (00000001) Stateless verifications (in) (asm)
2: - 1fffff7 (ffffffff8886eb00) (00000001) fw multik misc proto forwarding
3: - 1000000 (ffffffff88911d10) (00000003) SecureXL conn sync (secxl_sync)
4: 0 (ffffffff887ceee0) (00000001) fw VM inbound (fw)
5: 10 (ffffffff887e3690) (00000001) fw accounting inbound (acct)
6: 10000000 (ffffffff88910330) (00000003) SecureXL inbound (secxl)
7: 7f600000 (ffffffff88820bf0) (00000001) fw SCV inbound (scv)
8: 7f730000 (ffffffff88a35fb0) (00000001) passive streaming (in) (pass_str)
9: 7f750000 (ffffffff88c53260) (00000001) TCP streaming (in) (cpas)
10: 7f800000 (ffffffff8882bbe0) (ffffffff) IP Options Restore (in) (ipopt_res)
out chain (10):
0: -7f800000 (ffffffff8882b820) (ffffffff) IP Options Strip (out) (ipopt_strip)
1: - 1fffff0 (ffffffff88c534a0) (00000001) TCP streaming (out) (cpas)
2: - 1ffff50 (ffffffff88a35fb0) (00000001) passive streaming (out) (pass_str)
3: - 1f00000 (ffffffff8882d050) (00000001) Stateless verifications (out) (asm)
4: - 1ff (ffffffff88e6eb80) (00000001) NAC Packet Outbound (nac_tag)
5: 0 (ffffffff887ceee0) (00000001) fw VM outbound (fw)
6: 10000000 (ffffffff88910330) (00000003) SecureXL outbound (secxl)
7: 7f000000 (ffffffff887e3690) (00000001) fw accounting outbound (acct)
8: 7f700000 (ffffffff88c53690) (00000001) TCP streaming post VM (cpas)
9: 7f800000 (ffffffff8882bbe0) (ffffffff) IP Options Restore (out) (ipopt_res)
[Expert@mytestint-fwa:0]#
[Expert@mytestint-fwa:0]#
[Expert@mytestint-fwa:0]#

[Expert@mytestint-fwa:0]# fw monitor -po -0x1ffffe0

[Expert@mytestint-fwa:0]# fw ctl chain
in chain (13):
0: -7f800000 (ffffffff8882b820) (ffffffff) IP Options Strip (in) (ipopt_strip)
1: -70000000 (ffffffff88804d70) (ffffffff) fwmonitor (i/f side)
2: - 1fffff8 (ffffffff8882d050) (00000001) Stateless verifications (in) (asm)
3: - 1fffff7 (ffffffff8886eb00) (00000001) fw multik misc proto forwarding
4: - 1000000 (ffffffff88911d10) (00000003) SecureXL conn sync (secxl_sync)
5: 0 (ffffffff887ceee0) (00000001) fw VM inbound (fw)
6: 10 (ffffffff887e3690) (00000001) fw accounting inbound (acct)
7: 10000000 (ffffffff88910330) (00000003) SecureXL inbound (secxl)
8: 70000000 (ffffffff88804d70) (ffffffff) fwmonitor (IP side)
9: 7f600000 (ffffffff88820bf0) (00000001) fw SCV inbound (scv)
10: 7f730000 (ffffffff88a35fb0) (00000001) passive streaming (in) (pass_str)
11: 7f750000 (ffffffff88c53260) (00000001) TCP streaming (in) (cpas)
12: 7f800000 (ffffffff8882bbe0) (ffffffff) IP Options Restore (in) (ipopt_res)
out chain (12):
0: -7f800000 (ffffffff8882b820) (ffffffff) IP Options Strip (out) (ipopt_strip)
1: - 1fffff0 (ffffffff88c534a0) (00000001) TCP streaming (out) (cpas)
2: - 1ffffe0 (ffffffff88804d70) (ffffffff) fwmonitor (i/f side)
3: - 1ffff50 (ffffffff88a35fb0) (00000001) passive streaming (out) (pass_str)
4: - 1f00000 (ffffffff8882d050) (00000001) Stateless verifications (out) (asm)
5: - 1ff (ffffffff88e6eb80) (00000001) NAC Packet Outbound (nac_tag)
6: 0 (ffffffff887ceee0) (00000001) fw VM outbound (fw)
7: 10000000 (ffffffff88910330) (00000003) SecureXL outbound (secxl)
8: 70000000 (ffffffff88804d70) (ffffffff) fwmonitor (IP side)
9: 7f000000 (ffffffff887e3690) (00000001) fw accounting outbound (acct)
10: 7f700000 (ffffffff88c53690) (00000001) TCP streaming post VM (cpas)
11: 7f800000 (ffffffff8882bbe0) (ffffffff) IP Options Restore (out) (ipopt_res)
[Expert@mytestint-fwa:0]#

Here is an example where fw monitor is inserted as a chain module at pre-inbound position 4

[Expert@mytestint-fwa:0]# fw monitor -pi 4

[Expert@mytestint-fwa:0]# fw ctl chain
in chain (13):
0: -7f800000 (ffffffff8882b820) (ffffffff) IP Options Strip (in) (ipopt_strip)
1: - 1fffff8 (ffffffff8882d050) (00000001) Stateless verifications (in) (asm)
2: - 1fffff7 (ffffffff8886eb00) (00000001) fw multik misc proto forwarding
3: - 1000000 (ffffffff88911d10) (00000003) SecureXL conn sync (secxl_sync)
4: - 1 (ffffffff88804d70) (ffffffff) fwmonitor (i/f side)
5: 0 (ffffffff887ceee0) (00000001) fw VM inbound (fw)
6: 10 (ffffffff887e3690) (00000001) fw accounting inbound (acct)
7: 10000000 (ffffffff88910330) (00000003) SecureXL inbound (secxl)
8: 70000000 (ffffffff88804d70) (ffffffff) fwmonitor (IP side)
9: 7f600000 (ffffffff88820bf0) (00000001) fw SCV inbound (scv)
10: 7f730000 (ffffffff88a35fb0) (00000001) passive streaming (in) (pass_str)
11: 7f750000 (ffffffff88c53260) (00000001) TCP streaming (in) (cpas)
12: 7f800000 (ffffffff8882bbe0) (ffffffff) IP Options Restore (in) (ipopt_res)
out chain (12):
0: -7f800000 (ffffffff8882b820) (ffffffff) IP Options Strip (out) (ipopt_strip)
1: -70000000 (ffffffff88804d70) (ffffffff) fwmonitor (i/f side)
2: - 1fffff0 (ffffffff88c534a0) (00000001) TCP streaming (out) (cpas)
3: - 1ffff50 (ffffffff88a35fb0) (00000001) passive streaming (out) (pass_str)
4: - 1f00000 (ffffffff8882d050) (00000001) Stateless verifications (out) (asm)
5: - 1ff (ffffffff88e6eb80) (00000001) NAC Packet Outbound (nac_tag)
6: 0 (ffffffff887ceee0) (00000001) fw VM outbound (fw)
7: 10000000 (ffffffff88910330) (00000003) SecureXL outbound (secxl)
8: 70000000 (ffffffff88804d70) (ffffffff) fwmonitor (IP side)
9: 7f000000 (ffffffff887e3690) (00000001) fw accounting outbound (acct)
10: 7f700000 (ffffffff88c53690) (00000001) TCP streaming post VM (cpas)
11: 7f800000 (ffffffff8882bbe0) (ffffffff) IP Options Restore (out) (ipopt_res)
[Expert@bostestint-fwa:0]#

fw debug fwm - view fem.elg to find issues as SIC, Misconfigured rules, GUI Client problems and improper information.

Use the command fw ctl chain to study chain module behavior. Observe how policy changes impact the chain

Use the command fw debug fwm on and review the file fwm.elg to find such issues as SIC, mis configured rules, GUI client connectivity problems and improperly entered information.

Studying Chain Module behavior

Inbound and outbound

Independent and redundant

Unique inspection

fw ctl chain

fw monitor inspection points

inbound minus - coming in

Inbound - about to leave inbound

outbound minus - coming in

Outbound - about to leave outbound

Fw monitor –pi -vpn_ver –o monitor

Process of the Inspection Engine

Start -> NIC - Decision New Connection - No - to TCP/IPStack
Yes -> Decision Packet Match Rule - Yes - > Process Log/Alert -> Decision Pass Packet -Yes -to TCP/IPStack
No -> Decision Packet Rejected by Rule - Yes -> Process Send NAC - STOP
No - Drop the packet

Decision Packet Match Rule - NO - > Process is there another rule? -> NO -> Drop the packet ->Stop

Decision Packet Match Rule - Yes - > Process Packet Match Rule - Yes/NO

Packet Flow Through Checkpoint Firewall

Checkpoint process the packet in the ingress and the egress using two CHAINS.

Basic:

Physical layer - ingress interface
Data Link Layer/Ethernet
Inspect Driver [inspect Engine]

Network Layer/IP Routing

Inspect Driver
Data Link Layer/Ethernet
Physical layer - egress interface

Advance:

1. NIC hardware

-The network card receives electrical signalling from the link partner.

2. NIC driver

-Sanity checks

-The NIC hardware decodes the signal and passes it to the operating system's NIC driver via the PCI bus

-The frame is converted to an mbuf entry and the frame headers are stored for later use.

-NIC driver hands off the data to the operating system's mbuf memory space

3. Operating system IP protocol stack

-The OS performs sanity checks on the packet

-Hand off to SXL if enabled, or to Firewall Kernel if not

4. SecureXL (if enabled)

-SXL lookup is performed, if it matches, bypass the firewall kernel and proceed with (Operating system IP protocol stack, outbound side)

5. Firewall Kernel (inbound processing)

-FW Monitor starts here (so, perhaps you need to disable secureXL [fwaccel offCAUTION] ... )

-Connection state lookups, some protocol inspection, rulebase processing, antispoofing lookups etc

-Processing order can be seen via fw ctl chain

-Bypass complex inspection if not needed

6. Complex protocol inspection (AV is an example)

-Leave the kernel and process under userland.

-Enters back at this same stage if the traffic passed

(inbound processing stops here)

(outbound processing starts here)

7. Firewall Kernel

-Route lookup

-Check Point sanity checks etc

-FW Monitor ends here

-Pass to operating system

8. Operating system IP protocol stack

-The OS performs sanity checks on the packet

-Pass the mbuf to the NIC driver for the appropriate outbound interface

9. NIC driver

-Tag the packet as an ethernet frame by adding MAC addresses for source and destination

10. NIC hardware

-The NIC hardware encodes the signal and transmits it via wire

Fig. Ctl Chain

TIP:

1. DST NAT can happen between i & I (when client side NAT enabled [DEFAULT]) or between o & O (server side NAT)

2. SRC NAT will happen immediately after o (after routing) regardless the client side NAT enabled or not.

Static NAT - One to one translation
Hide/Dynamic NAT - Allows you to NAT multiple IPs behind one IP/Interface

Automatic NAT - Quick basic address NAT translation.
Manual NAT - Allows greater flexibility over automatic NAT; it is preferred over Automatic NAT. If you are using manual rules, you must configure proxy ARPs to associate the translated IP address with the MAC address of the Security Gateway interface that is on the same network as the translated addresses.Configuration depends of OS.
['Global Properties' - go to 'NAT' - go to 'Automatic NAT rules'

Server Side NAT - destination is NAT`d by the outbound kernel
Client Side NAT - destination is NAT`d by the inbound kernel
['Global Properties' - go to 'NAT' - go to 'Manual NAT rules'

Between all the steps there are queues. These queues accumulate packets and on intervals flush them to the next step. All of this happens very very quickly in small CPU time slices.

The INSPECT engine itself is more to do specifically with protocol inspection rather than all of the other steps. INSPECT runs traffic against definitions, if the definitions match it usually means that it hit a protection and the appropriate action is to (drop, log) the traffic.

VM is used for referring firewall Virtual Machine (VM, and it is CP terminology):

i (PREIN) – inbound direction before firewall VM .

I (POSTIN) – inbound direction after firewall VM.

o (PREOUT) – outbound direction before firewall VM,

O (POSTOUT) – outbound direction after firewall VM.

Fig. fw monitor in the Chain.

TIP: Source NAT occurs after outbound inspection (o), not between (I) and (o), so if we are doing src nat we won't see (O), the same will happen if the traffic flows via a vpn.

(00000001) new processed flows

(00000002) previous processed flows

(00000003) ciphered traffic

(ffffffff) Everything

[Expert@firewall]# fw ctl chain

in chain (19):

0: -7f800000 (f27f9c20) (ffffffff) IP Options Strip (in) (ipopt_strip)

1: -7d000000 (f1ee6000) (00000003) vpn multik forward in

2: - 2000000 (f1ecad30) (00000003) vpn decrypt (vpn)

3: - 1fffff8 (f1ed7550) (00000001) l2tp inbound (l2tp)

4: - 1fffff6 (f27faff0) (00000001) Stateless verifications (in) (asm)

5: - 1fffff5 (f2c6b240) (00000001) fw multik VoIP Forwarding

6: - 1fffff2 (f1ef30b0) (00000003) vpn tagging inbound (tagging)

7: - 1fffff0 (f1ecbb20) (00000003) vpn decrypt verify (vpn_ver)

8: - 1000000 (f2849190) (00000003) SecureXL conn sync (secxl_sync)

9: 0 (f27af6b0) (00000001) fw VM inbound (fw)

10: 1 (f2812680) (00000002) wire VM inbound (wire_vm)

11: 2000000 (f1ecdff0) (00000003) vpn policy inbound (vpn_pol)

12: 10000000 (f284e9e0) (00000003) SecureXL inbound (secxl)

13: 21500000 (f3b2c940) (00000001) RTM packet in (rtm)

14: 7f600000 (f27f0910) (00000001) fw SCV inbound (scv)

15: 7f730000 (f2928620) (00000001) passive streaming (in) (pass_str)

16: 7f750000 (f2a3bbf0) (00000001) TCP streaming (in) (cpas)

17: 7f800000 (f27f9fb0) (ffffffff) IP Options Restore (in) (ipopt_res)

18: 7fb00000 (f2a07540) (00000001) HA Forwarding (ha_for)

out chain (16):

0: -7f800000 (f27f9c20) (ffffffff) IP Options Strip (out) (ipopt_strip)

1: -78000000 (f1ee5fe0) (00000003) vpn multik forward out

2: - 1ffffff (f1eccbd0) (00000003) vpn nat outbound (vpn_nat)

3: - 1fffff0 (f2a3ba70) (00000001) TCP streaming (out) (cpas)

4: - 1ffff50 (f2928620) (00000001) passive streaming (out) (pass_str)

5: - 1ff0000 (f1ef30b0) (00000003) vpn tagging outbound (tagging)

6: - 1f00000 (f27faff0) (00000001) Stateless verifications (out) (asm)

7: 0 (f27af6b0) (00000001) fw VM outbound (fw)

8: 1 (f2812680) (00000002) wire VM outbound (wire_vm)

9: 2000000 (f1ecdaa0) (00000003) vpn policy outbound (vpn_pol)

10: 10000000 (f284e9e0) (00000003) SecureXL outbound (secxl)

11: 1ffffff0 (f1ed71e0) (00000001) l2tp outbound (l2tp)

12: 20000000 (f1ecce40) (00000003) vpn encrypt (vpn)

13: 24000000 (f3b2c940) (00000001) RTM packet out (rtm)

14: 7f700000 (f2a3b810) (00000001) TCP streaming post VM (cpas)

15: 7f800000 (f27f9fb0) (ffffffff) IP Options Restore (out) (ipopt_res)

[Expert@HostName]# fw ctl chain

Example:

in chain (17):

0: -7f800000 (f206df90) (ffffffff) IP Options Strip (in) (ipopt_strip)
1: - 2000000 (f149dd70) (00000003) vpn decrypt (vpn)
2: - 1fffff8 (f14a8b20) (00000001) l2tp inbound (l2tp)
3: - 1fffff6 (f206f290) (00000001) Stateless verifications (in) (asm)
4: - 1fffff2 (f14c4940) (00000003) vpn tagging inbound (tagging)
5: - 1fffff0 (f149bc10) (00000003) vpn decrypt verify (vpn_ver)
6: - 1000000 (f20c4980) (00000003) SecureXL conn sync (secxl_sync)
7: 0 (f201df50) (00000001) fw VM inbound (fw)
8: 1 (f2087ed0) (00000002) wire VM inbound (wire_vm)
9: 10 (f202f610) (00000001) fw accounting inbound (acct)
10: 2000000 (f149eaf0) (00000003) vpn policy inbound (vpn_pol)
11: 10000000 (f20ca740) (00000003) SecureXL inbound (secxl)
12: 7f600000 (f20646b0) (00000001) fw SCV inbound (scv)
13: 7f730000 (f21b11f0) (00000001) passive streaming (in) (pass_str)
14: 7f750000 (f231c540) (00000001) TCP streaming (in) (cpas)
15: 7f800000 (f206e320) (ffffffff) IP Options Restore (in) (ipopt_res)
16: 7fb00000 (f22de3b0) (00000001) HA Forwarding (ha_for)
out chain (15):
0: -7f800000 (f206df90) (ffffffff) IP Options Strip (out) (ipopt_strip)
1: - 1ffffff (f149cea0) (00000003) vpn nat outbound (vpn_nat)
2: - 1fffff0 (f231c3c0) (00000001) TCP streaming (out) (cpas)
3: - 1ffff50 (f21b11f0) (00000001) passive streaming (out) (pass_str)
4: - 1ff0000 (f14c4940) (00000003) vpn tagging outbound (tagging)
5: - 1f00000 (f206f290) (00000001) Stateless verifications (out) (asm)
6: 0 (f201df50) (00000001) fw VM outbound (fw)
7: 1 (f2087ed0) (00000002) wire VM outbound (wire_vm)
8: 2000000 (f149e5a0) (00000003) vpn policy outbound (vpn_pol)
9: 10000000 (f20ca740) (00000003) SecureXL outbound (secxl)
10: 1ffffff0 (f14a87b0) (00000001) l2tp outbound (l2tp)
11: 20000000 (f149d110) (00000003) vpn encrypt (vpn)
12: 7f000000 (f202f610) (00000001) fw accounting outbound (acct)
13: 7f700000 (f231e5b0) (00000001) TCP streaming post VM (cpas)
14: 7f800000 (f206e320) (ffffffff) IP Options Restore (out) (ipopt_res)

Module	Explanation
Kiss	Kernel I/s
Kissflow	Kernel I/s communication with other modules and policy enforcement
fw	Firewall, the most common module
h323	H323 VoIP traffic module
Multik	CoreXL module
uc	Usercheck module
dlpk	Data Loss Prevention module
cluster	ClusterXL module
CPAS	Check Point Active Streaming (unlike PSL)
cmi_loader	Signature loading for security blades
NRB	Next Rule Base, security blades rulebase
SGEN	Security Gateway Enforcement (stateful)
RAD_KERNEL	Very important I/S, DNS resolving, string matching, md5sum check against database and 3rd party, resource categorization
WS	Web Security module - HTTP inspection by all blades
APPI	Application Control module
CI	Content Inspection (Anti-Virus, Anti-Bot )
RTM	Real time monitoring
VPN	VPN module
SFT	Secure File Type

Common traffic flags in fw module

Module	Explanation
drop	the connection that was dropped, the function in the firewall that dropped it and the reason
conn	information from the connections table (everything that can be seen with this command #fw tab -t 8158) . Direction of packets in connections, source/destination of IP and ports, service type, handler, timeout, the rule that this packet is matched on, ETC
ld	this flag refers to the kernel dynamic tables infrastructure. It reads and write everything that is written to any kernel table. Adding this flag to the debug syntax may cause the machine high CPU and can even cause the machine to hang. Check load before performing this debug on production environment.
nat	basic NAT information
xlate	the firewall NAT infrastructure - basic NAT information + NAT cache
xltrc	NAT additional information, print NAT rulebase information and more information from the NAT kernel table (fwx_alloc)
packet	information on actions performed on packet - like accept, drop, fragment, inspection
packval	advanced information of the packet's header. stateless verifications - sequences, fragments, translations and other header changes and verifications.
vm	shows actions of all the virtual chains that can be seen with the command: # fw ctl chain on the traffic that go through the chain handle function fw_filter_chain
tcpstr	TCP streaming mechanism, organize TCP packets by order and checks protocol messages. debug the PSL I/S. (passive streaming layer)
chain	chain cookie information, chain modules
chainfwd	chain forwarding - related to fwha_perform_chain_forwarding global kernel variable (clusterXL only)
cookie	the kernel saves information and holds connections for inspection, this flag will show virtual de-fragmentation and cookie issues (cookies in the data structure holding the packets)
hold	holding mechanism and all packets being held / released

Module	Explanation
aspii	Accelerated Stateful Protocol Inspection Infrastructure (INPSECT streaming)
spii	Stateful Protocol Inspection Infrastructure
cmi	Context Management Infrastructure - IPS signature manager
advp	IPS and Application Control signatures, contexts and patterns
synatk	IPS protection "SYN Attack" (SYNDefender)

Kernel debug command line usage

Defaulting all kernel debug properties:

[Expert@HostName]#
fw ctl debug 0

Define a kernel debug buffer:

[Expert@HostName]#
fw ctl debug -buf <buffer_size>

If the machine has enough memory, it is recommended to set the maximum buffer of 32768.

The fact that the buffer size is 8000 or 32768 will not affect the file size.

The kernel writes the debug messages to the buffer. A user mode process named 'kdebug' writes the messages from the buffer to the file.

Check which debug flags are enabled:

[Expert@HostName]# fw ctl debug -m

Check which debug flags are on for specific module:

[Expert@HostName]# fw ctl debug -m <module_name>

Set the debugging flags for the required module:

[Expert@HostName]# fw ctl debug -m <module_name> + flag1 flag2
flag3

UnSet the debugging flags from the module:

[Expert@HostName]# fw ctl debug -m <module_name> - flag1 flag2
flag3

Add Timestamp (necessary to match the traffic capture to kernel debug):

[Expert@HostName]# fw ctl kdebug -T -f > /var/log/debug.txt

Dividing the debug output file into multiple file limited to certain size:

[Expert@HostName]# fw ctl kdebug -f -o <file_name> -m
<number> -s <size>

Kernel tables

The kernel writes all of its actions to relevant tables. Every kernel table has an assigned name and assigned kernel ID, and it holds specific information.
The forwarding, handling and decisions will be done based on informatin that is stored in different kernel tables.

Kernel tables command line usage

To view the kernel tables, run:

[Expert@HostName]# fw tab

To view a certain table, run:

[Expert@HostName]# fw tab -t <Table_Name | Table_ID>

To view all entries (unlimited number) in the table:

[Expert@HostName]# fw tab -t <Table_Name | Table_ID> -u

To view the table entries translated from HEX to DEC format:

[Expert@HostName]# fw tab -t <Table_Name | Table_ID> -u -f

To export the table contents to a file:

[Expert@HostName]# fw tab -t <Table_Name | Table_ID> -u >
/var/log/file_name.txt

To delete all entries from a certain kernel table (all involved connections will be immediately lost and disconnected):

[Expert@HostName]# fw tab -t <Table_Name | Table_ID> -x -y

Important Note: This operation deletes all information from the table. Do NOT use this command in production environment, as it might cause a traffic outage.

To delete specific entry:

[Expert@HostName]# fw tab -t <Table_Name | Table_ID> -x -e
<Entry_ID | Tuple>

Display a summary about entries in certain table:

[Expert@HostName]# fw tab -t <Table_Name | Table_ID> -s

Display a summary about entries in all tables:

[Expert@HostName]# fw tab -s

Connections Table

The most used kernel table is the Connections Table. Every connection that passed the firewall is written to the Connections Table. Timeouts, rule numbers, symbolic links, state decisions, all are written and decided by this table.

Real entry contains the following:

<DIRECTION,6-tuple-key;r_ctype,r_cflags,rule,service_id,handler,uuid1,uuid2,uuid3,uuid4,ifncin,ifncout,ifnsin,ifnsout,bits1,bits2,connection_module_kbufs@ttl/timeout>

Tuple is the basic information on the connection that is written to the connections table.

<direction,src_ip,src_port,dst_ip,dst_port,protocol>

Refer to sk65133 - Connections Table Format.

Example 1:

<00000001, C21D2B59, 0000AF6B, C21D24D9, 00000035, 00000011> - This is the first connection that was written to the Connections Table. The culmination of all six values named "six tuple" or - connections table EntryID

00000001 - first value of the tuple indicates the direction (00000001 - outbound)

C21D2B59 - Source IP - in our example, Security Gateway (in Dec = 194.29.43.89)

0000AF6B - Source port (in Dec = 44907)

C21D24D9 - Destination IP - in our example, DNS Server (in Dec = 194.29.36.217)

00000053 - Destination port (in Dec = 53)

00000017 - Protocol (in Dec = 11 = UDP)

Example 2:

The reply of the DNS request connection will be:

<00000000, C21D24D9, 00000035, C21D2B59, 0000AF6B, 00000011>

00000000 - first value of the tuple indicates the direction (00000000 - inbound)

C21D24D9 - Source IP- in our example, DNS Server (in Dec = 194.29.36.217)

00000053 - Source port (in Dec = 53)

C21D2B59 - Destination IP - in our example, Security Gateway (in Dec = 194.29.43.89)

0000AF6B - Destination port (in Dec = 44907)

00000017 - Protocol (in Dec = 11 = UDP)

Kernel debug analysis

In terms of analysis for the debug, the ability to analyze the debug depends on how familiar you are with the expected behavior.

Here is a basic flow of accepted inbound handling of a connection:

fwconn_lookup: connection lookup, the Security Gateway wants to see if there is a match for the connection that just arrived via the interface

[-- Stateful VM inbound: Entering (1244447195) --];

Before VM:

<192.168.121.1:4461 -> 10.0.121.13:23 IPP 6> (len-48) TCP flags-0x2 (SYN), seq-55a92b89, ack-0, data end-55a92b8a (ifn-0) (first seen) (looked up)
Since the connection is "not found in connections table", the next step is to record it

fwconn_record_conn: record conn
Stateful inspection - SYN:

fwconn_record_conn: SYN packet. Turn on SYN_SENT flag;

Since we have enabled the syn_sent flag, only RST or SYN-ACK are allowed. In the phase for example, if any other packet besides RST or SYN-ACK, it will be dropped on "out of state".
Getting further information from the Security Gateway connections table:

fwconn_get_service_timeout: Getting default protocol
Setting partial_handle bit: loading further code handling.

fwconn_record_conn: setting PARTIAL_SETUP flag
Creating REAL entry

fwconn_record_conn: created real

<dir 0, 192.168.121.1:4461 -> 10.0.121.13:23 IPP 6>
After we checked that stateful inspection is ok, rulebase matching:

fw_handle_first_packet: Rulebase returned
NAT rulebase match:

fw_xlate_match conn-<192.168.121.1:4461 -> 10.0.121.13:23 IPP 6>;
fw_xlate_find_all_matches_old: No match for conn
Creating symbolic links (after NAT decision)

fwconn_init_links: Creating links

fwconn_init_links: Creating links (inbound). One way links-0, Replies from any-0;

;fwconn_set_links_inbound: create link cls_o <dir 1, 10.0.121.13:23 -> 192.168.121.1:4461 IPP 6> -> <dir 0, 192.168.121.1:4461 -> 10.0.121.13:23 IPP 6>(0x5);

fwconn_set_links_inbound: create link srs_o <dir 1, 192.168.121.1:4461 -> 10.0.121.13:23 IPP 6> -> <dir 0, 192.168.121.1:4461 -> 10.0.121.13:23 IPP 6>(0x2);
Handling inbound is finished:

fwconn_init_links: connection completed, unset PARTIAL_SETUP bit;
fw_tcp_state_update: first SYN
VM Final action-ACCEPT;
----- Stateful VM inbound Completed -----
fwconn_lookup (outbound.......)

<10001,44000,1,1ae,0,4a2cc1db,0,4779a8c0,ffff,0,ffffffff,ffffffff,ffffffff,2000000,0,0,0,0,0,0,0,0,0,0,0,0,7c9c0800,0,0>

found in connections table dir-1;

After this phase, the packet is entering the outbound chains and the handling will continue in accordance with the chain forwarding.
[-- Stateful VM outbound: Entering (1244447195) --];
This point represents the chains between I and o.

Before VM:

<192.168.121.1:4461 -> 10.0.121.13:23 IPP 6> (len-48) TCP flags-0x2 (SYN), seq-55a92b89, ack-0, data end-55a92b8a (ifn-1) (looked up) ;

fwconn_chain_update_anti_spoofing_cache: updating anti-spoofing interface info. [cin-0, cout--1, sin--1, sout--1], dir-1, cdir-1, new ifn-1, spoof offset-3, previous ifn--1;
NAT rulebase match, cache and Slinks, after we have routing info.

fw_xlate_match: conn-<192.168.121.1:4461 -> 10.0.121.13:23 IPP 6>; ; 8Jun2009 10:46:35.420110;fw_xlate_match: cache hit!;
fw_conn_post_inspect: Setting SRS_OUTBOUND_SEEN flag;
Syn accepted, written, inspected by stateful inspection, rulebase and anti-spoofing.

fw_filter_chain: Final switch, action-ACCEPT;

After VM:

<192.168.121.1:4461 -> 10.0.121.13:23 IPP 6> (len-48) TCP flags-0x2 (SYN), seq-55a92b89, ack-0, data end-55a92b8a

VM Final action-ACCEPT
----- Stateful VM outbound Completed -----

When troubleshooting and trying to understand which chain is causing a problem on the Security Gateway, use the following command:

[Expert@HostName]# fw monitor -e "accept;" -p all

The '-p all' flag will show all the chains, through which the traffic passed.

To see it in kernel debug, use module fw with the flags monitorall, vm, chain and chainfwd

Note: These flags generate large number of messages, hence will cause high load on CPU.

To see all active modules on the Security Gateway, run:

[Expert@HostName]# fw ctl debug -m

Debugging Modules

When debugging the kernel, it is important to first understand the flow of the packets and connections and understand the failure point; it is important to understand which module you need to debug.

Every module has its own flags that were designed to print specific debug information. It is important to know the common flags to be more accurate when debugging the kernel.

Common IPS flags in fw module

Kernel debug command line usage

Defaulting all kernel debug properties:

Define a kernel debug buffer:

If the machine has enough memory, it is recommended to set the maximum buffer of 32768.

The fact that the buffer size is 8000 or 32768 will not affect the file size.

The kernel writes the debug messages to the buffer. A user mode process named 'kdebug' writes the messages from the buffer to the file.

Check which debug flags are enabled:

[Expert@HostName]# fw ctl debug -m

Check which debug flags are on for specific module:

[Expert@HostName]# fw ctl debug -m <module_name>

Set the debugging flags for the required module:

[Expert@HostName]# fw ctl debug -m <module_name> + flag1 flag2 flag3

UnSet the debugging flags from the module:

[Expert@HostName]# fw ctl debug -m <module_name> - flag1 flag2 flag3

Add Timestamp (necessary to match the traffic capture to kernel debug):

[Expert@HostName]# fw ctl kdebug -T -f > /var/log/debug.txt

Dividing the debug output file into multiple file limited to certain size:

[Expert@HostName]# fw ctl kdebug -f -o <file_name> -m <number> -s <size>

Kernel tables

The kernel writes all of its actions to relevant tables. Every kernel table has an assigned name and assigned kernel ID, and it holds specific information.

Kernel tables command line usage

To view the kernel tables, run:

[Expert@HostName]# fw tab

To view a certain table, run:

[Expert@HostName]# fw tab -t <Table_Name | Table_ID>

To view all entries (unlimited number) in the table:

[Expert@HostName]# fw tab -t <Table_Name | Table_ID> -u

To view the table entries translated from HEX to DEC format:

[Expert@HostName]# fw tab -t <Table_Name | Table_ID> -u -f

To export the table contents to a file:

[Expert@HostName]# fw tab -t <Table_Name | Table_ID> -u > /var/log/file_name.txt

To delete all entries from a certain kernel table (all involved connections will be immediately lost and disconnected):

[Expert@HostName]# fw tab -t <Table_Name | Table_ID> -x -y

To delete specific entry:

[Expert@HostName]# fw tab -t <Table_Name | Table_ID> -x -e <Entry_ID | Tuple>

Display a summary about entries in certain table:

[Expert@HostName]# fw tab -t <Table_Name | Table_ID> -s

Display a summary about entries in all tables:

[Expert@HostName]# fw tab -s

Connections Table

Real entry contains the following:

<DIRECTION,6-tuple-key;r_ctype,r_cflags,rule,service_id,handler,uuid1,uuid2,uuid3,uuid4,ifncin,ifncout,ifnsin,ifnsout,bits1,bits2,connection_module_kbufs@ttl/timeout>

Tuple is the basic information on the connection that is written to the connections table.

<direction,src_ip,src_port,dst_ip,dst_port,protocol>

Refer to sk65133 - Connections Table Format.

00000001 - first value of the tuple indicates the direction (00000001 - outbound)

C21D2B59 - Source IP - in our example, Security Gateway (in Dec = 194.29.43.89)

0000AF6B - Source port (in Dec = 44907)

C21D24D9 - Destination IP - in our example, DNS Server (in Dec = 194.29.36.217)

00000053 - Destination port (in Dec = 53)

00000017 - Protocol (in Dec = 11 = UDP)

Example 2:

The reply of the DNS request connection will be:

<00000000, C21D24D9, 00000035, C21D2B59, 0000AF6B, 00000011>

00000000 - first value of the tuple indicates the direction (00000000 - inbound)

C21D24D9 - Source IP- in our example, DNS Server (in Dec = 194.29.36.217)

00000053 - Source port (in Dec = 53)

C21D2B59 - Destination IP - in our example, Security Gateway (in Dec = 194.29.43.89)

0000AF6B - Destination port (in Dec = 44907)

00000017 - Protocol (in Dec = 11 = UDP)

Kernel debug analysis

In terms of analysis for the debug, the ability to analyze the debug depends on how familiar you are with the expected behavior.

Here is a basic flow of accepted inbound handling of a connection:

Currently as we see, the timeout is "25", as the handshake was not completed.

The SYN-ACK will go through the same inspection process, however the timeout will be also be "25". It will change only after the 3-way handshake will be completed.

In terms of stateful inspection, the only allowed flags according to the current state (SYN_SENT) are SYN-ACK and RST.

Every other packet will be dropped for "out of state".

Acceleration

Performance Pack is a software acceleration product installed on Security Gateways. Performance Pack uses SecureXL technology and other innovative network acceleration techniques to deliver wire-speed performance for Security Gateways.

In a SecureXL-enabled gateway, the firewall first uses the SecureXL API to query the SecureXL device and discover its capabilities. The firewall then implements a policy that determines which parts of what sessions are to be handled by the firewall, and which should be offloaded to the SecureXL device. When new sessions attempt to get established across the gateway, the first packet of each new session is inspected by the firewall to ensure that the connection is allowed by the security policy. As the packet is inspected, the firewall determines the required behavior for the session, and based on its policy it may then offload some or all of the session handling to the SecureXL device. Thereafter, the appropriate packets belonging to that session are inspected directly by the SecureXL device. The SecureXL device implements the security logic required for further analysis and handling of the traffic. If it identifies anomalies it then consults back with the firewall software and IPS engine. In addition, SecureXL provides a mode that allows for connection setup to be done entirely in the SecureXL device, thus providing extremely high session rate.

Fig. FW Paths and core processing

Medium Path is known as PXL (SXL + PSL [PSL passive Streaming Library IPS related])

Slow Path (Firewall Path) is known as F2F (Forwarded 2 Firewall)

SecureXL is implemented either in software (Core), or in hardware (SAM cards on Check Point 21000 appliances; ADP cards on IP Series appliances with CPUs inside). The Dispatcher + Performance Pack combination is known as SND (Secure Network Dispatch).

Fwaccell (Firewall Acceleation feature) is used to check/manage the SecureXL Acceleration Device

SIM (SecureXL implementation Device) Affinity (Association with a Core) can be managed automatically by checkpoint (each 60 seconds) or statically configured.

TIP: To achieve the best performance, pairs of interfaces carrying significant data flows (based on network topology) should be assigned to pairs of CPU cores on the same physical processor.

Active Streaming (CPAS) - Technology that sends streams of data to be inspected in the kernel, since more than a single packet at a time is needed in order to understand the application that is running (such as HTTP data). Active Streaming is Read- and Write-enabled, and works as a transparent proxy. Connections that pass through Active Streaming can not be accelerated by SecureXL.
Passive Streaming - Technology that sends streams of data to be inspected in the kernel, since more than a single packet at a time is needed in order to understand the application that is running (such as HTTP data). Passive Streaming is Read-only and it cannot hold packets, but the connections are accelerated by SecureXL.
Passive Streaming Library (PSL) - IPS infrastructure, which transparently listens to TCP traffic as network packets, and rebuilds the TCP stream out of these packets. Passive Streaming can listen to all TCP traffic, but process only the data packets, which belong to a previously registered connection. For more details, refer to sk95193 (ATRG: IPS).
PXL - Technology name for combination of SecureXL and PSL.
QXL - Technology name for combination of SecureXL and QoS (R77.10 and above).
F2F / F2Fed - Packets that can not be accelerated by SecureXL (refer to sk32578 (SecureXL Mechanism)) are Forwarded to Firewall.
F2P - Forward to PSL/Applications. Feature that allows to perform the PSL processing on the CPU cores, which are dedicated to the Firewall.

HyperThreading

SMT: New in R77. Minimices Context change inside a physical core and it may improve performance. (or, may not f.e memory used is increased and connection table size can de reduced)

It is not supported in Open Servers.

It is strongly recommended to disable Hyper-Threading in BIOS when CoreXL is enabled (on Check Point appliances this is disabled, by default). Applies to Intel processors prior to "Intel Nehalem (Core i7)", where this technology was improved (called Simultaneous Multi-Threading, or Intel® Hyper-Threading)

Multicore

Multi Core processing in Checkpoint is known as CoreXL. Thre are some limitations when CoreXL is enabled.

The following features/settings are not supported in CoreXL:

Check Point QoS (Quality of Service)⁽¹⁾
'Traffic View' in SmartView Monitor⁽²⁾ (all other views are available)
Route-based VPN
IP Pool NAT⁽³⁾ (refer to sk76800)
IPv6⁽⁴⁾
Firewall-1 GX
Overlapping NAT
SMTP Resource⁽³⁾
VPN Traditional Mode (refer to VPN Administration Guide - Appendix B for converting a Traditional policy to a Community-Based policy)

If any of the above features/settings is enabled/configured in SmartDashboard, then CoreXL acceleration will be automatically disabled on the Gateway (while CoreXL is still enabled). In order to preserve consistent configuration, before enabling one of the unsupported features, deactivate CoreXL via 'cpconfig' menu and reboot the Gateway (in cluster setup, CoreXL should be deactivated on all members).

Notes:

(1) - supported on R77.10 and above (refer to sk98229)
(2) - supported on R75.30 and above
(3) - supported on R75.40 and above
(4) - supported on R75.40 and above on SecurePlatform/Gaia/Linux only

There are two main roles for CPUs applicable to SecureXL and CoreXL:

SecureXL and CoreXL dispatcher CPU (the SND - Secure Network Distributor)
You can manually configure this using the sim affinity -s command. (Exception: cpmq if multiqueue)
CoreXL firewall instance CPU
You can manually configure this using the fw ctl affinity command

Traffic is processed by the CoreXL FW instances only when the traffic is not accelerated by SecureXL (if SecureXL is installed and enabled). So, if all the traffic is accelerated, we can have several fw instances idle.

CoreXL will improve performance with almost linear scalability in the following scenarios:
- Much of the traffic can not be accelerated by SecureXL
- Many IPS features enabled, which disable SecureXL functionality
- Large rulebase
- NAT rules
CoreXL will not improve performance in the following scenarios:

SecureXL accelerates much of the traffic
Traffic mostly consists of VPN (VPN traffic inspection occurs only in CoreXL FW instance #0)
Traffic mostly consists of VoIP (VoIP control connections are processed in only in CoreXLFW instance #0)

In some cases it may be advisable to change the distribution of kernel instances, the SND, and other processes, among the processing cores. This configuration change is known asPerformance Tuning. This is done by changing the affinities of different NICs (interfaces) and/or processes. However, to ensure CoreXL's efficiency, all interface traffic must be directed to cores not running kernel instances. Therefore, if you change affinities of interfaces or other processes, you will need to accordingly set the number of kernel instances and ensure that the instances run on other processing cores.

Automatic Mode — (default) Affinity is determined by analysis of the load on each NIC. If a NIC is not activated, Affinity is not set. NIC load is analyzed every 60 seconds.

Manual Mode — Configure Affinity settings for each interface: the processor numbers (separated by space) that handle this interface, or all. In Manual Mode, periodic NIC analysis is disabled

The default affinity setting for all interfaces is Automatic. Automatic affinity means that if Performance Pack is running, the affinity for each interface is automatically reset every 60 seconds, and balanced between available cores. If Performance Pack is not running, the default affinities of all interfaces are with one available core. In both cases, any processing core running a kernel instance, or defined as the affinity for another process, is considered unavailable and will not be set as the affinity for any interface. Poor decisions maybe done with automatic affinity.

Figure. Possible affinity setting.

Figure. MultiCore Processing and packet flow paths.

SecureXL and CoreXL connection info exchage:

Connection offload - Firewall kernel passes the relevant information about the connection from Firewall Connections Table to SecureXL Connections Table.
Note: In ClusterXL High Availability, the connections are not offloaded to SecureXL on Standby member.

Connection notification - SecureXL passes the relevant information about the accelerated connection from SecureXL Connections Table to Firewall Connections Table.

Partial connection - Connection that exists in the Firewall Connections Table, but not in the SecureXL Connections Table (versions R70 and above).
- In Cluster HA - partial connections are offloaded when member becomes Active
- In Cluster LS - partial connections are offloaded upon post-sync (only for NAT / VPN connections)
Such connections must be offloaded to SecureXL, since packets in these connections must not be dropped.
If a packet matched a partial connection in the outbound, then it should be dropped.

Delayed connection - Connection created from SecureXL Connection Templates without notifying the Firewall for a predefined period of time. The notified connections are deleted by the Firewall.

Dhansham - Engineer's Notebook Checkpoint Firewalls Gaia

Wednesday, July 6, 2016

Index

fw mon

Inspection Points (in the order they are passed):

For a complete two-way communicating traffic you should see 8 logs (also depends if you do want to see 8 or less)

Sometimes you won’t see 8 logs for a complete two-way communicating traffic and will only see 4 logs

Troubleshooting the inspection points

FW Monitor

fw_monitor

Starting with the first packet ID 27361:

Next packet ID 44853;

Next packet ID 27386;

Next packet ID is 27396;

Next packet IDs are 45365 and 45621;

Next packet ID is 27478;

Next packet ID is 27479;

Next packet ID is 11794;

Tuesday, July 5, 2016

Firewall Debugging

Disable SecureXL for capturing proper logs:

Display all 8 inspection points (iIoO) for a complete two-way traffic on the console:

Display 4 inspection points (iO) for a complete two-way traffic on the console:

Saving fw monitor logs to a .pcap file to analyse in wireshark:

Saving a TCP dump in a .pcap file:

Switches:

Chapter 2 Chain Module

Packet Flow Through Checkpoint Firewall

Checkpoint process the packet in the ingress and the egress using two CHAINS.

Acceleration