FWM - Troubleshoot

Installation failure on MGNT server occurs in verification or complication stage of the installation.  To troubleshoot this failure, follow these steps:

               

1.       Install policy from the CLI

fwm –d load $FWDIR/conf/PolicyName.W <target>

2.       Install the policy from the SmartDashboard

a.       Clean the old log files

cd $FWDIR/log

rm fwm.elg

echo  ‘ ‘> fwm.elg

                               

enable the fwm debug

fw debug fwm on TDERROR_ALL_ALL=5

fw debug fwm on OPSEC_DEBUG_LEVEL=9

                                                                                          .

Replicate the problem by installing the policy from the Dashboard GUI

                                Stop the fwm debug on the FWMGMT  server

fw debug fwm off TDERROR_ALL_ALL=1

fw debug fwm off OPSEC_DEBUG_LEVEL=1

Debug output are stored in fwm.elg, located under $FWDIR/log.

Policy Installation failure could be result of SIC, Misconfigured rules, GUI client connectivity problems or improper entered information, expired license. Evaluate output file fwm.elg to determine with of these may have cause the failure.

Lab   Troubleshoot SIC   - Policy Installation failure

From A-SMS (FW Management Server)

1.       cpca_client lscert    (use this command to validate certificate)

2.       fw debug fwm on Establish

3.       from Client GUI DashBoard, attempt to restablish SIC

4.       from A-SMS     fw debug fwm off

5.       review debug log output        less $FWDIR/log/fwm.elg 

6.       WinSCP to transfer fwm.elg file  and view in Test Editor to identity source of problem from debug file

Lab   Troubleshoot MisConfigured Rules  - Policy Installation Failure

From A-SMS (FW Management Server)

1.       In Expert mode

cd  $FWDIR/log

echo >fwm.elg

fw debug fwm on

2.       fw debug fwm on

TDERROR_ALL_ALL

fwm –d load $FWDIR/conf/standard.W A-GW

Note: you will see the output on the screen and fwm.elg. you can see additional info if you do a TDERROR_ALL_ALL=5

Stop the debugs

fw debug fwm off

unset TDERROR_ALL_ALL

                view fwm.elg     less $FWDIR/conf/fwm.elg

Lab   Troubleshoot GUI Client Connectivity Problem 

Look at less $FWDIR/con/gui-client file for allowed IP address

From A-SMS (FW Management Server)

cpconfig command

At the prompt type 3 and enter

Type Y and Enter , Type D and Enter  (delete your IP to test failure)

Enter 10.20.59.250   (your IP address)

Exit cpconfig option 9 enter

From the command prompt type

fw debug fwm on TDERROR_ALL_ALL=5

from your Client 10.20.59.250  Dashboard, launch the GUI

from the FWM stop the debug and view logs of the GUI client IP

fw debug fwm off

grep 10.20.59.250 $FWDIR/log/fwm.elg | less

How to uninstall and install another policy on firewall

ISSUE

There may be a time where you install the wrong policy onto a Check Point Firewall. This can block your connections, and screw which traffic is allowed through the firewall.

RESOLUTION

These steps will show you how to remove and reinstall the correct policy via the CLI on the manager (SCS),

  1. First of all we look at the policy history, so we can find out the name of the policy we need to reinstall.

fw stat -l [firewall ip]

  2. Next we remove the security policy from the firewall.

fwm unload [fwname]

  3. Finally we install the correct policy back onto the Firewall.

Note : Note how we add the .W to the policy name as it has yet to be be compiled into a .cf file (which is what is installed onto the Firewall/Gateway)

[PolicyName].W is  uncompelled policy

[PolicyName].CF is compiled Policy

fwm load [PolicyName].W [fwname]

fwm load RuleBaseName.W TargetGatewayName

[Expert@MYFWM01]# fwm load Standard samplegw

Installing policy on R77 compatible targets:

Standard.W: Security Policy Script generated into CustomerPolicy.pf

Standard:

Compiled OK.

Installing Security Gateway policy on: examplegw . ..

Security Gateway policy installed successfully on examplegw. ..

Security Gateway policy installation complete

Security Gateway policy installation succeeded for: examplegw

Installation failed; reason – load on module failed, failed to load security policy

Recently I tried a policy installation on a Security Gateway appliance which failed with the message: Installation failed; reason - load on module failed, failed to load security policy

From my experience I know that a cpstop ; cpstart normaly solves this problem.

But since I was dealing with a remote site gateway which also was stand-alone installation, issuing cpstop was no option since it would interrupt the service.

So I utilized the cpwd_admin command for stopping and restarting FWM and CPD.

cpwd_admin stop -name FWM -path "$FWDIR/bin/fw" -command "fw kill fwm"

cpwd_admin start -name FWM -path "$FWDIR/bin/fwm" -command "fwm"

cpwd_admin stop -name CPD -path "$CPDIR/bin/cpd_admin" -command "cpd_admin stop"

cpwd_admin start -name CPD -path "$CPDIR/bin/cpd" -command "cpd"

Then I checked the status of all my services with cpwd_admin list.

All services were up and I tried policy installation again, which worked as expected.

I was able to solve the issue using the following steps i stumbled on while surfing the net:

1.            tellpm process:monitord

2.            ps aux | grep cpd

3.            kill -15

4.            tellpm process:monitord t

5.            cpwd_admin start -name CPD -path “$CPDIR/bin/cpd” -command “cpd”

Here is the expected output:

[Expert@CP-DMZ:0]# tellpm process:monitord

[Expert@CP-DMZ:0]#

Message from syslogd@ at Tue Sep 16 10:27:42 2014 …

CP-M-DMZ monitord[4129]: monitord got killed

[Expert@CP-DMZ:0]#

[Expert@CP-DMZ:0]# ps aux | grep cpd

admin 4461 0.0 0.3 212412 3196 ? Dsl Aug06 42:47 cpd

admin 6905 0.0 0.0 1816 492 pts/2 S+ 10:27 0:00 grep cpd

[Expert@CP-DMZ:0]# kill -15 4461

[Expert@CP-DMZ:0]#

[Expert@CP-DMZ:0]# tellpm process:monitord t

[Expert@CP-DMZ:0]#

[Expert@CP-DMZ:0]# cpwd_admin start -name CPD -path “$CPDIR/bin/cpd” -command “cpd”

cpwd_admin:

Process CPD started successfully (pid=7030)

[Expert@CP-DMZ:0]#

So the problem was solved without service interruption.

SSL Inspection - UserCheck

UserCheck  (Enable Block Pages)

SK100571  SK97951  SK100281    Check URL Categorization

Exceptions - via regular expression
----------------------------------
(.*\.|)logmein\.com
(.*\.|)dropbox\.com
(.*\.|)gotomeet\.me
(.*\.|)citrixonline\.com
*.gotomeet.me

Packet Flow

1. Firewall Rules  - Firewall Rule base work on Layer 3 (IP/Networks) and Layer 4  (TCP Services)
2. Application Blade  works on  Layer 5-6-7 
1. IPS Blade - Streaming - Passive Streaming Session 

1. Checkpoint firewall object - Usercheck - Accessability - Edit- Through all Interfaces
2. Engine settings -  URL Filtering  uncheck (categorize HTTPS sites)  - must put back to allow other to w  Global Property
3. reset userset daemon
 # mpclient stop UserCheck
 # mpclient start UserCheck 

[Expert@mytestint-fwa:0]# mpclient stop UserCkeck
Exception: An exception has occurred during an RPC request: Portal not found
[Expert@mytestint-fwa:0]# mpclient stop UserCheck
Portal stopped
[Expert@mytestint-fwa:0]# mpclient start UserCheck
Portal started

[Expert@mytestint-fwa:0]# 

[Expert@mytestint-fwa:0]# ps -aux | grep UserCheck
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.7/FAQ
admin    26902  0.0  0.0   1740   528 pts/2    S+   13:07   0:00 grep UserCheck
[Expert@mytestint-fwa:0]# 

sk100571 
Webpage does not redirect to UserCheck page

Symptom

• When browsing to a block site, the site gets blocked however the page does not redirect to the UserCheck web page

Cause:
On SmartDashboard, the UserCheck Web Portal "Main URL" is configured with the incorrect IP. On a cluster, it is not configured as the virtual IP (VIP) of the cluster.

Solution

Change the UserCheck Web Portal "Main URL" to the correct IP address:
Open SmartDashboard > Edit Cluster/Gateway Object > UserCheck > Main URL
Once you have changed the Main URL, you will need to restart the UserCheck portal. From CLI run the following commands:
# mpclient stop UserCheck
# mpclient start UserCheck 

sk97951
UserCheck block page not displayed when accessing blocked site as per URL Filtering policy

Solution ID	sk97951
Product	Application Control, URL Filtering
Version	All
OS	Gaia, SecurePlatform 2.6
Platform / Model	All
Date Created	30-Dec-2013
Last Modified	29-Jun-2016

Symptoms

• UserCheck Block page is not displayed when users connect from the internal interfaces, even with the UserCheck portal or the Main URL field being configured as the primary URL for the web portal that shows the UserCheck notifications.
• HTTPS Inspection and HTTP/HTTPS Proxy is not enabled on the Security Gateway.

Cause

In the Security Gateway object properties - "UserCheck" pane - "Accessibility" section - the option "Through internal interfaces" is selected. However, the option "Including DMZ internal interfaces" or the option "Including undefined internal interfaces" was not selected.

Solution

Follow these steps:

1. Open Security Gateway object properties.
2. Go to 'UserCheck' pane.
3. In the 'Accessibility' section, click on 'Edit...' button.
4. Configure the interfaces on the Security Gateway, through which the UserCheck portal can be accessed.
   
   Select the options below, according to where the users are connecting from, so they are redirected to the UserCheck portal:
   
   
   • Through all interfaces
   • Through internal interfaces
     
     
     • Including undefined internal interfaces - select this option, if users are connecting through the internal interfaces
     • Including DMZ internal interfaces - select this option, if users are connecting through the DMZ internal interface
     • Including VPN encrypted interfaces
   • According to the Firewall Policy (select this option if there is a rule that states who can access the portal)
5. Click on 'OK' to apply the settings.
6. Save the changes: go to 'File' menu - click on 'Save'.
7. Install policy onto involved Security Gateways.

sk100281
Cannot access UserCheck page

Solution ID	sk100281
Product	Application Control, URL Filtering
Version	All
OS	Gaia, SecurePlatform, SecurePlatform 2.6, IPSO 6.2
Platform / Model	All
Date Created	15-Apr-2014
Last Modified	06-Jun-2016

Symptoms

• When using Application Control and URL Filtering with UserCheck enabled, blocked websites are successfully blocked, however the browser does not redirect to the UserCheck page. It will also not be possible to directly browse to the UserCheck page based on the alias set up in SmartDashboard.
• UserCheck connectivity debug (fw ctl zdebug + crypt) shows
  ;vm_mux_first_packet: Connection is for http(s)? = 1 (allow redirect = 1); ;vm_mux_first_packet: is mp port allowed? allow_80_443_ports = 0 or connection from internal interfaces = 0;

Cause

UserCheck is not enabled on the correct interfaces.

Solution

 To resolve this issue, perform one of the following two solutions:

1. Navigate to 'Firewall object -> UserCheck -> Accessibility' and enable:
   Through Internal Interfaces:* Including VPN encrypted interfaces* including undefined internal interfaces* Including DMZ internal interface
   
    
2. Navigate to 'Firewall object -> Topology -> Edit Topology'
   
   Confirm that the interfaces have been correctly configured as 'Internal' or 'External'. If an internal interface is misconfigured as an 'External' interface, UserCheck page will not be displayed. Only traffic from an 'Internal' interface will allow the UserCheck webpage to be displayed.
   The only exception would be if UserCheck is configured to be displayed through 'All Interfaces' in 'Firewall object -> UserCheck -> Accessibility'.

CPView CPKstats

1) CPView.

sk101878 | CPView Utility

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk101878&partition=General&product=Security


1a] To view a timestamp in history. We can also use this to verify against the times that the issue was reported.

# cpview -t <timestamp>

Example:
# cpview -t 01Mar2015 10:54:15


1b] When we are experiencing an issue with high CPU on a firewall worker.

Identify the worker.
# top

Go to CPView
# cpview

Advanced > CoreXL > Instances > Find the particular instance that we are seeing high CPU

'Statistics are NOT active (may affect performance). Press 'a' to activate.

You will then be prompted to type 'y' to proceed.

You should be able to scroll down in the instance pane and find the devices that are being handled by this firewall worker instance.

You can deactivate it by following the same steps to activate it.


1c] We can also look at the following areas to see what services may be causing this issue.

CPU > Top-Protocols > Instances > Find the particular instance that we are seeing high CPU

Network > Top-Connections (OR) Top-Protocols



2) CPKstats

Should we not be able to access CPView again, we can follow the below SK to get the same results.

sk103293 | How to get per CPU statistics and TOP FW-lock consumers with cpkstats

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk103293&partition=Expert&product=CoreXL%22



3) The CPView database failed to upload for some reason and I was hoping that we could upload that one more time from each firewall.

# tar czvf /var/log/cpview_`hostname`.tgz /var/log/CPView_history/CPViewDB.dat

# cpinfo -n -f /var/log/cpview* -s 1-8319371181 -u <UserCenter Username>



4) The core dump that was generated was a User Check core dump and I'm not sure that it would cause the issue that we saw as the failover time where we lost all connections to the gateways did not add up to when the core dump was generated.

-----------------------
User Mode Core Dumps
-----------------------
total 45868
-rw-r--r-- 1 admin root 46912760 Jul 29 15:32 usrchkd.28001.core.gz