ATRG: URL Filtering

Check Point's latest firewall innovation brings the industry's strongest URL Filtering, application and identity control to organizations of all sizes. You can easily create policies which detect or block thousands of applications and internet sites. Use the Application Control and URL Filtering blades to: Learn about the applications Use Check Point's comprehensive AppWiki to understand what applications are used for and what their risk levels are. Create a Granular Policy Make rules to allow or block applications or Internet sites, by individual application, application or URL categories, or risk levels. When you use Identity Awareness, you can easily make rules for individuals or different groups of users. You can also create an HTTPS policy that enables the gateway to inspect HTTPS traffic to prevent security risks related to the SSL protocol. Learn What Your Employees are Doing Use SmartView Tracker and SmartEvent to understand the application and site traffic that really occurs in your environment. Then change the policy to make it even more effective. Only administrators that have been assigned with relevant permissions can see all the fields in a log. Using these permissions makes sure that restricted data is kept private in logs and cannot be seen by all administrators. Keep Your Policies Updated Application and URL Filtering Database is updated regularly with applications and site categories to help you keep your policy current. The gateway connects to the Check Point Online Web Service to identify social networking widgets and website categories for URLs that it does not recognize. Results are stored on a local cache on each Security Gateway. Subsequent uncategorized URLs are first checked against the local cache before querying the Check Point Online Web Service. Custom Applications, Sites, Categories and Groups You can create applications, websites, categories and groups that are not in the Application and URL Filtering Database for use in the policy. Use these custom objects to create a Rule Base that meets your organization's requirements. It is also possible to contact Check Point to create customized application signatures that can be imported into the database. This file can contain, for example, a database with an organization's internal applications that are not necessarily web-based. URL Filtering Categorization Flow URL Categorization occurs mostly in the kernel: The kernel cache is queried for the URL. The cache lookup first checks for the host, and if there are members, it looks in the members array. If the URL is found in the cache, its categories are returned. If there is no URL in the cache, then a-sync request is issued, asking for online categorization. The user space daemon sends the URL to its user-space module that communicates with Check Point's online detection service. Once the online service responds with categories, the kernel cache is updated. Features Feature - HTTPS Filtering Categorization of HTTPS sites without HTTPS inspection (passive HTTPS). Supports URL Filtering on HTTPS traffic without HTTPS inspection. To enable it, enable the URL Filtering blade: Go to Application & URL Filtering tab -> Advanced -> Engine Settings -> Enable "Categorize HTTPS sites" and install Security policy How it works A cache table cptls_server_cn_cache is used here. The cache saves mapping between IP+Port to CN (certificate's Canonical Name) and a flag if the CN is valid. When SSL traffic is coming, the request (client hello) is recognized by PM (Pattern Matcher) matching on one of SSL request signatures internal procedure. Then the IP+Port will be extracted from the connection and will be searched in the cptls cache and a CN (certificate's Canonical Name) might be found. If a CN is found and valid, it will be sent to RAD (Resource ADvisor - internal engine responsible for URL Filtering) and the traffic will be categorized according to it. Otherwise, the traffic is continued to be inspected. When a response (server hello) is coming, it is recognized by PM (Pattern Matcher) matching on one of SSL response signatures internal procedure, the certificate will be extracted and sent to WSTLS process in User mode (in one or more traps). The process will try to resolve the certificate and find its CN. It will return an asynchronic ioctl with response to the kernel with the IP+Port and the CN (if found). If a CN is returned, it will be sent to RAD and will be categorized according to it. Otherwise, the IP address will be sent to RAD and will be categorized according to it. Example: if the certificate's CN is google.com, the categories will be "search engine/portal" Debugging & Troubleshooting Run the following debug on the Security Gateway: CLI command Description for PROC in $(pidof wstlsd) ; do fw debug $PROC on TDERROR_ALL_ALL=6 ; done Debug wstlsd in User space - refer to sk105559 (output will be in $FWDIR/log/wstlsd.elg) fw ctl debug 0 fw ctl debug -buf 32000 fw ctl debug -m fw + cptls fw ctl kdebug -T -f > /var/log/kernel_debug.txt Debug cptls in kernel (output will be in /var/log/kernel_debug.txt) fw ctl debug 0 fw ctl debug -buf 32000 fw ctl debug -m APPI all fw ctl kdebug -T -f > /var/log/appi_debug.txt  Debug APPI in kernel (output will be in /var/log/appi_debug.txt) fw ctl debug 0 fw ctl debug -buf 32000 fw ctl debug -m RAD_KERNEL all fw ctl kdebug -T -f > /var/log/rad_debug.txt Debug RAD in kernel (output will be in /var/log/rad_debug.txt) rad_admin rad debug on all Debug RAD in User space (output will be in $FWDIR/log/rad.elg) To stop, run: rad_admin rad debug off ALL Problem: HTTPS Filtering feature is not working. Troubleshooting: Make sure you are surfing via port 443 (and not through proxy). Check that the website you are surfing to is certificated by one of the trusted CAs configured on the SmartDashboard. Go to Application & URL Filtering Tab -> Advanced -> HTTPS inspection -> Trusted CAs. If not, add its certificate. Check that wstlsd process is up. Run ps -ax | grep wstlsd Check that the cptls_server_cn_cache cache table is loaded. Run fw tab -t cptls_server_cn_cache -s Try to find your connection in the table. Run fw tab -t cptls_server_cn_cache and search your IP+Port (in hex) in the 2 left columns. Analyze the debug output file: Check that the client side was mtached by the PM. Search: appi_rad_uf_cmi_handler_match_signature: group_data->group_id 10002002 Check if the IP+Port was found in cptls table. Search: fwtls_get_server_cn_from_cache_ex Check if the server side was matched by the PM. Search: appi_rad_uf_cmi_handler_match_signature: group_data->group_id 10002003 Check if the certificate was extracted from the traffic. Search: "CERTIFICATE" [in upper case] Check if the traps were sent to WSTLSD process. Search: appi_rad_uf_cmi_handler_ssl_sn_trap -> when you see _last 1 - it means it is the last trap. Check if you got ioctl from WSTLDS process. Search: appi_rad_uf_ssl_cn_cmi_ioctl Check if anything was sent to RAD. Search: rad_kernel_urlf_request_set_url Make sure you have connectivity to RAD and get response. Feature - Safe Search Safe Search feature in web browsers is designed to screen sites that contain sexually explicit content and remove them from the search results. While no filter is 100% accurate, Safe Search helps to avoid content you may prefer not to see or would rather your children did not stumble across. User can change the settings of Safe Search from the browser for each specific search engine. This feature supports Safe Search in search engines (currently Google, Bing and Yahoo). To enable it, enable the URL Filtering blade. Go to Application & URL Filtering tab -> Advanced -> Engine Settings -> Enable "Enforce Safe Search in search engines" and install Security policy How it works Each search engine has its saved "&meter=value" that is added to the URL when 'Safe search' feature is enabled or disabled. For example, when enabling Safe Search engine in Google, it adds "&safe=active" to the URL. And when disabling it, it adds "&safe=off" to the URL. Debugging & Troubleshooting Run the following debug on the Security Gateway: CLI command Description fw ctl debug 0 fw ctl debug -buf 32000 fw ctl debug -m APPI all fw ctl kdebug -T -f > /var/log/appi_debug.txt Debug Application Control in kernel (output will be in /var/log/appi_debug.txt) Problem: Safe Search feature is not working. Troubleshooting: Make sure that you use one of the supported search engines (Google, Bing or Yahoo). Capture the traffic using WireShark and see if you get redirect from the Security Gateway. Verify that your URL is matched on one of the signatures from $FWDIR/appi/update/appi_db.C: In the debug output file search for: appi_rad_uf_cmi_handler_match_signature: group_data->group_id #group number of relevant section of signatures from urlf_group# If one of the signatures was matched on your URL, analyze the debug output and see how the redirect URL was built.In the debug output file search for: appi_rad_uf_cmi_handler_safe_search_handler: redirect_url: #here comes the editted URL# If no signature was matched on your URL, send your URL to app_request@checkpoint.com with request to add a new signature to our list. Feature - Cached and Translated pages categorization With this feature disabled, when a user access the cached pages or Google translated pages, the matched category is ‘Search Engine’ since we drop the query string and send to RAD (Resource ADvisor) the Search Engine domain only. For example, when accessing http://www.ynet.co.il cached page from Google, the URL looks like: http://webcache.googleusercontent.com/search?ix=seb&sourceid=chrome&ie=UTF-8&q=cache%3Awww.ynet.co.il. The category will be the category of "webcache.googleusercontent.com" (Search Engine). This feature extracts the correct domain from the cached / Google translated URL buffer, sends it to RAD and then gets the exact category of the page. To enable it, enable the URL Filtering blade. Go to Application & URL Filtering tab -> Advanced -> Engine Settings -> Enable "Enforce Safe Search in search engines" and install Security policy How it works The signatures are compiled to one PM (Pattern Matcher) that recognizes URLs that try to access the cached/translated pages. When a URL is coming, the PM matching (built from the relevant signatures of this feature) is executed against it. If there is a match on one of the signatures, it means that it is cached or translated page. According to the signature match offset, the host is extracted and sent to RAD and the client will get the correct category of the page. Debugging & Troubleshooting Run the following debug on the Security Gateway: CLI command Description fw ctl debug 0 fw ctl debug -buf 32000 fw ctl debug -m RAD_KERNEL allfw ctl kdebug -T -f > /var/log/rad_debug.txt Debug RAD in kernel (output will be in /var/log/rad_debug.txt) rad_admin rad debug on all Debug RAD in User space (output will be in $FWDIR/log/rad.elg) To stop, run: rad_admin rad debug off ALL  fw ctl debug 0 fw ctl debug -buf 32000 fw ctl debug -m APPI all fw ctl kdebug -T -f > /var/log/appi_debug.txt Debug APPI in kernel (output will be in /var/log/appi_debug.txt) Problem: Cached/Translated pages feature is not working. Troubleshooting: Make sure that you are using one of the supported search engines on cached pages (Google, Bing or Yahoo). And Google for translated pages. Verify that your URL is matched with one of the signatures from $FWDIR/appi/update/appi_db.C. In the debug output file search for: appi_rad_uf_cmi_handler_match_signature: group_data->group_id 10002000 If one of the signatures was matched on your URL, run debug and see how the cached/translated domain is extracted from the URL. In the debug output file search for: appi_rad_uf_cmi_handler_extract_url_after_pm_match See that the extracted domain is sent to RAD. In the debug file search for rad_kernel_urlf_request_set_url Make sure you have connectivity to RAD and get response. If no signature was matched on your URL send your URL to app_request@checkpoint.com with request to add a new signature to our list. URL Filtering Agent General Description The new URL Filtering software blade uses a hosted categorization service (cloud-based categorization) that is being constantly updated to cope with the dynamic nature of the web. The Security Gateway is not required to download updates, the categorization database is not limited by the storage space on the Security Gateway, and customers receive the latest categorization to keep policies current. As requests are being sent to the cloud, responses are being cached locally on the Security Gateway so that subsequent requests for the same sites will not query the cloud, but will be answered by the local cache. In deployments of R75.20 URL Filtering, we have seen that more than 99% of requests are being met by the cache, providing customers with the most up-to-date categorization (with the cloud) on the one hand and minimal latency (as most requests are being met by the cache), on the other. When there are critical updates, for example, a previously benign site has been hacked and is now used to spread malware, the cache is invalidated and forced to retrieve updates from the cloud. Processes The R75.20 URL Filtering blade uses a single user mode process called RAD ($FWDIR/bin/rad) to communicate with the online URL Filtering service. The process is initiated once application control, or URL filtering blades, are active on the Security Gateway. Whenever RAD successfully categorizes a URL, it caches it in its kernel cache. The URL Filtering kernel cache holds the entire URL Family (i.e. all the URLs within the same host). Whenever there is a request for a new URL categorization, RAD sends to the online categorization service only the host part of the URL and receives the entire URL family. The response is then cached for future use. Important Files RAD holds its configuration in 3 files: $FWDIR/conf/rad_settings.C - RAD proxy settings (such as proxy settings) $FWDIR/conf/rad_scheme.C - a file describing where and how to connect to the online services; reloaded on each install policy event. $FWDIR/conf/rad_services.C - a database file that holds the kernel cache setting, such as cache table sizes and max allowed memory usage. URL Filtering Updates and local database (LDB) files The RAD kernel cache holds the entire URL Family. Whenever there is a request for a new URL categorization, RAD sends to the online categorization service only the host part of the URL and receives the entire URL family. To keep the response size short, the online service answers only if the URL family has 100 or less members. 99% of the URL database are part of families with 100 or less members and can be retrieved using the online service. The remaining 1% of the database is aggregated in a special file: $FWDIR/appi/update/urlf_db.bin The file is reloaded on each URL Filtering update. Before sending a new online categorization request to the service, RAD looks in the local LDB file for a match. Only if the URL does not exist in the file, a request to the service is made. To trigger a reload of the LDB file: rad_admin urlf update $FWDIR/appi/update/urlf_db.bin Website categorization mode You can select the mode that is used for web site categorization: Background: requests are allowed until categorization is complete. When a request cannot be categorized with a cached response, an uncategorized response is received. Access to the site is allowed. In the background, the Check Point Online Web Service continues the categorization procedure. The response is then cached locally for future requests (default). This option reduces latency in the categorization procedure. Hold: requests are blocked until categorization is complete. When a request cannot be categorized with the cached responses, it remains blocked until the Check Point Online Web Service completes categorization. RAD CLI Topic CLI Explanation Example Activation rad_admin start Start RAD process   Activation rad_admin stop Stop RAD process   Activation rad_admin install Install new RAD policy. RAD will reparse its rad_setting.C file.   Debug rad_admin rad debug on <topic> Start debug (output is written into $FWDIR/log/rad.elg) rad_admin rad debug on all Note: "all" must be in small letters Debug rad_admin rad debug off Stop debug rad_admin rad debug off ALL Note: "ALL" must be in capital letters Statistics rad_admin stats on <service> Start collecting statistics rad_admin stats on urlf Statistics rad_admin stats print <service> Print currents statistics to $FWDIR/log/<service>_rad_stats.csv rad_admin stats print urlf Statistics rad_admin stats reset <service> Reset all stats counters rad_admin stats reset urlf Statistics rad_admin stats print_reset <service> Print current statistics to $FWDIR/log/<service>_rad_stats.csv and reset stats counters rad_admin stats print_reset urlf Statistics rad_admin stats off <service> Stop collecting statistics rad_admin stats off urlf Update rad_admin urlf update <full path> Reload URL Filtering local database file   Fast Path rad_admin fastpath <service> Issue a request and update security categories in Security Gateway's cache  rad_admin fastpath urlf Main Debug Flags CLI Command GW/ Mgmt Kernel/ User Space Description fw ctl debug 0 fw ctl debug -buf 32000 fw ctl debug -m RAD_KERNEL all fw ctl kdebug -T -f > /var/log/debug_kernel.txt Security Gateway Kernel Start debug of RAD kernel module rad_admin rad debug on Security Gateway User Space Start debug of RAD process (debug is written into $FWDIR/log/rad.elg)