Default actions for all events of a hash (e.g., all requests) can be set using the '*' wildcard as the event. (Most hashes have default actions configured by default, these can be found in the description of the proxy classes.) It is important to note that setting the action using the '*' wildcard does NOT override an action explicitly defined for an event, even if the explicit setting precedes the general one in the Python code. This feature is illustrated in the example below.

class MyFtp1(FtpProxy):
	def config(self):
		self.request["APPE"] = (FTP_REQ_REJECT)
		self.request["*"] = (FTP_REQ_ACCEPT)

class MyFtp2(FtpProxy):
	def config(self):
		self.request["*"] = (FTP_REQ_ACCEPT)
		self.request["APPE"] = (FTP_REQ_REJECT)

Responses in certain protocols include numeric response codes, e.g., in the FTP protocol responses start with a three-digit code. In Zorp it is possible to filter these codes as well, furthermore, to filter them based on the command to which the response arrives to. In these cases the hash contains both the command and the answer, and an action as well. The '*' wildcard character can be used to match for every command or response code.

class MyFtp1(FtpProxy):
	def config(self):
		self.response["DELE", "250"] = (FTP_RSP_ACCEPT)
		self.response["*", "250"] = (FTP_RSP_REJECT)
		self.response["LIST", "*"] = (FTP_RSP_ACCEPT)

It is not necessary to specify the full response code, it is also possible to specify only the first, or the first two digits.

For example, all three response codes presented below are valid, but have different effects:

This kind of response code lookup is available in the following proxies: FTP, HTTP, NNTP, and SMTP. The precedence how the hash table entries are processed is the following:

Proxy stacking is mainly used to inspect embedded protocols, or perform virus filtering: e.g., to inspect HTTPS traffic, the external SSL protocol is examined with a Pssl proxy, and then a HTTP proxy is stacked to inspect the internal protocol. It is possible to stack several layers of proxies into each other if needed, e.g., in the above example, a further virus filtering solution (like a ZCV module) could be stacked into the HTTP proxy.

Stacking a proxy to inspect the embedded protocol is possible via the self.request_stack attribute; if another attribute has to be used, it is noted in the description of the given proxy. The HTTP proxy is special in the sense that it is possible to stack different proxies into the requests and the responses.

The parameters of the stack attribute has to specify the following:

The use of proxy stacking is illustrated in the following example:

class HttpsPsslProxy(PsslProxy):
	def config(self):
		PsslProxy.config(self)
		self.stack_proxy=(Z_STACK_PROXY, HttpProxy)

For additional information on proxy stacking, see Section 6.6.3, Analyzing embedded traffic in Zorp Professional 7 Administrator Guide, and the various tutorials available at the Balasys Documentation Page.

When stacking a program, the data received by a proxy within a protocol is directed to the standard input. Arbitrary commands (including command line scripts, or applications) working from the standard input can be run on this data stream. The original proxy obtains the processed data back from the standard output. When stacking a command, the command to be called has to be included in the proper stack attribute of the proxy between double-quotes. This is illustrated in the following example.

class MyHttp(HttpProxy):
	def config(self):
		HttpProxy.config(self)
		self.response_stack["GET"] = /
		(HTTP_STK_DATA, (Z_STACK_PROGRAM, "/bin/sed '/http:/s//https:/g'"))

The SSL framework inspects SSL/TLS connections, and also any other connections embedded into the encrypted SSL/TLS channel. SSL/TLS connections initiated from the client are terminated on the firewall, and two separate SSL/TLS connections are built: one between the client and the firewall, and one between the firewall and the server. If both connections match the configuration settings of Zorp (for example, the certificates are valid, and only the allowed encryption algorithms are used), Zorp inspects the protocol embedded into the secure channel as well. Note that the configuration settings can be different for the two connections, for example, it is possible to permit different protocol versions and encryption settings.

When a firewall rule matches an incoming connection, Zorp starts the Service specified in the firewall rule to inspect the connection. The Encryption policy set in the Service determines the encryption settings used in the connection.

Depending on the scenario (TwoSidedEncryption, ClientOnlyEncryption, and so on) set in the Encryption policy, the SSL framework selects the first peer to perform the SSL handshake with.

As part of the handshake process, Zorp checks if encryption is required on the given side. It is not necessary for SSL to be enabled on both sides - Zorp can handle one-sided SSL connections as well (for example, the firewall communicates in an unencrypted channel with the client, but in a secure channel with the server). If SSL is not enabled, the handshake is skipped for that side.

When SSL is needed, the Service collects the required parameters (keys, certificates, and so on) from the Encryption policy.

The SSL handshake is slightly different for the client (in this case Zorp behaves as an SSL server) and the server (when Zorp behaves as an SSL client):

Starting with version 6.0, Zorp supports session reuse in SSL and TLS connections. Zorp supports both session identifiers (RFC 8446) and session tickets (RFC 8446). Note that session tickets can be used only in TLS connections. Unless explicitly disabled in the configuration of the Encryption policy (for details, see Section 5.5, Module Encryption), Zorp attempts to use session tickets, and automatically falls back to using session identifiers if needed.

This section describes the configuration blocks of Encryption policies and objects used in Encryption policies. Encryption policies were designed to be flexible, and make encryption settings easy to reuse in different services.

An Encryption policy is an object that has a unique name, and references a fully-configured encryption scenario.

Encryption scenarios are actually Python classes that describe how encryption is used in a particular connection, for example, both the server-side and the client-side connection is encrypted, or the connection uses a one-sided SSL connection, and so on. Encryption scenarios also reference other classes that contain the actual settings for the scenario. Depending on the scenario, the following classes can be set for the client-side, the server-side, or both.

Zorp provides the following built-in encryption scenarios:

For example, on configuring Encryption policies, see How to configure SSL proxying in Zorp 7. For details on HTTPS-specific problems and the related solutions, see How to configure HTTPS proxying in Zorp 7.

To configure Encryption policies, you have to create an Encryption policy, and derive and configure your own scenario from the available built-in scenarios. To configure a scenario, you have to derive and configure your own certificate generator, certificate verifier, and protocol settings classes. (Do not change the built-in classes directly, because that changes the default behavior of Zorp, and can have unexpected and unwanted effects on the configuration of Zorp.)

For a details on configuring Encryption Policies, see the following procedure, or the How to configure SSL proxying in Zorp 7 tutorial.

Zorp is able to automatically verify the certificates received.

When checking revocation state for certificate chains Zorp offers two options:

To support both maximum security verification and the more common use-cases with less strict scenarios, the level of strictness for the verification can be configured separately for leaf and for non-leaf certificates. The configuration is done in encryption policies using options intermediate_revocation_check_type and leaf_revocation_check_type.

For both intermediate_revocation_check_type and leaf_revocation_check_type options 3 values are available:

intermediate_revocation_check_type also controls the chain verifier behavior of OCSP stapling. If no CRL list from the intermediate CA is available, the OCSP stapling signer's 'certification revoked' state cannot be determined. If intermediate_revocation_check_type has the HARD_FAIL value, the OCSP stapling message is not accepted as a valid revocation information. However, if the OCSP stapling message shows 'revoked state' the message is considered regardless of its chain verification.

The following tables describe how the values, that is NONE, HARD_FAIL or SOFT_FAIL influence the result of the revocation check type:

The types of accepted certificates can be controlled separately on the client and the server side using the attributes of the ClientCertificateVerifier and ServerCertificateVerifier classes (or your own classes derived from these), respectively.

By default (if the check_subject parameter is set to TRUE in the verifier), Zorp compares the domain name provided in the Subject field of the server certificate to application-level information about the server (that is, the domain name of the URL in HTTP and HTTPS connections).

EncryptionPolicy(
    name="MyTLSEncryption",
    encryption=TwoSidedEncryption(
        client_verify=ClientNoneVerifier(),
        server_verify=ServerCertificateVerifier(
            trust_level=TLS_TRUST_LEVEL_NONE, intermediate_revocation_check_type = 
            TLS_INTERMEDIATE_REVOCATION_NONE, leaf_revocation_check_type = 
            TLS_LEAF_REVOCATION_NONE,
            trusted_certs_directory="",
            verify_depth=4,
            verify_ca_directory="/etc/ca.d/certs/",
            verify_crl_directory="/etc/ca.d/crls/",
            check_subject=FALSE
            )

The following sections describe and show examples to common protocol-level TLS settings.

EncryptionPolicy(
    name="MyTLSEncryption",
    encryption=TwoSidedEncryption(
        client_verify=None,
        server_verify=ServerCertificateVerifier(
            ca_directory="/etc/ca.d/certs/",
            crl_directory="/etc/ca.d/crls/",
            )
        client_ssl_options=ServerSSLOptions(disable_tlsv1=TRUE),
        server_ssl_options=ServerSSLOptions(disable_tlsv1=TRUE),
            )
        )

Zorp supports the STARTTLS method for encrypting connections. STARTTLS support can be configured separately for the client- and server side. Currently, the following proxies support STARTTLS: Ftp proxy (to start FTPS sessions), Smtp proxy, and Pop3 proxy.

STARTTLS is enabled by default in the following encryption scenarios:

class FtpsProxy(FtpProxy):
    def config(self):
        FtpProxy.config(self)
        self.max_password_length=64

    EncryptionPolicy(name="ForwardSTARTTLS", encryption=ForwardStartTLSEncryption(client_verify=ClientCertificateVerifier(), client_ssl_options=ClientSSLOptions(), server_verify=ServerCertificateVerifier(), server_ssl_options=ServerSSLOptions(), client_certificate_generator=DynamicCertificate(private_key=PrivateKey.fromFile(key_file_path="/etc/key.d/ZMS_Engine/key.pem"), trusted_ca=Certificate.fromFile(certificate_file_path="/etc/ca.d/certs/my-trusted-ca-cert.pem", private_key=PrivateKey.fromFile("/etc/ca.d/keys/my-trusted-ca-cert.pem")), untrusted_ca=Certificate.fromFile(certificate_file_path="/etc/ca.d/certs/my-untrusted-ca-cert.pem", private_key=PrivateKey.fromFile("/etc/ca.d/keys/my-untrusted-ca-cert.pem")))))

    def demo() :
        Service(name='demo/MyFTPSService', router=TransparentRouter(), chainer=ConnectChainer(), proxy_class=FtpsProxy, max_instances=0, max_sessions=0, keepalive=Z_KEEPALIVE_NONE, encryption_policy="ForwardSTARTTLS")

    Rule(rule_id=2,
    proto=6,
    service='demo/MyFTPSService'
    )

A certificate name behaves as a string, and contains a DN in the following format (also known as one-line format):

/RDN=value/RDN=value/.../RDN=value/

The word RDN stands for relative distinguished name. For example, the DN cn=Root CA, ou=CA Group, o=Foo Ltd, l=Bar, st=Foobar State, c=US becomes /C=US/ST=Foobar State/L=Bar/O=Foo Ltd/OU=CA Group/CN=Root CA/

A certifying authority may revoke the issued certificates. A revocation means that the serial number and the revocation date is added to the list of revoked certificates. Revocations are published on a regular basis. This list is called the Certificate Revocation List, also known as CRL. A CRL always has an issuer, a date when the list was published, and the expected date of its next update.

Online Certificate Status Protocol (OCSP) stapling is an alternative to Certificate Revocation Lists (CRL) in verifying the validity of certificates. The protocol is described in details in IETF RFC 6960. It is now also possible to define to what level of strictness the encryption policies shall check the revocation status of the certificates. OCSP stapling provides a potentially faster revocation state with less traffic.

The proxy stores trusted CA certificates in a Certificate hash. This hash can be indexed by two different types. If an integer index is used, the slot specified by this value is looked up; if a string index is used, it is interpreted as a one-line DN value, and the appropriate certificate is looked up. Each slot in this hash contains an X.509 certificate.

Similarly to the certificate hash, a separate hash for storing Certificate Revocation Lists was defined. A CRL contains revocation lists associated to CAs.

This class encapsulates AnyPy, a proxy module calling a Python function to do all of its work. It can be used for defining proxies for protocols not directly supported.

This class encapsulates AnyPy, a proxy module calling a Python function to do all of its work. It can be used for defining proxies for protocols not directly supported.

Finger is a request/response based User Information Protocol using port TCP/79. The client opens a connection to the remote machine to initiate a request. The client sends a one line query based on the Finger query specification and waits for the answer. A remote user information program (RUIP) processes the query, returns the result and closes the connection. The response is a series of lines consisting of printable ASCII closed carriage return-line feed (CRLF, ASCII13, ASCII10). After receiving the answer the client closes the connection as well.

The following queries can be used:

Finger is a module built for parsing messages of the Finger protocol. It reads the QUERY at the client side, parses it and - if the local security policy permits - sends it to the server. When the RESPONSE arrives it processes the RESPONSE and sends it back to the client. It is possible to prepend and/or append a string to the response. Requests can also be manipulated in various ways using the fingerRequest function, which is called by the proxy if it is defined.

Length of the username, the line and the hostname can be limited by setting various attributes. Finger proxy also has the capability of limiting the number of hosts in a request, e.g.: finger user@domain@server normally results in fingering 'user@domain' performed by the host 'server'. By default, the proxy removes everything after and including the first '@' character. This behavior can be modified by setting the max_hop_count attribute to a non-zero value.

def MyFingerProxy(FingerProxy):
        def config(self):
                FingerProxy.config(self)
                self.max_hop_count = 2
                self.timeout = 30

This proxy implements the Finger protocol as specified in RFC 1288.

Simple FingerProxy based on AbstractFingerProxy.

File Transfer Protocol (FTP) is a protocol to transport files via a reliable TCP connection between a client and a server. FTP uses two reliable TCP connections to transfer files: a simple TCP connection (usually referred to as the Control Channel) to transfer control information and a secondary TCP connection (usually referred to as the Data Channel) to perform the data transfer. It uses a command/response based approach, i.e. the client issues a command and the server responds with a 3-digit status code and associated status information in text format. The Data Channel can be initiated either from the client or the server; the Control Channel is always started from the client.

The client is required to authenticate itself before other commands can be issued. This is performed using the USER and PASS commands specifying username and password, respectively.

220 FTP server ready
USER account
331 Password required.
PASS password
230 User logged in.
SYST
215 UNIX Type: L8
PASV
227 Entering passive mode (192,168,1,1,4,0)
LIST
150 Opening ASCII mode data connection for file list
226-Transferring data in separate connection complete.
226 Quotas off
QUIT
221 Goodbye

FtpProxy is a module built for parsing commands of the Control Channel in the FTP protocol. It reads the REQUEST at the client side, parses it and - if the local security policy permits - sends it to the server. The proxy parses the arriving RESPONSES and sends them to the client if the policy permits that. FtpProxy uses a PlugProxy to transfer the data arriving in the Data Channel. The proxy is capable of manipulating commands and stacking further proxies (for example, MimeProxy) into the Data Channel. Both transparent and non-transparent modes are supported.

The default low-level proxy implementation (AbstractFtpProxy) denies all requests by default. Different commands and/or responses can be enabled by using one of the several predefined proxy classes which are suitable for most tasks. Alternatively, use of the commands can be permitted individually using different attributes. This is detailed in the following two sections.

class AnonFtp(FtpProxy):
        def config(self):
                self.request["USER"] = (FTP_REQ_POLICY, self.pUser)
                self.request["*"] = (FTP_REQ_ACCEPT)

        def pUser(self,command):
                if self.request_parameter == "anonymous" or self.request_parameter == "Anonymous":
                        return FTP_REQ_ACCEPT
                return FTP_REQ_REJECT

class FtpsProxy(FtpProxy):
    def config(self):
        FtpProxy.config(self)
        self.max_password_length=64

    EncryptionPolicy(name="ForwardSTARTTLS", encryption=ForwardStartTLSEncryption(client_verify=ClientCertificateVerifier(), client_ssl_options=ClientSSLOptions(), server_verify=ServerCertificateVerifier(), server_ssl_options=ServerSSLOptions(), client_certificate_generator=DynamicCertificate(private_key=PrivateKey.fromFile(key_file_path="/etc/key.d/ZMS_Engine/key.pem"), trusted_ca=Certificate.fromFile(certificate_file_path="/etc/ca.d/certs/my-trusted-ca-cert.pem", private_key=PrivateKey.fromFile("/etc/ca.d/keys/my-trusted-ca-cert.pem")), untrusted_ca=Certificate.fromFile(certificate_file_path="/etc/ca.d/certs/my-untrusted-ca-cert.pem", private_key=PrivateKey.fromFile("/etc/ca.d/keys/my-untrusted-ca-cert.pem")))))

    def demo() :
        Service(name='demo/MyFTPSService', router=TransparentRouter(), chainer=ConnectChainer(), proxy_class=FtpsProxy, max_instances=0, max_sessions=0, keepalive=Z_KEEPALIVE_NONE, encryption_policy="ForwardSTARTTLS")

    Rule(rule_id=2,
    proto=6,
    service='demo/MyFTPSService'
    )

<ftp user>@<proxy user>@<remote site>[:<port>]

<ftp password>@<proxy password>

<ftp user>@<proxy user>@<remote site>[:<port>]:<ftp password>@<proxy password>

This proxy implements the FTP protocol as specified in RFC 959. All traffic and commands are denied by default. Consequently, either customized Ftp proxy classes derived from the abstract class should be used, or one of the predefined classes (e.g.: FtpProxy, FtpProxyRO, etc.).

A permitting Ftp proxy based on the AbstractFtpProxy, allowing all commands, responses, and features, including unknown ones. The connection is terminated if a response with the answer code 421 is received.

FTP proxy based on AbstractFtpProxy, enabling read-only access (i.e. only downloading) to anonymous users (uploads and usernames other than 'anonymous' or 'ftp' are disabled). Commands and return codes are strictly checked, unknown commands and responses are rejected. Every feature is accepted.

The ABOR; ACCT; AUTH; CDUP; CWD; EPRT; EPSV; FEAT; LIST; MODE; MDTM; NLST; NOOP; OPTS; PASV; PASS; PORT; PWD; QUIT; REST; RETR; SIZE; STAT; STRU; SYST; TYPE; and USER commands are permitted, the CLNT; XPWD; MACB commands are rejected.

FTP proxy based on AbstractFtpProxy, enabling full read-write access to anonymous users (the 'anonymous' and 'ftp' usernames are permitted). Commands and return codes are strictly checked, unknown commands and responses are rejected. Every feature is accepted.

The ABOR; ACCT; APPE; CDUP; CWD; DELE; EPRT; EPSV; LIST; MKD; MODE; MDTM; NLST; NOOP; OPTS; PASV; PASS; PORT; PWD; QUIT; RMD; RNFR; RNTO; REST; RETR; SIZE; STAT; STOR; STOU; STRU; SYST; TYPE; USER and FEAT commands are permitted, the AUTH; CLNT; XPWD; MACB commands are rejected.

FTP proxy based on AbstractFtpProxy, enabling read-only access to any user. Commands and return codes are strictly checked, unknown commands and responses are rejected. Every feature is accepted.

The ABOR; ACCT; AUTH; CDUP; CWD; EPRT; EPSV; FEAT; LIST; MODE; MDTM; NLST; NOOP; OPTS; PASV; PASS; PORT; PWD; QUIT; REST; RETR; SIZE; STAT; STRU; SYST; TYPE; and USER commands are permitted, the CLNT; XPWD; MACB commands are rejected.

FTP proxy based on AbstractFtpProxy, enabling full read-write access to any user. Commands and return codes are strictly checked, unknown commands and responses are rejected. Every feature is accepted.

The ABOR; ACCT; AUTH; CDUP; CWD; EPRT; EPSV; FEAT; LIST; MODE; MDTM; NLST; NOOP; OPTS; PASV; PASS; PORT; PWD; QUIT; REST; RETR; SIZE; STAT; STRU; SYST; TYPE; and USER commands are permitted, the CLNT; XPWD; MACB commands are rejected.

HTTP is an open application layer protocol for hypermedia information systems. It basically allows an open-ended set of methods to be applied to resources identified by Uniform Resource Identifiers (URIs).

GET /index.html HTTP/1.1
Host: www.example.com
Connection: keep-alive
User-Agent: My-Browser-Type 6.0

HTTP/1.1 200 OK
Connection: close
Content-Length: 14

<html>
</html>

The default low-level proxy implementation (AbstractHttpProxy) denies all requests by default. Different requests and/or responses can be enabled by using one of the several predefined proxy classes which are suitable for most tasks. Alternatively, a custom proxy class can be derived from AbstractHttpProxy and the requests and responses enabled individually using different attributes.

Several examples and considerations on how to enable virus filtering in the HTTP traffic are discussed in the Technical White Paper and Tutorial Virus filtering in HTTP, available at the BalaSys Documentation Page http://www.balasys.hu/documentation/.

GET http://www.example.com/index.html HTTP/1.1
Host: www.example.com
Connection: keep-alive
User-Agent: My-Browser-Type 6.0

HTTP/1.1 200 OK
Connection: close
Content-Length: 14

<html>
</html>

CONNECT www.example.com:443 HTTP/1.1
Host: www.example.com
User-agent: My-Browser-Type 6.0

HTTP/1.0 200 Connection established
Proxy-agent: My-Proxy/1.1

class DmzHTTP(HttpProxy):
        def config(self):
                HttpProxy.config(self)
                self.request["GET"] = (HTTP_REQ_POLICY, self.filterURL)

        def filterURL(self, method, url, version):
                if (url == "http://www.disallowedsite.com"):
                        self.error_info = 'Access of this content is denied by the local policy.'
                        return HTTP_REQ_REJECT
                return HTTP_REQ_ACCECT

class DmzHTTP(HttpProxy):
        def config(self):
                HttpProxy.config(self)
                self.response["GET", "404"] = (HTTP_RSP_POLICY, self.filter404)

        def filter404(self, method, url, version, response):
                self.error_status = 404
                self.error_info = "Requested page was not accessible."
                return HTTP_RSP_REJECT

class MyHttp(HttpProxy):
        def config(self):
                HttpProxy.config(self)
                self.request_header["User-Agent"] = (HTTP_HDR_CHANGE_VALUE, "Lynx 2.4.1")
                self.request_header["Cookie"] = (HTTP_HDR_POLICY, self.processCookies)
                self.response_header["Set-Cookie"] = (HTTP_HDR_DROP,)

        def processCookies(self, name, value):
                # You could change the current header in self.current_header_name
                # or self.current_header_value, the current request url is
                # in self.request_url
                return HTTP_HDR_DROP

class MyHttp(HttpProxy):
        def config(self):
                HttpProxy.config(self)
                self.request["GET"] = (HTTP_REQ_POLICY, self.filterURL)

        def filterURL(self, method, url, version):
                self.request_url = "http://www.example.com/"
                return HTTP_REQ_ACCEPT

class HttpProxyHttpsredirect(HttpProxy):
        def config(self):
                HttpProxy.config(self)
                self.error_silent = TRUE
                self.request["GET"] = (HTTP_REQ_POLICY, self.reqRedirect)

        def reqRedirect(self, method, url, version):
                self.error_status = 301
                #self.error_info = 'HTTP/1.0 301 Moved Permanently'
                self.error_headers="Location: https://%s/" % self.request_url_host
                return HTTP_REQ_REJECT

class MyHttp(HttpProxy):
        def config(self):
                HttpProxy.config(self)
                self.parent_proxy = "proxy.example.com"
                self.parent_proxy_port = 3128

def instance():
        Service("http", MyHttp, router=InbandRouter())
        Listener(SockAddrInet('10.0.0.1', 80), "http")

4.7.2.12. URL filtering in HTTP

From Zorp version 7.0.5 a new and enhanced version of the integrated category-based URL filtering solution is available, leveraging a permanently updated database. During the installation of the URL filtering database the administrator can choose the size of the URL filtering database. The database can be a smaller-sized, optimized database (the recommended version) for usual scenarios, which requires 1 GB storage space and 300 MB daily update traffic, or a normal database for more extensive scenarios, which requires 6 GB storage space and 2 GB daily update traffic. Note, that during the upgrade, the 6 GB storage space usage will temporarily increase to 18-20 GB.

For this new, extensive URL filter solution it is necessary to match the new filter categories manually with the corresponding old categories of the previous URL filter. The categories from the previous version of the URL filter will remain in the system and will be labeled as 'uncategorized' until they are defined differently. For the elements of the 'uncategorized' category the operations defined earlier (acceptance, prohibition, redirection) are applicable. Also note that the handling of the whitelists and blacklists has also changed.

• For an overview on how URL-filtering works, see Section How URL filtering works.

• For information on installing and selecting the size of the URL filter database, see Procedure 4.1.2, ZCV — Configuring the zorp-utils package in Zorp Professional 7 Installation Guide .

• To configure URL filtering, see Section Configuring URL filtering in HTTP.

• To customize or expand the URL-database, see Section Customizing the URL database.

• For the list of categories available by default, see Section List of URL-filtering categories.

How URL filtering works

The Zorp URL filter checks the URL of the HTTP requests and compares them to a categorized database. The URLs and the domains in the database are organized into thematic categories like adult, health, business, and so on. If the requested URL is listed in the database, the categories matching the URL are assigned to the request. Zorp can accept, redirect, or reject the HTTP request based on the category it belongs to.

Typically, accessing a webpage involves several HTTP requests. URL filtering is applied to every single HTTP request, meaning that different parts of the webpage can belong to different categories. Zorp can remove the parts belonging to unwanted categories, and permit access only to the remaining part. For example, this can be used to prohibit access to advertisements or videos, but to permit access to other parts of the website.

Note
URL filtering is handled by the Zorp Http proxy, without the need of using ZCV. The URL filtering capability of Zorp is available only after purchasing the url-filter license option. Updates to the URL database are automatically downloaded daily from the Balasys website using the zavupdate utility.

Configuring URL filtering in HTTP

Prior to the update to the new URL filter, it is necessary to check whether at least 1.7 GB free space is avaialble on the partition including the /var/lib/zorp/urlfilter/ folder. Following the update, the partition will utilize close to 850 MB space. After the update of Zorp Management Server (ZMS) to version 7.0.5, when signing in to Zorp Management Console (ZMC), the following notification pops up reminding on the changes with the URL fiter.

Figure 4.1. Notification by conversion

To enable url-filtering, follow the forthcoming steps:

1. Choose the proper URL filter proxy under Zorp component → Proxies menu item.

2. Edit the self.url_category under the Attribute menu item.

   You can find the categories from the previous version of the URL filter under the Key menu item. These items now belong to the 'uncategorized' group category.

   

   Figure 4.2. self.url_category attribute with the old categories yet

   
   

3. Click the Edit key button and the selection of the new categories will be displayed:

   

   Figure 4.3. The menu of the new category selection

   
   

4. Choose the new category corresponding to the old one, from the list.

   For the list of categories available by default, see Section List of URL-filtering categories. It is not necessary to change all the categories at once. The remaining elements will still remain as 'uncategorized' in the background.

   While searching for the categories, it might be helpful to use the Ctrl + F key combination. Alternatively, when starting typing, the findings matching the first keystrokes will pop up in a list of highlights. Among these highlighted matches we can choose with the 'up' and 'down' arrows.

   

   Figure 4.4. self.url_category attribute with partly new categories

   
   

5. After pressing the commit button, validate the changes with the check configuration option.

   

   Figure 4.5. Validating configuration

   
   

   The update of the URL filter database is performed by the zavupdate command, run by the cron job, every day at 11 pm by default. Use the login name and password necessary for accessing the apt repo as well. All these ensure a solution that can be updated more easily.

List of URL-filtering categories

The Zorp URL database contains the following thematic categories by default.

Main category	Subcategory
Adult	Abortion, Abortion Pro Choice, Abortion Pro Life, Child Inappropriate, Gambling, Gay, Lesbian, Bisexual, Lingerie, Suggestive and Pinap, Nudity, Pornography, R-Rated, Sex and Erotic, Sex Education and Pregnancy, Tobacco
Aggressive	Military, Violence, Weapons, Aggressive - Other
Arts	Fine Art, Arts - Other
Automotive	Auto Parts, Auto Repair, Buying/Selling Cars, Car Culture, Certified Pre-Owned, Convertible, Coupe, Crossover, Diesel, Electric Vehicle, Hatchback, Hybrid, Luxury, MiniVan, Motorcycles, Off-Road Vehicles, Performance Vehicles, Pickup, Road-Side Assistance, Sedan, Trucks & Accessories, Vintage Cars, Wagon, Automotive - Other
Business	Agriculture, Biotechnology, Business Software, Construction, Forestry, Government, Green Solutions & Conservation, Home & Office Furnishings, Human Resources, Manufacturing, Marketing Services, Metals, Physical Security, Productivity, Retirement Homes & Assisted Living, Shipping & Logistics,Business - Other
Careers	Career Advice, Career Planning, College, Financial Aid, Job Fairs, Job Search, Nursing, Resume Writing/Advice, Scholarships, Telecommuting, U.S. Military, Careers - Other
Criminal Activities	Child Abuse Images, Criminal Skills, Hacking, Hate Speech, Illegal Drugs, Marijuana, Piracy & Copyright Theft, School Cheating, Self Harm, Torrent Repository, Criminal Activities - Other
Dynamic	Anonymizer, Chat, Community Forums, Instant Messenger, Login Screens, Personal Pages & Blogs, Photo Sharing, Professional Networking, Redirect, Social Networking, Text Messaging & SMS, Translator, Web-based Email, Web-based Greeting Card
Education	7-12 Education, Adult Education, Art History, College Administration, College Life, Distance Learning, Educational Institutions, Educational Materials & Studies, English as a 2nd Language, Graduate School, Homeschooling, Homework/Study Tips, K-6 Educators, Language Learning, Literature & Books, Private School, Reference Materials & Maps, Special Education, Studying Business, Tutoring, Wikis, Education - Other
Entertainment	Entertainment News & Celebrity Sites, Entertainment Venues & Events, Humor, Movies, Music, Streaming & Downloadable Audio, Streaming & Downloadable Video, Television, Entertainment - Other
Family and Parenting	Adoption, Babies and Toddlers, Daycare/Pre School, Eldercare, Family Internet, Parenting - K-6 Kids, Parenting Teens, Pregnancy, Special Needs Kids, Family & Parenting - Other
Fashion	Accessories, Beauty, Body Art, Clothing, Fashion, Jewelry, Swimsuits, Fashion - Other
Finance	Accounting, Banking, Beginning Investing, Credit/Debt & Loans, Financial News, Financial Planning, Hedge Fund, Insurance, Investing, Mutual Funds, Online Financial Tools & Quotes, Options, Retirement Planning, Stocks, Tax Planning, Finance - Other
Food and Drink	American Cuisine, Barbecues & Grilling, Cajun/Creole, Chinese Cuisine, Cocktails/Beer, Coffee/Tea, Cuisine-Specific, Desserts & Baking, Dining Out, Food Allergies, French Cuisine, Health/Low fat Cooking, Italian Cuisine, Japanese Cuisine, Mexican Cuisine, Vegan, Vegetarian, Winer, Food & Drink - Other
Health	A.D.D., AIDS/HIV, Allergies, Alternative Medicine, Arthritis, Asthma, Autism/PDD, Bipolar Disorder, Brain Tumor, Cancer, Children's Health, Cholesterol, Chronic Fatigue, Chronic Pain, Cold & Flu, Cosmetic Surgery, Deafness, Dental Care, Depression, Dermatology, Diabetes, Disorders, Epilepsy, Exercise, GERD/Acid Reflux, Headaches/Migraines, Heart Disease, Herbs for Health, Holistic Healing, IBS/Crohn’s Disease, Incest/Abuse Support, Incontinence, Infertility, Men’s Health, Nutrition & Diet, Orthopedics, Panic/Anxiety, Pediatrics, Pharmaceuticals, Physical Therapy, Psychology/PsychiatrySelf-help & Addiction, Senior Health, Sexuality, Sleep Disorders, Smoking Cessation, Supplements & Compounds, Syndrome, Thyroid Disease, Weight Loss, Women’s Health, Health - Other
Hobbies and Interests	Art/Technology, Arts & Crafts, Beadwork, Birdwatching, Board Games/Puzzles, Candle & Soap Making, Card Games, Cartoons, Anime & Comic Books, Chess, Cigars, Collecting, Comic Books, Drawing/Sketching, Freelance Writing, Genealogy, Getting Published, Guitar, Home Recording,Investors & Patents, Jewelry Making, Magic & Illusion, Needlework, Painting, Photography, Radio, Roleplaying Games, Sci-Fi & Fantasy, Scrapbooking, Screenwriting, Stamps & Coins, Themes, Video & Computer Games, Woodworking, Hobbies & Interests - Other
House and Garden	Appliances, Entertaining, Environmental Safety, Gardening, Home Repair., Home Theater, Interior Decorating, Landscaping, Remodeling & Construction, Home & Garden - Other
Kids	Games, Kid's Pag, Toys, Kids - Other
Lifestyle	Dating & Relationships, Divorce Support, Ethnic Specific, Marriage, Parks, Rec Facilities & Gyms, Senior Living, Teens, Weddings, Lifestyle - Other
Malicious	Ad Fraud, Botnet, Command and Control Centers, Compromised & Links To Malware, Malware Call-Home, Malware Distribution Point, Phishing/Fraud, Spam URLs, Spyware & Questionable Software
Miscellaneous	Content Server, No Content Found, Parked & For Sale Domains, Private IP Address, Unreachable, Miscellaneous - Other
News, Portal and Search	Image Search, International News, Local News, Magazines, National News, Portal Sites, Search Engines, News, Portal & Search - Other
Online Ads	Pay-to-Surf, Online Ads - Other
Pets	Aquariums, Birds, Cats, Dogs, Large Animals, Reptiles, Veterinary Medicine, Pets - Other
Public, Government and Law	Advocacy Groups & Trade Associations, Commentary, Government Sponsored, Immigration, Legal Issues, Philanthropic Organizations, Politics, Social & Affiliation Organizations, U.S. Government Resources, Public, Government & Law - Other
Real Estate	Apartments, Architects, Buying/Selling Homes, Real Estate - Other
Religion	Alternative Religions, Atheism & Agnosticism, Buddhism, Catholicism, Christianity, Hinduism, Islam, Judaism, Latter-Day Saints, Non-traditional Religion and Occult, Pagan/Wiccan, Religion - Other
Science	Anatomy, Astrology and Horoscopes, Biology, Botany, Chemistry, Weather, Geography, Geology, Paranormal Phenomena, Physics, Space/Astronomy, Science - Other
Shopping	Auctions & Marketplaces, Catalogs, Contests & Surveys, Shopping - Online, Engines, Product Reviews & Price Comparisons, Coupons, Shopping - Other
Sports	Auto Racing, Baseball, Bicycling, Bodybuilding, Boxing, Canoeing/Kayaking, Cheerleading, Climbing, Cricket, Figure Skating, Fly Fishing, Football, Freshwater Fishing, Game & Fish, Golf, Horse Racing, Horses, Inline Skating, Martial Arts, Mountain Biking, NASCAR Racing, Olympics, Sports - Other, Paintball, Power & Motorcycles, Pro Basketball, Pro Ice Hockey, Rodeo, Rugby, Running/Jogging, Sailing, Saltwater Fishing, Scuba Diving, Skateboarding, Skiing, Snowboarding, Sport Hunting, Surfing/Bodyboarding, Swimming, Table Tennis/Ping-Pong, Tennis, Volleyball, Walking, Waterski/Wakeboard, World Soccer
Technology	3-D Graphics, Animation, Antivirus Software, C/C++, Cameras & Camcorders, Computer Certification, Computer Networking, Computer Peripherals, Computer Reviews, Databases, Desktop Publishing, Desktop Video, File Repositories, Graphics Software, Home Video/DVD, Information Security, Internet Phone & VOIP, Internet Technology, Java, Javascript, Linux, Mac OS, Technology - Other, Mac Support, Mobile Phones, MP3/MIDI, Net Conferencing, Net for Beginners, Network Security, Palmtops/PDAs, PC Support, Peer-to-Peer, Personal Storage, Portable, Entertainment, Remote Access, Shareware/Freeware, Unix, Utilities, Visual Basic, Web Clip Art, Web Design/HTML, Web Hosting, ISP & Telco, Windows, Online Information Management
Travel	Adventure Travel, Africa, Air Travel, Australia & New Zealand, Bed & Breakfast, Budget Travel, Business Travel, By US Locale, Camping, Canada, Caribbean, Cruises, Eastern Europe, Europe, Travel - Other, France, Greece, Honeymoons/Getaways, Hotels, Italy, Japan, Mexico & Central America, National Parks, Navigation, South America, Spas, Theme Parks, Traveling with Kids, United Kingdom

Table 4.14. URL filter categories

Customizing the URL database

Create blacklist and whitelist if necessary. From Zorp Professional version 7.0.5, this can be achieved with the help of free text editor, on the Zorp Node → New → Text editor → URL filter black and whitelists path.

Figure 4.6. Editor for URL filter blacklist and whitelist

The choices for blacklist, whitelist or * options are also put into the category list:

Figure 4.7. The 'whitelist' and 'blacklist' options are at the end of the category list

This class implements an abstract HTTP proxy - it serves as a starting point for customized proxy classes, but is itself not directly usable. Service definitions should refer to a customized class derived from AbstractHttpProxy, or one of the predefined proxy classes, such as HttpProxy or HttpProxyNonTransparent. AbstractHttpProxy denies all requests by default.

HttpProxy is a default HTTP proxy based on AbstractHttpProxy. It is transparent, and enables the most commonly used HTTP methods: "GET", "POST" and "HEAD".

HTTP proxy based on HttpProxy. This class is identical to HttpProxy with the only difference being that it is non-transparent (transparent_mode = FALSE). Consequently, clients must be explicitly configured to connect to this proxy instead of the target server and issue proxy requests. On the server side this proxy connects transparently to the target server.

For the correct operation the proxy must be able to set the server address on its own. This can be accomplished by using InbandRouter.

HTTP proxy based on HttpProxy, having URL filtering capability. The matcher attribute should be initialized to refer to a Matcher object. The initialization should be done in the class body as shown in the next example.

class MyHttp(HttpProxyURIFilter):
    matcher = RegexpFileMatcher('/etc/zorp/blacklist.txt',                                     '/etc/zorp/whitelist.txt')

HTTP proxy based on HttpProxyURIFilter, but operating in non-transparent mode (transparent_mode = FALSE).

HTTP proxy based on HttpProxy with enabled URL filtering (with DNS and reverse-DNS resolution) and preconfigured default category actions.

The following categories have policy action HTTP_URL_REJECT:

The following categories have policy action HTTP_URL_ACCEPT:

HTTP proxy based on HttpProxy, also capable of inspecting WebDAV extensions of the HTTP protocol.

The following requests are permitted: PROPFIND; PROPPATCH; MKCOL; COPY; MOVE; LOCK; UNLOCK.

HTTP proxy based on HttpProxyNonTransparent, operating in non-transparent mode (transparent_mode = FALSE) and capable of inspecting WebDAV extensions of the HTTP protocol.

The following requests are permitted: PROPFIND; PROPPATCH; MKCOL; COPY; MOVE; LOCK; UNLOCK.

This class implements a general plug proxy, and is capable of optionally disabling data transfer in either direction. Plug proxy reads connection on the client side, then creates another connection at the server side. Arriving responses are sent back to the client. However, it is not a protocol proxy, therefore PlugProxy does not implement any protocol analysis. It offers protection to clients and servers from lower level (e.g.: IP) attacks. It is mainly used to allow traffic pass the firewall for which there is no protocol proxy available.

By default plug copies all data in both directions. To change this behavior, set the copy_to_client or copy_to_server attribute to FALSE.

Plug supports the use of secondary sessions. For details, see Section 2.2, Secondary sessions.