The SSL framework starts its operation by inspecting the values set in the ssl.handshake_seq attribute. When this attribute is set to SSL_HSO_CLIENT_SERVER the client side, otherwise (SSL_HSO_SERVER_CLIENT) the server side handshake is performed first.

As part of the handshake process the proxy checks if SSL is enabled on the given side (ssl.client_connection_security and ssl.server_connection_security attributes). It is not necessary for SSL to be enabled on both sides - Zorp can handle one-sided SSL connections as well (e.g., the firewall communicates in an unencrypted channel with the client, but in a secure channel with the server). If SSL is not enabled, the handshake is skipped for that side.

When SSL is needed, the proxy will cooperate with the policy layer to have all required parameters (keys, certificates, etc.) set up. This is achieved using decision points in the hash named ssl.handshake_hash which is explained later in detail.

The SSL handshake is slightly different for the client (in this case Zorp behaves as an SSL server) and the server (when Zorp behaves as an SSL client).

As an SSL server the first thing to present to an SSL client is a certificate/key pair, thus a call to the 'setup_key' callback is made. It is expected that by the time this callback returns the attributes ssl.client_local_privatekey and ssl.client_local_certificate are filled appropriately.

If peer authentication is enabled (by setting the attribute ssl.client_verify_type) a list of trusted CA certificates must be set up (stored in the hash ssl.client_local_ca_list). The list can be set up by the 'setup_ca_list' function call. Peer certificates are verified against the trusted CA list and their associated revocation lists. Revocations can be set up in the 'setup_crl_list' callback.

At the end of the verification another callback named 'verify_cert' is called which can either ACCEPT or DENY the certificate possibly overriding the verification against the local CA database.

The server-side handshake is similar to the client-side handshake previously described. The difference is the order of certificate verification. On the server side Zorp verifies the server's certificate first and then sends its own certificate for verification. This is unlike the client side where the local certificate is sent first, and then the peer's certificate is verified.

So the callbacks are called in this order: 'setup_ca_list' and 'setup_crl_list' to set up CA and CRL information, 'verify_cert' to finalize certificate validation, and 'setup_key' to optionally provide a local certificate/key pair.

A certificate name behaves as a string, and contains a DN in the following format (also known as one-line format):

/RDN=value/RDN=value/.../RDN=value/

The word RDN stands for relative distinguished name. For example, the DN cn=Root CA, ou=CA Group, o=Foo Ltd, l=Bar, st=Foobar State, c=US becomes /C=US/ST=Foobar State/L=Bar/O=Foo Ltd/OU=CA Group/CN=Root CA/

A certifying authority may revoke the issued certificates. A revocation means that the serial number and the revocation date is added to the list of revoked certificates. Revocations are published on a regular basis. This list is called the Certificate Revocation List, also known as CRL. A CRL always has an issuer, a date when the list was published, and the expected date of its next update.

The proxy stores trusted CA certificates in a Certificate hash. This hash can be indexed by two different types. If an integer index is used, the slot specified by this value is looked up; if a string index is used, it is interpreted as a one-line DN value, and the appropriate certificate is looked up. Each slot in this hash contains an X.509 certificate.

Similarly to the certificate hash, a separate hash for storing Certificate Revocation Lists was defined. A CRL contains revocation lists associated to CAs.

Zorp is able to automatically verify the certificates received. The types of accepted certificates can be controlled separately on the client and the server side using the ssl.client_verify_type and the ssl.server_verify_type attributes. These attributes offer an easy way to restrict encrypted access only to sites having trustworthy certificates. The available options are summarized in the following table.

The ssl.server_check_subject can be used to compare the domain name provided in the Subject field of the server certificate to application level information about the server. Currently it can compare the Subject field to the domain name of the HTTP request in HTTPS communication. If the ssl.server_check_subject is set to TRUE and ssl.server_verify_type is SSL_VERIFY_REQUIRED_UNTRUSTED or SSL_VERIFY_REQUIRED_TRUSTED, the HTTP proxy using the SSL framework will deny access to the page and return an error if the Subject field does not match the domain name of the URL.

This proxy class can only be used as a basis for creating a custom proxy and cannot be used on its own. Please create a new proxy class with the AnyPyProxy as its parent and implement the proxyThread method for handling traffic.

Your code will be running as the proxy to transmit protocol elements, you'll have to take care and be security conscious not to make security vulnerabilities.

The basic protocol is as follows: the client issues a request (also called command in FTP terminology) and the server responds with the result. Both commands and responses are line based: commands are sent as complete lines starting with a keyword identifying the operation to be performed. A response spans one or more lines, each specifying the same 3-digit status code and possible explanation.

Certain commands (for example RETR, STOR or LIST) also have a data attachment which is transferred to the peer. Data attachments are transferred in a separate TCP connection. This connection is established on-demand on a random, unprivileged port when a data transfer command is issued.

Endpoint information of this data channel is exchanged via the PASV and PORT commands, or their newer equivalents (EPSV and EPRT).

The data connection can either be initiated by the client (passive mode) or the server (active mode). In passive mode (PASV or EPSV command) the server opens a listening socket and sends back the endpoint information in the PASV response. In active mode (PORT or EPRT command) the client opens a listening socket and sends its endpoint information as the argument of the PORT command. The source port of the server is usually either 20, or the port number of the Command Channel minus one.

220 FTP server ready
USER account
331 Password required.
PASS password
230 User logged in.
SYST
215 UNIX Type: L8
PASV
227 Entering passive mode (192,168,1,1,4,0)
LIST
150 Opening ASCII mode data connection for file list
226-Transferring data in separate connection complete.
226 Quotas off
QUIT
221 Goodbye

Changing the default behavior of commands can be done by using the hash attribute request, indexed by the command name (e.g.: USER or PWD). There is a similar attribute for responses called response, indexed by the command name and the response code. The possible values of these hashes are shown in the tables below. See Section 2.1, Policies for requests and responses for details. When looking up entries of the response attribute hash, the lookup precedence described in Section 2.1.2, Response codes is used.

class AnonFtp(FtpProxy):
        def config(self):
                self.request["USER"] = (FTP_REQ_POLICY, self.pUser)
                self.request["*"] = (FTP_REQ_ACCEPT)

        def pUser(self,command):
                if self.request_parameter == "anonymous" or self.request_parameter == "Anonymous":
                        return FTP_REQ_ACCEPT
                return FTP_REQ_REJECT

FTP servers send the list of supported features to the clients. For example, ProFTPD supports the following features: LANG en, MDTM, UTF8, AUTH TLS, PBSZ, PROT, REST STREAM, SIZE. The default behavior of FTP features can be changed using the hash attribute features, indexed by the name of the feature (e.g.: UTF8 or AUTH TLS). The possible actions are shown in the table below. See Section 2.1, Policies for requests and responses for details.

The built-in FTP proxies permit the use of every feature by default.

class FtpsProxy(FtpProxy):
    def config(self):
        FtpProxy.config(self)
        self.max_password_length=64

    EncryptionPolicy(name="ForwardSTARTTLS", encryption=ForwardStartTLSEncryption(client_verify=ClientCertificateVerifier(), client_ssl_options=ClientSSLOptions(), server_verify=ServerCertificateVerifier(), server_ssl_options=ServerSSLOptions(), client_certificate_generator=DynamicCertificate(private_key=PrivateKey.fromFile(key_file_path="/etc/key.d/ZMS_Engine/key.pem"), trusted_ca=Certificate.fromFile(certificate_file_path="/etc/ca.d/certs/my-trusted-ca-cert.pem", private_key=PrivateKey.fromFile("/etc/ca.d/keys/my-trusted-ca-cert.pem")), untrusted_ca=Certificate.fromFile(certificate_file_path="/etc/ca.d/certs/my-untrusted-ca-cert.pem", private_key=PrivateKey.fromFile("/etc/ca.d/keys/my-untrusted-ca-cert.pem")))))

    def demo() :
        Service(name='demo/MyFTPSService', router=TransparentRouter(), chainer=ConnectChainer(), proxy_class=FtpsProxy, max_instances=0, max_sessions=0, keepalive=Z_KEEPALIVE_NONE, encryption_policy="ForwardSTARTTLS")

    Rule(rule_id=2,
    proto=6,
    service='demo/MyFTPSService'
    )

The available stacking modes for this proxy module are listed in the following table. For additional information on stacking, see Section 2.3.1, Proxy stacking.

The Ftp proxy supports inband authentication as well to use the built-in authentication method of the FTP and FTPS protocols to authenticate the client. The authentication itself is performed by the backend configured for the service.

If the client uses different usernames on and the remote server (e.g., he uses his own username to authenticate to , but anonymous on the target FTP server), the client must specify the usernames and passwords in the following format:

Username:

<ftp user>@<proxy user>@<remote site>[:<port>]

Password:

<ftp password>@<proxy password>

Alternatively, all the above information can be specified as the username:

<ftp user>@<proxy user>@<remote site>[:<port>]:<ftp password>@<proxy password>

HTTP is a text based protocol where a client sends a request comprising of a METHOD, an URI and associated meta information represented as MIME-like headers, and possibly a data attachment. The server responds with a status code, a set of headers, and possibly a data attachment. Earlier protocol versions perform a single transaction in a single TCP connection, HTTP/1.1 introduces persistency where a single TCP connection can be reused to perform multiple transactions.

An HTTP method is a single word - usually spelled in capitals - instructing the server to apply a function to the resource specified by the URI. Commonly used HTTP methods are "GET", "POST" and "HEAD". HTTP method names are not restricted in any way, other HTTP based protocols (such as WebDAV) add new methods to the protocol while keeping the general syntax intact.

Headers are part of both the requests and the responses. Each header consists of a name followed by a colon (':') and a field value. These headers are used to specify content-specific and protocol control information.

The response to an HTTP request starts with an HTTP status line informing the client about the result of the operation and an associated message. The result is represented by three decimal digits, the possible values are defined in the HTTP RFCs.

The protocol has three variants, differentiated by their version number. Version 0.9 is a very simple protocol which allows a simple octet-stream to be transferred without any meta information (e.g.: no headers are associated with requests or responses).

Version 1.0 introduces MIME-like headers in both requests and responses; headers are used to control both the protocol (e.g.: the "Connection" header) and to give information about the content being transferred (e.g.: the "Content-Type" header). This version has also introduced the concept of name-based virtual hosts.

Building on the success of HTTP/1.0, version 1.1 of the protocol adds persistent connections (also referred to as "connection keep-alive") and improved proxy control.

Both requests and responses might have an associated data blob, also called an entity in HTTP terminology. The size of the entity is determined using one of three different methods:

GET /index.html HTTP/1.1
Host: www.example.com
Connection: keep-alive
User-Agent: My-Browser-Type 6.0

HTTP/1.1 200 OK
Connection: close
Content-Length: 14

<html>
</html>

HttpProxy is able to operate both in transparent and non-transparent mode. In transparent mode, the client does not notice (or even know) that it is communicating through a proxy. The client communicates using normal server-style requests.

In non-transparent mode, the address and the port of the proxy server must be set on the client. In this case the client sends proxy-style requests to the proxy.

GET http://www.example.com/index.html HTTP/1.1
Host: www.example.com
Connection: keep-alive
User-Agent: My-Browser-Type 6.0

HTTP/1.1 200 OK
Connection: close
Content-Length: 14

<html>
</html>

In non-transparent mode it is possible to request the use of the SSL protocol through the proxy, which means the client communicates with the proxy using the HTTP protocol, but the proxy uses HTTPS to communicate with the server. This technique is called data tunneling.

CONNECT www.example.com:443 HTTP/1.1
Host: www.example.com
User-agent: My-Browser-Type 6.0

HTTP/1.0 200 Connection established
Proxy-agent: My-Proxy/1.1

Changing the default behavior of requests is possible using the request attribute. This hash is indexed by the HTTP method names (e.g.: GET or POST). The response attribute (indexed by the request method and the response code) enables the control of HTTP responses. The possible actions are described in the following tables. See also Section 2.1, Policies for requests and responses. When looking up entries of the response attribute hash, the lookup precedence described in Section 2.1.2, Response codes is used.

class DmzHTTP(HttpProxy):
        def config(self):
                HttpProxy.config(self)
                self.request["GET"] = (HTTP_REQ_POLICY, self.filterURL)

        def filterURL(self, method, url, version):
                if (url == "http://www.disallowedsite.com"):
                        self.error_info = 'Access of this content is denied by the local policy.'
                        return HTTP_REQ_REJECT
                return HTTP_REQ_ACCECT

class DmzHTTP(HttpProxy):
        def config(self):
                HttpProxy.config(self)
                self.response["GET", "404"] = (HTTP_RSP_POLICY, self.filter404)

        def filter404(self, method, url, version, response):
                self.error_status = 404
                self.error_info = "Requested page was not accessible."
                return HTTP_RSP_REJECT

Both request and response headers can be modified by the proxy during the transfer. New header lines can be inserted, entries can be modified or deleted. To change headers in the requests and responses use the request_header hash or the response_header hash, respectively.

Similarly to the request hash, these hashes are indexed by the header name (like "User-Agent") and contain an actiontuple describing the action to take.

By default, the proxy modifies only the "Host", "Connection", "Proxy-Connection" and "Transfer-Encoding" headers. "Host" headers need to be changed when the proxy modifies the URL; "(Proxy-)Connection" is changed when the proxy turns connection keep-alive on/off; "Transfer-Enconding" is changed to enable chunked encoding.

class MyHttp(HttpProxy):
        def config(self):
                HttpProxy.config(self)
                self.request_header["User-Agent"] = (HTTP_HDR_CHANGE_VALUE, "Lynx 2.4.1")
                self.request_header["Cookie"] = (HTTP_HDR_POLICY, self.processCookies)
                self.response_header["Set-Cookie"] = (HTTP_HDR_DROP,)

        def processCookies(self, name, value):
                # You could change the current header in self.current_header_name
                # or self.current_header_value, the current request url is
                # in self.request_url
                return HTTP_HDR_DROP

URLs or sets of URLs can be easily rejected or redirected to a local mirror by modifying some attributes during request processing.

When an HTTP request is received, normative policy chains are processed (self.request, self.request_header). Policy callbacks for certain events can be configured with the HTTP_REQ_POLICY or HTTP_HDR_POLICY directives. Any of these callbacks may change the request_url attribute, instructing the proxy to fetch a page different from the one specified by the browser. Please note that this is transparent to the user and does not change the URL in the browser.

class MyHttp(HttpProxy):
        def config(self):
                HttpProxy.config(self)
                self.request["GET"] = (HTTP_REQ_POLICY, self.filterURL)

        def filterURL(self, method, url, version):
                self.request_url = "http://www.example.com/"
                return HTTP_REQ_ACCEPT

class HttpProxyHttpsredirect(HttpProxy):
        def config(self):
                HttpProxy.config(self)
                self.error_silent = TRUE
                self.request["GET"] = (HTTP_REQ_POLICY, self.reqRedirect)

        def reqRedirect(self, method, url, version):
                self.error_status = 301
                #self.error_info = 'HTTP/1.0 301 Moved Permanently'
                self.error_headers="Location: https://%s/" % self.request_url_host
                return HTTP_REQ_REJECT

Zorp differentiates between two request types: server requests and proxy request.

Zorp determines the type of the incoming request from the request URL, even if the Proxy-connection header exist. As there is no clear distinction between the two request types, the type of the request cannot always be accurately detected automatically, though all common cases are covered.

Requests are handled differently in transparent and non-transparent modes.

Parent proxies are non-transparent HTTP proxies used behind Zorp. Two things have to be set in order to use parent proxies. First, select a router which makes the proxy connect to the parent proxy, this can be either InbandRouter() or DirectedRouter(). Second, set the parent_proxy and parent_proxy_port attributes in the HttpProxy class. Setting these attributes results in proxy requests to be emitted to the target server both in transparent and non-transparent mode.

The parent proxy attributes can be set both in the configuration phase (e.g.: config() event), or later on a per-request basis. This is possible because the proxy re-connects.

class MyHttp(HttpProxy):
        def config(self):
                HttpProxy.config(self)
                self.parent_proxy = "proxy.example.com"
                self.parent_proxy_port = 3128

def instance():
        Service("http", MyHttp, router=InbandRouter())
        Listener(SockAddrInet('10.0.0.1', 80), "http")

In non-transparent mode it is possible to let Zorp process ftp:// URLs, effectively translating HTTP requests to FTP requests on the fly. This behaviour can be enabled by setting permit_ftp_over_http to TRUE and adding port 21 to target_port_range. Zorp currently supports passive mode transfers only.

There are cases when the HTTP proxy must return an error page to the client to indicate certain error conditions. These error messages are stored as files in the directory specified by the error_files_directory attribute, and can be customized by changing the contents of the files in this directory.

Each file contains plain HTML text, but some special macros are provided to dynamically add information to the error page. The following macros can be used:

It is generally recommended not to display error messages to untrusted clients, as they may leak confidential information. To turn error messages off, set the error_silent attribute to TRUE, or strip error files down to a minimum.

HTTP supports stacking proxies for both request and response entities (e.g.: data bodies). This is controlled by the request_stack and response_stack attribute hashes. See also Section 2.3.1, Proxy stacking.

There are two stacking modes available: HTTP_STK_DATA sends only the data portion to the downstream proxy, while HTTP_STK_MIME also sends all header information to make it possible to process the data body as a MIME envelope. Please note that while it is possible to change the data part in the stacked proxy, it is not possible to change the MIME headers - they can be modified only by the HTTP proxy. The possible parameters are listed in the following tables.

Please note that stacking is skipped altogether if there is no body in the message.

Certain webserver applications may return data entities in 205 responses. This is explicitly prohibited by the RFCs, but Zorp permits such responses for interoperability reasons.

Zorp’s HTTP proxy offers the ‘session persistence in load balancing’ feature, further enhancing Zorp’s load balancing capabilities by that.

With the help of this feature, the Round Robin chainer can identify connections by their session IDs and make sure that every connection with the same session ID is always addressed to the same server, so that the session persists.

For using the ‘session persistence in load balancing’ feature, the administrator has to configure the following three attributes for the HTTP proxy:

Starting with version 3.3FR1, Zorp supports category-based URL filtering using a regularly updated database.

class MyHTTPUrlFilter(HttpProxy):
    def config(self):
        HttpProxy.config(self)
        self.enable_url_filter=TRUE
        self.enable_url_filter_dns=TRUE
        self.url_category['adult']=(HTTP_URL_REJECT, (403, "Adult website",))
        self.url_category['porn']=(HTTP_URL_REJECT, (403, "Porn website",))
        self.url_category['malware']=(HTTP_URL_REJECT, (403, "Site contains malware",))
        self.url_category['phishing']=(HTTP_URL_REJECT, (403, "Phishing site",))
        self.url_category['warez']=(HTTP_URL_REJECT, (403, "Warez site",))
        self.url_category['*']=(HTTP_URL_ACCEPT,)

class MyHTTPUrlFilter(HttpProxy):
    def config(self):
        HttpProxy.config(self)
        self.enable_url_filter=TRUE
        self.enable_url_filter_dns=TRUE
        self.url_category['onlinegames']=(HTTP_URL_REDIRECT, "http://example.com")
        self.url_category['*']=(HTTP_URL_ACCEPT,)

The basic protocol is the following: the client issues a request (also called command in POP3 terminology) and the server responds with the result. Both commands and responses are line based, each command is sent as a complete line, a response is either a single line or - in case of mail transfer commands - multiple lines.

Commands begin with a case-insensitive keyword possibly followed by one or more arguments (such as RETR or DELE).

Responses begin with a status indicator ("+OK" or "-ERR") and a possible explanation of the status code (e.g.: "-ERR Permission denied.").

Responses to certain commands (usually mail transfer commands) also contain a data attachment, such as the mail body. See the Section 4.9.1.3, Bulk transfers for further details.

The protocol begins with the server displaying a greeting message, usually containing information about the server.

After the greeting message the client takes control and the protocol enters the AUTHORIZATION state where the user has to pass credentials proving his/her identity.

After successful authentication the protocol enters TRANSACTION state where mail access commands can be issued.

When the client has finished processing, it issues a QUIT command and the connection is closed.

Responses to certain commands (such as LIST or RETR) contain a long data stream. This is transferred as a series of lines, terminated by a "CRLF '.' CRLF" sequence, just like in SMTP.

+OK POP3 server ready
USER account
+OK User name is ok
PASS password
+OK Authentication successful
LIST
+OK Listing follows
1 5758
2 232323
3 3434
.
RETR 1
+OK Mail body follows
From: sender@sender.com
To: account@receiver.com
Subject: sample mail

This is a sample mail message. Lines beginning with
..are escaped, another '.' character is perpended which
is removed when the mail is stored by the client.
.
DELE 1
+OK Mail deleted
QUIT
+OK Good bye

By default, the proxy accepts all commands recommended in RFC 1939. Additionally, the following optional commands are also accepted: USER, PASS, AUTH. The proxy understands all the commands specified in RFC 1939 and the AUTH command. These additional commands can be enabled manually.

Changing the default behavior of commands can be done using the hash named request. The hash is indexed by the command name (e.g.: USER or AUTH). See Section 2.1, Policies for requests and responses for details.

class APop3(Pop3Proxy):
      def config(self):
              Pop3Proxy.config(self)
              self.request["USER"] = (POP3_REQ_REJECT)
              self.request["APOP"] = (POP3_REQ_ACCEPT)

class UToAPop3(Pop3Proxy):
      def config(self):
              Pop3Proxy.config(self)
              self.request["USER"] = (POP3_REQ_POLICY,self.DropUSER)
              self.request["PASS"] = (POP3_REQ_POLICY,self.UToA)

      def DropUSER(self,command):
              self.response_value = "+OK"
              self.response_param = "User ok Send Password"
              return POP3_REQ_REJECT

      def UToA(self,command):
              # Username is stored in self->username,
              # password in self->request_param,
              # and the server timestamp in self->timestamp,
              # consequently the digest can be calculated.
              # NOTE: This is only an example, calcdigest must be
              # implemented separately
              digest = calcdigest(self->timestamp+self->request_param)
              self->request_command = "APOP"
              self->request_param = name + " " + digest
              return POP3_REQ_ACCEPT

As in many other protocols, POP3 also starts with a server banner. This banner contains the protocol version the server uses, the possible protocol extensions that it supports and, in many situations, the vendor and exact version number of the POP3 server.

This information is useful only if the clients connecting to the POP3 server can be trusted, as it might make bug hunting somewhat easier. On the other hand, this information is also useful for attackers when targeting this service.

To prevent this, the banner can be replaced with a neutral one. Use the request hash with the 'GREETING' keyword as shown in the following example.

class NeutralPop3(Pop3Proxy):
      def config(self):
      Pop3Proxy.config(self)
      self.request["GREETING"] = (POP3_REQ_POLICY, None, self.rewriteBanner)

      def rewriteBanner(self, response)
              self.response_param = "Pop3 server ready"
              return POP3_RSP_ACCEPT

The available stacking modes for this proxy module are listed in the following table. For additional information on stacking, see Section 2.3.1, Proxy stacking.

When filtering messages for viruses or spam, the content vectoring modules reject infected and spam e-mails. In such cases the POP3 proxy notifies the client about the rejected message in a special e-mail.

To reject e-mail messages using the ERR protocol element, set the reject_by_mail attribute to FALSE. However, this is not recommended, because several client applications handle ERR responses incorrectly.