Proxedo API Security: Administration Guide

Transport Layer Security v1 (TLS) (successor of the now obsoleted Secure Socket Layer v3 (SSL)) is a widely used crypto protocol, guaranteeing data integrity and confidentiality in many PKI and e-commerce systems.

The TLS framework inspects TLS connections, and also any other connections embedded into the encrypted TLS channel. TLS connections initiated from the client are terminated on the Proxedo API Security, and two separate TLS connections are built: one between the client and the firewall, and one between the firewall and the server. If both connections match the configuration settings of PAS (for example, the certificates are valid, and only the allowed encryption algorithms are used), PAS inspects the protocol embedded into the secure channel as well. Note that the configuration settings can be different for the two connections, for example, it is possible to permit different protocol versions and encryption settings.

Proxedo API Security acts as an HTTP proxy and verifies that the traffic passing through conforms to HTTP’s specifications. By using OpenAPI schemas, as defined in OpenAPI specifications (also known as Swagger), it also verifies that the traffic passing through conforms to the API enpoint’s specification and can log or deny non-conforming traffic.

PAS also provides its own versatile filtering system to control passing traffic.

With Proxedo API Security it is possible to extract business-relevant information with extremely high resolution from the traffic and relay it to external data stores where further analysis can be implemented.

Thus, it is possible to feed Log Management solutions, Monitoring and SIEM systems, Data visualization tools with data extracted from the traffic, even to the level of specific fields deep inside API calls or URI parameters.

The security flow binds most of PAS’s features together. It allows flexible configuration for handling the traffic. Multiple Enforcement, Filter and Insigth plugins can be mix-and-matched with control over error policies.

The figure on Proxedo API Security architecture and the steps following that describe how client connection is handled. The following figure explains how calls are processed in more details:

Usually, Plugins are organized in the following manner:

Though the order of the plugins can be changed based on the needs, note the followings:

It is also worth noting that Insight Plugins instantly hand over data to the Insight Director, and let the execution continue.

Proxedo API Security has a version number in the form of major.minor.patch. The docker image labels control how the automatic update of the services are handled. Each image has 3 possible tags:

A single instance of Flow Director uses a single processor core. It is necessary to adjust the number of instances to use all the available cores. This is controlled by the PAS_FLOW_DIRECTOR_SCALE variable. As the Flow Director handles the most demanding duties among the components, it must be assigned most of the cores. If there are up to four cores available, assign three cores to the Flow Director, and the remaining one core will be suitable for the Transport and Insight Director. If there are more than four cores, assign two cores for the Transport and Insight Director and assign the rest to the Flow Director.

The configuration of listener can be completed by defining certain keys for the listeners.

Each of these list items has the following keys:

Thus, a simple configuration might look as follows:

When HTTPS is used in communicating with the clients, client_tls settings must be configured.

Configuration options for Client Side TLS:

Configuration for the X.509 certificate used for TLS connections on the listener:

TLS protocol options used on the listener:

Options for verifying client side X.509 certificates:

The Security Flow definition in an endpoint lists what happens to the traffic on a given endpoint.

To understand how requests flow through PAS, see Understanding processing flow. The Security Flow starts when the Transport Director has already set up client connection and routed the request to the Flow Director. At this point the TLS and HTTP layers are already processed, but the content in the body of the request is available only in raw format and has not been parsed yet.

At this stage, the configuration security flow decides on what happens to the traffic by applying a list of Plugins one by one. Plugin is a collective name for Enforcers, Insights, Filters, etc. Once, all the plugins have processed the request, the control is handed back to the Transport Director which routes the request to a backend server, and comes back with the response after handling TLS and HTTP. At this point, the Flow Director applies another list of Plugins to response, and once done, it hands back the response to the Transport Director which in turn returns that to the client.

If at any point an error occurs, the error policy is applied — which might either mean to lead to logging the error or to terminating processing and returning an error indication to the client.

Plugins can override the endpoint’s error policy.

Also note that different Plugins need different data. An Insight that applies a JMESPath query needs parsed JSON, while one that extracts value from an HTTP header field does not. Other Plugins provide these required values, like a JSON de-serializer Plugin. It is important that the Plugins are configured in such an order that the required data is made available beforehand.

Therefore a typical security flow is configured with the plugins in the following order:

Regardless of what they do, all Plugins share some common parameters:

Both of these are optional:

Plugins' configuration details are always defined under top level configuration keys that denote their type. They are always named so that their names refer to a Plugin that represents a certain configuration. The names themselves are referenced from the Security Flow.

The following example presents the configuration hierarchy of a Plugin:

A minimal example configuration is as follows:

Instead of referencing an existing entry, matchers and error policies can be defined inline.

The forthcoming two configurations are functionally equivalent therefore:

The Decompressor Plugin decompresses the body of the HTTP message.

The Compressor Plugin compresses the the body of the HTTP message.

These Plugins require no configuration. They understand the Transfer-Encoding HTTP header and can work with content optionally compressed by the gzip, deflate and brotli algorithms.

Since they require no configuration, an instance of both named default is available for inclusion in the security flow. They can be referenced as decompressor.default and compressor.default.

The Deserializer Plugin parses the body of the HTTP message’s body to structured data. This ensures that the message is well-formed.

The structured data is also be consumed by other Plugins that operate on the body of the message, such as the jmespath extractor.

The Serializer Plugin recreates the HTTP message’s body from the structured data.

The Deserializer Plugin works with decompressed body.

Serialization needs to be done before compression.

A typical Security Flow configuration therefore starts with a Decompressor followed by a Deserializer and finishes with a Serializer followed by a Compressor. This ensures that transferred HTTP bodies are syntactically correct and that they are reconstructed to avoid transferring potentially crafted content.

They understand the Content-Type HTTP header and can work with JSON and XML content.

The plugin accepts the following configuration options:

The Plugin does not override any of the default error policy options.

Problems are considered errors that lead to the termination of the call. Problems in the request are reported back to the client, while errors in the response are suppressed to avoid information leak.

See Error Policies to understand how defaults are applied.

There are two type of predefined (de)serializer plugin:

An example configuration is the plugin looks like this:

Filter Plugins are lightweight alternatives of Enforcer Plugins for filtering unwanted traffic. They only consist of a matcher and an error policy. If the matcher matches, the error policy is applied. This way you can use matchers inline instead of creating a whole schema-based Enforcer Plugin for the simple use cases.

The Plugin accepts the following configuration options:

The Plugin does not override any of the default error policy options.

Problems are considered errors that lead to the termination of the call. Problems in the request are reported back to the client, while errors in the response are suppressed to avoid information leak.

See Error Policies to understand how defaults are applied.

An example configuration for the Plugin is as follows:

An Enforcer Plugin validates calls against externally defined schemas.

The Plugin supports validation against OpenAPI2.0 (Swagger) schemas, XSD schemas or WSDL schema.

Understanding the format of these schemas is not in the scope of this document. Further information is available at:

The Insight plugin extracts various data from the call and sends it to external systems (log servers, SIEMs, and other data analysis tools).

The Insight plugin accepts the following configuration options:

The Plugin overrides the following fields of the default error policy:

Problems are considered errors that only need to be logged. If that is overridden then problems in the request are reported back to the client, while errors in the response are suppressed to avoid information leak.

See Error Policies to understand how defaults are applied.

The Plugin collects the information from all the selectors and sends them to all the targets.

The collected information from all the selectors is arranged into a dictionary: a list of key — value pairs. The key can be configured in each selector. Certain selectors might return complex data structures, that are made up of other dictionaries and/or lists. To ensure compatibility with a wide range of target types, such results are flattened. The path inside the complex data structure is encoded into the key for each value. More details are available on this in Data flattening.

Error Policies define how to proceed if a Plugin decides to have found an error. For example, when an Enforcer plugin decides that the call is invalid.

It is the error policy that enables to act differently in case the error appears in a request or a response.

An Error Policy contains the following settings:

The default values in the above table represent the hard coded default values. They form a strict security policy: all errors are fatal, and only mistakes made by the client are reported in detail.

The error_policy configuration tree contains named Error Policies with their respective configuration, as follows:

Such error policies can be used in Plugin configurations' error_policy option by referencing their name. It is also possible to write the Error Policy configuration directly inline, if you do not plan to reuse the policy.

Matchers decide if the Plugin should be executed by examining the content of the call. They provide an extremely versatile way of defining the circumstances that must be met for the Plugin to execute.

Matchers need three pieces of information:

To ease configuration, a matcher in its simplest form is defined as a key: a value pair where the key contains the Name and the Comparator and the value is the pattern, as in the following example:

An example is as follows:

The matcher configuration tree contains named Matchers with their respective configuration, as in the following example:

Such matcher can be used in Plugin configurations' match option by referencing their name. It is also possible to write the Matcher configuration directly inline, if you do not plan to reuse the matcher.

Matchers internally utilize Extractors to fetch the information from the call to compare with. The Name of the matcher resembles the name of the extractor that will be used.

All matchers have a default comparator that is applied implicitly, so the simplest from is as follows: url_path: /example/path.

The List of matchers contains the supported matchers, describing:

What each comparator does is described in List of comparators.

Selectors are responsible for collecting information from the call. They utilize Extractor bricks for this purpose.

They are used by Insight Plugins.

The selector accepts the following configuration options:

The selector configuration tree contains named Selectors with their respective configuration, as follows:

Most extractors return simple string values. However, some (might) return dictionaries. For example, you can get all the HTTP headers, or all the URI query parameters. For such extractors you can save the result directly on the top of the Insight’s dictionary.

Extractors are used to extract data from the call.

They are utilized by Matchers and Selectors. Extractors are configured inline in matchers and Selectors, there are no named extractors.

Extractors are included by their type in Selectors, and are used by a special syntax in matchers.

See the List of Extractors for the available extractor types and their configuration options.

Target bricks define where the data collected by the Insight Plugins will be sent to.

The target configuration tree contains named Targets with their respective configuration, as follows:

See the List of Targets for the available target types and their configuration options.

This matcher always matches.

This matcher never matches. It can be used to turn off a Plugin.

It matches the HTTP method of the request. Note that the method is case insensitive by definition, therefore the case will always be ignored.

It matches the status code of the response.

Matches the direction of the message (request or response).

It matches the value of an HTTP header.

Some HTTP headers can be present more than once in a call. To accommodate this, matching is completed against the value of each occurrence of the header. Matching occurs if there is any match.

For example, if the Accept header was repeated as follows:

Consequently, in this example above both header.accept: application/json and header.accept: application/xml would match.

The syntax of this matcher differs from the others because the name of the header_name must be added:

Therefore to match against the header named Server the key will be header.server, possibly completed with comparator specification, like header.server.regex.ignorecase.

It matches the original content of the message. If the content type is JSON, the body will be decompressed but not parsed.

It matches the content type of the message. It is a more robust solution than using the header on the Content-Type header because that can contain parameters as well.

It matches the data from the body of a JSON call with the help of the JMESPath expression.

JMESPath is a query language for JSON. It is a very versatile tool for extracting the needed information from the body of the call, and organize it according to needs.

A complete explanation on how to write JMESPath expressions is not in the scope of this document. To learn more about it visit the: main website:

Provide the JMESPath expression in the configuration.

The supported comparator parameters are described in the List of comparators.

A range of matchers is available to match different parts of the URI.

The structure of an URI looks as follows:

That is, for example:

These matchers use the URI extractors. It has an extensive list of examples of what each extractor extracts from the URI.

Any is a Compound matcher that matches if any of its sub-matchers matches.

All is a Compound matcher that matches if all of its sub-matchers match.

One is a Compound matcher that matches if exactly one of its sub-matchers match.

None is a Compound matcher that matches if none of its sub-matchers match.

It matches the data from the body of an XML call with the help of the Xpath expression.

Xpath is a query language for XML. It is a very versatile tool for extracting the needed information from the body of the call, and organize it according to needs.

A complete explanation on how to write Xpath expressions is not in the scope of this document. To learn more about it visit the main website.

Provide the Xpath expression in the configuration.

A range of matchers is available to match different parts of the SOAP message.

These matchers extend the xpath matcher with predefined expressions.

They use the SOAP extractors. It has an extensive list of examples of what each extractor extracts from the SOAP message.

You can extend the base query with unique expressions:

It matches if the parameter is exactly the same as the value matched.

Parameters:

It matches if the parameter is not exactly the same as the value matched.

Parameters:

It matches if the value starts exactly with the pattern.

Parameters:

It matches if the value ends exactly with the pattern.

Parameters:

It matches if the exact pattern is found somewhere in value.

Parameters:

Pattern treats input as Unix shell-style wildcards. The special characters used in shell-style wildcards are:

Parameters:

Regex treats input as a regular expression for matching. Consult Python’s regular expression documentation and their Regular Expression HOWTO.

It matches if pattern is larger or equal to value.

It matches if pattern is smaller or equal to value.

Matches if value is between the limits in the pattern, including boundaries. The format of the pattern must be min..max.

Status_class is a special matcher for conveniently matching HTTP status code classes. It takes the name of the class and checks if the status code is in the given range as follows:

It extracts the HTTP method of the request.

It does not require configuration.

It extracts the status code of the response.

It does not require configuration.

It extracts the value of an HTTP header.

It is valid for some HTTP headers to be present more than once in a call. In this case all the values are extracted as a list.

It provides the name of the header in the configuration.

It works like header but it returns a list even if there is only a single extracted value.

It works like header but it only returns the first extracted value even if there is a list of extracted values.

It extracts all the headers from the call.

The results are stored as a dictionary, so when you can choose to omit the save_as key if you use this from a selector.

It is valid for some HTTP headers to be present more than once in a call. In such cases all the values are stored under the header’s key as a list.

It does not require configuration.

A range of extractors is available to extract different parts of the URI.

The structure of an URI looks as follows:

That is, for example:

It does not require configuration.

It extracts data from the body of a JSON call with the help of a JMESPath expression.

JMESPath is a query language for JSON. It is a very versatile tool for extracting the needed information from the body of the call, and organize it according to requirements.

A complete explanation of how to write JMESPath expressions is not in the scope of this document. To learn more about it visit this website: main website:

Provide the JMESPath expression in the configuration.

Extracts the call direction (request, response).

It does not require configuration.

Extracts the current time.

Configuration:

It extracts the content.

It does not require configuration.

It extracts the content as a string.

It does not require configuration.

It extracts the content_type from the HTTP header.

It does not require configuration.

It extracts the charset from the content_type HTTP header.

It does not require configuration.

It extracts a string, integer, number, object, array, boolean as string from the configuration.

It extracts the session ID as string, generated by the Transport Director.

It does not require configuration.

It extracts data from the body of an XML call with the help of a Xpath expression.

Xpath is a query language for XML. It is a very versatile tool for extracting the needed information from the body of the call, and organize it according to needs.

A complete explanation on how to write Xpath expressions is not in the scope of this document. To learn more about it visit the main website.

Provide the Xpath expression in the configuration. Depending on the expression, the return value is a single node or a list of nodes. If you want a single value or a list independent from the expression, use xpath_first or xpath_force_list.

A range of extractors is available to extract different parts of the SOAP message.

These extractors extend the xpath with predefined expressions.

You can extend the base query with unique expressions:

It logs the result of the insight in the local system log.

The target accepts the following configuration options:

It sends the insight to a remote syslog server using the IETF syslog protocol defined in RFC5424.

Sends the insight to an Elasticsearch engine in JSON.

When HTTPS is used the backend_tls settings must be configured.

Backend TLS takes the following configuration options:

Configuration for the X.509 certificate used for TLS connections towards the backend servers:

TLS protocol options used on TLS connections towards the backend servers:

Options for verifying backendX.509 certificates:

pas-checkconfig can check the validity of PAS configuration. Namely:

Prior to upgrading services, make sure that the image tags point to the right version. See section docker-compose.conf for details.

There are two important concepts related to logs: categories and Session IDs.

It is usually in the form of svc/default/<listener>:<transport-director-session>/default/http#<http-request-count>/flow:<flow-director-id>/ch:<flow-director-channel>/<plugin_type>/<plugin_name>, for example: svc/default/httpbin:14/default/http#0/flow:1/ch:28/enforcer/manualtest.

Information that is not available at the time, will be missing from the Session ID. Generally, the part until /flow: belongs to the Transport Director. Consequently, the Transport Director will never see that part. The Flow Director however will fetch and include that information. Nevertheless, in early phases it might not be available, and the Session ID will start with flow:.

Despite some parts not being always available, the ID is constructed in such a manner that greping on any part will find other messages with extra information as well.