Copyright

Copyright © 2019 BalaSys IT Ltd.. All rights reserved. This document is protected by copyright and is distributed under licenses restricting its use, copying, distribution, and decompilation. No part of this document may be reproduced in any form by any means without prior written authorization of BalaSys.

This documentation and the product it describes are considered protected by copyright according to the applicable laws.

This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (http://www.openssl.org/). This product includes cryptographic software written by Eric Young (eay@cryptsoft.com)

Linux™ is a registered trademark of Linus Torvalds.

Windows™ 10 is registered trademarks of Microsoft Corporation.

The BalaSys™ name and the BalaSys™ logo are registered trademarks of BalaSys IT Ltd.

The Zorp™ name and the Zorp™ logo are registered trademarks of BalaSys IT Ltd.

The Proxedo™ name and the Proxedo™ logo are registered trademarks of BalaSys IT Ltd.

AMD Ryzen™ and AMD EPYC™ are registered trademarks of Advanced Micro Devices, Inc.

Intel® Core™ and Intel® Xeon™ are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries.

All other product names mentioned herein are the trademarks of their respective owners.

DISCLAIMER

BalaSys is not responsible for any third-party websites mentioned in this document. BalaSys does not endorse and is not responsible or liable for any content, advertising, products, or other material on or available from such sites or resources. BalaSys will not be responsible or liable for any damage or loss caused or alleged to be caused by or in connection with use of or reliance on any such content, goods, or services that are available on or through any such sites or resources.

2020-09-18 .Copyright

Preface

Typographical conventions

Before you start using this guide, it is important to understand the terms and typographical conventions used in the documentation. For more information on specialized terms and abbreviations used in the documentation, see the Glossary at the end of this document.

The following text formatting principles and icons identify special information in the document.

Tips provide best practices and recommendations.
Notes provide additional information on a topic, and emphasize important facts and considerations.
Warnings mark situations where loss of data or misconfiguration of the device is possible if the instructions are not obeyed.
Command

Commands you have to execute.

Emphasis

Reference items, additional readings.

/path/to/file

File names.

Parameters

Parameter and attribute names.

In the parameter listing tables the required parameters are also emphasized with bold text:

Key Description

param1

This is a required parameter.

param2

This is an optional parameter.

Contact and support information

This product is developed and maintained by BalaSys IT Ltd..

Contact:

BalaSys IT Ltd.
4 Alíz Street
H-1117 Budapest, Hungary
Tel: +36 1 646 4740
E-mail: <info@balasys.hu>
Web: http://balasys.hu/

Sales contact

You can directly contact us with sales-related topics at the e-mail address <sales@balasys.hu>, or leave us your contact information and we call you back.

Support contact

To access the BalaSys Support System, sign up for an account at the BalaSys Support System page. Online support is available 24 hours a day.

BalaSys Support System is available only for registered users with a valid support package.

Support e-mail address: <support@balasys.hu>.

Training

BalaSys IT Ltd. holds courses on using its products for new and experienced users. For dates, details, and application forms, visit the https://www.balasys.hu/hu/support/trainings webpage.

1. Introduction to Proxedo API Security

1.1. What is Proxedo API Security

The Proxedo API Security (PAS) is a security solution that protects API serving endpoints. It is positioned in the network flow between consumers of the APIs (clients) and backend solutions serving the API (servers) as a transparent HTTP proxy.

Proxedo API Security can:

  • handle incoming Transport Layer Security v1 (TLS) connections from clients & outgoing TLS connections to servers separately and selectively

  • verify that communication conforms to HTTP specifications

  • verify that the content of the messages conform to their specified content type

  • verify that the content of messages conform to API specification(s) as described in schemas

  • extract parts of the content of the messages and relay them to external data stores such as log servers, SIEM systems or other data warehouses

1.2. Where to start

Depending on what you need to do the following starting points are suggested:

2. Overview of Proxedo API Security

2.1. Main features

2.1.1. TLS

Transport Layer Security v1 (TLS) (successor of the now obsoleted Secure Socket Layer v3 (SSL)) is a widely used crypto protocol, guaranteeing data integrity and confidentiality in many PKI and e-commerce systems.

The TLS framework inspects TLS connections, and also any other connections embedded into the encrypted TLS channel. TLS connections initiated from the client are terminated on the Proxedo API Security, and two separate TLS connections are built: one between the client and the firewall, and one between the firewall and the server. If both connections match the configuration settings of PAS (for example, the certificates are valid, and only the allowed encryption algorithms are used), PAS inspects the protocol embedded into the secure channel as well. Note that the configuration settings can be different for the two connections, for example, it is possible to permit different protocol versions and encryption settings.

2.1.2. Enforcement

Proxedo API Security acts as an HTTP proxy and verifies that the traffic passing through conforms to HTTP’s specifications. By using OpenAPI schemas, as defined in OpenAPI specifications (also known as Swagger), it also verifies that the traffic passing through conforms to the API enpoint’s specification and can log or deny non-conforming traffic.

PAS also provides its own versatile filtering system to control passing traffic.

2.1.3. Insights

With Proxedo API Security it is possible to extract business-relevant information with extremely high resolution from the traffic and relay it to external data stores where further analysis can be implemented.

Thus, it is possible to feed Log Management solutions, Monitoring and SIEM systems, Data visualization tools with data extracted from the traffic, even to the level of specific fields deep inside API calls or URI parameters.

2.1.4. Security flow

The security flow binds most of PAS’s features together. It allows flexible configuration for handling the traffic. Multiple Enforcement, Filter and Insigth plugins can be mix-and-matched with control over error policies.

2.2. Main Concepts in Proxedo API Security

This chapter provides an overview of the Proxedo API Security solution, introduces its main concepts, and explains the relationship of the various components.

API Endpoint

Proxedo API Security protects API endpoints. An API endpoint is the serving part of the communication channel and is the collection of all functions of a service. It resides at a list of well-known top URIs under which all the functions are accessible. APIs have well-defined HTTP Endpoints for all exposed calls, resources etc., usually through providing a schema that describes all parameters of these URI paths, including possible HTTP response codes, the format and fields of the data structure in the request’s and response’s body.

Client

It is a consumer of API endpoints. It is the source of the requests.

Backend

One or more servers that serve the API endpoint. It receives the requests of the client and sends the responses.

HTTP message

An HTTP request coming from the client or an HTTP response coming from the backend.

Call

An HTTP conversation constitutes of a request — response interchange of HTTP messages between the client and the backend. Whenever the direction is irrelevant in the context — it applies to both requests and responses — the message is named Call.

Listener

The part of PAS that listens to incoming traffic for given API Endpoints. It is bound to a network port. Clients address this port when accessing API Endpoints through the gateway.

TLS

Transport Layer Security is the cryptographic protocol that secures HTTPS communications. PAS can apply TLS encryption both when communicating with Clients and Backends.

Security flow

A collection of security rules that PAS applies to a Call. It is two series of Plugins: one for requests and one for responses.

Plugin

An element of the security flow that applies a specific security function. It has different types based on the role they do.

Deserializer

It is a Plugin responsible for parsing the HTTP message’s body to structured data. This ensures that a message is well-formed. The structured data will also be consumed by other Plugins that operate on the body of the message.

Serializer

It is a Plugin responsible for serializing the structured data to the format of the HTTP message’s body.

Filter

It is a Plugin that rejects calls when they match defined rules.

Enforcer

It is a Plugin that validates calls against externally defined schemas.

Insight

It is a Plugin that extracts various data from the call and sends it to external systems (log servers, SIEMs, and other data analysis tools).

Brick

They are reuseable components of Plugins. They can be defined on their own and then shared by multiple Plugins.

Error policy

It is a brick that defines what happens if the Plugin has found an error. It decides if calls are rejected or merely logged, and defines the details of the HTTP error response sent to the client if a call is rejected.

Matcher

It is a brick that decides if the Plugin should be executed for a given call by checking various data in the HTTP message.

Selector

Selector is a brick that can extract a piece of information from a call. It is used by Insight plugins.

Target

It is a brick that defines an external system to send extracted data to. It is used by Insight plugins.

2.3. Architecture for Proxedo API Security

Proxedo API Security is based on a micro-services architecture.

The components of the architecture are each responsible for well-defined subset of handling traffic between the client and the backend. Proxedo API Security is built up of three components:

Transport Director

It manages the transport layer of API connections:

  • handles network connections from the client

  • handles network connections towards the backends

  • handles TLS on these connections

  • load-balances between multiple backend servers

  • load-balances between multiple Flow Directors

  • enforces HTTP protocol validity in calls

Flow Director

It is responsible for the execution of the Plugins in the Endpoint’s flow and for applying Error Policies as necessary.

Insight Director

It manages the connections to Targets. It is responsible for sending the data collected by Insight plugins to Target systems.

The handling of a connection with the help of components is shown in this figure:

PAS Architecture
Figure 1. PAS Architecture
  1. Incoming connections are accepted by the Transport Director.

    • It handles TLS with the client if necessary.

  2. It hands over the connection to the Flow Director.

    • The Flow Director chooses the Endpoint based on the URL.

    • The Flow Director applies the Endpoint specific Request Security Flow.

  3. If an Insight plugin needs to send data to an external Target it sends the collected data to the Insight Director.

  4. The Insight Director sends the data further to the Target with the appropriate protocol.

  5. The Flow Director hands the connection back to the Transport Director.

  6. The Transport Director then sends the data to the Backend.

    • It handles TLS with the backends if necessary.

    • It performs load balancing among Backend servers if necessary.

The same procedure is executed with the response coming from the Backend.

2.3.1. Understanding processing flow

The figure on Proxedo API Security architecture and the steps following that describe how client connection is handled. The following figure explains how calls are processed in more details:

PAS processing flow
Figure 2. PAS processing flow
  1. As shown in the figure above, the incoming connection from the client is handled by the Transport Director, applying TLS if needed.

  2. The Transport Director hands over the connection to the Flow Director, indicating which Listener the connection belongs to. The Flow Director chooses the Endpoint based on the URL in the request. First endpoint has matching URL is chosen.

  3. The Flow Director then starts applying the request part of the Security Flow definition. For each Plugin the Flow Director:

    • Checks if the Plugin's matcher matches the request.

    • If so, it executes the Plugin, if not, it executes the next Plugin.

    • If the Plugin indicates success it executes the next Plugin.

    • If the Plugin indicates an error it applies the Plugin's error policy. If the policy dictates to abort the connection:

      • It fills error details and hands back the connection to the Transport Director, aborting the execution of the flow.

      • The Transport Director closes the connection, sending error details to the client if allowed by the policy.

  4. Once, the last Plugin has been executed the connection is handed back to the Transport Director.

  5. The Transport Director initiates the connection towards the Backend:

    • It handles load balancing if necessary.

    • It handles TLS if necessary.

    • It sends the request itself to the Backend server.

  6. The Backend server sends its response to the Transport Director.

  7. Once, the response has been received the Transport Director again hands over the connection to the Flow Director.

  8. The Flow Director then starts applying the response part of the Security Flow definition, executing the Plugins as above.

  9. Once, the last Plugin has been executed the connection is handed back to the Transport Director.

  10. Finally, the Transport Director sends the response to the client.

Usually, Plugins are organized in the following manner:

  • Decompressor Plugin processes the compressed body

  • Deserializer Plugin processes the decompressed request to understand the details in the body

  • Filters are applied to filter unnecessary traffic.

  • Enforcers are applied for detailed validation of calls.

  • Insights are applied to collect data from the call.

  • Serializer Plugin serializes the body

  • Compressor Plugin compresses the serialized body

Though the order of the plugins can be changed based on the needs, note the followings:

  • When a Plugin needs access to the request body it requires Deserialized data. It is therefore strongly recommended that the first plugin is a Decompressor followed by a Deserializer.

  • At the end of the flow is strongly, it is recommended to place a Serializer plugin followed by a Compressor.

  • Generally Insights are applied after Filters and Enforcers so that they are not executed on possibly invalid calls.

  • Anything that operates on the HTTP headers or the body of the message will be aware of the call direction: The same Plugin in the request and response flow will act on the request or response data.

  • However, the Flow Director handles a request-response exchange together, so you can still use details from the request in Plugins of the response flow. The most notable example of this is using URI or method matchers in the response flow.

  • Plugins in the request flow, however, cannot access details of the response flow (since they are not available yet.)

It is also worth noting that Insight Plugins instantly hand over data to the Insight Director, and let the execution continue.

3. Installation of Proxedo API Security

The installation of Proxedo API Security is relatively straightforward. PAS is mainly distributed as Docker images, and is also completed with a .deb package that sets up the operational environment.

3.1. Prerequisites

The followings are needed prior to the installation of Proxedo API Security:

  • The licence for PAS.

  • A technical user for accessing Balasys' docker registry.

  • The PAS .deb package.

  • A server with Ubuntu 18.04 Operation System installed.

3.2. Installation steps

  1. Log in as root:

  2. Update the OS' package list: apt update.

  3. Install the PAS .deb package: apt install <path/to/deb>/proxedo-api-security_<version>.deb.

This will:

  • create a user named pas for running and configuring PAS.

    pas user must not be created manually beforehand.
  • Install the necessary configuration files and helper scripts under /opt/balasys.

  • Create systemd services for managing PAS.

    You need to use apt to locally install the .deb package as it installs its dependencies as well. dpkg will not resolve dependencies, and apt-get cannot install from a local file. Also note that to install PAS from the current directory, you must use the path ./ before the .deb package, or apt will try to download the package from a repository.
  1. Set up the number of Flow Director instances to run in /opt/balasys/etc/docker-compose.conf. If necessary, also change the version you want to follow. For details, see docker-compose.conf.

  2. Copy license.txt to /opt/balasys/etc.

  3. Copy license.txt to /opt/balasys/etc.

  4. Change to the PAS user: su - pas.

  5. Run pas-registry-login to set up authentication with the docker registry. Provide login credentials on the prompt. Contact support if you need assistance with your credentials.

    Docker will, by default, save your credentials unencrypted in the home directory of the pas user. Using a password-management tool like pass is not enforced, but it is recommended.
  6. Run pas-update to download the docker images.

  7. Set up initial configuration in /opt/balasys/etc/config.yml.

  8. Run pas-checkconfig to validate the configuration.

  9. Start PAS: systemctl start proxedo-api-security.

    This service is enabled by default, so the service starts on system restart.
  10. If you configured Certificate Revocation List (CRL) verification in any of your Backends or Listeners you need to enable CRL updates:

    sytemctl enable proxedo-api-security-crl-update.service
    sytemctl enable proxedo-api-security-crl-update.timer
    sytemctl start proxedo-api-security-crl-update.timer

4. Base system configuration

This chapter explains configuration details for setting up a working PAS. Configuration settings are detailed here, provided by the .deb package installed on an Ubuntu 18.04 LTS server.

The .deb package carries convenience tools for managing the Proxedo API Security installation, the actual work is done by Docker and docker-compose.

4.1. docker-compose.yml

The main configuration of the running environment is the /opt/balasys/etc/docker-compose.yml file. It describes the containers running the environment.

The format of the file must adhere to the YAML 1.1 specification. For a brief overview of the YAML format look at the example here. For an in-depth reference of docker-compose configuration see its documentation.

This file controls:

  • the image to run the container from

  • the persistent data storage (docker volumes) to attach to the container

  • the ports accessible from outside

  • intra container communication channels (links)

  • log target configuration

There are two configuration settings that need to be done in this file:

  1. All ports accessed by the clients, must be added to the list under services / traffic-director / ports. The first port is the port bound on the host, the second is inside the traffic-director container. This means that the Listener configuration needs to reference the latter in its port parameter.

  2. If you need to change the default behavior of logging into the system’s journal you must change the logging parameters under all the services. See more details in docker-compose’s documentation.

Do not use docker-compose directly to manage the installation. Always use systemctl as it handles dependencies and scaling.

4.2. docker-compose.conf

Some aspects of how the services are run by docker-compose are configured through /opt/balasys/etc/docker-compose.conf.

The format of this file is a shell environment file format: a key-value pair in each line, separated by an equal sign ("=").

There must not be spaces around the equal sign.

The following configuration options can be set:

Table 1. docker-compose.conf configuration options
Key Default Description

PAS_IMAGE_TAG

1.latest

The release track of Proxedo API Security to use. See Tracking version.

PAS_RESTART_POLICY

on-failure

Restart policy for failed containers. It must be one of no, on-failure, unless-stopped, always.

COMPOSE_FILE

/opt/balasys/etc/docker-compose.yml

Path to the compose file.

COMPOSE_PROJECT_NAME

pas

Name used for the compose project.

PAS_FLOW_DIRECTOR_SCALE

1

The number of flow-director instances to run.

Example:

PAS_IMAGE_TAG=1.latest
PAS_RESTART_POLICY=on-failure
COMPOSE_FILE=/opt/balasys/etc/docker-compose.yml
COMPOSE_PROJECT_NAME=pas
PAS_FLOW_DIRECTOR_SCALE=1
Changing any of the values requires the restart of all services.

4.2.1. Tracking version

Proxedo API Security has a version number in the form of major.minor.patch. The docker image labels control how the automatic update of the services are handled. Each image has 3 possible tags:

  • MAJOR.latest (for eg. 1.latest): These tags point to the latest release in the major release. That is, for the release of both x.y.z+1 and x.y+1.0 this tag will be updated, and the services will be upgraded at the restarts. There will not be upgrades for x+1.0.0.

  • MAJOR.MINOR.latest (for eg. 1.3.latest): These tags point to the latest release in the minor release. That is, for the release x.y.z+1 this tag will be updated, and the services will be upgraded at restarts, however not for the release x.y+1.0.

  • MAJOR.MINOR.PATCH (for e.g. 1.4.7): These tags point to a specific release and will never be changed once released.

4.2.2. Scaling Flow Director

A single instance of Flow Director uses a single processor core. It is necessary to adjust the number of instances to use all the available cores. This is controlled by the PAS_FLOW_DIRECTOR_SCALE variable. As the Flow Director handles the most demanding duties among the components, it must be assigned most of the cores. If there are up to four cores available, assign three cores to the Flow Director, and the remaining one core will be suitable for the Transport and Insight Director. If there are more than four cores, assign two cores for the Transport and Insight Director and assign the rest to the Flow Director.

4.3. License

The license file received from Balasys must be placed at /opt/balasys/etc/license.txt.

4.4. Configuration of dockerd

The docker daemon is configured through /etc/docker/daemon.json. The full documentation can be found in the official docker documentation.

Balasys recommends the use of the default configuration.

Do not use /etc/default/docker as it is ignored when systemd is used.

5. Reference configuration for Proxedo API Security

This chapter provides an overview on the configuration of the Proxedo API Security solution, describes configuration structure and provides detailed information on configuration parameters of different components.

5.1. Configuration format

Most user settings are configured from the common configuration file config.yml. The format of the file must adhere to the YAML 1.1 specification. For a brief overview look at the example here.

The file itself is a collection of the main configuration objects described under top level keys:

Table 2. Top level configuration keys
Key Description

global_default

Default configurations used by all components

listener

Configuration of network endpoints listening for incoming traffic

endpoint

Configuration for API endpoints

enforcer

Configuration of Enforcer plugins

filter

Configuration of Filter plugins

insight

Configuration of Insight plugins

target

Configuration of Insight target bricks

matcher

Configuration of matcher bricks

selector

Configuration of selector bricks

backend

Configuration of backend servers

The actual structure of each of these configuration trees is described in the respective sections.

The configuration details may include references to other files. For example, including swagger definitions or TLS certificates inline could be inconvenient, therefore they are provided separately. These files must be mounted as docker volumes to the component that handles them. Detailed information on their exact placement is also provided with their description in the forthcoming sections.

5.2. Listeners

Listeners are network endpoints where services are exposed to the network. They consist of a listening port, an optional client side TLS configuration if HTTPS is used, and list of endpoints that handles the traffic.

Since these are the entry points for client traffic it must be routed here on the network.

5.2.1. Configuring Listeners

The configuration of listener can be completed by defining certain keys for the listeners.

Each of these list items has the following keys:

Table 3. Listeners configuration options
Key Default Description

port

The number of the ports the listener binds to.

endpoints

The list of endpoints, as defined under Endpoints that serves traffic coming in on the listener.

client_tls

off

TLS configuration towards the clients. See Client side TLS for details. Its default value is off, which means not to use TLS (and therefore HTTPS).

Thus, a simple configuration might look as follows:

listener:
    - port: 80
      endpoints:
        - some_endpoint_1
        - some_endpoint_2
    - port: 8080
      endpoints:
        - other_endpoint_1
        - other_endpoint_2
      client_tls: off (1)
1 client_tls can be omitted
All endpoints in the list must have the same backend and backend URL configured.
While ports must be unique, as only one listener can bind to a specific port, it is perfectly valid to route incoming traffic from multiple listeners to the same endpoint.

5.2.2. Client side TLS

When HTTPS is used in communicating with the clients, client_tls settings must be configured.

These options are used by the Traffic Director. For options that reference a file the path is relative to /opt/balasys/var/persistent/ inside the Transport Director container. This directory is a docker volume and by default mounted from the /opt/balasys/var/persistent/transport-director directory in the host system.

Configuration options for Client Side TLS:

Table 4. client_tls configuration
Key Default Description

certificate

Configuration for the X.509 certificate used for TLS connections on the listener. See client_tls certificate configuration for details.

options

See individual options.

TLS protocol options used on the listener. See client_tls options configuration for details.

client_verification

off

Options for verifying client side X.509 certificates. See client_tls verification configuration for details. By default no client verification takes place.

Configuration for the X.509 certificate used for TLS connections on the listener:

Table 5. client_tls certificate configuration
Key Default Description

certificate_file

The path and filename to the certificate file. The certificate must be in PEM format.

key_file

The path and filename to the private key file. The private key must be in PEM format.

key_passphrase

Passphrase used to access the private key specified in key_file.

TLS protocol options used on the listener:

Table 6. client_tls options configuration
Key Default Description

cipher

ECDH+AESGCM: DH+AESGCM: ECDH+AES256: DH+AES256: ECDH+AES128: DH+AES: !aNULL: !MD5: !DSS

Specifies the allowed ciphers. Can be set to all, high, medium, low, or a string representation of the selected ciphers.

method

all

One of: all, sslv23, sslv3, tlsv1, tlsv1_1, tlsv1_2.

cipher_server_preference

no

Use server and not client preference order when determining which cipher suite, signature algorithm or elliptic curve to use for an incoming connection.

dh_param_file

The path and filename to the DH parameter file. The DH parameter file must be in PEM format.

disable_compression

off

Set this to 'on' to disable support for SSL/TLS compression.

disable_renegotiation

on

Set this to 'on' to disable client-initiated renegotiation.

disable_session_cache

off

Do not store session information in the session cache. Set this option to 'on' to disable SSL session reuse.

disable_ticket

off

Session tickets are a method for SSL session reuse, described in RFC 5077. Set this option to 'on' to disable SSL session reuse using session tickets.

disable_tlsv1

on

Do not allow using TLSv1 in the connection.

disable_tlsv1_1

off

Do not allow using TLSv1.1 in the connection.

disable_tlsv1_2

off

Do not allow using TLSv1.2 in the connection.

session_cache_size (integer)

20480

The number of sessions stored in the session cache for SSL session reuse.

timeout

300

Drop idle connection if the timeout value (in seconds) expires.

Options for verifying client side X.509 certificates:

Table 7. client_tls verification configuration
Key Default Description

permit_invalid_certificates

off

When permit_invalid_certificates is on and trusted is off, PAS accepts even invalid certificates, for example, expired or self-signed certificates.

permit_missing_crl

off

This option has effect only if the verify_crl_directory parameter is set. If PAS does not find a CRL in these directories that matches the CAs in the certificate chain and permit_missing_crl is set to off, PAS rejects the certificate. Otherwise, the certificate is accepted even if no matching CRL is found.

required

on

If the key required is on, PAS requires a certificate from the peer.

trusted

on

If the peer shows a certificate and the trusted parameter is on, only certificates signed by a trusted CA are accepted.

trusted_certs_directory

A directory where trusted IP address - certificate assignments are stored. When a peer from a specific IP address shows the certificate stored in this directory, it is accepted regardless of its expiration or issuer CA. Each file in the directory should contain a certificate in PEM format. The file name must be the IP address.

verify_ca_directory

Directory where the trusted CA certificates are stored. CA certificates are loaded on-demand from this directory when PAS verifies the certificate of the peer. When the Transport Director starts, it creates symlinks with the hash of the certificate. This is needed for its operation.

Along with the installation of the package, numerous ca-certificates are also provided, available at the /opt/balasys/etc/ca-certificates folder. In case these certificates are necessary, copy them to /opt/balasys/var/persistent/transport-director/ca-certificates to make them available for the Transport Director.

verify_crl_directory

Directory where the CRLs (Certificate Revocation Lists) associated with trusted CAs are stored. CRLs are loaded on-demand from this directory when PAS verifies the certificate of the peer. When this is set, the CRLs for the certificates in the verify_ca_directory will be automatically dowloaded.

You must enable the proxedo-api-security-crl-update.service and proxedo-api-security-crl-update.timer services and start the latter with systemctl on the host OS for this.

verify_depth (integer)

4

The length of the longest accepted CA verification chain. PAS automatically rejects longer CA chains.

Example
client_tls:
    certificate:
        certificate_file: example.pem
        key_file: example.key
    options:
        method: all
        disable_sslv2: true
        disable_sslv3: true
        disable_tlsv1: true
        disable_tlsv1_1: true
        disable_tlsv1_2: false
        cipher: high
        cipher_server_preference: true
        timeout: 300
        session_cache_size: 20480
        disable_session_cache: false
        disable_ticket: false
        disable_compression: false
        disable_renegotiation: true

5.3. Endpoints

An endpoint holds together all the policies that apply to a certain API endpoint:

  • List of URLs

  • The default error policy for the endpoint

  • The backend to which requests will be forwarded

  • The security flow that will be applied to the traffic

The endpoint tree contains named endpoints with their respective configuration, as follows:

endpoint:
    some_endpoint:
        ...
    other_endpoint:
        ...
Note that unlike listeners, endpoints are named. This is because they need to be referenced from other parts of the configuration. This is true for most top level configuration objects.

Each of the endpoints has the following configuration options:

Table 8. Endpoint configuration
Key Default Description

urls

The list of URLs, which clients use to address the API endpoint.

backend_url

keep url

The URL by which the backend servers understand incoming requests. When set, two transformations take place:

  • The original URL will be replaced by the new URL in the request.

  • The Host header will be replaced by the host indicated in the backend_url.

backend

The name of the backend to route traffic to. References an item in Backends.

backend and backend_url needs to be the same as for all endpoints configured to the same listener.

5.3.1. Security Flow

The Security Flow definition in an endpoint lists what happens to the traffic on a given endpoint.

To understand how requests flow through PAS, see Understanding processing flow. The Security Flow starts when the Transport Director has already set up client connection and routed the request to the Flow Director. At this point the TLS and HTTP layers are already processed, but the content in the body of the request is available only in raw format and has not been parsed yet.

At this stage, the configuration security flow decides on what happens to the traffic by applying a list of Plugins one by one. Plugin is a collective name for Enforcers, Insights, Filters, etc. Once, all the plugins have processed the request, the control is handed back to the Transport Director which routes the request to a backend server, and comes back with the response after handling TLS and HTTP. At this point, the Flow Director applies another list of Plugins to response, and once done, it hands back the response to the Transport Director which in turn returns that to the client.

If at any point an error occurs, the error policy is applied — which might either mean to lead to logging the error or to terminating processing and returning an error indication to the client.

Plugins can override the endpoint’s error policy.

Also note that different Plugins need different data. An Insight that applies a JMESPath query needs parsed JSON, while one that extracts value from an HTTP header field does not. Other Plugins provide these required values, like a JSON de-serializer Plugin. It is important that the Plugins are configured in such an order that the required data is made available beforehand.

Therefore a typical security flow is configured with the plugins in the following order:

  • A Decompressor Plugin that decompresses the content of the request

  • A Deserilaizer Plugin that parses the content of the request

  • Enforcer Plugin(s) that ensures the call is valid

  • Insight Plugin(s) that extract important data from certain calls

  • A Serializer Plugin that rebuilds the contents of the request

  • A Compressor Plugin that compresses the content of the request

5.4. Configuring security flow

The security flow process requires two separate configurations, each containing a list of Plugins as in the following example:

securtiy_flow:
    request:
        - decompressor.default
        - deserializer.default_json
        - plugin_type.plugin_name
        - plugin_type.plugin_name
        - ...
        - serializer.default_json
        - compressor.default
    response:
        - decompressor.default
        - deserializer.default_json
        - plugin_type.plugin_name
        - plugin_type.plugin_name
        - serializer.default_json
        - compressor.default
        - ...

The items of the list are the names of the Plugins they reference, not their actual configuration. The actual configuration is stored in top level configuration objects. A Plugin therefore, which is named as 'enforcer.example' references the configuration under the enforcer configuration.

This means that Plugin configurations are reusable.
(De)compressor Plugin requires no configuration. default must be used as plugin_name.

5.5. Plugins

A Plugin does something specific to the call. The following Plugins exist:

Deserializer

It is a Plugin responsible for parsing the HTTP message’s body to structured data. This ensures that a message is well-formed. The structured data will also be consumed by other Plugins that operate on the body of the message.

Serializer

It is a Plugin responsible for serializing the structured data to the format of the HTTP message’s body.

Filter

It is a Plugin that rejects calls when they match defined rules.

Enforcer

It is a Plugin that validates calls against externally defined schemas.

Insight

It is a Plugin that extracts various data from the call and sends it to external systems (log servers, SIEMs, and other data analysis tools).

5.5.1. Common Plugin parameters

Regardless of what they do, all Plugins share some common parameters:

Error Policy

It defines a custom error policy to be applied if the Plugin reports an error. The settings of the Error policy here override the Security Flow’s default error policy. For details see Error Policies.

Matcher

It decides if the Plugin should be executed based on the call’s details. For details see Matchers. If omitted the Plugin is always executed.

Both of these are optional:

  • If no matcher is configured, the Plugin is always executed.

  • If no error policy is configured, the plugin type’s default error policy is applied.

Only values defined in the custom error policy are overridden, the rest is inherited from the Plugin type’s default error policy, not from the Security Flow’s default error policy. See Error Policies for details on how the error policy hierarchy is applied.

5.5.2. Configuring Plugins

Plugins' configuration details are always defined under top level configuration keys that denote their type. They are always named so that their names refer to a Plugin that represents a certain configuration. The names themselves are referenced from the Security Flow.

The following example presents the configuration hierarchy of a Plugin:

<plugin_type>:
    <plugin_name>:
        [match: matcher name | matcher configuration]
        [error_policy: error policy name | error policy configuration]
        ...
        <plugin specific configuration>:
            ...
    <other_plugin_name>:
        match: ...
        error_policy: ...
        ...

A minimal example configuration is as follows:

enforcer:
    example_enforcer:
        match: v1_uri (1)
        error_policy: detail_log (1)
        swagger:
            schema: example_api_v1.json
1 Reference the name of a matcher and an error policy, respectively.

Instead of referencing an existing entry, matchers and error policies can be defined inline.

The forthcoming two configurations are functionally equivalent therefore:

enforcer:
    example_enforcer:
        match: v1_uri
        ...
matcher:
    v1_uri:
        uri_path: /v1/*
enforcer:
    example_enforcer:
        match:
            uri_path: /v1/*
        ...

5.5.3. Compressor and Decompressor plugins

The Decompressor Plugin decompresses the body of the HTTP message.

The Compressor Plugin compresses the the body of the HTTP message.

These Plugins require no configuration. They understand the Transfer-Encoding HTTP header and can work with content optionally compressed by the gzip, deflate and brotli algorithms.

Since they require no configuration, an instance of both named default is available for inclusion in the security flow. They can be referenced as decompressor.default and compressor.default.

5.5.4. Serialization and Deserialization plugins

The Deserializer Plugin parses the body of the HTTP message’s body to structured data. This ensures that the message is well-formed.

The structured data is also be consumed by other Plugins that operate on the body of the message, such as the jmespath extractor.

The Serializer Plugin recreates the HTTP message’s body from the structured data.

The Deserializer Plugin works with decompressed body.

Serialization needs to be done before compression.

A typical Security Flow configuration therefore starts with a Decompressor followed by a Deserializer and finishes with a Serializer followed by a Compressor. This ensures that transferred HTTP bodies are syntactically correct and that they are reconstructed to avoid transferring potentially crafted content.

They understand the Content-Type HTTP header and can work with JSON and XML content.

The plugin accepts the following configuration options:

Table 9. Deserializer, Serializer plugin configuration options
Key Default Description

parser

It defines the HTTP content parser type. Values: json, xml.

charset_conflict

drop

It defines the cases when charset conflict occurs. When the HTTP Content contains defined encoding and it’s not the same as HTTP Content charset. It works only with Xml content. Values: log, drop.

error_policy

The plugin’s default error policy, see below.

Defines a custom error policy to be applied if the plugin reports an error. Settings here override the Security Flow’s error policy. For details see Error Policies.

match

json_content, xml_content

Decides if the Plugin should be executed based on the call’s details. For details see Matchers.

The Plugin does not override any of the default error policy options.

Problems are considered errors that lead to the termination of the call. Problems in the request are reported back to the client, while errors in the response are suppressed to avoid information leak.

See Error Policies to understand how defaults are applied.

There are two type of predefined (de)serializer plugin:

JSON: 'deserializer.default_json', 'serializer.default_json'. These _Plugins_ match JSON content.
XML: 'deserializer.default_xml', 'serializer.default_xml'. These _Plugins_ match XML content.

An example configuration is the plugin looks like this:

XML deserializer plugin example configuration
deserializer:
    example_xml_deserializer:
        parser: xml
        charset_conflict: log
        match:
            content_type: application/xml
        error_policy: custom_error

5.5.5. Filter plugins

Filter Plugins are lightweight alternatives of Enforcer Plugins for filtering unwanted traffic. They only consist of a matcher and an error policy. If the matcher matches, the error policy is applied. This way you can use matchers inline instead of creating a whole schema-based Enforcer Plugin for the simple use cases.

The Plugin accepts the following configuration options:

Table 10. Filter plugin configuration options
Key Default Description

error_policy

Find the plugin’s default error policy below.

It defines a custom error policy to be applied if the plugin reports an error. The settings here override the Security Flow’s default error policy. For details see Error Policies.

match

always

It decides if the Plugin should be executed based on the call’s details. For details see Matchers. If omitted the plugin is always executed.

The Plugin does not override any of the default error policy options.

Problems are considered errors that lead to the termination of the call. Problems in the request are reported back to the client, while errors in the response are suppressed to avoid information leak.

See Error Policies to understand how defaults are applied.

An example configuration for the Plugin is as follows:

Filter plugin example configuration
filter:
    example_filter:
        match: v1_uri
        error_policy: custom_error
If you omit the matcher, the Plugin will always be executed. For Filter plugins this means aborting all calls.

5.5.6. Enforcer plugins

An Enforcer Plugin validates calls against externally defined schemas.

The Plugin supports validation against OpenAPI2.0 (Swagger) schemas, XSD schemas or WSDL schema.

Understanding the format of these schemas is not in the scope of this document. Further information is available at:

5.5.6.1. Swagger enforcer plugin

The swagger enforcer Plugin validates against OpenApi2.0 schemas.

In case of swagger the Plugin accepts the following configuration options:

Table 11. Swagger enforcer Plugin configuration options
Key Default Description

error_policy

Find the plugin’s default error policy below.

It defines a custom error policy to be applied if the plugin reports an error. The settings here override the Security Flow’s default error policy. For details see Error Policies.

match

always

It decides if the Plugin should be executed based on the call’s details. For details see Matchers. If omitted the plugin is always executed.

swagger

It defines the OpenAPI 2.0 related configuration. It includes a single sub-key named schema.

The Plugin overrides the following fields of the default error policy:

Table 12. Default Enforcer Error Policy
Policy Setting Default

request_code

422

request_message

Validation Error

Problems are considered errors that lead to the termination of the call. Problems in the request are reported back to the client, while errors in the response are suppressed to avoid information leak.

See Error Policies to understand how defaults are applied.

The Plugin needs the schema definition file of the API Endpoint. This file must be in JSON or YML format.

The path to the file is defined under the swagger/schema key in the configuration of the Plugin. The path is relative to /opt/balasys/var/persistent/ inside the flow-director container. This directory is a docker volume and by default mounted from the /opt/balasys/var/persistent/flow-director directory in the host system.

Enforcer plugin example configuration for swagger
enforcer:
    example_enforcer:
        match: v1_uri
        error_policy: detail_log
        swagger:
            schema: example_api_v1.json
5.5.6.2. XSD enforcer plugin

XSD enforcer Plugin validates against XSD schemas. Both XSD 1.0 and 1.1 are supported.

As XSD enforcer requires parsed XML content an xml desirializer plugin needs to be included before XSD enforcer.

In the XSD enforcer you can define operations. Each operation contains criteria for identifying the call, and path of an XSD schema. If the HTTP message meets all criteria, its content will be validated using the schema.

XSD enforcer schema must contain at least one operation.

Configuration of an operation:

Table 13. XSD enforcer plugin configuration options for operation
Key Default Description

error_policy

Find the plugin’s default error policy below.

It defines a custom error policy to be applied if the plugin reports an error. The settings here override the Security Flow’s default error policy. For details see Error Policies.

match

always

It decides if the Plugin should be executed based on the call’s details. For details see Matchers. If omitted the plugin is always executed.

uri_path

*

Defines the pattern for uri_path.

method

Defines the method of the HTTP message

status

Defines the status of the HTTP message

call_direction

Defines the direction of the message, must be request or response.

xsd

Defines the XSD schema

Enforcer Plugin example configuration for XSD
enforcer:
    example_enforcer:
        match: v1_uri
        error_policy: detail_log
        xsd:
            operations:
                - uri_path: /v1/resources/*
                  method: post
                      call_direction: request
                  xsd: example_xsd_v1_resources.xsd
5.5.6.3. WSDL enforcer plugin

WSDL enforcer Plugin validates against WSDL 1.0-1.1 schemas.

As WSDL enforcer requires parsed XML content, an xml desirializer plugin needs to be included before WSDL enforcer.

In case of WSDL, the Plugin accepts the following configuration options:

Table 14. WSDL enforcer Plugin configuration options
Key Default Description

error_policy

Find the plugin’s default error policy below.

It defines a custom error policy to be applied if the plugin reports an error. The settings here override the Security Flow’s default error policy. For details see Error Policies.

match

always

It decides if the Plugin should be executed based on the call’s details. For details see Matchers. If omitted the plugin is always executed.

wsdl

Defines the WSDL related configuration. It includes a single sub-key named schema.

The Plugin overrides the following fields of the default error policy:

Table 15. Default Enforcer Error Policy
Policy Setting Default

request_code

422

request_message

Validation Error

Problems are considered errors that lead to the termination of the call. Problems in the request are reported back to the client, while errors in the response are suppressed to avoid information leak.

See Error Policies to understand how defaults are applied.

The plugin needs the schema definition file. This file must be in XML format.

The path to the file is defined under the wsdl/schema key in the configuration of the plugin. The path is relative to /opt/balasys/var/persistent/ inside the flow-director container. This directory is a docker volume and by default mounted from the /opt/balasys/var/persistent/flow-director directory in the host system.

Enforcer plugin example configuration for WSDL
enforcer:
    example_enforcer:
        match: v1_uri
        error_policy: detail_log
        wsdl:
            schema: example_api_v1.wsdl
WSDL schema validates request and response as weell. Make sure that wsdl enforcer included in request and response flow as well.

In simple cases — when the listener/endpoint is serving a single version of a single API endpoint — a matcher is usually not needed as the schemas define all known URLs in the API.

If however multiple API endpoints are consolidated under a single endpoint definition, you can define multiple enforcers each matching on a sub-path by using an uri_path matcher and putting them all in the Security Flow.

5.5.7. Insight Plugins

The Insight plugin extracts various data from the call and sends it to external systems (log servers, SIEMs, and other data analysis tools).

The Insight plugin accepts the following configuration options:

Table 16. Insight configuration options
Key Default Description

error_policy

Find the plugin’s default error policy below.

It defines a custom error policy to be applied if the plugin reports an error. The settings here override the Security Flow’s default error policy. For details see Error Policies.

match

always

It decides if the Plugin should be executed based on the call’s details. For details see Matchers. If omitted the plugin is always executed.

selectors

A list of Selectors that collect information from the call. They can be referenced by their name or can be defined inline.

targets

A list of Targets where the collected information will be sent to.

The Plugin overrides the following fields of the default error policy:

Table 17. Default Insight Error Policy
Policy Setting Default

request

log

response

log

Problems are considered errors that only need to be logged. If that is overridden then problems in the request are reported back to the client, while errors in the response are suppressed to avoid information leak.

See Error Policies to understand how defaults are applied.

You can mix and match references and inline definitions in the list of selectors.

The Plugin collects the information from all the selectors and sends them to all the targets.

Insight plugin example configuration
insight:
    example_insight:
        match: example_matcher
        error_policy: custom_error_policy
        selectors:
            - some_selector
            - other_selector
        targets:
            - some_target

The collected information from all the selectors is arranged into a dictionary: a list of key — value pairs. The key can be configured in each selector. Certain selectors might return complex data structures, that are made up of other dictionaries and/or lists. To ensure compatibility with a wide range of target types, such results are flattened. The path inside the complex data structure is encoded into the key for each value. More details are available on this in Data flattening.

5.6. Bricks

Bricks are reusable components. They do not provide a complete security function themselves, instead they are used as building blocks elsewhere (hence the name). They can be used by Plugins (like Selectors), or utilized by other bricks (like Extractors).

5.6.1. Error Policies

Error Policies define how to proceed if a Plugin decides to have found an error. For example, when an Enforcer plugin decides that the call is invalid.

It is the error policy that enables to act differently in case the error appears in a request or a response.

An Error Policy contains the following settings:

Table 18. Error policy configuration options
Key Default Description

request

abort

Request error mode:

  • abort: request is denied if the Plugin fails. Use the other parameters to control the content of the error sent to the client.

  • log: invalid requests are allowed, but are logged.

request_silent

no

Do not report validation errors of the request to the client.
When turned off, the _Plugin_s have the ability to report the error in detail in the body of the HTTP error response.

request_code

400

HTTP status code to be used when denying invalid requests.

request_message

Request error

The reason to be used in the HTTP response line when denying invalid requests.

response

abort

Response error mode:

  • abort: response is denied if the Plugin fails. Use the other parameters to control the content of the error sent to the client.

  • log: invalid responses are allowed, but are logged.

response_silent

yes

Do not report validation errors of the response to the client.

response_code

502

HTTP status code to be used when denying invalid responses.

request_message

Response error

The reason to be used in the HTTP response line when denying invalid responses.

The default values in the above table represent the hard coded default values. They form a strict security policy: all errors are fatal, and only mistakes made by the client are reported in detail.

The error_policy configuration tree contains named Error Policies with their respective configuration, as follows:

error_policy:
    some_error_policy:
        ...
    other_error_policy:
        ...

Such error policies can be used in Plugin configurations' error_policy option by referencing their name. It is also possible to write the Error Policy configuration directly inline, if you do not plan to reuse the policy.

5.6.1.1. Error policy hierarchy

Error policy values are applied in a hierarchial manner. There are 3 layers of definition:

  • The hard-coded default error policy (see above).

  • The default error policy of the different Plugin types. These are needed because the expected default behavior depends on what a Plugin does.

  • A custom error policy applied on a Plugin instance that represents user policy.

An error policy value is searched for in a bottom-to-top manner: if it is present in a custom error policy, it will be applied, if not, it will be looked for in the layer above.

This lookup works on a configuration item basis, therefore one can only override a single configuration in case of a custom error policy.

Though, custom error policies can be named objects, they can behave differently when applied on different Plugins, as each Plugin has its own default error policy. The effective error policy of each Plugin instance is logged when the Flow Director starts.

5.6.2. Matchers

Matchers decide if the Plugin should be executed by examining the content of the call. They provide an extremely versatile way of defining the circumstances that must be met for the Plugin to execute.

Matchers need three pieces of information:

  • Name: The name of the matcher defines what part of the call needs to be checked.

  • Pattern: The pattern defines what it needs to be compared with.

  • Comparator: The Comparator shows by what means the collected value of the call is compared with the provided pattern. (Some comparators also take flags or arguments.)

To ease configuration, a matcher in its simplest form is defined as a key: a value pair where the key contains the Name and the Comparator and the value is the pattern, as in the following example:

"name[.comparator[.comparator_flag][.comparator_flag]..]: pattern"

An example is as follows:

"url_path.equals.ignorecase": /example/path

The matcher configuration tree contains named Matchers with their respective configuration, as in the following example:

matcher:
    some_matcher:
        ...
    other_matcher:
        ...

Such matcher can be used in Plugin configurations' match option by referencing their name. It is also possible to write the Matcher configuration directly inline, if you do not plan to reuse the matcher.

There are some named Matchers available without explicit configuration:

  • always and never are instances of always matcher and never matcher.

  • json_content that matches requests with the Content-Type headers representing JSON.

Based on the above information, the following matchers can be used in a Plugin’s configuration: match: never, match: always or match: json_content

Also note, that no other matchers can be defined with these names.

Matchers internally utilize Extractors to fetch the information from the call to compare with. The Name of the matcher resembles the name of the extractor that will be used.

All matchers have a default comparator that is applied implicitly, so the simplest from is as follows: url_path: /example/path.

If you want to use comparator parameters, the comparator name shoud be given even if the default comparator is used.

The List of matchers contains the supported matchers, describing:

  • what extractor the matcher uses

  • what the default comparator is

  • what comparators the matcher supports

What each comparator does is described in List of comparators.

5.6.2.1. Compound matcher

To combine the result of different matchers there are special matchers available — called compound matchers — that take a list of matchers and consider all of their results. Currently the any, all, one and none compound matchers are available.

Their configuration takes a list:

any:
    - "url_path.equals.ignorecase": /example/path
    - "url_path.equals.ignorecase": /other/path

Compound matchers can be put inside each other as in the following example:

all:
    - matcher: pattern
    - matcher: pattern
    - any:
        - matcher: pattern
        - matcher: pattern
5.6.2.2. Implicit Compound matchers

Normal matchers can take a list of patterns, and they automatically turn into an any Compound Matcher containing a matcher instance for all the patterns.

Consequently, these two definitions are equivalent:

matcher:
    path:
        any:
            - "url_path.equals.ignorecase": /example/path
            - "url_path.equals.ignorecase": /other/path
matcher:
    path:
        "url_path.equals.ignorecase":
            - /example/path
            - /other/path

5.6.3. Selectors

Selectors are responsible for collecting information from the call. They utilize Extractor bricks for this purpose.

They are used by Insight Plugins.

The selector accepts the following configuration options:

Table 19. Selector configuration options
Key Default Description

save_as

The key under which the results of a selector are saved in the Insight plugin’s dictionary. If omitted, the result will be directly merged as top level keys.

<extractor>

Exactly one extractor. The key must be the extractor type, the value must be the configuration of the extractor. For details, see Extractors.

The selector configuration tree contains named Selectors with their respective configuration, as follows:

selector:
    some_selector:
        ...
    other_selector:
        ...

Most extractors return simple string values. However, some (might) return dictionaries. For example, you can get all the HTTP headers, or all the URI query parameters. For such extractors you can save the result directly on the top of the Insight’s dictionary.

You can still use save_as for extractors returning dictionaries. For example, you can save all the headers under the headers' key and the URI query parameters under the parameters' key.

5.6.4. Extractors

Extractors are used to extract data from the call.

They are utilized by Matchers and Selectors. Extractors are configured inline in matchers and Selectors, there are no named extractors.

Extractors are included by their type in Selectors, and are used by a special syntax in matchers.

See the List of Extractors for the available extractor types and their configuration options.

5.6.5. Targets

Target bricks define where the data collected by the Insight Plugins will be sent to.

The target configuration tree contains named Targets with their respective configuration, as follows:

target:
    some_target:
        ...
    other_target:
        ...
Unlike other bricks, target configurations cannot be put inline in a Plugin’s configuration, they must always be configured here.

See the List of Targets for the available target types and their configuration options.

5.6.5.1. Data flattening

To ensure compatibility with a wide range of target types, results collected by the Insight plugin are flattened. The path inside the complex data structure is encoded into the key for each value:

  • The merged key describes the path to the value in the data structure as a string

  • The parts of the path will be separated by a forward slash character ("/").

  • Keys in nested dictionaries are added to the path by name

  • List items are added to the path by their index

For example you selected user details from the body by the jmespath as in the following example:

{
  "users": {
    "John Doe": {
      "phone": "555 33 11",
      "emails": [
        "john@example.com",
        "john.doe@example.com"
      ]
    },
    "Jane Doe": {
      "phone": "555 44 22",
      "emails": [
        "jane@example.com"
      ]
    }
  }
}

This would be sent to the target as in the following example:

{
  "users/John Doe/phone": "555 33 11",
  "users/John Doe/emails/0": "jane@example.com",
  "users/John Doe/emails/1": "john.doe@example.com",
  "users/Jane Doe/phone": "555 44 22"
}
You can control the separator with the flatten_separator configuration key that every target accepts.

5.7. List of matchers

5.7.1. always

This matcher always matches.

Example
always:

5.7.2. never

This matcher never matches. It can be used to turn off a Plugin.

Example
never:

5.7.3. method

Supported comparators

equals, not_equals

Default comparator

equals

It matches the HTTP method of the request. Note that the method is case insensitive by definition, therefore the case will always be ignored.

Example
method: get

5.7.4. status

Supported comparators

equals, not_equals, status_class, range, min, max

Default comparator

status_class

It matches the status code of the response.

See the default status_class comparator which allows convenient matching on HTTP status classes.
Example
status: server_error

# or

status.equals: 200

5.7.5. call_direction

Supported comparators

equals, not_equals

Default comparator

equals

Matches the direction of the message (request or response).

Example
call_direction: request

# or

call_direction.equals: response

5.7.6. header

Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches the value of an HTTP header.

Some HTTP headers can be present more than once in a call. To accommodate this, matching is completed against the value of each occurrence of the header. Matching occurs if there is any match.

For example, if the Accept header was repeated as follows:

Accept: application/json
Accept: application/xml

Consequently, in this example above both header.accept: application/json and header.accept: application/xml would match.

The syntax of this matcher differs from the others because the name of the header_name must be added:

"header.<header_name>[.comparator[.comparator_flag][.comparator_flag]..]"

Therefore to match against the header named Server the key will be header.server, possibly completed with comparator specification, like header.server.regex.ignorecase.

While the values are not, the HTTP header names are case insensitive, so you can write them all lowercase in the configuration key.
Example
header.server.pattern.ignorecase: nginx

5.7.7. raw_content

Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

substring

It matches the original content of the message. If the content type is JSON, the body will be decompressed but not parsed.

Example
raw_content: OK

5.7.8. content_type

Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches the content type of the message. It is a more robust solution than using the header on the Content-Type header because that can contain parameters as well.

Example
content_type: text/html

5.7.9. jmespath

Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex, min, max, range

Default comparator

pattern

It matches the data from the body of a JSON call with the help of the JMESPath expression.

JMESPath is a query language for JSON. It is a very versatile tool for extracting the needed information from the body of the call, and organize it according to needs.

A complete explanation on how to write JMESPath expressions is not in the scope of this document. To learn more about it visit the: main website:

Provide the JMESPath expression in the configuration.

Example:
jmespath:
    jmespath_expression: machines[?state=='running'].name
    comparator: pattern
    pattern: machine_name

The supported comparator parameters are described in the List of comparators.

Example of one comparator parameter:
    comparator_params: ignorecase
Example of many comparator parameters:
    comparator_params:
        - ignorecase
        - multiline

5.7.10. URI matchers

A range of matchers is available to match different parts of the URI.

The structure of an URI looks as follows:

scheme://[username[:password]@]host[:port][/path][?query][#fragment]

That is, for example:

https://john.doe:secret123@example.com:8443/some/resource?foo=bar&baz=qux#some-anchor
The fragment part is used by the client locally, and is never sent in the HTTP requests, therefore the PAS cannot do anything with it.

These matchers use the URI extractors. It has an extensive list of examples of what each extractor extracts from the URI.

5.7.10.1. uri
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches against the whole request URI as received from the client.

Example:
uri: http://*example.com/some/resource*
5.7.10.2. uri_netloc
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches the network location in the URI.

It includes:

  • username and password if present

  • host

  • port if present unless scheme default

If the port is the default port for the scheme — that is 80 and 443 for HTTP and HTTPS, respectively — the port will not be included even if explicitly sent by the client. So if the client used http://example.com:80/path then netloc would be example.com, not example.com:80
Example
uri_netloc: john.doe*@example.com*
5.7.10.3. uri_origin
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches the origin part of the URI.

It includes:

  • scheme

  • host

  • port if present, unless the default port for the scheme is used

If the port is the default port for the scheme — that is 80 and 443 for HTTP and HTTPS, respectively — the port will not be included, even if explicitly sent by the client. Therefore if the client used http://example.com:80/path, then the origin would be http://example.com, not http://example.com:80
Example
uri_origin: http://example.com*
5.7.10.4. uri_scheme
Supported comparators

equals, not_equals

Default comparator

equals

It matches the scheme of request (http or https).

Note that the scheme is case insensitive by definition, therefore the case will always be ignored.

Example
uri_scheme: http
5.7.10.5. uri_username
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches the username in the request if present.

Example
uri_username: john.doe
5.7.10.6. uri_password
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches the password in the request if present.

Example
uri_password: secret123
5.7.10.7. uri_host
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches the host in the request.

Example
uri_host: example.com
5.7.10.8. uri_port
Supported comparators

equals, not_equals, min, max, range

Default comparator

equals

It matches the port of the request.

Note that this matches the default port —  that is 80 and 443 for HTTP and HTTPS, respectively — even if it is not explicitly in the request.

Example
uri_port: 8443
5.7.10.9. uri_path
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches the path part of the URI.

It includes:

  • scheme

  • host

  • port if present, unless the default port for the scheme is used

If the port is the default port for the scheme — that is 80 and 443 for HTTP and HTTPS, respectively — the port will not be included, even if explicitly sent by the client. Therefore if the client used http://example.com:80/path, then the origin would be http://example.com, not http://example.com:80

If you need to match the path exactly as received, use uri_raw_path.

Example
uri_path: /*/resource
5.7.10.10. uri_raw_path
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches the path part of the URI, without the normalization of uri_path carried out.

If the path is missing, the match still runs against a single forward slash ("/").

It is recommended to use uri_path unless there is an explicit need for matching the raw path. One such example would be logging or filtering out badly formed requests.

Example
uri_raw_path: /some/resource
5.7.10.11. uri_raw_query
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern It matches the query part of the URI as a string.

It is recommended to use uri_query_param unless there is an explicit need for matching the raw string.

An example on this might be if there is a match on foo=barbar or tofoo=bar as well, even though it was not intended.

Example
uri_raw_query.substring: foo=bar
5.7.10.12. uri_query_param
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches the value of a query parameter.

It is also valid for URIs to include a query parameter more than once. That is, it could be foo=bar&qux=quz&foo=baz. To accommodate this, matching is done against the value of each occurrence of the parameter. Matching occurs if any value is matched. In the following example both uri_query_param.foo: bar and uri_query_param.foo: baz would match.

The syntax of this matcher differs from the others because the name of the query parameter must be added:

"uri_query_param.<query_param_name>[.comparator[.comparator_flag][.comparator_flag]..]"

In order to match the parameter named foo the key would be uri_query_param.foo, possibly with comparator specification, like uri_query_param.foo.equals.ignorecase.

Example
uri_query_param.foo: bar

5.7.11. any

Any is a Compound matcher that matches if any of its sub-matchers matches.

Example
any:
    - "url_path.equals.ignorecase": /example/path
    - "url_path.equals.ignorecase": /other/path

5.7.12. all

All is a Compound matcher that matches if all of its sub-matchers match.

Example
all:
    - host: example.com
    - url_path.equals.ignorecase: /example/path

5.7.13. one

One is a Compound matcher that matches if exactly one of its sub-matchers match.

Example
one:
    - url_path.equals.ignorecase: /example/path
    - url_path.equals.ignorecase: /other/path

5.7.14. none

None is a Compound matcher that matches if none of its sub-matchers match.

Example
none:
    - url_path.equals.ignorecase: /example/path
    - url_path.equals.ignorecase: /other/path

5.7.15. xpath

Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex, min, max, range

Default comparator

pattern

It matches the data from the body of an XML call with the help of the Xpath expression.

Xpath is a query language for XML. It is a very versatile tool for extracting the needed information from the body of the call, and organize it according to needs.

A complete explanation on how to write Xpath expressions is not in the scope of this document. To learn more about it visit the main website.

Table 20. Xpath matcher configuration options
Key Default Description

xpath_expression

The expression to extract the node from the call to match against.

namespaces

Defines the XML namespaces.

clear_text

False

Remove white spaces at the beginning and at the end of the string.

comparator

The name of the comparator to use for comparing the pattern with the result of the xpath expression. For details, see List of comparators.

comparator_params

The supported comparator parameters are described in the List of comparators.

pattern

The pattern to compare with the extracted value with.

Provide the Xpath expression in the configuration.

Example:
Input:
<bookstore>
    <book category="cooking">
      <title lang="en">Everyday Italian</title>
      <author>Giada De Laurentiis</author>
      <year>2005</year>
      <price>30.00</price>
      <available>true</available>
    </book>
</bookstore>
xpath:
    xpath_expression: 'bookstore/book[1]/author/text()'
    comparator: equals
    pattern: Giada De Laurentiis
Example of namespaces:
Input:
<m:bookstore xmlns:m = "http://www.xyz.org/m">
    <m:book xmlns:k = "http://www.xyz.org/k" category="cooking">
      <k:title lang="en">Everyday Italian</k:title>
      <k:author>Giada De Laurentiis</k:author>
      <k:year>2005</k:year>
      <k:price>30.00</k:price>
      <k:available>true</k:available>
    </m:book>
</m:bookstore>
xpath:
    xpath_expression: 'm:bookstore/m:book[1]/k:author/text()'
    namespaces:
        m: 'http://www.xyz.org/m'
        k: 'http://www.xyz.org/k'
    comparator: equals
    pattern: Giada De Laurentiis
Example of one comparator parameter:
    comparator_params: ignorecase
Example of many comparator parameters:
    comparator_params:
        - ignorecase
        - multiline

5.7.16. SOAP matchers

A range of matchers is available to match different parts of the SOAP message.

These matchers extend the xpath matcher with predefined expressions.

They use the SOAP extractors. It has an extensive list of examples of what each extractor extracts from the SOAP message.

Example:
soap_fault_code:
    xpath_expression: ''
    comparator: equals
    pattern: '<SOAP-ENV:faultcode>SOAP-ENV:Client</SOAP-ENV:faultcode>'

You can extend the base query with unique expressions:

Example of extension:
soap_fault_code:
    xpath_expression: '/text()'
    comparator: equals
    pattern: 'SOAP-ENV:Client'
5.7.16.1. soap_version
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches the the soap message version. It identify with the soap namespace.

Possible values:

  • soapv1_1 - the message version is SOAP v1.1

  • soapv1_2 - the message version is SOAP v1.2

Example:
soap_version:
    xpath_expression: ''
    comparator: equals
    pattern: soapv1_1
5.7.16.2. soap_envelope
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches with the soap envelope.

Example:
soap_envelope:
    xpath_expression: ''
    comparator: startswith
    pattern: '<SOAP-ENV:Envelope'
5.7.16.3. soap_header
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches with the soap header.

Example:
soap_header:
    xpath_expression: ''
    comparator: startswith
    pattern: '<SOAP-ENV:Header'
5.7.16.4. soap_body
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches with the soap body.

Example:
soap_body:
    xpath_expression: ''
    comparator: startswith
    pattern: '<SOAP-ENV:Body'
5.7.16.5. soap_fault
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches with the soap fault.

Example:
soap_fault:
    xpath_expression: ''
    comparator: startswith
    pattern: '<SOAP-ENV:Fault'
5.7.16.6. soap_fault_code
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches with the soap fault 'code'. The expression depends on the soap version.

  • faultcode - it is the SOAP v1.1 node tag.

  • Code - it is the SOAP v1.2 node tag.

Example of faultcode:
soap_fault_code:
    xpath_expression: ''
    comparator: startswith
    pattern: '<SOAP-ENV:faultcode'
Example of Code:
soap_fault_code:
    xpath_expression: ''
    comparator: startswith
    pattern: '<SOAP-ENV:Code'
5.7.16.7. soap_fault_detail
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches with the soap fault 'detail'. The expression depends on the soap version.

  • detail - it is the SOAP v1.1 node tag.

  • Detail - it is the SOAP v1.2 node tag.

Example of detail:
soap_fault_detail:
    xpath_expression: ''
    comparator: startswith
    pattern: '<SOAP-ENV:detail'
Example of Detail:
soap_fault_detail:
    xpath_expression: ''
    comparator: startswith
    pattern: '<SOAP-ENV:Detail'
5.7.16.8. soap11_fault_faultstring
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches with the soap fault 'faultstring'. This matcher only work with soap version 1.1.

Example:
soap11_fault_faultstring:
    xpath_expression: ''
    comparator: startswith
    pattern: '<SOAP-ENV:faultstring'
5.7.16.9. soap11_fault_faultactor
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches with the soap fault 'faultactor'. This matcher only work with soap version 1.1.

Example:
soap11_fault_faultactor:
    xpath_expression: ''
    comparator: startswith
    pattern: '<SOAP-ENV:faultactor'
5.7.16.10. soap12_fault_reason
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches with the soap fault 'Reason'. This matcher only work with soap version 1.2.

Example:
soap12_fault_reason:
    xpath_expression: ''
    comparator: startswith
    pattern: '<SOAP-ENV:Reason'
5.7.16.11. soap12_fault_node
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches with the soap fault 'Node'. This matcher only work with soap version 1.2.

Example:
soap12_fault_node:
    xpath_expression: ''
    comparator: startswith
    pattern: '<SOAP-ENV:Node'
5.7.16.12. soap12_fault_role
Supported comparators

equals, not_equals, starts_with, ends_with, substring, pattern, regex

Default comparator

pattern

It matches with the soap fault 'Role'. This matcher only work with soap version 1.2.

Example:
soap12_fault_role:
    xpath_expression: ''
    comparator: startswith
    pattern: '<SOAP-ENV:Role'

5.8. List of comparators

Comparators are utilized by Matchers to understand how their configured pattern is compared with the extracted value.

5.8.1. equals

It matches if the parameter is exactly the same as the value matched.

Parameters:

Name Description

ignorecase

Case differences are ignored. When the present VaLuE would match value.

5.8.2. not_equals

It matches if the parameter is not exactly the same as the value matched.

Parameters:

Name Description

ignorecase

Case differences are ignored. When present VaLuE would not match vAlUe.

5.8.3. starts_with

It matches if the value starts exactly with the pattern.

Parameters:

Name Description

ignorecase

Case differences are ignored. When present VaLuE would match value_given.

5.8.4. ends_with

It matches if the value ends exactly with the pattern.

Parameters:

Name Description

ignorecase

Case differences are ignored. When present VaLuE would match given_value.

5.8.5. substring

It matches if the exact pattern is found somewhere in value.

Parameters:

Name Description

ignorecase

Case differences are ignored. When present VaLuE would match some-value-given.

5.8.6. pattern

Pattern treats input as Unix shell-style wildcards. The special characters used in shell-style wildcards are:

Pattern Meaning

*

Matches everything.

?

Matches a single character.

[seq]

Matches any character in seq.

[!seq]

Matches any character not in seq.

For a literal match, wrap the meta-characters in brackets. For example, [?] matches a literal question-mark (?).

Parameters:

Name Description

ignorecase

Case differences are ignored. When present VaLuE would match some-value-given.

5.8.7. regex

Regex treats input as a regular expression for matching. Consult Python’s regular expression documentation and their Regular Expression HOWTO.

Name Description

ignorecase

Sets the IGNORECASE flag for the regex.

multiline

Sets the MULTILINE flag for the regex.

5.8.8. min

It matches if pattern is larger or equal to value.

5.8.9. max

It matches if pattern is smaller or equal to value.

5.8.10. range

Matches if value is between the limits in the pattern, including boundaries. The format of the pattern must be min..max.

Example
range: 200..210

5.8.11. status_class

Status_class is a special matcher for conveniently matching HTTP status code classes. It takes the name of the class and checks if the status code is in the given range as follows:

Pattern Status code range Description

info

1xx

Informational response

success

2xx

Successful response

redirect

3xx

Redirects

client_error

4xx

Client Errors

server_error

5xx

Server Errors

5.9. List of Extractors

5.9.1. method

It extracts the HTTP method of the request.

It does not require configuration.

Example
method:

5.9.2. status

It extracts the status code of the response.

It does not require configuration.

Example
status:

5.9.3. header

It extracts the value of an HTTP header.

It is valid for some HTTP headers to be present more than once in a call. In this case all the values are extracted as a list.

It provides the name of the header in the configuration.

Example:
header: server

5.9.4. header_force_list

It works like header but it returns a list even if there is only a single extracted value.

Example:
header_force_list: server

5.9.5. header_first

It works like header but it only returns the first extracted value even if there is a list of extracted values.

Example:
header_first: server

5.9.6. headers

It extracts all the headers from the call.

The results are stored as a dictionary, so when you can choose to omit the save_as key if you use this from a selector.

It is valid for some HTTP headers to be present more than once in a call. In such cases all the values are stored under the header’s key as a list.

It does not require configuration.

Example
headers:

5.9.7. URI extractors

A range of extractors is available to extract different parts of the URI.

The structure of an URI looks as follows:

scheme://[username[:password]@]host[:port][/path][?query][#fragment]

That is, for example:

https://john.doe:secret123@example.com:8443/some/resource?foo=bar&baz=qux#some-anchor
The fragment part is used by the client locally, and is never sent in the HTTP requests, therefore the PAS cannot do anything with it.

It does not require configuration.

5.9.7.1. uri

It matches against the whole request URI as received from the client.

Example
uri:

It does not require configuration.

5.9.7.2. uri_netloc

Extracts the network location in the URI.

It includes:

  • username and password if present

  • host

  • port if present unless scheme default

If the port is the default port for the scheme — that is 80 and 443 for HTTP and HTTPS, respectively — the port will not be included even if explicitly sent by the client. So if the client used http://example.com:80/path then netloc would be example.com, not example.com:80

It does not require configuration.

Example
uri_netloc:
5.9.7.3. uri_origin

Extracts the origin part of the URI.

It includes:

  • scheme

  • host

  • port if present, unless the default port for the scheme is used

If the port is the default port for the scheme — that is 80 and 443 for HTTP and HTTPS, respectively — the port will not be included, even if explicitly sent by the client. Therefore if the client used http://example.com:80/path, then the origin would be http://example.com, not http://example.com:80

It does not require configuration.

Example
uri_origin:
5.9.7.4. uri_scheme

Extracts the scheme of the request (http or https).

It does not require configuration.

Example
uri_scheme:
5.9.7.5. uri_username

Extracts the username in the request if present.

It does not require configuration.

Example
uri_username:
5.9.7.6. uri_password

Extracts the password in the request if present.

It does not require configuration.

Example
uri_password:
5.9.7.7. uri_host

Extracts the host in the request.

It does not require configuration.

Example
uri_host:
5.9.7.8. uri_port

Extracts the port of the request.

Extracts the default port —  that is 80 and 443 for HTTP and HTTPS, respectively — even if not displayed explicitly in the request.

It does not require configuration.

Example
uri_port:
5.9.7.9. uri_path

It extracts the path part of the URI.

It includes:

  • scheme

  • host

  • port if present, unless the default port for the scheme is used

If the port is the default port for the scheme — that is 80 and 443 for HTTP and HTTPS, respectively — the port will not be included, even if explicitly sent by the client. Therefore if the client used http://example.com:80/path, then the origin would be http://example.com, not http://example.com:80

If you need to extract the path exactly as received, use uri_raw_path.

It does not require configuration.

Example
uri_path:
5.9.7.10. uri_raw_path

It extracts the path part of the URI, without the normalization of uri_path carried out.

If the path is missing a single forward slash ("/") is extracted.

It does not require configuration.

Example
uri_raw_path:
5.9.7.11. uri_query

It extracts the query part of the URI.

It does not require configuration.

Example
uri_query:
5.9.7.12. uri_raw_query

It extracts the query part of the URI as a string.

It does not require configuration.

Example
uri_raw_query:
5.9.7.13. uri_query_param

It extracts the value of a query parameter.

It is also valid for URIs to include a query parameter more than once. That is, it could be foo=bar&qux=quz&foo=baz. In this case both values are extracted as a list.

Provide the name of the parameter in the configuration.

Example
uri_query_param: foo
5.9.7.14. uri_query_param_force_list

It works like uri_query_param but it returns a list even if there is only a single extracted value.

Example
uri_query_param_force_list: foo
5.9.7.15. uri_query_param_first

It works like uri_query_param but it only returns the first extracted value even if there is a list of extracted values.

Example
uri_query_param_first: foo

5.9.8. jmespath

It extracts data from the body of a JSON call with the help of a JMESPath expression.

JMESPath is a query language for JSON. It is a very versatile tool for extracting the needed information from the body of the call, and organize it according to requirements.

A complete explanation of how to write JMESPath expressions is not in the scope of this document. To learn more about it visit this website: main website:

Provide the JMESPath expression in the configuration.

Example:
jmespath: machines[?state=='running'].name

5.9.9. call_direction

Extracts the call direction (request, response).

It does not require configuration.

Example:
call_direction:

5.9.10. timestamp

Extracts the current time.

Configuration:

Name Default Description

timezone

'UTC'

Set the timezone.

  • An str describing a timezone, similar to ‘US/Pacific’, or ‘Europe/Berlin’. See: Timezones

  • An str in ISO 8601 style, as in ‘+07:00’.

  • An str, one of the following: ‘local’, ‘utc’, ‘UTC’.

format

'YYYY-MM-DDTHH:mm:ss.SSSSSSZZ'

Set the format. See: format options

Table 21. format options
Token Output

Year

YYYY
YY

2000, 2001, 2002 … 2012, 2013
00, 01, 02 … 12, 13

Month

MMMM
MMM
MM
M

January, February, March
Jan, Feb, Mar
01, 02, 03 … 11, 12
1, 2, 3 … 11, 12

Day of Year

DDDD
DDD

001, 002, 003 … 364, 365
1, 2, 3 … 364, 365

Day of Month

DD
D
Do

01, 02, 03 … 30, 31
1, 2, 3 … 30, 31
1st, 2nd, 3rd … 30th, 31st

Day of Week

dddd
ddd
d

Monday, Tuesday, Wednesday
Mon, Tue, Wed
1, 2, 3 … 6, 7

Hour

HH
H
hh
h

00, 01, 02 … 23, 24
0, 1, 2 … 23, 24
01, 02, 03 … 11, 12
1, 2, 3 … 11, 12

AM / PM

A
a

AM, PM, am, pm
am, pm

Minute

mm
m

00, 01, 02 … 58, 59
0, 1, 2 … 58, 59

Second

ss
s

00, 01, 02 … 58, 59
0, 1, 2 … 58, 59

Sub-second

S…

0, 02, 003, 000006, 123123123123
the result is truncated to microseconds, with half-to-even rounding

Timezone

ZZZ
ZZ
Z

Asia/Baku, Europe/Warsaw, GMT
-07:00, -06:00 … +06:00, +07:00, +08, Z
-0700, -0600 … +0600, +0700, +08, Z

Seconds Timestamp

X

1381685817, 1381685817.915482

ms or µs Timestamp

x

1569980330813, 1569980330813221

Example
timestamp:
    timezone: 'UTC'
    format: 'YYYY-MM-DDTHH:mm:ss.SSSSSSZZ'

5.9.11. content

It extracts the content.

It does not require configuration.

Example
content:

5.9.12. raw_content

It extracts the content as a string.

It does not require configuration.

Example
raw_content:

5.9.13. content_type

It extracts the content_type from the HTTP header.

It does not require configuration.

Example
content_type:

5.9.14. content_type_charset

It extracts the charset from the content_type HTTP header.

It does not require configuration.

Example
content_type_charset:

5.9.15. static

It extracts a string, integer, number, object, array, boolean as string from the configuration.

Example of string
    static: test
Example of array
    static:
        - first
        - second

5.9.16. session_id

It extracts the session ID as string, generated by the Transport Director.

It does not require configuration.

Example
session_id:

5.9.17. xpath

It extracts data from the body of an XML call with the help of a Xpath expression.

Xpath is a query language for XML. It is a very versatile tool for extracting the needed information from the body of the call, and organize it according to needs.

A complete explanation on how to write Xpath expressions is not in the scope of this document. To learn more about it visit the main website.

Table 22. Xpath extractor configuration options
Key Default Description

xpath_expression

The expression to extract the node from the call to match against.

namespaces

Defines the XML namespaces.

clear_text

False

Remove white spaces at the beginning and at the end of the string.

Provide the Xpath expression in the configuration. Depending on the expression, the return value is a single node or a list of nodes. If you want a single value or a list independent from the expression, use xpath_first or xpath_force_list.

Example:
Input:
<bookstore>
    <book category="cooking">
      <title lang="en">Everyday Italian</title>
      <author>Giada De Laurentiis</author>
      <year>2005</year>
      <price>30.00</price>
      <available>true</available>
    </book>
</bookstore>
xpath:
    xpath_expression: 'bookstore/book[1]/author/text()'
Output:
    Giada De Laurentiis
Example of namespaces:
Input:
<m:bookstore xmlns:m = "http://www.xyz.org/m">
    <m:book xmlns:k = "http://www.xyz.org/k" category="cooking">
      <k:title lang="en">Everyday Italian</k:title>
      <k:author>Giada De Laurentiis</k:author>
      <k:year>2005</k:year>
      <k:price>30.00</k:price>
      <k:available>true</k:available>
    </m:book>
</m:bookstore>
xpath:
    xpath_expression: 'm:bookstore/m:book[1]/k:author/text()'
    namespaces:
        m: 'http://www.xyz.org/m'
        k: 'http://www.xyz.org/k'
Output:
    Giada De Laurentiis
5.9.17.1. xpath_force_list

It works like xpath but it returns a list even if there is only a single extracted value.

Example:
xpath_force_list:
    xpath_expression: 'bookstore/book[1]/author/text()'
5.9.17.2. xpath_first

It works like xpath but it only returns the first extracted value even if there is a list of extracted values.

Example:
xpath_first:
    xpath_expression: 'bookstore/book/author/text()'

5.9.18. SOAP extractors

A range of extractors is available to extract different parts of the SOAP message.

These extractors extend the xpath with predefined expressions.

Example:
soap_body:
    xpath_expression: ''
Base query of soap_body:
/*[local-name() = 'Envelope']/*[local-name() = 'Body']

You can extend the base query with unique expressions:

Example of extension:
soap_body:
    xpath_expression: '/text()'
Extended query of soap_body:
/*[local-name() = 'Envelope']/*[local-name() = 'Body']/text()
5.9.18.1. soap_version

It extracts the soap message version. It identify with the soap namespace.

Possible values:

  • soapv1_1 - the message version is SOAP v1.1

  • soapv1_2 - the message version is SOAP v1.2

Example:
soap_version:
    xpath_expression: ''
5.9.18.2. soap_envelope

It extracts the soap envelope.

Base query:
/*[local-name() = 'Envelope']
Example:
soap_envelope:
    xpath_expression: ''
5.9.18.3. soap_header

It extracts the soap header.

Base query:
/*[local-name() = 'Envelope']/*[local-name() = 'Header']
Example:
soap_header:
    xpath_expression: ''
5.9.18.4. soap_body

It extracts the soap body.

Base query:
/*[local-name() = 'Envelope']/*[local-name() = 'Header']
Example:
soap_body:
    xpath_expression: ''
5.9.18.5. soap_fault

It extracts the soap fault.

Base query:
/*[local-name() = 'Envelope']/*[local-name() = 'Body']/*[local-name() = 'Fault']
Example:
soap_fault:
    xpath_expression: ''
5.9.18.6. soap_fault_code

It extracts the soap fault 'code'. This extractor expression depend from the soap version.

  • faultcode - it is the SOAP v1.1 node tag

  • Code - it is the SOAP v1.2 node tag

Base queries:
faultcode: /*[local-name() = 'Envelope']/*[local-name() = 'Body']/*[local-name() = 'Fault']/*[local-name() = 'faultcode']
Code: /*[local-name() = 'Envelope']/*[local-name() = 'Body']/*[local-name() = 'Fault']/*[local-name() = 'Code']
Example:
soap_fault_code:
    xpath_expression: ''
5.9.18.7. soap_fault_detail

It extracts the soap fault 'detail'. This matcher expression depend from the soap version.

  • detail - it is the SOAP v1.1 node tag

  • Detail - it is the SOAP v1.2 node tag

Base queries:
detail: /*[local-name() = 'Envelope']/*[local-name() = 'Body']/*[local-name() = 'Fault']/*[local-name() = 'detail']
Detail: /*[local-name() = 'Envelope']/*[local-name() = 'Body']/*[local-name() = 'Fault']/*[local-name() = 'Detail']
Example:
soap_fault_detail:
    xpath_expression: ''
5.9.18.8. soap11_fault_faultstring

It extracts the soap fault 'faultstring'. This extractor only work with soap version 1.1.

Base query:
/*[local-name() = 'Envelope']/*[local-name() = 'Body']/*[local-name() = 'Fault']/*[local-name() = 'faultstring']
Example:
soap11_fault_faultstring:
    xpath_expression: ''
5.9.18.9. soap11_fault_faultactor

It extracts the soap fault 'faultactor'. This extractor only work with soap version 1.1.

Base query:
/*[local-name() = 'Envelope']/*[local-name() = 'Body']/*[local-name() = 'Fault']/*[local-name() = 'faultactor']
Example:
soap11_fault_faultactor:
    xpath_expression: ''
5.9.18.10. soap12_fault_reason

It extracts the soap fault 'Reason'. This extractor only work with soap version 1.2.

Base query:
/*[local-name() = 'Envelope']/*[local-name() = 'Body']/*[local-name() = 'Fault']/*[local-name() = 'Reason']
Example:
soap12_fault_reason:
    xpath_expression: ''
5.9.18.11. soap12_fault_node

It extracts the soap fault 'Node'. This extractor only work with soap version 1.2.

Base query:
/*[local-name() = 'Envelope']/*[local-name() = 'Body']/*[local-name() = 'Fault']/*[local-name() = 'Node']
Example:
soap12_fault_node:
    xpath_expression: ''
5.9.18.12. soap12_fault_role

It extracts the soap fault 'Role'. This extractor only work with soap version 1.2.

Base query:
/*[local-name() = 'Envelope']/*[local-name() = 'Body']/*[local-name() = 'Fault']/*[local-name() = 'Role']
Example:
soap12_fault_role:
    xpath_expression: ''

5.10. List of Targets

Targets are utilized by Insight Plugins for sending the data to external systems.

5.10.1. local_log

It logs the result of the insight in the local system log.

The target accepts the following configuration options:

Table 23. local_log target configuration options
Key Default Description

level

3

The log level for the logged message.

tag

info

The log tag for the logged message.

nmessage

The message of the insight if present, otherwise it is empty.

The message part of the log message.

5.10.2. syslog

It sends the insight to a remote syslog server using the IETF syslog protocol defined in RFC5424.

Table 24. syslog target configuration options
Key Default Description

protocol

tcp

The transport protocol to send messages over. Either tcp or udp.

port

514 for TCP
601 for UDP

The port number to connect to on the remote system.

flush_lines

0

Specifies how many lines are flushed to a destination at a time. The Insights Director waits for this number of lines to accumulate and sends them off in a single batch. Increasing this number increases throughput as more messages are sent in a single batch, but also increases message latency.

ip_protocol

4

Determines the internet protocol version of the given driver. The possible values are 4 and 6, corresponding to IPv4 and IPv6.

time_zone

GMT

The name of the timezone, or the timezone offset.
The timezone can be specified by using the name, for example, "Europe/Budapest", or as the timezone offset in +/-HH:MM format, for example, +01:00).

data_format

sdata

The data format of the insight. Possible values: sdata, json.

flatten

True

Flatten the target message.

flatten_separator

/

The separator in the flattened message.

5.10.3. elastic

Sends the insight to an Elasticsearch engine in JSON.

Table 25. elastic target configuration options
Key Default Description

host

The hostname of the Elastic search instance.

port

9200

The port to connect to on the Elastic search instance.

index

The name of the index in the Elastic search instance.

doc_type

_doc

The doc_type to use when sending the data.

flatten

False

Flatten the target message.

flatten_separator

/

The separator in the flattened message.

5.11. Backends

Backends are a set of servers for a given API endpoint.

Their configuration is made up of 2 main parts:

  • A list of servers: port pairs and how to route traffic to them

  • TLS configuration for talking to the servers

The backend configuration tree contains named Backends with their respective configuration, as follows:

backend:
    some_backend:
        ...
    other_backend:
        ...

Backends take the following configuration options:

Table 26. Backend configuration
Key Default Description

servers

The list of servers that serve API endpoint(s). See Backend servers configuration for details.

backend_use_client_address

no

If it is set to yes, PAS uses the client’s source address as the source of the server-side connection. Otherwise it uses the IP address of the interface connected to the server.

forge_port

any

This parameter defines the source port that PAS uses in the server-side connection. The source port can be one of the followings:

  • any: Select a random port between 1024 and 65535.

  • group: Select a random port in the same group as the port used by the client. The following groups are defined: 0-513, 514-1024,1025-.

  • exact: Use the same port as the client.

  • random: Select a random port using a cryptographically secure function.

lb_method

direct

Load balancing method to use. One of the following methods can be used:

  • direct: always use the first server

  • failover: use the first server while available, then fail over to the next

  • rr: use all servers in a round-robin fashion

backend_timeout

30000

Connection timeout in milliseconds for a server considered to be down.

backend_retry_in:

600000

Timeout in milliseconds before a server -that has been considered to be down- is restarted again.

backend_tls

TLS configuration towards backend servers. See Backend side TLS for details.

Each entry in the list under the server parameter takes the following configuration options:

Table 27. Backend servers configuration
Key Default Description

host

The name or IP address of the host to connect to.

port

The port on host to connect to.

5.11.1. Backend side TLS

When HTTPS is used the backend_tls settings must be configured.

These options are used by the Transport Director. For options that reference a file the path is relative to /opt/balasys/var/persistent/ inside the transport-director container. This directory is a docker volume and by default mounted from the /opt/balasys/var/persistent/transport-director directory in the host system.

Backend TLS takes the following configuration options:

Table 28. Backend TLS configuration
Key Default Description

certificate

No client certificate is sent.

Configuration for the X.509 certificate used as a client certificate for TLS connections towards the backend servers. See Backend TLS certificate configuration for details.

options

See individual options.

TLS protocol options used on TLS connections towards the backend servers. See Backend TLS options configuration for details.

backend_verification

off

Options for verifying backend X.509 certificates. See Backend TLS verification configuration for details. By default no backend verification takes place.

Configuration for the X.509 certificate used for TLS connections towards the backend servers:

Table 29. Backend TLS certificate configuration
Key Default Description

certificate_file

The path and filename to the certificate file. The certificate must be in PEM format.

key_file

The path and filename to the private key file. The private key must be in PEM format.

key_passphrase

Passphrase used to access the private key specified in key_file.

TLS protocol options used on TLS connections towards the backend servers:

Table 30. Backend TLS options configuration
Key Default Description

method

all

One of: all, sslv23, sslv3, tlsv1, tlsv1_1, tlsv1_2

cipher

ECDH+AESGCM: DH+AESGCM: ECDH+AES256: DH+AES256: ECDH+AES128: DH+AES: !aNULL: !MD5: !DSS

Specifies the allowed ciphers. Can be set to all, high, medium, low, or a string representation of the selected ciphers.

disable_compression

off

Set this to on to disable support for SSL/TLS compression.

disable_session_cache

off

Do not store session information in the session cache. Set this option to 'on' to disable SSL session reuse.

disable_ticket

off

Session tickets are a method for SSL session reuse, described in RFC 5077. Set this option to 'on' to disable SSL session reuse using session tickets.

disable_tlsv1

on

Do not allow the usage of TLSv1 in the connection.

disable_tlsv1_1

off

Do not allow the usage of TLSv1.1 in the connection.

disable_tlsv1_2

off

Do not allow the usage of TLSv1.2 in the connection.

session_cache_size (integer)

20480

The number of sessions stored in the session cache for SSL session reuse.

Options for verifying backendX.509 certificates:

Table 31. Backend TLS verification configuration
Key Default Description

permit_invalid_certificates

off

When permit_invalid_certificates is on and trusted is off, PAS accepts even invalid certificates, for example, expired or self-signed certificates.

permit_missing_crl

off

This option has effect only if the verify_crl_directory parameter is set. If PAS does not find a CRL in these directories that matches the CAs in the certificate chain and permit_missing_crl is set to off, PAS rejects the certificate. Otherwise, the certificate is accepted even if no matching CRL is found.

required

on

If the required key is set to on, PAS requires a certificate from the peer.

trusted

on

If the peer shows a certificate and the trusted parameter is on, only certificates signed by a trusted CA are accepted.

trusted_certs_directory

A directory where trusted IP address - certificate assignments are stored. When a peer from a specific IP address shows the certificate stored in this directory, it is accepted regardless of its expiration or issuer CA. Each file in the directory should contain a certificate in PEM format. The filename must be the IP address.

verify_ca_directory

A directory where the trusted CA certificates are stored. CA certificates are loaded on-demand from this directory when PAS verifies the certificate of the peer. When the Transport Director starts it creates symlinks with the hash of the certificate. This is needed for its operation.

verify_crl_directory

A directory where the CRLs (Certificate Revocation Lists) associated with trusted CAs are stored. CRLs are loaded on-demand from this directory when PAS verifies the certificate of the peer. When this is set, CRLs for the certificates in verify_ca_directory will be automatically dowloaded.

You must enable the proxedo-api-security-crl-update.service and proxedo-api-security-crl-update.timer services and start the latter with systemctl on the host OS for this.

verify_depth (integer)

4

The length of the longest accepted CA verification chain. PAS will automatically reject longer CA chains.

check_subject

on

If this key is set to 'on', PAS compares the Subject of the server-side certificate with application-layer information (for example, it checks whether the Subject matches the hostname in the URL).

Example
backend_tls:
    certificate:
        certificate_file: example.pem
        key_file: example.key
    options:
        method: all
        disable_tlsv1: true
        disable_tlsv1_1: true
        disable_tlsv1_2: false
        cipher: high
        timeout: 30

5.12. Transport director settings

The transport_director configuration tree contains the Transport Director-specific configuration, as in the following example:

tranposrt director:
    parameter: value
    ...

The following settings control the Transport Director container’s startup.

Table 32. Transport Director configuration
Key Default Description

logtags

yes

Prepend log category and log level to each message.

logescape

yes

Escape non-printable characters to avoid binary log files. Each character less than 0x20 and greater than 0x7F are escaped in the form <XX>.

enablecore

no

Enables core dumps on failures.

fdlimitmin

0

The minimum number of the required file descriptors.

restartmax

0

The maximum number of restarts within a specified interval.

restartinterval

0

Set the length of the interval in seconds to check process restarts.

notifyinterval

0

Interval between sending 2 notifications in seconds.

threads

0

Set the maximum number of threads that can be used in parallel.

deadlockchecktimeout

0

Timeout for deadlock detection queries in seconds.

5.13. Logging

The log configuration tree contains log level configuration, as in the following example:

log:
    loglevel: value
    logspec: value
Table 33. Log configuration

loglevel

3

Log level to log. Must be between 1-9.

logspec

"*.accounting:4,core.summary:4"

Set verbosity mask on a per category basis. Each log message has an assigned multi-level category, where levels are separated by a dot. A single log specification consists of a wildcard matching log category, a colon, and a number specifying the verbosity level of that given category. Categories match from left to right. For example: http.*:5,core:3. The last matching entry will be used as the verbosity of the given category. If no match is found the default verbosity specified with loglevel is used.

6. Operations

6.1. Operation of dockerd

Dockerd is managed through 'systemd', so common administration tasks are carried out through its interfaces.

Checking the status of docker

systemctl status docker

Example output
docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2017-07-10 08:25:38 CEST; 4h 1min ago
     Docs: https://docs.docker.com
 Main PID: 2148 (dockerd)
    Tasks: 177 (limit: 4915)
   Memory: 119.1M
      CPU: 1min 36.272s
   CGroup: /system.slice/docker.service
           ├─2148 /usr/bin/dockerd
           ├─2185 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcon
           ├─2542 docker-containerd-shim fef20e5205c47b5cc18e612903a33e749ebd89a4bf30fd5bb8fb4a801450c84f /var/run/docker/libcontainerd/fef20e5205c47b5cc18e612903a33e749ebd8
           ├─2582 docker-containerd-shim 410f0bc67c731635a7d60e9f259d2f62ef8a845e09595254217decd3b3885473 /var/run/docker/libcontainerd/410f0bc67c731635a7d60e9f259d2f62ef8a8
           ├─2704 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 5000 -container-ip 172.18.0.2 -container-port 5000
           ├─2732 docker-containerd-shim 3853efde62d1767e70372584812df07968a647f40039691d82ccd5cbc66ee32d /var/run/docker/libcontainerd/3853efde62d1767e70372584812df07968a64
           ├─2770 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8484 -container-ip 172.18.0.2 -container-port 443
           ├─2806 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 8181 -container-ip 172.18.0.2 -container-port 80
           ├─2832 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 2222 -container-ip 172.18.0.2 -container-port 22
           ├─2837 docker-containerd-shim e24a8f2f189467601edb6bee0e63451e7230726feab50d43556e6c66a8f9fc56 /var/run/docker/libcontainerd/e24a8f2f189467601edb6bee0e63451e72307
           ├─2921 docker-containerd-shim 8ac62e1eee0d162e632eab95b08ea36aff69abd5d1eeac475bfee3f393cba179 /var/run/docker/libcontainerd/8ac62e1eee0d162e632eab95b08ea36aff69a
           ├─2974 docker-containerd-shim 6df61a17c29a132cb5886a494fc34e38ff38f2cf470919289c783fada579a70c /var/run/docker/libcontainerd/6df61a17c29a132cb5886a494fc34e38ff38f
           ├─3043 docker-containerd-shim d00a1de3994e2b11ecd93d938dc94702f4f6d0364d2f3c1c423ab2a1ec5c843a /var/run/docker/libcontainerd/d00a1de3994e2b11ecd93d938dc94702f4f6d
           ├─3123 docker-containerd-shim b9e93059835c2d343c912c7f7154b14625dcd2e8d242fd67328e9532e5829d64 /var/run/docker/libcontainerd/b9e93059835c2d343c912c7f7154b14625dcd
           ├─3187 docker-containerd-shim 2d058ab3987f2461c5f0029505eca264f94d34ed23c8464bfd83362ad9bcd142 /var/run/docker/libcontainerd/2d058ab3987f2461c5f0029505eca264f94d3
           └─3258 docker-containerd-shim 882c51a1a693230ea2d84f2f1a422655f9051d3a21a5f916a03e62614b17ed4a /var/run/docker/libcontainerd/882c51a1a693230ea2d84f2f1a422655f9051
Starting docker

systemctl start docker

Stopping docker

systemctl stop docker

Configuring docker to start automatically

systemctl enable docker

Configuring docker not to start automatically

systemctl disable docker

6.2. Operation of services

The services of PAS are consolidated under the user pas who has privileges for common administration tasks.

Unless otherwise noted administrative commands should therefore be run as pas and not as root. This is especially true for docker-compose commands.

6.2.1. Checking configuration with pas-checkconfig

pas-checkconfig can check the validity of PAS configuration. Namely:

  • it makes sure that the mandatory containers are defined in docker-compose.yml.

  • it checks config.yml against the defined containers one by one.

docker-compose.yml is only checked for making sure that services have a proper image tag.

6.2.2. Checking services

Checking the status of PAS

systemctl status proxedo-api-security

List the status of the services

docker-compose ps

Example output
          Name                        Command               State                    Ports
------------------------------------------------------------------------------------------------------------
pas_flow-director_1        /opt/balasys/bin/go-init - ...   Up
pas_flow-director_2        /opt/balasys/bin/go-init - ...   Up
pas_insight-director_1     /opt/balasys/bin/go-init - ...   Up
pas_transport-director_1   /opt/balasys/bin/go-init - ...   Up      0.0.0.0:443->443/tcp, 0.0.0.0:80->80/tcp
Check which images are used by the services

docker-compose images

Example output
       Container                              Repository                                  Tag                  Image Id      Size
----------------------------------------------------------------------------------------------------------------------------------
pas_flow-director_1        docker.balasys.hu/api-security/flow-director        1.0.0   94bced587138   215 MB
pas_flow-director_2        docker.balasys.hu/api-security/flow-director        1.0.0   94bced587138   215 MB
pas_insight-director_1     docker.balasys.hu/api-security/insight-director     1.0.0   47da6789a496   480 MB
pas_transport-director_1   docker.balasys.hu/api-security/transport-director   1.0.0   26f7e79c5c4e   279 MB

6.2.3. Starting and stopping services

Starting PAS

systemctl start proxedo-api-security

Stopping PAS

systemctl stop proxedo-api-security

Restarting PAS

systemctl restart proxedo-api-security

pas-checkconfig is invoked prior to (re)starting the service.
Configuring PAS to start automatically

systemctl enable proxedo-api-security

Configuring PAS not to start automatically

systemctl disable proxedo-api-security

6.2.4. Upgrading services

Prior to upgrading services, make sure that the image tags point to the right version. See section docker-compose.conf for details.

The upgrade process will cause a service disruption.
Upgrade all services

/opt/balasys/bin/pas-update

6.3. Checking Logs

All the 'components' logs are collected in the system journal. Component logs are prefixed with the name of the component such as pas-[transport|insight|flow]-director.

You can check the system journal with the journalctl command. It accepts various possibilites for filtering, consults its manual page for details.

When using the --unit option of journalctl, note that the services are docker containers and their logs show up under the docker service, and not under proxedo-api-security.

6.3.1. Understanding logs

There are two important concepts related to logs: categories and Session IDs.

  • Categories help filtering logs based on their relevance. They are composed of a component, a tag, and a severity, for example: http.info(3).

    • The component helps to identify the part of the solution. For the Transport Director this is usually core or http, for the Flow Director it is either core, or the Plugin’s type, such as serializer or enforcer.

    • The tag helps to define the type of the message. Usually one of info, error, debug, policy or accounting.

    • The severity defines how important the message is. It is a number between 1-9 where 1 is the highest.

  • Session ID helps identifying log lines that belong to the same session. This is especially important as the calls travel between the Transport Director and the Flow Director.

It is usually in the form of svc/default/<listener>:<transport-director-session>/default/http#<http-request-count>/flow:<flow-director-id>/ch:<flow-director-channel>/<plugin_type>/<plugin_name>, for example: svc/default/httpbin:14/default/http#0/flow:1/ch:28/enforcer/manualtest.

Information that is not available at the time, will be missing from the Session ID. Generally, the part until /flow: belongs to the Transport Director. Consequently, the Transport Director will never see that part. The Flow Director however will fetch and include that information. Nevertheless, in early phases it might not be available, and the Session ID will start with flow:.

Despite some parts not being always available, the ID is constructed in such a manner that greping on any part will find other messages with extra information as well.

6.4. Backup and restore

Configuration

The following files and folders need to be backed up or restored:

  • /opt/balasys/etc

  • /opt/docker

Data

The following files or folders need to be backed up or restored:

  • /opt/balasys/var/persistent

Process to backup files or folders
  • Pack files or folders mentioned earlier (optional)

  • Copy (packed) configuration and data to backup server

Process to restore files or folders
  • Stop all PAS services

  • Copy (packed) configuration and data from remote server

  • Unpack files or folders mentioned earlier (optional)

  • Start all PAS services

6.5. Recreating services

Recreating services will cause a service disruption.
Factory reset for PAS services

Remove all persistent data from the host.

This operation must be run as root.

/opt/balasys/bin/pas-factory-reset

Resetting an individual service without removing persistent data
  • Use docker-compose ps to find the container name of the service the container of which you want to reset.

  • Stop the services by systemctl stop proxedo-api-security

  • Remove the containers by docker rm <name-of-container>

  • Start the services by systemctl start proxedo-api-security

6.6. Troubleshooting services

Inspect running processes inside services

docker-compose top [SERVICE]

Example output
pas_flow-director_1
UID    PID    PPID   C   STIME   TTY     TIME                                                                    CMD
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
root   3303   3251   0   22:35   ?     00:00:00   /opt/balasys/bin/go-init -pre /opt/balasys/bin/pre.sh -main twistd -ny /opt/balasys/etc/twisted.tac -post /opt/balasys/bin/post.sh
root   3654   3303   0   22:35   ?     00:00:00   /usr/bin/python3 /usr/local/bin/twistd -ny /opt/balasys/etc/twisted.tac

pas_flow-director_2
UID    PID    PPID   C   STIME   TTY     TIME                                                                    CMD
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
root   3611   3567   0   22:35   ?     00:00:00   /opt/balasys/bin/go-init -pre /opt/balasys/bin/pre.sh -main twistd -ny /opt/balasys/etc/twisted.tac -post /opt/balasys/bin/post.sh
root   3849   3611   0   22:35   ?     00:00:00   /usr/bin/python3 /usr/local/bin/twistd -ny /opt/balasys/etc/twisted.tac
Gain a shell to a running service

pas-login SERVICE

Appendix A: Timezones

Timezones
Country Code Timezone Name

AD

Europe/Andorra

AE

Asia/Dubai

AF

Asia/Kabul

AG

America/Antigua

AI

America/Anguilla

AL

Europe/Tirane

AM

Asia/Yerevan

AO

Africa/Luanda

AQ

Antarctica/McMurdo

AQ

Antarctica/Casey

AQ

Antarctica/Davis

AQ

Antarctica/DumontDUrville

AQ

Antarctica/Mawson

AQ

Antarctica/Palmer

AQ

Antarctica/Rothera

AQ

Antarctica/Syowa

AQ

Antarctica/Troll

AQ

Antarctica/Vostok

AR

America/Argentina/Buenos_Aires

AR

America/Argentina/Cordoba

AR

America/Argentina/Salta

AR

America/Argentina/Jujuy

AR

America/Argentina/Tucuman

AR

America/Argentina/Catamarca

AR

America/Argentina/La_Rioja

AR

America/Argentina/San_Juan

AR

America/Argentina/Mendoza

AR

America/Argentina/San_Luis

AR

America/Argentina/Rio_Gallegos

AR

America/Argentina/Ushuaia

AS

Pacific/Pago_Pago

AT

Europe/Vienna

AU

Australia/Lord_Howe

AU

Antarctica/Macquarie

AU

Australia/Hobart

AU

Australia/Currie

AU

Australia/Melbourne

AU

Australia/Sydney

AU

Australia/Broken_Hill

AU

Australia/Brisbane

AU

Australia/Lindeman

AU

Australia/Adelaide

AU

Australia/Darwin

AU

Australia/Perth

AU

Australia/Eucla

AW

America/Aruba

AX

Europe/Mariehamn

AZ

Asia/Baku

BA

Europe/Sarajevo

BB

America/Barbados

BD

Asia/Dhaka

BE

Europe/Brussels

BF

Africa/Ouagadougou

BG

Europe/Sofia

BH

Asia/Bahrain

BI

Africa/Bujumbura

BJ

Africa/Porto-Novo

BL

America/St_Barthelemy

BM

Atlantic/Bermuda

BN

Asia/Brunei

BO

America/La_Paz

BQ

America/Kralendijk

BR

America/Noronha

BR

America/Belem

BR

America/Fortaleza

BR

America/Recife

BR

America/Araguaina

BR

America/Maceio

BR

America/Bahia

BR

America/Sao_Paulo

BR

America/Campo_Grande

BR

America/Cuiaba

BR

America/Santarem

BR

America/Porto_Velho

BR

America/Boa_Vista

BR

America/Manaus

BR

America/Eirunepe

BR

America/Rio_Branco

BS

America/Nassau

BT

Asia/Thimphu

BW

Africa/Gaborone

BY

Europe/Minsk

BZ

America/Belize

CA

America/St_Johns

CA

America/Halifax

CA

America/Glace_Bay

CA

America/Moncton

CA

America/Goose_Bay

CA

America/Blanc-Sablon

CA

America/Toronto

CA

America/Nipigon

CA

America/Thunder_Bay

CA

America/Iqaluit

CA

America/Pangnirtung

CA

America/Atikokan

CA

America/Winnipeg

CA

America/Rainy_River

CA

America/Resolute

CA

America/Rankin_Inlet

CA

America/Regina

CA

America/Swift_Current

CA

America/Edmonton

CA

America/Cambridge_Bay

CA

America/Yellowknife

CA

America/Inuvik

CA

America/Creston

CA

America/Dawson_Creek

CA

America/Fort_Nelson

CA

America/Vancouver

CA

America/Whitehorse

CA

America/Dawson

CC

Indian/Cocos

CD

Africa/Kinshasa

CD

Africa/Lubumbashi

CF

Africa/Bangui

CG

Africa/Brazzaville

CH

Europe/Zurich

CI

Africa/Abidjan

CK

Pacific/Rarotonga

CL

America/Santiago

CL

America/Punta_Arenas

CL

Pacific/Easter

CM

Africa/Douala

CN

Asia/Shanghai

CN

Asia/Urumqi

CO

America/Bogota

CR

America/Costa_Rica

CU

America/Havana

CV

Atlantic/Cape_Verde

CW

America/Curacao

CX

Indian/Christmas

CY

Asia/Nicosia

CY

Asia/Famagusta

CZ

Europe/Prague

DE

Europe/Berlin

DE

Europe/Busingen

DJ

Africa/Djibouti

DK

Europe/Copenhagen

DM

America/Dominica

DO

America/Santo_Domingo

DZ

Africa/Algiers

EC

America/Guayaquil

EC

Pacific/Galapagos

EE

Europe/Tallinn

EG

Africa/Cairo

EH

Africa/El_Aaiun

ER

Africa/Asmara

ES

Europe/Madrid

ES

Africa/Ceuta

ES

Atlantic/Canary

ET

Africa/Addis_Ababa

FI

Europe/Helsinki

FJ

Pacific/Fiji

FK

Atlantic/Stanley

FM

Pacific/Chuuk

FM

Pacific/Pohnpei

FM

Pacific/Kosrae

FO

Atlantic/Faroe

FR

Europe/Paris

GA

Africa/Libreville

GB

Europe/London

GD

America/Grenada

GE

Asia/Tbilisi

GF

America/Cayenne

GG

Europe/Guernsey

GH

Africa/Accra

GI

Europe/Gibraltar

GL

America/Godthab

GL

America/Danmarkshavn

GL

America/Scoresbysund

GL

America/Thule

GM

Africa/Banjul

GN

Africa/Conakry

GP

America/Guadeloupe

GQ

Africa/Malabo

GR

Europe/Athens

GS

Atlantic/South_Georgia

GT

America/Guatemala

GU

Pacific/Guam

GW

Africa/Bissau

GY

America/Guyana

HK

Asia/Hong_Kong

HN

America/Tegucigalpa

HR

Europe/Zagreb

HT

America/Port-au-Prince

HU

Europe/Budapest

ID

Asia/Jakarta

ID

Asia/Pontianak

ID

Asia/Makassar

ID

Asia/Jayapura

IE

Europe/Dublin

IL

Asia/Jerusalem

IM

Europe/Isle_of_Man

IN

Asia/Kolkata

IO

Indian/Chagos

IQ

Asia/Baghdad

IR

Asia/Tehran

IS

Atlantic/Reykjavik

IT

Europe/Rome

JE

Europe/Jersey

JM

America/Jamaica

JO

Asia/Amman

JP

Asia/Tokyo

KE

Africa/Nairobi

KG

Asia/Bishkek

KH

Asia/Phnom_Penh

KI

Pacific/Tarawa

KI

Pacific/Enderbury

KI

Pacific/Kiritimati

KM

Indian/Comoro

KN

America/St_Kitts

KP

Asia/Pyongyang

KR

Asia/Seoul

KW

Asia/Kuwait

KY

America/Cayman

KZ

Asia/Almaty

KZ

Asia/Qyzylorda

KZ

Asia/Qostanay

KZ

Asia/Aqtobe

KZ

Asia/Aqtau

KZ

Asia/Atyrau

KZ

Asia/Oral

LA

Asia/Vientiane

LB

Asia/Beirut

LC

America/St_Lucia

LI

Europe/Vaduz

LK

Asia/Colombo

LR

Africa/Monrovia

LS

Africa/Maseru

LT

Europe/Vilnius

LU

Europe/Luxembourg

LV

Europe/Riga

LY

Africa/Tripoli

MA

Africa/Casablanca

MC

Europe/Monaco

MD

Europe/Chisinau

ME

Europe/Podgorica

MF

America/Marigot

MG

Indian/Antananarivo

MH

Pacific/Majuro

MH

Pacific/Kwajalein

MK

Europe/Skopje

ML

Africa/Bamako

MM

Asia/Yangon

MN

Asia/Ulaanbaatar

MN

Asia/Hovd

MN

Asia/Choibalsan

MO

Asia/Macau

MP

Pacific/Saipan

MQ

America/Martinique

MR

Africa/Nouakchott

MS

America/Montserrat

MT

Europe/Malta

MU

Indian/Mauritius

MV

Indian/Maldives

MW

Africa/Blantyre

MX

America/Mexico_City

MX

America/Cancun

MX

America/Merida

MX

America/Monterrey

MX

America/Matamoros

MX

America/Mazatlan

MX

America/Chihuahua

MX

America/Ojinaga

MX

America/Hermosillo

MX

America/Tijuana

MX

America/Bahia_Banderas

MY

Asia/Kuala_Lumpur

MY

Asia/Kuching

MZ

Africa/Maputo

NA

Africa/Windhoek

NC

Pacific/Noumea

NE

Africa/Niamey

NF

Pacific/Norfolk

NG

Africa/Lagos

NI

America/Managua

NL

Europe/Amsterdam

NO

Europe/Oslo

NP

Asia/Kathmandu

NR

Pacific/Nauru

NU

Pacific/Niue

NZ

Pacific/Auckland

NZ

Pacific/Chatham

OM

Asia/Muscat

PA

America/Panama

PE

America/Lima

PF

Pacific/Tahiti

PF

Pacific/Marquesas

PF

Pacific/Gambier

PG

Pacific/Port_Moresby

PG

Pacific/Bougainville

PH

Asia/Manila

PK

Asia/Karachi

PL

Europe/Warsaw

PM

America/Miquelon

PN

Pacific/Pitcairn

PR

America/Puerto_Rico

PS

Asia/Gaza

PS

Asia/Hebron

PT

Europe/Lisbon

PT

Atlantic/Madeira

PT

Atlantic/Azores

PW

Pacific/Palau

PY

America/Asuncion

QA

Asia/Qatar

RE

Indian/Reunion

RO

Europe/Bucharest

RS

Europe/Belgrade

RU

Europe/Kaliningrad

RU

Europe/Moscow

UA

Europe/Simferopol

RU

Europe/Kirov

RU

Europe/Astrakhan

RU

Europe/Volgograd

RU

Europe/Saratov

RU

Europe/Ulyanovsk

RU

Europe/Samara

RU

Asia/Yekaterinburg

RU

Asia/Omsk

RU

Asia/Novosibirsk

RU

Asia/Barnaul

RU

Asia/Tomsk

RU

Asia/Novokuznetsk

RU

Asia/Krasnoyarsk

RU

Asia/Irkutsk

RU

Asia/Chita

RU

Asia/Yakutsk

RU

Asia/Khandyga

RU

Asia/Vladivostok

RU

Asia/Ust-Nera

RU

Asia/Magadan

RU

Asia/Sakhalin

RU

Asia/Srednekolymsk

RU

Asia/Kamchatka

RU

Asia/Anadyr

RW

Africa/Kigali

SA

Asia/Riyadh

SB

Pacific/Guadalcanal

SC

Indian/Mahe

SD

Africa/Khartoum

SE

Europe/Stockholm

SG

Asia/Singapore

SH

Atlantic/St_Helena

SI

Europe/Ljubljana

SJ

Arctic/Longyearbyen

SK

Europe/Bratislava

SL

Africa/Freetown

SM

Europe/San_Marino

SN

Africa/Dakar

SO

Africa/Mogadishu

SR

America/Paramaribo

SS

Africa/Juba

ST

Africa/Sao_Tome

SV

America/El_Salvador

SX

America/Lower_Princes

SY

Asia/Damascus

SZ

Africa/Mbabane

TC

America/Grand_Turk

TD

Africa/Ndjamena

TF

Indian/Kerguelen

TG

Africa/Lome

TH

Asia/Bangkok

TJ

Asia/Dushanbe

TK

Pacific/Fakaofo

TL

Asia/Dili

TM

Asia/Ashgabat

TN

Africa/Tunis

TO

Pacific/Tongatapu

TR

Europe/Istanbul

TT

America/Port_of_Spain

TV

Pacific/Funafuti

TW

Asia/Taipei

TZ

Africa/Dar_es_Salaam

UA

Europe/Kiev

UA

Europe/Uzhgorod

UA

Europe/Zaporozhye

UG

Africa/Kampala

UM

Pacific/Midway

UM

Pacific/Wake

US

America/New_York

US

America/Detroit

US

America/Kentucky/Louisville

US

America/Kentucky/Monticello

US

America/Indiana/Indianapolis

US

America/Indiana/Vincennes

US

America/Indiana/Winamac

US

America/Indiana/Marengo

US

America/Indiana/Petersburg

US

America/Indiana/Vevay

US

America/Chicago

US

America/Indiana/Tell_City

US

America/Indiana/Knox

US

America/Menominee

US

America/North_Dakota/Center

US

America/North_Dakota/New_Salem

US

America/North_Dakota/Beulah

US

America/Denver

US

America/Boise

US

America/Phoenix

US

America/Los_Angeles

US

America/Anchorage

US

America/Juneau

US

America/Sitka

US

America/Metlakatla

US

America/Yakutat

US

America/Nome

US

America/Adak

US

Pacific/Honolulu

UY

America/Montevideo

UZ

Asia/Samarkand

UZ

Asia/Tashkent

VA

Europe/Vatican

VC

America/St_Vincent

VE

America/Caracas

VG

America/Tortola

VI

America/St_Thomas

VN

Asia/Ho_Chi_Minh

VU

Pacific/Efate

WF

Pacific/Wallis

WS

Pacific/Apia

YE

Asia/Aden

YT

Indian/Mayotte

ZA

Africa/Johannesburg

ZM

Africa/Lusaka

ZW

Africa/Harare

Glossary

API

Application Programming Interface

CA

Certification Authority

CRL

Certificate Revocation List

HTTP

HyperText Transport Protocol

HTTPS

HyperText Transport Protocol Secure

JSON

Javascript Object Notation

PEM

Privacy Enhanced Mail

SSL

Secure Socket Layer

SIEM

Security Information and Event Management

TLS

Transport Layer Security

URI

Universal Resource Indicator

URL

Universal Resource Locator