Apache Traffic Control

My notes on creating a prototype CDN using ATC and ATS. All of this is specifically for CentOS 8.

Traffic Ops

The Traffic Ops host will host both the Postgresql database and the ATC “traffic_ops” package. “traffic_ops” is the central engine of ATC - most other components connect to “traffic_ops” and treat it as the source of truth. The current recommendation is to install the database and “traffic_ops” on the same host to avoid having to configure external access to the database.

  1. Install Postgresql. An install script can be obtained from here or this one can be used (which was obtained from that webpage).

    # Install the repository RPM:
    sudo dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm
    
    # Disable the built-in PostgreSQL module:
    sudo dnf -qy module disable postgresql
    
    # Install PostgreSQL:
    sudo dnf install -y postgresql13-server
    
    # Optionally initialize the database and enable automatic start:
    sudo /usr/pgsql-13/bin/postgresql-13-setup initdb
    sudo systemctl enable postgresql-13
    sudo systemctl start postgresql-13
    

    Differences:

    • No configuration to enable access from other hosts. All access will be from this host.

    • Root access is via user id, not password.

    As of Traffic Control 6.0.0 and Postgresql 13 (at this time, the stable version of each), there is a mismatch between how Postgresql and Traffic Control expect to perform password authentication. To fix this, Postgresql needs to be adjusted to use an older method.

    Edit “/var/lib/pgsql/13/data/postgresql.conf” and make this change

    password_encryption = md5       # md5 or scram-sha-256
    

    It may or not be necessary to also change “/var/lib/pgsql/13/data/pg_hba.conf” to enable “md5” password authentication. This involves changing “scram-sha-256” to “md5”.

    # IPv4 local connections:
    host    all             all             127.0.0.1/32            md5
    # IPv6 local connections:
    host    all             all             ::1/128                 md5
    

    If this is not done, you may see this error from “traffic_ops”

    pq: unknown authentication response: 10
    
  2. Install EPEL which provides some of the “traffic_ops” requirements.

    sudo dnf install -y epel-release
    
  3. Install the “traffic_ops” RPM. This should automatically install all dependencies.

    sudo dnf install --enablerepo=powertools --enablerepo=epel -y traffic_ops-6.0.0-11512.e4469f55.el8.x86_64.rpm
    
  4. Initialize the databases.

    Run a shell as the “postgres” user.

    sudo su -c "bash -l" - postgress
    

    Create the “traffic_ops” user, in this case with the password “vgl”. In production this should be something a bit more secure.

    psql
    CREATE USER traffic_ops with encrypted password 'vgl';
    \q
    

    If doing this by hand the terminal output should look like

    $ psql
    psql (13.4)
    Type "help" for help.
    
    postgres=# CREATE USER traffic_ops with encrypted password 'vgl';
    CREATE ROLE
    postgres=# \q
    $
    

    Then at the shell command prompt (not in the psql environment) create the databases.

    createdb traffic_ops --owner traffic_ops
    createdb traffic_vault --owner traffic_ops
    

    Then exit from the “postgres” user shell.

  5. Finalization

    The script “/opt/traffic_ops/install/bin/postinstall” needs to be invoked as root. Unfortunately the check for postgresql superuser may not work. Line 38 may need to be changed to

    if [[ ! $(su -c 'psql -c "show is_superuser"' - postgres </dev/null 2>/dev/null) =~ 'on' ]]; then
    

    At some point I will submit a PR to fix this.

    For the “secrets”, this is still a bit of a mystery to me.

  6. Network Configuration

    TLS requires a certificate and corresponding key files. These can be anywhere as long as they are accessible to the “traffic_ops” process. The best locations are either in “/etc/pki” or somewhere in the “traffic_ops” install directory (generally “/opt/traffic_ops”). The location of these files is stored in the “cdn.conf” file in “/opt/traffic_ops/app/conf”. For legacy reasons these are referenced by editing the URL for the listen key under the hypnotoad key. In the query string of this URL are two keys, “cert” and “key”. The “cert” value must be the absolute path to the certificate (“.pem”) file. The “key” value must the absolute path to the key (”*.key”) file.

    To allow external connections from Traffic Portal the TLS port must opened in the firewall. This is done with

    sudo firewall-cmd --permanent --zone=public --add-service=https
    sudo firewall-cmd --reload
    

References

Notes

postgresql

createdb is a command line tool, not an SQL command used inside the psql context.

Switching user id to “postgres” is needed only for root database control. This setup will be listening on the standard postgresql port on localhost (IPv4 and IPv6) which is what “traffic_ops” will use to perform its operations, logging as the user created for it.

Do not install the standard postgresql package - that causes problems. One effect seems to be the command line utilities don’t get installed in “/usr/bin” which causes other problems.

Traffic Portal

The base instructions worked.

curl --silent --location https://rpm.nodesource.com/setup_14.x | sudo bash -
sudo dnf install -y nodejs
sudo dnf install -y /tmp/traffic_portal-6.0.0-11512.e4469f55.el8.x86_64.rpm

It is expected that Traffic Portal will accept incoming browser requests and then make requests to the Traffic Ops host for data.

The Traffic Portal listening ports must be opened in the CentOS firewall.

sudo firewall-cmd --permanent --zone=public --add-service=http
sudo firewall-cmd --permanent --zone=public --add-service=https
sudo firewall-cmd --reload

References

Traffic Server

This requires building an RPM for Traffic Server. There is a base “.spec” file in “tools/packages/trafficserver.spec”. This is close but probably not quite what is needed. Among the changes I intend to try

  • Change the --layout option to “opt” instead of “Gentoo”.

To build the RPM several packages are needed on the build machine.

sudo dnf install -y epel-release
sudo dnf install -y rpmdevtools
sudo dnf --enablerepo=epel --enablerepo=powertools install -y expat-devel hwloc-devel libcurl-devel ncurses-devel

The rpmbuild command will put the output in “~/rpmbuild/RPMS” and I cannot find any way to change that.

Notes

The output directory can be overriden in the “.spec” file with

%_topdir /path

Administration

It is not possible to create “servers” (records for ATS instances) immediately. Information about the CDN must be created first. The steps for that are

  1. Define a “division”.

  2. Define a “region”.

  3. Define a “location” (aka “physical location”).

  4. Define a “cache group”.

  5. Define a “profile” of type ATS_PROFILE.

After this a “server” can be created to represent an ATS instance.

Tools

Communications should be over TLS and that requires certificates. For the prototype I generated my own root certificate and used it to sign the deployed certificates. It is expected that instead of answering prompts, the scripts will be edited to put the appropriate values in the certificates.

Note - this is fine for proof of concept and prototyping, but not for actual production use.

Both scripts depend on a file name “openssl.cnf” which is defined by OpenSSL. This is the base content, which should be tweaked as needed for local conditions.

# SPDX-License-Identifier: Apache-2.0

# Required environment variables
# PREFIX - used to set the common name.
# DN - used to select a root or signed distinguished name. Must be "CA" or "Signed".

[ ca ]
default_ca = CA_default

[ CA_default ]
new_certs_dir = .

[ req ]
# Disable prompting so the commands use data from here!
prompt = no
distinguished_name = DN-${ENV::DN}

[DN-Signed]
CN = base.ex:role.${ENV::PREFIX}
O = TxnBox
C = S3
ST = Amber
L = Kolvir

[DN-CA]
CN = TxnBox CA ${ENV::PREFIX}
O = TxnBox
C = S3
ST = Confusion
L = Dazed

The script to generate the root certificate.

# SPDX-License-Identifier: Apache-2.0
# Generate root certificate.
# Note - after running this, no existing signed certificate will work with the new root CA.

# the openssl.cnf file requires two environment variables
# PREFIX - the certificate prexifx. This is used to decorate the distinguished name.
# DN - this must be "CA" for root certs and "Signed" for signed certs.

# Set serial number for signed certs.
echo 1 > ca-serial-no.txt

# Generate the root cert key.
rm -f vgl-ca.key
openssl genrsa -passout pass:claudio -des3 -out vgl-ca.key 2048

# Create the root cert.
rm -f vgl-ca.pem
PREFIX=vgl DN=CA openssl req -passin pass:claudio -config ./openssl.cnf -x509 -new -nodes \
   -key vgl-ca.key -sha256 -days 36500 -batch -out vgl-ca.pem

Once the root certificate has been generated, signed certificates for deployment are generated with this script.

# SPDX-License-Identifier: Apache-2.0
# Bash script to generate signed certificates.

# the openssl.cnf file requires two environment variables
# PREFIX - the certificate prefix. This is used to decorate the distinguished name.
# DN - this must be "CA" for root certs and "Signed" for signed certs.

let N=$(cat ca-serial-no.txt) # signed certificate serial number.

# Generate key for the signed cert.
name="vgl-signed-${N}"
key_file="${name}.key"
cert_file="${name}.pem"

rm -f ${key_file}
PREFIX=vgl DN=Signed openssl genrsa -passout pass:claudio -out ${key_file} 2048

# Generate the CSR for the signed cert.
rm -f tmp.csr
PREFIX=vgl DN=Signed openssl req -passin pass:claudio -config ./openssl.cnf -new -key ${key_file} -batch -out tmp.csr

# Generate the signed certificate.
rm -f ${cert_file}
PREFIX=vgl DN=Signed openssl x509 -passin pass:claudio -req -in tmp.csr -CA vgl-ca.pem -CAkey vgl-ca.key \
     -set_serial 0$N -out ${cert_file} -days 36500 -sha256

# cleanup.
rm -f tmp.csr
let N=N+1
echo ${N} > ca-serial-no.txt

Note - install a certificate for use requires both the certificate file “.pem” and the key file “.key”. The root certificate may need to be installed but the root key must never be deployed on a production machine. It is used only to generate the signed certificates that are deployed.

Common Issues

  • If the Portal page comes up completely blank, this likely due to “traffic_portal” not being able to connect to “traffic_ops”. Check that * “traffic_ops” host has port 443 open * The name of the “traffic_ops” host resolved correctly on the “traffic portal” host. * Check that “/etc/traffic_portal/conf/config.js” has been updated to have the “traffic ops” host.

  • If the initial admin login doesn’t work, verify the password tweaks have been done. If not, it will be necessary to make then changes and then reset the password for the “traffic_ops” user. This is most easily done by logging in to the Postgresql host with this command.

    psql --host 127.0.0.1 --user traffic_ops
    

    There should be a password prompt - use the password specified in the “traffic ops” post install.

    \password
    

    Enter the password (twice) when prompted. Then exit Postgresql with the following command.

    \q