Maintenance and updates

Before doing an update, check the Release Notes page, to learn what has been changed.

Logging and Monitoring

To continuously watch the logs of all services included in the current Docker Compose, type

docker compose logs -f

To see the current logs of individual containers running with Docker Compose, execute

docker compose logs -f --tail=100 <service-name>

This command prints the last hundred log lines, but also keeps watching for new entries.

If you prefer to access the logs directly, go to aura/logs. Here you can find the log files for all services. Use tail to watch the log file:

tail -f -n100 <file>

The file in aura/logs can be integrated to your favorite third-party monitoring software.

Prometheus Monitoring

In some future release, we will provide a native Prometheus Monitoring integration.

Backup Strategy

/* to be defined */

Update Containers

For updates of the components containers:

  1. Pull the aura repository:

git pull

and check out the new release:

git checkout tags/<release-version>

  1. Compare your .env file with the (now updated) sample.env and change your .env file accordingly. Take special note of new versions of the components. For aura-playout: If you use the docker container of engine-core you also want to compare the sample.engine-core.ini with your engine-core.ini

  2. For the components having new versions: Check the release notes for changes you might need to take into account.

  3. Pull the new images from Docker Hub:

docker compose pull

  1. Recreate the containers:

docker compose up -d

Automating Image Updates

Watchtower can be used to keep the images up to date.

Adding the Watchtower service to docker-compose like this will poll every 60 seconds for new images.

It will then pull the new images, stop the outdated containers and restart the services.

...
services:
  watchtower:
    command: --interval 60
    container_name: watchtower
    image: containrrr/watchtower:1.5.3
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

This compose-file includes the Watchtower service.

Update the database by applying migrations

Manual steps for updating of the databases should only be necessary for steering. The other databases should be updated automatically by their services.

The Django migration scripts for steering are created during development and only need to be applied by the service to the database. This has to be done after updating the service.

To display any available migration run:

docker compose run --rm steering showmigrations

To apply the migrations simply run:

docker compose run --rm steering migrate

Create OpenID Connect clients

If you need to create a new or additional OpenID Connect client for dashboard, you can update AURA_DASHBOARD_OIDC_CLIENT_ID in your .env and run:

$ docker compose run --rm steering create_oidc_client.dashboard

If you need to create a new or additional OpenID Connect client for tank, you can update AURA_TANK_OIDC_CLIENT_ID in your .env and run:

$ docker compose run --rm steering create_oidc_client.tank

These commands will fail if the client_id is duplicated.

Upgrade the database version

The postgres-version is saved in the POSTGRES_VERSION variable. When upgrading Postgres, it is not sufficient to change this version though. New major versions of Postgres cannot read the databases created by older major versions. The data has to be exported from a running instance of the old version and imported by the new version.

Thankfully, there is a Docker container available to automate this process. You can use the following snippet to upgrade your database in the volume aura-web_steering_db_data, keeping a backup of the old version in aura-web_steering_db_data_old:

# Replace "9.4" and "11" with the versions you are migrating between.
export OLD_POSTGRES=9.4
export NEW_POSTGRES=11
doker-compose stop steering-postgres
docker volume create aura-web_steering_db_data_new
docker run --rm \
  -v aura-web_steering_db_data:/var/lib/postgresql/${OLD_POSTGRES}/data \
  -v aura-web_steering_db_data_new:/var/lib/postgresql/${NEW_POSTGRES}/data \
  tianon/postgres-upgrade:${OLD_POSTGRES}-to-${NEW_POSTGRES}
# Add back the access control rule that doesn't survive the upgrade
docker run --rm -it -v aura-web_steering_db_data_new:/data alpine ash -c "echo 'host all all all trust' | tee -a /data/pg_hba.conf
# Swap over to the new database
docker volume create aura-web_steering_db_data_old
docker run --rm -it -v aura-web_steering_db_data:/from -v aura-web_steering_db_data_old:/to alpine ash -c "cd /from ; mv . /to"
docker run --rm -it -v aura-web_steering_db_data_new:/from -v aura-web_steering_db_data:/to alpine ash -c "cd /from ; mv . /to"
docker volume rm aura-web_steering_db_data_new

Please double check all values and that your local setup matches this. Of course this needs to be done for all postgres-dbs.

Overriding the docker-compose.yml

If you need to make changes to the docker-compose.yml you can create a docker-compose.override.yml and make the necessary adjustments in there. That way, your adjustments won’t create conflicts.

Useful docker and docker compose commands

Here you can find the official documentation of docker compose commands.

Generally the commands are to be called from the folder in which the respective docker-compose.yml is in.

Below you find some useful examples.

List running services

To get a list of services currently running use:

docker ps

(Re)Create and start the containers

docker compose up -d

Starts all containers of services defined in the docker-compose.yml file of the folder the command is called from (and creates them if they didn’t exist before). If services, that are already running and were changed in any way, Docker Compose re-creates them.

The parameter “-d” means it starts them as daemons in the background.

Stop and remove services

If you wish to delete your deployment, all you need to do is shutting it down:

docker compose down

If you also wish to delete all data, including the Docker Volumes, you can run the following command:

docker compose down -v

Pull images from Docker Hub

To pull the newest images from Docker Hub:

docker compose pull

Delete unused images

Since Docker does not automatically delete old images it can happen, that too much space is used by them on the server. To delete images of containers not currently running, use:

docker system prune

This will not delete the Docker volumes (where the databases and therefore the persistent data lives).

Log into a container

To bash into an already running container execute

docker compose exec -it steering bash

To log into the database of a PostgreSQL container you can run

docker compose exec steering-postgres psql -U steering

Deploy other Docker Images

If you prefer some individual deployment scenario, you can also run single Docker Images.

These Docker images are hosted on https://hub.docker.com/u/autoradio.

Work in progress

These images are not yet fully documented. We will update the documentation on Docker Hub as we move along. If you need such image please consult the documentation and Makefiles in the relevant repositories.