Update and Maintenance

This guide helps to update and monitor your AURA installation.

Docker commands can be executed from any location. Docker Compose commands are to be called from the docker-compose.yml folder.

To learn about additional commands and arguments, consult the official Docker Compose command reference.

Upgrade to a new release

Update Containers

Before doing an update, check the Release Notes page, to learn what has been changed. For the case of an failed update, ensure you have backups of your data.

Follow these steps to update your services.

1. Switch to the new release

   Pull the aura repository:

$ git pull

list available releases:

$ git tag
  1.0.0-alpha1
  1.0.0-alpha2
  1.0.0-alpha3

and check out the new release:

$ git checkout tags/<release-version>

Replace <release-version> with any of the releases shown by git tag.

2. Update the configuration

Compare your .env file with the (potentially changed) sample.env and change your .env file accordingly. Take special note of new versions of the services. Carefully check the release notes for changes you might need to take into account. For aura-playout, you also want to compare the sample.engine-core.ini with your engine-core.ini.

3. Pull the new images from Docker Hub

$ docker compose pull

4. Re-create the containers

$ docker compose up -d

Logging and Monitoring

List running services

To get a list of services currently running use:

$ docker ps

Alternatively you can also run, giving additional details on the container composition:

$ docker compose ps

Logs

To continuously watch the logs of all services included in the current Docker Compose, type:

$ docker compose logs -f

To see the current logs of individual containers running with Docker Compose, execute:

$ docker compose logs -f --tail=100 <service-name>

This command prints the last hundred log lines, but also keeps watching for new entries.

If you prefer to access the logs directly, go to logs. Here you can find the log files for all services. Use tail to watch the log file:

$ tail -f -n100 <file>

The files in logs can be integrated to your favorite third-party monitoring software.

Prometheus Monitoring

In some future release, we will provide a native Prometheus Monitoring integration.

Resetting, shutting down and cleaning up

Enforce creation of new containers

docker compose up -d --force-recreate

This recreates containers even if their configuration and image have not changed.

Stop and remove containers

If you wish to remove your deployment, all you need to do is shutting it down:

docker compose down

If you also wish to delete all data, including the Docker Volumes, you can run the following command:

docker compose down -v

Delete unused images

Since Docker does not automatically delete old images, they may pile up and occupy to much space. So you should delete them once in a while.

To delete images of containers not currently running, use:

docker system prune --all

This will not delete the Docker volumes. Volume are the place, where the databases and the persistent data lives.

The --all option also removes unused images, not just dangling ones.

Delete all data

If you want a complete fresh installation, you may need to delete all data, avoiding any conflicts.

Be aware this deletes all volume data, including the ones not part of the AURA installation.

docker system prune --all --volumes

Backup Strategy

Include following filesystem locations to your backup system. Some of these are only available in certain deployment bundles, such as aura-web, aura-playout or aura-recorder.

Filesystem location	Deployment bundle	Description
aura/config	all	All custom and sample configuration files
aura/log	all	The reported log files
aura/audio	all	The audio store, holding all audio and playlists
/var/lib/docker/volumes	all	All Docker Volumes

Every Docker Volume is prefixed with the specific deployment bundle. Below you find commands for backing up all Docker Volumes.

For Database specific backups you can use the description from Upgrade the database version

Backup & Restore Docker Volumes

To backup all AURA related Docker Volumes, commands for backup and restore are provided.

To create a backup run:

sudo make aura-volume.backup BACKUP_PATH=/your/backup/path

This will create a tar.gz archive for each volume added with a timestamp, to your desired path.

To restore a volume backup run:

sudo make aura-volume.backup BACKUP_FILE=/your/backup.tar.gz VOLUME_NAME=aura-steering

Avoid data corruption, ensure volume is not in use

Before running the restore procedure, make sure the specific volume is not in use, otherwise you may end up with corrupted data!

Special Operations

Log into a container

To bash into an already running container execute:

docker compose exec -it steering bash

To log into the database of a PostgreSQL container you can run:

docker compose exec steering-postgres psql -U steering

Manage OpenID Clients

Open ID clients are used for authorized access to Steering.

For example Dashboard and Tank need to be registered as clients, in order to communicate with Steering.

Usually they are created during the initialization step of Aura Web. In some case you may want to update them, to fix an broken installation. Or you might want to register any 3rd party service at Steering.

Create OpenID Connect clients

To create an OpenID Connect client for Dashboard, update AURA_DASHBOARD_OIDC_CLIENT_ID in your .env and execute the following with the containers running:

$ docker compose exec steering poetry run ./manage.py create_oidc_client dashboard public --client-id "${DASHBOARD_OIDC_CLIENT_ID}" --client-secret "${DASHBOARD_OIDC_CLIENT_SECRET}" -r "id_token token" -u "${DASHBOARD_CALLBACK_BASE_URL}/oidc_callback.html" -u "${DASHBOARD_CALLBACK_BASE_URL}/oidc_callback_silentRenew.html" -p "${DASHBOARD_CALLBACK_BASE_URL}" -p "${DASHBOARD_CALLBACK_BASE_URL}/"

To create an OpenID Connect client for Tank, update AURA_TANK_OIDC_CLIENT_ID in your .env and execute the following with the containers running:

$ docker compose exec steering poetry run ./manage.py create_oidc_client tank confidential --client-id "${TANK_OIDC_CLIENT_ID}" --client-secret "${TANK_OIDC_CLIENT_SECRET}" -r "code" -u "${TANK_CALLBACK_BASE_URL}/tank/auth/oidc/callback"

These commands will fail if the client_id is already existing in Steering. In that case you need to delete the existing client first.

Delete OpenID Connect clients

If you need to replace an existing client_id, you can delete them in Steering’s administration interface at “Steering > OpenID Connect Provider > Clients”.

If you cannot access Steering’s administration interface, you can delete the clients by running:

$ docker compose exec steering poetry run ./manage.py delete_oidc_clients

This will delete the existing OpenID Connect clients for Dashboard and Tank.

You can then re-create as described above.

Renew schedules for a new year

By default, steering generates timeslots from the first date of a schedule, until either the last date of the schedule, the end of the year or defined number of days after the first date.

To renew the schedules for a new year you can run the addtimeslots management command, specifying the year you want to add timeslots for.

For example, for the year 2024 you execute:

$ docker compose exec steering poetry run ./manage.py addtimeslots 2024

This will add timeslots to the schedules that don’t have a “once” recurrence and where the last date is not set.

You can add the --dry-run option to see the results, without actually updating anything.

$ docker compose exec steering poetry run ./manage.py addtimeslots --dry-run 2024

Upgrade the database version

The postgres-version is saved in the POSTGRES_VERSION variable. When upgrading Postgres, it is not sufficient to change this version though. New major versions of Postgres cannot read the databases created by older major versions. The data has to be exported from a running instance of the old version and imported by the new version.

Thankfully, there is a Docker container available to automate this process. You can use the following snippet to upgrade your database in the volume aura-web_steering_db_data, keeping a backup of the old version in aura-web_steering_db_data_old:

# Replace "9.4" and "11" with the versions you are migrating between.
export OLD_POSTGRES=9.4
export NEW_POSTGRES=11
doker-compose stop steering-postgres
docker volume create aura-web_steering_db_data_new
docker run --rm \
  -v aura-web_steering_db_data:/var/lib/postgresql/${OLD_POSTGRES}/data \
  -v aura-web_steering_db_data_new:/var/lib/postgresql/${NEW_POSTGRES}/data \
  tianon/postgres-upgrade:${OLD_POSTGRES}-to-${NEW_POSTGRES}
# Add back the access control rule that doesn't survive the upgrade
docker run --rm -it -v aura-web_steering_db_data_new:/data alpine ash -c "echo 'host all all all trust' | tee -a /data/pg_hba.conf
# Swap over to the new database
docker volume create aura-web_steering_db_data_old
docker run --rm -it -v aura-web_steering_db_data:/from -v aura-web_steering_db_data_old:/to alpine ash -c "cd /from ; mv . /to"
docker run --rm -it -v aura-web_steering_db_data_new:/from -v aura-web_steering_db_data:/to alpine ash -c "cd /from ; mv . /to"
docker volume rm aura-web_steering_db_data_new

Please double check all values and that your local setup matches this. Of course this needs to be done for all postgres-dbs.

Automating Image Updates

You may want to have a testing instance, where you automatically update to the latest images.

In that case Watchtower can be used to keep the images up to date.

By adding the Watchtower service to your compose file will poll for new images every 60 seconds:

---
services:
  watchtower:
    command: --interval 60
    container_name: watchtower
    image: containrrr/watchtower:1.5.3
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Watchtower will stop and restart outdated containers automatically.

There is a sample Docker Compose file including Watchtower.

Overriding the docker-compose.yml

If you need to make changes to the docker-compose.yml you can create a docker-compose.override.yml and make the necessary adjustments in there. That way, your adjustments won’t create conflicts.

Deploy other Docker Images

If you prefer some individual deployment scenario, you can also run single Docker Images.

These Docker images are hosted on https://hub.docker.com/u/autoradio.

Work in progress

These images are not yet fully documented. We will update the documentation on Docker Hub as we move along. If you need such image please consult the documentation and Makefiles in the relevant repositories.