# Update and Maintenance

This guide helps to update and monitor your AURA installation.

Docker commands can be executed from any location. Docker Compose commands are to be called from
the `docker-compose.yml` folder.

To learn about additional commands and arguments, consult the
[official Docker Compose command reference](https://docs.docker.com/compose/reference/).

## Upgrade to a new release

### Update Containers

Before doing an update, check the [Release Notes](../release-notes.md) page, to learn what has been
changed. For the case of an failed update, ensure you have [backups](#backup-strategy) of your data.

Follow these steps to update your services.

1. **Switch to the new release**

   Pull the `aura` repository:

```properties
git pull
```

list available releases:

```properties
git tag
  1.0.0-alpha1
  1.0.0-alpha2
  1.0.0-alpha3
```

and check out the new release:

```properties
git checkout tags/<release-version>
```

Replace `<release-version>` with any of the releases shown by `git tag`.

2. **Update the configuration**

Compare your `.env` file with the (potentially changed) `sample.env` and change your `.env` file
accordingly. Take special note of new versions of the services. Carefully check the release notes
for changes you might need to take into account. For `aura-playout`, you also want to compare the
`sample.engine-core.ini` with your `engine-core.ini`.

3. **Pull the new images from Docker Hub**

```properties
docker compose pull
```

4. **Re-create the containers**

```properties
docker compose up -d
```

## Logging and Monitoring

### List running services

To get a list of services currently running use:

```properties
docker ps
```

Alternatively you can also run, giving additional details on the container composition:

```properties
docker compose ps
```

### Logs

To continuously watch the logs of all services included in the current _Docker Compose_, type:

```properties
docker compose logs -f
```

To see the current logs of individual containers running with _Docker Compose_, execute:

```properties
docker compose logs -f --tail=100 <service-name>
```

This command prints the last hundred log lines, but also keeps watching for new entries.

If you prefer to access the logs directly, go to `logs`. Here you can find the log files
for all services. Use `tail` to watch the log file:

```properties
tail -f -n100 <file>
```

The files in `logs` can be integrated to your favorite third-party monitoring software.

```{admonition} Prometheus Monitoring
:class: tip

In some [future release](https://gitlab.servus.at/aura/aura/-/issues/98), we will provide a native [Prometheus Monitoring](https://prometheus.io/) integration.
```

## Resetting, shutting down and cleaning up

### Enforce creation of new containers

```properties
docker compose up -d --force-recreate
```

This recreates containers even if their configuration and image have not changed.

### Stop and remove containers

If you wish to remove your deployment, all you need to do is shutting it down:

```properties
docker compose down
```

If you also wish to delete all data, including the Docker Volumes, you can run the following
command:

```properties
docker compose down -v
```

### Delete unused images

Since Docker does not automatically delete old images, they may pile up and occupy to much space.
So you should delete them once in a while.

To delete images of containers not currently running, use:

```properties
docker system prune --all
```

This will <u>not delete the Docker volumes</u>. Volume are the place, where the databases and the
persistent data lives.

The `--all` option also removes unused images, not just dangling ones.

### Delete all data

If you want a complete fresh installation, you may need to delete all data, avoiding any conflicts.

Be aware <u>this deletes all volume data</u>, including the ones not part of the AURA installation.

```properties
docker system prune --all --volumes
```

## Backup Strategy

Include following filesystem locations to your backup system. Some of these are only available in
certain deployment bundles, such as `aura-web`, `aura-playout` or `aura-recorder`.

| Filesystem location     | Deployment bundle | Description                                      |
| ----------------------- | ----------------- | ------------------------------------------------ |
| aura/config             | all               | All custom and sample configuration files        |
| aura/log                | all               | The reported log files                           |
| aura/audio              | all               | The audio store, holding all audio and playlists |
| /var/lib/docker/volumes | all               | All Docker Volumes                               |

Every Docker Volume is prefixed with the specific deployment bundle. Below you find commands for
backing up all Docker Volumes.

For Database specific backups you can use the description from `Upgrade the database version`

### Backup & Restore Docker Volumes

To backup all AURA related Docker Volumes, commands for backup and restore are provided.

To create a backup run:

```properties
sudo make aura-volume.backup BACKUP_PATH=/your/backup/path
```

This will create a `tar.gz` archive for each volume added with a timestamp, to your desired path.

To restore a volume backup run:

```properties
sudo make aura-volume.backup BACKUP_FILE=/your/backup.tar.gz VOLUME_NAME=aura-steering
```

```{admonition} Avoid data corruption, ensure volume is not in use
:class: alert
Before running the restore procedure, make sure the specific volume is not in use, otherwise you may end up with corrupted data!
```

## Special Operations

### Log into a container

To _bash_ into an already running container execute:

```properties
docker compose exec -it steering bash
```

To log into the database of a PostgreSQL container you can run:

```properties
docker compose exec steering-postgres psql -U steering
```

### Manage OpenID Clients

Open ID clients are used for authorized access to _Steering_.

For example _Dashboard_ and _Tank_ need to be registered as clients, in order to communicate with
_Steering_.

Usually they are created during the initialization step of _Aura Web_. In some case you may want to
update them, to fix an broken installation. Or you might want to register any 3rd party service at
_Steering_.

#### Create OpenID Connect clients

To create an OpenID Connect client for Dashboard, update `AURA_DASHBOARD_OIDC_CLIENT_ID` in your
`.env` and execute the following with the containers running:

```properties
docker compose exec steering poetry run ./manage.py create_oidc_client dashboard public --client-id "${DASHBOARD_OIDC_CLIENT_ID}" --client-secret "${DASHBOARD_OIDC_CLIENT_SECRET}" -r "id_token token" -u "${DASHBOARD_CALLBACK_BASE_URL}/oidc_callback.html" -u "${DASHBOARD_CALLBACK_BASE_URL}/oidc_callback_silentRenew.html" -p "${DASHBOARD_CALLBACK_BASE_URL}" -p "${DASHBOARD_CALLBACK_BASE_URL}/"
```

To create an OpenID Connect client for Tank, update `AURA_TANK_OIDC_CLIENT_ID` in your `.env`
and execute the following with the containers running:

```properties
docker compose exec steering poetry run ./manage.py create_oidc_client tank confidential --client-id "${TANK_OIDC_CLIENT_ID}" --client-secret "${TANK_OIDC_CLIENT_SECRET}" -r "code" -u "${TANK_CALLBACK_BASE_URL}/tank/auth/oidc/callback"
```

These commands will fail if the `client_id` is already existing in Steering. In that case you need
to delete the existing client first.

#### Delete OpenID Connect clients

If you need to replace an existing `client_id`, you can delete them in Steering's administration
interface at "_Steering &gt; OpenID&nbsp;Connect&nbsp;Provider &gt; Clients_".

<img src="../_static/screenshots/admin/oidc_provider-client.png" />

If you cannot access Steering’s administration interface, you can delete the clients by running:

```properties
docker compose exec steering poetry run ./manage.py delete_oidc_clients
```

This will delete the existing OpenID Connect clients for Dashboard and Tank.

You can then re-create as described [above](#create-openid-connect-clients).

### Renew schedules for a new year

By default, `steering` generates timeslots from the first date of a schedule,
until either the last date of the schedule, the end of the year or defined
number of days after the first date.

To renew the schedules for a new year you can run the `addtimeslots` management
command, specifying the year you want to add timeslots for.

For example, for the year 2024 you execute:

```properties
docker compose exec steering poetry run ./manage.py addtimeslots 2024
```

This will add timeslots to the schedules that don’t have a "once" recurrence
and where the last date is not set.

You can add the `--dry-run` option to see the results, without actually
updating anything.

```properties
docker compose exec steering poetry run ./manage.py addtimeslots --dry-run 2024
```

### Upgrade the database version

The postgres-version is saved in the `POSTGRES_VERSION` variable. When upgrading Postgres, it is
not sufficient to change this version though. New major versions of Postgres cannot read the
databases created by older major versions. The data has to be exported from a running instance
of the old version and imported by the new version.

Thankfully, there is a Docker container available to automate this process. You can use the
following snippet to upgrade your database in the volume `aura-web_steering_db_data`, keeping a
backup of the old version in `aura-web_steering_db_data_old`:

```properties
# Replace "9.4" and "11" with the versions you are migrating between.
export OLD_POSTGRES=9.4
export NEW_POSTGRES=11
doker compose stop steering-postgres
docker volume create aura-web_steering_db_data_new
docker run --rm \
  -v aura-web_steering_db_data:/var/lib/postgresql/${OLD_POSTGRES}/data \
  -v aura-web_steering_db_data_new:/var/lib/postgresql/${NEW_POSTGRES}/data \
  tianon/postgres-upgrade:${OLD_POSTGRES}-to-${NEW_POSTGRES}
# Add back the access control rule that doesn't survive the upgrade
docker run --rm -it -v aura-web_steering_db_data_new:/data alpine ash -c "echo 'host all all all trust' | tee -a /data/pg_hba.conf
# Swap over to the new database
docker volume create aura-web_steering_db_data_old
docker run --rm -it -v aura-web_steering_db_data:/from -v aura-web_steering_db_data_old:/to alpine ash -c "cd /from ; mv . /to"
docker run --rm -it -v aura-web_steering_db_data_new:/from -v aura-web_steering_db_data:/to alpine ash -c "cd /from ; mv . /to"
docker volume rm aura-web_steering_db_data_new
```

Please double check all values and that your local setup matches this. Of course this needs to
be done for all postgres-dbs.

### Automating Image Updates

You may want to have a testing instance, where you automatically update to the latest images.

In that case [Watchtower](https://containrrr.dev/watchtower/) can be used to keep the images up
to date.

By adding the Watchtower service to your compose file will poll for new images every 60 seconds:

```yaml
---
services:
  watchtower:
    command: --interval 60
    container_name: watchtower
    image: containrrr/watchtower:1.5.3
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
```

Watchtower will stop and restart outdated containers automatically.

There is a sample [Docker Compose file including Watchtower](https://gitlab.servus.at/aura/aura/-/blob/main/config/aura-web/docker-compose.watchtower.yml).

### Overriding the docker-compose.yml

If you need to make changes to the `docker-compose.yml` you can create a
`docker-compose.override.yml` and make the necessary adjustments in there. That way, your
adjustments won't create conflicts.

### Deploy other Docker Images

If you prefer some individual deployment scenario, you can also run single Docker Images.

These Docker images are hosted on [https://hub.docker.com/u/autoradio](https://hub.docker.com/u/autoradio).

```{admonition} Work in progress
:class: alert
These images are not yet fully documented. We will update the documentation on Docker Hub as we
move along. If you need such image please consult the documentation and _Makefiles_ in the
relevant repositories.
```