# Container Monitoring and Management The following tasks and information will be useful as you continue through this guide. Familiarise yourself with the commands and information on this page. ## Controlling our `adsb` application ### Stopping the `adsb` application If you need to bring the environment down (for example, if you need to unplug the RTL-SDR USB dongle for maintenance), you can issue the command `docker compose down` from the application directory (the directory containing your `docker-compose.yml` file). ### Starting the `adsb` application To start the environment, or apply any changes made to your `docker-compose.yml` file, you can issue the command `docker compose up -d` from the application directory. ### Viewing container logs To "tail" the consolidated logs of all containers that make up the `adsb` application, you can issue the command `docker compose logs -f` from the application directory. Individual container logs can be "tailed" with `docker logs -f `. If you want to limit the output to, for example, the last 100 lines, you can add `--tail=100` to either of the logs commands above. If you want to search for a specific word or phrase in the output, you can add a `| grep ` to the end of the commands. ### Updating containers to latest version If you would like to manually update your containers to the latest versions of their images, you can run the following commands from the application directory: ```shell docker compose pull docker compose up -d ``` ### Updating container configuration If you need to update a container's configuration in `docker-compose.yml` or in the `.env` file, once complete, issue the command `docker compose up -d` (in the `docker-compose.yml` file's directory) and the affected containers will be recreated by `docker compose` to reflect the updated configuration. ### Start containers on system boot All of the containers defined within this document will be configured with the directive `restart: unless-stopped`. This will ensure the containers are automatically started if the host is rebooted unless you have manually stopped the container(s) previously. ## Information on Healthchecks Images can implement [healthchecks](https://docs.docker.com/engine/reference/builder/). A healthcheck is a script that docker runs within the container periodically that tells docker whether the container is operating as expected. For example, in the `ghcr.io/sdr-enthusiasts/docker-tar1090` container, the [healthcheck script](https://github.com/sdr-enthusiasts/docker-tar1090/blob/main/rootfs/healthcheck.sh) does the following: * For each expected network connection, make sure the connection exists * Make sure that messages are being received from the SDR * Make sure that the services running within the container aren't dying over and over for some reason If all of the checks above pass, the container is considered healthy. If any fail, the container is considered unhealthy. Earlier, we ran the command `docker ps`, to see our newly created `ultrafeeder` container up and running: ``` CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7b9c4be5a410 ghcr.io/sdr-enthusiasts/docker-adsb-ultrafeeder:latest "/init" 17 hours ago Up 17 hours (healthy) 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp ultrafeeder ``` Notice that next to the container status, there is some information about the container's health. This may be one of the following: * No health information: Not all images have healthchecks implemented. If an image doesn't report health, this is why. * `(health: starting)`: The container will wait up to a predefined start-period (defined in the [Dockerfile](https://github.com/sdr-enthusiasts/docker-tar1090/blob/main/Dockerfile#L179)) or until the healthcheck script returns a healthy result. * `(healthy)`: The container is operating as expected. * `(unhealthy)`: The container is not operating as expected. Later in this guide we will implement a method to automatically restart any unhealthy containers. This way, if your SDR "wedges", the container will eventually go unhealthy and be restarted. Where practical, we try to include healthchecks in all our images, so as you go through this guide and deploy more containers, you should see a health status whenever you issue the `docker ps` command. ### Reason for healthy/unhealthy? You can inspect the container for some (hopefully) meaningful output from the healthcheck script. If you issue the command `docker inspect ` (replacing `` with the name of the container you're interested in), you'll see lots of information about the container, including the output of the most recent run of the healthcheck script. This output is in JSON format, so with the help of the `jq` utility we can easily find our `ultrafeeder` container's most recent health information. First make sure `jq` is installed with `sudo apt install -y jq`, then we can check the `ultrafeeder` container's health with the following command: ```shell docker inspect ultrafeeder | jq .[0].State.Health.Log | jq .[-1].Output | awk '{gsub(/\\n/,"\n")}1' ``` Which will return something like this: ``` "readsb last updated: 1683575756.737, now: 1683575757.348255120, delta: .611255120. HEALTHY nginx deaths: 0. HEALTHY readsb deaths: 0. HEALTHY tar1090 deaths: 0. HEALTHY " ``` An `ExitCode` of `0` represents a healthy result. An `ExitCode` of `1` represents an unhealthy result. The first line is showing that we've received messages from the SDR in the past 15 minutes. The remaining lines show the number of times a service has had an "abnormal death" (crashed for some reason). ### Disabling Healthchecks On systems with with low spec CPU/memory, you may wish to disable healthchecks to gain back some precious CPU cycles. There are ways to disable the docker healthchecks in the `docker-compose.yml` file. However, versions of these images have been published that have the healthcheck removed, so it may be easier to simply change the image's `latest` tag to `latest_nohealthcheck`.