Implement health check for the index application

[josecelano]

In the composer.yaml file, we have the definition for the backend service. The healthcheck still needs to be implemented.

# todo: implement healthcheck
#healthcheck:
#  test:
#    [
#      "CMD-SHELL",
#      "cargo run healthcheck"
#    ]
#  interval: 10s
#  retries: 5
#  start_period: 10s
#  timeout: 3s

For that reason, workflows take longer than they should. In the script to run E2E tests, we wait 20 seconds instead of waiting only until the container is healthy.

wait_for_container_to_be_healthy torrust-mysql-1 10 3
# todo: implement healthchecks for tracker and backend and wait until they are healthy
#wait_for_container torrust-tracker-1 10 3
#wait_for_container torrust-idx-back-1 10 3
sleep 20s

See: https://github.com/torrust/torrust-index-backend/blob/develop/docker/bin/run-e2e-tests.sh#L47-L51

Proposal 1

Add a new endpoint tho the API: GET /status that should return a 200 code is the app is ready to process requests. It could contains a json body with more verbose report like:

{ 
  "status": "Ok",
  "database": "Ok",
  "tracker": "Ok"
}

When something is wrong could return something like:

{ 
  "status": "Error",
  "error": "Cannot connect to database",
}

We can define this later. For now we just check that return a 200 code.

The healthcheck could be something like

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:3030/health"]
  interval: 30s
  timeout: 10s
  retries: 3

Links

• Docker Healthchecks: Why Not To Use curl or iwr
• https://stackoverflow.com/a/68822241/3012842

[josecelano]

Hi @da2ce7 I saw you added this line to the Containerfile:

# HEALTHCHECK CMD ["/usr/bin/wget", "--no-verbose", "--tries=1", "--spider", "localhost:${API_PORT}/version"]

After reading the article "Docker Healthchecks: Why Not To Use curl or iwr" I'm thinking about an alternative solution without using curl or wget.

We could add a new binary, src/bin/health_check. We can call that binary. However the new binary has to check if the main process is healthy. What I do not like about the healthcheck.js example in the article is that it says you should not expose the endpoint but is calling an endpoint anyway. I think calling an endpoint localhost:${API_PORT}/version or localhost:${API_PORT}/health is the easiest way to implement communication between both processes.

Maybe the trick is that the endpoint is only exposed locally. With docker the API socket should be bound to the wildcard address (0.0.0.0), so it should work with localhost. Maybe we can check in the Axum handler that the request is from the loopback IP.

On the other hand, I like the idea of exposing a public localhost:${API_PORT}/health endpoint. Because we can quickly implement smoke tests after deployments. But exposing a public endpoint is always a risk. I see it like this Twitter accounts that show you the status of a public service.

What do you think? I would like to implement this feature (the most basic one) because sometimes tests fail because a container image was not started correctly, and you waste a lot of time trying to figure out why the test failed. I want the pipeline to stop immediately if any of the containers could not be started correctly.

[josecelano]

I've just realized that the containerized application has more than one service:

• The Index REST API.
• The tracker statistics importer: a cronjob to import the tracker torrents statistics.

So the healthcheck should ensure both services are OK.

In the future, I think we should refactor that part:

• We should remove the importer from the API binary
• We should launch two independent containers, each one with its own binary, even if most of the Containerfile is the same. In production but services could be deployed independently.

but before doing that maybe we will need to extract the common logic for both services. The "API" and the "importer" share a common core "Index" package.

For the time being, just keep in mind the HEALTCHECK should not be only for the API.