1. Some kind of monitoring software, like the Grafana stack. I like email and Discord notifications.
2. The Dockerfile will have a HEALTHCHECK statement, but in my experience this is pretty rare. Most of the time I set up a health check in the docker compose file or I extended the Dockerfile and add my own. You sometimes need to add a tool (like curl) to do the health check anyway.
3. It’s a feature of the container, but the app needs to support some way of signaling “health”, such as through a web API.
4. It depends on your needs. You can do all of the above. You can do so-called black box monitoring where you’re just monitoring whether your webapp is up or down. Easy. However, for a business you may want to know about problems before they happen, so you add white box monitoring for sub-components (database, services), timing, error counts, etc.

To add to that: health checks in Docker containers are mostly for self-healing purposes. Think about a system where you have a web app running in many separate containers across some number of nodes. You want to know if one container has become too slow or non-responsive so you can restart it before the rest of the containers are overwhelmed, causing more serious downtime. So, a health check allows Docker to restart the container without manual intervention. You can configure it to give up if it restarts too many times, and then you would have other systems (like a load balancer) to direct traffic away from the failed subsystems.

It’s useful to remember that containers are “cattle not pets”, so a restart or shutdown of a container is a “business as usual” event and things should continue to run in a distributed system.

When something doesn’t work, I do sudo docker ps lol.

I just put a healthcheck in my compose files and then run an autoheal container that will automatically restart them if they are “unhealthy”.

Dozzle will tell you just about everything you want to know about the health of a container. Sadly, to my knowledge, it does not integrate with any notification platforms like nfty, even though there is a long standing request for that feature.

i just let kubernetes handle it for me. k3s specifically.

I use uptimekuma with notifications through home assistant. I get notifications on my phone and watch. I had notifications set up to go to a room on my matrix homeserver but recently migrated it and don’t feel like messing with the room.

You can read the Dockerfile for the HEALTHCHECK clause however not all have it as it’s been introduced in later docker versions.

You can also write your own using things like curl.

What happened to grafana and Prometheus?

I have been putting off rebuilding my home cluster since moving but that used to be the default for much of this and I’m not hearing that in these responses.

I use Gatus in conjunction with http APIs for health checking. For services that don’t support that, you can always pattern match the HTML code.

I decided that at my scale, NixOS is easier to maintain. So for me its just a `systemctl status <thing I host>ˋ

2, no, just check the docs.

3, yup

You can make your own health checks in docker compose, so for instance, I had etcd running provided by another company, and I just set up the check in compose using the etcdctl commands (etcdctl endpoint health).

https://docs.docker.com/reference/compose-file/services/#healthcheck

I rely on the developers putting in a health check, but few do.

I’ve also got uptime kuma setup, which is kinda like an external healthcheck.

If I go to its web interface (because everything is a web interface) and it’s down, then I know it has a problem.

I could set up monitoring, but I wouldn’t care enough to fix it until I had free time to use it either.

How do you notify yourself about the status of a container?

I usually notice if a container or application is down because that usually results in something in my house not working. Sounds stupid, but I’m not hosting a hyper available cluster at home.

Is there a “quick” way to know if a container has healthcheck as a feature.

Check the documentation

Does healthcheck feature simply depend on the developer of each app, or the person building the container?

If the developer adds a healthcheck feature, you should use that. If there is none, you can always build one yourself. If it’s a web app, a simple HTTP request does the trick, just validate the returned HTML - if the status code is 200 and the output contains a certain string, it seems to be up. If it’s not a web app, like a database, a simple SELECT 1 on the database could tell you if it’s reachable or not.

Is it better to simply monitor the http(s) request to each service? (I believe this in my case would make Caddy a single point of failure for this kind of monitor).

If you only run a bunch of web services that you use on demand, monitoring the HTTP requests to each service is more than enough. Caddy being a single point of failure is not a problem because your caddy being dead still results in the service being unusable. And you will immediately know if caddy died or the service behind it because the error message looks different. If the upstream is dead, caddy returns a 502, if caddy is dead, you’ll get a “Connection timed out”

docker inspect --format='{{json .State.Health}}' <container_name>

HEALTHCHECK is part of the Dockerfile syntax and ought to be supported by all your container runtimes

https://docs.docker.com/reference/dockerfile/#healthcheck

You could extend all the dockerfiles that don’t have a health check to implement this feature with whatever health check makes sense for the application, even if for now it’s just a curl of an endpoint.