Skip to content
SHC Docs

Restarting what is unhealthy

This page decides what happens to a replica whose healthcheck keeps failing. A check only reports it; the node restarts it once it has been unhealthy for long enough, five minutes unless the service says otherwise. x-shc-health.restart_after is that time for one service.

Give web thirty seconds: a page that has not answered for half a minute is better restarted than waited on.

~/learn/guestbook/compose.yaml
services:
web:
image: nginx:1.27-alpine
expose:
- "80"
volumes:
- ./site:/usr/share/nginx/html:ro
healthcheck:
test: ["SHC-HTTP", ":80/"]
interval: 10s
timeout: 3s
retries: 3
deploy:
replicas: 2
x-shc-primary: true
x-shc-health:
restart_after: 30s
api:
image: traefik/whoami:v1.10
command: ["--port", "8080", "--name", "guestbook-api"]
expose:
- "8080"
healthcheck:
test: ["SHC-HTTP", ":8080/"]
interval: 10s
timeout: 3s
retries: 3
x-shc-path: /api
x-shc-path-strip: true
counter:
image: redis:7-alpine
command: ["redis-server", "--appendonly", "yes"]
expose:
- "6379"
volumes:
- data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 12
x-shc-scale:
max: 1
volumes:
data:
home01
$ shc update guestbook
...

Now make one replica stop answering without stopping. pkill -STOP freezes every nginx process in its container. A frozen process still exists, so the container stays up while every request to it hangs. That is the failure a healthcheck is for.

home01
$ shc exec -s guestbook -S web -- pkill -STOP nginx

Three failed checks ten seconds apart make it unhealthy, and thirty seconds later its node restarts it. The node says so in its journal.

home01
$ journalctl --no-pager -u shc -g X201026CTRRST -o cat -n 1
...

A restart starts nginx afresh, and its healthcheck passes again.

home01
$ shc status -s guestbook
...
prod guestbook prod web-r0 home01 ready
...

Next: The same app in shc.yaml.