Service Watchdog and Automatic Restart (systemd WatchdogSec/Restart)
Description
Configures systemd service units with watchdog monitoring (WatchdogSec=) and automatic restart policies (Restart=on-failure or Restart=always, RestartSec=) to recover services that become unresponsive or are killed due to resource exhaustion. When a memory leak causes the OOM killer to terminate a service or the process hangs due to resource starvation, systemd automatically restarts the service from a clean state. The watchdog mechanism requires the service to periodically notify systemd (via sd_notify WATCHDOG=1); failure to notify within the WatchdogSec interval triggers a restart. Combined with cgroup memory limits (CME-704), this provides both containment and recovery — the cgroup bounds the leak, and systemd restarts the service when the bound is hit.
CVSS Vector Impacts
| Metric | Transition | Rationale |
|---|---|---|
| Availability (A) | H → L | Service recovers automatically from resource exhaustion conditions via systemd restart policy; availability impact is bounded by the restart interval (typically seconds) rather than mean-time-to-manual-recovery (minutes to hours). The watchdog timer detects service hangs caused by resource starvation even when the process has not been OOM-killed. |
CWE Relationships
Verification
Check that critical services have restart policies and watchdog timers configured
# Expected: Restart=on-failure (or always); WatchdogSec set to a positive value
# Expected: Critical services have Restart= and WatchdogSec= directives
Remediation
mkdir -p /etc/systemd/system/<service>.service.d
printf '[Service]\nWatchdogSec=30\nRestart=on-failure\nRestartSec=5s\nStartLimitIntervalSec=180\nStartLimitBurst=5\n' > /etc/systemd/system/<service>.service.d/watchdog.conf
systemctl daemon-reload
systemctl restart <service>
kubectl patch deployment <deployment> -n <namespace> --type=json -p '[{"op":"add","path":"/spec/template/spec/containers/0/livenessProbe","value":{"httpGet":{"path":"/healthz","port":8080},"initialDelaySeconds":10,"periodSeconds":30,"failureThreshold":3}}]'