ntfy is a minimal, self-hostable HTTP push notification service. Send an HTTP POST to a topic URL, and any subscribed client receives a push notification. No account required for basic use, no infrastructure complexity.
ntfy is the single alerting bus for all failure conditions across the home lab infrastructure. A Fail2Ban ban, a systemd service crash, a backup failure, an rsyslog staleness event — all route through ntfy to the same phone.
What ntfy Is#
ntfy is a service that listens on an HTTP endpoint. Any application can POST a JSON payload to https://ntfy.sh/<topic> (or self-hosted at https://yourserver/<topic>) with a message and optional title. Any device (phone, desktop, browser tab) can subscribe to that topic and receive push notifications whenever a message is published.
There's no authentication layer for public topics on ntfy.sh. If you know the topic name, you can post to it. If you post to a public topic that someone else has guessed, they see your messages. This is not a security issue for monitoring alerts — alerts are not sensitive data — but it's worth understanding.
For sensitive workloads, run ntfy on a private server (like the Pi) that only internal services can reach.
Fail2Ban Integration#
Fail2Ban monitors /var/log/auth.log and /var/log/samba/log.* for authentication failures. When a threshold is hit (e.g., 5 failures in 10 minutes), Fail2Ban issues a ban — it adds the offending IP to an ipset or firewall rule, blocking future attempts.
A custom action file at /etc/fail2ban/action.d/nftables-ntfy.conf runs a curl POST to ntfy when a ban is issued:
curl -d "Fail2Ban Ban: $(<IP>) in $(<JAIL>)" https://ntfy.sh/<your-topic>
The message includes the banned IP and the jail name. Knowing when a ban expires matters — if the same IP gets banned three times in a day, that's a suspicious pattern worth investigating. ntfy alerts on both ban and unban events, so you see the full lifecycle.
systemd OnFailure Hooks#
A script /usr/local/bin/notify-failure.sh is called by OnFailure= directives in systemd service overrides for critical services: Technitium DNS, Samba, Chrony NTP, and rpi-connect. If any of these services crash and the restart limit is exceeded, the script fires an ntfy alert before you notice the outage.
Backup Monitoring#
The daily pi-backup.sh script runs on a systemd timer. The timer has an OnFailure= override that calls ntfy if the backup script exits with a non-zero status. A failed backup is silent by default — you'd only notice when you try to recover from the backup and find it's corrupt. ntfy makes backup failures loud.
rsyslog Staleness Monitoring#
A cron job runs every 5 minutes checking whether /var/log/ucgfiber.log (the UCG Fiber syslog feed) has been written to in the last 5 minutes. If the log goes stale — the gateway syslog service stopped sending, rsyslog crashed, the network link to the gateway died — ntfy fires.
This catches a class of failures that no service health check would detect. The gateway is running and responding to ping, but its logs have gone silent. The silence is itself a symptom.
The Principle: Silence Is Failure#
If something is important enough to run, it's important enough to alert on failure. If you don't care when it breaks, it doesn't need to be running — remove it.
ntfy makes alerting cheap. An HTTP POST is trivial. There's no infrastructure cost. A $5/month VPS running ntfy can handle thousands of alerts. There's no reason not to alert comprehensively.
The result: you find out about failures in minutes, not hours. You fix them before they compound into bigger problems.