Guide · Airflow

Monitor Apache Airflow with a dead man's switch

Airflow's own UI tells you a DAG failed, if the scheduler is up to notice. It won't tell you when the scheduler itself stalls, a worker queue backs up, or a DAG gets silently paused. A check-in from the DAG closes that gap: no run, no ping, you get alerted.

1. Create a check

Create a check in Mortemain with the same schedule as your DAG (plus a grace window for normal run time), and copy its ping URL.

2. Add success and failure callbacks to the DAG

Airflow calls on_success_callback once when every task in the run succeeds, and on_failure_callback when any task fails. Set both at the DAG level so one ping covers the whole run, not just a single task:

dags/nightly_etl.py

import requests
from airflow import DAG
from airflow.operators.python import PythonOperator

PING_URL = "https://ping.mortemain.com/your-check-uuid"

def notify_success(context):
    requests.get(PING_URL, timeout=10)

def notify_failure(context):
    requests.get(PING_URL + "/fail", timeout=10)

with DAG(
    dag_id="nightly_etl",
    schedule="0 3 * * *",
    on_success_callback=notify_success,
    on_failure_callback=notify_failure,
) as dag:
    # ... your tasks ...
    pass

Both callbacks receive the run's context, so you can also POST a short body (the DAG run ID, a task's log URL) as the check-in's payload if you want it on hand later.

3. No callbacks? Use a final BashOperator

If you'd rather keep the ping out of Python, add it as tasks instead. One pings on success, downstream of everything else; the other pings /fail and only runs if something upstream failed:

dags/nightly_etl.py

from airflow.operators.bash import BashOperator

PING_URL = "https://ping.mortemain.com/your-check-uuid"

ping_success = BashOperator(
    task_id="ping_mortemain_success",
    bash_command=f"curl -fsS -m 10 {PING_URL}",
    trigger_rule="all_success",
)

ping_failure = BashOperator(
    task_id="ping_mortemain_failure",
    bash_command=f"curl -fsS -m 10 {PING_URL}/fail",
    trigger_rule="one_failed",
)

extract >> transform >> load >> [ping_success, ping_failure]

The trigger_rule is what makes it work: all_success only fires when everything upstream passed, one_failed only fires when something didn't.

4. Track run duration (optional)

Ping /start from the first task so Mortemain can measure how long the DAG run actually took, useful for catching a job that's still "succeeding" but creeping slower every night:

dags/nightly_etl.py

ping_start = BashOperator(
    task_id="ping_mortemain_start",
    bash_command=f"curl -fsS -m 10 {PING_URL}/start",
)

ping_start >> extract >> transform >> load >> [ping_success, ping_failure]

Test it

Trigger a manual DAG run and confirm the check flips to up. Then fail a task on purpose (a bad import, a forced exception) and confirm the down alert arrives immediately, that's the failure mode a schedule-only view of Airflow can miss.