← All guides

Guide · Data orchestration

Monitor Dagster with a dead man's switch

Dagster's UI shows you a run once it happens. It can't tell you a schedule stopped firing, a sensor stalled, or the daemon died, because nothing runs to report the problem. A check-in from the job itself closes that gap.

1. Create a check

Create a check in Mortemain with the same period as your job's schedule (plus a sensible grace window), and copy its unique ping URL.

2. Ping with success and failure hooks

Dagster's @success_hook and @failure_hook decorators wrap an HookContext callback you attach to a job. Point them at the ping URL and /fail:

 jobs.py
from dagster import success_hook, failure_hook, HookContext, job, op
import requests

PING_URL = "https://ping.mortemain.com/your-check-uuid"

@success_hook(required_resource_keys=set())
def ping_success(context: HookContext):
    requests.get(PING_URL, timeout=10)

@failure_hook(required_resource_keys=set())
def ping_failure(context: HookContext):
    requests.get(PING_URL + "/fail", timeout=10)

@op
def extract():
    ...

@job(hooks={ping_success, ping_failure})
def nightly_etl():
    extract()

Job-level hooks run once per op, so a multi-op job pings ping_success several times on a clean run. That's harmless, Mortemain only needs one check-in per period, but it does mean duration or payload data from a single hook call won't represent the whole run.

3. Prefer one ping per run: a run status sensor

For a single check-in that reflects the whole run rather than each op, use a run_status_sensor scoped to the job instead of job hooks:

 sensors.py
from dagster import run_status_sensor, RunStatusSensorContext, DagsterRunStatus
import requests

PING_URL = "https://ping.mortemain.com/your-check-uuid"

@run_status_sensor(run_status=DagsterRunStatus.SUCCESS, monitored_jobs=[nightly_etl])
def nightly_etl_succeeded(context: RunStatusSensorContext):
    requests.get(PING_URL, timeout=10)

@run_status_sensor(run_status=DagsterRunStatus.FAILURE, monitored_jobs=[nightly_etl])
def nightly_etl_failed(context: RunStatusSensorContext):
    requests.get(PING_URL + "/fail", timeout=10)

Register both sensors in your Definitions and turn them on from the Dagster UI. Optionally add a third sensor on DagsterRunStatus.STARTED that calls PING_URL + "/start", so Mortemain can measure how long each run takes and flag ones that run unusually long.

Test it

Materialize the job once from the UI or CLI and watch the check flip to up. Then disable the schedule (or point the sensor at a job that will never run) and confirm the down alert arrives after the grace window, that's the failure mode Dagster's own logging won't surface.