Airflow DAG for monthly exports
The monthly category export as a production Airflow DAG — idempotent, resumable via the export id, with the estimate step as a cost gate.
← All recipesData API
The problem
Our data platform runs on Airflow; I want Magpie exports scheduled, retried and observable like every other pipeline.
How it works
- Task 1 estimates the export and fails the run if the cost exceeds the configured ceiling.
- Task 2 submits and stores the export_id in XCom, so retries re-poll instead of re-buying.
- Task 3 polls until ready and lands the files in your object store.
A taste of the code
est = exports.estimate(**SCOPE)
if est.credits > MAX_CREDITS:
raise AirflowFailException(f"Export would cost {est.credits} credits")
job = exports.submit(**SCOPE, format="parquet")
ti.xcom_push(key="export_id", value=job.export_id)The full, runnable pipeline — argument parsing, retries, partial-result handling, delivery to S3/GCS — is in the repo: open on GitHub →
What it costs
Same as the plain monthly export — the DAG's estimate gate just makes the spend a reviewable config value.
Estimates are always free — see how credits work.