Automating Daily Data Pipelines with Apache Airflow: A Tutorial

October 09, 2026

A daily data load sounds simple. Pull some rows, clean them, push them into a warehouse, repeat at 04:00. Then the source API times out at 03:58, the job dies quietly, and someone in finance spots the stale numbers before you do. Apache Airflow exists to make that failure visible, repeatable and easy to fix — and it takes less time to set up than most people expect.

What Airflow Actually Does

Airflow is an orchestrator, not a data processing engine. It decides when things run and in what order; the work itself happens in whatever Airflow triggers — a Python function, a SQL statement, a Docker container, a call to an external API.

Everything you build lives inside a DAG, a directed acyclic graph. The "acyclic" part matters: tasks can depend on each other, but nothing loops back on itself. A daily pipeline might fetch source data, validate the row count, load it into the warehouse, then post a summary to a channel. Four boxes, three arrows, one run a day. In code that reads as fetch >> validate >> load >> notify, and Airflow will not start load until validate has finished successfully.

That is the part a cron job cannot do. Cron knows nothing about dependencies, nothing about retries, and nothing about what happened last Tuesday at 04:00. Airflow records every attempt in its metadata database, so "did the load run, and what did it say?" becomes a question you can answer in ten seconds instead of an afternoon spent grepping log files.

Underneath, a scheduler decides what is ready to run, workers run it, the metadata database records every attempt, and the web interface shows you what happened. For local development, SQLite and the built-in executor are perfectly adequate.

Getting a Local Instance Running

Install into a virtual environment so you don't end up fighting your system Python. The quickest route is pip install apache-airflow, followed by airflow standalone. That creates a metadata database in ~/airflow, starts the scheduler and web server, and prints the URL along with a generated admin password. On port 8080 you get the UI: the DAG list, run history, and task logs.

The standalone command also ships with example DAGs, and they are worth fifteen minutes of clicking. They show what a finished, working pipeline looks like — dependencies, retries, templated dates — before you write your own. Once you are comfortable, set load_examples = False in airflow.cfg to keep the interface uncluttered.

For anything beyond a first experiment, use the official Docker Compose setup from the Airflow documentation. It gives you Postgres, a scheduler, a web server and a worker, already wired together. Pin your Airflow version while you're at it — upgrades occasionally require a database migration, and you don't want that happening by accident on a Tuesday afternoon.

Building the DAG, Task by Task

DAG files are Python, stored in a folder the scheduler scans. Name the pipeline, anchor it to a start date, and give it a schedule: DAG("daily_orders", start_date=..., schedule="0 4 * * *", catchup=False). That cron string means 04:00 every day; @daily is the shorthand if the hour doesn't matter.

Then define the tasks. The TaskFlow API keeps this readable: decorate a function with @task, return a value, and pass it downstream to the next one.

Extract. Pull yesterday's orders from the source. Use the DAG's logical date rather than datetime.now(), so a rerun fetches exactly the same window.
Validate. Compare the row count against a sensible range and fail loudly if it is zero or absurdly high. A pipeline that quietly loads nothing is worse than one that crashes.
Load. Write to a staging table, then merge into the target on a stable key.
Transform. Run the SQL or your transformation tool. A database is usually a better place for this than a worker process.
Notify. Post a one-line summary of rows loaded to a channel people actually read.

Chain them with extract >> validate >> load, and Airflow handles the ordering, retries and logging for you.

Scheduling, Catchup and Idempotency

Two settings cause most first-week pain. The first is catchup, which Airflow enables by default: it will run every interval between your start date and now. Set a start date of January 2023 and you'll return to thousands of queued runs. Leave catchup=False while you're learning, and backfill deliberately once the pipeline is trusted.

The second is idempotency. A rerun of the same date should produce the same result, not double it. Appending an extra day to a table with an unconditional INSERT turns your revenue figure into fiction. Practical fixes:

Delete the target partition for the logical date, then insert.
Use a MERGE or UPSERT on a stable key.
Tag rows with a run identifier so repeated attempts overwrite rather than append.

Also use timezone-aware start dates. Airflow stores times in UTC and converts for display, and naive datetimes produce confusing off-by-one-hour runs when the clocks change.

Watching It Run

The Grid view is where you'll spend most of your time: each column is a run, each row a task. A green square means success, red means failure, and the outline around a column marks the current schedule interval. Click any square and you get the logs, the duration, the retry count, and a button to clear that task and run it again without rerunning the whole DAG.

Failures are where Airflow earns its keep. Give every task a sensible retries value — two or three is usually plenty — plus a retry_delay measured in minutes, then set email_on_failure or point an alerting callback at the chat tool or paging service your team already watches. A transient timeout at 03:58 then becomes a task that succeeded on the second attempt, not an awkward phone call.

Do not retry blindly, though. A validation task that failed because the source returned zero rows will fail again just as fast, three times in a row. Retries are for flaky networks and locked tables; logic errors need a human.

Testing Before You Turn It Loose

You can run a single task without touching the scheduler at all: airflow tasks test daily_orders extract 2024-06-01 executes the extract function for that logical date and prints the result straight to your terminal. It is the fastest loop for debugging a new task, and it leaves no trace in the run history.

Keep the real logic in ordinary Python functions and test them with pytest. A function that takes a list of rows and returns a cleaned list is trivial to test; the same logic buried inside an operator is not. Thin DAG files that call well-tested modules are easier to review, faster to debug, and portable if you ever move off Airflow.

From Laptop to Production

When the pipeline starts mattering to other people, a few things change:

Move the metadata database to Postgres. SQLite copes badly with a scheduler and a web server writing at the same time.
Keep DAG files in version control and deploy them like any other code, ideally through CI.
Consider a managed runtime — Cloud Composer, MWAA, Astronomer — if you would rather not operate a scheduler yourself.
Watch the scheduler's own metrics. A DAG that takes longer to parse than its schedule interval is a problem nobody notices until the queue backs up.

Keep Secrets Out of the DAG

Hardcoding an API key in a DAG file is the quickest way to leak it. Airflow has a place for that. Connections store hostnames, logins and passwords under a name and encrypt them in the metadata database; Variables hold smaller pieces of configuration, such as a threshold or a channel name, that you would rather change without redeploying code.

Push the Heavy Work Downstream

Airflow is at its best when it starts something and checks on it. Loading ten million rows through a Python worker is slow, memory-hungry and awkward to retry. Doing it with INSERT INTO ... SELECT inside the warehouse takes seconds and lets the database do what it is built for.