data pipeline

Data Pipeline

Any automated process that moves data from where it's produced to where it's used — extracting, transforming, and loading along the way.

When you'd see it: Anywhere data shows up somewhere it didn't start. Pipelines run on schedules (every hour, every night), on triggers (a new file arrives), or continuously (event streams). When a dashboard is "broken," the pipeline behind it is usually what actually broke.

Why it matters: Pipelines are the plumbing the rest of the data world depends on. Reports, dashboards, ML models, customer-facing features — all of them assume the pipeline ran, ran on time, and produced clean output. Pipeline reliability is the unsexy foundation under every "data-driven" claim.

Common mistakes: Confusing "the pipeline ran" with "the data is correct." Pipelines can succeed technically while delivering wrong numbers — a missed source, a silent schema change, a bug in the transform. Data quality is its own discipline on top of pipeline reliability.

Related ETL, API