Market Pulse is a stock-market analytics lakehouse that runs entirely on a local machine: daily price data lands in an Iceberg bronze table, gets cleaned into silver, and rolls up into a gold analytics layer (returns, moving averages, volatility, volume-anomaly detection) computed in dbt-on-Trino SQL and orchestrated by Airflow. It's a data-engineering demonstration project, not a trading system: the market data comes from a free, delayed third-party feed, and nothing here is investment advice.
Key technologies
Apache Iceberg · MinIO · Trino · dbt · Airflow · Terraform · Superset
github.com/nhatminh06/market-pulse
Why I built a lakehouse
Most of the data pipelines I'd built before this were "read a CSV, write a table," which teaches you almost nothing about the actual hard problems in data engineering: what happens when you re-run an ingestion job over a window you already loaded, what happens when a source revises historical data, and how you catch a bad transformation before it becomes a bad dashboard. I wanted a project where those questions weren't optional.
Bronze, Silver, Gold
The medallion layers aren't just a naming convention here, each one has a different write pattern and a different job:
| Layer | Table | Write pattern |
|---|---|---|
| Bronze | bronze.prices | Delete rows in the fetched ticker/date window, then append the replacement batch |
| Silver | silver.stg_prices | Incremental merge on (ticker, trade_date) |
| Gold | gold.fct_daily_metrics | Full rebuild each run |
Bronze is raw, but not blindly appended: a delete-and-replace-partition pattern scoped to the exact window that was fetched, so re-running an ingest for the same days doesn't duplicate rows. Silver merges incrementally. Gold, where the rolling-window math lives, is cheap enough to just rebuild in full every run rather than reason about incremental correctness for window functions.
MinIO, Iceberg, Trino
MinIO gives an S3-compatible object store without a cloud account, which is the only reason this can run entirely local. Iceberg on top of that gives ACID writes, schema evolution, and time travel: the properties that make "just re-run it" actually safe instead of a manual reconciliation project. Trino is the query engine that ties dbt models to the Iceberg tables regardless of which layer they're reading or writing.
Why data tests, not just pipeline tests
A pipeline can run green and still produce wrong numbers: a join that fans out, a null that should have failed a filter, a currency or unit mismatch nobody caught. dbt tests catch a different class of bug than unit tests do: not-null, uniqueness, referential integrity between tables, accepted-value checks, and a custom test asserting a daily bar's low is never above its high. None of that requires knowing anything about the transformation logic; it just requires stating what should always be true about the data and failing loudly when it isn't.
On top of the dbt tests, a Python data-quality gate runs after the dbt build and re-checks the gold layer's invariants independently, with its check-interpretation logic unit-tested on its own so the gate itself isn't a silent single point of failure.
Fixtures and idempotent, incremental ingestion
Two separate ingestion modes exist: a full backfill from a start date, and an incremental run over the last N days, both hitting the same idempotent write path so re-running either one doesn't corrupt bronze. For CI, none of that touches live data or the real warehouse; the fast path is fixture-based: a small, deterministic dataset checked into the repo that exercises the same code paths without a network call or a live Trino cluster. That's what makes the tests fast enough to run on every pull request.
Why E2E verification matters more than a fixture pass
Fixture-based tests prove the code is internally consistent. They don't prove the real stack (a live Trino cluster, a real Iceberg catalog, actual SQL execution) behaves the way the fixtures assumed. Market Pulse runs a separate, manual/weekly end-to-end workflow that boots the full stack and runs the real dbt build and dbt test against live Trino, plus the quality gate's actual SQL checks, instead of trusting that "the fixtures passed" is the same claim as "the pipeline works." It isn't run on every PR because it's slower and needs the live stack, but it's the thing that actually verifies correctness end to end rather than in simulation.
Why I removed the unsupported performance claims
An earlier version of this project's documentation had a benchmark-shaped number in it: the kind of thing that looks great in a portfolio bullet and means nothing without a methodology behind it. I pulled it. A single favorable timing number, measured once, on one machine, under unknown conditions, isn't a benchmark; it's an anecdote wearing a benchmark's clothes. The honest version is a benchmark framework: a documented methodology, a way to reproduce the measurement, and a result you'd trust enough to defend if someone asked how you got it. I'd rather ship a project with no performance number than one with a number I can't stand behind.
What I learned about data engineering
- Idempotency is the real correctness bar for ingestion. "It loaded the data" is a much weaker claim than "it loads the data the same way no matter how many times you run it."
- Tests on data are a different discipline than tests on code. They're about invariants that should hold regardless of how the transformation is implemented.
- Fixtures and E2E verification answer different questions. Fast fixture tests keep CI usable; a slower, real-stack run is what actually proves the pipeline works.
- An unverified metric is worse than no metric. It costs credibility the moment someone asks you to reproduce it.
Closing
Market Pulse taught me that the hard part of data engineering isn't the SQL, it's the discipline around re-runs, invariants, and honest verification. A pipeline that "worked once" is not the same claim as a pipeline you'd trust with someone else's dashboard.