• Consistent correct answers and hallucinations

• Low-latency answers and waiting forever

• Maxing out your token limit in a day and being able to use a cheaper model

In this article, we will go over how data modeling is changing in the age of AI and what you need to know

• How data engineers are drifting toward a new role: context engineers

• One Big Table vs Dimensional Modeling vs Data Vault vs AI-native Modeling

  • Each of these techniques has tradeoffs, and some work better than others for AI

The Rise of the Context Engineer

Data engineering is not about Spark and Airflow like it was in 2020. The roles that maintain pipelines and optimize queries are starting to slow down. (Although getting into Data Engineering in 2026 is still very possible)

Companies will start realizing that data engineers can do so much more. AI allows a shift from analytics-oriented architecture to action-oriented architectures.

Enabling action-oriented architectures will be the main duty of data engineers in the coming age. This means they need to be:

• More deeply connected to the business since decisions happen fast. They need to take on both the analyst and data engineer roles to keep up!

• More well-versed with concepts like Change Data capture and Kafka streams, because data will come to LLMs faster

• Well-versed in handling unstructured data because documents are often critical in the decision-making processes of LLMs!

• They need to sharpen their system design skills to handle the three Vs of big data: velocity, variety, and volume!

All of this will allow for AI to have all the right context at the right time to make genuinely real-time decisions for the business!

Since this is more than just data, the more accurate title is:

Context Engineer

One Big Table (OBT) vs Dimensional Modeling vs AI-native Modeling

Getting the right context for your AI is complicated because:

• Too little context and your AI will make stuff up

• Too much context and your AI will get confused and burn your token budget

• You truly need the Goldilocks zone of context to maximize your ROI from AI

One Big Table (OBT) is the newest kid on the block. It puts all your columns into a single table. This feels kind of insane, but it has quite a few benefits too.

• AI really struggles with generating JOINs between tables, as is done in dimensional data modeling. This allows OBT to shine because getting the context is usually a simple WHERE clause without complex SQL generation. (As much as Databricks likes to lie to us, text-to-SQL models are still pretty bad)

• OBT’s biggest strength is also its biggest weakness. You get ALL the context. Not just the RELEVANT context.

Dimensional Modeling (also called Kimball data modeling) is the “old reliable” version of data warehouse modeling.

• Dimensional modeling’s strengths are OBT’s weaknesses and vice versa. Dimensional modeling allows you to get THE RIGHT context. But often relies too heavily on text-to-SQL to get it right.

• If you want your dimensional models to be more AI-effective, you need to:

  • Properly model your columns

    • Dimension columns are labeled as such (e.g., dim_name)

    • Fact/measure columns are labeled as such (e.g., m_revenue)

    • Column comments and table comments were rarely used by data engineers before 2023. Now they are CRITICAL for text-to-SQL models for generating and reasoning about the right values to give to the AI.

The real question is: can we have a modeling technique that gives us the retrieval simplicity of One Big Table with the “just the right amount of context” of dimensional modeling?

Enter: AI-native modeling, the question-first orientation

AI thrives on the right context at the right time.

Context engineers who summarize and create data marts on top of OBT models will succeed for many reasons.

• AI thrives at mapping “question” → “appropriate data set.”

  • “What has happened with churn recently?” cleanly maps the dataset “customer_churn_detail.”

  • “Who recently bought our product” cleanly maps to the dataset “customer_purchase_detail.”

These new AI-native aggregations will help dramatically reduce token cost by processing this context with normal tools like Spark and BigQuery.

Remember, always use the right tool for the right job. This applies to data modeling as well! Both dimensional modeling and OBT are wrong in the age of AI. But they both support the downstream models very well!

What hits the spot in this article? How will you change your data modeling from here? Are you excited to be a context engineer in the future?

In the next edition of the DataExpert.io blog, we will cover unstructured data and how it fits into all of this with embeddings and RAG retrieval!

At DataExpert.io, you can get an OpenAI key, an Anthropic Key, a production Databricks account, and a production Snowflake account to build the portfolio project of your dreams for just $97/month!

Read more

But if you zoom out and look at the product direction over the last few years, a much different pattern emerges.

Read more

If AI can generate pipelines, DAGs, tests, and even migrations…
What’s left for data engineers to actually work on?

Conceptual knowledge is no …

Read more

In this article, we will attack the following beliefs:

• Only Spark can be used for terabytes of data (or it is ALWAYS the best choice)

• You need a lot of time to process TBs of data

We want to leave your head spinning at the end of this article. Wondering if everything you learned about MapReduce was wrong!

This Article is brought to you by

We want to give a shout-out to MotherDuck, who is sponsoring this article and providing the infrastructure for the benchmarks!

You Said to Use DuckDB On Small Data

Previously, I was a champion of using DuckDB for any dataset that was “small” (< 20GBs). Recently, I was challenged on that remark on LinkedIn by some astute data engineers, who said I had a misconception about what DuckDB was capable of. Being the curious data engineer I am, I took a bite on that bait and decided to roll up my sleeves and benchmark much larger datasets.

But how much larger?

Also - if you want to subscribe to Matt’s Substack, you can click here.

I first decided to go after ~200 GBs. DuckDB read that data in <10 seconds.

This was too fast. It felt magical. What about 500 GBs? Then I hit a wall: a physical wall. The hard drive on my Mac M2 didn’t have enough space for 500GBs. I strolled to my local Best Buy and picked up this thing:

Side Note - A 4 TB external hard drive might seem like overkill; this was one of those “Go big or go home” moments. I figured in my mind “well if 500gb works, I want to have enough runway for much larger tests down the road”

I created a 500GB dataset on the external drive in DuckDB. It read that data in ~40 seconds

This made me realize I needed to set my sights on the big kahuna. 1 full TB of data!

Building A 1 TB Dataset for DuckDB

In my previous articles, you will see that I use a script that leverages DuckDB’s' generate_series' function to generate rows of data quickly. The gist of it looks like this:

It’s straightforward:

• You pass in a row count and let it generate a Parquet file.

• If we want to do this at scale and not wait several hours (or go dreaded serialized). What do you do?

• Bring in the good ol’ Python ProcessPoolExecutor and go parallel. (code here)

Ok, But Did You Really Generate A Full TB Of Data?

Yes, it took my M2 Pro (16GB of RAM) ~70 minutes to fry this egg with 10 workers in parallel. Here’s the proof:

The dataset is: 400 files, each ~2.76GB in size.

So now, without further ado, let’s get cracking and run some benchmarks on this.

Side Note - If you have not moved from the old Python virtual environment of “-m venv” over to UV, do yourself a favor and do it now; uv loads packages faster, makes targeting specific Python environments easier; I could go on…you’ll thank me later

The Benchmark…And What Exactly Are You Doing Here?

Today’s benchmark will:

• run a common aggregation query across the 1TB dataset;

• It will group by a date, count rows, and sum a value.

This is a common analytics query I have seen in my last two decades as a data engineer and BI leader; I did not cherry-pick this to just make DuckDB look good. This is what the benchmark query boils down to:

For our benchmark, we will run it 5 times. Below are the results:

Average processing time locally: 1 minute, 29 seconds.

Hold On - You Said We Could Crush A Full TB In Under 30 Seconds!?

I did say that. This first benchmark was on my laptop and local computer, which is impressive.

What would happen if I were to create a full 1TB dataset in MotherDuck and try this again?

Time To Join The Flock

On MotherDuck, we have excellent options to load data. We could do stuff like:

• store CSVs and Parquet in S3/Azure/GCP

• import the TCP-H dataset

• Use our local CLI to generate data

For this article, I chose the third option. Here is the script that created the 1TB dataset in MotherDuck:

I created a view in a MotherDuck database that leveraged the generate_series function (like in the previous local benchmark). After that, I ran the script to iterate over and insert the data multiple times.

After 10 iterations, I saw it wasn’t quite at 1TB; I manually ran the load process several times more until I got 1TB.

We now have over 1TB of data. Now it’s time to choose our compute capacity.

In MD, we have four standard options for compute capacity: Pulse, Standard, Jumbo, and Mega. I went with Mega, given we are dealing with a full TB of data:

For the benchmark, we ran the exact same query we did locally, but with a caveat;

• MotherDuck has intelligent caching; running the same query five times will have results 2-5 be about 5 seconds or less because it will read from cache vs. actually scanning the data.

So how do we get around that?

• Simple - for each iteration, we will have our aggregation values sum a different lower and upper range, which makes the query non-deterministic, and removes the ability for it to just hit the cache layer. The query template looks like this:

Below are the results of our benchmark:

Holy Smokes -

• That was clocking in at under 17 seconds on average. You might also ask - “Hey what is that run 0 a.k.a the cold start?”

  • Remember, we are now dealing with a cloud data warehouse, which will not always keep our data readily available to rip from RAM; sometimes, it will be on disk and have to get read in; thus, the iteration 0 is a warm up run, incase its having to read off of disk for the first usage; for datasets you will query often in MotherDuck, this won’t be an issue, as your data will more than likely be ready to process and sit in hot ram.

Ok, Great Stuff! We Are Done, Right?

We blew our promise of scanning a full TB in under 30 seconds out of the water, by a factor of 2. But what if we wanted faster?

I’m Ready - Let’s Go Deeper

DuckDB supports indexes, but they don’t really push the concept much. The Zonemap index is a secret weapon that allows you to take advantage of pre-sorted data through min/max tracking of the metadata.

How would I implement these zone maps?

• Let’s reload our dataset where we sort and insert on the rand date. The load process looked like this:

Once that was complete, I created another benchmark, and here were the results:

The sorted (zonemap) dataset improved our benchmark time by roughly 30%. That is an amazing tweak, simply by loading the table in a sorted order by the field we are grouping by.

Summary

This article showcased a paradigm shift in DuckDB’s capabilities! We have now shattered the belief of what “small” data is and what the duck can do. Even on my local laptop, we were still scanning 1TB of data in <2 minutes. If my batch jobs refreshed my reports in 2 minutes without Spark, I would be very happy!

Here are all the code examples.

• Local Data Generator

• Motherduck Data Generator (Unsorted)

• Motherduck Data Generator (Sorted)

• Local Benchmark

• Motherduck Unsorted Data Benchmark

• Motherduck Sorted Data Benchmark

Thanks again, MotherDuck, for providing us with the environment to showcase this capability!

Thanks for reading and happy holidays! If you found value in this article, make sure to comment and share with your friends!

Matt and Zach

Data engineers sit at the intersection of value and vulnerability. You build the pipelines that power analytics, AI, and business decisions. But the same systems that enable innovation also become prime targets for attackers.

You don’t need to become a full-time security engineer to protect your data. But you do need to think like one — because no amount…

Read more

Sahar and I learned a lot during out time working in core growth together at Facebook working in friending and notifications.

Let’s talk about the pain of unlearning and then let’s get to the magic.

Imagine you’re an analyst at a social media company. The retention team asks: “For users who now have 1000+ followers but had under 200 three months ago – what device were they primarily using back then? And of the posts they viewed during that growth period, how many were from accounts that were mutuals at the time?”

You need to join user data (follower counts then and now), device data (primary device then and now), relationships (who was a mutual then vs now), and post views – all as of 3 months ago.

With most data warehouse setups, this query is somewhere between “nightmare” and “impossible.”  

You’re dealing with state, not actions. State in the past, across multiple tables. There’s a word for this problem – slowly changing dimensions. Whole chapters of textbooks deal with various approaches. You could try logs (if you logged the right stuff). You could try slowly changing dimensions with `valid_from/valid_to` dates. You could try separate history tables. All of these approaches are painful, error-prone, and make backfilling a living hell.

There’s a better way. Through the magic of ✨datestamps✨ and idempotent pipelines, this query becomes straightforward. And backfills? They become a button you push.

Part 1 fixed weird columns, janky tables, and trusting your SQL. Part 3 will cover scaling your team and warehouse. But now – now we fix: backfills, 3am alerts, time complexity, data recovery, and historical queries.1

The old way was a mess

Here’s what most teams do when they start out:

Option 1: Overwrite everything daily

Your pipeline runs every night, updates dim_users with today’s snapshot, overwrites yesterday’s data. Simple! Until six months later when someone asks “how many followers did users have in March?” and you realize: that data is gone. You have no history. You can’t answer the question. Oops.

(Jargon alert – Apparently this is SCD Type-1 ¯\_(ツ)_/¯ )

Option 2: Try to track history manually

Okay, you think, let’s be smarter. Add an updated_at column. Or maybe valid_from and valid_to dates, with an is_current flag. When a user’s follower count changes, don’t update their row – instead, mark the old row as outdated and insert a new one.

(Jargon alert – This is SCD Type-2. Booo)

This is better! You have history. But now:

• Your pipelines need custom logic to “close out” old rows before inserting new ones

• If you mess up the valid_to dates, you get gaps or overlaps in history

• Backfilling becomes a nightmare – you can’t just rerun a pipeline, you need to carefully update dates without breaking everything downstream

• Querying becomes a nightmare. To get user data “as of 3 months ago”, you need:

SELECT * FROM dim_users WHERE user_id = 123 AND valid_from <= ‘2024-10-01’ AND (valid_to > ‘2024-10-01’ OR valid_to IS NULL)

Now imagine joining MULTIPLE historical tables (users, devices, relationships). Every join needs that BETWEEN logic. Miss one and your results are silently wrong. Get the date math slightly off and you’re joining snapshots from different points in time. Good luck debugging that.

Option 3: Separate current and history tables

Some teams maintain dim_users (current snapshot) and dim_users_history (everything else). Now you’ve got two sources of truth to keep in sync. Analysts need to remember which table to query. Any analysis spanning current and historical data requires stitching across tables with UNION ALL. It’s a mess.

And, depending on how the dim_users_history table works – it won’t solve any of the problems you’d have in option 2!

All of these approaches share a problem: they’re trying to be clever about storage. They made sense when disk was expensive. They don’t anymore.

(Jargon alert – This is SCD Type-4. Note that I didn’t know this when I started writing this blog post because it’s useless, boring, outdated jargon. Ignore it.)

Sponsorship

If you want to learn more about data modeling and data architecture in detail, you can use code SCDSUCKS by November 14th to get 35% off the DataExpert.io Snowflake + dbt boot camp

The new way: Just append everything

You solve it with date stamps. You solve it with “functional data engineering”.

What you really want is a sort of table that tracks state – a dimension table –, but where you can access a version that tracks information about the world today, and another version that tracks information about the world in the past.

Maxime Beauchemin wrote the seminal public work on the idea here. But, honestly, I think the concept can be explained more plainly and directly. So here we are.

The thinking goes like this:

• We’re getting new data all the time.

• Let’s simplify it and say – we get new data every day. We copy over snapshots from our production database each evening.

• There are complex, convoluted ways to keep track of what data is new and useful, and what data is a duplicate of yesterday.

• But wait. Storage is cheap. Compute is cheap. Pipelines can run jobs for us while we sleep.

• It’s annoying to have a table with the data we need as of right now, and either some specialized columns or tables to track history..

• Instead, what if we just kept adding data to existing tables? Add a column for “date this information was true” to keep track.

Here’s what it looks like in practice. Instead of overwriting your dimension tables every day, you append to them:

dim_users
┌─────────┬───────────┬────────────┐
│ user_id │ followers │ ds         │
├─────────┼───────────┼────────────┤
│ 123     │ 150       │ 2024-10-01 │
│ 123     │ 180       │ 2024-10-02 │
│ 123     │ ...       │ ...        │
│ 123     │ 1200      │ 2025-01-16 │
└─────────┴───────────┴────────────┘

dim_devices
┌─────────┬─────────┬────────────┐
│ user_id │ device  │ ds         │
├─────────┼─────────┼────────────┤
│ 123     │ mobile  │ 2024-10-01 │
│ 123     │ mobile  │ 2024-10-02 │
│ 123     │ ...     │ ...        │
│ 123     │ desktop │ 2025-01-16 │
└─────────┴─────────┴────────────┘

dim_relationships:
┌─────────┬───────────┬───────────┬────────────┐
│ user_id │ friend_id │ is_mutual │ ds         │
├─────────┼───────────┼───────────┼────────────┤
│ 123     │ 789       │ true      │ 2024-10-01 │
│ 123     │ 789       │ true      │ 2024-10-02 │
│ ...     │ ...       │ ...       │ ...        │
│ 123     │ 789       │ false     │ 2025-01-16 │ ← changed
└─────────┴───────────┴───────────┴────────────┘

fct_post_views:
┌─────────┬───────────┬───────────┬────────────┐
│ post_id │ viewer_id │ poster_id │ ds         │
├─────────┼───────────┼───────────┼────────────┤
│ 5001    │ 123       │ 789       │ 2024-10-01 │
│ 5002    │ 123       │ 456       │ 2024-10-01 │
│ 5003    │ 123       │ 789       │ 2024-10-05 │
│ ...     │ ...       │ ...       │ ...        │
│ 9999    │ 123       │ 789       │ 2025-01-15 │
└─────────┴───────────┴───────────┴────────────┘

Now that impossible retention query becomes straightforward. No BETWEEN clauses, no valid_from/valid_to logic – just filter each table to the date you want:

-- For fast-growing users, what device did they use back then?

WITH
  today_users as (SELECT user_id, followers as today_followers
      FROM dim_users WHERE ds = ‘2025-01-16’ AND followers >= 1000),
  past_users as (SELECT user_id, followers as past_followers
      FROM dim_users WHERE ds = ‘2024-10-01’ AND followers < 200),
  past_device as (SELECT user_id, device
      FROM dim_devices WHERE ds = ‘2024-10-01’),
  user_device as (
      SELECT tu.user_id, today_followers, past_followers, pd.device
      FROM past_users pu
      JOIN today_users tu ON pu.user_id = tu.user_id
      JOIN past_device pd ON tu.user_id = pd.user_id),
  views as (
      SELECT post_id, viewer_id, poster_id, ds
      FROM fct_post_views 
      WHERE ds BETWEEN ‘2024-10-01’ AND ‘2025-01-16’)
  SELECT
      ud.user_id,
      ud.device as device_during_growth,
      COUNT(DISTINCT views.post_id) as posts_from_mutuals
  FROM user_device ud
  LEFT JOIN views
      ON ud.user_id = views.viewer_id
  LEFT JOIN dim_relationships past_rels
      ON views.viewer_id = past_rels.user_id
      AND views.poster_id = past_rels.friend_id
      AND views.ds = past_rels.ds -- mutual status AS OF view date
      AND past_rels.is_mutual = true
  GROUP BY 1, 2

Is this query complex? Sure.2 But the complexity is in the business logic (what you’re trying to measure), not in fighting with valid_from/valid_to dates. Each query just filters to ds = {the date I want}. That’s it.

The idea is that you’re not overwriting existing tables. You are appending. 3

Sidebar: Common Table Expressions

If I had a SECOND “one weird trick” for data engineering, CTEs would be it. CTEs are just fucking fantastic. With liberal use of common table expressions (the WITH clause you saw in the retention query above), you can treat subqueries like variables – and then manipulating data feels more like code. Make sure your query engine (like Presto/Trino) flattens them for free – but if it does: wowee! SQL just got dirt simple. (a free one hour course on CTEs here)

When you grab data into your warehouse4, append a special column. That column is usually called “ds” – probably short for datestamp. You want something small and unobtrusive. (Notice that “date” would be a bad name – because you’d confuse people between this (date of ingestion of data) and the more obvious sort of date – date the action happened.) For snapshots, copy over the entire data of the snapshot, and have your “ds” column be <today’s date>. For logs, you can just grab the logs since yesterday, and set the ds column to <today’s date>.

Sidebar: Date stamps vs Date partitions
I’ll mostly say “date stamps” in this piece – the concept of marking each row with when that data was valid/ingested.

“Date partitions” is how most warehouse tools *implement* date stamps. A partition is how your warehouse physically organizes data. Think of it like: all rows with ds=2025-01-15 get grouped together in one chunk, ds=2025-01-16 in another chunk, and so on. (In older systems, each partition was literally a separate folder. Modern cloud warehouses abstract this, but the concept remains.)

Why does this matter? When you query `WHERE ds=’2025-01-15`, your warehouse only scans that one partition instead of the entire table. This makes queries faster and cheaper (especially in cloud warehouses where you pay per data scanned).

People use the terms interchangeably. The important thing is the concept: tables with a date column that lets you query any point in history.

Every table emanating from your input tables should add a filter (WHERE ds={today}), and similarly append the data to the table (WHERE ds={today}). (Except special circumstances where a pipeline might want to look into the past).

That’s it! Now your naive setup (overwriting everything every day) has only changed a bit (append everything each day, and keep track of what you appended when) – but everything has become so much nicer.

This is huge

This has two major implications:

First, many types of analysis become much easier. Want to know about the state of the world yesterday? Filter with WHERE ds = {yesterday}. Need data from a month ago? Filter with WHERE ds = {a month ago}. You can even mix and match – comparing today’s data with historical data, all within simple queries.

Second, data engineering becomes both easier and much less error prone. You can rerun jobs, create tables with historical data, and fix bugs in the past. Your pipeline will produce consistent, fast, reliable results consistently

What “functional” actually means

(Aka “I don’t know what idempotent means and at this point I’m afraid to ask”)

So, in Maxime’s article (link) there’s all this talk about “functional data engineering”. What does that even mean? Let’s discuss.

First, we’re borrowing an idea from traditional programming. “Functional programs” (or functions) meet certain conditions:

1. If you give it the same input, you get the same output. Every time.

2. State doesn’t change. Your inputs won’t change, hidden variables won’t change. It’s clean. (AKA “no side effects”)

Okay, so what does that mean for pipelines? Functional pipelines:

• Given the same input, will give the same output

• Don’t use (or rely on) magic secret variables

This is what people mean when they say “idempotent” pipelines or “reproducible” data.

And here’s how to implement it: datestamps.

• Your rawest/most upstream data should never be deleted – just keep appending with datestamps

• Pipelines work the same in backfill mode vs normal daily runs

• If you find bugs, fix the pipeline and rerun – the corrected data overwrites the bad data

• Time travel is built in – just filter to any ds you need

Datestamps also give you the nice side-effect of having it be very clear how fresh the data you’re looking at is. If the latest datestamp on your table is from a week ago -- it’s instantly understandable not only what’s wrong, but also you have hints about why.

Sidebar – what this looks like in practice:
Your SQL will look something like: WHERE ds=’{{ ds }}’ (Airflow’s templating syntax) or WHERE ds=@run_date (parameter binding).

Your orchestrator injects the date - whether it’s today’s scheduled run or a backfill from three months ago. Same SQL, different parameter. That’s the whole trick.

Backfilling is now easy, simple, magical

Remember that retention query? Now imagine you built that analysis pipeline three months ago, but you just discovered a bug in your dim_relationships table. The is_mutual flag was wrong for two weeks in November. You fixed the bug going forward, but now all your retention metrics from that period are wrong.

With the old SCD Type-2 approach, you’re in hell:

You can’t just “rerun November.” Because each day’s pipeline depended on the previous day’s state. Day 15 updated rows from Day 14, which updated rows from Day 13, and so on. To fix November 15th, you’d need to:

1. Rerun November 1st (building from October 31st’s state)

2. Wait for it to finish

3. Rerun November 2nd (building from your new November 1st)

4. Wait for it to finish

5. Rerun November 3rd...

6. ...keep going for 30 days, sequentially, one at a time

And this is assuming nothing breaks along the way. If Day 18 fails? Start over. Need to fix December too? Add another 31 sequential runs.

In Airflow terms, this is what depends_on_past=True does to you. Each day is blocked until the previous day completes. Backfilling becomes painfully slow. But that’s by no means the worst part.

You can’t just hit “backfill” and walk away. Your normal daily pipeline logic doesn’t work for backfills. Why? Because SCD Type-2 requires you to:

• Close out existing rows (set their valid_to date)

• Insert new rows (with new valid_from dates)

• Update is_current flags

• Handle the case where a row changed multiple times during your backfill period

Your daily pipeline probably has logic like:

-- Daily SCD Type-2 pipeline (simplified)
-- Step 1: Close out changed rows
UPDATE dim_users
SET valid_to = CURRENT_DATE - 1, is_current = false
WHERE user_id IN (
SELECT user_id FROM users_source_today
WHERE <something changed>
)
AND is_current = true;

-- Step 2: Insert new versions
INSERT INTO dim_users (user_id, followers, valid_from, valid_to, is_current)
SELECT user_id, followers, CURRENT_DATE, NULL, true
FROM users_source_today;

This works fine when you’re processing “today.” But for a backfill? You need different SQL:

• You need to carefully reconstruct valid_from/valid_to for historical dates

• And handle the fact that a user might have changed multiple times during your backfill window

• This gets messy fast.

• You’re essentially rewriting your pipeline. (WHY?)

So now you’re not just waiting 30 sequential days - you’re maintaining two separate codebases: one for daily runs, one for backfills. And every time you change your daily logic, you need to update your backfill logic to match. More code to write, more code to test, more places for bugs to hide. It’s completely useless and unnecessary.

Sidenote – even worse, if you’re outside your retention window (say, the source data from 90 days ago has been deleted), you can’t backfill at all. You’d need to completely rebuild the entire table from scratch, from whatever historical snapshots you still have. Which probably means... datestamped snapshots anyway. Womp womp.

With datestamps, backfilling is trivial:

Your pipeline for any given day just needs:

• Input tables filtered to ds=’2024-11-15’ (or whatever day you’re processing)

• Write output to ds=’2024-11-15’

That’s it. November 15th doesn’t need November 14th. It just needs the snapshot from November 15th.

So to fix your broken November data:

# In Airflow (or whatever orchestrator)
> airflow dags backfill my_retention_pipeline \--start-date 2024-11-01 \--end-date 2024-11-30

What happens behind the scenes?

• All 30 days kick off in parallel (up to your concurrency limits)

• Each day independently reads from its ds partition

• Each day independently writes to its ds partition

• No coordination needed between days

• The whole month finishes in the time it takes to run one day

The exact same SQL that runs daily also handles backfills - no special logic, no custom code

This changes everything:

No more custom SQL for backfills - It’s just a button you push. Your orchestrator handles it. The same pipeline code that runs daily also handles backfills. No special logic needed.

New tables get history for free - Created a new dim_users_enriched table today but want to populate it with the last year of data? Just backfill 365 days. Since your input tables have datestamps, the data is sitting there waiting.

Bugs in old data become fixable - Fix your pipeline logic, backfill the affected date range, done. The old (wrong) data gets overwritten with the new (correct) data for those specific partitions. Everything downstream can reprocess automatically.

Upstream changes cascade easily - Fixed a bug in dim_users? All downstream tables that depend on it can backfill the affected dates in parallel. The whole warehouse stays in sync.

This is possible because your pipelines are idempotent. Run them once, run them a thousand times - given the same input date, you get the same output. No hidden state, no “current” vs “historical” logic, no manual date math.

One pattern to avoid: Tasks that depend on the previous day’s partition of their own table. If computing today’s dim_users requires yesterday’s dim_users, you’ve created a chain - backfilling 90 days means 90 sequential runs that can’t be parallelized. This is sometimes necessary for cumulative metrics, but most dimension tables don’t need it - just recompute from raw sources each day.

For most datestamped pipelines, depends_on_past should be False. Each day is independent - the only dependency is “does the upstream data exist for this ds?”

Welcome to the magic of easy DE work

We started this article staring at the prospect of valid_from/valid_to logic, sequential backfills that take days, and custom SQL for every backfill and cascading for every bugfix. Yuck. Ew!

Or maybe – worse – with no sense of history at all. No ability to ask “how did the world look like yesterday”, much less “3 months ago”. I’ve seen startups and presidential campaigns and 500 million dollar operations operate like this. 🙃

Now you know the secret. Now you have the magic. What mature companies have been doing all along: snapshot your data daily, append it with datestamps, and write idempotent pipelines on top.

That’s it. That’s the whole One Weird Trick. Add a ds column to every table. Filter on it. Write your pipelines to be independent of each other. Have every pipeline be ds-aware. Storage is cheap. Your time is expensive. Getting your data wrong is extra expensive.

What you get in return:

• Backfills that run in parallel and finish in minutes instead of days

• Backfills that are a button push instead of custom SQL mess.

• Historical queries that are simple WHERE ds=’2024-10-01’ filters instead of date-range gymnastics

• Pipelines that are the same whether you’re processing today or reprocessing last year

• A built-in time machine for your entire warehouse

• Bugs that are fixable instead of permanent scars on your data

This is functional data engineering. Functional as in idempotent. And functional as in “it works”.

Your backfills are easy now. Your 3am alerts will be rarer. Time complexity is solved. Data recovery is trivial. Your job just became so much easier.

But we’re not done yet. Part 3 will tackle: how to scale your team and your warehouse. Parts 4 and 5 are gonna get me back on my “he who controls metrics controls the galaxy” soapbox.

For now, go add some datestamps. Your future self will thank you.

Hey, it’s Zach again. Sahar is currently open to work in NYC. Make sure to follow Sahar’s blog to understand growth and what comes next, both personally and for your business! And more at sahar.io

Data engineering can actually be easy, fast, and resilient! All you have to embrace is a simple concept: Date-stamping all your data.

Why isn’t this the norm? Because – even in 2025 — , institutions haven’t really understood the implications that STORAGE IS CHEAP! (And your data team’s time is expensive).

Datestamping solves so many problems. But you won’t find it in a standard textbook. They’ll teach you “slowly changing dimensions Type 2” when the real answer is simpler and more powerful. You will find the answer in Maxime Beauchemin’s seminal article on functional data engineering. Here’s the thing – I love Max, but that article is not helpful to the majority of people who could learn from it.

What if I told you:

• We can have resilient pipelines.

• We can master changes to data over time.

• We can use One Weird Trick to marry the benefits of order and structure with the benefits of chaos and exploration.

That’s where this article comes in. It’s been 7 years in the making – all the stuff that you should know, but no one bothered to tell you yet. (At least, in plain english – sorry Max!)

• Part One: How to set up a simple warehouse (and which small bits of jargon actually matter)

• Part Two: Date-stamping. Understand this and everyone’s life will become easier, happier, and 90% more bug-free.

• Part Three: Plugging metrics into AB testing. Warehousing enables experimentation. Experimentation enables business velocity.

• Part Four: The limits of metrics and KPIs. It can be so captivating to chase short-term metrics to long-term doom.

I’ll show you a practical intro to scalable analytics warehousing, where date stamps are the organizing principle, not an afterthought. In plain language, not tied to any specific tool, and useful to you today Meta used this architecture even back in the early 2010s. It worked with Hive metastore. It still works with Iceberg, Delta, and Hudi.

But first, to understand why all this matters, you need some context about how warehouses work. Then I’ll show you the magic.

Sponsorship

Cut Code Review Time & Bugs in Half

Code reviews are critical but time-consuming. CodeRabbit acts as your AI co-pilot, providing instant Code review comments and potential impacts of every pull request.

Beyond just flagging issues, CodeRabbit provides one-click fix suggestions and lets you define custom code quality rules using AST Grep patterns, catching subtle issues that traditional static analysis tools might miss.

CodeRabbit has so far reviewed more than 10 million PRs, installed on 1 million repositories, and used by 70 thousand Open-source projects. CodeRabbit is free for all open-source repo’s.

Part one — A Simple Explanation of Modern Data Warehousing

Our goals and our context

We are here to build a system that gets all company data, tidily, in one place. That allows us to make dashboards that executives and managers look at, charts and tools that analysts and product managers can use to do deep dives, alerts on anomalies, and a breadth of linked data that allows data scientists and researchers to look for magic or product insights. The basic building blocks are tables, and the pipelines that create and maintain them.

Sidebar: DB vs Data lake? OLTP vs OLAP? Production vs warehouse? Here’s what you need to know.

A basic point about a data warehouse (or lake, or pond, or whatever trendy buzzword people use today) is that it is not production. It must be a separate system from “the databases we use to power the product”.

Both are “databases”, both have “data”, including “tables” that might be similar or mirrored – but the similarity should end there.

• Your production database is meant to be fast, serve your product and users. It is optimized for code to read and write.

• Your warehouse is meant to be human-usable, and serve people inside the business. It is optimized for breadth, for use by human analysts, and to have historical records.

Put it this way – your ecommerce webapp needs to look up an item’s price and return it as fast as possible. Your warehouse needs to look up an item from a year ago, and look at how the price changed over the course of months. The database powering the webapp won’t even store the information, much less make it easy to compute. Meanwhile if you run a particularly difficult query, you don’t want your webapp to slow down.

So – split them. (You might hear people talking about OLTP vs OLAP – it’s just this distinction. Ignore the confusing terminology. Here’s a deep dive into the two types of OLAP data model (Kimball and One Big Table) )

So, we want a warehouse. Ideally, it should:

• Be separate from our production databases

• Collect all data that is useful to the company

• Have tables that make queries easy

• Be correct – with accurate, trusted, information

• Be reasonably up to date – with perhaps a daily lag, rather than a weekly or monthly one

• Power charts and interactive tools, while also being useful for automatic and local queries

This used to be difficult! (It is not anymore!) There was a tradeoff between “big enough to have all the data we need” and “give answers fast enough to be useful”. A lot of hard work was put into reconciling those two needs.

Since circa 2015 or so, this pretty much no longer a problem. Presto/Trino, Spark, and hosted databases (BigQuery, Snowflake, the AWS offerings) and other tools allow you to have arbitrarily huge data, accessed quickly. We live in a golden age.

Sidebar: At my old school…
At Meta, they used HDFS and Hive to power their data lake and MySQL to power production. Once a day they took a “snapshot” of production with a corresponding date stamp and moved the data from MySQL to Hive.

In a world where storage is cheap, access to data can be measured in seconds rather than minutes or hours, and data is overflowing, the bottleneck is engineering time and conceptual complexity. Solving that bottleneck allows us to break with annoyingly fiddly past best practices. That’s what I’m here to talk about.

A basic setup

Imagine your warehouse as a giant box, holding many, many tables. Think of data flowing downhill through it.

• At the top: raw copies from production databases, marketing APIs, payment processors, whatever.

• At the bottom: clean, trusted tables that analysts actually query.

• In between: pipelines that flow data from table to table.

[Raw Input Tables]
├─ users_production
├─ events_raw
├─ transactions_raw [Pipelines]
└─ ... ↓

     Clean → Join → Enrich

                 ↓

[Clean Output Tables]
├─ dim_users
├─ fct_events
└─ grp_daily_revenue

How do we get from raw input to clean tables? Pipelines. (See buzzwords like ETL, ELT? Ignore the froth – replace with “pipelines” and move on).

Pipelines are the #1 tool of data engineering. At their most basic form, they’re pieces of code that take in one or more input tables, do something to the data, and output a different table.

What language do you write pipelines in? Like it or not, the lingua franca of editing large-scale data is SQL. The lingua franca of accessing large scale data is SQL. SQL is a constrained enough language that it can parallelize easily. The tools that invisibly translate your simple snippets into complex mechanisms to grab data from different machines, transform it, join it, etc – they not only are literally set up with SQL in mind, they figuratively cannot do the same for python, java, etc. Why? Because a traditional programming language gives you too much flexibility -- there’s no guarantee that your imperative code can be parallelized nicely.

Sidebar: When non-SQL makes sense (or doesn’t)

If you’re ingesting data from the outside world (calling APIs, reading streams, and so on), then python, javascript, etc could make sense. But once data is in the warehouse, beware anything that isn’t SQL – it’s likely unnecessary, and almost certainly going to be much slower than everything else.

Your tooling might offer a way to “backdoor” a bit of code (e.g. “write some java code that calls an API and then writes the resultant variable to a column”). Think twice before you use it. Often, it’s easier and faster to import a new dataset into your warehouse so that you can recreate with SQL joins what you would have done using an imperative language.

You may be tempted to transform or analyze data in R, pandas, or whatnot – that’s fine, but you do that by interactively reading from the warehouse. Rule of thumb: if you’re writing between tables in a warehouse – SQL. Into a warehouse – you probably need some glue code somewhere. Out of a warehouse – that’s on you.

So here’s the simple setup:

Then, set up a system of pipelines to this, every day, as soon as the upstream data is ready:

• As each of these input tables gets the latest dump of data from outside: take that latest day’s data, deduplicate, clean it up a bit, rename the columns, and cascade it to a nicer, cleaner version of that table. (this is your silver tier data in medallion architecture)

• Then, from that nicer input table, perform a host of transformations, joins, etc to write to other downstream tables. (this is your master data)

• Master data is highly trusted which makes building metrics and powering dashboards easy!1

Every day, new data comes in, and your pipeline setup cascades new information in a host of tables downstream of it. That’s the setup.

A well-ordered table structure

Okay, so to review: the basic useful item in a warehouse is a table. Tables are created (and filled up by) pipelines.

“Great, great,” you might say – “but which tables do I build?”

For the sake of example, let’s imagine our product is a social network. But this typology should work just as well for whichever business you are in – from b2b saas to ecommerce to astrophysics.

From the perspective of the data warehouse as a product, there are only three kinds of tables: input tables (copied from outside), staging tables (used by pipelines and machines), and output tables – also known as user-facing tables.

Output tables (in fact, almost all tables) really only have three types:

• Tables where each row corresponds to a noun. (E.g. “user”, or even “post” or “comment”). When done right, these are called dimension tables. Prefix their names with dim_

• Tables where each row corresponds to an action. Think of them as fancier versions of logs. (E.g. “user X wrote post Y at time Z”). When done right, these are called fact tables. Prefix their names with fct_

• Everything else. Often these will be summary tables. (e.g. “number of users who made at least 1 post, per country, per day). If you’re proud of these, prefix them with sum_ or agg_.

Sidebar: more on naming

YMMV, but I generally don’t prefix input tables. Input tables should be an exact copy of the table you’re importing from outside the warehouse. Changing names breaks that – and an unprefixed table name is a good sign that the table cannot be trusted.

Staging and temporary tables are prefixed with stg_ or tmp_.

Let’s talk more about dimension and fact tables, since they’re the core part of any clean warehouse.

Dimension tables are the clean, user-friendly, mature form of noun tables.

• Despite being focused on nouns (say, users), they can also roll up useful verby information (leveraging cumulative table design)

• For instance, a dim_users table might both include stuff like: user id, date created, datetime last seen, number of friends, name; AND more aggregate “verby” information like: total number of posts written, comments made in the last 7 days, number of days active in the last month, number of views yesterday.

• If a data analyst might consistently want that data – maybe add it to the table! Your small code tweak will save them hours of waiting a week.2

(Now, what’s to stop the table from being unusably wide? Say, with 500+ columns? Well, that’s mostly an internal culture problem, and somewhat a tooling problem. You could imagine, say, dim_user getting too large, so the more extraneous information is in a dim_user_extras table, to be joined in when necessary. Or using complex data types to reduce the number of columns)

Fact tables are the clean, user-friendly, mature form of logs (or actions or verb tables).

• Despite being verb focused, fact tables contains noun information. (Zach chimes in: here’s a free 4 hour course on everything you need to know about fact tables)

• Unlike a plain log, which will be terse, they can also be enriched with data that might probably live in a dim table.

• The essence of a good fact table is providing all the necessary context to do analysis of the event in question.

• A fact table, fundamentally, helps you understand: “Thing X happened at time Y. And here’s a bunch of context Z that you might enjoy”.

• So a log containing “User Z made comment Xa on post Xb at time Y” could turn into a fct_comment table, with fields like: commenter id, comment id, post id, time, time at commenter timezone, comment text, post text, userid of owner of post, time zone of owner of parent post. Some of these fields are strictly speaking unnecessary – you could in theory do some joins to grab the post text, or the comment text, or time zone of the owner of the parent post. But they’re useful to have handy for your users, so why not save them time and grab them anyway.

Q: Wait – so if dim tables also have verb data, and fact tables also have noun data, what’s the difference?

A: Glad you asked. Here’s what it boils down to – is there one row per noun in the table? Dim. One row per “a thing happened?” Fact. That’s it. You’re welcome.

Here, as in so much, we are spending space freely. We are duplicating data. We are also doing a macro form of caching – rather than forcing users to join or group data on the fly, we have pipelines do it ahead of time.

Compute is cheap, storage is cheap. Staff time is not. We want analysis to be fluid and low latency – both technically in terms of compute, and in terms of mental overhead.

Q: Wait! What about data stamps? Where’s the magic? You promised magic.

A: Patience, young grasshopper. Part of enlightenment is the journey. Part of understanding the magic is understanding what it builds on. And – hey – would YOU read a huge blog post all at once? Or would you prefer to read it in chunks. Yeah, you with your Tiktok problem and inability to focus. I’m surprised you even made this far.

Stay tuned for part two where we:

• Show you how to make warehousing dirt easy

• Behold the glory of date stamping

  • Through better data quality

  • How it avoids backfill nightmares

  • Bug fixes are easy now

  • You get a time machine for free

• Explore the dream of functional data engineering (what is that weird phrase?)

• Throw SCD-2 and other outdated “solutions” to the dustbin of history

Make sure to follow Sahar’s blog to understand growth and what comes next, both personally and for your business! Sahar is currently open to work, he is interested in DevRel and engineering leadership roles in New York City.

Knowing where to start and how to get a handle on this requires some guidance. This newsletter is going to unveil the st…

Read more

Read more

Introduction

Apache Spark has been the de facto open source data processing for fifteen years. It was invented to solve a major problem that traditional data warehousing was not built to solve - processing massive amounts of data horizontally at scale (Zach used Spark to process 2000 TBs per day at Netflix), whether in a structured or semi-structured for…

Read more

Read more

This month, I’m excited to present three fantastic opportunities to learn from top-tier practitioners without spending a dime.
Whether you’re trying to break into tech or looking to make a strategic pivot in your career, these free bootcamps are packed with real-world lessons, hands-on skills, and instruction from people who’ve been in the trenches.

We a…

Read more

There are some conversations that confirm what you already believe, and then there are conversations that take what you believe and frame it in a way that makes it actionable, precise and inevitable.

That’s what happened when I sat down with Chip Huyen during a Tech Talk we held at DataExpert.io AI Engineering Boot Camp.

Chip is an AI researcher, former big tech engineer, best-selling author and one of the most lucid thinkers on AI systems working today. Recently, she’s been in the public spotlight with her new book AI Engineering, which has quickly become the most comprehensive, well-structured guide to the essential aspects of building generative AI systems (we covered a lot of its content in the boot camp too).

This talk was neither a book launch nor a formal Q&A. It was a honest, refreshing & grounded conversation that can be distilled into seven core takeaways, each one capturing ideas Chip shared that stuck with me, challenged me or reshaped how I think about building and leading in AI. This article covers the following:

• Chip’s journey into AI

• Where most GenAI products go wrong

• The underrated value of UX

• How to build functional AI agents

• What really takes to ship value in the modern AI stack.

• How to stay informed without burning out

• Building with clarity and conviction

If you want to learn from other brilliant minds like Chip, we are launching a 10-week Challenge Boot Camp on Sep 15th where will be covering insightful tech talks with 15 industry leaders in the data, analytics and AI engineering space. The first 5 people to register can use code CHIP for 30% off!

✍🏻 #1: The Primacy of Compute and Data

To understand Chip’s journey into AI, we have to rewind to 2012, the year Deep Learning truly exploded onto the scene. That was the year AlexNet, a deep convolutional neural network, won the ImageNet competition by a massive margin (over 10 percentage points better than the next best model).

AlexNet rewrote the rules that have defined the last decade of AI. And notably, one of the paper's co-authors, Ilya Sutskever, would go on to co-found OpenAI, the organization that would later lead the charge on scaling up large language models.

Chip recounted how a single sentence from that 2012 paper changed her life trajectory:

“Our experiments show that we can achieve better results by just waiting for more compute and more data.”

That line reframed AI not as a field of breakthroughs, but as one of compounding scale. Chip went on to work at NVIDIA to understand compute infrastructure and later joined Snorkel AI to understand data workflows.

She also reflected on how OpenAI was initially dismissed for simply scaling up, with many academics saying it wasn't real research. But the turning point came in 2020 when the GPT-2 paper received a best paper award, and the conversation suddenly shifted.

Everyone was like, ‘Wow, now it’s real research.’

💡 I remember similar skepticism back in my days at Facebook. People dismissed what OpenAI was doing as brute force. It turns out brute force was the insight.

This key takeaways reframes my view of AI progress not as a parade of novel ideas but as an engineering problem of sufficient scale. Chip's clarity here gives me language to explain why things like GPT-5 didn't just appear but emerged from compute and data discipline.

✍🏻 #2: The GenAI Hype Cycle and the Misuse of ML

What makes Chip’s take on GenAI refreshing is that her relationship with AI long predates the hype. Before ChatGPT became a buzzword, she was building simple algorithms to test logic, even designing games her smartest friends couldn’t win. Not to outsmart them but to understand how reasoning could be codified.

Read more

Fake accounts come and go. Some are flagged incorrectly, others are later verified, and many are caught by internal ML systems. The goal of our pipeline was to trace the inflows and outflows of fake accounts daily. That meant building a reliable dataset that could track:

• Accounts unlabeled as fake (i.e. after submitting a valid ID)

• Accounts relabeled as fake

• Accounts flagged as fake for the first time

• Accounts that remained fake

The pattern was very straightforward: a classic cumulative table design. But the way it was wired to upstream data, specifically how it “waited” for inputs, created a non-deterministic nightmare. For weeks, I chased what I thought was a bug in my code, only to discover that the real problem had been there from day one.

In this article, I’ll cover the following aspects:

• Why relying on “latest” partition data broke everything

• How upstream non-deterministic leads to silent data mismatches

• The simple fix using explicit partition dates

• The tradeoff between latency and reproducibility

• Engineering lessons that go beyond code

If you want to learn more in depth about patterns like this, the DataExpert.io academy subscription has 200+ hours of content about system design, streaming pipelines, etc. The first 5 people can use code FAKE for 30% off!

If you enjoy this article, here are some more from my time in big tech:

• How I prepared for Airbnb’s staff data engineer interview

• How I achieved a 12x speed up on Facebook notification pipelines

• How I used the “Psycho” pattern to detect threats at Netflix

• How I cut Airbnb’s pricing data backfill time by 95%

Understanding Fake Account Flows

At a high level, the pipeline’s job was to compare today’s and yesterday’s fake account snapshots, determine who had entered or exited the fake state, and store those inflow/outflow transitions for downstream analytics.

This was a classic cumulative table design pattern, built in plain vanilla SQL and tracked four main fake states:

• New fakes → New people who got labeled as fake

• Resolved accounts → People who were labeled fake earlier than today and passed a challenge to remove the label

• Persisting fakes → People who were labeled fake earlier than today and haven’t passed a challenge

• Relabeled fakes → People who were labeled fake, passed a challenge, and then continued to do fake activity

Originally, it was set up like this:

1. Fake accounts pipeline waited on the “latest” partition of the users table

2. Then joined it with fake_accounts to compute transitions

3. The job ran daily and published results

-- Define struct and enum types

CREATE TYPE IF NOT EXISTS daily_detection_stats AS (
    detection_date DATE,
    login_attempts INTEGER,
    friend_requests_sent INTEGER,
    posts_created INTEGER,
    flagged_reports INTEGER
);

CREATE TYPE IF NOT EXISTS fake_classification AS ENUM('new', 'resolved', 'persisting', 'relabeled');

-- Create cumulative table
CREATE TABLE IF NOT EXISTS fake_accounts
(
    account_id TEXT,
    country TEXT,
    sign_up_method TEXT,
    sign_up_date DATE,
    daily_detection_stats daily_detection_stats[],
    fake_classification fake_classification,
    days_since_last_detected INTEGER,
    current_detection_date DATE,
    PRIMARY KEY (account_id, current_detection_date)
);


-- Dynamically pick the "latest" date
WITH latest_date AS (
    SELECT MAX(detection_date) AS detection_date FROM account_daily_signals
),
yesterday AS (
    SELECT * 
    FROM fake_accounts 
    WHERE current_detection_date = (SELECT detection_date - INTERVAL '1 day' FROM latest_date)
),
today AS (
    SELECT * FROM account_daily_signals 
    WHERE detection_date = (SELECT detection_date FROM latest_date)
)

-- Non-idempotent insert
INSERT INTO fake_accounts
SELECT
    COALESCE(t.account_id, y.account_id) AS account_id,
    COALESCE(t.country, y.country) AS country,
    COALESCE(t.sign_up_method, y.sign_up_method) AS sign_up_method,
    COALESCE(t.sign_up_date, y.sign_up_date) AS sign_up_date,
    CASE
        WHEN y.daily_detection_stats IS NULL THEN ARRAY[ROW(
            t.detection_date,
            t.login_attempts,
            t.friend_requests_sent,
            t.posts_created,
            t.flagged_reports
        )::daily_detection_stats]
        WHEN t.detection_date IS NOT NULL THEN y.daily_detection_stats || ARRAY[ROW(
            t.detection_date,
            t.login_attempts,
            t.friend_requests_sent,
            t.posts_created,
            t.flagged_reports
        )::daily_detection_stats]
        ELSE y.daily_detection_stats
    END AS daily_detection_stats,
    CASE
        WHEN y.account_id IS NULL THEN 'new'
        WHEN t.account_id IS NULL THEN 'resolved'
        WHEN t.detection_date IS NOT NULL AND y.fake_classification = 'resolved' THEN 'relabeled'
        ELSE 'persisting'
    END::fake_classification AS fake_classification,

    CASE
        WHEN t.detection_date IS NOT NULL THEN 0
        ELSE y.days_since_last_detected + 1
    END AS days_since_last_detected,
    COALESCE(t.detection_date, y.current_detection_date + INTERVAL '1 day')::DATE AS current_detection_date

FROM today t
FULL OUTER JOIN yesterday y
ON t.account_id = y.account_id;

The Architecture That Broke Everything

Read more

One of my most challenging assignments was owning the pipeline that deduplicated all notification events. This dataset drove downstream metrics like CTRs, conversions, and even machine learning signal quality.

This pipeline presented one big problem: it was slow. Very, very slow.

When I joined Facebook, the deduped notifications pipeline ran a giant Hive GROUP BY job once a day at UTC midnight which took 9.5 hours to complete. This latency issue represented a huge bottleneck for all downstream models in the Core Growth team.

This article is the story of how I brought 9.5-hour latency down to 45 minutes, and what it taught me about I/O, orchestration, and never trusting a “simple” DAG.

Here I’ll be covering the following:

• Why streaming deduplication at scale failed

• The hourly dedup job that exploded compute usage

• A tree-based DAG design that saved the day

• Key orchestration lessons from building a 300-step daily DAG

• How this project got me promoted and why I almost gave up

If you want to learn more in depth about patterns like this, the DataExpert.io academy subscription has 200+ hours of content about system design, streaming pipelines, etc. The first 5 people can use code DEDUP for 30% off!

The Stakes: Notifications at Facebook Scale

The notif_events table contained every event tied to a notification:

• Sent to your phone

• Delivered to device

• Clicked

• Converted

Since one person might click the same notification multiple times, we had to dedup those. If we counted every click, we’d get click-through rates above 100%, which negatively impacted metric tracking and, worse, model training.

But the problem wasn’t logic. It was latency.

As I mentioned earlier, the dedup job ran once a day via Hive & took 9.5 hours to complete. My managers wanted me to reduce latency dramatically.

Approach 1: Stream It 😣

At first, my manager’s request was straightforward: ‘Let’s dedup in real-time’.

So, I tried. I built a Spark Streaming job that listened to notification events and tried to hold recent activity in-memory for comparison. But this approach was holding as much as 50+ terabytes in RAM to do real-time deduping.

This wasn’t feasible. The streaming job crumbled under memory pressure. I had to come up with a better solution.

Approach 2: Hourly Dedup + Merge 🤔

Our second approach was to dedup every hour. The idea was simple:

1. Dedup the current hour table and write it to a sorted, bucketed table (to minimize shuffle later with SMB join)

2. Merge with the previous hour table (a cumulative of all previous hours’ deduped data).

3. Output a deduped table up to the current hour.

The read and merge hourly deduped table to the previous hour can be implemented with this simple SQL pattern inside your DAG:

-- Hourly dedup logic 
-- Remember this table is sorted and bucketed on user_id

INSERT OVERWRITE TABLE notif_deduped_hourly(ds='{{ ds }}', hour={{ current_hour}}, channel='{{ channel }}')
  SELECT 
     notif_id,
     user_id, 
     -- count the number of events of each type
     -- a custom UDF that returned a MAP {"sent":1, "clicked":3}
     COUNT_MAP(event_type) as event_map_count
  FROM notif_events
  WHERE event_hour = '{{ current_hour }}'
  AND ds = '{{ ds }}'
  AND channel = '{{ channel }}'
  GROUP BY notif_id, user_id

-- Then we merged the cumulative previous hours and the current hour with FULL OUTER JOIN

INSERT OVERWRITE TABLE notif_deduped_combined_hourly(ds='{{ ds }}', hour='{{ current_hour }}', channel='{{ channel }}')
WITH dedup_current_hour AS (
  SELECT 
     *
  FROM notif_deduped_hourly
  WHERE hour = '{{ current_hour }}'
  AND ds = '{{ ds }}'
  AND channel = '{{ channel }}'
),
previous_hour AS (
   SELECT 
     *
  FROM notif_deduped_combined_hourly
  WHERE hour = '{{ previous_hour }}'
  AND ds = '{{ ds }}'
  AND channel = '{{ channel }}'
)

SELECT  
   COALESCE(c.notif_id, p.notif_id) as notif_id,
   COALESCE(c.user_id, p.user_id) as user_id,
   -- udf that merges the keys of two maps
   -- {"sent": 1, "clicked": 3} + {"clicked":4} 
   -- = {"sent":1, "clicked": 7}
   COMBINE_MAPS(c.event_map_count, p.event_map_count)  as event_map_count
FROM dedup_current_hour c FULL OUTER JOIN previous_hour p 
ON c.notif_id = p.notif_id 
-- This condition triggers the SMB join because both tables are sorted and bucketed on user_id
AND c.user_id = p.user_id

This approach actually worked. It lowered latency and produced correct results.

But it had a huge flaw: compute usage exploded. It used 15 times more compute than the original 9.5-hour GROUP BY job.

How was that possible? Quite straightforward:

• Hour 1 processes 1 hour of data

• Hour 2 reads and merges 2 hours

• Hour 3 reads and merges 3 hours…

By hour 22, you find yourself reprocessing nearly the entire day’s data on every run.

It looked like this:

Once again, this solution was not sustainable, especially not for one of Facebook’s biggest datasets.

Approach 3: Tree-Style Merge DAG 🤩

Read more

At the time, we were leveraging a creative hybrid architecture internally dubbed as the “Psycho Pattern.” Think of i…

Read more

In this deep dive, I’ll walk you through the challenges I faced, the architectural mistakes I inherited and the solutions that made Airbnb earn millions.

This article covers the following topics:

• The subtle nuance in ‘availability’ definitions

• The original P&A pipeline design and its pain points

• Why massive backfills were so slow (and expensive)

• Introducing staging tables for rapid iteration

• What valuable lessons I learned

• The business impact of my work and some personal reflections

There’s a summary infographic of the entire data orchestration pipeline at the end of this article!

The True meaning of “Available”

Airbnb’s legacy definition of an “available” night was simply:

A host has not blocked this night, and it is not already reserved

On the surface, that sounds reasonable but, in reality, it diverged from what guests actually could book. Therefore, Airbnb aimed to establish a more reliable meaning of what’s “available”.

A trip can be booked that contains this night.

In fact, the two definitions only matched 96% of the time. But this subtle change captures the nuances between both definitions.

Key edge cases:

• Minimum stay requirements: Hosts or local regulations (e.g., 30-day minimum in New York) made many unblocked nights unbookable.

• Last minute/ time zone bugs: The system evaluated availability one second before midnight UTC. So Asia or Europe-based listings were sometimes asking, “Can I book yesterday?”

Original Pipeline Architecture & Pain Points

Here’s what I inherited in the P&A pipeline:

1. Fifteen raw datasets for blocked nights, calendar entries, regional regulations, minimum stays, etc.

2. A single Spark job that:

   1. Joins all fifteen tables in one massive operation

   2. Calls the Airbnb Java P&A library (via Scala Spark) to calculate availability

   3. Writes out the master P&A table for downstream models (i.e. Smart Pricing)

Why this was a problem

• Massive, repeated joins: Every time we tweaked a rule, the pipeline re-joined all 15 tables across eight years of historical data.

• Unpredictable runtimes: Backfilling could take 2½ weeks—despite only ~10 GB of daily data.

• High compute costs: Multiple backfill attempts (due to late requirements changes) meant wasted weeks and tens of thousands of dollars.

Why Backfills were glacially slow

I kept asking myself: “This isn’t even Big Data. It is not Netflix-scale… therefore, why so slow?” A few realizations:

• Monolithic joins: Spark spent most of its time shuffling data across executors for each join.

• Lack of decoupling: The join logic (inputs) and the calculation logic (P&A library) were tightly coupled, with every change rippled across the entire dataset.

• Zero incrementalism: No opportunity to reuse intermediate results; every run was a full historical sweep.

A solution: Staging Tables & Incremental Orchestration

The breakthrough came when I introduced a staging table and materialize all raw P&A inputs into one “inputs” dataset.

Read more

The content can be found both on the DataExpert.io Community Academy, as well as on YouTube!

If you want to take it for credit, you will need to watch through the DataExpert.io platform and submit your homework there as well. AI will automatically grade your homework and…

Read more

Conceptual knowledge is becoming king in 2025 and onward!

Read more