In 2026, Data Engineering is no longer about pipelines1. When your LLM writes SQL faster and better than you, when it debugs infra errors, when it can even write the logging code — we can’t pretend otherwise any longer. So what’s left?

Our job is to figure out what product (data) will find market fit with our customers (coworkers). And then build it.

And if they don’t use your stuff — that’s your fault, not theirs.

First, a story. Does it sound familiar?

A data engineer is tasked with fixing a swamp of google sheets, bespoke code, copies of prod databases, and random tables. They read books. They do research. They will Do It Right. They’re still setting things up by the time management loses faith in them. If they’re lucky, the project is cancelled. If they’re unlucky, they’re fired.

Variations of this story happen across the industry every day. They happen because we haven’t taught data engineers the big secret to the job – you may think you’re “just” a version of backend engineer. No. Data Engineers are product owners for data.

And in the AI age, it’s getting more and more obvious.

Our robot minions make the engineering grunt work easier2. That means everything else got more important: engineering strategy, and the people/product stuff.

Historically, the path from “here’s a data swamp no one uses or trusts, fix it” to “wow, thank you, here’s a promotion and the key to the city” has been tough. You gotta do 2/3 of one person’s job doing the human-relationship PMish stuff, and 2/3 of one person’s job doing the engineering. Plus another 2/3 of a person’s job responding to stochastic urgent requests from your team.

Finally, there’s a way forward. The job has never been more fun.

Beyond the warehouse.

This is part 3 of our “The Data Warehouse Setup No One Taught You” series. So far, we’ve focused on a key set of tools – pipelines, the warehouse, and ingesting data in and out. Mostly expressed via SQL.

But of course, there’s a whole galaxy of other types of tools!3 Enough material that it’d take weeks of coursework just to cover it all. We’ll get into the most important of these in a future post in this series. (Part 4? Part 5?)

But that’s not the point right now. There’s an invisible universe of skills, consideration, and work that DEs do outside of tools. Outside of the “engineering” space. Things that involve judgement, office politics, product sense, communication, and a whole different type of thinking. That’s what most trainings, blog posts, and textbooks completely miss, and that’s what we’re covering today.

You are a data PM. PMs are strategic. You are strategic.

Like it or not, you’re part of the product and management team.

They will set goal metrics for the team – and if you don’t step in, they might set really stupid metrics that actively push everyone in the wrong direction. We all have a sacred duty to stop this from happening.

They make daily decisions such as cutting features, prioritizing between stability and iteration, resourcing teams, or whatever else. This is all based on their limited understanding of *your data*. Meanwhile, you’re constantly making data modeling decisions big and small. You need an intimate sense of your team’s product and users – what is, what could be, and what should be.

If their assumptions don’t match yours, the team/org/company/product will be in pain.4 If they don’t trust your data, nothing else matters. And if they don’t trust you, then they will not trust your data.

If all you do is respond to tickets and build artifacts, your career will suffer, your team will suffer, and you will have besmirched the honorable mission of our shared craft. Not to mention, a robot might take your job in a few months.

Say you inherit a small warehouse setup with easily a year’s worth of obvious work ahead of you. You’re thrust into a situation with an existing data system that people don’t really trust.

“I’m going to roll up my sleeves and make more tables” is not the answer to a small warehouse.
“I’m going to write data quality checks and validate our flows” is not the complete answer to untrusted tables.

A data engineer’s job is to help their customers (coworkers) make the right decisions.

To do that:

1. Understand the data and data system under your purview

2. Understand the customers of that data and system

3. Identify the most important use cases, customers, bottlenecks, and improvements for the above

4. Prioritize and deliver

5. Market, sell, iterate, and even showboat …

6. So that customers are happier, trust the data and system, and use it to make key decisions

   1. Bonus: so that more customers join in

Many data engineering manuals focus on #1 and #2. Maybe #4. Meh.

True senior data engineers know what to build, for whom. And then they make goddamn sure it’s not just used, but making an impact.

Showmanship is good, actually.

A story. I was three months into being a DE at Facebook and I was CONFUSED.

What was my job, exactly? What were the types of outputs that I was supposed to make? Tables and pipelines seemed to be part of it. Charts too. But no one really spelled it out.

I met my skip. He asked me what I was working on. I showed him a table I was painstakingly building and polishing. Our team was tracking how users (often abroad, often on android) were upgrading (or not) their version of the app. The table tracked each user/app/device tuple per day, so we could both analyze and instrument metrics around app upgrades.

He told me: “you think you’re making a table for your team. That’s fine. But actually what you’re making is a table that’s useful for the entire company. Rename it to something more general and tell people about it.”

I did it. If I recall correctly my blog post was titled something like “One Weird Trick for Much Better Growth Statistics”. It was my first little viral hit inside the company. Years later I’d still get calls from coworkers I never met, telling me how the table was core to their work and asking about the finer details.

The difference between a table that only three people in my team cared about and the viral hit that materially helped the company5 was just … pizzazz? (Plus a bit of insight)

To trust your data, they need to trust you

What I learned from the most successful DE I ever met

The data engineer I’ve seen reach the highest pinnacle of career success, the one who has so much power and influence that they’re known to power players in DC – I knew him way back when we were both IC’s at Facebook. Let’s call him Moriarty.

I’ve thought a fair bit about how he pulled it off. Moriarty had two key moves. The first was more minor, and mildly technical6. But the other key move was, in retrospect, killer.

Moriarty never let a question go unanswered for more than a few minutes. I thought maturity meant blocking off time to focus and batching my responses to pings. He did the opposite. He sacrificed everything on the altar of being maximally available, helpful, and low-latency to key members of his team.

While I thought my job was making data pipelines (“it’d be rude to query them and step on the toes of data scientists”), he made sure to fluently answer any data or analytics question within minutes. Infra, analysis, whatever. And if it took more than a few minutes, respond with an ETA.

Honestly, I wish I had followed his lead.

Yes, your data needs to be trustworthy, but that’s different than being trusted.

What does trust mean? Does it mean timely numbers, metrics that can be defended, or numbers that match their numbers? Or simply that you’re pingable and reliable?

It depends on your workplace, but some things seem universal.

Let’s go back to the data swamp. Imagine you’re trying to disentangle a system that people simply don’t have faith in.

To trust you, they need to trust your priorities

In your company/org/team, what do your customers actually pay attention to? What do they use as the numbers they actually refer to again and again?

Often, it’d be a surface that you don’t actually control. I’ve seen it be an automated daily email. I’ve seen it be a Google Sheet. I’ve even seen it as an Excel doc that someone compiles each day and emails around.

That’s the thin pillar of truth that the company brass are holding on to.

If you threaten that pillar (even in favor of your shiny new solution that’s better for XYZ reasons), they will react with overwhelming force.

Here’s the mantra: The surface they obsess about, you obsess about.

There’s an engineering lens that junior DEs often have. “This is a jank system”, they might say. “Why don’t we move this process7 to our actual data infra? It’s technically superior and easier and automatable in all these good ways”. BEWARE! If you try to blow it up and tell them to use your dashboard instead, you are asking for trouble!8

Leave a comment

It’s better to use a product lens. Here’s what that might look like:

In this example, our customers are the C-suite. They like surface X (let’s say a Google sheet). Your job is not just to understand why they like the metrics/charts/numbers on that sheet, but to get to a place where they are sure that you understand why. Then have a gentle, iterative, plan to upgrade that surface. Maybe it’s piping in better numbers. Maybe it’s doing some caching work in the backend to get the sheet to load faster or be redundant.

Didn’t think your job involved making DQ checks for a Salesforce export? At the most senior levels, your job is to build trust with the people who matter so that they give you the go-ahead to do the bigger stuff.

The customer is not always right.

None of this means “only do what my coworkers want”, or “the job is just being a fancy analyst, no engineering required”.

The point of credibility is to spend it.

• To argue for better goal metrics for your team

• To replace bad numbers with better ones

• To get support for the infra you need to build

• To get buy-in on investing in your data model and architecture

• To do what you do best

Prioritization: aka how I stopped worrying and learned to avoid overengineering.

It’s all about prioritization. It’s about asking WHY.

Here’s who might matter:

• The PM, EM, or other leads on your product team (if you’re in a product team)

• The c-suite, VPs, executives, founders (if you’re at the right kind of company)

• Finance!

• SWEs, Ops teams, and others who are day-to-day consumers of your product

And here’s what might matter, in roughly descending order, depending on your office culture:

• Making the company more money

• Moving impact metrics upwards

• Making metrics to track impact

• Helping others to look good / make decisions / do their job

• Avoiding losing the company money

• …. (big gap here)
  …
  … (yep, still a gap)
  …

• The aesthetic beauty of your code, system, setup, etc

A story about what matters

I was once young and foolish. I was working at a series B startup, and I had Plans. We were going to make a data warehouse. We were going to use Spark, the new hotness. I spent a month laying out every table we would need.

Reader, I never built those tables. We didn’t need Spark.

The thing I actually did, the thing actually worth putting on my resume, was just this: spending a few days going through the janky, messy, weird data in the prod replica database. Dumping it. Aggregating it via raw python. Manually finding the business metrics most useful for our pitch deck for getting to series C.

Thanks in part to me, we got that series C. It had nothing to do with big data, machine learning, causal inference, whatever. Just counting, averaging, and understanding metrics. Python scripts. That’s what we actually needed.

Whatever your job (engineer? lawyer? executive chef? marine biologist?) – you gotta know who your actual customers are. What they need from you. And deliver it.

Often, what they need might be:

• Analysis now rather than a table that makes analysis easier next week

• Prove that you can match previous (bad) numbers exactly before you show your new better numbers and account for why they’re different

• A straight, raw arithmetic answer with 1 minute latency during a meeting vs a nuanced margin-of-error answer after it’s over

Maybe the key is to have lots of tables, but each can be less trusted. Or vice versa. Or maybe it’s about chart latency. Or something else. Depends on the role!

Now, I’m not saying deliver what they ask for. We have to figure out what they actually need. It’s hard to put it into special frameworks or charts. It’s wisdom. It comes from experience.

But that’s the job.

DataExpert.io and Adal are launching a free vibe coding hackathon on July 4th and 5th! The winners of this hackathon will get into DataExpert’s next Databricks bootcamp for free! ($3000 value). You can join here

Thanks for reading. You can always reach me here.
Yours truly, Sahar Massachi.

This article is brought to you by:

Eon and Google Cloud are hosting BigQuery Day

Read more

While Leetcode has historically been important for filtering candidates out, it has zero impact on the level candidates are hired at in Big Tech. The behavioral round and leadersh…

Read more

• Consistent correct answers and hallucinations

• Low-latency answers and waiting forever

• Maxing out your token limit in a day and being able to use a cheaper model

In this article, we will go over ho…

Read more

Read more

But if you zoom out and look at the product direction over the last few years, a much different pattern emerges.

Here are three observations that show that Databricks has moved on from data engineers:

• Physical Data Modeling is a skill that doesn’t matter (a quick YouTube video on WHY IT STILL DOES)

• Databricks spent over a $1B on Tabular to slow its progress

• Most businesses don’t want to learn data. They want to get value from it

1. Physical data modeling is a skill that no longer matters (according to Databricks)

Since the early 2010s, Data engineers have been physically modeling data via partitioning, sorting, bucketing, etc.

These types of physical data modeling techniques created things like zorder, SMB join (a join without shuffle), file compactions, and data index skipping.

Now, Databricks says “don’t partition or sort or bucket” your tables. Your expertise is no longer needed here.

Instead of partitioning or bucketing, you now have “liquid clustering.”

Instead of file compaction and table layouts, you now have “predictive optimization.”

Databricks has steadily abstracted away the entire physical data modeling skillset. This is fantastic for analysts.

They don’t want to think about small files, metadata bloat, or clustering keys. They want queries to run fast and solve business problems even faster.

But engineers?
Engineers care about mechanics. File pruning behavior. Snapshot retention. Metadata growth. Table property tuning. Failure modes.

The philosophical shift is clear: reduce footguns by reducing access to the gun.

Once you upgrade beyond Databricks 13.3, you can retire your physical data modeling skills and put them in the trash.

2. Databricks spent $1B on Tabular to slow Iceberg’s progress

With open-source Iceberg, you get:

• Data portability

• Hidden partitioning

• Deterministic file layout (You decide the sorting, the partitioning, and the bucketing)

• File compaction

• Snapshot management

Nearly two years after the acquisition, you can create managed Iceberg tables in Databricks. But you cannot optimize them. You cannot call file compaction. You cannot manage the snapshots. You cannot use hidden partitioning.

Databricks’ support of Iceberg is flawed.

• It doesn’t support the hidden partitioning using things like days(event_time), year(event_time), etc

• Managed Iceberg doesn’t allow you to do your own file compaction. You can do auto file compaction with liquid clustering only

• To use managed Iceberg, predictive optimization HAS TO BE ENABLED.

  

All of this is saying, “We bought Tabular not to support the strengths of Iceberg.”

When Databricks deprecated the Tabular metastore, I migrated my 13,000 tables to Glue catalog instead of Unity Catalog because its offering was flawed, and I prefer to teach true physical data modeling, not “Databricks magic.”

Data engineers love determinism, explicit table layouts, visibility, and control.

Databricks now offers ease of use, unpredictable table layouts, opacity, and a lack of control.

Don’t get me wrong. This isn’t a bad thing for Databricks. The trade-off makes their product more available to less technical people, but it alienates experts who know what they’re doing.

3. Most businesses don’t actually want to learn data.

Data is seen as an impediment to business value. It’s raw, unrefined, and very easy to do wrong.

Also, data engineers are expensive, burn out quickly, and are increasingly automatable. The parts of the job that companies used to pay a premium for: writing boilerplate Spark jobs, tuning clusters, managing file layouts, optimizing joins. These are exactly the things AI can now scaffold in seconds.

What most companies actually want is not better data engineering.

They want:

• Dashboards that load instantly

• Models that work

• Pipelines that don’t break

• Costs that don’t spike unpredictably

• Governance without reading a whitepaper

They don’t want to debate partitioning strategies.
They don’t want to argue Iceberg vs Delta.
They don’t want to tune shuffle partitions.

They want it to just work.

If you have:

• Analysts writing transformations in SQL

• AI generating pipeline code and tests

• Databricks handling optimization (Liquid Clustering, Photon, Predictive Optimization)

• Unity Catalog abstracting governance

You’ve effectively collapsed the traditional data engineering function into the platform.

That combination is extremely powerful — and usually cheaper than hiring a senior data engineer.

From an executive’s perspective, this is obvious:

• Lower headcount

• Faster iteration

• Fewer specialized bottlenecks

• Less infrastructure decision risk

This is the Snowflake playbook, and Databricks loves to follow their lead

The uncomfortable truth for data engineers is this: The market is rewarding abstraction, not control.

The companies growing fastest aren’t selling “deep control over distributed systems.”

They’re selling “you don’t have to think about it.”

And most businesses will choose that every time.

So what does this mean?

Databricks isn’t bad for data engineers. But teaching Databricks has become increasingly more boring.

When the answer to the question: “How to make my job more efficient?” is “Just let Databricks handle it.” It undermines the value of truly understanding how data works.

The pendulum is swinging in favor of analysts with these changes. The closer you are to the business, the more valuable you will become. Because Databricks is boldly telling you, “Your physical data modeling skills are no longer needed.”

I want to leave you with a few questions. Please comment on what you think:

• With Snowflake and Databricks both leaning into “it just works” magic, where is the home for the data experts who actually know what they are doing?

• How will we discover new physical data modeling techniques if all the control is ceded to Databricks?

If AI can generate pipelines, DAGs, tests, and even migrations…
What’s left for data engineers to actually work on?

Conceptual knowledge is no …

Read more

In this article, we will attack the following beliefs:

• Only Spark can be used for terabytes of data (or it is ALWAYS the best choice)

• You need a lot of time to process TBs of data

We want to leave your head spinning at the end of this article. Wondering if everything you learned about MapReduce was wrong!

This Article is brought to you by

We want to give a shout-out to MotherDuck, who is sponsoring this article and providing the infrastructure for the benchmarks!

You Said to Use DuckDB On Small Data

Previously, I was a champion of using DuckDB for any dataset that was “small” (< 20GBs). Recently, I was challenged on that remark on LinkedIn by some astute data engineers, who said I had a misconception about what DuckDB was capable of. Being the curious data engineer I am, I took a bite on that bait and decided to roll up my sleeves and benchmark much larger datasets.

But how much larger?

Also - if you want to subscribe to Matt’s Substack, you can click here.

I first decided to go after ~200 GBs. DuckDB read that data in <10 seconds.

This was too fast. It felt magical. What about 500 GBs? Then I hit a wall: a physical wall. The hard drive on my Mac M2 didn’t have enough space for 500GBs. I strolled to my local Best Buy and picked up this thing:

Side Note - A 4 TB external hard drive might seem like overkill; this was one of those “Go big or go home” moments. I figured in my mind “well if 500gb works, I want to have enough runway for much larger tests down the road”

I created a 500GB dataset on the external drive in DuckDB. It read that data in ~40 seconds

This made me realize I needed to set my sights on the big kahuna. 1 full TB of data!

Building A 1 TB Dataset for DuckDB

In my previous articles, you will see that I use a script that leverages DuckDB’s' generate_series' function to generate rows of data quickly. The gist of it looks like this:

It’s straightforward:

• You pass in a row count and let it generate a Parquet file.

• If we want to do this at scale and not wait several hours (or go dreaded serialized). What do you do?

• Bring in the good ol’ Python ProcessPoolExecutor and go parallel. (code here)

Ok, But Did You Really Generate A Full TB Of Data?

Yes, it took my M2 Pro (16GB of RAM) ~70 minutes to fry this egg with 10 workers in parallel. Here’s the proof:

The dataset is: 400 files, each ~2.76GB in size.

So now, without further ado, let’s get cracking and run some benchmarks on this.

Side Note - If you have not moved from the old Python virtual environment of “-m venv” over to UV, do yourself a favor and do it now; uv loads packages faster, makes targeting specific Python environments easier; I could go on…you’ll thank me later

The Benchmark…And What Exactly Are You Doing Here?

Today’s benchmark will:

• run a common aggregation query across the 1TB dataset;

• It will group by a date, count rows, and sum a value.

This is a common analytics query I have seen in my last two decades as a data engineer and BI leader; I did not cherry-pick this to just make DuckDB look good. This is what the benchmark query boils down to:

For our benchmark, we will run it 5 times. Below are the results:

Average processing time locally: 1 minute, 29 seconds.

Hold On - You Said We Could Crush A Full TB In Under 30 Seconds!?

I did say that. This first benchmark was on my laptop and local computer, which is impressive.

What would happen if I were to create a full 1TB dataset in MotherDuck and try this again?

Time To Join The Flock

On MotherDuck, we have excellent options to load data. We could do stuff like:

• store CSVs and Parquet in S3/Azure/GCP

• import the TCP-H dataset

• Use our local CLI to generate data

For this article, I chose the third option. Here is the script that created the 1TB dataset in MotherDuck:

I created a view in a MotherDuck database that leveraged the generate_series function (like in the previous local benchmark). After that, I ran the script to iterate over and insert the data multiple times.

After 10 iterations, I saw it wasn’t quite at 1TB; I manually ran the load process several times more until I got 1TB.

We now have over 1TB of data. Now it’s time to choose our compute capacity.

In MD, we have four standard options for compute capacity: Pulse, Standard, Jumbo, and Mega. I went with Mega, given we are dealing with a full TB of data:

For the benchmark, we ran the exact same query we did locally, but with a caveat;

• MotherDuck has intelligent caching; running the same query five times will have results 2-5 be about 5 seconds or less because it will read from cache vs. actually scanning the data.

So how do we get around that?

• Simple - for each iteration, we will have our aggregation values sum a different lower and upper range, which makes the query non-deterministic, and removes the ability for it to just hit the cache layer. The query template looks like this:

Below are the results of our benchmark:

Holy Smokes -

• That was clocking in at under 17 seconds on average. You might also ask - “Hey what is that run 0 a.k.a the cold start?”

  • Remember, we are now dealing with a cloud data warehouse, which will not always keep our data readily available to rip from RAM; sometimes, it will be on disk and have to get read in; thus, the iteration 0 is a warm up run, incase its having to read off of disk for the first usage; for datasets you will query often in MotherDuck, this won’t be an issue, as your data will more than likely be ready to process and sit in hot ram.

Ok, Great Stuff! We Are Done, Right?

We blew our promise of scanning a full TB in under 30 seconds out of the water, by a factor of 2. But what if we wanted faster?

I’m Ready - Let’s Go Deeper

DuckDB supports indexes, but they don’t really push the concept much. The Zonemap index is a secret weapon that allows you to take advantage of pre-sorted data through min/max tracking of the metadata.

How would I implement these zone maps?

• Let’s reload our dataset where we sort and insert on the rand date. The load process looked like this:

Once that was complete, I created another benchmark, and here were the results:

The sorted (zonemap) dataset improved our benchmark time by roughly 30%. That is an amazing tweak, simply by loading the table in a sorted order by the field we are grouping by.

Summary

This article showcased a paradigm shift in DuckDB’s capabilities! We have now shattered the belief of what “small” data is and what the duck can do. Even on my local laptop, we were still scanning 1TB of data in <2 minutes. If my batch jobs refreshed my reports in 2 minutes without Spark, I would be very happy!

Here are all the code examples.

• Local Data Generator

• Motherduck Data Generator (Unsorted)

• Motherduck Data Generator (Sorted)

• Local Benchmark

• Motherduck Unsorted Data Benchmark

• Motherduck Sorted Data Benchmark

Thanks again, MotherDuck, for providing us with the environment to showcase this capability!

Thanks for reading and happy holidays! If you found value in this article, make sure to comment and share with your friends!

Matt and Zach

Data engineers sit at the intersection of value and vulnerability. You build the pipelines that power analytics, AI, and business decisions. But the same systems that enable innovation also become prime targets for attackers.

You don’t need to become a full-time security engineer to protect your data. But you do need to think like one — because no amount…

Read more

Sahar and I learned a lot during out time working in core growth together at Facebook working in friending and notifications.

Let’s talk about the pain of unlearning and then let’s get to the magic.

Imagine you’re an analyst at a social media company. The retention team asks: “For users who now have 1000+ followers but had under 200 three months ago – what device were they primarily using back then? And of the posts they viewed during that growth period, how many were from accounts that were mutuals at the time?”

You need to join user data (follower counts then and now), device data (primary device then and now), relationships (who was a mutual then vs now), and post views – all as of 3 months ago.

With most data warehouse setups, this query is somewhere between “nightmare” and “impossible.”  

You’re dealing with state, not actions. State in the past, across multiple tables. There’s a word for this problem – slowly changing dimensions. Whole chapters of textbooks deal with various approaches. You could try logs (if you logged the right stuff). You could try slowly changing dimensions with `valid_from/valid_to` dates. You could try separate history tables. All of these approaches are painful, error-prone, and make backfilling a living hell.

There’s a better way. Through the magic of ✨datestamps✨ and idempotent pipelines, this query becomes straightforward. And backfills? They become a button you push.

Part 1 fixed weird columns, janky tables, and trusting your SQL. Part 3 will cover scaling your team and warehouse. But now – now we fix: backfills, 3am alerts, time complexity, data recovery, and historical queries.1

The old way was a mess

Here’s what most teams do when they start out:

Option 1: Overwrite everything daily

Your pipeline runs every night, updates dim_users with today’s snapshot, overwrites yesterday’s data. Simple! Until six months later when someone asks “how many followers did users have in March?” and you realize: that data is gone. You have no history. You can’t answer the question. Oops.

(Jargon alert – Apparently this is SCD Type-1 ¯\_(ツ)_/¯ )

Option 2: Try to track history manually

Okay, you think, let’s be smarter. Add an updated_at column. Or maybe valid_from and valid_to dates, with an is_current flag. When a user’s follower count changes, don’t update their row – instead, mark the old row as outdated and insert a new one.

(Jargon alert – This is SCD Type-2. Booo)

This is better! You have history. But now:

• Your pipelines need custom logic to “close out” old rows before inserting new ones

• If you mess up the valid_to dates, you get gaps or overlaps in history

• Backfilling becomes a nightmare – you can’t just rerun a pipeline, you need to carefully update dates without breaking everything downstream

• Querying becomes a nightmare. To get user data “as of 3 months ago”, you need:

SELECT * FROM dim_users WHERE user_id = 123 AND valid_from <= ‘2024-10-01’ AND (valid_to > ‘2024-10-01’ OR valid_to IS NULL)

Now imagine joining MULTIPLE historical tables (users, devices, relationships). Every join needs that BETWEEN logic. Miss one and your results are silently wrong. Get the date math slightly off and you’re joining snapshots from different points in time. Good luck debugging that.

Option 3: Separate current and history tables

Some teams maintain dim_users (current snapshot) and dim_users_history (everything else). Now you’ve got two sources of truth to keep in sync. Analysts need to remember which table to query. Any analysis spanning current and historical data requires stitching across tables with UNION ALL. It’s a mess.

And, depending on how the dim_users_history table works – it won’t solve any of the problems you’d have in option 2!

All of these approaches share a problem: they’re trying to be clever about storage. They made sense when disk was expensive. They don’t anymore.

(Jargon alert – This is SCD Type-4. Note that I didn’t know this when I started writing this blog post because it’s useless, boring, outdated jargon. Ignore it.)

Sponsorship

If you want to learn more about data modeling and data architecture in detail, you can use code SCDSUCKS by November 14th to get 35% off the DataExpert.io Snowflake + dbt boot camp

The new way: Just append everything

You solve it with date stamps. You solve it with “functional data engineering”.

What you really want is a sort of table that tracks state – a dimension table –, but where you can access a version that tracks information about the world today, and another version that tracks information about the world in the past.

Maxime Beauchemin wrote the seminal public work on the idea here. But, honestly, I think the concept can be explained more plainly and directly. So here we are.

The thinking goes like this:

• We’re getting new data all the time.

• Let’s simplify it and say – we get new data every day. We copy over snapshots from our production database each evening.

• There are complex, convoluted ways to keep track of what data is new and useful, and what data is a duplicate of yesterday.

• But wait. Storage is cheap. Compute is cheap. Pipelines can run jobs for us while we sleep.

• It’s annoying to have a table with the data we need as of right now, and either some specialized columns or tables to track history..

• Instead, what if we just kept adding data to existing tables? Add a column for “date this information was true” to keep track.

Here’s what it looks like in practice. Instead of overwriting your dimension tables every day, you append to them:

dim_users
┌─────────┬───────────┬────────────┐
│ user_id │ followers │ ds         │
├─────────┼───────────┼────────────┤
│ 123     │ 150       │ 2024-10-01 │
│ 123     │ 180       │ 2024-10-02 │
│ 123     │ ...       │ ...        │
│ 123     │ 1200      │ 2025-01-16 │
└─────────┴───────────┴────────────┘

dim_devices
┌─────────┬─────────┬────────────┐
│ user_id │ device  │ ds         │
├─────────┼─────────┼────────────┤
│ 123     │ mobile  │ 2024-10-01 │
│ 123     │ mobile  │ 2024-10-02 │
│ 123     │ ...     │ ...        │
│ 123     │ desktop │ 2025-01-16 │
└─────────┴─────────┴────────────┘

dim_relationships:
┌─────────┬───────────┬───────────┬────────────┐
│ user_id │ friend_id │ is_mutual │ ds         │
├─────────┼───────────┼───────────┼────────────┤
│ 123     │ 789       │ true      │ 2024-10-01 │
│ 123     │ 789       │ true      │ 2024-10-02 │
│ ...     │ ...       │ ...       │ ...        │
│ 123     │ 789       │ false     │ 2025-01-16 │ ← changed
└─────────┴───────────┴───────────┴────────────┘

fct_post_views:
┌─────────┬───────────┬───────────┬────────────┐
│ post_id │ viewer_id │ poster_id │ ds         │
├─────────┼───────────┼───────────┼────────────┤
│ 5001    │ 123       │ 789       │ 2024-10-01 │
│ 5002    │ 123       │ 456       │ 2024-10-01 │
│ 5003    │ 123       │ 789       │ 2024-10-05 │
│ ...     │ ...       │ ...       │ ...        │
│ 9999    │ 123       │ 789       │ 2025-01-15 │
└─────────┴───────────┴───────────┴────────────┘

Now that impossible retention query becomes straightforward. No BETWEEN clauses, no valid_from/valid_to logic – just filter each table to the date you want:

-- For fast-growing users, what device did they use back then?

WITH
  today_users as (SELECT user_id, followers as today_followers
      FROM dim_users WHERE ds = ‘2025-01-16’ AND followers >= 1000),
  past_users as (SELECT user_id, followers as past_followers
      FROM dim_users WHERE ds = ‘2024-10-01’ AND followers < 200),
  past_device as (SELECT user_id, device
      FROM dim_devices WHERE ds = ‘2024-10-01’),
  user_device as (
      SELECT tu.user_id, today_followers, past_followers, pd.device
      FROM past_users pu
      JOIN today_users tu ON pu.user_id = tu.user_id
      JOIN past_device pd ON tu.user_id = pd.user_id),
  views as (
      SELECT post_id, viewer_id, poster_id, ds
      FROM fct_post_views 
      WHERE ds BETWEEN ‘2024-10-01’ AND ‘2025-01-16’)
  SELECT
      ud.user_id,
      ud.device as device_during_growth,
      COUNT(DISTINCT views.post_id) as posts_from_mutuals
  FROM user_device ud
  LEFT JOIN views
      ON ud.user_id = views.viewer_id
  LEFT JOIN dim_relationships past_rels
      ON views.viewer_id = past_rels.user_id
      AND views.poster_id = past_rels.friend_id
      AND views.ds = past_rels.ds -- mutual status AS OF view date
      AND past_rels.is_mutual = true
  GROUP BY 1, 2

Is this query complex? Sure.2 But the complexity is in the business logic (what you’re trying to measure), not in fighting with valid_from/valid_to dates. Each query just filters to ds = {the date I want}. That’s it.

The idea is that you’re not overwriting existing tables. You are appending. 3

Sidebar: Common Table Expressions

If I had a SECOND “one weird trick” for data engineering, CTEs would be it. CTEs are just fucking fantastic. With liberal use of common table expressions (the WITH clause you saw in the retention query above), you can treat subqueries like variables – and then manipulating data feels more like code. Make sure your query engine (like Presto/Trino) flattens them for free – but if it does: wowee! SQL just got dirt simple. (a free one hour course on CTEs here)

When you grab data into your warehouse4, append a special column. That column is usually called “ds” – probably short for datestamp. You want something small and unobtrusive. (Notice that “date” would be a bad name – because you’d confuse people between this (date of ingestion of data) and the more obvious sort of date – date the action happened.) For snapshots, copy over the entire data of the snapshot, and have your “ds” column be <today’s date>. For logs, you can just grab the logs since yesterday, and set the ds column to <today’s date>.

Sidebar: Date stamps vs Date partitions
I’ll mostly say “date stamps” in this piece – the concept of marking each row with when that data was valid/ingested.

“Date partitions” is how most warehouse tools *implement* date stamps. A partition is how your warehouse physically organizes data. Think of it like: all rows with ds=2025-01-15 get grouped together in one chunk, ds=2025-01-16 in another chunk, and so on. (In older systems, each partition was literally a separate folder. Modern cloud warehouses abstract this, but the concept remains.)

Why does this matter? When you query `WHERE ds=’2025-01-15`, your warehouse only scans that one partition instead of the entire table. This makes queries faster and cheaper (especially in cloud warehouses where you pay per data scanned).

People use the terms interchangeably. The important thing is the concept: tables with a date column that lets you query any point in history.

Every table emanating from your input tables should add a filter (WHERE ds={today}), and similarly append the data to the table (WHERE ds={today}). (Except special circumstances where a pipeline might want to look into the past).

That’s it! Now your naive setup (overwriting everything every day) has only changed a bit (append everything each day, and keep track of what you appended when) – but everything has become so much nicer.

This is huge

This has two major implications:

First, many types of analysis become much easier. Want to know about the state of the world yesterday? Filter with WHERE ds = {yesterday}. Need data from a month ago? Filter with WHERE ds = {a month ago}. You can even mix and match – comparing today’s data with historical data, all within simple queries.

Second, data engineering becomes both easier and much less error prone. You can rerun jobs, create tables with historical data, and fix bugs in the past. Your pipeline will produce consistent, fast, reliable results consistently

What “functional” actually means

(Aka “I don’t know what idempotent means and at this point I’m afraid to ask”)

So, in Maxime’s article (link) there’s all this talk about “functional data engineering”. What does that even mean? Let’s discuss.

First, we’re borrowing an idea from traditional programming. “Functional programs” (or functions) meet certain conditions:

1. If you give it the same input, you get the same output. Every time.

2. State doesn’t change. Your inputs won’t change, hidden variables won’t change. It’s clean. (AKA “no side effects”)

Okay, so what does that mean for pipelines? Functional pipelines:

• Given the same input, will give the same output

• Don’t use (or rely on) magic secret variables

This is what people mean when they say “idempotent” pipelines or “reproducible” data.

And here’s how to implement it: datestamps.

• Your rawest/most upstream data should never be deleted – just keep appending with datestamps

• Pipelines work the same in backfill mode vs normal daily runs

• If you find bugs, fix the pipeline and rerun – the corrected data overwrites the bad data

• Time travel is built in – just filter to any ds you need

Datestamps also give you the nice side-effect of having it be very clear how fresh the data you’re looking at is. If the latest datestamp on your table is from a week ago -- it’s instantly understandable not only what’s wrong, but also you have hints about why.

Sidebar – what this looks like in practice:
Your SQL will look something like: WHERE ds=’{{ ds }}’ (Airflow’s templating syntax) or WHERE ds=@run_date (parameter binding).

Your orchestrator injects the date - whether it’s today’s scheduled run or a backfill from three months ago. Same SQL, different parameter. That’s the whole trick.

Backfilling is now easy, simple, magical

Remember that retention query? Now imagine you built that analysis pipeline three months ago, but you just discovered a bug in your dim_relationships table. The is_mutual flag was wrong for two weeks in November. You fixed the bug going forward, but now all your retention metrics from that period are wrong.

With the old SCD Type-2 approach, you’re in hell:

You can’t just “rerun November.” Because each day’s pipeline depended on the previous day’s state. Day 15 updated rows from Day 14, which updated rows from Day 13, and so on. To fix November 15th, you’d need to:

1. Rerun November 1st (building from October 31st’s state)

2. Wait for it to finish

3. Rerun November 2nd (building from your new November 1st)

4. Wait for it to finish

5. Rerun November 3rd...

6. ...keep going for 30 days, sequentially, one at a time

And this is assuming nothing breaks along the way. If Day 18 fails? Start over. Need to fix December too? Add another 31 sequential runs.

In Airflow terms, this is what depends_on_past=True does to you. Each day is blocked until the previous day completes. Backfilling becomes painfully slow. But that’s by no means the worst part.

You can’t just hit “backfill” and walk away. Your normal daily pipeline logic doesn’t work for backfills. Why? Because SCD Type-2 requires you to:

• Close out existing rows (set their valid_to date)

• Insert new rows (with new valid_from dates)

• Update is_current flags

• Handle the case where a row changed multiple times during your backfill period

Your daily pipeline probably has logic like:

-- Daily SCD Type-2 pipeline (simplified)
-- Step 1: Close out changed rows
UPDATE dim_users
SET valid_to = CURRENT_DATE - 1, is_current = false
WHERE user_id IN (
SELECT user_id FROM users_source_today
WHERE <something changed>
)
AND is_current = true;

-- Step 2: Insert new versions
INSERT INTO dim_users (user_id, followers, valid_from, valid_to, is_current)
SELECT user_id, followers, CURRENT_DATE, NULL, true
FROM users_source_today;

This works fine when you’re processing “today.” But for a backfill? You need different SQL:

• You need to carefully reconstruct valid_from/valid_to for historical dates

• And handle the fact that a user might have changed multiple times during your backfill window

• This gets messy fast.

• You’re essentially rewriting your pipeline. (WHY?)

So now you’re not just waiting 30 sequential days - you’re maintaining two separate codebases: one for daily runs, one for backfills. And every time you change your daily logic, you need to update your backfill logic to match. More code to write, more code to test, more places for bugs to hide. It’s completely useless and unnecessary.

Sidenote – even worse, if you’re outside your retention window (say, the source data from 90 days ago has been deleted), you can’t backfill at all. You’d need to completely rebuild the entire table from scratch, from whatever historical snapshots you still have. Which probably means... datestamped snapshots anyway. Womp womp.

With datestamps, backfilling is trivial:

Your pipeline for any given day just needs:

• Input tables filtered to ds=’2024-11-15’ (or whatever day you’re processing)

• Write output to ds=’2024-11-15’

That’s it. November 15th doesn’t need November 14th. It just needs the snapshot from November 15th.

So to fix your broken November data:

# In Airflow (or whatever orchestrator)
> airflow dags backfill my_retention_pipeline \--start-date 2024-11-01 \--end-date 2024-11-30

What happens behind the scenes?

• All 30 days kick off in parallel (up to your concurrency limits)

• Each day independently reads from its ds partition

• Each day independently writes to its ds partition

• No coordination needed between days

• The whole month finishes in the time it takes to run one day

The exact same SQL that runs daily also handles backfills - no special logic, no custom code

This changes everything:

No more custom SQL for backfills - It’s just a button you push. Your orchestrator handles it. The same pipeline code that runs daily also handles backfills. No special logic needed.

New tables get history for free - Created a new dim_users_enriched table today but want to populate it with the last year of data? Just backfill 365 days. Since your input tables have datestamps, the data is sitting there waiting.

Bugs in old data become fixable - Fix your pipeline logic, backfill the affected date range, done. The old (wrong) data gets overwritten with the new (correct) data for those specific partitions. Everything downstream can reprocess automatically.

Upstream changes cascade easily - Fixed a bug in dim_users? All downstream tables that depend on it can backfill the affected dates in parallel. The whole warehouse stays in sync.

This is possible because your pipelines are idempotent. Run them once, run them a thousand times - given the same input date, you get the same output. No hidden state, no “current” vs “historical” logic, no manual date math.

One pattern to avoid: Tasks that depend on the previous day’s partition of their own table. If computing today’s dim_users requires yesterday’s dim_users, you’ve created a chain - backfilling 90 days means 90 sequential runs that can’t be parallelized. This is sometimes necessary for cumulative metrics, but most dimension tables don’t need it - just recompute from raw sources each day.

For most datestamped pipelines, depends_on_past should be False. Each day is independent - the only dependency is “does the upstream data exist for this ds?”

Welcome to the magic of easy DE work

We started this article staring at the prospect of valid_from/valid_to logic, sequential backfills that take days, and custom SQL for every backfill and cascading for every bugfix. Yuck. Ew!

Or maybe – worse – with no sense of history at all. No ability to ask “how did the world look like yesterday”, much less “3 months ago”. I’ve seen startups and presidential campaigns and 500 million dollar operations operate like this. 🙃

Now you know the secret. Now you have the magic. What mature companies have been doing all along: snapshot your data daily, append it with datestamps, and write idempotent pipelines on top.

That’s it. That’s the whole One Weird Trick. Add a ds column to every table. Filter on it. Write your pipelines to be independent of each other. Have every pipeline be ds-aware. Storage is cheap. Your time is expensive. Getting your data wrong is extra expensive.

What you get in return:

• Backfills that run in parallel and finish in minutes instead of days

• Backfills that are a button push instead of custom SQL mess.

• Historical queries that are simple WHERE ds=’2024-10-01’ filters instead of date-range gymnastics

• Pipelines that are the same whether you’re processing today or reprocessing last year

• A built-in time machine for your entire warehouse

• Bugs that are fixable instead of permanent scars on your data

This is functional data engineering. Functional as in idempotent. And functional as in “it works”.

Your backfills are easy now. Your 3am alerts will be rarer. Time complexity is solved. Data recovery is trivial. Your job just became so much easier.

But we’re not done yet. Part 3 will tackle: how to scale your team and your warehouse. Parts 4 and 5 are gonna get me back on my “he who controls metrics controls the galaxy” soapbox.

For now, go add some datestamps. Your future self will thank you.

Hey, it’s Zach again. Sahar is currently open to work in NYC. Make sure to follow Sahar’s blog to understand growth and what comes next, both personally and for your business! And more at sahar.io

Data engineering can actually be easy, fast, and resilient! All you have to embrace is a simple concept: Date-stamping all your data.

Why isn’t this the norm? Because – even in 2025 — , institutions haven’t really understood the implications that STORAGE IS CHEAP! (And your data team’s time is expensive).

Datestamping solves so many problems. But you won’t find it in a standard textbook. They’ll teach you “slowly changing dimensions Type 2” when the real answer is simpler and more powerful. You will find the answer in Maxime Beauchemin’s seminal article on functional data engineering. Here’s the thing – I love Max, but that article is not helpful to the majority of people who could learn from it.

What if I told you:

• We can have resilient pipelines.

• We can master changes to data over time.

• We can use One Weird Trick to marry the benefits of order and structure with the benefits of chaos and exploration.

That’s where this article comes in. It’s been 7 years in the making – all the stuff that you should know, but no one bothered to tell you yet. (At least, in plain english – sorry Max!)

• Part One: How to set up a simple warehouse (and which small bits of jargon actually matter)

• Part Two: Date-stamping. Understand this and everyone’s life will become easier, happier, and 90% more bug-free.

• Part Three: Plugging metrics into AB testing. Warehousing enables experimentation. Experimentation enables business velocity.

• Part Four: The limits of metrics and KPIs. It can be so captivating to chase short-term metrics to long-term doom.

I’ll show you a practical intro to scalable analytics warehousing, where date stamps are the organizing principle, not an afterthought. In plain language, not tied to any specific tool, and useful to you today Meta used this architecture even back in the early 2010s. It worked with Hive metastore. It still works with Iceberg, Delta, and Hudi.

But first, to understand why all this matters, you need some context about how warehouses work. Then I’ll show you the magic.

Sponsorship

Cut Code Review Time & Bugs in Half

Code reviews are critical but time-consuming. CodeRabbit acts as your AI co-pilot, providing instant Code review comments and potential impacts of every pull request.

Beyond just flagging issues, CodeRabbit provides one-click fix suggestions and lets you define custom code quality rules using AST Grep patterns, catching subtle issues that traditional static analysis tools might miss.

CodeRabbit has so far reviewed more than 10 million PRs, installed on 1 million repositories, and used by 70 thousand Open-source projects. CodeRabbit is free for all open-source repo’s.

Part one — A Simple Explanation of Modern Data Warehousing

Our goals and our context

We are here to build a system that gets all company data, tidily, in one place. That allows us to make dashboards that executives and managers look at, charts and tools that analysts and product managers can use to do deep dives, alerts on anomalies, and a breadth of linked data that allows data scientists and researchers to look for magic or product insights. The basic building blocks are tables, and the pipelines that create and maintain them.

Sidebar: DB vs Data lake? OLTP vs OLAP? Production vs warehouse? Here’s what you need to know.

A basic point about a data warehouse (or lake, or pond, or whatever trendy buzzword people use today) is that it is not production. It must be a separate system from “the databases we use to power the product”.

Both are “databases”, both have “data”, including “tables” that might be similar or mirrored – but the similarity should end there.

• Your production database is meant to be fast, serve your product and users. It is optimized for code to read and write.

• Your warehouse is meant to be human-usable, and serve people inside the business. It is optimized for breadth, for use by human analysts, and to have historical records.

Put it this way – your ecommerce webapp needs to look up an item’s price and return it as fast as possible. Your warehouse needs to look up an item from a year ago, and look at how the price changed over the course of months. The database powering the webapp won’t even store the information, much less make it easy to compute. Meanwhile if you run a particularly difficult query, you don’t want your webapp to slow down.

So – split them. (You might hear people talking about OLTP vs OLAP – it’s just this distinction. Ignore the confusing terminology. Here’s a deep dive into the two types of OLAP data model (Kimball and One Big Table) )

So, we want a warehouse. Ideally, it should:

• Be separate from our production databases

• Collect all data that is useful to the company

• Have tables that make queries easy

• Be correct – with accurate, trusted, information

• Be reasonably up to date – with perhaps a daily lag, rather than a weekly or monthly one

• Power charts and interactive tools, while also being useful for automatic and local queries

This used to be difficult! (It is not anymore!) There was a tradeoff between “big enough to have all the data we need” and “give answers fast enough to be useful”. A lot of hard work was put into reconciling those two needs.

Since circa 2015 or so, this pretty much no longer a problem. Presto/Trino, Spark, and hosted databases (BigQuery, Snowflake, the AWS offerings) and other tools allow you to have arbitrarily huge data, accessed quickly. We live in a golden age.

Sidebar: At my old school…
At Meta, they used HDFS and Hive to power their data lake and MySQL to power production. Once a day they took a “snapshot” of production with a corresponding date stamp and moved the data from MySQL to Hive.

In a world where storage is cheap, access to data can be measured in seconds rather than minutes or hours, and data is overflowing, the bottleneck is engineering time and conceptual complexity. Solving that bottleneck allows us to break with annoyingly fiddly past best practices. That’s what I’m here to talk about.

A basic setup

Imagine your warehouse as a giant box, holding many, many tables. Think of data flowing downhill through it.

• At the top: raw copies from production databases, marketing APIs, payment processors, whatever.

• At the bottom: clean, trusted tables that analysts actually query.

• In between: pipelines that flow data from table to table.

[Raw Input Tables]
├─ users_production
├─ events_raw
├─ transactions_raw [Pipelines]
└─ ... ↓

     Clean → Join → Enrich

                 ↓

[Clean Output Tables]
├─ dim_users
├─ fct_events
└─ grp_daily_revenue

How do we get from raw input to clean tables? Pipelines. (See buzzwords like ETL, ELT? Ignore the froth – replace with “pipelines” and move on).

Pipelines are the #1 tool of data engineering. At their most basic form, they’re pieces of code that take in one or more input tables, do something to the data, and output a different table.

What language do you write pipelines in? Like it or not, the lingua franca of editing large-scale data is SQL. The lingua franca of accessing large scale data is SQL. SQL is a constrained enough language that it can parallelize easily. The tools that invisibly translate your simple snippets into complex mechanisms to grab data from different machines, transform it, join it, etc – they not only are literally set up with SQL in mind, they figuratively cannot do the same for python, java, etc. Why? Because a traditional programming language gives you too much flexibility -- there’s no guarantee that your imperative code can be parallelized nicely.

Sidebar: When non-SQL makes sense (or doesn’t)

If you’re ingesting data from the outside world (calling APIs, reading streams, and so on), then python, javascript, etc could make sense. But once data is in the warehouse, beware anything that isn’t SQL – it’s likely unnecessary, and almost certainly going to be much slower than everything else.

Your tooling might offer a way to “backdoor” a bit of code (e.g. “write some java code that calls an API and then writes the resultant variable to a column”). Think twice before you use it. Often, it’s easier and faster to import a new dataset into your warehouse so that you can recreate with SQL joins what you would have done using an imperative language.

You may be tempted to transform or analyze data in R, pandas, or whatnot – that’s fine, but you do that by interactively reading from the warehouse. Rule of thumb: if you’re writing between tables in a warehouse – SQL. Into a warehouse – you probably need some glue code somewhere. Out of a warehouse – that’s on you.

So here’s the simple setup:

Then, set up a system of pipelines to this, every day, as soon as the upstream data is ready:

• As each of these input tables gets the latest dump of data from outside: take that latest day’s data, deduplicate, clean it up a bit, rename the columns, and cascade it to a nicer, cleaner version of that table. (this is your silver tier data in medallion architecture)

• Then, from that nicer input table, perform a host of transformations, joins, etc to write to other downstream tables. (this is your master data)

• Master data is highly trusted which makes building metrics and powering dashboards easy!1

Every day, new data comes in, and your pipeline setup cascades new information in a host of tables downstream of it. That’s the setup.

A well-ordered table structure

Okay, so to review: the basic useful item in a warehouse is a table. Tables are created (and filled up by) pipelines.

“Great, great,” you might say – “but which tables do I build?”

For the sake of example, let’s imagine our product is a social network. But this typology should work just as well for whichever business you are in – from b2b saas to ecommerce to astrophysics.

From the perspective of the data warehouse as a product, there are only three kinds of tables: input tables (copied from outside), staging tables (used by pipelines and machines), and output tables – also known as user-facing tables.

Output tables (in fact, almost all tables) really only have three types:

• Tables where each row corresponds to a noun. (E.g. “user”, or even “post” or “comment”). When done right, these are called dimension tables. Prefix their names with dim_

• Tables where each row corresponds to an action. Think of them as fancier versions of logs. (E.g. “user X wrote post Y at time Z”). When done right, these are called fact tables. Prefix their names with fct_

• Everything else. Often these will be summary tables. (e.g. “number of users who made at least 1 post, per country, per day). If you’re proud of these, prefix them with sum_ or agg_.

Sidebar: more on naming

YMMV, but I generally don’t prefix input tables. Input tables should be an exact copy of the table you’re importing from outside the warehouse. Changing names breaks that – and an unprefixed table name is a good sign that the table cannot be trusted.

Staging and temporary tables are prefixed with stg_ or tmp_.

Let’s talk more about dimension and fact tables, since they’re the core part of any clean warehouse.

Dimension tables are the clean, user-friendly, mature form of noun tables.

• Despite being focused on nouns (say, users), they can also roll up useful verby information (leveraging cumulative table design)

• For instance, a dim_users table might both include stuff like: user id, date created, datetime last seen, number of friends, name; AND more aggregate “verby” information like: total number of posts written, comments made in the last 7 days, number of days active in the last month, number of views yesterday.

• If a data analyst might consistently want that data – maybe add it to the table! Your small code tweak will save them hours of waiting a week.2

(Now, what’s to stop the table from being unusably wide? Say, with 500+ columns? Well, that’s mostly an internal culture problem, and somewhat a tooling problem. You could imagine, say, dim_user getting too large, so the more extraneous information is in a dim_user_extras table, to be joined in when necessary. Or using complex data types to reduce the number of columns)

Fact tables are the clean, user-friendly, mature form of logs (or actions or verb tables).

• Despite being verb focused, fact tables contains noun information. (Zach chimes in: here’s a free 4 hour course on everything you need to know about fact tables)

• Unlike a plain log, which will be terse, they can also be enriched with data that might probably live in a dim table.

• The essence of a good fact table is providing all the necessary context to do analysis of the event in question.

• A fact table, fundamentally, helps you understand: “Thing X happened at time Y. And here’s a bunch of context Z that you might enjoy”.

• So a log containing “User Z made comment Xa on post Xb at time Y” could turn into a fct_comment table, with fields like: commenter id, comment id, post id, time, time at commenter timezone, comment text, post text, userid of owner of post, time zone of owner of parent post. Some of these fields are strictly speaking unnecessary – you could in theory do some joins to grab the post text, or the comment text, or time zone of the owner of the parent post. But they’re useful to have handy for your users, so why not save them time and grab them anyway.

Q: Wait – so if dim tables also have verb data, and fact tables also have noun data, what’s the difference?

A: Glad you asked. Here’s what it boils down to – is there one row per noun in the table? Dim. One row per “a thing happened?” Fact. That’s it. You’re welcome.

Here, as in so much, we are spending space freely. We are duplicating data. We are also doing a macro form of caching – rather than forcing users to join or group data on the fly, we have pipelines do it ahead of time.

Compute is cheap, storage is cheap. Staff time is not. We want analysis to be fluid and low latency – both technically in terms of compute, and in terms of mental overhead.

Q: Wait! What about data stamps? Where’s the magic? You promised magic.

A: Patience, young grasshopper. Part of enlightenment is the journey. Part of understanding the magic is understanding what it builds on. And – hey – would YOU read a huge blog post all at once? Or would you prefer to read it in chunks. Yeah, you with your Tiktok problem and inability to focus. I’m surprised you even made this far.

Stay tuned for part two where we:

• Show you how to make warehousing dirt easy

• Behold the glory of date stamping

  • Through better data quality

  • How it avoids backfill nightmares

  • Bug fixes are easy now

  • You get a time machine for free

• Explore the dream of functional data engineering (what is that weird phrase?)

• Throw SCD-2 and other outdated “solutions” to the dustbin of history

Make sure to follow Sahar’s blog to understand growth and what comes next, both personally and for your business! Sahar is currently open to work, he is interested in DevRel and engineering leadership roles in New York City.

Knowing where to start and how to get a handle on this requires some guidance. This newsletter is going to unveil the steps needed to break into data engineering in 2025 and how AI fits into the picture.

A lot of people still think that after you’ve done a magical number of Leetcode problems, a job falls into your lap. That’s almost never the case!

To get a job in 2025, you’ll need the following things:

• Demonstrable skills with:

  • SQL and Python

  • Distributed compute (Snowflake, Spark, BigQuery, DuckDB)

  • Orchestration knowledge (Airflow, Mage, or Databricks workflows)

  • Data modeling and data quality

  • AI/data integrations (vector databases, embeddings, RAG pipelines)

• An opportunity to demonstrate those skills via a portfolio project

• A personal brand that radiates above the noise both on LinkedIn and in interviews

Let’s dig into each of these areas and see how you can fast-track your way to success!

Sponsorship

Tinybird helps companies like Vercel and Framer transform massive streams of real-time data into instant insights and real-time user experiences.

They built a free course on real-time data designed for data engineers who want to master the basics and build amazing data products.

Learning SQL

Avoiding SQL is the same as avoiding a job in data engineering. This is the most fundamental language you need to know.

There are many resources out there to learn it! The ones I recommend are:

• DataExpert.io free SQL question practice

• DataLemur

• StrataScratch

Key things you should know in this bucket are:

• The basics

  • JOINs

    • INNER, LEFT, FULL OUTER

      • Remember you should almost never use RIGHT JOIN

  • Aggregations with GROUP BY

    • Know the differences between COUNT and COUNT(DISTINCT)

      • Remember that COUNT(DISTINCT) is much slower in distributed environments like Spark

    • Know how to use aggregation functions with CASE WHEN statements

      • example: COUNT(CASE WHEN status = ‘expired’ THEN order_id END) this counts the number of expired orders

    • Know about cardinality reduction and bucketing your dimensions

      • example:

        • SELECT CASE WHEN age > 30 THEN ‘old’ ELSE ‘young’ END as age_bucket, COUNT(1) FROM users GROUP BY 1

      • You’ll see in this query, we take a high cardinality dimension like age and make it a lower cardinality (just two values “old” and “young”)

• Common Table Expressions vs Subquery vs View vs Temp Table (a great YouTube video here)

  • The key things here are:

    • You should very rarely be using subquery (it hurts readability of pipelines)

    • You should use temp table if you need to reuse some logic since Temp Table gets materialized and will improve the performance of your pipeline

    • You should use View when you need to store logic for longer than the duration of the pipeline execution

    • In all other cases, you should use common table expressions to improve readability!

• Understand how SQL works in distributed environments

  • Know what keywords trigger shuffle

    • JOIN, GROUP BY, ORDER BY

  • Know what keywords are extremely scalable (this means they’re executed entirely on the map-side)

    • SELECT, FROM, WHERE, LIMIT

• Know window functions thoroughly (great YouTube video here)

  • The basics

    • RANK() OVER (PARTITION BY <partition> ORDER BY <order by> ROWS BETWEEN <preceding> PRECEDING AND <following rows>)

    • You have the function (e.g. RANK, SUM, etc).

    • You have PARTITION BY this divides the window up. Maybe you want to do window functions per department or country?

    • You have ORDER BY this determines the sorting of the window

    • You have the ROWS BETWEEN clause to determine how many rows you should include in your window. If you don’t specify this, it defaults to ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW. So the default is the “cumulative” row definition.

  • Understand RANK vs DENSE_RANK vs ROW_NUMBER(a quick two minute YouTube video about this)

    • Key things:

      • When there is no tie in your ORDER BY clause, these functions are identical

      • When there are ties,

        • RANK skips values (e.g. a tie for 1st place means the next place is third)

        • DENSE_RANK does not skip values (e.g. a tie for 1st place means the next place is second)

        • ROW_NUMBER guarantees unique values with no ties (a tie for first place means one of them will get second place, this is based on the natural ordering of the data)

    • Understand how to do “rolling” calculations

      • Rolling average and sum by department is a common interview question. You can solve it with a query like this:

        • SELECT revenue_date, SUM(revenue) OVER (PARTITION BY department ORDER BY revenue_date ROWS BETWEEN 30 PRECEDING AND CURRENT ROW) as thirty_day_rolling_revenue FROM daily_sales

        • You’ll see we split the window by department, then we look at the rolling 30 day period for each day and sum it up. You need to be careful here and ensure that there is data (even if it’s zero) for each date otherwise you’ll calculations will be wrong!

  • Know about the differences between INSERT INTO, INSERT OVERWRITE and MERGE

    • INSERT INTO just copies the data from the result query into the table. THIS IS PRONE TO DUPLICATES! If you’re using INSERT INTO, it should always be coupled with either TRUNCATE or a DELETE statement!

    • INSERT OVERWRITE is nice because it copies the data and replaces whatever existing data is in that partition. This is the most common one they use in big tech!

    • MERGE is nice because it looks at the existing data and copies only the rows that are updated and/or deletes the rows that aren’t in the incoming data. The only minus of MERGE is the comparisons it needs to do to accomplish this can be very slow for large data sets!

What’s new in 2025?

• AI copilots like Databricks Genie and Snowflake Cortex will happily write SQL for you. But interviewers now test whether you can verify, debug, and optimize AI-generated SQL.

• Understanding how SQL runs in distributed environments (shuffle, partitioning, scalability) is still your edge over someone blindly trusting a copilot.

Learning Python (and AI Integration)

SQL is great and can accomplish a lot in data engineering. There are limitations to it though. To overcome these limitations you need a more complete language like Python.

Here are the concepts you need to learn:

• Data types and data structures

  • Basics

    • strings, integers, decimals, booleans

  • complex

    • lists (or arrays), dictionaries, stacks and queues

  • Other data structures you probably don’t need to learn (although these might show up in interviews which sucks!)

    • heaps, trees, graphs, self-balancing trees, tries

• Algorithms

  • Basics

    • Loops, linear search, binary search

    • Big O Notation

      • You should be able to write down both the space and time complexity of algorithms

  • Algorithms you probably don’t need to know (although these might show up in interviews which sucks!)

    • Dijkstra's algorithm, dynamic programming, greedy algorithms

• Using Python as an orchestrator

  • One of the most common use cases for Python in data engineering is to construct Airflow DAGs

• Using Python to interact with REST APIs

  • A common data source for data engineers is REST data. Learn about GET, POST, PUT request in Python. The requests package in Python is great for this!

• Know how to test your code with pytest (in Python) or JUnit (in Scala)

  • There’s a really solid library called Chispa that works well with pytest for testing your PySpark jobs you should check out!

New in 2025:
Python is now the glue between data engineering and AI. You’ll need to know:

• Calling LLM APIs (OpenAI, Anthropic, open-source models)

• Generating and storing embeddings

• Working with vector databases (Pinecone, Milvus, Weaviate, pgvector)

• Building lightweight RAG (Retrieval-Augmented Generation) pipelines

• Writing “AI validators” — Python jobs that use an LLM to check data quality or generate documentation

Learning Distributed Compute (either Spark, BigQuery or Snowflake)

Back in the late 2000s, Hadoop was born and so was the notion of distributed compute. This means that instead of having one fancy computer process all your data you have a team of computers each process a small chunk!

This concept unlocks the possibility of computing vast amounts of data in a small amount of time by leveraging teamwork! But this does not come without complexity!

Here are some things to consider:

• Shuffle

  • If we are using teamwork, we need to guarantee certain data is on a certain machine (like if we are counting how many messages each user has received). The team accomplishes this guarantee by passing all of your data to one machine via shuffle (example in the diagram below). We only HAVE to do this when we do GROUP BY, JOIN, or ORDER BY. (a quick 2 minute video about how I managed this at petabyte scale at Netflix)

• Shuffling isn’t a bad thing remember! It actually is really good because it makes distributed compute mostly the same as single node compute! The only time it gets in the way of things is at very large scale!

  • Things you should consider to reduce shuffling at very large scale

    • Broadcast JOIN

      • If one side of your JOIN is small (< 5 GBs), you can “broadcast” the entire data set to your executors. This allows you to do the join without shuffle which is much faster

    • Bucket JOIN

      • If both sides of your JOIN are large, you can bucket them first and then do the join, this allows. Remember you’ll still have to shuffle the data once to bucket it, but if you’re doing multiple JOIN with this data set it will be worth it!

    • Partitioning your data set

      • Sometimes you’re just trying to JOIN too much data because you should JOIN one day of data not multiple. Think about how you could do your JOIN with less data

    • Leverage cumulative table design

      • Sometimes you’ll be asked to aggregate multiple days of data for things like “monthly active users.” Instead of scanning thirty days of data, leverage cumulative table design to dramatically improve your pipeline’s performance!

  • Shuffle can have problems too! What if one team member gets a lot more data than the rest? This is called skew and happens rather frequently! There are a few options here:

    • In Spark 3+, you can enable adaptive execution. This solves the problem very quickly and I love Databricks for adding this feature!

    • In Spark <3, you can salt the JOIN or GROUP BY. Salting allows you to leverage random numbers so you get a more even distribution of your workload among your team members!

• Output data

  • Most often when you’re using Spark, you’re going to be writing Parquet files to S3 or GCP or Azure. Parquet files have some interesting properties that allow for dramatic file size reduction if you use them right. (an in depth YouTube video about this).

    • The key property of parquet files that you need to know about is a thing called run length encoding compression. I shrunk pricing and availability datasets at Airbnb by over 90% by leveraging this technique!

  • If you’re using Snowflake or BigQuery, they use their own proprietary format that I know less about. So find another newsletter for that! Maybe Seattle Data Guy’s?

What’s new in 2025?

• AI workloads run on your data stack (i.e. Shift Left)

  • Embedding generation at scale (GPU clusters with Spark or Ray)

  • Hybrid query engines that support both structured tables + vector search

  • Streaming inference pipelines with Kafka + Flink

Data Modeling and Data Quality

Data engineering is ultimately about delivering usable, correct, privacy-compliant data.

Here are the ways you can assess data quality:

• Data should be correct

  • You should check for duplicates, NULLs, proper formatting. Also checking that there is the right number of rows in the data set!

• Data should be usable and efficient

  • This means it should have proper documentation and good column names. The query patterns should also allow for fast answers to questions! Answers shouldn’t give Jeff Bezos millions of dollars either!

• Data should be privacy-compliant

  • An often overlooked part of the puzzle. You shouldn’t be hurting user privacy to make your analytics better!

So how do you achieve this output data set nirvana dream? It has multiple parts!

• Correctness should be handled in a few ways

  • The first pass should be via validation from a data analyst. This a part of the powerful MIDAS process at Airbnb that you should check out!

  • After that, you should build in automated data quality checks into your pipeline with something like Great Expectations. Remember to follow the write-audit-publish pattern here so you don’t publish bad data into production that doesn’t pass data quality checks!

• Usability and efficiency are handled with a few things

  • Documentation should be a big part of the process. Spec building and stakeholder sign-off should happen BEFORE you start building your pipeline. This will prevent a lot of redoing and undoing of work!

  • Leveraging efficient practices for your data lake:

    • Manage your Apache Iceberg snapshots for time-traveling and disaster recovery

    • Set good retention policies on your data so you don’t have a huge cloud bill for data you do not use!

  • Data modeling is going to be the other big piece of this puzzle

    • There are a few camps here:

      • Relational data modeling

        • This type prioritizes data deduplication at the cost of more complex queries. Think of this as prioritizing storage at the cost of compute.

      • Dimensional (or Kimball) data modeling

        • This denormalizes the data into facts and dimensions which prioritizes larger queries but duplicates data a bit. Think of this as trying to balance the costs of storage and compute.

      • One Big Table

        • This denormalizes the data even more where the facts and dimensions are in one table. You duplicate data more but you get extremely efficient queries from it. Think of this as prioritizing compute at the cost of storage.

      • I wrote a long-form article detailing the differences and when to pick which one here. If you prefer video format you can find it here

• Privacy compliant data sets

  • Be mindful where you have personally identifiable information in your data sets and don’t hold onto that longer than you need.

    • Remember that anything that can bring you back to a user is personally identifiable!

  • Anonymizing the data so you can hold onto it longer is a great strategy that balances user privacy and long-term analytical capabilities

New in 2025:

• AI-assisted quality checks. Tools now let you point LLMs at logs or query results to spot anomalies.

• Semantic modeling. Using embeddings to cluster or enrich tables with “semantic meaning.”

• Privacy in AI. You must ensure embeddings don’t leak personally identifiable information. Vector search can accidentally memorize sensitive data if you’re careless.

  

Building a Portfolio Project

This is where you separate yourself from the pack.

You should pick a project that:

• You care about deeply. I was obsessed about Magic the Gathering and that was what allowed me to keep working on that project even though I wasn’t getting paid

• You can work on it for 3 months at 5-10 hours per week. A portfolio piece shouldn’t be easy to create. If it was easy to create, everybody would do it and then you wouldn’t stand out!

• You build a piece that has a frontend! Another portfolio piece I build was HaloGods.com a website I built that allowed me to reach 20th in the world in Halo 5 back in 2016. Without a frontend, your data pipelines are a little bit harder to show off. This is why learning a skill like Tableau or Power BI can be a really solid way to make your portfolio shine even if those aren’t skills you end up using on the job!

• You implement the following things in your portfolio

  • A comprehensive documentation that details all the inputs and output data sets, the data quality checks, how often the pipeline runs, etc.

  • You have a pipeline running in production. Using something like Databricks Free Edition would be a great place to get started!

  • You leverage hot technologies like Spark, Snowflake, Vector Databases, Iceberg, and/or Delta Lake.

  • If you want to stand out, you build a JavaScript front end like I did with HaloGods and you’ll find yourself landing jobs in big tech even though you didn’t go to Stanford, MIT or IIT.

In 2025, you’ll want something that shows both data pipelines and AI integration. Remember you can use AI to get the boilerplate and you add your own details to get started!

Ideas:

• E-commerce pipeline → warehouse model → AI recommendation system

• Ingesting data from Polygon.io → stock market model → AI stock picker system

• YouTube transcript ingestion → Iceberg tables → semantic search UI with RAG

• Automated data quality alerts using GPT validators

Remember, you might have to pay a tiny bit of money to build these projects. But it will be worth it over the long run!

Building a Personal Brand

So you have the skills and the sexy portfolio project. Now all you got to do is not mess up the job interview and you’ll be golden.

Here are some things you should do to get there:

• Build relationships on LinkedIn

  • You should be building relationships with

    • Hiring managers and recruiters

    • Peers

  • You should start talking with people BEFORE you ask for a referral and start building up friendships and your network.

    • If you need a referral, it would be better to send out DMs to people and ask them what the job is like. Leading with questions instead of tasks will have a much higher hit rate!

    • Also remember that this DM game is a low percentage play. If you send out 20 DMs, 1 or 2 might respond, especially if you’re early in your career. Finding creators and employees that are near your same level will help increase that hit rate!

  • Create your own content and talk about your learning journey. You’d be surprised how effective this is at landing you opportunities you didn’t even think were possible. For example, I made $600k from LinkedIn in 7 months after quitting my job! Content creation and branding are a very powerful combo that can change your life!

    • Nowadays with AI, you can use AI aid in the content creation process! Just remember to remove the ugly emojis, em dashes, and

• Interview like a person, not a robot

  • When you go into the interview make sure you have:

    • Researched the people who are interviewing you. You should know

      • How long they’ve worked for the company

      • What they do for the company

    • Asked the recruiter a lot of detailed questions about the role

      • What technologies will I be using

      • How many people will be on my team

      • What is the culture like

  • During the interview you should radiate

    • positivity, enthusiasm and excitement for the role

    • competence and calm when asked questions

    • curiosity to engage in stupid interview questions and curiosity about the role and what you’ll be doing by asking good follow up questions

  • Demonstrate technical skills during the interview

    • The SQL Interview

    • The Data Modeling Interview

    • The Data Architecture Interview

    • The Data Structures and Algorithms Interview

Conclusion

Getting into data engineering in 2025 is still hard. The job market is competitive. But the best engineers and the best startups are built in times like this.

The difference now? AI is part of the job. You don’t need to be an AI researcher, but you do need to know how to integrate AI responsibly into your pipelines.

Follow this roadmap and you’ll be much closer to landing the data engineering role of your dreams.

I am launching a new AI Engineering Boot camp starting on October 20th where we are covering all the new AI topics necessary to be a good AI-enabled data engineer. You can get 30% off with code AIROADMAP here.

What else do you think is critical to learn in data engineering to excel in the field? Anything I missed here that you would add? Please share this with your friends and on LinkedIn if you found it useful!

Read more

Introduction

Apache Spark has been the de facto open source data processing for fifteen years. It was invented to solve a major problem that traditional data warehousing was not built to solve - processing massive amounts of data horizontally at scale (Zach used Spark to process 2000 TBs per day at Netflix), whether in a structured or semi-structured for…

Read more

Read more

This month, I’m excited to present three fantastic opportunities to learn from top-tier practitioners without spending a dime.
Whether you’re trying to break into tech or looking to make a strategic pivot in your career, these free bootcamps are packed with real-world lessons, hands-on skills, and instruction from people who’ve been in the trenches.

We a…

Read more

There are some conversations that confirm what you already believe, and then there are conversations that take what you believe and frame it in a way that makes it actionable, precise and inevitable.

That’s what happened when I sat down with Chip Huyen during a Tech Talk we held at DataExpert.io AI Engineering Boot Camp.

Chip is an AI researcher, former big tech engineer, best-selling author and one of the most lucid thinkers on AI systems working today. Recently, she’s been in the public spotlight with her new book AI Engineering, which has quickly become the most comprehensive, well-structured guide to the essential aspects of building generative AI systems (we covered a lot of its content in the boot camp too).

This talk was neither a book launch nor a formal Q&A. It was a honest, refreshing & grounded conversation that can be distilled into seven core takeaways, each one capturing ideas Chip shared that stuck with me, challenged me or reshaped how I think about building and leading in AI. This article covers the following:

• Chip’s journey into AI

• Where most GenAI products go wrong

• The underrated value of UX

• How to build functional AI agents

• What really takes to ship value in the modern AI stack.

• How to stay informed without burning out

• Building with clarity and conviction

If you want to learn from other brilliant minds like Chip, we are launching a 10-week Challenge Boot Camp on Sep 15th where will be covering insightful tech talks with 15 industry leaders in the data, analytics and AI engineering space. The first 5 people to register can use code CHIP for 30% off!

✍🏻 #1: The Primacy of Compute and Data

To understand Chip’s journey into AI, we have to rewind to 2012, the year Deep Learning truly exploded onto the scene. That was the year AlexNet, a deep convolutional neural network, won the ImageNet competition by a massive margin (over 10 percentage points better than the next best model).

AlexNet rewrote the rules that have defined the last decade of AI. And notably, one of the paper's co-authors, Ilya Sutskever, would go on to co-found OpenAI, the organization that would later lead the charge on scaling up large language models.

Chip recounted how a single sentence from that 2012 paper changed her life trajectory:

“Our experiments show that we can achieve better results by just waiting for more compute and more data.”

That line reframed AI not as a field of breakthroughs, but as one of compounding scale. Chip went on to work at NVIDIA to understand compute infrastructure and later joined Snorkel AI to understand data workflows.

She also reflected on how OpenAI was initially dismissed for simply scaling up, with many academics saying it wasn't real research. But the turning point came in 2020 when the GPT-2 paper received a best paper award, and the conversation suddenly shifted.

Everyone was like, ‘Wow, now it’s real research.’

💡 I remember similar skepticism back in my days at Facebook. People dismissed what OpenAI was doing as brute force. It turns out brute force was the insight.

This key takeaways reframes my view of AI progress not as a parade of novel ideas but as an engineering problem of sufficient scale. Chip's clarity here gives me language to explain why things like GPT-5 didn't just appear but emerged from compute and data discipline.

✍🏻 #2: The GenAI Hype Cycle and the Misuse of ML

What makes Chip’s take on GenAI refreshing is that her relationship with AI long predates the hype. Before ChatGPT became a buzzword, she was building simple algorithms to test logic, even designing games her smartest friends couldn’t win. Not to outsmart them but to understand how reasoning could be codified.

Read more

Fake accounts come and go. Some are flagged incorrectly, others are later verified, and many are caught by internal ML systems. The goal of our pipeline was to trace the inflows and outflows of fake accounts daily. That meant building a reliable dataset that could track:

• Accounts unlabeled as fake (i.e. after submitting a valid ID)

• Accounts relabeled as fake

• Accounts flagged as fake for the first time

• Accounts that remained fake

The pattern was very straightforward: a classic cumulative table design. But the way it was wired to upstream data, specifically how it “waited” for inputs, created a non-deterministic nightmare. For weeks, I chased what I thought was a bug in my code, only to discover that the real problem had been there from day one.

In this article, I’ll cover the following aspects:

• Why relying on “latest” partition data broke everything

• How upstream non-deterministic leads to silent data mismatches

• The simple fix using explicit partition dates

• The tradeoff between latency and reproducibility

• Engineering lessons that go beyond code

If you want to learn more in depth about patterns like this, the DataExpert.io academy subscription has 200+ hours of content about system design, streaming pipelines, etc. The first 5 people can use code FAKE for 30% off!

If you enjoy this article, here are some more from my time in big tech:

• How I prepared for Airbnb’s staff data engineer interview

• How I achieved a 12x speed up on Facebook notification pipelines

• How I used the “Psycho” pattern to detect threats at Netflix

• How I cut Airbnb’s pricing data backfill time by 95%

Understanding Fake Account Flows

At a high level, the pipeline’s job was to compare today’s and yesterday’s fake account snapshots, determine who had entered or exited the fake state, and store those inflow/outflow transitions for downstream analytics.

This was a classic cumulative table design pattern, built in plain vanilla SQL and tracked four main fake states:

• New fakes → New people who got labeled as fake

• Resolved accounts → People who were labeled fake earlier than today and passed a challenge to remove the label

• Persisting fakes → People who were labeled fake earlier than today and haven’t passed a challenge

• Relabeled fakes → People who were labeled fake, passed a challenge, and then continued to do fake activity

Originally, it was set up like this:

1. Fake accounts pipeline waited on the “latest” partition of the users table

2. Then joined it with fake_accounts to compute transitions

3. The job ran daily and published results

-- Define struct and enum types

CREATE TYPE IF NOT EXISTS daily_detection_stats AS (
    detection_date DATE,
    login_attempts INTEGER,
    friend_requests_sent INTEGER,
    posts_created INTEGER,
    flagged_reports INTEGER
);

CREATE TYPE IF NOT EXISTS fake_classification AS ENUM('new', 'resolved', 'persisting', 'relabeled');

-- Create cumulative table
CREATE TABLE IF NOT EXISTS fake_accounts
(
    account_id TEXT,
    country TEXT,
    sign_up_method TEXT,
    sign_up_date DATE,
    daily_detection_stats daily_detection_stats[],
    fake_classification fake_classification,
    days_since_last_detected INTEGER,
    current_detection_date DATE,
    PRIMARY KEY (account_id, current_detection_date)
);


-- Dynamically pick the "latest" date
WITH latest_date AS (
    SELECT MAX(detection_date) AS detection_date FROM account_daily_signals
),
yesterday AS (
    SELECT * 
    FROM fake_accounts 
    WHERE current_detection_date = (SELECT detection_date - INTERVAL '1 day' FROM latest_date)
),
today AS (
    SELECT * FROM account_daily_signals 
    WHERE detection_date = (SELECT detection_date FROM latest_date)
)

-- Non-idempotent insert
INSERT INTO fake_accounts
SELECT
    COALESCE(t.account_id, y.account_id) AS account_id,
    COALESCE(t.country, y.country) AS country,
    COALESCE(t.sign_up_method, y.sign_up_method) AS sign_up_method,
    COALESCE(t.sign_up_date, y.sign_up_date) AS sign_up_date,
    CASE
        WHEN y.daily_detection_stats IS NULL THEN ARRAY[ROW(
            t.detection_date,
            t.login_attempts,
            t.friend_requests_sent,
            t.posts_created,
            t.flagged_reports
        )::daily_detection_stats]
        WHEN t.detection_date IS NOT NULL THEN y.daily_detection_stats || ARRAY[ROW(
            t.detection_date,
            t.login_attempts,
            t.friend_requests_sent,
            t.posts_created,
            t.flagged_reports
        )::daily_detection_stats]
        ELSE y.daily_detection_stats
    END AS daily_detection_stats,
    CASE
        WHEN y.account_id IS NULL THEN 'new'
        WHEN t.account_id IS NULL THEN 'resolved'
        WHEN t.detection_date IS NOT NULL AND y.fake_classification = 'resolved' THEN 'relabeled'
        ELSE 'persisting'
    END::fake_classification AS fake_classification,

    CASE
        WHEN t.detection_date IS NOT NULL THEN 0
        ELSE y.days_since_last_detected + 1
    END AS days_since_last_detected,
    COALESCE(t.detection_date, y.current_detection_date + INTERVAL '1 day')::DATE AS current_detection_date

FROM today t
FULL OUTER JOIN yesterday y
ON t.account_id = y.account_id;

The Architecture That Broke Everything

Read more

One of my most challenging assignments was owning the pipeline that deduplicated all notification events. This dataset drove downstream metrics like CTRs, conversions, and even machine learning signal quality.

This pipeline presented one big problem: it was slow. Very, very slow.

When I joined Facebook, the deduped notifications pipeline ran a giant Hive GROUP BY job once a day at UTC midnight which took 9.5 hours to complete. This latency issue represented a huge bottleneck for all downstream models in the Core Growth team.

This article is the story of how I brought 9.5-hour latency down to 45 minutes, and what it taught me about I/O, orchestration, and never trusting a “simple” DAG.

Here I’ll be covering the following:

• Why streaming deduplication at scale failed

• The hourly dedup job that exploded compute usage

• A tree-based DAG design that saved the day

• Key orchestration lessons from building a 300-step daily DAG

• How this project got me promoted and why I almost gave up

If you want to learn more in depth about patterns like this, the DataExpert.io academy subscription has 200+ hours of content about system design, streaming pipelines, etc. The first 5 people can use code DEDUP for 30% off!

The Stakes: Notifications at Facebook Scale

The notif_events table contained every event tied to a notification:

• Sent to your phone

• Delivered to device

• Clicked

• Converted

Since one person might click the same notification multiple times, we had to dedup those. If we counted every click, we’d get click-through rates above 100%, which negatively impacted metric tracking and, worse, model training.

But the problem wasn’t logic. It was latency.

As I mentioned earlier, the dedup job ran once a day via Hive & took 9.5 hours to complete. My managers wanted me to reduce latency dramatically.

Approach 1: Stream It 😣

At first, my manager’s request was straightforward: ‘Let’s dedup in real-time’.

So, I tried. I built a Spark Streaming job that listened to notification events and tried to hold recent activity in-memory for comparison. But this approach was holding as much as 50+ terabytes in RAM to do real-time deduping.

This wasn’t feasible. The streaming job crumbled under memory pressure. I had to come up with a better solution.

Approach 2: Hourly Dedup + Merge 🤔

Our second approach was to dedup every hour. The idea was simple:

1. Dedup the current hour table and write it to a sorted, bucketed table (to minimize shuffle later with SMB join)

2. Merge with the previous hour table (a cumulative of all previous hours’ deduped data).

3. Output a deduped table up to the current hour.

The read and merge hourly deduped table to the previous hour can be implemented with this simple SQL pattern inside your DAG:

-- Hourly dedup logic 
-- Remember this table is sorted and bucketed on user_id

INSERT OVERWRITE TABLE notif_deduped_hourly(ds='{{ ds }}', hour={{ current_hour}}, channel='{{ channel }}')
  SELECT 
     notif_id,
     user_id, 
     -- count the number of events of each type
     -- a custom UDF that returned a MAP {"sent":1, "clicked":3}
     COUNT_MAP(event_type) as event_map_count
  FROM notif_events
  WHERE event_hour = '{{ current_hour }}'
  AND ds = '{{ ds }}'
  AND channel = '{{ channel }}'
  GROUP BY notif_id, user_id

-- Then we merged the cumulative previous hours and the current hour with FULL OUTER JOIN

INSERT OVERWRITE TABLE notif_deduped_combined_hourly(ds='{{ ds }}', hour='{{ current_hour }}', channel='{{ channel }}')
WITH dedup_current_hour AS (
  SELECT 
     *
  FROM notif_deduped_hourly
  WHERE hour = '{{ current_hour }}'
  AND ds = '{{ ds }}'
  AND channel = '{{ channel }}'
),
previous_hour AS (
   SELECT 
     *
  FROM notif_deduped_combined_hourly
  WHERE hour = '{{ previous_hour }}'
  AND ds = '{{ ds }}'
  AND channel = '{{ channel }}'
)

SELECT  
   COALESCE(c.notif_id, p.notif_id) as notif_id,
   COALESCE(c.user_id, p.user_id) as user_id,
   -- udf that merges the keys of two maps
   -- {"sent": 1, "clicked": 3} + {"clicked":4} 
   -- = {"sent":1, "clicked": 7}
   COMBINE_MAPS(c.event_map_count, p.event_map_count)  as event_map_count
FROM dedup_current_hour c FULL OUTER JOIN previous_hour p 
ON c.notif_id = p.notif_id 
-- This condition triggers the SMB join because both tables are sorted and bucketed on user_id
AND c.user_id = p.user_id

This approach actually worked. It lowered latency and produced correct results.

But it had a huge flaw: compute usage exploded. It used 15 times more compute than the original 9.5-hour GROUP BY job.

How was that possible? Quite straightforward:

• Hour 1 processes 1 hour of data

• Hour 2 reads and merges 2 hours

• Hour 3 reads and merges 3 hours…

By hour 22, you find yourself reprocessing nearly the entire day’s data on every run.

It looked like this:

Once again, this solution was not sustainable, especially not for one of Facebook’s biggest datasets.

Approach 3: Tree-Style Merge DAG 🤩

Read more

At the time, we were leveraging a creative hybrid architecture internally dubbed as the “Psycho Pattern.” Think of i…

Read more