This article is brought to you by:

Eon and Google Cloud are hosting BigQuery Day, a free one-day virtual event for teams running BigQuery in production. Sessions from Google’s VP of BigQuery Engineering, Lead Data Engineer at L.L.Bean, Google Cloud Developer Advocates, and more.

AI has evolved into new UX patterns. This will continue.

So, we have robots (LLMs, agents, whatever you want to call them) now. What can we use them for?

Well, robots have changed their shape over the last few years. We went through these affordances:

• GPT playground (autocomplete)

• ChatGPT (chatbot)

• custom (RAG) setups (understand your files specifically)

• Claude Code (agents, dispatchers)

• “Agent” workflows

• OpenClaw (autonomy + skills + texting interface)

Each was useful! Businesses are being built on them.

So -- what next? And what can we learn?

Well, we’re about to see a new layer – the army of scribes. You’ll be tempted to use them directly. You’ll then be tempted to hook them up to your robot assistant. Fine. But the real action will be in giving that assistant enough context, power, and team-aware-capabilities so that it can actually be a useful clerk or secretary.

The gold rush of AI scribes is coming

Surely you also see the coming gold rush. Seemingly everyone is working on versions of this vision (to greater and lesser degrees of insight). Via “owning your docs”, via “owning your communications”. Maybe pitched as “specially for developers” or a sales-calls-to-product-insight loop. Maybe as a panicked pivot from an entirely different product. Maybe over a specific dataset. They’re all the same thing – scribes, taking notes, collating them, and giving them back to you.

This will happen.1

And it’ll be fine! It’ll be useful. But how often do you currently read meeting notes? How will it feel, really, to have action items more efficiently thrown your way? You’re still tied to the tyranny of staring at a screen and typing on a keyboard.

What’s more interesting is what will happen next.

We can do better than delegating notes to robots

Think about depictions of principals doing magnificent work across media.

A soccer coach rallying his team. Space captains conducting negotiation with aliens. Ad executives making deals in the 1960s. A fantasy king leading his men into battle. Business executives jockeying for power.

What do these people have in common? They have fun, creative jobs. They have meetings. They have big thoughts. They made decisions. They do not slave over a typewriter or word document. They do not take meticulous notes. They have staff for that. (Notably, being the staff is generally a lot less pleasant.2)

In the near future – that could be you. But if you want that life, you need staff to do the less fun work. Beyond just scribes taking notes. You need clerks. You need secretaries. And you need them to be as reliable and competent as those fantasy secretaries. But since they’re robots, you don’t actually need to worry about whether their jobs are rewarding.

But what do real-life secretaries actually do?

Share

Well, in large part, you’re currently a secretary to yourself. Modern real-world workplaces spend a lot less money on logistical support than fantasy TV offices do. You do a lot of typing. You send memos. You route information between people on your team. You update documentation. You schedule and bargain and procure. You might communicate with vendors and outsiders or do other logistics. You type. You do it all in the background as part of supporting your “real” work. It may feel tactically rote, but it relies on a wealth of knowledge and trust and context that you can’t hand off cleanly – at least, until now.

Now what does this look like in our near-future?

Mostly-reliable scribes are a commodity

Your company will set up scribes. You will have an invisible system of robots doing certain menial tasks. Your conversations will be filed, your notes will be taken. The notes will be parsed into decisions. Jira tickets will be filed and closed. Executives will get daily summaries of what their teams are up to. Action items will be filed after each meeting.

This might even be built, in a half-baked way, in your team right now.

That’s the basic, obvious vision. Robots swarm around writing notes, reading notes, answering your questions about your notes. An army of scribes writing into a giant filing cabinet.

Importantly – you’ll need these scribes to be so good that you don’t spend a lot of time fixing their filing mistakes. Or, more interestingly, you’ll need some other entity to make sure they got it right.

(”So far, so good! But weren’t we talking about secretaries?” -- yes, shh. We’re going deeper).

SIDEBAR: What about OpenClaw?

There’s an obvious reaction to all this: “hey, isn’t this just openclaw? OpenClaw/Hermes/IronClaw is already my personal assistant”. Well, kinda. Not really. The first prototypes of AI secretaries we see will likely be built on a foundation of *claws, but we need AI systems that have these core design principles:

• Built for teams (of humans and similar robots) from the ground up.

• Coordinating across systems of code, people, and other AIs (invisibly, constantly)

• Will integrate with company systems (and be trusted by IT not to leak or delete everything)

• Likely multi-server and multi-model.

• Opinionated on what happens invisibly/continuously and need user approval or visibility.

And more (see below).

This will require a new and different type of architecture than looping agents on a server and connecting it to Telegram.

A scribe writes things down. A secretary proactively preps.

But wait. A beautifully organized filing system doesn’t work well unless you can use it; at the same time you don’t want to access it yourself. Otherwise you’re back in the drudgery of paperwork.

And you need to be able to give orders -- often implicitly phrased as “if X, then Y”. (Current tooling is pretty bad at this right now)

You need someone to proactively and reactively hand you the perfect manila folder each time you need it.

And, suddenly, we’re not talking about a scribe at all. We’re discussing something more interesting. The layer on top of that. The secretary. A clerk that talks to the invisible army of scribes on your behalf, sure, but so much more than that. It must continuously manage attention, context, information, and execution across time.

Let’s go further. You’re at work.

When you go to a meeting with someone else on your team, you want your secretary to know ahead of time, talk to their secretary, compare notes, write briefing docs for the meeting, hand them to you on a just-in-time basis (or maybe send you the pre-read in the morning and the prep doc 10 minutes ahead of the meeting) -- AND understand any decisions you make in the meeting and implement them.

When you work on an architecture plan, you want your clerk to interrupt you, and tell you that team ABC tried something similar in the company 2 years ago.

The clerk has:

• Simple retrieval: Pulled all the notes

• Social graph querying with sophisticated privacy / sensitivity features: Talked the clerk of the Albert, the lead architect of ABC, and got from them the sensitive learnings that weren’t in the official document.

  • (“Joe is great at code but horrible at explaining things well – we should not have made him a project lead on this”, “direction X was really promising but couldn’t pursue it due to office politics”, “I’ve subsequently learned about Y pattern that we should have used instead”)

  • Followed up with the clerks of Joe, Sally, and Martha who were also on that project.

• Negotiation with other clerks: Booked coffee with Albert, who will be in town next week (but does not show his free time on his calendar)

• Proactive pushing of info: Prepped Albert’s clerk with all your context

• Context aware synthesis: Written you a report on what, from project ABC, can you actually learn

That’s a vision. Maybe a compelling vision. It is to me.

And I think we can build it.

Can you trust your robot secretary?

For all this to work, you need reliability, trust, and even discretion.

• You need to know that stuff won’t get dropped. Dropped is terrible. Forgotten about? Also terrible, but as long as it is filed somewhere you can recover.

• You need to be able to fire your secretary. In this case, that means that if you stop using your particular AI system, porting to a new system needs to be incredibly easy. The secretary’s notes need to play well with the filing/scribe system the company uses. It needs to be flexible enough to take a filing system as it exists and invisibly clean it up to its standard over time.

• Some robots need to know about the existence of, say, an email or meeting (”the boss is meeting a lawyer to talk about his divorce”) but NEVER leak the content of it. And we need to trust that they never double-book, and might even do polite fictions if necessary.

SIDEBAR: Your secretary is many agents in a trenchcoat

Note that I’m not using the word “agent” a lot here. That’s because your clerk, secretary, even your scribe – they’re not going to be one agent. Why should they?

An agent is analogous to a process on a machine. Processes can fork, or call other processes. They can die without implicating each other. A traditional “program” on your laptop has many processes running at once. But a SASS app has so many different computers involved that talking about processes feels like you’re missing the point.

Share

So too with anthropomorphized robot helpers. Maybe on one server there’s a dispatcher agent that calls research agents, runs async scripts, negotiates with the company data layer. Maybe it coordinates with a dispatcher on another server that message and info passes with other assistants. A third dispatcher on a third server might be the layer that talks to you (and also runs subagents for odd jobs rather than message passing to the other two).

I’m not saying this is a good architecture, by the way. But it’s entirely feasible. And, to my point: who is the “agent” you’re talking to in this scenario? Even “Multi-agent system” is too low-level. We don’t quite have the right words the emerging thing we’re describing. Agent OS? AI persona?

I have some thoughts on how to build it. I’ve even started. The keys, so far, seem to be:

• Start from an assumption that your secretary is working on a team

• Set up the right dispersion of deterministic and LLM control flow

• Use a really rich schema for how secretaries communicate in structured ways.

Some primitives that I’ve found useful:

• Dispatcher/subagent structure

• Code. Gatekeepers, hooks, a message queue, and injection of prompts to deterministically both nudge and enforce behavior.

• Rich schema for discretion, communication, decisions, and memories.

• Secure enclaves of local LLMs that monitor the frontier model “kernel” and gatekeep its communication for the correct discretion sensitivity.

• Federated CRMs and knowledge graphs within a team

• An “If this, then that” data structure that points to the right memory when agents match a certain behavior (solving the “remember to look up a memory telling you to remember something problem).

• Logs and health check monitors everywhere

But the crux, maybe, is this: the world of OpenClaw clones is running into two bottlenecks: security/privacy/reliability, and the need for an actual killer app. Those are related. Imagine a world where everyone has some sort of robot butler -- but their *claws talk to each other well. Invisibly, carefully.

Leave a comment

SIDEBAR: Clerks and Secretaries and CoS

Why “secretary”? Why not just “clerk?” Well, the term clerk doesn’t quite bring up the level of familiarity and trust that I’m going for. Does a clerk know about your sensitive meeting even as they don’t put it on the calendar? Clerks are closer to paper pushers than autonomous agents.

Another word you could use was “chief of staff”. But that has its own limitations – mostly the term means very different things. The CoS in a government agency basically runs it. A CoS in tech has extremely different responsibilities than one in nonprofits. And so on.

So, when using “secretary”, I’m trying to evoke a level of trust, competence, autonomy and brilliance that we see in a Joan or Peggy. Without anyone involved being as poor a boss or bad a person as a Don Draper.

Against communication explosion

That’s the future I see. We shouldn’t just throw our robots into Slack3 and call it a day. You don’t teach students by throwing them all into a gym and having them run around yelling at each other. You don’t run a company by having CEOs spying on how each worker is sending notes to each other.

We will have our AIs morph between copilot and HUD. They’ll share information in a smart way. They’ll know they’re on a team and work accordingly.

So don’t obsess over the coming scribes. That’s tomorrow’s tech. Build assuming that it exists. Focus on using it in smart ways rather than staying stuck as your own secretary.

Make sure to follow Sahar’s blog to understand growth and what comes next, both personally and for your business! Sahar is currently pursing full-time product and engineering leadership roles in New York City.

DataExpert is now open for more sponsorship. Like this post? Want people to know about how great your product is? Please email us at mitali@dataexpert.io to start a conversation.

Leave a comment

While Leetcode has historically been important for filtering candidates out, it has zero impact on the level candidates are hired at in Big Tech. The behavioral round and leadersh…

Read more

• Consistent correct answers and hallucinations

• Low-latency answers and waiting forever

• Maxing out your token limit in a day and being able to use a cheaper model

In this article, we will go over ho…

Read more

Read more

But if you zoom out and look at the product direction over the last few years, a much different pattern emerges.

Read more

If AI can generate pipelines, DAGs, tests, and even migrations…
What’s left for data engineers to actually work on?

Conceptual knowledge is no …

Read more

In this article, we will attack the following beliefs:

• Only Spark can be used for terabytes of data (or it is ALWAYS the best choice)

• You need a lot of time to process TBs of data

We want to leave your head spinning at the end of this article. Wondering if everything you learned about MapReduce was wrong!

This Article is brought to you by

We want to give a shout-out to MotherDuck, who is sponsoring this article and providing the infrastructure for the benchmarks!

You Said to Use DuckDB On Small Data

Previously, I was a champion of using DuckDB for any dataset that was “small” (< 20GBs). Recently, I was challenged on that remark on LinkedIn by some astute data engineers, who said I had a misconception about what DuckDB was capable of. Being the curious data engineer I am, I took a bite on that bait and decided to roll up my sleeves and benchmark much larger datasets.

But how much larger?

Also - if you want to subscribe to Matt’s Substack, you can click here.

I first decided to go after ~200 GBs. DuckDB read that data in <10 seconds.

This was too fast. It felt magical. What about 500 GBs? Then I hit a wall: a physical wall. The hard drive on my Mac M2 didn’t have enough space for 500GBs. I strolled to my local Best Buy and picked up this thing:

Side Note - A 4 TB external hard drive might seem like overkill; this was one of those “Go big or go home” moments. I figured in my mind “well if 500gb works, I want to have enough runway for much larger tests down the road”

I created a 500GB dataset on the external drive in DuckDB. It read that data in ~40 seconds

This made me realize I needed to set my sights on the big kahuna. 1 full TB of data!

Building A 1 TB Dataset for DuckDB

In my previous articles, you will see that I use a script that leverages DuckDB’s' generate_series' function to generate rows of data quickly. The gist of it looks like this:

It’s straightforward:

• You pass in a row count and let it generate a Parquet file.

• If we want to do this at scale and not wait several hours (or go dreaded serialized). What do you do?

• Bring in the good ol’ Python ProcessPoolExecutor and go parallel. (code here)

Ok, But Did You Really Generate A Full TB Of Data?

Yes, it took my M2 Pro (16GB of RAM) ~70 minutes to fry this egg with 10 workers in parallel. Here’s the proof:

The dataset is: 400 files, each ~2.76GB in size.

So now, without further ado, let’s get cracking and run some benchmarks on this.

Side Note - If you have not moved from the old Python virtual environment of “-m venv” over to UV, do yourself a favor and do it now; uv loads packages faster, makes targeting specific Python environments easier; I could go on…you’ll thank me later

The Benchmark…And What Exactly Are You Doing Here?

Today’s benchmark will:

• run a common aggregation query across the 1TB dataset;

• It will group by a date, count rows, and sum a value.

This is a common analytics query I have seen in my last two decades as a data engineer and BI leader; I did not cherry-pick this to just make DuckDB look good. This is what the benchmark query boils down to:

For our benchmark, we will run it 5 times. Below are the results:

Average processing time locally: 1 minute, 29 seconds.

Hold On - You Said We Could Crush A Full TB In Under 30 Seconds!?

I did say that. This first benchmark was on my laptop and local computer, which is impressive.

What would happen if I were to create a full 1TB dataset in MotherDuck and try this again?

Time To Join The Flock

On MotherDuck, we have excellent options to load data. We could do stuff like:

• store CSVs and Parquet in S3/Azure/GCP

• import the TCP-H dataset

• Use our local CLI to generate data

For this article, I chose the third option. Here is the script that created the 1TB dataset in MotherDuck:

I created a view in a MotherDuck database that leveraged the generate_series function (like in the previous local benchmark). After that, I ran the script to iterate over and insert the data multiple times.

After 10 iterations, I saw it wasn’t quite at 1TB; I manually ran the load process several times more until I got 1TB.

We now have over 1TB of data. Now it’s time to choose our compute capacity.

In MD, we have four standard options for compute capacity: Pulse, Standard, Jumbo, and Mega. I went with Mega, given we are dealing with a full TB of data:

For the benchmark, we ran the exact same query we did locally, but with a caveat;

• MotherDuck has intelligent caching; running the same query five times will have results 2-5 be about 5 seconds or less because it will read from cache vs. actually scanning the data.

So how do we get around that?

• Simple - for each iteration, we will have our aggregation values sum a different lower and upper range, which makes the query non-deterministic, and removes the ability for it to just hit the cache layer. The query template looks like this:

Below are the results of our benchmark:

Holy Smokes -

• That was clocking in at under 17 seconds on average. You might also ask - “Hey what is that run 0 a.k.a the cold start?”

  • Remember, we are now dealing with a cloud data warehouse, which will not always keep our data readily available to rip from RAM; sometimes, it will be on disk and have to get read in; thus, the iteration 0 is a warm up run, incase its having to read off of disk for the first usage; for datasets you will query often in MotherDuck, this won’t be an issue, as your data will more than likely be ready to process and sit in hot ram.

Ok, Great Stuff! We Are Done, Right?

We blew our promise of scanning a full TB in under 30 seconds out of the water, by a factor of 2. But what if we wanted faster?

I’m Ready - Let’s Go Deeper

DuckDB supports indexes, but they don’t really push the concept much. The Zonemap index is a secret weapon that allows you to take advantage of pre-sorted data through min/max tracking of the metadata.

How would I implement these zone maps?

• Let’s reload our dataset where we sort and insert on the rand date. The load process looked like this:

Once that was complete, I created another benchmark, and here were the results:

The sorted (zonemap) dataset improved our benchmark time by roughly 30%. That is an amazing tweak, simply by loading the table in a sorted order by the field we are grouping by.

Summary

This article showcased a paradigm shift in DuckDB’s capabilities! We have now shattered the belief of what “small” data is and what the duck can do. Even on my local laptop, we were still scanning 1TB of data in <2 minutes. If my batch jobs refreshed my reports in 2 minutes without Spark, I would be very happy!

Here are all the code examples.

• Local Data Generator

• Motherduck Data Generator (Unsorted)

• Motherduck Data Generator (Sorted)

• Local Benchmark

• Motherduck Unsorted Data Benchmark

• Motherduck Sorted Data Benchmark

Thanks again, MotherDuck, for providing us with the environment to showcase this capability!

Thanks for reading and happy holidays! If you found value in this article, make sure to comment and share with your friends!

Matt and Zach

Data engineers sit at the intersection of value and vulnerability. You build the pipelines that power analytics, AI, and business decisions. But the same systems that enable innovation also become prime targets for attackers.

You don’t need to become a full-time security engineer to protect your data. But you do need to think like one — because no amount…

Read more

Sahar and I learned a lot during out time working in core growth together at Facebook working in friending and notifications.

Let’s talk about the pain of unlearning and then let’s get to the magic.

Imagine you’re an analyst at a social media company. The retention team asks: “For users who now have 1000+ followers but had under 200 three months ago – what device were they primarily using back then? And of the posts they viewed during that growth period, how many were from accounts that were mutuals at the time?”

You need to join user data (follower counts then and now), device data (primary device then and now), relationships (who was a mutual then vs now), and post views – all as of 3 months ago.

With most data warehouse setups, this query is somewhere between “nightmare” and “impossible.”  

You’re dealing with state, not actions. State in the past, across multiple tables. There’s a word for this problem – slowly changing dimensions. Whole chapters of textbooks deal with various approaches. You could try logs (if you logged the right stuff). You could try slowly changing dimensions with `valid_from/valid_to` dates. You could try separate history tables. All of these approaches are painful, error-prone, and make backfilling a living hell.

There’s a better way. Through the magic of ✨datestamps✨ and idempotent pipelines, this query becomes straightforward. And backfills? They become a button you push.

Part 1 fixed weird columns, janky tables, and trusting your SQL. Part 3 will cover scaling your team and warehouse. But now – now we fix: backfills, 3am alerts, time complexity, data recovery, and historical queries.1

The old way was a mess

Here’s what most teams do when they start out:

Option 1: Overwrite everything daily

Your pipeline runs every night, updates dim_users with today’s snapshot, overwrites yesterday’s data. Simple! Until six months later when someone asks “how many followers did users have in March?” and you realize: that data is gone. You have no history. You can’t answer the question. Oops.

(Jargon alert – Apparently this is SCD Type-1 ¯\_(ツ)_/¯ )

Option 2: Try to track history manually

Okay, you think, let’s be smarter. Add an updated_at column. Or maybe valid_from and valid_to dates, with an is_current flag. When a user’s follower count changes, don’t update their row – instead, mark the old row as outdated and insert a new one.

(Jargon alert – This is SCD Type-2. Booo)

This is better! You have history. But now:

• Your pipelines need custom logic to “close out” old rows before inserting new ones

• If you mess up the valid_to dates, you get gaps or overlaps in history

• Backfilling becomes a nightmare – you can’t just rerun a pipeline, you need to carefully update dates without breaking everything downstream

• Querying becomes a nightmare. To get user data “as of 3 months ago”, you need:

SELECT * FROM dim_users WHERE user_id = 123 AND valid_from <= ‘2024-10-01’ AND (valid_to > ‘2024-10-01’ OR valid_to IS NULL)

Now imagine joining MULTIPLE historical tables (users, devices, relationships). Every join needs that BETWEEN logic. Miss one and your results are silently wrong. Get the date math slightly off and you’re joining snapshots from different points in time. Good luck debugging that.

Option 3: Separate current and history tables

Some teams maintain dim_users (current snapshot) and dim_users_history (everything else). Now you’ve got two sources of truth to keep in sync. Analysts need to remember which table to query. Any analysis spanning current and historical data requires stitching across tables with UNION ALL. It’s a mess.

And, depending on how the dim_users_history table works – it won’t solve any of the problems you’d have in option 2!

All of these approaches share a problem: they’re trying to be clever about storage. They made sense when disk was expensive. They don’t anymore.

(Jargon alert – This is SCD Type-4. Note that I didn’t know this when I started writing this blog post because it’s useless, boring, outdated jargon. Ignore it.)

Sponsorship

If you want to learn more about data modeling and data architecture in detail, you can use code SCDSUCKS by November 14th to get 35% off the DataExpert.io Snowflake + dbt boot camp

The new way: Just append everything

You solve it with date stamps. You solve it with “functional data engineering”.

What you really want is a sort of table that tracks state – a dimension table –, but where you can access a version that tracks information about the world today, and another version that tracks information about the world in the past.

Maxime Beauchemin wrote the seminal public work on the idea here. But, honestly, I think the concept can be explained more plainly and directly. So here we are.

The thinking goes like this:

• We’re getting new data all the time.

• Let’s simplify it and say – we get new data every day. We copy over snapshots from our production database each evening.

• There are complex, convoluted ways to keep track of what data is new and useful, and what data is a duplicate of yesterday.

• But wait. Storage is cheap. Compute is cheap. Pipelines can run jobs for us while we sleep.

• It’s annoying to have a table with the data we need as of right now, and either some specialized columns or tables to track history..

• Instead, what if we just kept adding data to existing tables? Add a column for “date this information was true” to keep track.

Here’s what it looks like in practice. Instead of overwriting your dimension tables every day, you append to them:

dim_users
┌─────────┬───────────┬────────────┐
│ user_id │ followers │ ds         │
├─────────┼───────────┼────────────┤
│ 123     │ 150       │ 2024-10-01 │
│ 123     │ 180       │ 2024-10-02 │
│ 123     │ ...       │ ...        │
│ 123     │ 1200      │ 2025-01-16 │
└─────────┴───────────┴────────────┘

dim_devices
┌─────────┬─────────┬────────────┐
│ user_id │ device  │ ds         │
├─────────┼─────────┼────────────┤
│ 123     │ mobile  │ 2024-10-01 │
│ 123     │ mobile  │ 2024-10-02 │
│ 123     │ ...     │ ...        │
│ 123     │ desktop │ 2025-01-16 │
└─────────┴─────────┴────────────┘

dim_relationships:
┌─────────┬───────────┬───────────┬────────────┐
│ user_id │ friend_id │ is_mutual │ ds         │
├─────────┼───────────┼───────────┼────────────┤
│ 123     │ 789       │ true      │ 2024-10-01 │
│ 123     │ 789       │ true      │ 2024-10-02 │
│ ...     │ ...       │ ...       │ ...        │
│ 123     │ 789       │ false     │ 2025-01-16 │ ← changed
└─────────┴───────────┴───────────┴────────────┘

fct_post_views:
┌─────────┬───────────┬───────────┬────────────┐
│ post_id │ viewer_id │ poster_id │ ds         │
├─────────┼───────────┼───────────┼────────────┤
│ 5001    │ 123       │ 789       │ 2024-10-01 │
│ 5002    │ 123       │ 456       │ 2024-10-01 │
│ 5003    │ 123       │ 789       │ 2024-10-05 │
│ ...     │ ...       │ ...       │ ...        │
│ 9999    │ 123       │ 789       │ 2025-01-15 │
└─────────┴───────────┴───────────┴────────────┘

Now that impossible retention query becomes straightforward. No BETWEEN clauses, no valid_from/valid_to logic – just filter each table to the date you want:

-- For fast-growing users, what device did they use back then?

WITH
  today_users as (SELECT user_id, followers as today_followers
      FROM dim_users WHERE ds = ‘2025-01-16’ AND followers >= 1000),
  past_users as (SELECT user_id, followers as past_followers
      FROM dim_users WHERE ds = ‘2024-10-01’ AND followers < 200),
  past_device as (SELECT user_id, device
      FROM dim_devices WHERE ds = ‘2024-10-01’),
  user_device as (
      SELECT tu.user_id, today_followers, past_followers, pd.device
      FROM past_users pu
      JOIN today_users tu ON pu.user_id = tu.user_id
      JOIN past_device pd ON tu.user_id = pd.user_id),
  views as (
      SELECT post_id, viewer_id, poster_id, ds
      FROM fct_post_views 
      WHERE ds BETWEEN ‘2024-10-01’ AND ‘2025-01-16’)
  SELECT
      ud.user_id,
      ud.device as device_during_growth,
      COUNT(DISTINCT views.post_id) as posts_from_mutuals
  FROM user_device ud
  LEFT JOIN views
      ON ud.user_id = views.viewer_id
  LEFT JOIN dim_relationships past_rels
      ON views.viewer_id = past_rels.user_id
      AND views.poster_id = past_rels.friend_id
      AND views.ds = past_rels.ds -- mutual status AS OF view date
      AND past_rels.is_mutual = true
  GROUP BY 1, 2

Is this query complex? Sure.2 But the complexity is in the business logic (what you’re trying to measure), not in fighting with valid_from/valid_to dates. Each query just filters to ds = {the date I want}. That’s it.

The idea is that you’re not overwriting existing tables. You are appending. 3

Sidebar: Common Table Expressions

If I had a SECOND “one weird trick” for data engineering, CTEs would be it. CTEs are just fucking fantastic. With liberal use of common table expressions (the WITH clause you saw in the retention query above), you can treat subqueries like variables – and then manipulating data feels more like code. Make sure your query engine (like Presto/Trino) flattens them for free – but if it does: wowee! SQL just got dirt simple. (a free one hour course on CTEs here)

When you grab data into your warehouse4, append a special column. That column is usually called “ds” – probably short for datestamp. You want something small and unobtrusive. (Notice that “date” would be a bad name – because you’d confuse people between this (date of ingestion of data) and the more obvious sort of date – date the action happened.) For snapshots, copy over the entire data of the snapshot, and have your “ds” column be <today’s date>. For logs, you can just grab the logs since yesterday, and set the ds column to <today’s date>.

Sidebar: Date stamps vs Date partitions
I’ll mostly say “date stamps” in this piece – the concept of marking each row with when that data was valid/ingested.

“Date partitions” is how most warehouse tools *implement* date stamps. A partition is how your warehouse physically organizes data. Think of it like: all rows with ds=2025-01-15 get grouped together in one chunk, ds=2025-01-16 in another chunk, and so on. (In older systems, each partition was literally a separate folder. Modern cloud warehouses abstract this, but the concept remains.)

Why does this matter? When you query `WHERE ds=’2025-01-15`, your warehouse only scans that one partition instead of the entire table. This makes queries faster and cheaper (especially in cloud warehouses where you pay per data scanned).

People use the terms interchangeably. The important thing is the concept: tables with a date column that lets you query any point in history.

Every table emanating from your input tables should add a filter (WHERE ds={today}), and similarly append the data to the table (WHERE ds={today}). (Except special circumstances where a pipeline might want to look into the past).

That’s it! Now your naive setup (overwriting everything every day) has only changed a bit (append everything each day, and keep track of what you appended when) – but everything has become so much nicer.

This is huge

This has two major implications:

First, many types of analysis become much easier. Want to know about the state of the world yesterday? Filter with WHERE ds = {yesterday}. Need data from a month ago? Filter with WHERE ds = {a month ago}. You can even mix and match – comparing today’s data with historical data, all within simple queries.

Second, data engineering becomes both easier and much less error prone. You can rerun jobs, create tables with historical data, and fix bugs in the past. Your pipeline will produce consistent, fast, reliable results consistently

What “functional” actually means

(Aka “I don’t know what idempotent means and at this point I’m afraid to ask”)

So, in Maxime’s article (link) there’s all this talk about “functional data engineering”. What does that even mean? Let’s discuss.

First, we’re borrowing an idea from traditional programming. “Functional programs” (or functions) meet certain conditions:

1. If you give it the same input, you get the same output. Every time.

2. State doesn’t change. Your inputs won’t change, hidden variables won’t change. It’s clean. (AKA “no side effects”)

Okay, so what does that mean for pipelines? Functional pipelines:

• Given the same input, will give the same output

• Don’t use (or rely on) magic secret variables

This is what people mean when they say “idempotent” pipelines or “reproducible” data.

And here’s how to implement it: datestamps.

• Your rawest/most upstream data should never be deleted – just keep appending with datestamps

• Pipelines work the same in backfill mode vs normal daily runs

• If you find bugs, fix the pipeline and rerun – the corrected data overwrites the bad data

• Time travel is built in – just filter to any ds you need

Datestamps also give you the nice side-effect of having it be very clear how fresh the data you’re looking at is. If the latest datestamp on your table is from a week ago -- it’s instantly understandable not only what’s wrong, but also you have hints about why.

Sidebar – what this looks like in practice:
Your SQL will look something like: WHERE ds=’{{ ds }}’ (Airflow’s templating syntax) or WHERE ds=@run_date (parameter binding).

Your orchestrator injects the date - whether it’s today’s scheduled run or a backfill from three months ago. Same SQL, different parameter. That’s the whole trick.

Backfilling is now easy, simple, magical

Remember that retention query? Now imagine you built that analysis pipeline three months ago, but you just discovered a bug in your dim_relationships table. The is_mutual flag was wrong for two weeks in November. You fixed the bug going forward, but now all your retention metrics from that period are wrong.

With the old SCD Type-2 approach, you’re in hell:

You can’t just “rerun November.” Because each day’s pipeline depended on the previous day’s state. Day 15 updated rows from Day 14, which updated rows from Day 13, and so on. To fix November 15th, you’d need to:

1. Rerun November 1st (building from October 31st’s state)

2. Wait for it to finish

3. Rerun November 2nd (building from your new November 1st)

4. Wait for it to finish

5. Rerun November 3rd...

6. ...keep going for 30 days, sequentially, one at a time

And this is assuming nothing breaks along the way. If Day 18 fails? Start over. Need to fix December too? Add another 31 sequential runs.

In Airflow terms, this is what depends_on_past=True does to you. Each day is blocked until the previous day completes. Backfilling becomes painfully slow. But that’s by no means the worst part.

You can’t just hit “backfill” and walk away. Your normal daily pipeline logic doesn’t work for backfills. Why? Because SCD Type-2 requires you to:

• Close out existing rows (set their valid_to date)

• Insert new rows (with new valid_from dates)

• Update is_current flags

• Handle the case where a row changed multiple times during your backfill period

Your daily pipeline probably has logic like:

-- Daily SCD Type-2 pipeline (simplified)
-- Step 1: Close out changed rows
UPDATE dim_users
SET valid_to = CURRENT_DATE - 1, is_current = false
WHERE user_id IN (
SELECT user_id FROM users_source_today
WHERE <something changed>
)
AND is_current = true;

-- Step 2: Insert new versions
INSERT INTO dim_users (user_id, followers, valid_from, valid_to, is_current)
SELECT user_id, followers, CURRENT_DATE, NULL, true
FROM users_source_today;

This works fine when you’re processing “today.” But for a backfill? You need different SQL:

• You need to carefully reconstruct valid_from/valid_to for historical dates

• And handle the fact that a user might have changed multiple times during your backfill window

• This gets messy fast.

• You’re essentially rewriting your pipeline. (WHY?)

So now you’re not just waiting 30 sequential days - you’re maintaining two separate codebases: one for daily runs, one for backfills. And every time you change your daily logic, you need to update your backfill logic to match. More code to write, more code to test, more places for bugs to hide. It’s completely useless and unnecessary.

Sidenote – even worse, if you’re outside your retention window (say, the source data from 90 days ago has been deleted), you can’t backfill at all. You’d need to completely rebuild the entire table from scratch, from whatever historical snapshots you still have. Which probably means... datestamped snapshots anyway. Womp womp.

With datestamps, backfilling is trivial:

Your pipeline for any given day just needs:

• Input tables filtered to ds=’2024-11-15’ (or whatever day you’re processing)

• Write output to ds=’2024-11-15’

That’s it. November 15th doesn’t need November 14th. It just needs the snapshot from November 15th.

So to fix your broken November data:

# In Airflow (or whatever orchestrator)
> airflow dags backfill my_retention_pipeline \--start-date 2024-11-01 \--end-date 2024-11-30

What happens behind the scenes?

• All 30 days kick off in parallel (up to your concurrency limits)

• Each day independently reads from its ds partition

• Each day independently writes to its ds partition

• No coordination needed between days

• The whole month finishes in the time it takes to run one day

The exact same SQL that runs daily also handles backfills - no special logic, no custom code

This changes everything:

No more custom SQL for backfills - It’s just a button you push. Your orchestrator handles it. The same pipeline code that runs daily also handles backfills. No special logic needed.

New tables get history for free - Created a new dim_users_enriched table today but want to populate it with the last year of data? Just backfill 365 days. Since your input tables have datestamps, the data is sitting there waiting.

Bugs in old data become fixable - Fix your pipeline logic, backfill the affected date range, done. The old (wrong) data gets overwritten with the new (correct) data for those specific partitions. Everything downstream can reprocess automatically.

Upstream changes cascade easily - Fixed a bug in dim_users? All downstream tables that depend on it can backfill the affected dates in parallel. The whole warehouse stays in sync.

This is possible because your pipelines are idempotent. Run them once, run them a thousand times - given the same input date, you get the same output. No hidden state, no “current” vs “historical” logic, no manual date math.

One pattern to avoid: Tasks that depend on the previous day’s partition of their own table. If computing today’s dim_users requires yesterday’s dim_users, you’ve created a chain - backfilling 90 days means 90 sequential runs that can’t be parallelized. This is sometimes necessary for cumulative metrics, but most dimension tables don’t need it - just recompute from raw sources each day.

For most datestamped pipelines, depends_on_past should be False. Each day is independent - the only dependency is “does the upstream data exist for this ds?”

Welcome to the magic of easy DE work

We started this article staring at the prospect of valid_from/valid_to logic, sequential backfills that take days, and custom SQL for every backfill and cascading for every bugfix. Yuck. Ew!

Or maybe – worse – with no sense of history at all. No ability to ask “how did the world look like yesterday”, much less “3 months ago”. I’ve seen startups and presidential campaigns and 500 million dollar operations operate like this. 🙃

Now you know the secret. Now you have the magic. What mature companies have been doing all along: snapshot your data daily, append it with datestamps, and write idempotent pipelines on top.

That’s it. That’s the whole One Weird Trick. Add a ds column to every table. Filter on it. Write your pipelines to be independent of each other. Have every pipeline be ds-aware. Storage is cheap. Your time is expensive. Getting your data wrong is extra expensive.

What you get in return:

• Backfills that run in parallel and finish in minutes instead of days

• Backfills that are a button push instead of custom SQL mess.

• Historical queries that are simple WHERE ds=’2024-10-01’ filters instead of date-range gymnastics

• Pipelines that are the same whether you’re processing today or reprocessing last year

• A built-in time machine for your entire warehouse

• Bugs that are fixable instead of permanent scars on your data

This is functional data engineering. Functional as in idempotent. And functional as in “it works”.

Your backfills are easy now. Your 3am alerts will be rarer. Time complexity is solved. Data recovery is trivial. Your job just became so much easier.

But we’re not done yet. Part 3 will tackle: how to scale your team and your warehouse. Parts 4 and 5 are gonna get me back on my “he who controls metrics controls the galaxy” soapbox.

For now, go add some datestamps. Your future self will thank you.

Hey, it’s Zach again. Sahar is currently open to work in NYC. Make sure to follow Sahar’s blog to understand growth and what comes next, both personally and for your business! And more at sahar.io

Data engineering can actually be easy, fast, and resilient! All you have to embrace is a simple concept: Date-stamping all your data.

Why isn’t this the norm? Because – even in 2025 — , institutions haven’t really understood the implications that STORAGE IS CHEAP! (And your data team’s time is expensive).

Datestamping solves so many problems. But you won’t find it in a standard textbook. They’ll teach you “slowly changing dimensions Type 2” when the real answer is simpler and more powerful. You will find the answer in Maxime Beauchemin’s seminal article on functional data engineering. Here’s the thing – I love Max, but that article is not helpful to the majority of people who could learn from it.

What if I told you:

• We can have resilient pipelines.

• We can master changes to data over time.

• We can use One Weird Trick to marry the benefits of order and structure with the benefits of chaos and exploration.

That’s where this article comes in. It’s been 7 years in the making – all the stuff that you should know, but no one bothered to tell you yet. (At least, in plain english – sorry Max!)

• Part One: How to set up a simple warehouse (and which small bits of jargon actually matter)

• Part Two: Date-stamping. Understand this and everyone’s life will become easier, happier, and 90% more bug-free.

• Part Three: Plugging metrics into AB testing. Warehousing enables experimentation. Experimentation enables business velocity.

• Part Four: The limits of metrics and KPIs. It can be so captivating to chase short-term metrics to long-term doom.

I’ll show you a practical intro to scalable analytics warehousing, where date stamps are the organizing principle, not an afterthought. In plain language, not tied to any specific tool, and useful to you today Meta used this architecture even back in the early 2010s. It worked with Hive metastore. It still works with Iceberg, Delta, and Hudi.

But first, to understand why all this matters, you need some context about how warehouses work. Then I’ll show you the magic.

Sponsorship

Cut Code Review Time & Bugs in Half

Code reviews are critical but time-consuming. CodeRabbit acts as your AI co-pilot, providing instant Code review comments and potential impacts of every pull request.

Beyond just flagging issues, CodeRabbit provides one-click fix suggestions and lets you define custom code quality rules using AST Grep patterns, catching subtle issues that traditional static analysis tools might miss.

CodeRabbit has so far reviewed more than 10 million PRs, installed on 1 million repositories, and used by 70 thousand Open-source projects. CodeRabbit is free for all open-source repo’s.

Part one — A Simple Explanation of Modern Data Warehousing

Our goals and our context

We are here to build a system that gets all company data, tidily, in one place. That allows us to make dashboards that executives and managers look at, charts and tools that analysts and product managers can use to do deep dives, alerts on anomalies, and a breadth of linked data that allows data scientists and researchers to look for magic or product insights. The basic building blocks are tables, and the pipelines that create and maintain them.

Sidebar: DB vs Data lake? OLTP vs OLAP? Production vs warehouse? Here’s what you need to know.

A basic point about a data warehouse (or lake, or pond, or whatever trendy buzzword people use today) is that it is not production. It must be a separate system from “the databases we use to power the product”.

Both are “databases”, both have “data”, including “tables” that might be similar or mirrored – but the similarity should end there.

• Your production database is meant to be fast, serve your product and users. It is optimized for code to read and write.

• Your warehouse is meant to be human-usable, and serve people inside the business. It is optimized for breadth, for use by human analysts, and to have historical records.

Put it this way – your ecommerce webapp needs to look up an item’s price and return it as fast as possible. Your warehouse needs to look up an item from a year ago, and look at how the price changed over the course of months. The database powering the webapp won’t even store the information, much less make it easy to compute. Meanwhile if you run a particularly difficult query, you don’t want your webapp to slow down.

So – split them. (You might hear people talking about OLTP vs OLAP – it’s just this distinction. Ignore the confusing terminology. Here’s a deep dive into the two types of OLAP data model (Kimball and One Big Table) )

So, we want a warehouse. Ideally, it should:

• Be separate from our production databases

• Collect all data that is useful to the company

• Have tables that make queries easy

• Be correct – with accurate, trusted, information

• Be reasonably up to date – with perhaps a daily lag, rather than a weekly or monthly one

• Power charts and interactive tools, while also being useful for automatic and local queries

This used to be difficult! (It is not anymore!) There was a tradeoff between “big enough to have all the data we need” and “give answers fast enough to be useful”. A lot of hard work was put into reconciling those two needs.

Since circa 2015 or so, this pretty much no longer a problem. Presto/Trino, Spark, and hosted databases (BigQuery, Snowflake, the AWS offerings) and other tools allow you to have arbitrarily huge data, accessed quickly. We live in a golden age.

Sidebar: At my old school…
At Meta, they used HDFS and Hive to power their data lake and MySQL to power production. Once a day they took a “snapshot” of production with a corresponding date stamp and moved the data from MySQL to Hive.

In a world where storage is cheap, access to data can be measured in seconds rather than minutes or hours, and data is overflowing, the bottleneck is engineering time and conceptual complexity. Solving that bottleneck allows us to break with annoyingly fiddly past best practices. That’s what I’m here to talk about.

A basic setup

Imagine your warehouse as a giant box, holding many, many tables. Think of data flowing downhill through it.

• At the top: raw copies from production databases, marketing APIs, payment processors, whatever.

• At the bottom: clean, trusted tables that analysts actually query.

• In between: pipelines that flow data from table to table.

[Raw Input Tables]
├─ users_production
├─ events_raw
├─ transactions_raw [Pipelines]
└─ ... ↓

     Clean → Join → Enrich

                 ↓

[Clean Output Tables]
├─ dim_users
├─ fct_events
└─ grp_daily_revenue

How do we get from raw input to clean tables? Pipelines. (See buzzwords like ETL, ELT? Ignore the froth – replace with “pipelines” and move on).

Pipelines are the #1 tool of data engineering. At their most basic form, they’re pieces of code that take in one or more input tables, do something to the data, and output a different table.

What language do you write pipelines in? Like it or not, the lingua franca of editing large-scale data is SQL. The lingua franca of accessing large scale data is SQL. SQL is a constrained enough language that it can parallelize easily. The tools that invisibly translate your simple snippets into complex mechanisms to grab data from different machines, transform it, join it, etc – they not only are literally set up with SQL in mind, they figuratively cannot do the same for python, java, etc. Why? Because a traditional programming language gives you too much flexibility -- there’s no guarantee that your imperative code can be parallelized nicely.

Sidebar: When non-SQL makes sense (or doesn’t)

If you’re ingesting data from the outside world (calling APIs, reading streams, and so on), then python, javascript, etc could make sense. But once data is in the warehouse, beware anything that isn’t SQL – it’s likely unnecessary, and almost certainly going to be much slower than everything else.

Your tooling might offer a way to “backdoor” a bit of code (e.g. “write some java code that calls an API and then writes the resultant variable to a column”). Think twice before you use it. Often, it’s easier and faster to import a new dataset into your warehouse so that you can recreate with SQL joins what you would have done using an imperative language.

You may be tempted to transform or analyze data in R, pandas, or whatnot – that’s fine, but you do that by interactively reading from the warehouse. Rule of thumb: if you’re writing between tables in a warehouse – SQL. Into a warehouse – you probably need some glue code somewhere. Out of a warehouse – that’s on you.

So here’s the simple setup:

Then, set up a system of pipelines to this, every day, as soon as the upstream data is ready:

• As each of these input tables gets the latest dump of data from outside: take that latest day’s data, deduplicate, clean it up a bit, rename the columns, and cascade it to a nicer, cleaner version of that table. (this is your silver tier data in medallion architecture)

• Then, from that nicer input table, perform a host of transformations, joins, etc to write to other downstream tables. (this is your master data)

• Master data is highly trusted which makes building metrics and powering dashboards easy!1

Every day, new data comes in, and your pipeline setup cascades new information in a host of tables downstream of it. That’s the setup.

A well-ordered table structure

Okay, so to review: the basic useful item in a warehouse is a table. Tables are created (and filled up by) pipelines.

“Great, great,” you might say – “but which tables do I build?”

For the sake of example, let’s imagine our product is a social network. But this typology should work just as well for whichever business you are in – from b2b saas to ecommerce to astrophysics.

From the perspective of the data warehouse as a product, there are only three kinds of tables: input tables (copied from outside), staging tables (used by pipelines and machines), and output tables – also known as user-facing tables.

Output tables (in fact, almost all tables) really only have three types:

• Tables where each row corresponds to a noun. (E.g. “user”, or even “post” or “comment”). When done right, these are called dimension tables. Prefix their names with dim_

• Tables where each row corresponds to an action. Think of them as fancier versions of logs. (E.g. “user X wrote post Y at time Z”). When done right, these are called fact tables. Prefix their names with fct_

• Everything else. Often these will be summary tables. (e.g. “number of users who made at least 1 post, per country, per day). If you’re proud of these, prefix them with sum_ or agg_.

Sidebar: more on naming

YMMV, but I generally don’t prefix input tables. Input tables should be an exact copy of the table you’re importing from outside the warehouse. Changing names breaks that – and an unprefixed table name is a good sign that the table cannot be trusted.

Staging and temporary tables are prefixed with stg_ or tmp_.

Let’s talk more about dimension and fact tables, since they’re the core part of any clean warehouse.

Dimension tables are the clean, user-friendly, mature form of noun tables.

• Despite being focused on nouns (say, users), they can also roll up useful verby information (leveraging cumulative table design)

• For instance, a dim_users table might both include stuff like: user id, date created, datetime last seen, number of friends, name; AND more aggregate “verby” information like: total number of posts written, comments made in the last 7 days, number of days active in the last month, number of views yesterday.

• If a data analyst might consistently want that data – maybe add it to the table! Your small code tweak will save them hours of waiting a week.2

(Now, what’s to stop the table from being unusably wide? Say, with 500+ columns? Well, that’s mostly an internal culture problem, and somewhat a tooling problem. You could imagine, say, dim_user getting too large, so the more extraneous information is in a dim_user_extras table, to be joined in when necessary. Or using complex data types to reduce the number of columns)

Fact tables are the clean, user-friendly, mature form of logs (or actions or verb tables).

• Despite being verb focused, fact tables contains noun information. (Zach chimes in: here’s a free 4 hour course on everything you need to know about fact tables)

• Unlike a plain log, which will be terse, they can also be enriched with data that might probably live in a dim table.

• The essence of a good fact table is providing all the necessary context to do analysis of the event in question.

• A fact table, fundamentally, helps you understand: “Thing X happened at time Y. And here’s a bunch of context Z that you might enjoy”.

• So a log containing “User Z made comment Xa on post Xb at time Y” could turn into a fct_comment table, with fields like: commenter id, comment id, post id, time, time at commenter timezone, comment text, post text, userid of owner of post, time zone of owner of parent post. Some of these fields are strictly speaking unnecessary – you could in theory do some joins to grab the post text, or the comment text, or time zone of the owner of the parent post. But they’re useful to have handy for your users, so why not save them time and grab them anyway.

Q: Wait – so if dim tables also have verb data, and fact tables also have noun data, what’s the difference?

A: Glad you asked. Here’s what it boils down to – is there one row per noun in the table? Dim. One row per “a thing happened?” Fact. That’s it. You’re welcome.

Here, as in so much, we are spending space freely. We are duplicating data. We are also doing a macro form of caching – rather than forcing users to join or group data on the fly, we have pipelines do it ahead of time.

Compute is cheap, storage is cheap. Staff time is not. We want analysis to be fluid and low latency – both technically in terms of compute, and in terms of mental overhead.

Q: Wait! What about data stamps? Where’s the magic? You promised magic.

A: Patience, young grasshopper. Part of enlightenment is the journey. Part of understanding the magic is understanding what it builds on. And – hey – would YOU read a huge blog post all at once? Or would you prefer to read it in chunks. Yeah, you with your Tiktok problem and inability to focus. I’m surprised you even made this far.

Stay tuned for part two where we:

• Show you how to make warehousing dirt easy

• Behold the glory of date stamping

  • Through better data quality

  • How it avoids backfill nightmares

  • Bug fixes are easy now

  • You get a time machine for free

• Explore the dream of functional data engineering (what is that weird phrase?)

• Throw SCD-2 and other outdated “solutions” to the dustbin of history

Make sure to follow Sahar’s blog to understand growth and what comes next, both personally and for your business! Sahar is currently open to work, he is interested in DevRel and engineering leadership roles in New York City.

Knowing where to start and how to get a handle on this requires some guidance. This newsletter is going to unveil the steps needed to break into data engineering in 2025 and how AI fits into the picture.

A lot of people still think that after you’ve done a magical number of Leetcode problems, a job falls into your lap. That’s almost never the case!

To get a job in 2025, you’ll need the following things:

• Demonstrable skills with:

  • SQL and Python

  • Distributed compute (Snowflake, Spark, BigQuery, DuckDB)

  • Orchestration knowledge (Airflow, Mage, or Databricks workflows)

  • Data modeling and data quality

  • AI/data integrations (vector databases, embeddings, RAG pipelines)

• An opportunity to demonstrate those skills via a portfolio project

• A personal brand that radiates above the noise both on LinkedIn and in interviews

Let’s dig into each of these areas and see how you can fast-track your way to success!

Sponsorship

Tinybird helps companies like Vercel and Framer transform massive streams of real-time data into instant insights and real-time user experiences.

They built a free course on real-time data designed for data engineers who want to master the basics and build amazing data products.

Learning SQL

Avoiding SQL is the same as avoiding a job in data engineering. This is the most fundamental language you need to know.

There are many resources out there to learn it! The ones I recommend are:

• DataExpert.io free SQL question practice

• DataLemur

• StrataScratch

Key things you should know in this bucket are:

• The basics

  • JOINs

    • INNER, LEFT, FULL OUTER

      • Remember you should almost never use RIGHT JOIN

  • Aggregations with GROUP BY

    • Know the differences between COUNT and COUNT(DISTINCT)

      • Remember that COUNT(DISTINCT) is much slower in distributed environments like Spark

    • Know how to use aggregation functions with CASE WHEN statements

      • example: COUNT(CASE WHEN status = ‘expired’ THEN order_id END) this counts the number of expired orders

    • Know about cardinality reduction and bucketing your dimensions

      • example:

        • SELECT CASE WHEN age > 30 THEN ‘old’ ELSE ‘young’ END as age_bucket, COUNT(1) FROM users GROUP BY 1

      • You’ll see in this query, we take a high cardinality dimension like age and make it a lower cardinality (just two values “old” and “young”)

• Common Table Expressions vs Subquery vs View vs Temp Table (a great YouTube video here)

  • The key things here are:

    • You should very rarely be using subquery (it hurts readability of pipelines)

    • You should use temp table if you need to reuse some logic since Temp Table gets materialized and will improve the performance of your pipeline

    • You should use View when you need to store logic for longer than the duration of the pipeline execution

    • In all other cases, you should use common table expressions to improve readability!

• Understand how SQL works in distributed environments

  • Know what keywords trigger shuffle

    • JOIN, GROUP BY, ORDER BY

  • Know what keywords are extremely scalable (this means they’re executed entirely on the map-side)

    • SELECT, FROM, WHERE, LIMIT

• Know window functions thoroughly (great YouTube video here)

  • The basics

    • RANK() OVER (PARTITION BY <partition> ORDER BY <order by> ROWS BETWEEN <preceding> PRECEDING AND <following rows>)

    • You have the function (e.g. RANK, SUM, etc).

    • You have PARTITION BY this divides the window up. Maybe you want to do window functions per department or country?

    • You have ORDER BY this determines the sorting of the window

    • You have the ROWS BETWEEN clause to determine how many rows you should include in your window. If you don’t specify this, it defaults to ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW. So the default is the “cumulative” row definition.

  • Understand RANK vs DENSE_RANK vs ROW_NUMBER(a quick two minute YouTube video about this)

    • Key things:

      • When there is no tie in your ORDER BY clause, these functions are identical

      • When there are ties,

        • RANK skips values (e.g. a tie for 1st place means the next place is third)

        • DENSE_RANK does not skip values (e.g. a tie for 1st place means the next place is second)

        • ROW_NUMBER guarantees unique values with no ties (a tie for first place means one of them will get second place, this is based on the natural ordering of the data)

    • Understand how to do “rolling” calculations

      • Rolling average and sum by department is a common interview question. You can solve it with a query like this:

        • SELECT revenue_date, SUM(revenue) OVER (PARTITION BY department ORDER BY revenue_date ROWS BETWEEN 30 PRECEDING AND CURRENT ROW) as thirty_day_rolling_revenue FROM daily_sales

        • You’ll see we split the window by department, then we look at the rolling 30 day period for each day and sum it up. You need to be careful here and ensure that there is data (even if it’s zero) for each date otherwise you’ll calculations will be wrong!

  • Know about the differences between INSERT INTO, INSERT OVERWRITE and MERGE

    • INSERT INTO just copies the data from the result query into the table. THIS IS PRONE TO DUPLICATES! If you’re using INSERT INTO, it should always be coupled with either TRUNCATE or a DELETE statement!

    • INSERT OVERWRITE is nice because it copies the data and replaces whatever existing data is in that partition. This is the most common one they use in big tech!

    • MERGE is nice because it looks at the existing data and copies only the rows that are updated and/or deletes the rows that aren’t in the incoming data. The only minus of MERGE is the comparisons it needs to do to accomplish this can be very slow for large data sets!

What’s new in 2025?

• AI copilots like Databricks Genie and Snowflake Cortex will happily write SQL for you. But interviewers now test whether you can verify, debug, and optimize AI-generated SQL.

• Understanding how SQL runs in distributed environments (shuffle, partitioning, scalability) is still your edge over someone blindly trusting a copilot.

Learning Python (and AI Integration)

SQL is great and can accomplish a lot in data engineering. There are limitations to it though. To overcome these limitations you need a more complete language like Python.

Here are the concepts you need to learn:

• Data types and data structures

  • Basics

    • strings, integers, decimals, booleans

  • complex

    • lists (or arrays), dictionaries, stacks and queues

  • Other data structures you probably don’t need to learn (although these might show up in interviews which sucks!)

    • heaps, trees, graphs, self-balancing trees, tries

• Algorithms

  • Basics

    • Loops, linear search, binary search

    • Big O Notation

      • You should be able to write down both the space and time complexity of algorithms

  • Algorithms you probably don’t need to know (although these might show up in interviews which sucks!)

    • Dijkstra's algorithm, dynamic programming, greedy algorithms

• Using Python as an orchestrator

  • One of the most common use cases for Python in data engineering is to construct Airflow DAGs

• Using Python to interact with REST APIs

  • A common data source for data engineers is REST data. Learn about GET, POST, PUT request in Python. The requests package in Python is great for this!

• Know how to test your code with pytest (in Python) or JUnit (in Scala)

  • There’s a really solid library called Chispa that works well with pytest for testing your PySpark jobs you should check out!

New in 2025:
Python is now the glue between data engineering and AI. You’ll need to know:

• Calling LLM APIs (OpenAI, Anthropic, open-source models)

• Generating and storing embeddings

• Working with vector databases (Pinecone, Milvus, Weaviate, pgvector)

• Building lightweight RAG (Retrieval-Augmented Generation) pipelines

• Writing “AI validators” — Python jobs that use an LLM to check data quality or generate documentation

Learning Distributed Compute (either Spark, BigQuery or Snowflake)

Back in the late 2000s, Hadoop was born and so was the notion of distributed compute. This means that instead of having one fancy computer process all your data you have a team of computers each process a small chunk!

This concept unlocks the possibility of computing vast amounts of data in a small amount of time by leveraging teamwork! But this does not come without complexity!

Here are some things to consider:

• Shuffle

  • If we are using teamwork, we need to guarantee certain data is on a certain machine (like if we are counting how many messages each user has received). The team accomplishes this guarantee by passing all of your data to one machine via shuffle (example in the diagram below). We only HAVE to do this when we do GROUP BY, JOIN, or ORDER BY. (a quick 2 minute video about how I managed this at petabyte scale at Netflix)

• Shuffling isn’t a bad thing remember! It actually is really good because it makes distributed compute mostly the same as single node compute! The only time it gets in the way of things is at very large scale!

  • Things you should consider to reduce shuffling at very large scale

    • Broadcast JOIN

      • If one side of your JOIN is small (< 5 GBs), you can “broadcast” the entire data set to your executors. This allows you to do the join without shuffle which is much faster

    • Bucket JOIN

      • If both sides of your JOIN are large, you can bucket them first and then do the join, this allows. Remember you’ll still have to shuffle the data once to bucket it, but if you’re doing multiple JOIN with this data set it will be worth it!

    • Partitioning your data set

      • Sometimes you’re just trying to JOIN too much data because you should JOIN one day of data not multiple. Think about how you could do your JOIN with less data

    • Leverage cumulative table design

      • Sometimes you’ll be asked to aggregate multiple days of data for things like “monthly active users.” Instead of scanning thirty days of data, leverage cumulative table design to dramatically improve your pipeline’s performance!

  • Shuffle can have problems too! What if one team member gets a lot more data than the rest? This is called skew and happens rather frequently! There are a few options here:

    • In Spark 3+, you can enable adaptive execution. This solves the problem very quickly and I love Databricks for adding this feature!

    • In Spark <3, you can salt the JOIN or GROUP BY. Salting allows you to leverage random numbers so you get a more even distribution of your workload among your team members!

• Output data

  • Most often when you’re using Spark, you’re going to be writing Parquet files to S3 or GCP or Azure. Parquet files have some interesting properties that allow for dramatic file size reduction if you use them right. (an in depth YouTube video about this).

    • The key property of parquet files that you need to know about is a thing called run length encoding compression. I shrunk pricing and availability datasets at Airbnb by over 90% by leveraging this technique!

  • If you’re using Snowflake or BigQuery, they use their own proprietary format that I know less about. So find another newsletter for that! Maybe Seattle Data Guy’s?

What’s new in 2025?

• AI workloads run on your data stack (i.e. Shift Left)

  • Embedding generation at scale (GPU clusters with Spark or Ray)

  • Hybrid query engines that support both structured tables + vector search

  • Streaming inference pipelines with Kafka + Flink

Data Modeling and Data Quality

Data engineering is ultimately about delivering usable, correct, privacy-compliant data.

Here are the ways you can assess data quality:

• Data should be correct

  • You should check for duplicates, NULLs, proper formatting. Also checking that there is the right number of rows in the data set!

• Data should be usable and efficient

  • This means it should have proper documentation and good column names. The query patterns should also allow for fast answers to questions! Answers shouldn’t give Jeff Bezos millions of dollars either!

• Data should be privacy-compliant

  • An often overlooked part of the puzzle. You shouldn’t be hurting user privacy to make your analytics better!

So how do you achieve this output data set nirvana dream? It has multiple parts!

• Correctness should be handled in a few ways

  • The first pass should be via validation from a data analyst. This a part of the powerful MIDAS process at Airbnb that you should check out!

  • After that, you should build in automated data quality checks into your pipeline with something like Great Expectations. Remember to follow the write-audit-publish pattern here so you don’t publish bad data into production that doesn’t pass data quality checks!

• Usability and efficiency are handled with a few things

  • Documentation should be a big part of the process. Spec building and stakeholder sign-off should happen BEFORE you start building your pipeline. This will prevent a lot of redoing and undoing of work!

  • Leveraging efficient practices for your data lake:

    • Manage your Apache Iceberg snapshots for time-traveling and disaster recovery

    • Set good retention policies on your data so you don’t have a huge cloud bill for data you do not use!

  • Data modeling is going to be the other big piece of this puzzle

    • There are a few camps here:

      • Relational data modeling

        • This type prioritizes data deduplication at the cost of more complex queries. Think of this as prioritizing storage at the cost of compute.

      • Dimensional (or Kimball) data modeling

        • This denormalizes the data into facts and dimensions which prioritizes larger queries but duplicates data a bit. Think of this as trying to balance the costs of storage and compute.

      • One Big Table

        • This denormalizes the data even more where the facts and dimensions are in one table. You duplicate data more but you get extremely efficient queries from it. Think of this as prioritizing compute at the cost of storage.

      • I wrote a long-form article detailing the differences and when to pick which one here. If you prefer video format you can find it here

• Privacy compliant data sets

  • Be mindful where you have personally identifiable information in your data sets and don’t hold onto that longer than you need.

    • Remember that anything that can bring you back to a user is personally identifiable!

  • Anonymizing the data so you can hold onto it longer is a great strategy that balances user privacy and long-term analytical capabilities

New in 2025:

• AI-assisted quality checks. Tools now let you point LLMs at logs or query results to spot anomalies.

• Semantic modeling. Using embeddings to cluster or enrich tables with “semantic meaning.”

• Privacy in AI. You must ensure embeddings don’t leak personally identifiable information. Vector search can accidentally memorize sensitive data if you’re careless.

  

Building a Portfolio Project

This is where you separate yourself from the pack.

You should pick a project that:

• You care about deeply. I was obsessed about Magic the Gathering and that was what allowed me to keep working on that project even though I wasn’t getting paid

• You can work on it for 3 months at 5-10 hours per week. A portfolio piece shouldn’t be easy to create. If it was easy to create, everybody would do it and then you wouldn’t stand out!

• You build a piece that has a frontend! Another portfolio piece I build was HaloGods.com a website I built that allowed me to reach 20th in the world in Halo 5 back in 2016. Without a frontend, your data pipelines are a little bit harder to show off. This is why learning a skill like Tableau or Power BI can be a really solid way to make your portfolio shine even if those aren’t skills you end up using on the job!

• You implement the following things in your portfolio

  • A comprehensive documentation that details all the inputs and output data sets, the data quality checks, how often the pipeline runs, etc.

  • You have a pipeline running in production. Using something like Databricks Free Edition would be a great place to get started!

  • You leverage hot technologies like Spark, Snowflake, Vector Databases, Iceberg, and/or Delta Lake.

  • If you want to stand out, you build a JavaScript front end like I did with HaloGods and you’ll find yourself landing jobs in big tech even though you didn’t go to Stanford, MIT or IIT.

In 2025, you’ll want something that shows both data pipelines and AI integration. Remember you can use AI to get the boilerplate and you add your own details to get started!

Ideas:

• E-commerce pipeline → warehouse model → AI recommendation system

• Ingesting data from Polygon.io → stock market model → AI stock picker system

• YouTube transcript ingestion → Iceberg tables → semantic search UI with RAG

• Automated data quality alerts using GPT validators

Remember, you might have to pay a tiny bit of money to build these projects. But it will be worth it over the long run!

Building a Personal Brand

So you have the skills and the sexy portfolio project. Now all you got to do is not mess up the job interview and you’ll be golden.

Here are some things you should do to get there:

• Build relationships on LinkedIn

  • You should be building relationships with

    • Hiring managers and recruiters

    • Peers

  • You should start talking with people BEFORE you ask for a referral and start building up friendships and your network.

    • If you need a referral, it would be better to send out DMs to people and ask them what the job is like. Leading with questions instead of tasks will have a much higher hit rate!

    • Also remember that this DM game is a low percentage play. If you send out 20 DMs, 1 or 2 might respond, especially if you’re early in your career. Finding creators and employees that are near your same level will help increase that hit rate!

  • Create your own content and talk about your learning journey. You’d be surprised how effective this is at landing you opportunities you didn’t even think were possible. For example, I made $600k from LinkedIn in 7 months after quitting my job! Content creation and branding are a very powerful combo that can change your life!

    • Nowadays with AI, you can use AI aid in the content creation process! Just remember to remove the ugly emojis, em dashes, and

• Interview like a person, not a robot

  • When you go into the interview make sure you have:

    • Researched the people who are interviewing you. You should know

      • How long they’ve worked for the company

      • What they do for the company

    • Asked the recruiter a lot of detailed questions about the role

      • What technologies will I be using

      • How many people will be on my team

      • What is the culture like

  • During the interview you should radiate

    • positivity, enthusiasm and excitement for the role

    • competence and calm when asked questions

    • curiosity to engage in stupid interview questions and curiosity about the role and what you’ll be doing by asking good follow up questions

  • Demonstrate technical skills during the interview

    • The SQL Interview

    • The Data Modeling Interview

    • The Data Architecture Interview

    • The Data Structures and Algorithms Interview

Conclusion

Getting into data engineering in 2025 is still hard. The job market is competitive. But the best engineers and the best startups are built in times like this.

The difference now? AI is part of the job. You don’t need to be an AI researcher, but you do need to know how to integrate AI responsibly into your pipelines.

Follow this roadmap and you’ll be much closer to landing the data engineering role of your dreams.

I am launching a new AI Engineering Boot camp starting on October 20th where we are covering all the new AI topics necessary to be a good AI-enabled data engineer. You can get 30% off with code AIROADMAP here.

What else do you think is critical to learn in data engineering to excel in the field? Anything I missed here that you would add? Please share this with your friends and on LinkedIn if you found it useful!

Read more

Introduction

Apache Spark has been the de facto open source data processing for fifteen years. It was invented to solve a major problem that traditional data warehousing was not built to solve - processing massive amounts of data horizontally at scale (Zach used Spark to process 2000 TBs per day at Netflix), whether in a structured or semi-structured for…

Read more

Read more

This month, I’m excited to present three fantastic opportunities to learn from top-tier practitioners without spending a dime.
Whether you’re trying to break into tech or looking to make a strategic pivot in your career, these free bootcamps are packed with real-world lessons, hands-on skills, and instruction from people who’ve been in the trenches.

We a…

Read more

There are some conversations that confirm what you already believe, and then there are conversations that take what you believe and frame it in a way that makes it actionable, precise and inevitable.

That’s what happened when I sat down with Chip Huyen during a Tech Talk we held at DataExpert.io AI Engineering Boot Camp.

Chip is an AI researcher, former big tech engineer, best-selling author and one of the most lucid thinkers on AI systems working today. Recently, she’s been in the public spotlight with her new book AI Engineering, which has quickly become the most comprehensive, well-structured guide to the essential aspects of building generative AI systems (we covered a lot of its content in the boot camp too).

This talk was neither a book launch nor a formal Q&A. It was a honest, refreshing & grounded conversation that can be distilled into seven core takeaways, each one capturing ideas Chip shared that stuck with me, challenged me or reshaped how I think about building and leading in AI. This article covers the following:

• Chip’s journey into AI

• Where most GenAI products go wrong

• The underrated value of UX

• How to build functional AI agents

• What really takes to ship value in the modern AI stack.

• How to stay informed without burning out

• Building with clarity and conviction

If you want to learn from other brilliant minds like Chip, we are launching a 10-week Challenge Boot Camp on Sep 15th where will be covering insightful tech talks with 15 industry leaders in the data, analytics and AI engineering space. The first 5 people to register can use code CHIP for 30% off!

✍🏻 #1: The Primacy of Compute and Data

To understand Chip’s journey into AI, we have to rewind to 2012, the year Deep Learning truly exploded onto the scene. That was the year AlexNet, a deep convolutional neural network, won the ImageNet competition by a massive margin (over 10 percentage points better than the next best model).

AlexNet rewrote the rules that have defined the last decade of AI. And notably, one of the paper's co-authors, Ilya Sutskever, would go on to co-found OpenAI, the organization that would later lead the charge on scaling up large language models.

Chip recounted how a single sentence from that 2012 paper changed her life trajectory:

“Our experiments show that we can achieve better results by just waiting for more compute and more data.”

That line reframed AI not as a field of breakthroughs, but as one of compounding scale. Chip went on to work at NVIDIA to understand compute infrastructure and later joined Snorkel AI to understand data workflows.

She also reflected on how OpenAI was initially dismissed for simply scaling up, with many academics saying it wasn't real research. But the turning point came in 2020 when the GPT-2 paper received a best paper award, and the conversation suddenly shifted.

Everyone was like, ‘Wow, now it’s real research.’

💡 I remember similar skepticism back in my days at Facebook. People dismissed what OpenAI was doing as brute force. It turns out brute force was the insight.

This key takeaways reframes my view of AI progress not as a parade of novel ideas but as an engineering problem of sufficient scale. Chip's clarity here gives me language to explain why things like GPT-5 didn't just appear but emerged from compute and data discipline.

✍🏻 #2: The GenAI Hype Cycle and the Misuse of ML

What makes Chip’s take on GenAI refreshing is that her relationship with AI long predates the hype. Before ChatGPT became a buzzword, she was building simple algorithms to test logic, even designing games her smartest friends couldn’t win. Not to outsmart them but to understand how reasoning could be codified.

Read more

Fake accounts come and go. Some are flagged incorrectly, others are later verified, and many are caught by internal ML systems. The goal of our pipeline was to trace the inflows and outflows of fake accounts daily. That meant building a reliable dataset that could track:

• Accounts unlabeled as fake (i.e. after submitting a valid ID)

• Accounts relabeled as fake

• Accounts flagged as fake for the first time

• Accounts that remained fake

The pattern was very straightforward: a classic cumulative table design. But the way it was wired to upstream data, specifically how it “waited” for inputs, created a non-deterministic nightmare. For weeks, I chased what I thought was a bug in my code, only to discover that the real problem had been there from day one.

In this article, I’ll cover the following aspects:

• Why relying on “latest” partition data broke everything

• How upstream non-deterministic leads to silent data mismatches

• The simple fix using explicit partition dates

• The tradeoff between latency and reproducibility

• Engineering lessons that go beyond code

If you want to learn more in depth about patterns like this, the DataExpert.io academy subscription has 200+ hours of content about system design, streaming pipelines, etc. The first 5 people can use code FAKE for 30% off!

If you enjoy this article, here are some more from my time in big tech:

• How I prepared for Airbnb’s staff data engineer interview

• How I achieved a 12x speed up on Facebook notification pipelines

• How I used the “Psycho” pattern to detect threats at Netflix

• How I cut Airbnb’s pricing data backfill time by 95%

Understanding Fake Account Flows

At a high level, the pipeline’s job was to compare today’s and yesterday’s fake account snapshots, determine who had entered or exited the fake state, and store those inflow/outflow transitions for downstream analytics.

This was a classic cumulative table design pattern, built in plain vanilla SQL and tracked four main fake states:

• New fakes → New people who got labeled as fake

• Resolved accounts → People who were labeled fake earlier than today and passed a challenge to remove the label

• Persisting fakes → People who were labeled fake earlier than today and haven’t passed a challenge

• Relabeled fakes → People who were labeled fake, passed a challenge, and then continued to do fake activity

Originally, it was set up like this:

1. Fake accounts pipeline waited on the “latest” partition of the users table

2. Then joined it with fake_accounts to compute transitions

3. The job ran daily and published results

-- Define struct and enum types

CREATE TYPE IF NOT EXISTS daily_detection_stats AS (
    detection_date DATE,
    login_attempts INTEGER,
    friend_requests_sent INTEGER,
    posts_created INTEGER,
    flagged_reports INTEGER
);

CREATE TYPE IF NOT EXISTS fake_classification AS ENUM('new', 'resolved', 'persisting', 'relabeled');

-- Create cumulative table
CREATE TABLE IF NOT EXISTS fake_accounts
(
    account_id TEXT,
    country TEXT,
    sign_up_method TEXT,
    sign_up_date DATE,
    daily_detection_stats daily_detection_stats[],
    fake_classification fake_classification,
    days_since_last_detected INTEGER,
    current_detection_date DATE,
    PRIMARY KEY (account_id, current_detection_date)
);


-- Dynamically pick the "latest" date
WITH latest_date AS (
    SELECT MAX(detection_date) AS detection_date FROM account_daily_signals
),
yesterday AS (
    SELECT * 
    FROM fake_accounts 
    WHERE current_detection_date = (SELECT detection_date - INTERVAL '1 day' FROM latest_date)
),
today AS (
    SELECT * FROM account_daily_signals 
    WHERE detection_date = (SELECT detection_date FROM latest_date)
)

-- Non-idempotent insert
INSERT INTO fake_accounts
SELECT
    COALESCE(t.account_id, y.account_id) AS account_id,
    COALESCE(t.country, y.country) AS country,
    COALESCE(t.sign_up_method, y.sign_up_method) AS sign_up_method,
    COALESCE(t.sign_up_date, y.sign_up_date) AS sign_up_date,
    CASE
        WHEN y.daily_detection_stats IS NULL THEN ARRAY[ROW(
            t.detection_date,
            t.login_attempts,
            t.friend_requests_sent,
            t.posts_created,
            t.flagged_reports
        )::daily_detection_stats]
        WHEN t.detection_date IS NOT NULL THEN y.daily_detection_stats || ARRAY[ROW(
            t.detection_date,
            t.login_attempts,
            t.friend_requests_sent,
            t.posts_created,
            t.flagged_reports
        )::daily_detection_stats]
        ELSE y.daily_detection_stats
    END AS daily_detection_stats,
    CASE
        WHEN y.account_id IS NULL THEN 'new'
        WHEN t.account_id IS NULL THEN 'resolved'
        WHEN t.detection_date IS NOT NULL AND y.fake_classification = 'resolved' THEN 'relabeled'
        ELSE 'persisting'
    END::fake_classification AS fake_classification,

    CASE
        WHEN t.detection_date IS NOT NULL THEN 0
        ELSE y.days_since_last_detected + 1
    END AS days_since_last_detected,
    COALESCE(t.detection_date, y.current_detection_date + INTERVAL '1 day')::DATE AS current_detection_date

FROM today t
FULL OUTER JOIN yesterday y
ON t.account_id = y.account_id;

The Architecture That Broke Everything

Read more

One of my most challenging assignments was owning the pipeline that deduplicated all notification events. This dataset drove downstream metrics like CTRs, conversions, and even machine learning signal quality.

This pipeline presented one big problem: it was slow. Very, very slow.

When I joined Facebook, the deduped notifications pipeline ran a giant Hive GROUP BY job once a day at UTC midnight which took 9.5 hours to complete. This latency issue represented a huge bottleneck for all downstream models in the Core Growth team.

This article is the story of how I brought 9.5-hour latency down to 45 minutes, and what it taught me about I/O, orchestration, and never trusting a “simple” DAG.

Here I’ll be covering the following:

• Why streaming deduplication at scale failed

• The hourly dedup job that exploded compute usage

• A tree-based DAG design that saved the day

• Key orchestration lessons from building a 300-step daily DAG

• How this project got me promoted and why I almost gave up

If you want to learn more in depth about patterns like this, the DataExpert.io academy subscription has 200+ hours of content about system design, streaming pipelines, etc. The first 5 people can use code DEDUP for 30% off!

The Stakes: Notifications at Facebook Scale

The notif_events table contained every event tied to a notification:

• Sent to your phone

• Delivered to device

• Clicked

• Converted

Since one person might click the same notification multiple times, we had to dedup those. If we counted every click, we’d get click-through rates above 100%, which negatively impacted metric tracking and, worse, model training.

But the problem wasn’t logic. It was latency.

As I mentioned earlier, the dedup job ran once a day via Hive & took 9.5 hours to complete. My managers wanted me to reduce latency dramatically.

Approach 1: Stream It 😣

At first, my manager’s request was straightforward: ‘Let’s dedup in real-time’.

So, I tried. I built a Spark Streaming job that listened to notification events and tried to hold recent activity in-memory for comparison. But this approach was holding as much as 50+ terabytes in RAM to do real-time deduping.

This wasn’t feasible. The streaming job crumbled under memory pressure. I had to come up with a better solution.

Approach 2: Hourly Dedup + Merge 🤔

Our second approach was to dedup every hour. The idea was simple:

1. Dedup the current hour table and write it to a sorted, bucketed table (to minimize shuffle later with SMB join)

2. Merge with the previous hour table (a cumulative of all previous hours’ deduped data).

3. Output a deduped table up to the current hour.

The read and merge hourly deduped table to the previous hour can be implemented with this simple SQL pattern inside your DAG:

-- Hourly dedup logic 
-- Remember this table is sorted and bucketed on user_id

INSERT OVERWRITE TABLE notif_deduped_hourly(ds='{{ ds }}', hour={{ current_hour}}, channel='{{ channel }}')
  SELECT 
     notif_id,
     user_id, 
     -- count the number of events of each type
     -- a custom UDF that returned a MAP {"sent":1, "clicked":3}
     COUNT_MAP(event_type) as event_map_count
  FROM notif_events
  WHERE event_hour = '{{ current_hour }}'
  AND ds = '{{ ds }}'
  AND channel = '{{ channel }}'
  GROUP BY notif_id, user_id

-- Then we merged the cumulative previous hours and the current hour with FULL OUTER JOIN

INSERT OVERWRITE TABLE notif_deduped_combined_hourly(ds='{{ ds }}', hour='{{ current_hour }}', channel='{{ channel }}')
WITH dedup_current_hour AS (
  SELECT 
     *
  FROM notif_deduped_hourly
  WHERE hour = '{{ current_hour }}'
  AND ds = '{{ ds }}'
  AND channel = '{{ channel }}'
),
previous_hour AS (
   SELECT 
     *
  FROM notif_deduped_combined_hourly
  WHERE hour = '{{ previous_hour }}'
  AND ds = '{{ ds }}'
  AND channel = '{{ channel }}'
)

SELECT  
   COALESCE(c.notif_id, p.notif_id) as notif_id,
   COALESCE(c.user_id, p.user_id) as user_id,
   -- udf that merges the keys of two maps
   -- {"sent": 1, "clicked": 3} + {"clicked":4} 
   -- = {"sent":1, "clicked": 7}
   COMBINE_MAPS(c.event_map_count, p.event_map_count)  as event_map_count
FROM dedup_current_hour c FULL OUTER JOIN previous_hour p 
ON c.notif_id = p.notif_id 
-- This condition triggers the SMB join because both tables are sorted and bucketed on user_id
AND c.user_id = p.user_id

This approach actually worked. It lowered latency and produced correct results.

But it had a huge flaw: compute usage exploded. It used 15 times more compute than the original 9.5-hour GROUP BY job.

How was that possible? Quite straightforward:

• Hour 1 processes 1 hour of data

• Hour 2 reads and merges 2 hours

• Hour 3 reads and merges 3 hours…

By hour 22, you find yourself reprocessing nearly the entire day’s data on every run.

It looked like this:

Once again, this solution was not sustainable, especially not for one of Facebook’s biggest datasets.

Approach 3: Tree-Style Merge DAG 🤩

Read more

At the time, we were leveraging a creative hybrid architecture internally dubbed as the “Psycho Pattern.” Think of i…

Read more

In this deep dive, I’ll walk you through the challenges I faced, the architectural mistakes I inherited and the solutions that made Airbnb earn millions.

This article covers the following topics:

• The subtle nuance in ‘availability’ definitions

• The original P&A pipeline design and its pain points

• Why massive backfills were so slow (and expensive)

• Introducing staging tables for rapid iteration

• What valuable lessons I learned

• The business impact of my work and some personal reflections

There’s a summary infographic of the entire data orchestration pipeline at the end of this article!

The True meaning of “Available”

Airbnb’s legacy definition of an “available” night was simply:

A host has not blocked this night, and it is not already reserved

On the surface, that sounds reasonable but, in reality, it diverged from what guests actually could book. Therefore, Airbnb aimed to establish a more reliable meaning of what’s “available”.

A trip can be booked that contains this night.

In fact, the two definitions only matched 96% of the time. But this subtle change captures the nuances between both definitions.

Key edge cases:

• Minimum stay requirements: Hosts or local regulations (e.g., 30-day minimum in New York) made many unblocked nights unbookable.

• Last minute/ time zone bugs: The system evaluated availability one second before midnight UTC. So Asia or Europe-based listings were sometimes asking, “Can I book yesterday?”

Original Pipeline Architecture & Pain Points

Here’s what I inherited in the P&A pipeline:

1. Fifteen raw datasets for blocked nights, calendar entries, regional regulations, minimum stays, etc.

2. A single Spark job that:

   1. Joins all fifteen tables in one massive operation

   2. Calls the Airbnb Java P&A library (via Scala Spark) to calculate availability

   3. Writes out the master P&A table for downstream models (i.e. Smart Pricing)

Why this was a problem

• Massive, repeated joins: Every time we tweaked a rule, the pipeline re-joined all 15 tables across eight years of historical data.

• Unpredictable runtimes: Backfilling could take 2½ weeks—despite only ~10 GB of daily data.

• High compute costs: Multiple backfill attempts (due to late requirements changes) meant wasted weeks and tens of thousands of dollars.

Why Backfills were glacially slow

I kept asking myself: “This isn’t even Big Data. It is not Netflix-scale… therefore, why so slow?” A few realizations:

• Monolithic joins: Spark spent most of its time shuffling data across executors for each join.

• Lack of decoupling: The join logic (inputs) and the calculation logic (P&A library) were tightly coupled, with every change rippled across the entire dataset.

• Zero incrementalism: No opportunity to reuse intermediate results; every run was a full historical sweep.

A solution: Staging Tables & Incremental Orchestration

The breakthrough came when I introduced a staging table and materialize all raw P&A inputs into one “inputs” dataset.

Read more