So is trying to pivot into data engineering a lost cause at this point, with so many people actively trying to automate it out of existence? Hard to find any motivation to learn anything when it feels like the certain individuals are working hard to make sure nobody has a future.
But I wonder if this isn't just the natural evolution we've always seen. Research moved from books → online papers → Stack Overflow → now AI. Development went from text editors → notebooks → IDEs → now AI agents. Even navigation from paper maps → GPS → Google Maps predicting traffic before we hit it.
Each step removed friction. The goal stayed the same.
Maybe DE is going through the same shift. If we're not tuning knobs anymore, maybe the value just moves to better modeling, sharper system thinking, cost awareness, understanding what the business actually needs.
I'll be honest , I'm in the same boat. 10 years in, I sometimes wonder how relevant my core skills will be in a few years. But I still catch myself adding value in weird moments like connecting dots in meetings, making sense of messy conversations, asking the questions no one else thinks to ask.
Great article! I agree that most companies don’t even know what good data engineering is. They just care if the dashboard loads on time and if the data is correct
The answer is simple. Reposition yourself as data contract expert or data architect. If data engineering work is automated and databricks is doing it efficiently. Do check out "Data Engineer Weekly - data engineer after AI" Article.
Very interesting article, I have always see the other side: AI agents could build reports and dashboards, boosting the data analytics and replacing some data analysts. Now, abstracting the data contro, also Data Engineers are replaceable. Should we become experts of what?
This shift mirrors what's happening across the entire enterprise AI stack, abstraction is winning. The companies that thrive will be the ones that let practitioners focus on the business problem instead of infrastructure. It's the same trajectory we've seen in automation: the less technical the interface, the wider the adoption.
Pretty interesting article Zach! I really appreciate this.
What would you recommend for aspiring Data Engineers? I've been in this role for two years, and I still feel there's a great deal to learn. At the same time, the market appears to be moving toward higher levels of abstraction, which makes me question what skills I should prioritize to remain current and competitive.
We've been working on our new data engineering startup based on very similar predictions. We believe that a lot of the complexity around DE can be automated in the coming two years. Most companies setting up their data stacks won't have to deal with all the layers of complexity to get intelligent answers from their data - much of the stack will be simplified, unified, and automated. Only time will tell if we're right, but we're working hard to make it a reality. We'd love any and all feedback on our product (Nile: https://getnile.ai)
This article also makes me think about what is next for data engineers, I have always been pondering that question, Now it seems like no future? Or data engineers are moving to AI engineers?
This is so accurate!! Even with AWS slowly everything is moving from self managed to AWS managed. Their mantra is you write the code and let AWS scale your app.
So is trying to pivot into data engineering a lost cause at this point, with so many people actively trying to automate it out of existence? Hard to find any motivation to learn anything when it feels like the certain individuals are working hard to make sure nobody has a future.
100% agree with this
Seems like now you have to still learn the fundamentals and then learn the vendor-specific stuff and then lay on some AI engineering knowledge too.
Great article, Zach , really resonated.
But I wonder if this isn't just the natural evolution we've always seen. Research moved from books → online papers → Stack Overflow → now AI. Development went from text editors → notebooks → IDEs → now AI agents. Even navigation from paper maps → GPS → Google Maps predicting traffic before we hit it.
Each step removed friction. The goal stayed the same.
Maybe DE is going through the same shift. If we're not tuning knobs anymore, maybe the value just moves to better modeling, sharper system thinking, cost awareness, understanding what the business actually needs.
I'll be honest , I'm in the same boat. 10 years in, I sometimes wonder how relevant my core skills will be in a few years. But I still catch myself adding value in weird moments like connecting dots in meetings, making sense of messy conversations, asking the questions no one else thinks to ask.
Maybe the knobs change. The thinking doesn't.
Curious how you see it playing out.
Great article! I agree that most companies don’t even know what good data engineering is. They just care if the dashboard loads on time and if the data is correct
True. Unless they really care about efficiency and effectiveness. All they want is the real time dashboards most of the cases
The answer is simple. Reposition yourself as data contract expert or data architect. If data engineering work is automated and databricks is doing it efficiently. Do check out "Data Engineer Weekly - data engineer after AI" Article.
Great article. An eye opener. Thanks Zach.
Great Article.
Very interesting article, I have always see the other side: AI agents could build reports and dashboards, boosting the data analytics and replacing some data analysts. Now, abstracting the data contro, also Data Engineers are replaceable. Should we become experts of what?
Another "we aren't going to need blah anymore" article.
This shift mirrors what's happening across the entire enterprise AI stack, abstraction is winning. The companies that thrive will be the ones that let practitioners focus on the business problem instead of infrastructure. It's the same trajectory we've seen in automation: the less technical the interface, the wider the adoption.
Pretty interesting article Zach! I really appreciate this.
What would you recommend for aspiring Data Engineers? I've been in this role for two years, and I still feel there's a great deal to learn. At the same time, the market appears to be moving toward higher levels of abstraction, which makes me question what skills I should prioritize to remain current and competitive.
Let's at which point cloud/compute costs become too big and privacy / proprietary knowledge becomes a question mark.
Nice analysis as always Zach.
We've been working on our new data engineering startup based on very similar predictions. We believe that a lot of the complexity around DE can be automated in the coming two years. Most companies setting up their data stacks won't have to deal with all the layers of complexity to get intelligent answers from their data - much of the stack will be simplified, unified, and automated. Only time will tell if we're right, but we're working hard to make it a reality. We'd love any and all feedback on our product (Nile: https://getnile.ai)
Thanks again for a fantastic read!
Oh dog, this is gonna be fun.
This article also makes me think about what is next for data engineers, I have always been pondering that question, Now it seems like no future? Or data engineers are moving to AI engineers?
This is so accurate!! Even with AWS slowly everything is moving from self managed to AWS managed. Their mantra is you write the code and let AWS scale your app.