A well-architected secretary is 76 agents in a trenchcoat

Beyond “take meeting notes I’ll never read”

Hundreds of startups (and maybe an internal team at your company) are trying, right now, to build and sell you on an army of AI scribes. That’s fine, and it’ll be useful. But what you’ll really need are competent, trusted, proactive secretaries. That’s much harder. But doable!

This article is brought to you by:

Eon and Google Cloud are hosting BigQuery Day, a free one-day virtual event for teams running BigQuery in production. Sessions from Google’s VP of BigQuery Engineering, Lead Data Engineer at L.L.Bean, Google Cloud Developer Advocates, and more.

AI has evolved into new UX patterns. This will continue.

So, we have robots (LLMs, agents, whatever you want to call them) now. What can we use them for?

Well, robots have changed their shape over the last few years. We went through these affordances:

• GPT playground (autocomplete)

• ChatGPT (chatbot)

• custom (RAG) setups (understand your files specifically)

• Claude Code (agents, dispatchers)

• “Agent” workflows

• OpenClaw (autonomy + skills + texting interface)

Each was useful! Businesses are being built on them.

So -- what next? And what can we learn?

Well, we’re about to see a new layer – the army of scribes. You’ll be tempted to use them directly. You’ll then be tempted to hook them up to your robot assistant. Fine. But the real action will be in giving that assistant enough context, power, and team-aware-capabilities so that it can actually be a useful clerk or secretary.

The gold rush of AI scribes is coming

Surely you also see the coming gold rush. Seemingly everyone is working on versions of this vision (to greater and lesser degrees of insight). Via “owning your docs”, via “owning your communications”. Maybe pitched as “specially for developers” or a sales-calls-to-product-insight loop. Maybe as a panicked pivot from an entirely different product. Maybe over a specific dataset. They’re all the same thing – scribes, taking notes, collating them, and giving them back to you.

This will happen.¹

And it’ll be fine! It’ll be useful. But how often do you currently read meeting notes? How will it feel, really, to have action items more efficiently thrown your way? You’re still tied to the tyranny of staring at a screen and typing on a keyboard.

What’s more interesting is what will happen next.

We can do better than delegating notes to robots

Think about depictions of principals doing magnificent work across media.

A soccer coach rallying his team. Space captains conducting negotiation with aliens. Ad executives making deals in the 1960s. A fantasy king leading his men into battle. Business executives jockeying for power.

How will it feel, really, to have action items more efficiently thrown your way? You’re still tied to the tyranny of staring at screen and typing on a keyboard

What do these people have in common? They have fun, creative jobs. They have meetings. They have big thoughts. They made decisions. They do not slave over a typewriter or word document. They do not take meticulous notes. They have staff for that. (Notably, being the staff is generally a lot less pleasant.²)

In the near future – that could be you. But if you want that life, you need staff to do the less fun work. Beyond just scribes taking notes. You need clerks. You need secretaries. And you need them to be as reliable and competent as those fantasy secretaries. But since they’re robots, you don’t actually need to worry about whether their jobs are rewarding.

But what do real-life secretaries actually do?

Well, in large part, you’re currently a secretary to yourself. Modern real-world workplaces spend a lot less money on logistical support than fantasy TV offices do. You do a lot of typing. You send memos. You route information between people on your team. You update documentation. You schedule and bargain and procure. You might communicate with vendors and outsiders or do other logistics. You type. You do it all in the background as part of supporting your “real” work. It may feel tactically rote, but it relies on a wealth of knowledge and trust and context that you can’t hand off cleanly – at least, until now.

What do secretaries actually do? Well, in large part, you’re currently a secretary to yourself.

Now what does this look like in our near-future?

Mostly-reliable scribes are a commodity

Your company will set up scribes. You will have an invisible system of robots doing certain menial tasks. Your conversations will be filed, your notes will be taken. The notes will be parsed into decisions. Jira tickets will be filed and closed. Executives will get daily summaries of what their teams are up to. Action items will be filed after each meeting.

This might even be built, in a half-baked way, in your team right now.

That’s the basic, obvious vision. Robots swarm around writing notes, reading notes, answering your questions about your notes. An army of scribes writing into a giant filing cabinet.

Importantly – you’ll need these scribes to be so good that you don’t spend a lot of time fixing their filing mistakes. Or, more interestingly, you’ll need some other entity to make sure they got it right.

(”So far, so good! But weren’t we talking about secretaries?” -- yes, shh. We’re going deeper).

SIDEBAR: What about OpenClaw?

There’s an obvious reaction to all this: “hey, isn’t this just openclaw? OpenClaw/Hermes/IronClaw is already my personal assistant”. Well, kinda. Not really. The first prototypes of AI secretaries we see will likely be built on a foundation of *claws, but we need AI systems that have these core design principles:

• Built for teams (of humans and similar robots) from the ground up.

• Coordinating across systems of code, people, and other AIs (invisibly, constantly)

• Will integrate with company systems (and be trusted by IT not to leak or delete everything)

• Likely multi-server and multi-model.

• Opinionated on what happens invisibly/continuously and need user approval or visibility.

And more (see below).

This will require a new and different type of architecture than looping agents on a server and connecting it to Telegram.

A scribe writes things down. A secretary proactively preps.

But wait. A beautifully organized filing system doesn’t work well unless you can use it; at the same time you don’t want to access it yourself. Otherwise you’re back in the drudgery of paperwork.

And you need to be able to give orders -- often implicitly phrased as “if X, then Y”. (Current tooling is pretty bad at this right now)

You need someone to proactively and reactively hand you the perfect manila folder each time you need it.

And, suddenly, we’re not talking about a scribe at all. We’re discussing something more interesting. The layer on top of that. The secretary. A clerk that talks to the invisible army of scribes on your behalf, sure, but so much more than that. It must continuously manage attention, context, information, and execution across time.

Let’s go further. You’re at work.

When you go to a meeting with someone else on your team, you want your secretary to know ahead of time, talk to their secretary, compare notes, write briefing docs for the meeting, hand them to you on a just-in-time basis (or maybe send you the pre-read in the morning and the prep doc 10 minutes ahead of the meeting) -- AND understand any decisions you make in the meeting and implement them.

When you work on an architecture plan, you want your clerk to interrupt you, and tell you that team ABC tried something similar in the company 2 years ago.

The clerk has:

• Simple retrieval: Pulled all the notes

• Social graph querying with sophisticated privacy / sensitivity features: Talked the clerk of the Albert, the lead architect of ABC, and got from them the sensitive learnings that weren’t in the official document.

  • (“Joe is great at code but horrible at explaining things well – we should not have made him a project lead on this”, “direction X was really promising but couldn’t pursue it due to office politics”, “I’ve subsequently learned about Y pattern that we should have used instead”)

  • Followed up with the clerks of Joe, Sally, and Martha who were also on that project.

• Negotiation with other clerks: Booked coffee with Albert, who will be in town next week (but does not show his free time on his calendar)

• Proactive pushing of info: Prepped Albert’s clerk with all your context

• Context aware synthesis: Written you a report on what, from project ABC, can you actually learn

That’s a vision. Maybe a compelling vision. It is to me.

And I think we can build it.

Can you trust your robot secretary?

For all this to work, you need reliability, trust, and even discretion.

• You need to know that stuff won’t get dropped. Dropped is terrible. Forgotten about? Also terrible, but as long as it is filed somewhere you can recover.

• You need to be able to fire your secretary. In this case, that means that if you stop using your particular AI system, porting to a new system needs to be incredibly easy. The secretary’s notes need to play well with the filing/scribe system the company uses. It needs to be flexible enough to take a filing system as it exists and invisibly clean it up to its standard over time.

• Some robots need to know about the existence of, say, an email or meeting (”the boss is meeting a lawyer to talk about his divorce”) but NEVER leak the content of it. And we need to trust that they never double-book, and might even do polite fictions if necessary.

SIDEBAR: Your secretary is many agents in a trenchcoat

Note that I’m not using the word “agent” a lot here. That’s because your clerk, secretary, even your scribe – they’re not going to be one agent. Why should they?

An agent is analogous to a process on a machine. Processes can fork, or call other processes. They can die without implicating each other. A traditional “program” on your laptop has many processes running at once. But a SASS app has so many different computers involved that talking about processes feels like you’re missing the point.

So too with anthropomorphized robot helpers. Maybe on one server there’s a dispatcher agent that calls research agents, runs async scripts, negotiates with the company data layer. Maybe it coordinates with a dispatcher on another server that message and info passes with other assistants. A third dispatcher on a third server might be the layer that talks to you (and also runs subagents for odd jobs rather than message passing to the other two).

I’m not saying this is a good architecture, by the way. But it’s entirely feasible. And, to my point: who is the “agent” you’re talking to in this scenario? Even “Multi-agent system” is too low-level. We don’t quite have the right words the emerging thing we’re describing. Agent OS? AI persona?

I have some thoughts on how to build it. I’ve even started. The keys, so far, seem to be:

• Start from an assumption that your secretary is working on a team

• Set up the right dispersion of deterministic and LLM control flow

• Use a really rich schema for how secretaries communicate in structured ways.

Some primitives that I’ve found useful:

• Dispatcher/subagent structure

• Code. Gatekeepers, hooks, a message queue, and injection of prompts to deterministically both nudge and enforce behavior.

• Rich schema for discretion, communication, decisions, and memories.

• Secure enclaves of local LLMs that monitor the frontier model “kernel” and gatekeep its communication for the correct discretion sensitivity.

• Federated CRMs and knowledge graphs within a team

• An “If this, then that” data structure that points to the right memory when agents match a certain behavior (solving the “remember to look up a memory telling you to remember something problem).

• Logs and health check monitors everywhere

But the crux, maybe, is this: the world of OpenClaw clones is running into two bottlenecks: security/privacy/reliability, and the need for an actual killer app. Those are related. Imagine a world where everyone has some sort of robot butler -- but their *claws talk to each other well. Invisibly, carefully.

SIDEBAR: Clerks and Secretaries and CoS

Why “secretary”? Why not just “clerk?” Well, the term clerk doesn’t quite bring up the level of familiarity and trust that I’m going for. Does a clerk know about your sensitive meeting even as they don’t put it on the calendar? Clerks are closer to paper pushers than autonomous agents.

Another word you could use was “chief of staff”. But that has its own limitations – mostly the term means very different things. The CoS in a government agency basically runs it. A CoS in tech has extremely different responsibilities than one in nonprofits. And so on.

So, when using “secretary”, I’m trying to evoke a level of trust, competence, autonomy and brilliance that we see in a Joan or Peggy. Without anyone involved being as poor a boss or bad a person as a Don Draper.

Against communication explosion

That’s the future I see. We shouldn’t just throw our robots into Slack³ and call it a day. You don’t teach students by throwing them all into a gym and having them run around yelling at each other. You don’t run a company by having CEOs spying on how each worker is sending notes to each other.

We will have our AIs morph between copilot and HUD. They’ll share information in a smart way. They’ll know they’re on a team and work accordingly.

So don’t obsess over the coming scribes. That’s tomorrow’s tech. Build assuming that it exists. Focus on using it in smart ways rather than staying stuck as your own secretary.

Make sure to follow Sahar’s blog to understand growth and what comes next, both personally and for your business! Sahar is currently pursing full-time product and engineering leadership roles in New York City.

DataExpert is now open for more sponsorship. Like this post? Want people to know about how great your product is? Please email us at mitali@dataexpert.io to start a conversation.

1. The interesting part of it is how. Which framing will resonate with your company? Which strategy to integrate to your company will actually succeed? (I think the jargon for this is “change management”. )

2. Some of these very shows -- Mad Men and Ted Lasso, for example -- put a lot of narrative weight into underscoring the importance and brilliance of these support staff and what happens when the relationship gets sour.

3. For one thing -- slacks are terrible at distinguishing between urgent and important. Don’t use Slack!