The Anatomy of Agency: Tools, Planning, and Memory

Transcript

SPEAKER_1: Ok, so last time we landed on this idea that the agent is goal-oriented, not just reactive — that Perceive-Reason-Act loop. Now I want to get into what's actually inside the machine. Because an agent isn't just a model; it also combines reasoning, planning, memory, and ways to interact with tools or environments. SPEAKER_2: Right, and that's exactly the right pressure to apply. The LLM is the reasoning core — it decides what to do next — but it can't act on its own. Around it, you need at least two other things: tools that let it reach into the world, and memory that lets it carry context across steps. SPEAKER_1: So walk through those. Start with tools — what does that actually mean in practice? SPEAKER_2: Think of a tool as any external capability the agent can invoke. A calculator, a search engine, a calendar, a code interpreter, a database query. The agent doesn't run those itself — it issues a structured request, gets a result back, and folds that into its next reasoning step. That's the function-calling mechanism we touched on before. SPEAKER_1: So the model is essentially saying, 'I need this piece of information, go get it,' rather than trying to recall it from training. SPEAKER_2: Exactly. And that matters enormously for reliability. The ReAct framework makes this explicit. The agent interleaves reasoning traces with actual actions and then observations. Reason, act, observe, repeat. On knowledge-intensive tasks, that interaction with external sources — like a Wikipedia API — directly reduced hallucination compared to reasoning alone. SPEAKER_1: Mm-hmm. So grounding the reasoning in real retrieved data cuts the drift. SPEAKER_2: Significantly. And on interactive decision-making benchmarks — ALFWorld and WebShop — ReAct outperformed comparison methods by absolute success-rate gains of 34 and 10 percentage points respectively. That's not marginal. SPEAKER_1: Wait — but how does the model learn which tool to call and when? That seems like it could go wrong fast. SPEAKER_2: [inhale] That's where something like Toolformer is interesting. Toolformer is a training approach that teaches a language model to decide which API to call, when to call it, what arguments to provide, and how to use the result — using notably a handful of demonstrations per API rather than requiring a large human-annotated tool-use dataset. The model essentially teaches itself by checking whether a tool call improved its ability to predict what came next. SPEAKER_1: That's counterintuitive. Self-supervised tool learning from almost no examples. SPEAKER_2: Right. And here's the flip side that's worth flagging: more tools don't automatically mean better performance. Each tool call adds another point where the agent has to choose an API, provide arguments, interpret an observation, and decide the next step. The key idea is that capability and controllability can pull in opposite directions. SPEAKER_1: So now memory. Because the context window is finite — the agent can't just hold everything in working memory indefinitely. SPEAKER_2: Correct. Researchers distinguish several types: working memory, which is what's in the active context right now; episodic memory, specific past experiences; semantic memory, general knowledge; and procedural memory, how to do things. For anything beyond a short task, you need external storage. SPEAKER_1: For example — how does that external retrieval actually work? SPEAKER_2: Think of a system like MemGPT. It treats memory the way an operating system treats RAM and disk. Information that doesn't fit in the active context gets paged out to external storage and pulled back in when relevant. It was evaluated on documents larger than the model's context window and on multi-session conversations — exactly the cases where a flat context fails. SPEAKER_1: And there's a catch here, right? Because I've seen claims that agents perform well on memory benchmarks but then fall apart on real tasks. SPEAKER_2: [short pause] Yes — and this is one of the more sobering findings. A study called MemoryArena found that agents performing near saturation on standard long-context benchmarks performed poorly on interdependent multi-session agentic tasks. The benchmarks were measuring something easier than what real deployment actually requires. SPEAKER_1: So the benchmark gap is real. Now — planning. Because memory and tools are components, but something has to sequence them. SPEAKER_2: Planning is what converts a high-level goal into a sequence of executable steps — task decomposition. And critically, a plan isn't a fixed script. Agents can revise after receiving observations or hitting errors. Reflexion is a good example: it converts feedback into natural-language reflections stored in memory, then uses those on the next attempt — without retraining the model. It reported 91 percent pass-at-one accuracy on HumanEval, compared to 80 percent for the GPT-4 baseline. SPEAKER_1: So the takeaway for anyone designing one of these systems: you're not just picking a model. You're assembling a stack — reasoning engine, tools, memory, planner — and each layer has its own failure modes. SPEAKER_2: That's the right frame. And remember, the failure points compound: bad instructions, flawed decomposition, a tool returning garbage, a memory retrieval miss — any one of those can derail the whole loop. That's why implementation-level safeguards matter: constrained tool access, sandboxing, human approval checkpoints at high-stakes steps. The architecture gives you power; the guardrails are what make it usable. Next, we'll look at what happens when these systems operate at scale — and what that means for the people whose jobs sit inside those automated workflows.