Holiday Hacks, cont. 🎄

I have been tinkering with a few "agentic" patterns, just to look inside a bit after reading about a few on the Manus blog.

I built a little agent lab to play around with a few techniques.

Tool Masking vs Tool Removal

You might remove irrelevant tools from their schema to encourage the model to pick a tool you think is best. (They can get overlaoded with context and options.) But dynamic tool removal breaks your KV cache every time the tool set changes. This masking technique is cute: they don't change the tools, but they do use prefill to say something like, "I used tool file_xyz..." which starts the LLM down the narrower path. Doesn't blow the KV.

Prefill

Put words in the LLM's mouth. You can, for example, strongly encourage a JSON-only response by prefilling with {", or you can skip the hedging with Absolutely!, or even extract structured data by starting the response for it. Compare to elaborate prompts.

File as Context

Context windows are big now, but stuffing everything in still isn't free. Try offloading content to files and retrieving on-demand vs. front-loading everything. The retrieval overhead often beats the attention cost.

Sequential vs Subagents

Pretty straightforward for anyone in distsys, the idea is that spawning (potentially parallel) subagents without the global context can sometimes work more effectively with constrained context rather than global context that gets polluted from all of the separate tasks.