Turning AI Agents into Reliable Assistive Tech: Lessons from 1 Billion Tokens

From Hype to Habit: My First Two Months with an AI Agent

Two months ago, I decided to build a "digital twin" that would handle 80% of my work. I chose Codex, OpenAI's coding agent, as my primary tool. By August 11, I had already burned through 300 million tokens. At that pace, I'll hit a billion by month's end.

What did I actually use it for? Everything from data updates and weekly reports to user segmentation, financial reconciliation, revenue forecasts, project handoffs, and even automating backend systems. It even turned a video into a work manual. So yes, Codex has infiltrated almost every corner of my job.

But if you ask me whether I've built that elusive digital twin—the one that does 80% of the work without me—the answer is still no. My perspective, though, has shifted. The real value isn't in a magic button; it's in building a system that works.

SOPs: The Unsexy Secret to Reliable Automation

The task I run most often is embarrassingly ordinary: refreshing a data dashboard. It starts with downloading raw files from two internal systems, merging them by ID (removing duplicates), writing 400,000 rows into a master sheet, then applying a convoluted logic to calculate metrics, update a summary table, and generate an HTML dashboard.

At first, I had to write long, detailed prompts every time. But after enough repetitions, I decided to codify the whole thing. I made a slide deck explaining exactly what to query, which date ranges to use, how to handle duplicate orders, and what to verify before writing. Codex read it, asked a few clarifying questions, and we ran a test.

It was a mess. Excel files with data were flagged as empty because of odd size markers. 20-digit IDs got mangled into scientific notation. New product names didn't fit existing categories. And the dashboard showed stale data because of a caching issue. But we fixed each problem, and every fix went back into the Skill. Now I just say, "Here's my work for the week; update the data through the 15th and generate the weekly report." Codex figures out the rest.

The lesson? For standardized tasks, the quality of your agent depends less on prompt engineering and more on having a proven, battle-tested SOP. The counterintuitive truth is that the more you let it improvise, the more likely it is to screw up. Before you automate anything, you need to answer five questions: What's the input? What's the processing rule? Where does the output go? How do you verify correctness? What do you do when something goes wrong? Only then should you build a Skill or schedule a recurring task.

Browser Automation: A Word of Caution

If your task involves browser actions—like logging into a system to pull data or filling out a form—be prepared for pain. Codex can use its built-in browser, a Chrome extension, or full computer control. The extension is the sweet spot; the built-in one can't upload local files, and computer control is slow and gets stuck on permissions. Watching Codex move a mouse pixel by pixel is like watching an 80-year-old try to win a fight. It's not fast. But it works, as long as you've got a clear SOP.

Context Is King: Why Your Agent Needs a Memory

Beyond fixed workflows, I used Codex for a long-running project where every analysis built on the last. Revenue dropped, so I analyzed why. Then I forecasted H2 revenue. That revealed a product issue, so I planned a special campaign. That required user segmentation and historical data. Each task looked independent, but they were all links in a chain.

If I started a fresh chat and said, "Make a plan," I'd get a generic, useless answer. What made the difference was saving every analysis into a project folder. The revenue decline report became the basis for a strategy. The forecast became the target for the campaign. The user segmentation informed the outreach design. Over time, I realized that what makes an agent smart is the validated history it can pull up—what researchers call "context."

For any long-term project, your knowledge base should include four things: data definitions (how metrics are calculated), decision rationale (why we chose this path), validated conclusions (what has been proven), and version history (what changed and why). Don't just record the conclusion; record the "why." Tell it "we raised the price to $X," but also why, for whom, and what business goal it serves. That way, next time, it can make a judgment call that fits reality.

AGENTS.md: A Constitution, Not a Diary

When I started, the first thing I did was create an AGENTS.md file. It tells Codex who I am, what I do, my preferences, and where to find key resources. It's powerful. But I quickly learned not to stuff it with everything. A product's current sales figure or this quarter's goals are temporary—they'll be outdated in weeks. If you put them in AGENTS.md, Codex will treat them as permanent rules, and you'll get confidently wrong answers.

I now think of context in four layers: AGENTS.md for stable identity and principles, Skills for task execution, project knowledge base for evolving history, and the current conversation for immediate context. A simple test: if a statement won't be true in three months, it doesn't belong in AGENTS.md. Treat it like a company's constitution—review it periodically, but amend it sparingly.

The First Draft Is Just a Whiteboard

Codex produces beautiful Word docs, Excel files, and PPTs. They're structured, comprehensive, and full of charts. That's also why they often feel "AI-ish." The problem isn't that they're bad; it's that they're too complete. They hide the crucial questions: What's the core judgment? Does the data support this? Is this just industry common sense? What's actually different about us? What decision do we need from leadership?

For one presentation, the first version had everything: industry analysis, product ops, campaign design, monetization. But it took eight revisions, 35 messages, and 6,377 words of back-and-forth to get it right. Another went through five iterations, with 20 rounds and nearly 4,000 words. I've learned to treat Codex's first draft as a whiteboard, not an answer. The real value comes from arguing with it: Is this number solid? Is this just a nice-to-have? Which assumption will a client challenge? What can we cut? AI is great at generating, but the final judgment—the responsibility—is yours.

Redefining the Agent: A System, Not a Magic Wand

I used to think of an agent as a combination of brain, tools, skills, knowledge, and memory. Now I add one more element: the human. The brain handles planning, tools handle files and systems, skills define standard processes, knowledge provides context, and the human sets goals, makes calls, and takes responsibility.

I once compared the ideal agent to a pair of disappearing chopsticks—you just say "I want meat," and it appears. But the process doesn't disappear; it just moves earlier. The first time I built that data workflow, it took longer than doing it manually. But once it worked, each run was effortless.

So, is an agent worth it? Absolutely—if you use it right. For tasks you've already figured out, it can run them endlessly without complaining. For tasks you haven't figured out, it's invaluable at turning vague ideas into a concrete draft you can react to. My advice for anyone starting: pick one repetitive, well-defined task. Run it with the agent, document the SOP, fix the edge cases, and only then automate. Build a proper file structure for long-term projects. Keep AGENTS.md lean. And always treat the first draft as just the beginning. The real work—and the real value—is in the back-and-forth that turns a generic output into something that's truly yours.

Two months ago, I thought the key was asking better questions. Now I know the questions are just the start. The real edge comes from turning a good conversation into a reusable process, building a living knowledge base, and using your own judgment to shape the AI's output. That's how you turn AI from a parlor trick into assistive technology that actually helps you work.