Building Geny-Harness: A Personal AI Assistant with Memory

Geny-Harness Engineering: Building Your Own Personal Assistant

Introduction: Enough Theory

In Part 1, I analyzed MemoryOS: its hierarchical memory structure, the STM → MTM → LTM transition policy, and the ~50% improvement on the LoCoMo benchmark.

In Part 2, I compared Mem0, Letta, Zep, MAGMA, and SimpleMem — what's fast, what's accurate, what supports temporal reasoning.

And honestly, after writing all of that, I was left with one thought:

"So when do I actually apply any of this to my own code?"

Reading papers and studying benchmarks is one thing. Building an agent that actually remembers things and maintains a coherent conversation is something else entirely. That's what this post is about.

What we built, why we built it, and what broke along the way.

---

1. Why Build from Scratch?

1.1 "Why Not Just Use LangChain?"

That question always comes up.

Honestly, I did use it at first. I tried LangGraph, tried LlamaIndex. But when you start building agents seriously, you keep running into the same wall:

The framework hides too much.

You can't tell why the agent called a particular tool.
You can't see how the context was assembled.
You can't track what's driving costs.
Changing one thing breaks another.

And critically — it still feels like you're stacking something on top of a framework. When something goes wrong, you have to dig into the framework internals, and at that point you have no idea what you're actually fixing.

So I decided to build from scratch. Starting from zero, with nothing but the Anthropic SDK.

1.2 Inspiration: Claude Code's Agent Loop

Using Claude Code gave me the first practical answer I'd seen to the question of how an agent should actually work — what to prepare before an LLM call, how to handle tool execution results, when to terminate the loop.

Watching that, I realized: an agent loop is a pipeline. Each step is distinct, each step has a clear role. What would that look like in code?

That was the starting point for geny-executor.

---

2. geny-executor: The Agent Pipeline Engine

2.1 A 16-Stage Pipeline

I didn't know how many stages I'd need going in. As I built it, I ended up with 16.

Phase A (once):  [1: Input]
Phase B (loop):  [2: Context] → [3: System] → [4: Guard] → [5: Cache]
                 → [6: API] → [7: Token] → [8: Think] → [9: Parse]
                 → [10: Tool] → [11: Agent] → [12: Evaluate] → [13: Loop]
Phase C (once):  [14: Emit] → [15: Memory] → [16: Yield]

Each stage does exactly one thing.

At first, some stages felt unnecessary — Token, Emit. But in practice, without the Token stage you have no idea where costs are coming from, and without Emit, changing the output format means touching core logic. The 16-stage design wasn't an upfront architectural decision; it was the result of necessity.

2.2 Dual Abstraction: Two Layers of Replaceability

This was the part I thought hardest about.

┌─ Level 1: Stage ───────────────────────────────────────┐
│  Entire stages can be swapped out in the pipeline      │
│                                                        │
│  ┌─ Level 2: Strategy ──────────────────────────────┐  │
│  │  Only the internal logic of a stage is replaced  │  │
│  │                                                  │  │
│  │  ContextStage can use any of these strategies:   │  │
│  │    → SimpleLoadStrategy        (default)         │  │
│  │    → ProgressiveDisclosureStrategy               │  │
│  │    → VectorSearchStrategy                        │  │
│  │    → MyCustomStrategy                            │  │
│  └──────────────────────────────────────────────────┘  │
└────────────────────────────────────────────────────────┘

At Level 1, you can replace the entire APIStage with your own custom provider.
At Level 2, you can swap only the internal logic of ContextStage.

# Write a custom context strategy
class MyVectorStrategy(Strategy):
    async def load(self, state):
        # Pull relevant context from my vector DB
        return await my_vector_db.search(state.query, top_k=5)

# Plug it into the pipeline — the other 15 stages are untouched
pipeline = (
    PipelineBuilder("my-agent", api_key="sk-ant-...")
    .with_context(strategy=MyVectorStrategy())  # only this changes
    .with_system(prompt="You are a helpful assistant.")
    .with_guard(cost_budget_usd=1.0)
    .build()
)

Why does this matter? When you're building real agents, you constantly hit the situation where you want to change just one thing but end up having to rewrite everything. Two layers of replaceability handle that cleanly.

2.3 Event System: Eliminating the Black Box

The most frustrating thing about using a framework is not knowing what's happening internally.

geny-executor emits events on every stage entry and exit.

@pipeline.on("stage.enter")
async def on_stage(event):
    print(f"→ {event.stage}")

@pipeline.on("stage.exit")
async def on_exit(event):
    print(f"← {event.stage} ({event.elapsed_ms:.1f}ms)")

@pipeline.on("*")  # catch all events
async def on_any(event):
    logger.debug(event)

You can see exactly which stage the agent is in, where costs are accumulating, and how many times each tool has been called. This event stream later becomes the channel connecting to the Geny UI.

2.4 What Broke During Development

First mistake: sharing too much state

In the initial design, every stage shared one giant PipelineState object. Convenient at first, but it caused problems. Stage A would mutate the state and Stage B would operate on stale data without knowing it. Debugging was a nightmare.

I eventually switched to defining explicit input/output types per stage: Stage[InputType, OutputType]. Type mismatches get caught at compile time.

Second mistake: deferring strategy selection to runtime

Making strategies selectable at runtime instead of build time made testing painful. There was no way to observe from the outside which strategy was actually running.

I added a configure() method and config_schema to make strategy configuration explicit. Now you can see exactly which strategy is running and with what settings.

class MyStrategy(Strategy):
    @property
    def config_schema(self) -> dict:
        return {
            "top_k": {"type": "int", "default": 5},
            "threshold": {"type": "float", "default": 0.7},
        }

    def configure(self, config: dict) -> None:
        self.top_k = config.get("top_k", 5)
        self.threshold = config.get("threshold", 0.7)

---

3. geny: The Multi-Agent Management System

If geny-executor is the engine, geny is the car it powers.

"Autonomous multi-agent system — manage multiple agent sessions, orchestrate autonomous tasks, and visualize it all in a 3D city playground."

3.1 Env: The Agent's Environment Unit

The core concept in geny is the Env (environment) — a container that holds the complete configuration for one agent (or team of agents).

One Env = {
  Which model to use           (model)
  Which tools are available    (tools)
  Permission level             (permissions)
  Memory management config     (memory config)
  Hook definitions             (hooks)
  Skill definitions            (skills)
  MCP server connections       (mcp)
}

All of this is managed through a GUI. You can change an agent's behavior without opening a code editor. Settings like "this agent can read files but not write them" take a few clicks.

3.2 The Skill System

A Skill is a predefined capability unit you give to an agent, defined as a Markdown file (SKILL.md).

# Code Reviewer
category: engineering
effort: medium

## Description
Review the given code, identify areas for improvement and bugs.

## Examples
- Input: "Review this Python function"
- Expected behavior: Read the function → List issues → Suggest improvements

In the GUI, you can create Skills with a live Markdown preview and immediately test them with a dry run — no code required.

It's a similar concept to Claude Code's / commands, but in geny, Skills are bound to the agent environment itself. Any agent running in that Env always has access to those Skills.

3.3 Permission System

Granting permissions to an agent is a double-edged sword: too many and it's dangerous, too few and it's useless.

geny manages permissions along two axes:

runner_mode:    how the agent behaves during execution
executor_mode:  what is permitted at the geny-executor level

There's also a narrowing feature at the Env level. Env-level settings can be more restrictive than global settings. In production agents, this separation actually matters.

Phase 9.9 was a major cleanup of this area. runner_mode became meaningful in practice, and executor_mode started propagating correctly to the pipeline level. Before that, there were fields that existed in name only and did nothing. (Embarrassing to admit, but that's what happens when you're moving fast.)

3.4 Version History: Building in Phase Increments

Looking at the commit messages, you can see development proceeded in discrete phases.

Phase 9.3: Skill form modal (beginner-friendly)
Phase 9.4: Hook form modal
Phase 9.5: Permission form modal
Phase 9.6: Skill Markdown live preview
Phase 9.7: Skill dry-run (test directly from form)
Phase 9.8: Inline list ↔ form view (modal → inline transition)
Phase 9.9: runner_mode / executor_mode cleanup

Each phase is a small, self-contained deliverable. Keep the big picture in mind, build in small units. geny-executor followed the same approach — developed in Cycle A, B, C, D increments.

---

4. Opsidian: The Memory Viewer We Built Ourselves

4.1 Why We Built It Instead of Using an Existing App

Most developers know Obsidian — a markdown-based knowledge management tool with backlinks and a graph view. It's well-made.

But there's a reason Obsidian doesn't work well for our use case.

An agent's memory files are fundamentally different from the "notes" Obsidian is designed to handle.

Agent memory is:

• Not something a human edits directly — it's a file the agent writes to in real time
• Managed per session (.geny/sessions/<sid>/memory/)
• Written by geny-executor's Stage 15 (Memory) and read by Stage 2 (Context) in a cycle
• Something you need to observe in real time — which memories the agent has currently activated
• In need of backlinks and a graph view, but from the perspective of the agent's memory network, not a human's knowledge base

So we built Opsidian. Inspired by Obsidian, but purpose-built for agent memory management.

The name comes from that inspiration: Op(erations) + (Ob)sidian. Similar enough to Obsidian to feel familiar, distinct enough to make it clearly ours.

4.2 What Opsidian Does

It's organized around three views.

┌──────────────────────────────────────────────────────┐
│                     Opsidian                         │
│                                                      │
│  [Editor]          [Graph]           [Search]        │
│                                                      │
│  Markdown file     Memory network    Full-text        │
│  read/edit         visualization     memory search   │
└──────────────────────────────────────────────────────┘

Editor view: Renders the agent-authored memory files as markdown. Composed of file tabs, a sidebar (files / tags / backlinks), and a right panel.

Graph view: A visual graph of how memories connect to each other. Nodes are memory files; edges are backlink and tag connections. At a glance, you can see what kind of knowledge structure the agent is building.

Search view: Full-text search across all agent memory. Gives you an immediate answer to "What does this agent remember about X?"

4.3 Integration with geny-executor

Opsidian isn't a standalone viewer. It syncs in real time with geny-executor's Memory Stage.

Agent execution flow:

Stage 2 (Context)  ──read───►  .geny/sessions/<sid>/memory/
Stage 15 (Memory)  ──write──►  .geny/sessions/<sid>/memory/
                                        │
                              Opsidian Sync Hook
                                        │
                                        ▼
                             Geny UI (Opsidian view)

When the agent remembers something during a session, Stage 15 writes it to a file. Opsidian reflects that change in real time.

The reverse also holds: if you edit a memory file in Opsidian, Stage 2 will read that edited content in the next session. You can directly modify what the agent remembers.

This matters for a concrete reason. When an agent has remembered something incorrectly, you can fix it without touching any code. Part 1 described "selective forgetting" as a hard problem — direct editing through Opsidian is the practical workaround.

4.4 Opsidian as a Tool Family

Inside geny, Opsidian is also a tool family — a set of tools prefixed with opsidian_.

// genyToolFamily.ts
const SUB_FAMILY_PREFIXES = [
  { id: 'memory',   prefix: 'memory_'   },
  { id: 'opsidian', prefix: 'opsidian_' },  // ← here
  { id: 'session',  prefix: 'session_'  },
  // ...
];

You can give the agent tools like opsidian_read and opsidian_browse, enabling it to explore its own memory store directly.

Where Letta used tools like core_memory_append to let the LLM manage its memory directly, Opsidian takes that a step further by giving the agent a richer navigation experience: file-level browsing, tag-based navigation, and backlink traversal.

---

5. The Full Picture: How the Three Pieces Fit Together

┌──────────────────────────────────────────────────────────┐
│                       Geny (UI)                          │
│                                                          │
│  ┌──────────────┐  ┌────────────────┐  ┌─────────────┐  │
│  │  Env Mgmt    │  │ Session Mgmt   │  │  Opsidian   │  │
│  │(config/perms)│  │(multi-agent)   │  │(memory view)│  │
│  └──────┬───────┘  └───────┬────────┘  └──────┬──────┘  │
└─────────┼──────────────────┼──────────────────┼─────────┘
          │                  │                  │
          ▼                  ▼                  ▼
┌──────────────────────────────────────────────────────────┐
│                  geny-executor (Pipeline)                 │
│                                                          │
│  [1:Input]→[2:Context]→[3:System]→[4:Guard]→[5:Cache]   │
│  →[6:API]→[7:Token]→[8:Think]→[9:Parse]→[10:Tool]       │
│  →[11:Agent]→[12:Evaluate]→[13:Loop]                     │
│  →[14:Emit]→[15:Memory]→[16:Yield]                       │
│                                                          │
│  Stage 2  ◄──read────  .geny/sessions/<sid>/memory/      │
│  Stage 15 ──write───►  .geny/sessions/<sid>/memory/      │
└──────────────────────────────────────────────────────────┘

Setting an Env in Geny UI → changes the geny-executor pipeline configuration.
The agent runs → Stage 15 writes memory to files.
Opsidian → displays those files in real time.
Next session → Stage 2 reads those files and injects them into context.

This cycle is the concrete reality of "an agent that has memory."

---

6. In Practice: What Works and What Doesn't Yet

What's Working Well

The 16-stage pipeline is genuinely useful.

"Why did costs spike?" → Stage 7 (Token) logs answer that precisely. "Why won't this agent exit the loop?" → Stage 13 (Loop) logs show exactly why. Because each stage is clearly separated, you can pinpoint where a problem originated immediately.

The biggest surprise was Stage 4 (Guard). Handling "block execution when cost budget is exceeded" at the pipeline level eliminated situations where an agent stuck in an infinite loop burned through API quota. That class of mistake just stopped happening.

Dual Abstraction turned out to be genuinely necessary.

Initially it felt like over-engineering, so we only built Level 1. Then "I just want to change the context loading strategy" kept coming up, roughly once a month. After adding Level 2, those situations resolved cleanly.

Opsidian's graph view is more useful than expected.

Seeing how the agent has structured its memory in graph form makes visible things like "this agent is connecting these concepts in this way." It's especially useful in long-running sessions for understanding what kind of knowledge structure the agent is building. We didn't expect to use this view much — it's turned out to be the one we open most often during agent debugging.

What Still Falls Short

The memory summarization and compression pipeline is unfinished.

If Stage 15 (Memory) is the writer, the downstream processing — "automatically summarize and compress memory as it accumulates" — needs significantly more work to function properly. Accumulation works fine; the cleanup is still crude. The goal is to implement SimpleMem's Semantic Density Gating concept at the stage level.

The Opsidian edit → agent reflection cycle is still one-directional.

The original goal was for edits a human makes to agent memory to be fully reflected in the agent's next session. Reading works. Post-edit synchronization isn't clean yet. The part where the agent recognizes "I was remembering X incorrectly" is still underdeveloped.

Multi-agent memory sharing is still too simplistic.

When multiple agents share a session, conflict handling for memory updates is rough. If agent A remembers "Kim Cheolsu lives in Seoul" and agent B remembers "Kim Cheolsu moved to Busan," what happens when both update shared memory? The upsert vs append problem from Part 1 becomes significantly more complex when extended to multi-agent scenarios.

---

Closing: On Building a Personal Assistant

The title of this series is "Let's Build Our Own Personal Assistant." Having actually built one, what I've learned is that "personal assistant" is a more complex concept than it first appears.

A good assistant needs to:

• Remember tomorrow what you talked about today (Memory)
• Know what you dislike (Personalization)
• Only do what you've authorized (Permission)
• Let you see what it's doing (Observability)
• Allow incorrect memories to be corrected (Editability)

geny-executor structures these requirements as a pipeline.
geny wraps that pipeline in a UI people can actually interact with.
Opsidian makes the agent's memory visible to humans.

There's still a long way to go. Memory compression needs to be built. Bidirectional edit synchronization needs to work. Multi-agent memory conflicts need to be solved.

But at some point you watch an agent remember yesterday's conversation, behave exactly within the constraints you configured, and see its memory graph slowly growing — and you get the feeling that you're actually building something. That feeling is what keeps you going.

Part 1 posed a question:

"Does my agent remember today, tomorrow?"

We're in the process of building the answer.

---

Links

• geny-executor GitHub
• geny GitHub
• Previous post: Harness Engineering: Mechanical Memory Methods
• Previous post: Harness Engineering: Mechanical Memory Methods - 2