Harness Engineering: How to Give AI Agents Real Memory

Harness Engineering: Mechanical Memory Methods

Introduction: Agents That Forget

You start a conversation. The agent responds well.
Twenty minutes later, you're repeating yourself to the same agent.
An hour in, it seems to have forgotten what you originally wanted.
The next day, you start a new session and everything from yesterday is gone.

This is the fundamental problem with most AI agents today. The model is smart. But it can't remember.

More precisely: there is no memory architecture.

The context window acts like working memory, but it's capacity-limited and evaporates when the session ends. Parametric memory (model weights) retains only what the model saw during training and can't incorporate real-time experience. The gap between these two — that's the battlefield we're exploring today.

Starting in late 2025, a wave of papers began tackling this problem head-on: Memory in the Age of AI Agents: A Survey, MemoryOS, Agentic Memory, Episodic Memory is the Missing Piece...

They all converge on the same point:

"Memory is a core capability for agents. And we still don't know how to build it properly."

---

1. The Anatomy of Memory: What Needs to Be Remembered

Let's establish terminology first. Everyone uses "long-term memory" and "short-term memory," but the Memory in the Age of AI Agents survey argues this taxonomy is too coarse — insufficient to capture the diversity of modern agent systems.

The paper proposes a finer-grained classification along two axes: function (what for) and form (how stored).

1.1 Memory by Function (What For)

Factual Memory

The agent's declarative knowledge base. Things like "this user's name is John," or "the company's refund policy is 30 days." This memory enforces consistency. User profiles, environment state, and domain knowledge all fall here.

Experiential Memory

Procedural knowledge — what the agent knows how to do. Three subtypes:

• Case-based: Raw experience trajectories. "Last time I was in this situation, I did X."
• Strategy-based: Abstracted workflows. "For this class of problem, this approach works well."
• Skill-based: Executable code or tool APIs. Reusable capability units.

Working Memory

The live context of ongoing reasoning: the current task state, intermediate computation results, and the agent's immediate focus. The context window plays exactly this role.

1.2 Memory by Form (How Stored)

The same paper also classifies memory by storage medium:

Token memory is transparent and mutable. Parametric memory is persistent but costly to update. Latent memory is fast but fleeting. All three serve different roles, and all three are necessary.

The key insight here: the traditional long/short-term dichotomy only looks at storage duration. This classification also considers purpose and form. That makes it a far more actionable design framework.

---

2. MemoryOS: Managing Memory Like an Operating System

In a previous post, I used the metaphor of "Harness as operating system." The MemoryOS paper applies this metaphor directly to memory management.

Selected as an EMNLP 2025 Oral, the paper's core insight is straightforward:

"What if we applied OS memory management principles directly to AI agents?"

When RAM fills up, an OS swaps to disk. It evicts the least-recently-used pages first and keeps frequently accessed data in cache. MemoryOS applies this same principle to conversational memory.

2.1 Three-Tier Storage Architecture

┌─────────────────────────────────────┐
│     Working Memory (context window) │  ← Current conversation (fast, capacity-limited)
└──────────────┬──────────────────────┘
               │  FIFO (per dialogue chain)
               ▼
┌─────────────────────────────────────┐
│        Short-Term Memory (STM)      │  ← Recent conversation summaries (dozens)
└──────────────┬──────────────────────┘
               │  Segment page strategy
               ▼
┌─────────────────────────────────────┐
│       Mid-Term Memory (MTM)         │  ← Topic-organized conversations (hundreds)
└──────────────┬──────────────────────┘
               │  Importance-based compression
               ▼
┌─────────────────────────────────────┐
│      Long-Term Memory (LTM)         │  ← Personalization profiles, patterns (persistent)
└─────────────────────────────────────┘

STM → MTM transition: FIFO based on dialogue chains. When old conversations exceed STM capacity, they get pushed to MTM. Rather than simply deleting old entries, the system preserves and migrates semantic units (dialogue chains).

MTM → LTM transition: Segment page organization strategy. Patterns that repeat in MTM, or information above an importance threshold, gets promoted to LTM. This process extracts user preferences, recurring request patterns, and personalization signals.

2.2 Four Core Modules

MemoryOS consists of four modules:

1. Memory Storage: Manages the three-tier store
2. Memory Updating: Executes the inter-tier transition rules
3. Memory Retrieval: Retrieves relevant memories using semantic segmentation
4. Response Generation: Integrates retrieved memories as context for response generation

class MemoryOS:
    def __init__(self):
        self.working = WorkingMemory(capacity=128_000)   # tokens
        self.stm = ShortTermMemory(capacity=50)           # dialogue chains
        self.mtm = MidTermMemory(capacity=500)            # segments
        self.ltm = LongTermMemory()                       # unbounded

    def update(self, dialogue_turn):
        self.working.add(dialogue_turn)

        # Working overflow → STM transition
        if self.working.is_near_capacity():
            chain = self.working.extract_dialogue_chain()
            self.stm.add_fifo(chain)

        # STM overflow → MTM/LTM transition
        if self.stm.is_near_capacity():
            segment = self.stm.get_oldest_segment()
            if segment.importance > IMPORTANCE_THRESHOLD:
                self.ltm.promote(segment)   # important items go to LTM
            else:
                self.mtm.archive(segment)   # everything else goes to MTM

    def retrieve(self, query: str) -> list[Memory]:
        # Per-tier retrieval using semantic segmentation
        ltm_results = self.ltm.semantic_search(query, top_k=5)
        mtm_results = self.mtm.semantic_search(query, top_k=10)
        stm_results = self.stm.recent(n=5)
        return merge_by_relevance(ltm_results, mtm_results, stm_results)

2.3 Measured Performance

On the LoCoMo benchmark (long-context conversation understanding) with GPT-4o-mini:

Nearly 50% improvement — without changing the model, just the memory management architecture.

This is what Harness-level engineering buys you.

---

3. Agentic Memory: Memory as a Tool

Where MemoryOS defines the structure of memory, Agentic Memory (AgeMem) turns memory management into an agent action.

What's wrong with existing approaches?

Long-term and short-term memory are built as separate components, each managed by its own heuristics and controllers. This means the two systems operate without awareness of each other — the left hand doesn't know what the right hand is doing. Adaptability is low, and end-to-end optimization is impossible.

AgeMem's solution: expose memory operations as tools and let the agent decide.

# Memory tools AgeMem exposes to the agent
memory_tools = [
    Tool(
        name="store_to_ltm",
        description="Store important information in long-term memory. Use for things you'll need again later.",
        fn=lambda content: ltm.store(content)
    ),
    Tool(
        name="retrieve_from_ltm",
        description="Search long-term memory for relevant information.",
        fn=lambda query: ltm.search(query)
    ),
    Tool(
        name="update_stm",
        description="Update the current task state in short-term memory.",
        fn=lambda state: stm.update(state)
    ),
    Tool(
        name="summarize_and_compress",
        description="Compress the current context to essentials when it gets too long.",
        fn=lambda ctx: compress(ctx)
    ),
    Tool(
        name="forget",
        description="Remove information from memory that is no longer needed.",
        fn=lambda item_id: memory.delete(item_id)
    )
]

The agent decides for itself: "I'll probably need this later — put it in long-term memory." This is a learned policy, not hardcoded rules.

3.1 Training the Memory Policy with Reinforcement Learning

AgeMem's key innovation is training this memory policy with reinforcement learning (RL), using a three-stage progressive strategy:

1. Stage 1: Learn basic memory operations (when to store, when to retrieve)
2. Stage 2: Learn LTM-STM integration (how to use both memories together)
3. Stage 3: Strategic memory management on complex, long-horizon tasks

Reward design is the crux of the problem. Because memory operations are intermediate steps, rewards are sparse and discontinuous. To address this, AgeMem uses staged GRPO (Group Relative Policy Optimization).

3.2 Performance Comparison

One notable pattern: smaller models benefit more from unified memory management. The smaller the model, the weaker its intrinsic context management ability — so a well-designed memory policy fills that gap more dramatically.

4. Episodic Memory: The Missing Piece

After reviewing the taxonomy and systems above, something feels absent.

The paper Episodic Memory is the Missing Piece for Long-Term LLM Agents identifies exactly that gap.

Current systems focus on Semantic Memory and Procedural Memory — "what is known" and "how to do things." But what long-term agents actually need most is Episodic Memory.

What is episodic memory? It is memory organized around events — specific instances of experience bound to context: when, where, who, what, and why.

An example makes this immediately clear:

• "A server error occurred." → Semantic memory
• "At 2:00 AM on March 15, 2026, a cache invalidation bug immediately after deployment caused a server error, resolved by deploying a hotfix." → Episodic memory

4.1 Five Core Properties of Episodic Memory

The paper defines five essential attributes of episodic memory:

The last two properties are the crux. Existing memory systems record what happened but discard when, why, and under what circumstances. Episodic memory preserves that full context.

With episodic memory, an agent can:

• "What did I do last time I hit a similar situation?" → immediate recall
• Avoid repeating the same mistakes
• Deliver genuine personalization based on a user's history and preferences

4.2 Implementation: External Memory System

The architecture the paper proposes is a three-tier bridge structure:

┌──────────────────────┐
│   Parametric Memory  │  ← Model weights (general knowledge, patterns)
│   (Parametric)       │
└──────────┬───────────┘
           │  consolidation / generalization
           ▼
┌──────────────────────┐
│ External Episodic    │  ← Specific event store (fast encode/retrieve)
│ Memory (External     │
│ Episodic)            │
└──────────┬───────────┘
           │  offload / load
           ▼
┌──────────────────────┐
│  In-Context Memory   │  ← Current context window
│  (In-Context)        │
└──────────────────────┘

External episodic memory acts as the bridge between parametric memory and in-context memory. When the context overflows, episodes are offloaded to external memory and reloaded when needed. Recurring patterns are consolidated into parametric memory.

class EpisodicMemorySystem:
    def encode_episode(self, event: dict) -> str:
        """Encode a single event as an episode with its full context"""
        episode = {
            "timestamp": event["time"],
            "context": {
                "who": event.get("user"),
                "where": event.get("location"),
                "why": event.get("intent")
            },
            "content": event["description"],
            "outcome": event.get("result"),
            "tags": self.extract_tags(event)
        }
        return self.storage.save(episode)

    def recall(self, cue: str, context: dict = None) -> list[Episode]:
        """Context-based episode recall"""
        # Semantic similarity + context matching
        candidates = self.storage.semantic_search(cue, top_k=20)
        if context:
            # Prioritize episodes from similar situations
            candidates = self.context_filter(candidates, context)
        return candidates[:5]

    def consolidate(self, episodes: list[Episode]):
        """Consolidate recurring patterns into parametric knowledge"""
        patterns = self.find_patterns(episodes)
        for pattern in patterns:
            self.parametric_memory.integrate(pattern)

---

5. Practical Application in Agent Flows

Enough theory. How does this actually apply when building real agents?

5.1 Memory Tier Decision Tree

Storing everything the same way is inefficient. Every piece of incoming information should pass through these questions:

Where should this information be stored?
│
├─ Is it only needed right now?
│   └─ YES → Working Memory (context window)
│
├─ Is it needed for this session?
│   └─ YES → STM (conversation history summary)
│
├─ Is it needed repeatedly for this user/project?
│   └─ YES → LTM (user profile, project context)
│
├─ Might the specifics of this event be needed later?
│   └─ YES → Episodic Memory (concrete event + context record)
│
└─ Is it a general pattern or rule?
    └─ YES → Semantic Memory (AGENTS.md, knowledge base)

5.2 Harness Implementation: Memory Middleware

Applying the Middleware pattern introduced in the previous post to memory:

class MemoryMiddleware:
    def __init__(self):
        self.memory_os = MemoryOS()
        self.episodic = EpisodicMemorySystem()

    def before_agent(self, state):
        """Load relevant memories at agent start (Progressive Disclosure)"""
        user_profile = self.memory_os.ltm.get_user_profile(state.user_id)
        recent_context = self.memory_os.stm.get_recent(n=10)
        relevant_episodes = self.episodic.recall(
            cue=state.initial_query,
            context={"user": state.user_id}
        )

        # Inject only relevant memories — not everything
        state.inject_context(f"""
## User Profile
{user_profile.summary}

## Recent Context
{recent_context.summary}

## Relevant Past Experiences
{format_episodes(relevant_episodes)}
""")
        return state

    def before_model(self, state):
        """Prevent context overload: offload low-priority memories"""
        usage = state.context_used / state.context_limit
        if usage > 0.75:
            state.offload_low_priority_memories(self.memory_os)
        return state

    def after_agent(self, state):
        """Update memories after agent completes"""
        # Store significant events in episodic memory
        if state.has_significant_event():
            self.episodic.encode_episode({
                "time": now(),
                "user": state.user_id,
                "intent": state.original_intent,
                "description": state.summary,
                "result": state.outcome
            })

        # Update user profile
        self.memory_os.ltm.update_user_profile(
            user_id=state.user_id,
            interaction=state.to_interaction_record()
        )

        # Add conversation to STM (tier transitions handled automatically)
        self.memory_os.update(state.to_dialogue_record())
        return state

One thing worth noting in before_agent when injecting relevant memories: don't inject all of them. This is the same principle as Progressive Disclosure. Just as you inject only the tools you need, you inject only the memories relevant to the current query.

5.3 Three Anti-Patterns

These are the patterns that come up most often when memory is implemented poorly.

Anti-pattern 1: Stuffing everything into context

# ❌ Bad example
context = f"""
{entire_conversation_history}   # hundreds of turns
{entire_user_data}              # megabytes
{entire_knowledge_base}         # even more
"""

This is the opposite of Progressive Disclosure. Irrelevant information becomes noise and burns through the context window quickly. "More information means the agent knows more, right?" — in practice, it's the opposite.

Anti-pattern 2: Not remembering anything

# ❌ Bad example
def after_session(state):
    pass  # everything is lost

An agent with only short-term memory starts from scratch every time. No personalization. It repeats the same mistakes forever.

Anti-pattern 3: Putting everything in LTM

# ❌ Bad example
def after_every_turn(turn):
    ltm.store(turn)  # permanently store everything, including junk

LTM becomes a garbage dump. Retrieval quality degrades and storage costs explode. Without judgment about what is worth remembering, the entire memory system collapses.

---

6. Current Limitations and the Road Ahead

These papers are candid about what remains unsolved.

Problem 1: Tier transition orchestration

Most systems implement two tiers (STM/LTM) reasonably well, but rely on crude heuristics for transitions between them — simple rules like "how old is it" or "how frequently has it been accessed." Learning an optimal transition policy remains hard. AgeMem tackles this with RL, but production-stable deployments are still rare.

Problem 2: Memory reliability

What happens when an agent remembers something incorrectly, or misapplies a past memory to the current situation? Memory systems risk cementing an agent's biases. If a wrong pattern — "this user always does it this way" — gets written to LTM, that bias becomes permanent.

Problem 3: Multi-agent memory

How should memory be shared across multiple agents? Conflicting memories, synchronization issues, and the question of how much to trust any given agent's experience all remain open. In an orchestrator-worker pattern, when workers each carry different memories, how does the orchestrator reconcile them?

Problem 4: Multimodal memory

The transition from systems that remember only text to systems that remember images, audio, and video together. How to integrate and retrieve memories across different modalities is still in early stages.

A pattern emerges across all of these: memory problems are ultimately problems of judgment about what matters — what to store, what to discard, what to retrieve and when. Whether that judgment is delegated to hardcoded rules or learned policies is the central tension in memory research.

---

Closing: Real Agents Need Real Memory

Let's return to one basic fact.

An agent without memory is a stranger you meet for the first time, every time — no matter how capable it is. An agent with a good memory structure knows your history, remembers your preferences, and is a partner that learns from past mistakes.

Summarizing the direction these papers point toward:

• MemoryOS: Manage memory hierarchically, like an OS. STM → MTM → LTM — define each tier's role and its transition rules clearly.
• AgeMem: Turn memory operations into tools so the agent makes the decisions itself, then train the decision policy.
• Episodic Memory: Remember not just what but when, why, and in what context. Preserve the instance of the event.

All three share one thing in common. They don't treat short-term and long-term memory as separate systems. They are a single continuum. Harness is the operating system that manages that continuum.

One more point. Just as Progressive Disclosure applies to tools, it applies to memory too. Only relevant memories, at the moment they're needed, in the quantity required. Stuffing all memories into the context is the same as loading every tool at once.

If you're building a Harness, ask yourself this:

"Does my agent remember today, tomorrow?"

---

References

• Memory in the Age of AI Agents: A Survey (arXiv:2512.13564)
• Memory OS of AI Agent / MemoryOS (arXiv:2506.06326)
• Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management (arXiv:2601.01885)
• Position: Episodic Memory is the Missing Piece for Long-Term LLM Agents (arXiv:2502.06975)