AI Agent Memory
Persistent memory across sessions => users don’t repeatedly explain project context, preferences, and past decisions.
Types
Working Memory - Short-term, in-context scratchpad to hold the current task state, parameters, intermediate thoughts, and next steps while the agent reasons (to prevent context bloating)
Long-Term Memory - Persistent storage across sessions to retain knowledge and experience so the agent can recall information later.
Episodic Memory - past events, interactions, outcomes, user preferences (what/when happened).
Semantic Memory - general facts and domain knowledge independent of events.
Procedural Memory - learned routines, skills, and decision workflows (how to do things).
• Memory Operations
Working m. - Save task state (offload progress), load task state (resume work). Long-term m. - retrieve from long-term memory (contextual recall), and save to long-term memory (persist new knowledge).
Storage Options
Memory files: Markdown stored in ~/.copilot/memory/
Embedding database: for semantic search (SQLite w/vector embeddings)
File structure: Flat directory, each memory = separate .md file with YAML frontmatter (metadata: timestamp, source, tags)
Memory Flow
Write - User message → LLM extracts memory-worthy info → Deduplication check via embeddings → If novel, write to markdown file + generate embedding → Store in SQLite.
Deduplicate - Before writing, compare new memory embedding against existing memories; skip if similarity > threshold.
Fact Extraction Prompt: Instructs LLM to identify: user preferences, project conventions, architectural decisions, tool configurations, recurring patterns. Long memories split into smaller chunks for better retrieval.
Read - User query → Generate query embedding → Vector similarity search in SQLite → Retrieve. Retrieval Parameters - a) Top-k: typically 5-10 memories, b) Similarity threshold: filters low-relevance matches, c) Context window budget: limits total injected memory tokens.
Limitations: memory bloat over time (needs pruning strategies), remove outdated memory, embedding quality affects retrieval accuracy
Cross-Agent Memory
Memory enhancement for multi-agent ecosystem (like GitHub Copilot which spans coding, CLI, code review, security, debugging, deployment, maintenance agents) - cross-agent memory allows agents to remember / learn across the entire development cycle, reducing the need to re-establish context at the start of each session by allowing validated information to persist across agentic workflows.
Cross-Agent Memory Lifecycle
Creation: Agents invoke Memory API’s store_memory tool to store a fact in the Memory DB when LLM discovers actionable patterns (e.g., a special logging convention or how to handle databases). Decentralized learning: example - Copilot code review finds a pattern (e.g., "Log files must follow app-YYYYMMDD.log"). Copilot coding agent later verifies and applies it to a new service. Copilot CLI uses it to locate debug logs.
Storage with Citations: Memories include references to specific code locations supporting the fact.
{ subject: "API version synchronization",
fact: "API version must match between client SDK, server routes, and documentation.",
citations: ["src/client/sdk/constants.ts:12", "server/routes/api.go:8", "docs/api-reference.md:37"],
reason: "If the API version is not kept properly synchronized, the integration can fail or exhibit subtle bugs. Remembering these locations will help ensure they are kept syncronized in future updates." }
Retrieval: At session start, the system retrieves recent repository memories and injects them into the agent prompt.
Verification: At retrieval time before use, agents verify citations dynamically against the current codebase to ensure info is relevant and to prevent invalid data (from abandoned branches) or memory attacks.
Reference: https://github.blog/ai-and-ml/github-copilot/building-an-agentic-memory-system-for-github-copilot/
Context Management - not optimization, but a necessity (OpenAI)
Tool calling is easy, context survival is the real challenge. Successful agents - long-run context, tools, memory and caching stay stable as complexity grows. Manus AI solved it => Meta purchase them.
Typical agentic workflow can make ~50 tool calls (hundreds of turns in prod). Architectures w/short sessions break by default. Goal - unbounded tool-call traces.
To prevent quality rot - repetitions, slower inference and weaker decisions, identified a pre-rot trigger (~128K to 200K tokens) & run context reduction:
Reversible compaction: drops payloads, but keeps their references (file path, URL, IDs).
Irreversible summarization: last resort only when compaction stops buying space AND keep logs => agent can recover details later.
Compact older history first. Keep the newest tool calls in full detail so the model retains fresh examples of correct tool usage and schema.
Offload token-heavy outputs, pre-summary conversation logs, and other big artifacts to files, keep short pointers in context, agent can retrieve these on demand using simple file-based search.
Cache engineering keeps costs and latency down. Agents process ~100 tokens per 1 token of output => use fixed-prefix prompts (identical beginning of prompts across requests) with predictable structure (consistent formatting) and only add new content at the end (append-only prompts)—this lets the AI reuse cached computations instead of reprocessing. Higher cache hits - a critical metric => faster responses and lower costs, since the AI skips re-reading identical content.
Source: here (see attached pdf there for details).