Supermemory is the new State-of-the-Art in agent memory

A new memory architecture that solves long-term forgetting in LLMs, delivering state-of-the-art performance on LongMemEval by enabling reliable recall, temporal reasoning, and knowledge updates at scale.

LongMemEval-S Benchmark: Supermemory vs Zep vs Full context

Introduction

Large Language Models (LLMs) fundamentally suffer from "forgetting". They treat every interaction as a new, discrete event, lacking the persistent continuity required for personalized user experiences. While Context Windows are growing, LLMs are still prone to forgetting context in the middle of the context window as shown by Liu et al. [1] and high latency.

In this report, we introduce Supermemory, a memory engine designed to solve the problem of long-term coherence. We demonstrate that Supermemory achieves State-of-the-Art (SOTA) results on LongMemEval_s [2], effectively solving the challenges of temporal reasoning and knowledge conflicts in high-noise environments (115k+ tokens).

Authors

Soham Daga - AI Researcher, Supermemory
Sreeram Sreedhar - AI Researcher, Supermemory
Dhravya Shah - CEO, Supermemory

The Evaluation Landscape: Why LongMemEval?

Current benchmarks in the LLM memory space often fail to capture the chaos of real-world production environments. Benchmarks like LoCoMo [3] are insufficient for modern models due to their limited context size and lack of knowledge updates (testing ability to overwrite or update old and obsolete information with newer information).

We utilized LongMemEval [2] for validation because it represents the most rigorous approximation of real-world chat history. It challenges the retrieval system not just on recall, but on reasoning over time and filtering out noise. Unlike other benchmarks (which present human-human interaction), LongMemEval tests for human-assistant interactions, which is representative of real-world usage, as also highlighted by Rasmussen et al. [4].

LongMemEval_s spans 500 questions split into 6 categories and evaluates five core memory capabilities:

• Information Extraction: Accurately extracting and storing factual information from conversations.
  • single-session-user: Retrieving literal context mentioned by the user within a single session.
  • single-session-assistant: Retrieving literal context mentioned by the assistant within a single session.
  • single-session-preference: Extracting implicit user preferences to inform personalized responses.
• Multi-Session Reasoning: Synthesizing information scattered across multiple conversation sessions. Categories: multi-session
• Knowledge Update: Handling scenarios where newer information contradicts or supersedes older facts. Categories: knowledge-update
• Temporal Reasoning: Understanding the sequence of events, calculating time intervals, and reasoning about relative timestamps. Categories: temporal-reasoning
• Abstaining on Unanswerable Questions: Recognizing when sufficient information is not available and appropriately declining to answer. Categories: all

These capabilities cover a broad variety of general real-world use-cases.

Methodology: Supermemory's Architecture

Supermemory outperforms existing solutions by minimizing semantic ambiguity, which is a big reason for context not being utilized effectively in LLMs as demonstrated by Keluskar et al. [5]. We achieve this by coupling memories with temporal metadata, relations, and raw chunks.

1. Chunk-based Ingestion & Contextual Memories

Standard RAG (Retrieval Augmented Generation) [6] often fails because it retrieves raw chunks that lack context when isolated from the conversation.

Example of where standard RAG fails due to ambiguity.

• Chunking: We decompose large sessions into manageable semantic blocks.
• Memory Generation: As we index the chunk, we also generate memories — single (atomic) pieces of information that resolve ambiguous references within the chunk using a modified version of Contextual Retrieval. [8]

2. Relational Versioning & Knowledge Chains

Supermemory also defines semantic relationships between new and existing memories. This allows us to map evolution of facts:

• updates (State Mutation): Handles contradictions or corrections (e.g., "My favorite color is now Green" updates "My favorite color is Blue"), creating a version history of sorts.
• extends (Refinement): Supplements existing nodes with new details without contradiction (e.g., adding a job title to an existing employment memory).
• derives (Inference): Captures second-order logic inferred from combining multiple distinct memories.

Example of relational versioning in Supermemory.

3. Temporal Grounding

A core differentiator in our architecture is the dual-layer time-stamping approach, which drove our high scores in the temporal-reasoning, knowledge-update, and multi-session categories. For every memory, we extract:

• documentDate: The time the conversation took place.
• eventDate: The extracted timestamp of when the event described in the conversation actually occurred.

Example of temporal grounding in Supermemory.

4. Hybrid Search Strategy

We perform semantic search on the memories to identify relevant concepts. As memories encapsulate singular pieces of information, i.e. high signal and low noise, this is generally more accurate than directly searching for the noisy chunks as noted by several sources [9][10].

Once a hit is found, we inject the original source chunk for the memory into the result output. This allows the LLM to access the "finer details" required for nuance while relying on the atomicity of the memory for high-precision retrieval. This serves to resolve the concern of information loss brought up in Section 5.2 of the LongMemEval [2] paper.

5. Session-Based Ingestion

Unlike the evaluation methodology described in the LongMemEval [2] paper, which processes conversation history round-by-round (one user message followed by one assistant response), we choose to ingest the dataset session-by-session.

Performance Results

Supermemory demonstrates superior performance across all categories on LongMemEval_s. The system shows particular strength in Multi Session (71.43%) and Temporal Reasoning (76.69%), areas where standard vector-store approaches historically struggle.

LLM-as-Judge Evaluation

How to reproduce these results

We believe in transparency and rigorous cross-validation.

• Data & Prompts: The full prompt used for answering is provided in our appendix. For answer evaluation, we used gpt-4o with the question-specific prompts provided in the LongMemEval paper [2].
• Codebase: The ingestion pipeline, search logic, and evaluation scripts are available in our GitHub repository.

Conclusion

The ability to accurately recall user details, respect temporal sequences, and update knowledge over time is not a "feature" — it is a prerequisite for Agentic AI.

By moving beyond simple vector similarity and implementing this form of disambiguation, Supermemory provides a robust backend for enterprise applications. It transforms the LLM from a stateless processor into a stateful assistant, capable of maintaining long-term, personalized user narratives with high fidelity.