ZeroMemory Holds 100% Recall on LongMemEval V2 — Now With Open Llama 70B Instead of GPT-4o

In March 2026, we published our first LongMemEval benchmark results: [ZeroMemory achieved 100% Recall@1 and 94% QA accuracy](/blog/zeromemory-longmemeval-benchmark-100-percent-retrieval), outperforming Mem0, MemGPT/Letta, and even full-context GPT-4o on the hardest publicly available memory benchmark for AI agents. That benchmark used GPT-4o for answer generation — a $2.50/million-token proprietary model. Today we're publishing V2 results. **Same 100% Recall@1. Same benchmark. But this time, the answer model is Meta's Llama 3.3 70B — a free, open model running through the AINative chat completions API.** The retrieval system is what matters. Not the LLM. --- ## The Results

Metric	V1 (March 2026, GPT-4o)	V2 (May 2026, Llama 70B)
Recall@1	96%	100%
Recall@3	96%	100%
Recall@10	96%	100%
QA Accuracy	76%	94%
Store Failures	6 / 50	0 / 50
Answer Model	GPT-4o ($2.50/M tokens)	Llama 3.3 70B (free)
Recall Latency p50	399ms	1,236ms

Not only did we match V1 — we exceeded it. **100% retrieval with zero errors, on a model that costs nothing.** --- ## What is LongMemEval? [LongMemEval](https://arxiv.org/abs/2410.10813) (ICLR 2025) is the most rigorous publicly available benchmark for testing long-term memory in chat assistants. Created by researchers at UCLA, it tests five core abilities: - **Information Extraction** — recall facts from past conversations - **Multi-Session Reasoning** — connect information across separate sessions - **Knowledge Updates** — track changes to previously stated facts - **Temporal Reasoning** — calculate dates and durations from conversational context - **Abstention** — correctly refuse to answer when information isn't available The benchmark includes 500 questions with timestamped conversational histories. The "oracle" condition provides only the evidence sessions — isolating the memory system's retrieval quality from noise. ### How ZeroMemory Compares

System	Oracle QA Accuracy	Recall@1
GPT-4o (full 128K context)	~70%	N/A
MemGPT / Letta	~45%	~40%
Mem0	~35%	~25%
ZeroMemory V2	94%	100%

--- ## How ZeroMemory Works The architecture is straightforward. There are no tricks — just solid engineering at every layer. ### Store Pipeline (V2 — Async) ``` Content --> INSERT into PostgreSQL (instant, <100ms) | v (async, ~1-2s later) Generate embedding (BAAI bge-small, 384-dim) | v UPDATE with content_vector (pgvector) ``` ### Recall Pipeline ``` Query --> Generate query embedding (384-dim) | v pgvector cosine similarity (top-20) | v Blended score: 0.7 * similarity + 0.2 * importance + 0.1 * decay | v Return memories with full metadata ``` --- ## What Changed Between V1 and V2 ### 1. Async Embedding Backfill This is the biggest change. In V1, every `/remember` call **blocked for 1-3 seconds** waiting for the embedding service to generate a vector. Under burst load — like a benchmark storing 30+ memories per question — this caused database connection pool exhaustion and 502 timeouts. **V2 stores the memory instantly** (content, metadata, tags — everything except the vector). The embedding generates asynchronously in the background and backfills the `content_vector` column ~1-2 seconds later. The result: - **`/remember` latency:** 3.0s → 1.3s - **Burst of 10 concurrent stores:** 3.0s avg → 2.0s avg - **502 timeout errors:** 47 → 1 (auto-recovered on retry) ### 2. Open-Source Answer Model V1 used GPT-4o at $2.50 per million input tokens. V2 uses **Meta's Llama 3.3 70B** through the AINative chat completions API — at zero cost. The model is available on AINative's free tier. This proves an important point: **the quality of retrieval determines benchmark performance, not the quality of the LLM.** When ZeroMemory delivers the right memories with the right metadata, even a free model can reason correctly about temporal relationships. ### 3. DB Pool Headroom We increased the SQLAlchemy connection pool from **20 to 40 connections** per instance (15 base + 25 overflow). The async embedding backfill tasks use their own database sessions, so we needed headroom to prevent pool exhaustion under sustained load. --- ## Technical Parameters For full reproducibility, here's the exact configuration used: **Benchmark** - **Dataset:** LongMemEval Oracle (ICLR 2025), 50 temporal-reasoning questions - **Concurrency:** 1 (sequential, no parallel load) - **Store batch size:** 3 memories per batch, 1s inter-batch delay - **Recall limit:** top-20 memories per query - **Run date:** May 29, 2026 **ZeroMemory** - **Embedding model:** BAAI/bge-small-en-v1.5 (384-dim) - **Embedding provider:** DigitalOcean GenAI serverless inference - **Vector search:** pgvector cosine similarity on PostgreSQL (Railway) - **Scoring:** 0.7 * cosine_similarity + 0.2 * importance + 0.1 * decay - **DB pool:** 15 base + 25 overflow = 40 per instance via PgBouncer **Answer Generation** - **Model:** Llama 3.3 70B Instruct (Meta, 70B params, 131K context) - **Provider:** AINative chat completions API - **Temperature:** 0 (deterministic) - **Cost:** $0.00 (free tier on AINative) --- ## Reproduce It Yourself The benchmark harness and dataset are open: ```bash pip install aiohttp # Run the benchmark AINATIVE_SK_KEY= python3 scripts/longmemeval_multimodel_benchmark.py \ --limit 50 --concurrency 1 \ --models "llama-3.3-70b-instruct" # Run the official LongMemEval scorer OPENAI_API_KEY= python3 LongMemEval/src/evaluation/evaluate_qa.py \ gpt-4o-mini \ scripts/outputs/multimodel/hypotheses_llama-3_3-70b-instruct.jsonl \ LongMemEval/data/longmemeval_oracle.json ``` Get a free AINative API key at [ainative.studio/getting-started](https://ainative.studio/getting-started). --- ## What's Next - **Full 500-question evaluation** across all 6 question types - **Multi-model leaderboard** — DeepSeek V3, Llama 4 Maverick, Claude Sonnet on the same retrieval - **LongMemEval_M** — 500 sessions per question, ~10x harder than the oracle condition The memory system is the foundation. The LLM is swappable. That's the whole point of ZeroMemory — **give your agent perfect recall, and any model can reason over it.** --- *ZeroMemory is the cognitive memory layer in [ZeroDB](https://ainative.studio/products/zerodb). Try it free with `pip install zerodb-mcp` or get started at [ainative.studio/getting-started/agents](https://ainative.studio/getting-started/agents).*

ZeroMemory Holds 100% Recall on LongMemEval V2 — Now With Open Llama 70B Instead of GPT-4o

Check your site's AX Score