Memory

In this part, we will learn about memory in generative AI systems. Memory is a very important part that lets AI models remember information over time. This helps them give answers that make more sense and fit the conversation better. We will look at different types of memory, how to add memory to AI systems, and some simple tips to manage memory the right way.

Context Windows

Before we talk about memory, we first need to understand what a context window is. A context window is the amount of information an AI model can look at, or “keep in mind,” at one time while it is writing a reply. The bigger the context window, the more information the model can remember at once. If the context window is small, older information gets removed sooner.

LLMs (Large Language Models) read text using something called a sliding context window. This means the model can only “see” a fixed number of tokens (small pieces of text) at one time.

Think of it like a box that can only hold 10 letters.

+---+---+---+---+---+---+---+---+---+---+
| A | B | C | D | E | F | G | H | I | J |
+---+---+---+---+---+---+---+---+---+---+

Now say you want to add a new letter K. Since the box is already full, there is no empty space left. To make room, the oldest letter (A) is removed, everything else moves one step to the left, and the new letter is added at the end.

Before

+---+---+---+---+---+---+---+---+---+---+
| A | B | C | D | E | F | G | H | I | J |
+---+---+---+---+---+---+---+---+---+---+

Add K

                 ↓

After

+---+---+---+---+---+---+---+---+---+---+
| B | C | D | E | F | G | H | I | J | K |
+---+---+---+---+---+---+---+---+---+---+

This is almost exactly how a context window works inside an LLM. The model has only a fixed amount of space for tokens. When new tokens come in, the old tokens at the start get removed to make space for the new ones.

For example, say the context window can only hold 10 tokens. If an 11th token comes in, the oldest token gets removed. If a 12th token comes in, the second-oldest token gets removed too, and this keeps happening. Since the window keeps moving forward as new tokens come in, we call it a sliding context window.

Because of this, if you ask about A after K has been added, the model will no longer have A in its context window. So it will not be able to talk about A in its answer anymore.

Note: real LLMs actually work with tokens, not letters. We used letters here only because they are much easier to picture and understand.

Types of Memory

In generative AI, memory can be split into a few types, and each one is used for a different purpose. The main types of memory are:

• Short-term Memory: This type of memory holds information for a short time, usually just during one conversation or session. It helps the model stay on topic and keep track of what is being talked about right now.

• Long-term Memory: Long-term memory lets the model remember information across many conversations or sessions, not just one. This is important for things like personal assistants or recommendation systems, where the AI needs to remember things for a long time. We usually store this in a database or a vector database so we can look it up later. Long-term memory can be built using things like knowledge graphs, embeddings, or outside databases. In this type of memory, we first add the knowledge while building the system, kind of similar to how RAG works.

  We can also split long-term memory into three main types:

  • Factual Memory: This stores basic facts about the user, like their name, age, and so on. This is the kind of information that stays in the context window all the time. It is very small in size, so we can always keep it available to help make the response more personal.
  • Episodic Memory: This type stores specific events or things that happened before, so the model can remember past conversations and give answers that fit better. For example, think about a time you told the AI about your future goals. The AI does not need to keep this in the context window every single time. Instead, it can save this in episodic memory. Then, when you talk about your future goals again, it can bring back that information and give you a more personal answer. Here, we usually use a tool call to search for the right information, similar to how RAG searches, or we can use some other method to fetch it from episodic memory.
  • Semantic Memory: This stores general knowledge, like “Delhi is the capital of India,” and other facts like that.

  The problem with long-term memory is that it keeps growing over time, and we cannot fit everything into the context window because it has a limit. So we need a good strategy to decide what to pull from long-term memory and add into the context window when needed.

Mem0

Mem0 is a tool that helps us manage memory in generative AI systems. It makes it much easier to add memory to AI models and to keep track of which information matters. Mem0 can handle both short-term and long-term memory, so it is a handy tool for developers who want to build AI systems that can remember things over time.

Docs: docs.mem0.ai | docs.mem0.ai/open-source/python-quickstart

Mem0 can store memory in two main ways: a vector store (good for meaning-based search) and a graph store (good for relationship-based search). We will look at both, one by one.

Vector Store

A vector store is a type of database that is good at storing and searching for information based on meaning, not just exact words. This is useful for long-term memory because it lets us find relevant information quickly, even if the wording is a bit different.

Installation:

uv add mem0ai

Configuration:

from mem0 import Memory
import os
from dotenv import load_dotenv

load_dotenv()

API_KEY = os.getenv("LLM_API_KEY")

config = {
  "version": "v1.1",
  "embedder": {
    "provider": "openai",
    "config": {
      "api_key": API_KEY,
      "model": "text-embedding-3-small"
    }
  },
  "llm": {
    "provider": "openai",
    "config": {
      "api_key": API_KEY,
      "model": "gpt-4o"
    }
  },
  "vector_store": {
    "provider": "qdrant",
    "config": {
      "host": "localhost",
      "port": 6333
    }
  }
}

memory_client = Memory.from_config(config) # we use this to store and fetch memory

We run Qdrant locally using Docker, so we do not need to set up anything by hand:

# docker-compose.yml
services:
  qdrant:
    image: qdrant/qdrant
    container_name: qdrant
    ports:
      - "6333:6333"
    volumes:
      - ./.qdrantData:/qdrant/storage

Tip: You do not need to install any extra package just to talk to Qdrant, since Mem0 already comes with what it needs to connect to it. But if you want better keyword search (called BM25 hybrid search) along with meaning-based search, install fastembed as well:

uv add fastembed

Without fastembed, Mem0 still works fine, it just quietly skips the keyword search part and uses meaning-based search only.

Now let us use the OpenAI SDK along with Mem0 to build a small working example of a memory system. We will use the Memory class to store and fetch memory.

from mem0 import Memory
from dotenv import load_dotenv
import os
from openai import OpenAI
import json

load_dotenv()

API_KEY = os.getenv("OPENAI_API_KEY")

client = OpenAI(api_key=API_KEY)

API_KEY = os.getenv("LLM_API_KEY")

config = {
  "version": "v1.1",
  "embedder": {
    "provider": "openai",
    "config": {
      "api_key": API_KEY,
      "model": "text-embedding-3-small"
    }
  },
  "llm": {
    "provider": "openai",
    "config": {
      "api_key": API_KEY,
      "model": "gpt-4o"
    }
  },
  "vector_store": {
    "provider": "qdrant",
    "config": {
      "host": "localhost",
      "port": 6333
    }
  }
}

memory_client = Memory.from_config(config)

while True:
    user_input = input("User: ")
    if user_input.lower() == "exit":
        break

    user_id = "user_1"  # In a real app, you would get this from the user's session or login info

    search_memory = memory_client.search(query=user_input, user_id=user_id)

    memories = [
      f"ID:{memory.get('id')}\n\nMemory:{memory.get('memory')}"
      for memory in search_memory.get("results", [])
    ]

    SYSTEM_PROMPT = f"""
      You are a helpful assistant that remembers past conversations and information. Use the following memories to provide context for the current conversation. If the memories are not relevant, you can ignore them.

      Memories: {json.dumps(memories)}
    """

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_input}
        ]
    )

    answer = response.choices[0].message.content

    print(f"AI: {answer}")

    memory_client.add(
        messages=[
          {"role": "system", "content": SYSTEM_PROMPT},
          {"role": "user", "content": user_input},
          {"role": "assistant", "content": answer}
        ],
        user_id=user_id
    )

    print("Memory added successfully!")

Graph Store

We can also store memory in a graph store, which is a type of database that is good at storing and searching for information based on relationships between things, not just meaning. This is useful when your memory has a lot of connections in it, like “Bob is the manager of the user” or “the user prefers email over calls.”

A graph store generally uses the Cypher query language to store and fetch data. You can use Neo4j, Memgraph, Kuzu, or any other supported graph database. You do not need to learn Cypher yourself, Mem0 handles it for you behind the scenes. Cypher docs, if you are curious: neo4j.com/docs/cypher-manual/current

Why use a graph store?

A vector store is good at finding information based on meaning, but it is not great with relationships. A graph store is good at finding information based on relationships, but it is not great with meaning. So if the memory you want to store has a lot of connections in it, like a knowledge graph, then a graph store is a better fit.

Installation - a graph store needs some extra packages on top of the base mem0ai package, so install the graph extra instead of the plain one:

uv add "mem0ai[graph]"

This single command installs everything needed for graph memory to work, including the Neo4j driver, langchain-neo4j, and rank-bm25 for better search ranking. You do not need to install these one by one.

If you were already using an external graph database (like Neo4j) with an older version of Mem0, and you upgrade, you can remove those separate graph drivers. Newer Mem0 versions can build their own graph memory without needing a separate database, but this notes section is about connecting Mem0 to a real Neo4j database yourself, which is still fully supported through the graph_store config shown below.

Configuration:

from mem0 import Memory
import os
from dotenv import load_dotenv

load_dotenv()

API_KEY = os.getenv("LLM_API_KEY")

config = {
  "version": "v1.1",
  "embedder": {
    "provider": "openai",
    "config": {
      "api_key": API_KEY,
      "model": "text-embedding-3-small"
    }
  },
  "llm": {
    "provider": "openai",
    "config": {
      "api_key": API_KEY,
      "model": "gpt-4o"
    }
  },
  "graph_store": {
    "provider": "neo4j",
    "config": {
      "uri": os.getenv("NEO4J_URI"),
      "username": os.getenv("NEO4J_USERNAME"),
      "password": os.getenv("NEO4J_PASSWORD")
    }
  }
}

memory_client = Memory.from_config(config) # we use this to store and fetch memory

You can run a local Neo4j instance using Docker. Instructions are here. But running Neo4j locally needs a fairly strong machine. If your machine is not powerful enough, you can use Neo4j’s free cloud version called AuraDB instead. Instructions are here.

Now let us use the OpenAI SDK along with Mem0 to build a small working example of a memory system that uses Neo4j as the graph store. We will use the Memory class to store and fetch memory.

from mem0 import Memory
from dotenv import load_dotenv
import os
from openai import OpenAI

load_dotenv()

API_KEY = os.getenv("OPENAI_API_KEY")

client = OpenAI(api_key=API_KEY)

config = {
  "version": "v1.1",
  "embedder": {
    "provider": "openai",
    "config": {
      "api_key": API_KEY,
      "model": "text-embedding-3-small"
    }
  },
  "llm": {
    "provider": "openai",
    "config": {
      "api_key": API_KEY,
      "model": "gpt-4o"
    }
  },
  "graph_store": {
    "provider": "neo4j",
    "config": {
      "uri": os.getenv("NEO4J_URI"),
      "username": os.getenv("NEO4J_USERNAME"),
      "password": os.getenv("NEO4J_PASSWORD")
    }
  }
}

memory_client = Memory.from_config(config)

while True:
    user_input = input("User: ")
    if user_input.lower() == "exit":
        break

    user_id = "user_1"  # In a real app, you would get this from the user's session or login info

    search_memory = memory_client.search(query=user_input, user_id=user_id)

    memories = [
      f"ID:{memory.get('id')}\n\nMemory:{memory.get('memory')}"
      for memory in search_memory.get("results", [])
    ]

    SYSTEM_PROMPT = f"""
      You are a helpful assistant that remembers past conversations and information. Use the following memories to provide context for the current conversation. If the memories are not relevant, you can ignore them.

      Memories: {json.dumps(memories)}
    """

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_input}
        ]
    )

    answer = response.choices[0].message.content

    print(f"AI: {answer}")

    memory_client.add(
        messages=[
          {"role": "system", "content": SYSTEM_PROMPT},
          {"role": "user", "content": user_input},
          {"role": "assistant", "content": answer}
        ],
        user_id=user_id
    )

    print("Memory added successfully!")

Using Both Vector Store and Graph Store Together

You do not have to pick just one. Mem0 lets you combine a vector store (Qdrant) and a graph store (Neo4j) in the same config. This way, you get meaning-based search from Qdrant and relationship-based search from Neo4j, both at once.

Full Code (Vector + Graph)

from mem0 import Memory
from dotenv import load_dotenv
import os
from openai import OpenAI

load_dotenv()

API_KEY = os.getenv("OPENAI_API_KEY")

client = OpenAI(api_key=API_KEY)

config = {
  "version": "v1.1",
  "embedder": {
    "provider": "openai",
    "config": {
      "api_key": API_KEY,
      "model": "text-embedding-3-small"
    }
  },
  "llm": {
    "provider": "openai",
    "config": {
      "api_key": API_KEY,
      "model": "gpt-4o"
    }
  },
  "graph_store": {
    "provider": "neo4j",
    "config": {
      "uri": os.getenv("NEO4J_URI"),
      "username": os.getenv("NEO4J_USERNAME"),
      "password": os.getenv("NEO4J_PASSWORD")
    }
  },
  "vector_store": {
    "provider": "qdrant",
    "config": {
      "host": "localhost",
      "port": 6333
    }
  }
}

memory_client = Memory.from_config(config)

while True:
    user_input = input("User: ")
    if user_input.lower() == "exit":
        break

    user_id = "user_1"  # In a real app, you would get this from the user's session or login info

    search_memory = memory_client.search(query=user_input, user_id=user_id)

    memories = [
      f"ID:{memory.get('id')}\n\nMemory:{memory.get('memory')}"
      for memory in search_memory.get("results", [])
    ]

    SYSTEM_PROMPT = f"""
      You are a helpful assistant that remembers past conversations and information. Use the following memories to provide context for the current conversation. If the memories are not relevant, you can ignore them.

      Memories: {json.dumps(memories)}
    """

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_input}
        ]
    )

    answer = response.choices[0].message.content

    print(f"AI: {answer}")

    memory_client.add(
        messages=[
          {"role": "system", "content": SYSTEM_PROMPT},
          {"role": "user", "content": user_input},
          {"role": "assistant", "content": answer}
        ],
        user_id=user_id
    )

    print("Memory added successfully!")