Building A Local-First AI Memory Agent In Rust

Personal context gets scattered fast.

Commits are in GitHub. Follow-ups are in email. Meetings are in Calendar. Team context might live in Slack. Notes may be in Obsidian. When I want to remember why I made a decision, what I shipped last week, or which problem keeps repeating, the information is rarely in one place.

RustMarrow is my local-first experiment for that problem. It is a Rust CLI named

marrow

The important word is selected. A memory agent should not swallow everything blindly.

The Privacy Model Comes First

Before the features, the rule is simple: private context should stay local by default.

Marrow stores memory chunks in SQLite on your machine. Credentials live outside the repository.

.env

Example local setup:

mkdir -p ~/.marrow
cp .env.example ~/.marrow/.env
$EDITOR ~/.marrow/.env

The environment file uses local placeholders:

GITHUB_TOKEN=your_github_pat_here
GITHUB_USERNAME=your_github_username_here
ANTHROPIC_API_KEY=your_anthropic_key_here
MARROW_DB_PATH=~/.marrow/marrow.db

Data leaves your machine only when Marrow calls configured source APIs or sends selected context to Claude for

ask

digest

The Core Commands

Marrow’s CLI is organized around a small set of actions:

marrow pull
marrow pull --source github
marrow pull --source gmail
marrow pull --source calendar
marrow pull --source slack
marrow search "android crash"
marrow ask "what am I working on this week?"
marrow digest

There are also management commands:

marrow status
marrow clear github
marrow forget 42
marrow open

The shape is deliberate. Pull data, search locally, ask only when useful, and keep delete/clear commands close at hand.

SQLite As The Boring Center

SQLite is a good fit for a local memory agent because it is boring in exactly the right way. It is file-based, fast enough for personal context, easy to back up, and does not require running a server.

Marrow stores memory chunks with source metadata:

pub struct Chunk {
    pub source: String,
    pub source_id: String,
    pub title: String,
    pub content: String,
    pub url: Option<String>,
    pub tags: Vec<String>,
}

That gives every memory item enough identity to support incremental updates, search results, and later cleanup.

The store writes chunks with an upsert:

pub fn ingest(&self, chunk: &Chunk) -> Result<()> {
    let tags_json = serde_json::to_string(&chunk.tags)?;
    self.db.conn.execute(
        "INSERT INTO memory_chunks (source, source_id, title, content, url, tags, fetched_at)
         VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7)
         ON CONFLICT(source, source_id) DO UPDATE SET
           title      = excluded.title,
           content    = excluded.content,
           url        = excluded.url,
           tags       = excluded.tags,
           fetched_at = excluded.fetched_at",
        rusqlite::params![
            chunk.source,
            chunk.source_id,
            chunk.title,
            chunk.content,
            chunk.url,
            tags_json,
            Utc::now().to_rfc3339(),
        ],
    )?;

    Ok(())
}

This is the part I care about most: repeated pulls should not create endless duplicates. Each source owns stable IDs, and the local store updates the existing row when the same item comes back.

Backup And Restore

The v1.1.0 release added JSON export and import:

marrow export --json marrow-backup.json
marrow import --json marrow-backup.json

The export contains memory chunks only:

• Source.
• Source ID.
• Title.
• Content.
• URL.
• Tags.
• Fetched timestamp.

It does not include provider credentials or

.env

The Rust model is plain:

#[derive(Debug, Serialize, Deserialize)]
pub struct ExportedChunk {
    pub source: String,
    pub source_id: String,
    pub title: String,
    pub content: String,
    pub url: Option<String>,
    pub tags: Vec<String>,
    pub fetched_at: String,
}

That gives Marrow a safer migration story. If I move machines, reset a database, or want a backup before testing a new puller, I can export memory chunks without bundling secrets.

[!WARNING] A memory export can still contain private work context. Treat backup JSON files like sensitive documents, even when they do not contain API keys.

Obsidian Mirroring

Marrow can optionally mirror memory chunks into an Obsidian vault. That is useful because not every memory workflow should happen through a CLI or a model.

Markdown files are easy to browse, search, link, and archive. If a memory chunk is useful enough to keep, it can also become part of a broader notes system.

The optional mirror is controlled by a local path:

OBSIDIAN_VAULT_PATH=/Users/you/ObsidianVault

That should stay local. Generated memory notes may contain private context from commits, emails, calendar events, or messages.

Where Claude Fits

Marrow can ask Claude questions using recent local memory as context:

marrow ask "which issues look urgent?"

This is useful, but it is also the sensitive boundary. Search can happen locally without sending text to a model. Asking a question means selecting context and sending it to the configured model provider.

That is why I like keeping both commands:

• code Copy

  marrow search

  for local-only recall.
• code Copy

  marrow ask

  when model reasoning is worth the privacy tradeoff.

The user should choose when context leaves the machine.

Key Takeaways

• A personal AI memory agent should start with a clear privacy model.
• SQLite is a strong default for local-first memory because it is simple and portable.
• Provider pullers should ingest selected context, not everything by default.
• JSON backup and restore should exclude credentials but still be treated as sensitive.
• Local search and model-backed answers should be separate commands.

RustMarrow is not finished, but the direction is clear: local-first storage, explicit source pulls, searchable memory, careful model boundaries, and backup tools that respect the fact that personal context is private context.