ASP

ASP is a ground-up, single-user Rust rewrite of Karpathy's arxiv-sanity: one CLI binary bundling the full arXiv ingestion pipeline and an Axum web server, with nothing to operate beyond a bundled SQLite file. A paper-recommendation machine you run for yourself, not an account on someone else's Flask app.

The biggest performance wins in my C++ work rarely come from writing better code; they come from implementing a better idea: a smarter index structure, a tighter estimator, an algorithm with better asymptotics than the one I was about to hand-roll. Those ideas live in papers, so finding them is part of the job, and neither scrolling arXiv listings nor trusting social media to surface them scales; the first costs hours, the second optimizes for hype. ASP is the machine I built instead: it crawls arXiv on my topics, learns my taste from what I save, ranks everything else against it, and keeps the result one browser tab away.

01Started from arxiv-sanity, kept zero lines of it

Credit first: the concept is Karpathy's arxiv-sanity, and the pipeline skeleton (fetch, PDF, text, TF-IDF, per user SVM) is his design. But nothing of the original code survives. The rebuild came in two waves:

1. UI reworked from zero.
2. Backend fully rewritten from Python to Rust, every module reimplemented, not translated. ~8k lines, one crate.

Before, the original:

After:

The deepest change is not the language, it is the operating model:

That inversion quietly decided the index structure, the storage, the training loop, and the frontend. Everything below follows from it.

02The backend is a straight line, on purpose

The whole system is one binary: the ingestion pipeline as CLI subcommands, plus an Axum web server as one more subcommand. State is a .pipeline/ directory and one bundled SQLite file. There is no service mesh because there are no services.

  arXiv Atom API        OpenAlex API
        │                    │
        v                    v
 ┌───────────────────────────────────┐
 │ fetch-papers      → db.jsonl      │   metadata, citations
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ download-pdfs     → pdf/          │   rate limited, resumable
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ parse-pdf-to-text → txt/          │   pdftotext, length fences
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ thumb-pdf         → thumb/        │   ImageMagick
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ analyze           → tfidf, hnsw   │   vectors + ANN index
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ buildsvm          → user_sim      │   Personal Lib Recommender
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ make-cache        → serve cache   │   search dicts, sort orders
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ serve   (Axum, snapshot swap)     │──> browser
 └───────────────────────────────────┘

Each stage reads the artifacts of the stage above it, from disk, and writes its own. There is almost no horizontal communication between modules. No stage calls into another's internals, no shared mutable state, no daemon coordinating them. That is an axiom I hold for uptime and maintainability:

• A crash in any stage corrupts nothing upstream and nothing downstream; you rerun that stage.
• Every stage is restartable and testable in isolation, from real on-disk inputs.
• The contract between modules is a file format, which is the most honest interface there is: versionable, diffable, inspectable with less.
• When something looks wrong, the debugging surface is a directory listing, not a distributed trace.

run-all chains the stages; a recover subcommand audits the database against the actual pdf/, txt/, thumb/ contents and repairs drift. More on why that subcommand exists later.

One deliberate quirk: the pipeline subcommands are hidden from --help. The web UI is the front door; it spawns the same binary as a subprocess with those hidden flags. This routes every download through the Settings flow that enforces arXiv's contact email etiquette, and it guarantees the UI and the pipeline can never version apart, because they are the same program. Old Karpathy era corpora migrate in place: the loaders read legacy pickle alongside native bincode and JSON, so compatibility is a data format concern, never a code one.

03The text stack: classical, inspectable, replaceable

No embedding model, no GPU. Deliberately.

The rationale is operational. A self hosted tool must run on whatever box you have, train in seconds, and be auditable: when a recommendation looks wrong, I read the exact term weights that produced it. And the representation is the most replaceable module in the system; the index, ingest path, and serving layer are all vector agnostic, so swapping in embeddings later is an isolated change at a designed seam.

04HNSW is here for inserts, not queries

An admission: at personal corpus scale, exact nearest neighbor scan would be fast enough. HNSW earns its place because of what single user hosting does to the write pattern.

A shared server ingests in bulk on a schedule. I ingest one paper at a time, the moment I read something worth keeping. The original architecture answered any corpus change with a full pipeline rerun: recompute TF-IDF over everything, rebuild the index, retrain. Hours of compute to add one paper. In practice that means you stop adding papers, and the tool dies of friction.

The incremental path does the minimum work that preserves correctness:

// Incremental TF-IDF policy: keep the existing IDF fixed for
// single-paper ingest. Full recomputation is handled by run_analyze.
let vector = vectorize_document_text(&text, &meta);   // frozen vocab + IDF

if let Some(&idx) = meta.ptoi.get(&pid) {
    tfidf.vectors[idx] = vector.clone();              // re-ingest: replace in place
} else {
    meta.ptoi.insert(pid.clone(), tfidf.vectors.len());
    tfidf.vectors.push(vector.clone());               // new paper: append
}
// ...
index.insert(pid.clone(), vector.clone())?;           // O(log n) graph walk

Three decisions packed in there:

• Frozen IDF. One paper among thousands moves corpus statistics by noise. Recomputing per insert buys nothing; drift is repaid at the next full analyze.
• insert instead of rebuild. HNSW insertion is a logarithmic graph walk, the same operation construction is made of. Adding a paper costs about one query. Hours become seconds.
• Designed failure mode. If the stored index length disagrees with the vector matrix, or a paper is replaced (the graph has no update in place), the code rebuilds from scratch instead of serving a stale index. Correctness degrades to slowness, never to wrong neighbors.

05Serving: readers never wait for a rebuild

Request handlers read from an immutable snapshot of the world (paper db, HNSW index, recommendation lists, precomputed search dictionaries):

data: Arc<RwLock<Arc<ServeData>>>,

The double Arc is the pattern. A handler holds the read lock just long enough to clone the inner Arc, a refcount bump, then works on its private snapshot for the whole request. A recompute builds an entire new snapshot off to the side and swaps the inner pointer under a brief write lock.

 requests ──> clone Arc ──> read snapshot A ──> respond
                                  ^
                                  │   atomic pointer swap
 pipeline ──> build snapshot B ───┘ (A freed when last reader drops)

RCU flavored: reads are lock free in the steady state, a request in flight keeps a consistent view while the world is replaced beneath it, and nothing is ever half updated. Search itself never touches a model at request time; queries hit per paper term weight dictionaries built at cache time (title terms x3 with a per term cap, categories pinned, abstract x1), so the request path is hash lookups and additions.

06Politeness is enforced by the kernel

Multi host means every user is a crawler, so API courtesy has to be structural. ASP runs below the published limits on purpose:

Two mechanisms make the limits hold where naive limiters break, across process boundaries:

• The metadata limiter persists its last request timestamp to a file using wall clock time. Every fresh process (each CLI run, each server spawned ingest subprocess) inherits the previous one's state on construction.
• A global flock(2) on .pipeline/arxiv.lock serializes arXiv access between the CLI and the server outright. The kernel releases the lock if a process crashes, so there is no stale lockfile failure mode to clean by hand.

07The frontend ships inside the binary

The frontend mirrors the backend's philosophy: the browser is a thin, honest view over server state, not a second application holding its own truth.

• Server rendered minijinja templates, vanilla JS with a small jQuery layer: ~4.3k lines of templates, ~2.1k of CSS, no build step, no bundler, no framework version to chase. The UI ships inside the binary and cannot drift from it.
• Rendering is cheap because the expensive work happened at cache time; a page is template substitution over a precomputed snapshot.
• The one place async state genuinely matters is long running pipeline jobs. The server exposes them as jobs with status endpoints; the page polls and drives a progress panel. The payoff is the whole point of the tool: paste an arXiv ID, watch it move through fetch, extract, vectorize, index, and see the feed update, without opening a terminal.
• Reading loop conveniences exist for the same reason: MathJax renders LaTeX in abstracts, d3 draws topic views, thumbnails give visual memory, the theme persists because this page is open at 2am.

Every feature above maps to one loop: scan, judge, save, let the model learn. Anything that did not serve that loop did not get built.

08What it traded, what it bought

And operationally, against the original:

On Hermes, my homelab, the container runs persistently, recrawls on schedule, and its library survives restarts. One process, one directory.

09Further reading

• arxiv-sanity-preserver: Karpathy's original, conceptual ancestor of the pipeline.
• Malkov & Yashunin, HNSW: the index behind the incremental ingest path.
• Malisiewicz et al., Exemplar-SVMs: the "my library against the world" formulation.
• OpenAlex: citation metadata behind the impact score.