ASP

ASP is a ground-up, single-user Rust rewrite of Karpathy's arxiv-sanity: one CLI binary bundling the full arXiv ingestion pipeline and an Axum web server, with nothing to operate beyond a bundled SQLite file. A paper-recommendation machine you run for yourself, not an account on someone else's Flask app.

The biggest performance wins in my C++ work usually come from a better idea, not better code: a smarter index, a tighter estimator, better asymptotics. Those ideas live in papers, and neither scrolling arXiv nor trusting social media surfaces them at scale. ASP is the machine I built instead: it crawls arXiv on my topics, learns my taste from what I save, ranks everything else against it, and keeps the result one browser tab away.

01Started from arxiv-sanity, kept zero lines of it

Credit first: the concept is Karpathy's arxiv-sanity, and the pipeline skeleton (fetch, PDF, text, TF-IDF, per-user SVM) is his design. Nothing of the original code survives. The rebuild came in two waves:

1. UI reworked from zero.
2. Backend fully rewritten from Python to Rust, every module reimplemented, not translated. ~8k lines, one crate.

Before, the original:

After:

The deepest change is the operating model:

That inversion decided the index structure, storage, training loop, and frontend.

02The backend is a straight line, on purpose

One binary: the ingestion pipeline as CLI subcommands, plus an Axum web server as one more subcommand. State is a .pipeline/ directory and one bundled SQLite file. No service mesh, no services.

  arXiv Atom API        OpenAlex API
        │                    │
        v                    v
 ┌───────────────────────────────────┐
 │ fetch-papers      → db.jsonl      │   metadata, citations
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ download-pdfs     → pdf/          │   rate limited, resumable
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ parse-pdf-to-text → txt/          │   pdftotext, length fences
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ thumb-pdf         → thumb/        │   ImageMagick
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ analyze           → tfidf, hnsw   │   vectors + ANN index
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ buildsvm          → user_sim      │   Personal Lib Recommender
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ make-cache        → serve cache   │   search dicts, sort orders
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ serve   (Axum, snapshot swap)     │──> browser
 └───────────────────────────────────┘

Each stage reads the artifacts of the stage above it, from disk, and writes its own. No horizontal communication between modules: no shared mutable state, no coordinating daemon.

• A crash in any stage corrupts nothing upstream or downstream; you rerun that stage.
• Every stage is restartable and testable in isolation, from real on-disk inputs.
• The contract between modules is a file format: versionable, diffable, inspectable with less.
• Debugging surface is a directory listing, not a distributed trace.

run-all chains the stages; recover audits the database against actual pdf/, txt/, thumb/ contents and repairs drift.

One deliberate quirk: pipeline subcommands are hidden from --help. The web UI is the front door; it spawns the same binary as a subprocess with those hidden flags, routing every download through the Settings flow that enforces arXiv's contact-email etiquette, and guaranteeing the UI and pipeline can never version apart. Old Karpathy-era corpora migrate in place: loaders read legacy pickle alongside native bincode and JSON.

03The text stack: classical, inspectable, replaceable

No embedding model, no GPU. Deliberately.

A self-hosted tool must run on whatever box you have, train in seconds, and stay auditable: when a recommendation looks wrong, I read the exact term weights that produced it. The representation is the most replaceable module: index, ingest path, and serving layer are all vector-agnostic, so swapping in embeddings later is an isolated change.

04HNSW is here for inserts, not queries

At personal corpus scale, exact nearest-neighbor scan would be fast enough. HNSW earns its place because of the write pattern: a shared server ingests in bulk on a schedule; I ingest one paper at a time. The original architecture answered any corpus change with a full pipeline rerun, hours of compute to add one paper, which is how you stop adding papers.

The incremental path does the minimum work that preserves correctness:

// Incremental TF-IDF policy: keep the existing IDF fixed for
// single-paper ingest. Full recomputation is handled by run_analyze.
let vector = vectorize_document_text(&text, &meta);   // frozen vocab + IDF

if let Some(&idx) = meta.ptoi.get(&pid) {
    tfidf.vectors[idx] = vector.clone();              // re-ingest: replace in place
} else {
    meta.ptoi.insert(pid.clone(), tfidf.vectors.len());
    tfidf.vectors.push(vector.clone());               // new paper: append
}
// ...
index.insert(pid.clone(), vector.clone())?;           // O(log n) graph walk

• Frozen IDF. One paper among thousands moves corpus statistics by noise; drift is repaid at the next full analyze.
• insert instead of rebuild. HNSW insertion is a logarithmic graph walk, the same operation construction is made of: adding a paper costs about one query.
• Designed failure mode. If the index length disagrees with the vector matrix, or a paper is replaced (no update-in-place in the graph), the code rebuilds from scratch. Correctness degrades to slowness, never wrong neighbors.

05Serving: readers never wait for a rebuild

Request handlers read from an immutable snapshot (paper db, HNSW index, recommendation lists, precomputed search dictionaries):

data: Arc<RwLock<Arc<ServeData>>>,

A handler holds the read lock just long enough to clone the inner Arc, then works on its private snapshot for the whole request. A recompute builds an entire new snapshot off to the side and swaps the inner pointer under a brief write lock.

 requests ──> clone Arc ──> read snapshot A ──> respond
                                  ^
                                  │   atomic pointer swap
 pipeline ──> build snapshot B ───┘ (A freed when last reader drops)

RCU flavored: reads are lock-free in the steady state, an in-flight request keeps a consistent view while the world is replaced beneath it. Search never touches a model at request time; queries hit per-paper term-weight dictionaries built at cache time (title terms x3 with a per-term cap, categories pinned, abstract x1), so the request path is hash lookups and additions.

06Politeness is enforced by the kernel

Multi-host means every user is a crawler, so ASP runs below the published limits:

Two mechanisms hold the limits across process boundaries:

• The metadata limiter persists its last request timestamp to a file using wall-clock time; every fresh process inherits the previous one's state on construction.
• A global flock(2) on .pipeline/arxiv.lock serializes arXiv access between CLI and server. The kernel releases the lock if a process crashes, so there's no stale-lockfile cleanup.

07The frontend ships inside the binary

The browser is a thin, honest view over server state, not a second app with its own truth.

• Server-rendered minijinja templates, vanilla JS with a small jQuery layer: ~4.3k lines of templates, ~2.1k of CSS, no build step, no bundler, no framework version to chase.
• Rendering is cheap because the expensive work happened at cache time; a page is template substitution over a precomputed snapshot.
• Long-running pipeline jobs are exposed as jobs with status endpoints; the page polls and drives a progress panel: paste an arXiv ID, watch it move through fetch, extract, vectorize, index, without opening a terminal.
• MathJax renders LaTeX in abstracts, d3 draws topic views, thumbnails give visual memory, theme persists because this page is open at 2am.

Every feature maps to one loop: scan, judge, save, let the model learn.

08What it traded, what it bought

And operationally, against the original:

On Hermes, my homelab, the container runs persistently, recrawls on schedule, and its library survives restarts.

09Further reading

• arxiv-sanity-preserver: Karpathy's original, conceptual ancestor of the pipeline.
• Malkov & Yashunin, HNSW: the index behind the incremental ingest path.
• Malisiewicz et al., Exemplar-SVMs: the "my library against the world" formulation.
• OpenAlex: citation metadata behind the impact score.