ASP

ASP: single-user Rust rewrite of Karpathy's arxiv-sanity. One binary, arXiv ingestion pipeline plus Axum server. State is a .pipeline/ directory and one bundled SQLite file.

01Rewrite, not a port

Skeleton is Karpathy's: fetch, PDF, text, TF-IDF, per-user SVM. Zero original lines survive. UI rebuilt from zero. Backend Python to Rust, ~8k lines, one crate.

Before:

After:

The operating model inverted, which decided index structure, storage, training loop and frontend:

02Pipeline

  arXiv Atom API        OpenAlex API
        │                    │
        v                    v
 ┌───────────────────────────────────┐
 │ fetch-papers      → db.jsonl      │   metadata, citations
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ download-pdfs     → pdf/          │   rate limited, resumable
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ parse-pdf-to-text → txt/          │   pdftotext, length fences
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ thumb-pdf         → thumb/        │   ImageMagick
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ analyze           → tfidf, hnsw   │   vectors + ANN index
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ buildsvm          → user_sim      │   Personal Lib Recommender
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ make-cache        → serve cache   │   search dicts, sort orders
 └───────────────┬───────────────────┘
                 v
 ┌───────────────────────────────────┐
 │ serve   (Axum, snapshot swap)     │──> browser
 └───────────────────────────────────┘

Stages communicate only through on-disk artifacts. No shared state, no daemon. A crash costs one stage rerun. Each stage is restartable and testable alone. The module contract is a file format: versionable, diffable, inspectable with less.

run-all chains stages. recover audits the db against pdf/, txt/, thumb/.

Pipeline subcommands are hidden from --help; the UI spawns the same binary with those flags. Downloads carry arXiv's contact email. UI cannot version apart from pipeline. Legacy pickle loaders migrate old corpora in place.

03Text stack

No embedding model, no GPU.

Constraints: any box, seconds to train, auditable. A wrong recommendation resolves to its term weights. Index, ingest and serving stay vector-agnostic; embeddings drop in later.

04HNSW is here for inserts, not queries

Exact scan is fast enough at this corpus size. HNSW earns its place on the write pattern: one paper at a time. The original rebuilt everything per paper, hours of compute.

// Incremental TF-IDF policy: keep the existing IDF fixed for
// single-paper ingest. Full recomputation is handled by run_analyze.
let vector = vectorize_document_text(&text, &meta);   // frozen vocab + IDF

if let Some(&idx) = meta.ptoi.get(&pid) {
    tfidf.vectors[idx] = vector.clone();              // re-ingest: replace in place
} else {
    meta.ptoi.insert(pid.clone(), tfidf.vectors.len());
    tfidf.vectors.push(vector.clone());               // new paper: append
}
// ...
index.insert(pid.clone(), vector.clone())?;           // O(log n) graph walk

One paper moves corpus statistics by noise, so IDF stays frozen until the next analyze. Insertion is the logarithmic walk construction is made of: a paper costs one query. Index length mismatched with the vector matrix, or a replaced paper, forces a full rebuild. Slow, never wrong neighbors.

05Serving

data: Arc<RwLock<Arc<ServeData>>>,

Handlers clone the inner Arc under the read lock, then work on a private snapshot. Recompute builds a snapshot aside, swaps the pointer under a brief write lock. RCU flavored: reads are lock-free, in-flight requests keep a consistent view.

No model at request time. Queries hit per-paper term-weight dicts built at cache time: title x3 with per-term cap, categories pinned, abstract x1.

06Rate limits, enforced by the kernel

Multi-host means every user is a crawler. ASP runs below published limits:

Limits hold across processes:

• The metadata limiter persists its last request timestamp to a file. A fresh process inherits it.
• A global flock(2) on .pipeline/arxiv.lock serializes arXiv access between CLI and server. The kernel releases it on crash, so no stale-lockfile cleanup.

07Frontend, in the binary

• minijinja templates, vanilla JS, small jQuery layer. ~4.3k lines templates, ~2.1k CSS. No build step, no bundler.
• Pages are template substitution over the precomputed snapshot.
• Pipeline jobs expose status endpoints; the page polls a progress panel. Paste an arXiv ID, watch fetch, extract, vectorize, index.
• MathJax abstracts, d3 topic views, thumbnails, persisted theme.

08Trade-offs

Against the original:

Runs as a container on Hermes, my homelab. Scheduled recrawl, library survives restarts.

09Further reading

• arxiv-sanity-preserver: the original.
• Malkov & Yashunin, HNSW: the index.
• Malisiewicz et al., Exemplar-SVMs: library against the world.
• OpenAlex: impact score.