Local arXiv paper manager with semantic search. Crawls arXiv categories, downloads PDFs, chunks content, and indexes with FAISS + Ollama embeddings. No cloud API keys required — everything runs locally.
An arXiv paper crawler with local semantic search (FAISS), topic management, and optional LLM summarization. All embedding is done locally — no cloud APIs required.
Powers the 🔬 ArXiv app in PrivateApp.
python3 scripts/install.py
This will:
faiss-cpu, pdfplumber, arxiv, numpy, tiktoken)qwen3-embedding:0.6b)~/workspace/arxivkb/# Browse available categories
akb topics browse
akb topics browse "machine learning"
# List enabled categories
akb topics list
# Enable categories
akb topics add cs.AI cs.CV cs.RO stat.ML
# Disable a category
akb topics delete cs.AI
# Ingest papers from the last 7 days
akb ingest --days 7
# Dry run (show what would be fetched)
akb ingest --days 3 --dry-run
# Expire old papers
akb expire --days 30
# Semantic search (requires embedding model)
python3 scripts/search.py "transformer attention mechanism" --top 10
# Paper details
akb paper 2310.00001
akb stats
Papers are stored in ~/workspace/arxivkb/:
arxivkb.db — SQLite database (papers, chunks, categories)pdfs/ — Downloaded PDF filesfaiss/ — FAISS vector index filesconfig.json — Per-user configurationBy default, ArXivKB uses qwen3-embedding:0.6b via Ollama. Make sure Ollama is running:
ollama serve
ollama pull qwen3-embedding:0.6b
Alternative models can be configured in ~/workspace/arxivkb/config.json.
The installer schedules daily paper ingestion:
# Linux — systemd timer
systemctl --user status akb-crawler.timer
systemctl --user start akb-crawler.service # run now
# macOS — launchd
launchctl list | grep arxivkb
MIT
Category:science-education