Sci-Papers MCP Architecture

Sci-Papers MCP is a scientific paper management service that provides both an MCP tool interface and a REST API for searching, ingesting, and organizing academic papers from multiple sources. It aggregates content from arXiv, Semantic Scholar, OpenAlex, USPTO Patents, YouTube transcripts, and manual PDF uploads into a unified, topic-organized knowledge base with vector search capabilities.

The server runs as a dual-mode FastAPI + FastMCP application: REST endpoints at /api/* serve the web GUI and programmatic clients, while the MCP Streamable HTTP transport at /mcp exposes the same functionality as 14 MCP tools for AI agent consumption.

Key value proposition: AI agents (Claude Code, n8n workflows) can search across all ingested academic papers, patents, and YouTube transcripts using natural language via MCP tools, while human users get a web GUI for browsing and managing the same knowledge base.

Python LOC

3,480

MCP Tools

REST Endpoints

Data Sources

Providers

Test Files

2. Runtime Architecture

Diagram A: Runtime Request Flow

Request Flow

MCP path: Claude Code connects via Streamable HTTP to /mcp and invokes tools (e.g., search_papers, ingest_from_arxiv).
REST path: The web GUI or programmatic clients call /api/* endpoints for the same operations.
Both paths invoke the same core modules: TopicManager for CRUD and SearchEngine for hybrid search.
TopicManager reads/writes topic directories and JSON knowledge bases on the NAS filesystem.
SearchEngine combines bag-of-words cosine similarity (SimpleVectorStore) with sentence-transformer embeddings (Qdrant) for ranked results.

3. Ingestion Architecture

Diagram B: Data Ingestion Flow

Ingestion Pipeline

User triggers ingestion via MCP tool or REST endpoint, specifying a query and topic.
The appropriate Provider (arXiv, S2, OpenAlex, Patents, YouTube) searches the external API.
Papers/transcripts are downloaded; PDFs are extracted via PyMuPDF or pdfplumber.
Text is split into chunks by the Chunker and filenames generated by the Namer.
Chunks are stored in both SimpleVectorStore (BoW, JSON on NAS) and Qdrant (sentence-transformer embeddings).
An IngestionJob tracks progress (queued, downloading, extracting, ingesting, completed/failed).

4. Storage Model

Dual Vector Store Architecture

The system uses two complementary vector stores for search:

Store	Technology	Embedding	Use Case
`SimpleVectorStore`	JSON files on NAS	Bag-of-words (cosine)	Per-topic paper search, zero-dependency fallback
`QdrantStore`	Qdrant v1.13.6 (Docker)	all-MiniLM-L6-v2 (384-dim)	Semantic search, YouTube transcripts, cross-topic queries

NAS Filesystem Layout

/mnt/nas/Audiobooks/Organized/Sci_Papers/
  {topic_name}/
    _topic_meta.json              # display name, description, created_at
    {date}_{source}-{id}_{title}.pdf
    knowledge_vector/
      knowledge_base.json         # full text chunks + metadata
      vstore.json                 # bag-of-words vectors
  MD_Youtube/
    {channel_name}/
      {video_title}.md            # YouTube transcript markdown

Qdrant Collections

Each topic or content source maps to a Qdrant collection. Collections are auto-created on first ingestion. Point IDs are deterministic (MD5 of paper_id:chunk_index) to enable idempotent upserts.

5. Public Interfaces

MCP Tools (14 tools via Streamable HTTP at `/mcp`)

Tool	Description	Key Parameters
`list_topics`	List all paper topics with paper counts and statistics	none
`get_topic_stats`	Get detailed statistics for a specific topic	`topic`
`search_papers`	Search across all topics or a specific topic for papers	`query`, `topic?`, `k?`
`ingest_from_arxiv`	Search arXiv and ingest papers into a topic	`query`, `topic`, `max_results?`, `days_back?`, `field?`
`ingest_from_s2`	Search Semantic Scholar and ingest papers into a topic	`query`, `topic`, `max_results?`
`ingest_from_openalex`	Search OpenAlex (250M+ works, free, no key) and ingest	`query`, `topic`, `max_results?`
`search_patents`	Search USPTO patents and ingest into a topic	`query`, `topic?`, `max_results?`
`upload_paper`	Register a PDF already on disk into a topic's knowledge base	`topic`, `file_path`, `title?`
`reindex_topic`	Re-index existing PDFs in a topic directory into the vector store	`topic`, `source_dir?`
`ingest_youtube`	Ingest YouTube transcript markdown files from a channel directory	`source_dir`, `channel?`
`youtube_download`	Download YouTube subtitles via yt-dlp and ingest into Qdrant	`url`, `topic?`, `lang?`, `max_videos?`
`purge_topic`	Delete all documents from a topic's vector store	`topic`
`get_paper`	Get metadata for a specific paper by ID	`paper_id`

REST API Endpoints (18 endpoints at `/api/*`)

Method	Path	Description
GET	`/health`	Health check with topic/paper/chunk counts
GET	`/`	Web GUI (HTML)
GET	`/api/topics`	List all topics
POST	`/api/topics`	Create a new topic
GET	`/api/topics/{topic}`	Get topic details + papers
DELETE	`/api/topics/{topic}`	Delete an empty topic
GET	`/api/papers`	List papers (optional topic/source filter)
GET	`/api/papers/{id}`	Get paper metadata
GET	`/api/papers/{id}/pdf`	Download paper PDF
GET	`/api/search`	Search papers (hybrid BoW + Qdrant)
POST	`/api/ingest/arxiv`	Ingest from arXiv
POST	`/api/ingest/s2`	Ingest from Semantic Scholar
POST	`/api/ingest/upload`	Upload PDF (multipart)
POST	`/api/ingest/patents`	Ingest from USPTO
POST	`/api/ingest/openalex`	Ingest from OpenAlex
POST	`/api/ingest/reindex`	Re-index existing PDFs
POST	`/api/ingest/youtube`	Ingest YouTube transcripts
POST	`/api/ingest/youtube-download`	Download + ingest YouTube subtitles
GET	`/api/ingest/jobs`	List ingestion jobs
GET	`/api/ingest/jobs/{id}`	Get job status
POST	`/api/purge`	Purge topic vector store
GET	`/api/stats`	Global statistics

6. Repository Structure

Component Classification

Classification	Component	Path	Purpose
Primary	Server	`server.py`	FastAPI + FastMCP dual-mode server, all MCP tools and REST endpoints
Primary	TopicManager	`core/topic_manager.py`	Topic CRUD, paper listing, stats, NAS filesystem operations
Primary	SearchEngine	`core/search.py`	Hybrid search combining SimpleVectorStore and Qdrant results
Secondary	QdrantStore	`core/qdrant_store.py`	Qdrant-backed vector store with sentence-transformer embeddings
Secondary	SimpleVectorStore	`core/vector_store.py`	JSON-file BoW vector store (zero-dependency fallback)
Secondary	ArxivProvider	`providers/arxiv.py`	arXiv API search + PDF download + ingestion (292 LOC)
Secondary	S2Provider	`providers/semantic_scholar.py`	Semantic Scholar API search + ingestion (152 LOC)
Secondary	OpenAlexProvider	`providers/openalex.py`	OpenAlex API search + ingestion (198 LOC)
Secondary	PatentProvider	`providers/patents.py`	USPTO PatentsView API search + ingestion (175 LOC)
Auxiliary	YouTube MD Ingest	`providers/youtube_md.py`	Ingest pre-existing YouTube transcript markdown files
Auxiliary	YouTube Download	`providers/youtube_download.py`	Download subtitles via yt-dlp + save as markdown
Auxiliary	UploadProvider	`providers/upload.py`	Manual PDF upload ingestion
Auxiliary	PDF Extractor	`core/pdf_extract.py`	Text extraction from PDFs (PyMuPDF + pdfplumber)
Auxiliary	Chunker	`core/chunker.py`	Text splitting into chunks
Auxiliary	Namer	`core/namer.py`	Generate standardized filenames for papers
Auxiliary	Reindexer	`core/reindexer.py`	Re-index existing PDFs from disk into vector stores
Auxiliary	Models	`models/`	Pydantic models (PaperMetadata, Topic, IngestionJob, enums)
Auxiliary	Web GUI	`gui/index.html`	Single-page web interface for browsing papers (33K HTML)
External	Qdrant	Docker container	Vector database (qdrant/qdrant:v1.13.6)

Directory Tree

_sci-papers-mcp/
  server.py                     # 704 LOC — FastAPI app + all MCP tools + REST endpoints
  Dockerfile                    # Python 3.12-slim, uvicorn :8000
  docker-compose.yml            # 2 services: sci-papers-mcp + sci-papers-qdrant
  requirements.txt              # 12 dependencies
  README.md
  core/
    topic_manager.py            # 173 LOC — Topic CRUD backed by NAS filesystem
    search.py                   # 126 LOC — Hybrid search engine
    qdrant_store.py             # 166 LOC — Qdrant vector store + sentence-transformers
    vector_store.py             # 102 LOC — SimpleVectorStore (BoW, JSON files)
    pdf_extract.py              #  75 LOC — PDF text extraction
    reindexer.py                # 152 LOC — Re-index existing PDFs
    chunker.py                  #  21 LOC — Text chunking
    namer.py                    #  34 LOC — Filename generation
  providers/
    base.py                     #  20 LOC — AbstractProvider ABC
    arxiv.py                    # 292 LOC — arXiv ingestion
    semantic_scholar.py         # 152 LOC — Semantic Scholar ingestion
    openalex.py                 # 198 LOC — OpenAlex ingestion
    patents.py                  # 175 LOC — USPTO patent ingestion
    upload.py                   # 135 LOC — Manual PDF upload
    youtube_md.py               # 214 LOC — YouTube transcript markdown ingestion
    youtube_download.py         # 222 LOC — yt-dlp subtitle download
    audiobook.py                # 179 LOC — Audiobook provider (planned)
    ebook.py                    # 243 LOC — Ebook provider (planned)
  models/
    enums.py                    #  19 LOC — PaperSource, IngestionStatus
    paper.py                    #  66 LOC — PaperMetadata, Topic, IngestionJob, SearchResult
  gui/
    index.html                  # 33K — Single-page web GUI
  tests/                        # 10 test files (68K total)
  docs/

7. Deployment & Operations

Docker Compose (2 containers)

Container	Image	Port	Network	Volumes
`sci-papers-mcp`	Custom (Python 3.12-slim)	8000	hosting_web	`/mnt/nas/Audiobooks/Organized` → `/data/library`
`sci-papers-qdrant`	qdrant/qdrant:v1.13.6	6333 (internal)	hosting_web	`sci_papers_qdrant` named volume

Environment Variables

Variable	Default	Purpose
`LIBRARY_DIR`	`/data/library`	Base path for topic storage
`QDRANT_URL`	`http://sci-papers-qdrant:6333`	Qdrant connection URL
`S2_API_KEY`	(empty)	Semantic Scholar API key (optional, higher rate limits)
`AUDIOBOOKSHELF_API_KEY`	(empty)	Audiobookshelf API key (planned)
`USPTO_API_KEY`	(empty)	USPTO PatentsView API key (optional)
`ARXIV_BATCH_DELAY`	`3.0`	Delay between arXiv API requests (seconds)

Health Check

Both containers have Docker health checks. The application responds at GET /health with topic count, paper count, and chunk count.

Access

Interface	URL
Web GUI	`https://sci-papers.home/`
Health	`https://sci-papers.home/health`
MCP endpoint	`https://sci-papers.home/mcp`
REST API docs	`https://sci-papers.home/docs`

Operations

# Start
cd /home/rod/_rod/_sci-papers-mcp && docker compose up -d

# Logs
docker logs -f sci-papers-mcp

# Health check
curl -s https://sci-papers.home/health | jq .

# Rebuild after code changes
docker compose build sci-papers-mcp && docker compose up -d sci-papers-mcp

8. Dependencies

Python Dependencies (requirements.txt)

Package	Version	Role	Classification
`fastapi`	>=0.110.0	REST API framework	Primary
`uvicorn[standard]`	>=0.27.0	ASGI server	Primary
`mcp[cli]`	>=1.0.0	MCP protocol SDK (FastMCP)	Primary
`pydantic`	>=2.0.0	Data validation and models	Primary
`httpx`	>=0.27.0	Async HTTP client for external APIs	Primary
`qdrant-client`	>=1.12.0	Qdrant vector DB client	Secondary
`sentence-transformers`	>=3.0.0	Embedding model (all-MiniLM-L6-v2)	Secondary
`PyMuPDF`	>=1.24.0	PDF text extraction (primary)	Secondary
`pdfplumber`	>=0.10.0	PDF text extraction (fallback)	Auxiliary
`python-multipart`	>=0.0.9	File upload support for FastAPI	Auxiliary
`yt-dlp`	>=2024.0.0	YouTube subtitle download	Auxiliary
`pytest`	>=8.0.0	Test framework	Auxiliary

External Services

Service	Auth Required	Rate Limits
arXiv API (export.arxiv.org)	No	3-second delay enforced via `ARXIV_BATCH_DELAY`
Semantic Scholar API	Optional (S2_API_KEY)	100 req/5min without key, higher with key
OpenAlex API	No	Free, polite pool (~10 req/sec)
USPTO PatentsView API	Optional	Standard rate limits
YouTube (via yt-dlp)	No	Standard YouTube rate limits
Qdrant (local container)	No	N/A (local)

9. Constraints & Risks

Item	Severity	Description
MCP tools use synchronous `asyncio.run()`	WARN	MCP tool functions call `asyncio.run()` inside sync functions, which blocks the event loop. Works for single-user but will not scale to concurrent MCP sessions.
SimpleVectorStore uses BoW vectors	WARN	Bag-of-words cosine similarity has limited semantic understanding. Qdrant with sentence-transformers is the preferred search path.
NAS dependency for storage	WARN	All paper storage requires `/mnt/nas` to be mounted. If NAS is offline, ingestion and browsing will fail.
`upload_paper` MCP tool is a stub	INFO	The MCP tool returns `{"status": "stub"}`; the REST upload endpoint works via multipart form.
Ebook and Audiobook providers are planned	INFO	Provider classes exist (179 and 243 LOC) but are not wired into MCP tools yet.
In-memory job tracking	INFO	Ingestion jobs are stored in a Python dict and lost on restart. No persistent job queue.
sentence-transformers cold start	INFO	First embedding request loads the MiniLM model into memory (~90MB). Subsequent requests are fast.

Generated by /repo-architecture v2.0.0 on 2026-03-22. This document is auto-generated — regenerate after significant changes.

Sci-Papers MCP Architecture

2. Runtime Architecture

Request Flow

3. Ingestion Architecture

Ingestion Pipeline

4. Storage Model

Dual Vector Store Architecture

NAS Filesystem Layout

Qdrant Collections

5. Public Interfaces

MCP Tools (14 tools via Streamable HTTP at /mcp)

REST API Endpoints (18 endpoints at /api/*)

6. Repository Structure

Component Classification

Directory Tree

7. Deployment & Operations

Docker Compose (2 containers)

Environment Variables

Health Check

Access

Operations

8. Dependencies

Python Dependencies (requirements.txt)

External Services

9. Constraints & Risks

MCP Tools (14 tools via Streamable HTTP at `/mcp`)

REST API Endpoints (18 endpoints at `/api/*`)