---
name: ocr-and-documents
description: "Extract text from PDFs/scans (pymupdf, marker-pdf)."
version: 2.3.0
author: Hermes Agent
license: MIT
metadata:
  hermes:
    tags: [PDF, Documents, Research, Arxiv, Text-Extraction, OCR]
    related_skills: [powerpoint]
---

# PDF & Document Extraction

For DOCX: use `python-docx` (parses actual document structure, far better than OCR).
For PPTX: see the `powerpoint` skill (uses `python-pptx` with full slide/notes support).
This skill covers **PDFs and scanned documents**.

## Step 1: Remote URL Available?

If the document has a URL, **always try `web_extract` first**:

```
web_extract(urls=["https://arxiv.org/pdf/2402.03300"])
web_extract(urls=["https://example.com/report.pdf"])
```

This handles PDF-to-markdown conversion via Firecrawl with no local dependencies.

Only use local extraction when: the file is local, web_extract fails, or you need batch processing.

## Step 2: Choose Local Extractor

### Recommended Standard Pipeline (updated after scanned-political-report OCR, 2026-05-31)

Use a **layered pipeline** instead of jumping directly to page-image Tesseract. The goal is: avoid OCR when text already exists, preserve layout/tables when possible, and only use heavy OCR/VLM where it materially improves quality.

1. **Preflight: identify PDF type**
   ```bash
   pdfinfo document.pdf
   pdftotext -layout document.pdf /tmp/preflight.txt
   wc -c /tmp/preflight.txt
   ```
   If extracted text is substantial and readable, do **not** OCR the PDF. Use PyMuPDF/pdfplumber/docling on the existing text layer.

2. **Text-based PDFs: PyMuPDF/pdfplumber first**
   - Use **PyMuPDF** for fast full-text extraction, metadata, page counts, splitting/merging, image extraction.
   - Use **pdfplumber** when text layout, columns, or table-like alignment matter.
   - Use **Camelot/Tabula** only when the PDF has real text/vector tables; they are poor for pure scans.

3. **Structured parser: Docling as the main GenAI/RAG parser**
   - Use **Docling** for structured Markdown/JSON, layout, reading order, tables, formulas, and mixed document understanding.
   - For image-only pages, split/render pages and feed Docling page images when the PDF conversion path fails.
   - Docling is the preferred structured output stage, not merely a fallback OCR tool.

4. **Scanned PDFs: OCRmyPDF + Tesseract before ad-hoc page OCR**
   - Use **OCRmyPDF** to create a searchable PDF with a text layer, then rerun PyMuPDF/pdfplumber/Docling on the OCRed PDF.
   - This is usually better than manual `pdftoppm | tesseract` because it preserves page structure and creates a reusable searchable artifact.
   - Manual page rendering + Tesseract remains a fallback when OCRmyPDF is unavailable or fails.

5. **Difficult scanned tables/images: PaddleOCR / PP-Structure as specialist fallback**
   - Use **PaddleOCR/PP-Structure** for difficult scanned tables, forms, screenshots, and image-heavy pages where Tesseract or Docling loses structure.
   - It is heavier to install and maintain, so keep it as an escalation path rather than the default.

6. **Verification**
   - Always report extraction method, page count, text character count, and any failed/low-confidence pages.
   - For political/medical/legal documents, sample-check key pages visually against the extracted text before final analysis.

### Quick Decision Tree

```
Does the PDF already contain real text?
  → YES → PyMuPDF/pdfplumber; Docling for structured Markdown/JSON; Camelot/Tabula for real text tables.
  → NO / image-only → OCRmyPDF + Tesseract to add a text layer, then rerun PyMuPDF/pdfplumber/Docling.
       → If tables/images remain poor → PaddleOCR/PP-Structure or Docling VLM/page-image processing.
```

### Full Comparison (2026-05-10)

| Feature | pymupdf | **docling** | marker-pdf | MarkItDown |
|---------|---------|-------------|------------|------------|
| **Text-based PDF** | ✅ (instant) | ✅ | ✅ | ✅ |
| **Scanned PDF (OCR)** | ❌ | ✅ | ✅ (90+ lang) | ⚠️ basic only |
| **Tables** | ✅ (basic) | ✅ (high accuracy) | ✅ (high accuracy) | ❌ |
| **Equations / LaTeX** | ❌ | ✅ | ✅ (via Texify) | ❌ |
| **Code blocks** | ❌ | ✅ | ✅ | ❌ |
| **Forms** | ❌ | ✅ | ✅ | ❌ |
| **Charts understanding** | ❌ | ✅ (Bar/Pie/Line → tables) | ❌ | ❌ |
| **Headers/footers removal** | ❌ | ✅ | ✅ | ❌ |
| **Reading order detection** | ❌ | ✅ | ✅ | ❌ |
| **Audio transcription** | ❌ | ✅ | ❌ | ✅ |
| **EPUB** | ✅ | ✅ | ✅ | ✅ |
| **Markdown output** | ✅ | ✅ | ✅ | ✅ |
| **LLM hybrid mode** | ❌ | ❌ | ✅ (`--use_llm`) | ❌ |
| **Install size** | ~25MB | ~1-2GB | ~3-5GB (PyTorch+models) | ~50MB |
| **License** | MIT | **MIT** | GPL-3.0 | MIT |
| **Speed** | Instant | ~0.5-2s/page | ~1-14s/page (CPU) | Instant |
| **Stars** | 9.6K | **59.5K** | 34.9K | **122K** |

**Decision:** Use **docling** as the default for OCR/advanced extraction — MIT license, IBM-backed, full-featured. Use pymupdf for quick text-only PDFs. Use marker only when `--use_llm` boost is needed for maximum accuracy.

### Why Docling > Marker (2026-05-10 evaluation)

See `references/pdf-ocr-comparison.md` for full benchmark analysis.

**Docling wins because:**
- MIT license (vs GPL-3.0 for Marker) — no commercial restrictions
- IBM-backed, LF AI & Data Foundation — better long-term sustainability
- Chart understanding (Bar/Pie/Line → tables) — Marker lacks this
- MCP server for agent integration
- VLM support (GraniteDocling and others)
- Audio transcription built-in
- Same accuracy as Marker for tables and layout

**Marker wins because:**
- LLM hybrid mode (`--use_llm`) — 0.907 table score with Gemini
- Slightly higher heuristic scores (95.7% vs ~86.7% for Docling)
- Texify integration — better equation recognition
- Multi-GPU chunk conversion for batch processing

**MarkItDown** (Microsoft) has 122K stars but is NOT suitable for OCR/scanned documents — basic EXIF-OCR only, no table structure, no equations. Good for office documents with digital text.

**For medical/laboratory PDFs with complex tables** (Blutbilder, Laborberichte): Use **docling** (preferred) or **camelot** (`pip install camelot-py[cv]`) — both extract table structures with high accuracy.

**For scanned medical documents**: Use **docling** for OCR + layout analysis; combine with **pytesseract** for pure image-based PDFs if docling fails.

If the user needs advanced OCR but the system lacks disk space:
> "This document needs OCR/advanced extraction (docling or marker-pdf), which requires ~1-5GB for models. Your system has [X]GB free. Options: free up space, provide a URL so I can use web_extract, or I can try pymupdf which works for text-based PDFs but not scanned documents or equations."

---

## Tesseract OCR (for scanned PDFs)

### Large scanned PDFs / political dossiers

When a PDF is image-only and `pdftotext` returns almost no text, do **not** assume it was read. Verify page count and extracted character count first. For large scanned reports, a practical workflow is:

1. `pdfinfo report.pdf` to record page count and metadata.
2. `pdftotext -layout report.pdf report.txt`; if output is empty or only form-feeds, treat it as scanned.
3. Render pages with `pdftoppm` at a moderate DPI first (`-r 120` or `-r 80`) instead of immediately using 300 DPI. This is often sufficient for political reports and much faster.
4. OCR key sections first from the table of contents: summary, findings, recommendations, outlook/next steps, and politically sensitive sections (e.g. traffic, costs, climate, governance). Then OCR all remaining pages only if the task truly needs exhaustive coverage.
5. Use `tesseract page.png out -l deu+eng --psm 4 --dpi 120` for report pages with mixed headings/paragraphs; fall back to `--psm 11` for sparse/graphic-heavy pages. Wrap each page OCR in a timeout and continue, rather than letting one graphic-heavy page block the whole job.
6. Combine page texts with explicit `--- SEITE N ---` separators so later analysis can cite or audit where claims came from.

This pattern is preferable to full high-DPI OCR when the deliverable is a political/municipal briefing: it gives fast grounded coverage of the decision-relevant pages while preserving the option to complete OCR afterwards.

```bash
# Check available languages
tesseract --list-langs

# Basic OCR
tesseract input.png output -l deu+eng

# For mixed text+tables (laboratory reports):
tesseract input.png output -l deu+eng --psm 6 --oem 3
# PSM=6: Assume a single uniform block of text (BEST for tabular data)
# PSM=4: Single column (good for structured forms)
# PSM=11: Sparse text (words as separate lines — good for extracting individual values)
# PSM=13: Raw line (fastest, lowest accuracy)

# For scanned PDFs: first convert to PNG, then OCR
# Using PyMuPDF: convert page to image at 300 DPI for best OCR quality
```

**Pitfall:** Tesseract OCR on tabular PDFs often produces fragmented output. Use PSM=6 for mixed content, PSM=11 when you need individual values as separate tokens. For medical lab reports with obscured values in the PDF itself, OCR may not recover hidden data.

**Python environment note:** PyMuPDF (fitz) and pytesseract are installed in `~/.local/lib/python3.12/site-packages`, NOT in the hermes-agent venv. Always set `PYTHONPATH=~/.local/lib/python3.12/site-packages` when running Python scripts that import these packages.

## camelot (PDF Table Extraction — Elite)

```bash
pip install camelot-py[cv]
```

**Required:** Ghostscript installed (`apt install ghostscript`) — camelot depends on it.

```python
import camelot
tables = camelot.read_pdf('laborbericht.pdf', flavor='lattice')  # Grid-based tables
# OR
tables = camelot.read_pdf('laborbericht.pdf', flavor='stream')    # Space-separated tables

# Extract to DataFrame
df = tables[0].df  # First table as DataFrame
df.to_csv('output.csv')  # Save to CSV
```

**Decision:** Use camelot for laboratory reports, medical charts, and any PDF with structured tabular data. It preserves column alignment and cell structure far better than pymupdf.

**Pitfall:** camelot requires Ghostscript. If `gs` is not found, install with `apt install ghostscript`. Without it, camelot will silently fail or raise `GhostscriptNotFoundError`.

**Pitfall:** For scanned PDFs, camelot may not work — use PyMuPDF + pytesseract for OCR instead.

---

## pymupdf (lightweight)

```bash
pip install pymupdf pymupdf4llm
```

**Via helper script**:
```bash
python scripts/extract_pymupdf.py document.pdf              # Plain text
python scripts/extract_pymupdf.py document.pdf --markdown    # Markdown
python scripts/extract_pymupdf.py document.pdf --tables      # Tables
python scripts/extract_pymupdf.py document.pdf --images out/ # Extract images
python scripts/extract_pymupdf.py document.pdf --metadata    # Title, author, pages
python scripts/extract_pymupdf.py document.pdf --pages 0-4   # Specific pages
```

**Inline**:
```bash
python3 -c "
import pymupdf
doc = pymupdf.open('document.pdf')
for page in doc:
    print(page.get_text())
"
```

---

## Multi-Page PDFs with Mixed Content (Text + Scanned Pages)

**Problem:** Docling extracts text-based pages but skips image-only pages (scanned pages). Page 2 of a PDF may be entirely images.

**Solution:** Split the PDF into individual pages, then process image-only pages as images with Docling (which runs OCR on images):

```python
import fitz  # PyMuPDF

pdf_path = "document.pdf"
doc = fitz.open(pdf_path)

for page_idx in range(len(doc)):
    # Extract page as image at 300 DPI for best OCR
    page = doc[page_idx]
    pix = page.get_pixmap(dpi=300)
    pix.save(f"/tmp/page_{page_idx+1}.png")
    doc.close()

# Process each page image with Docling
from docling.document_converter import DocumentConverter
converter = DocumentConverter()
result = converter.convert("/tmp/page_2.png")
print(result.document.export_to_markdown())
```

**Pitfall:** Always check `len(doc.pages)` first to know how many pages to process. Docling's `convert()` on a multi-page PDF may only process the first text page.

**Pitfall:** Use `pip install pymupdf` (NOT `PyMuPDF` — the lowercase package name). Install in the hermes-agent venv: `cd /home/agent/.hermes/hermes-agent && uv pip install pymupdf`.

**Pitfall:** Docling on images triggers OCR automatically. No separate OCR model needed.

## docling (PDF-to-Markdown — Preferred)

**Installation** (hermes-agent venv):
```bash
cd /home/agent/.hermes/hermes-agent && uv pip install docling
```
⚠️ **Do NOT use `pip install docling`** — the hermes-agent venv uses `uv` for package management. Regular pip installs to system Python which is externally managed.

**CLI:**
```bash
docling document.pdf                          # Markdown output
docling document.pdf --pipeline vlm           # VLM mode (GraniteDocling)
docling https://example.com/file.pdf          # URL support
docling --help                                # All options
```
⚠️ **First run downloads models** (~1-2GB to `~/.cache/huggingface/`) — `docling --help` may timeout on first invocation. Wait for model download to complete before running CLI commands.

**Python API:**
```python
import sys
sys.path.insert(0, "/home/agent/.hermes/hermes-agent/venv/lib/python3.11/site-packages")

from docling.document_converter import DocumentConverter

converter = DocumentConverter()
result = converter.convert("document.pdf")
print(result.document.export_to_markdown())
```

**Features:**
- MIT license (vs GPL-3.0 for Marker)
- IBM-backed, LF AI & Data Foundation
- OCR, tables, equations, code, charts, forms
- VLM support (GraniteDocling, 258M params)
- MCP server for agent integration
- Audio transcription (ASR)
- Export: Markdown, HTML, DocTags, lossless JSON
- Chart understanding: Bar/Pie/Line → tables
- 59.5K stars, very active (last push: 2 days ago)

**Decision:** Use docling as the default for OCR/advanced extraction. Install size ~1-2GB.

**Pitfall:** Docling extracts tables as Markdown tables. Use the `references/docling-markdown-extraction.md` guide for parsing and converting to structured formats (Excel, CSV, etc.).

---

## MarkItDown (Microsoft — basic conversion only)

```bash
pip install 'markitdown[all]'
```

**CLI:**
```bash
markitdown document.pdf > output.md
```

**Limitations:** No real OCR, no equations, no table structure. Only suitable for office documents with digital text. NOT suitable for scanned documents or medical PDFs. 122K stars but basic quality.

---

## marker-pdf (high-quality OCR — LLM boost)

See `references/marker-pdf-evaluation.md` for full benchmarks, LLM service options, and pitfall notes.

```bash
# Check disk space first
python scripts/extract_marker.py --check

pip install marker-pdf[full]
```

**Via helper script:**
```bash
python scripts/extract_marker.py document.pdf                # Markdown
python scripts/extract_marker.py document.pdf --json         # JSON with metadata
python scripts/extract_marker.py document.pdf --output_dir out/  # Save images
python scripts/extract_marker.py scanned.pdf                 # Scanned PDF (OCR)
python scripts/extract_marker.py document.pdf --use_llm      # LLM-boosted accuracy
```

**CLI** (installed with marker-pdf):
```bash
marker_single document.pdf --output_dir ./output            # Single file
marker /path/to/folder --workers 4                          # Batch
marker_single document.pdf --force_ocr                      # Force OCR
marker_single document.pdf --page_range "0,5-10"            # Specific pages
marker_single document.pdf --use_llm --llm_service marker.services.ollama.OllamaService  # Local LLM
```

**Decision:** Use marker only when `--use_llm` boost is needed for maximum accuracy. GPL-3.0 license OK for internal use.

---

## Arxiv Papers

```
# Abstract only (fast)
web_extract(urls=["https://arxiv.org/abs/2402.03300"])

# Full paper
web_extract(urls=["https://arxiv.org/pdf/2402.03300"])

# Search
web_search(query="arxiv GRPO reinforcement learning 2026")
```

## Split, Merge & Search

pymupdf handles these natively — use `execute_code` or inline Python:

```python
# Split: extract pages 1-5 to a new PDF
import pymupdf
doc = pymupdf.open("report.pdf")
new = pymupdf.open()
for i in range(5):
    new.insert_pdf(doc, from_page=i, to_page=i)
new.save("pages_1-5.pdf")
```

```python
# Merge multiple PDFs
import pymupdf
result = pymupdf.open()
for path in ["a.pdf", "b.pdf", "c.pdf"]:
    result.insert_pdf(pymupdf.open(path))
result.save("merged.pdf")
```

```python
# Search for text across all pages
import pymupdf
doc = pymupdf.open("report.pdf")
for i, page in enumerate(doc):
    results = page.search_for("revenue")
    if results:
        print(f"Page {i+1}: {len(results)} match(es)")
        print(page.get_text("text"))
```

No extra dependencies needed — pymupdf covers split, merge, search, and text extraction in one package.

---

## Notes

- `web_extract` is always first choice for URLs
- **Multi-page PDFs:** Docling may only extract the first text page. Split with PyMuPDF and process image pages separately as images for OCR. See multi-page workflow above.
- **Image-only pages:** Pages that are scanned images (no selectable text) require conversion to PNG + Docling for OCR. Docling on images = automatic OCR.
- pymupdf is the safe default for text-only PDFs — instant, no models
- marker-pdf is for maximum accuracy with `--use_llm` boost — GPL-3.0 license
- MarkItDown: basic only, NOT suitable for scanned documents
- Both helper scripts accept `--help` for full usage
- marker-pdf downloads ~2.5GB of models to `~/.cache/huggingface/` on first use
- docling downloads ~1-2GB of models on first use
- **Docling Evaluation (2026-05-10):** ✅ Successfully tested with text-based and table-containing PDFs. Extracts text, tables, and preserves formatting correctly. Install via `uv pip install docling` in hermes-agent venv. First run downloads models (~1-2GB).
- **For medical/laboratory PDFs:** **docling** (preferred) or **camelot** for table extraction
- For Word docs: `pip install python-docx` (better than OCR — parses actual structure)
- For PowerPoint: see the `powerpoint` skill (uses python-pptx)
- For PDF tables: `camelot-py[cv]` requires Ghostscript (`apt install ghostscript`)
- **Health data processing:** Medical documents stored at `/home/agent/.hermes/assets/Gesundheit/` with master DB at `health_data.db`. See `health-data-management` skill for the full workflow. Only 9/59 documents have searchable extracted text as of 2026-05-12 — Docling reprocessing recommended.

## References

- `references/tesseract-lab-report-ocr.md` — Session-specific OCR patterns for medical lab reports (PSM modes, Python env, obscured values)
- `references/marker-pdf-evaluation.md` — Marker (datalab-to/marker) evaluation: benchmarks, LLM services, CLI usage, pitfalls, licensing
- `references/pdf-ocr-comparison.md` — Full comparison: docling vs marker vs MarkItDown vs pymupdf (benchmarks, licensing, decision matrix)
- `references/docling-markdown-extraction.md` — Pattern for parsing Docling Markdown tables and converting to Excel/CSV
- `references/multi-page-pdf-ocr-pattern.md` — Split PDF + render pages as PNG + Docling OCR for mixed text/image PDFs