Extract structured data from scientific literature across multiple formats (PDF, HTML, images, plain text). Auto-detects scientific domain to recommend specialized tools for chemistry/materials when appropriate. Use this skill when: extracting numerical data from papers, digitizing graphs/plots, parsing tables from PDFs, extracting chemical properties or reactions, or converting unstructured scientific text to structured formats. Key capabilities: format detection and routing, domain-specific extraction (chemistry/materials), multi-method validation, table extraction, graph digitization, LLM-enhanced extraction with verification, confidence scoring.
A comprehensive guidance skill for extracting structured data from scientific literature across multiple formats (PDF, HTML, images, plain text). The skill auto-detects scientific domains to recommend specialized tools for chemistry and materials science, and provides a hierarchical extraction approach with multi-method validation.
# Quick extraction with PyMuPDF4LLM
import pymupdf4llm
text = pymupdf4llm.to_markdown("paper.pdf")
# Standard extraction with GROBID
import scipdf_parser
article = scipdf_parser.parse_pdf_to_dict("paper.pdf")
# Table extraction with Camelot
import camelot
tables = camelot.read_pdf("paper.pdf", flavor='lattice')
df = tables[0].df
# Property extraction with ChemDataExtractor
from chemdataextractor import Document
doc = Document.from_file("paper.pdf")
for record in doc.records:
print(record.serialize())
# Reaction extraction with OpenChemIE
from openchemie import OpenChemIE
model = OpenChemIE()
reactions = model.extract_reactions_from_text(text)
| Table Type | Tool | Code |
|------------|------|------|
| Bordered | Camelot | camelot.read_pdf(f, flavor='lattice') |
| Borderless | Camelot | camelot.read_pdf(f, flavor='stream') |
| Complex | pdfplumber | page.extract_table(table_settings) |
| General | Tabula | tabula.read_pdf(f) |
# Basic PDF and text processing
pip install pymupdf4llm pdfplumber beautifulsoup4 lxml spacy pandas
# Table extraction
pip install camelot-py[cv] tabula-py
# Install spaCy model
python -m spacy download en_core_web_sm
# ChemDataExtractor v2
pip install chemdataextractor2
# OpenChemIE
pip install openchemie
# GROBID client (requires running GROBID server)
pip install scipdf_parser
# IBM Docling
pip install docling
# Marker-PDF (OCR-capable)
pip install marker-pdf
# OCR
pip install pytesseract # requires tesseract system install
# or
pip install surya-ocr # ML-based OCR
# macOS
brew install ghostscript tesseract poppler openjdk
# Ubuntu/Debian
sudo apt-get install ghostscript tesseract-ocr poppler-utils default-jdk
# For GROBID server (optional)
docker pull lfoppiano/grobid:0.8.0
docker run -t --rm -p 8070:8070 lfoppiano/grobid:0.8.0
| Tool | Speed | Structure | Tables | Figures | OCR | Best For | |------|-------|-----------|--------|---------|-----|----------| | PyMuPDF4LLM | Very Fast | Basic | Limited | No | Optional | Quick screening, RAG | | GROBID | Medium | Excellent | Good | Metadata | No | Academic papers, references | | Docling | Medium | Good | Excellent | Basic | Optional | Complex layouts | | Marker-PDF | Slow | Good | Good | Good | Yes | Scanned documents | | pdfplumber | Fast | Detailed | Excellent | No | No | Table-heavy documents |
| Tool | Bordered | Borderless | Complex | Speed | Accuracy | |------|----------|------------|---------|-------|----------| | Camelot (lattice) | Excellent | Poor | Fair | Fast | High | | Camelot (stream) | Good | Good | Fair | Fast | Medium | | Tabula | Good | Good | Fair | Fast | Medium | | pdfplumber | Excellent | Fair | Excellent | Medium | High | | Table Transformer | Good | Good | Excellent | Slow | High |
| Tool | Properties | Reactions | Entities | Tables | Figures | |------|------------|-----------|----------|--------|---------| | ChemDataExtractor v2 | Excellent | Fair | Excellent | Good | No | | OpenChemIE | Good | Excellent | Good | Good | Yes | | LLM (GPT-4/Claude) | Good | Good | Good | Fair | Good |
For high-confidence extraction, use multiple methods:
def calculate_confidence(primary_result, secondary_result, llm_verified, db_match):
"""Calculate extraction confidence score."""
score = 0.0
# Base score from primary extraction
score = 0.5
# Agreement between methods
if primary_result == secondary_result:
score += 0.25
# LLM verification
if llm_verified:
score += 0.15
# Database match
if db_match:
score += 0.10
# Classify confidence level
if score >= 0.9:
level = "HIGH"
elif score >= 0.7:
level = "MEDIUM"
elif score >= 0.5:
level = "LOW"
else:
level = "REVIEW"
return {"score": score, "level": level}
scientific-data-extraction/
├── SKILL.md # Main skill file
├── README.md # This documentation
├── QUICK_REFERENCE.md # Quick lookup guide
├── references/
│ ├── pdf-tools.md # PDF tool comparison and usage
│ ├── table-extraction.md # Table extraction methods
│ ├── graph-digitization.md # Graph data extraction
│ ├── chemistry-tools.md # ChemDataExtractor, OpenChemIE
│ └── llm-extraction.md # LLM-based extraction patterns
└── examples/
├── extract-from-pdf.md # Complete PDF workflow
├── extract-table-data.md # Table extraction examples
├── digitize-graph.md # Graph digitization guide
└── chemistry-extraction.md # Chemistry-specific workflow
Swain & Cole (2016). "ChemDataExtractor: A Toolkit for Automated Extraction of Chemical Information from the Scientific Literature." J. Chem. Inf. Model. DOI: 10.1021/acs.jcim.6b00207
Mavracic et al. (2021). "ChemDataExtractor 2.0: Autopopulated Ontologies for Materials Science." J. Chem. Inf. Model. DOI: 10.1021/acs.jcim.1c00446
Fan et al. (2024). "OpenChemIE: An Information Extraction Toolkit for Chemistry Literature." J. Chem. Inf. Model. DOI: 10.1021/acs.jcim.4c00572
Zheng et al. (2024). "ChatExtract: Accurate Data Extraction from Scientific Publications via Large Language Models." arXiv:2308.02214
To improve this skill:
references/examples/This skill documentation is provided under MIT License. Individual tools have their own licenses:
Category:science-education