1 line
23 KiB
JSON
1 line
23 KiB
JSON
{"repo": "allenai/olmocr", "files": {"README.md": {"f7cfe4c22098b154c76b6ec950d1c0a464eecf8d": {"t": 1785674593.7942934, "n": 22311, "h": "5cd5ac0de8d5b15c", "c": "<div align=\"center\">\n <img width=\"350\" alt=\"olmocr-2-full@2x\" src=\"https://github.com/user-attachments/assets/24f1b596-4059-46f1-8130-5d72dcc0b02e\" />\n<hr/>\n</div>\n<p align=\"center\">\n <a href=\"https://github.com/allenai/OLMo/blob/main/LICENSE\">\n <img alt=\"GitHub License\" src=\"https://img.shields.io/github/license/allenai/OLMo\">\n </a>\n <a href=\"https://github.com/allenai/olmocr/releases\">\n <img alt=\"GitHub release\" src=\"https://img.shields.io/github/release/allenai/olmocr.svg\">\n </a>\n <a href=\"https://arxiv.org/abs/2502.18443\">\n <img alt=\"Tech Report v1\" src=\"https://img.shields.io/badge/Paper_v1-olmOCR-blue\">\n </a>\n <a href=\"https://arxiv.org/abs/2510.19817\">\n <img alt=\"Tech Report v2\" src=\"https://img.shields.io/badge/Paper_v2-olmOCR-blue\">\n </a>\n <a href=\"https://olmocr.allenai.org\">\n <img alt=\"Demo\" src=\"https://img.shields.io/badge/Ai2-Demo-F0529C\">\n </a>\n <a href=\"https://discord.gg/sZq3jTNVNG\">\n <img alt=\"Discord\" src=\"https://img.shields.io/badge/Discord%20-%20blue?style=flat&logo=discord&label=Ai2&color=%235B65E9\">\n </a>\n</p>\n\nA toolkit for converting PDFs and other image-based document formats into clean, readable, plain text format.\n\nTry the online demo: [https://olmocr.allenai.org/](https://olmocr.allenai.org/)\n\nFeatures:\n - Convert PDF, PNG, and JPEG based documents into clean Markdown\n - Support for equations, tables, handwriting, and complex formatting\n - Automatically removes headers and footers\n - Convert into text with a natural reading order, even in the presence of\n figures, multi-column layouts, and insets\n - Efficient, less than $200 USD per million pages converted\n - (Based on a 7B parameter VLM, so it requires a GPU)\n\n### News\n - October 21, 2025 - v0.4.0 - [New model release](https://huggingface.co/allenai/olmOCR-2-7B-1025-FP8), boosts olmOCR-bench score by ~4 points using synthetic data and introduces RL training.\n - August 13, 2025 - v0.3.0 - [New model release](https://huggingface.co/allenai/olmOCR-7B-0825-FP8), fixes auto-rotation detection, and hallucinations on blank documents.\n - July 24, 2025 - v0.2.1 - [New model release](https://huggingface.co/allenai/olmOCR-7B-0725-FP8), scores 3 points higher on [olmOCR-Bench](https://github.com/allenai/olmocr/tree/main/olmocr/bench), also runs significantly faster because it's default FP8, and needs much fewer retries per document.\n - July 23, 2025 - v0.2.0 - New cleaned up [trainer code](https://github.com/allenai/olmocr/tree/main/olmocr/train), makes it much simpler to train olmOCR models yourself.\n - June 17, 2025 - v0.1.75 - Switch from sglang to vllm based inference pipeline, updated docker image to CUDA 12.8.\n - May 23, 2025 - v0.1.70 - Official docker support and images are now available! [See Docker usage](#using-docker)\n - May 19, 2025 - v0.1.68 - [olmOCR-Bench](https://github.com/allenai/olmocr/tree/main/olmocr/bench) launch, scoring 77.4. Launch includes 2 point performance boost in olmOCR pipeline due to bug fixes with prompts.\n - Mar 17, 2025 - v0.1.60 - Performance improvements due to better temperature selection in sampling.\n - Feb 25, 2025 - v0.1.58 - Initial public launch and demo.\n\n### Benchmark\n\n[**olmOCR-Bench**](https://github.com/allenai/olmocr/tree/main/olmocr/bench):\nWe also ship a comprehensive benchmark suite covering over 7,000 test cases across 1,400 documents to help measure performance of OCR systems. \n\n<table>\n <thead>\n <tr>\n <th></th>\n <th>ArXiv</th>\n <th>Old<br>scans<br>math</th>\n <th>Tables</th>\n <th>Old<br>scans</th>\n <th>Headers<br>&<br>footers</th>\n <th>Multi<br>column</th>\n <th>Long<br>tiny<br>text</th>\n <th>Base</th>\n <th>Overall</th>\n </tr>\n </thead>\n <tbody>\n <tr>\n <td>Mistral OCR API</td>\n <td>77.2</td>\n <td>67.5</td>\n <td>60.6</td>\n <td>29.3</td>\n <td>93.6</td>\n <td>71.3</td>\n <td>77.1</td>\n <td>99.4</td>\n <td>72.0±1.1</td>\n </tr>\n <tr>\n <td>Marker 1.10.1</td>\n <td>83.8</td>\n <td>66.8</td>\n <td>72.9</td>\n <td>33.5</td>\n <td>86.6</td>\n <td>80.0</td>\n <td>85.7</td>\n <td>99.3</td>\n <td>76.1±1.1</td>\n </tr>\n <tr>\n <td>MinerU 2.5.4*</td>\n <td>76.6</td>\n <td>54.6</td>\n <td>84.9</td>\n <td>33.7</td>\n <td>96.6</td>\n <td>78.2</td>\n <td>83.5</td>\n <td>93.7</td>\n <td>75.2±1.1</td>\n </tr>\n <tr>\n <td>DeepSeek-OCR</td>\n <td>77.2</td>\n <td>73.6</td>\n <td>80.2</td>\n <td>33.3</td>\n <td>96.1</td>\n <td>66.4</td>\n <td>79.4</td>\n <td>99.8</td>\n <td>75.7±1.0</td>\n </tr>\n <tr>\n <td>Nanonets-OCR2-3B</td>\n <td>75.4</td>\n <td>46.1</td>\n <td>86.8</td>\n <td>40.9</td>\n <td>32.1</td>\n <td>81.9</td>\n <td>93.0</td>\n <td>99.6</td>\n <td>69.5±1.1</td>\n </tr>\n <tr>\n <td>PaddleOCR-VL*</td>\n <td>85.7</td>\n <td>71.0</td>\n <td>84.1</td>\n <td>37.8</td>\n <td>97.0</td>\n <td>79.9</td>\n <td>85.7</td>\n <td>98.5</td>\n <td>80.0±1.0</td>\n </tr>\n <tr>\n <td>Infinity-Parser 7B*</td>\n <td>84.4</td>\n <td>83.8</td>\n <td>85.0</td>\n <td>47.9</td>\n <td>88.7</td>\n <td>84.2</td>\n <td>86.4</td>\n <td>99.8</td>\n <td>82.5±?</td>\n </tr>\n <tr>\n <td>Chandra OCR 0.1.0*</td>\n <td>82.2</td>\n <td>80.3</td>\n <td>88.0</td>\n <td>50.4</td>\n <td>90.8</td>\n <td>81.2</td>\n <td>92.3</td>\n <td>99.9</td>\n <td>83.1±0.9</td>\n </tr>\n <tr>\n <td colspan=\"10\"><hr></td>\n </tr>\n <tr>\n <td><strong>olmOCR v0.4.0</strong></td>\n <td>83.0</td>\n <td>82.3</td>\n <td>84.9</td>\n <td>47.7</td>\n <td>96.1</td>\n <td>83.7</td>\n <td>81.9</td>\n <td>99.7</td>\n <td>82.4±1.1</td>\n </tr>\n </tbody>\n</table>\n\n\n### Installation\n\n#### System Dependencies\n\nYou will need to install poppler-utils and additional fonts for rendering PDF images.\n\nInstall dependencies (Ubuntu/Debian):\n```bash\nsudo apt-get update\nsudo apt-get install poppler-utils ttf-mscorefonts-installer msttcorefonts fonts-crosextra-caladea fonts-crosextra-carlito gsfonts lcdf-typetools\n```\n\n#### Python Installation\n\nSet up a conda environment and install olmocr. The requirements for running olmOCR\nare difficult to install in an existing python environment, so please do make a clean python environment to install into.\n\n```bash\nconda create -n olmocr python=3.11\nconda activate olmocr\n```\n\nChoose the installation option that matches your use case:\n\n**Option 1: Remote Inference (Lightweight)**\n\nIf you plan to use a remote vLLM server with the `--server` flag, install the base package:\n```bash\npip install olmocr\n```\nThis avoids installing heavy GPU dependencies like PyTorch (~2GB+).\n\n**Option 2: Local GPU Inference**\n\nRequirements:\n - Recent NVIDIA GPU (tested on RTX 4090, L40S, A100, H100) with at least 12 GB of GPU RAM\n - 30GB of free disk space\n\nFor running inference with your own GPU:\n```bash\npip install olmocr[gpu] --extra-index-url https://download.pytorch.org/whl/cu128\n\n# Recommended: Install flash infer for faster inference on GPU\npip install https://download.pytorch.org/whl/cu128/flashinfer/flashinfer_python-0.2.5%2Bcu128torch2.7-cp38-abi3-linux_x86_64.whl\n```\n\n**Option 3: Beaker Cluster Execution**\n\nFor submitting jobs to Beaker clusters with the `--beaker` flag:\n```bash\npip install olmocr[beaker]\n```\n\n**Option 4: Benchmark Suite**\n\nFor running the olmOCR benchmark suite:\n```bash\npip install olmocr[bench]\n```\n\n**Combined Installation**\n\nYou can combine multiple options:\n```bash\n# GPU + Beaker support\npip install olmocr[gpu,beaker] --extra-index-url https://download.pytorch.org/whl/cu128\n\n# GPU + Benchmark support\npip install olmocr[gpu,bench] --extra-index-url https://download.pytorch.org/whl/cu128\n```\n\n**Troubleshooting**\n\nIf you run into errors about `too many open files`, update your ulimit:\n```bash\nulimit -n 65536\n```\n\n### Usage Examples\n\nFor quick testing, try the [web demo](https://olmocr.allen.ai/).\n\n**Convert a Single PDF (Local GPU):**\n```bash\n# Download a sample PDF\ncurl -o olmocr-sample.pdf https://olmocr.allenai.org/papers/olmocr_3pg_sample.pdf\n\n# Convert it to markdown\nolmocr ./localworkspace --markdown --pdfs olmocr-sample.pdf\n```\n\n**Convert an Image file:**\n```bash\nolmocr ./localworkspace --markdown --pdfs random_page.png\n```\n\n**Convert Multiple PDFs:**\n```bash\nolmocr ./localworkspace --markdown --pdfs tests/gnarly_pdfs/*.pdf\n```\n\n**Use Remote Inference Server:**\n```bash\nolmocr ./localworkspace --server http://remote-server:8000/v1 --model allenai/olmOCR-2-7B-1025-FP8 --markdown --pdfs *.pdf\n```\n\nWith the `--markdown` flag, results will be stored as markdown files inside of `./localworkspace/markdown/`.\n\n> **Note:** You can also use `python -m olmocr.pipeline` instead of `olmocr` if you prefer. \n\n#### Viewing Results\n\nThe `./localworkspace/` workspace folder will then have both [Dolma](https://github.com/allenai/dolma) and markdown files (if using `--markdown`).\n\n\n```bash\ncat localworkspace/markdown/olmocr-sample.md \n```\n\n```\nolmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models\n...\n```\n\n### Using an Inference Provider or External Server\n\nIf you have a vLLM server already running elsewhere (or any inference platform implementing the OpenAI API), you can point olmOCR to use it instead of spawning a local instance.\n\n**Installation for Remote Inference:**\n```bash\n# Lightweight installation - no GPU dependencies needed\npip install olmocr\n```\n\n**Using an External Server:**\n```bash\n# Use external vLLM server instead of local one\nolmocr ./localworkspace --server http://remote-server:8000/v1 --model allenai/olmOCR-2-7B-1025-FP8 --markdown --pdfs tests/gnarly_pdfs/*.pdf\n```\n\nThe served model name in vLLM needs to match the value provided in `--model`.\n\n**Example vLLM Server Launch:**\n```bash\nvllm serve allenai/olmOCR-2-7B-1025-FP8 --max-model-len 16384\n```\n\n#### Verified External Providers\n\nWe have tested `olmOCR-2-7B-1025-FP8` on these external model providers and confirmed that they work\n\n| | $/1M Input tokens | $/1M Output tokens | Example Command |\n|-----------------------------------------------------------------------------|-------------------|--------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| [Cirrascale](https://ai2endpoints.cirrascale.ai/models/overview) | $0.07 | $0.15 | `olmocr ./workspace --server https://ai2endpoints.cirrascale.ai/api --api_key sk-XXXXXXX --workers 1 --max_concurrent_requests 20 --model olmOCR-2-7B-1025 --pdfs tests/gnarly_pdfs/*.pdf` |\n| [DeepInfra](https://deepinfra.com/) | $0.09 | $0.19 | `olmocr ./workspace --server https://api.deepinfra.com/v1/openai --api_key DfXXXXXXX --workers 1 --max_concurrent_requests 20 --model allenai/olmOCR-2-7B-1025 --pdfs tests/gnarly_pdfs/*.pdf` |\n| [Parasail](https://www.saas.parasail.io/serverless?name=olmocr-7b-1025-fp8) | $0.10 | $0.20 | `olmocr ./workspace --server https://api.parasail.io/v1 --api_key psk-XXXXX --workers 1 --max_concurrent_requests 20 --model allenai/olmOCR-2-7B-1025 --pdfs tests/gnarly_pdfs/*.pdf` |\n\n\nNotes on arguments\n- `--server`: Defines the OpenAI-compatible endpoint: ex `https://api.deepinfra.com/v1/openai`\n- `--api_key`: Your API key, bassed in via Authorization Bearer HTTP header\n- `--max_concurrent_requests`: Max concurrent requests that will be in-flight to the inference provider at one time\n- `--workers`: Max number of page groups that will be processed at once. You may want to set this to `1` so that you finish one group of stuff before moving on.\n- `--pages_per_group`: You may want a smaller number of pages per group as many external provides have lower concurrent request limits\n- `--model`: The model identifier, ex. `allenai/olmOCR-2-7B-1025`, different providers have different names, and if you run locally, you can use `olmocr`\n- Other arguments work the same as with local inference\n\n\n### Multi-node / Cluster Usage\n\nIf you want to convert millions of PDFs using multiple nodes running in parallel, olmOCR supports\nreading PDFs from AWS S3 and coordinating work using an AWS S3 output bucket.\n\n**Start the first worker node:**\n```bash\nolmocr s3://my_s3_bucket/pdfworkspaces/exampleworkspace --pdfs s3://my_s3_bucket/jakep/gnarly_pdfs/*.pdf\n```\n\nThis sets up a simple work queue in your AWS bucket and starts converting PDFs.\n\n**On subsequent worker nodes:**\n```bash\nolmocr s3://my_s3_bucket/pdfworkspaces/exampleworkspace\n```\n\nThey will automatically start grabbing items from the same workspace queue.\n\n#### Using Beaker for Cluster Execution\n\nIf you are at Ai2 and want to linearize millions of PDFs efficiently using [beaker](https://www.beaker.org), install with Beaker support:\n\n```bash\npip install olmocr[gpu,beaker] --extra-index-url https://download.pytorch.org/whl/cu128\n```\n\nThen use the `--beaker` flag to prepare the workspace locally and launch N GPU workers in the cluster:\n\n```bash\nolmocr s3://my_s3_bucket/pdfworkspaces/exampleworkspace --pdfs s3://my_s3_bucket/jakep/gnarly_pdfs/*.pdf --beaker --beaker_gpus 4\n```\n\n\n### Using Docker\n\nPull the Docker image (large, includes the model, ~30GB):\n```bash\ndocker pull alleninstituteforai/olmocr:latest-with-model\n```\n\nFor advanced users who want to manage their own model downloads, we also provide a base image without the model:\n```bash\ndocker pull alleninstituteforai/olmocr:latest\n```\n\n#### Quick Start - Process PDFs\n\nProcess a single PDF in your current directory:\n```bash\ndocker run --gpus all \\\n -v $(pwd):/workspace \\\n alleninstituteforai/olmocr:latest-with-model \\\n -c \"olmocr /workspace/output --markdown --pdfs /workspace/sample.pdf\"\n```\n\nProcess multiple PDFs:\n```bash\ndocker run --gpus all \\\n -v /path/to/pdfs:/input \\\n -v /path/to/output:/output \\\n alleninstituteforai/olmocr:latest-with-model \\\n -c \"olmocr /output --markdown --pdfs /input/*.pdf\"\n```\n\n#### Interactive Mode\n\nRun the container interactively for exploration and debugging:\n```bash\ndocker run -it --gpus all alleninstituteforai/olmocr:latest-with-model\n```\n\n> Visit our Docker repository on [Docker Hub](https://hub.docker.com/r/alleninstituteforai/olmocr) for more information.\n\n### Full Documentation\n\nTo see all available options:\n```bash\nolmocr --help\nusage: pipeline.py [-h] [--pdfs [PDFS ...]] [--model MODEL] [--workspace_profile WORKSPACE_PROFILE] [--pdf_profile PDF_PROFILE] [--pages_per_group PAGES_PER_GROUP] [--max_page_retries MAX_PAGE_RETRIES] [--max_page_error_rate MAX_PAGE_ERROR_RATE] [--workers WORKERS]\n [--apply_filter] [--stats] [--markdown] [--target_longest_image_dim TARGET_LONGEST_IMAGE_DIM] [--target_anchor_text_len TARGET_ANCHOR_TEXT_LEN] [--guided_decoding] [--gpu-memory-utilization GPU_MEMORY_UTILIZATION] [--max_model_len MAX_MODEL_LEN]\n [--tensor-parallel-size TENSOR_PARALLEL_SIZE] [--data-parallel-size DATA_PARALLEL_SIZE] [--port PORT] [--server SERVER] [--beaker] [--beaker_workspace BEAKER_WORKSPACE] [--beaker_cluster BEAKER_CLUSTER] [--beaker_gpus BEAKER_GPUS] [--beaker_priority BEAKER_PRIORITY]\n workspace\n\nManager for running millions of PDFs through a batch inference pipeline\n\npositional arguments:\n workspace The filesystem path where work will be stored, can be a local folder, or an s3 path if coordinating work with many workers, s3://bucket/prefix/\n\noptions:\n -h, --help show this help message and exit\n --pdfs [PDFS ...] Path to add pdfs stored in s3 to the workspace, can be a glob path s3://bucket/prefix/*.pdf or path to file containing list of pdf paths\n --model MODEL Path where the model is located, allenai/olmOCR-7B-0725-FP8 is the default, can be local, s3, or hugging face.\n --workspace_profile WORKSPACE_PROFILE\n S3 configuration profile for accessing the workspace\n --pdf_profile PDF_PROFILE\n S3 configuration profile for accessing the raw pdf documents\n --pages_per_group PAGES_PER_GROUP\n Aiming for this many pdf pages per work item group\n --max_page_retries MAX_PAGE_RETRIES\n Max number of times we will retry rendering a page\n --max_page_error_rate MAX_PAGE_ERROR_RATE\n Rate of allowable failed pages in a document, 1/250 by default\n --workers WORKERS Number of workers to run at a time\n --apply_filter Apply basic filtering to English pdfs which are not forms, and not likely seo spam\n --stats Instead of running any job, reports some statistics about the current workspace\n --markdown Also write natural text to markdown files preserving the folder structure of the input pdfs\n --target_longest_image_dim TARGET_LONGEST_IMAGE_DIM\n Dimension on longest side to use for rendering the pdf pages\n --target_anchor_text_len TARGET_ANCHOR_TEXT_LEN\n Maximum amount of anchor text to use (characters), not used for new models\n --guided_decoding Enable guided decoding for model YAML type outputs\n\nVLLM arguments:\n --gpu-memory-utilization GPU_MEMORY_UTILIZATION\n Fraction of VRAM vLLM may pre-allocate for KV-cache (passed through to vllm serve).\n --max_model_len MAX_MODEL_LEN\n Upper bound (tokens) vLLM will allocate KV-cache for, lower if VLLM won't start\n --tensor-parallel-size TENSOR_PARALLEL_SIZE, -tp TENSOR_PARALLEL_SIZE\n Tensor parallel size for vLLM\n --data-parallel-size DATA_PARALLEL_SIZE, -dp DATA_PARALLEL_SIZE\n Data parallel size for vLLM\n --port PORT Port to use for the VLLM server\n --server SERVER URL of external vLLM (or other compatible provider)\n server (e.g., http://hostname:port). If provided,\n skips spawning local vLLM instance\n\nbeaker/cluster execution:\n --beaker Submit this job to beaker instead of running locally\n --beaker_workspace BEAKER_WORKSPACE\n Beaker workspace to submit to\n --beaker_cluster BEAKER_CLUSTER\n Beaker clusters you want to run on\n --beaker_gpus BEAKER_GPUS\n Number of gpu replicas to run\n --beaker_priority BEAKER_PRIORITY\n Beaker priority level for the job\n```\n\n## Code overview\n\nThere are some nice reusable pieces of the code that may be useful for your own projects:\n - A prompting strategy to get really good natural text parsing using ChatGPT 4o - [buildsilver.py](https://github.com/allenai/olmocr/blob/main/olmocr/data/buildsilver.py)\n - Basic filtering by language and SEO spam removal - [filter.py](https://github.com/allenai/olmocr/blob/main/olmocr/filter/filter.py)\n - SFT Finetuning code for Qwen2.5-VL - [train.py](https://github.com/allenai/olmocr/blob/main/olmocr/train/train.py)\n - GRPO RL Trainer - [grpo_train.py](https://github.com/allenai/olmocr/blob/main/olmocr/train/grpo_train.py)\n - Synthetic data generation - [mine_html_templates.py](https://github.com/allenai/olmocr/blob/main/olmocr/synth/mine_html_templates.py)\n - Processing millions of PDFs through a finetuned model using VLLM - [pipeline.py](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py)\n - Viewing [Dolma docs](https://github.com/allenai/dolma) created from PDFs - [dolmaviewer.py](https://github.com/allenai/olmocr/blob/main/olmocr/viewer/dolmaviewer.py)\n\n\n\n## Team\n\n<!-- start team -->\n\n**olmOCR** is developed and maintained by the AllenNLP team, backed by [the Allen Institute for Artificial Intelligence (AI2)](https://allenai.org/).\nAI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering.\nTo learn more about who specifically contributed to this codebase, see [our contributors](https://github.com/allenai/olmocr/graphs/contributors) page.\n\n<!-- end team -->\n\n## License\n\n<!-- start license -->\n\n**olmOCR** is licensed under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0).\nA full copy of the license can be found [on GitHub](https://github.com/allenai/olmocr/blob/main/LICENSE).\n\n<!-- end license -->\n\n## Citing\n\nFor olmOCR v1 and OlmOCR-bench:\n```bibtex\n@misc{olmocrbench,\n title={{olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models}},\n author={Jake Poznanski and Jon Borchardt and Jason Dunkelberger and Regan Huff and Daniel Lin and Aman Rangapur and Christopher Wilhelm and Kyle Lo and Luca Soldaini},\n year={2025},\n eprint={2502.18443},\n archivePrefix={arXiv},\n primaryClass={cs.CL},\n url={https://arxiv.org/abs/2502.18443},\n}\n```\n\nFor olmOCR v2 Unit Testing Rewards with RL:\n```bibtex\n@misc{olmocr2,\n title={olmOCR 2: Unit Test Rewards for Document OCR}, \n author={Jake Poznanski and Luca Soldaini and Kyle Lo},\n year={2025},\n eprint={2510.19817},\n archivePrefix={arXiv},\n primaryClass={cs.CV},\n url={https://arxiv.org/abs/2510.19817}, \n}\n```\n"}}}} |