EpubCCPDF to EPUB conversion benchmark
Check the files. Compare the results.
In EpubCC's benchmark of November 3, 2026, the EpubCC engine reached a mean 8-gram recall of 0.767 on 12 PDFs with an independent reference text, against 0.743 for pdftotext and 0.746 for PDFium; 15 of its 15 EPUBs passed EPUBCheck with no errors.
Updated:
The files, by document type
| Document | Two columns | Three columns | Footnotes | Repeated headers | Chapters | Arabic (RTL) | Japanese | Spanish | French | German | Charts | Scanned | Wrapped images |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Source PDF | file (PDF) | file (PDF) | file (PDF) | file (PDF) | file (PDF) | file (PDF) | file (PDF) | file (PDF) | file (PDF) | file (PDF) | file (PDF) | file (PDF) | file (PDF) |
| Result EPUB | file (EPUB) | file (EPUB) | file (EPUB) | file (EPUB) | file (EPUB) | file (EPUB) | file (EPUB) | file (EPUB) | file (EPUB) | file (EPUB) | file (EPUB) | file (EPUB) | file (EPUB) |
| EPUBCheck errors / warnings | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 | 0 / 0 |
Methodology
- Same input. 15 PDFs, every one public domain or under a free licence, run through 8 text-extraction engines on November 3, 2026.
- Automated scoring. 12 of the PDFs have an independent reference text of the same document (the HTML of the Wikipedia or arXiv article, the Project Gutenberg text, or govinfo's plain-text edition), never another tool's output. Text quality is measured against it with n-gram metrics.
- Valid EPUB. Every EPUB our engine writes is checked with EPUBCheck: 15 of 15 had no errors.
- 4-gram recall
- Share of the reference text's 4-word sequences found in the output: content recovered.
- 4-gram precision
- Share of the output's 4-word sequences found in the reference: content that belongs.
- 8-gram recall
- Like 4-gram recall with 8 words: drops when reading order or line joins break sentences (continuity).
- Merged words / 1,000
- Words glued to their neighbour, per 1,000 words.
- Split words / 1,000
- Words broken in two, per 1,000 words.
- Margin leak
- Share of repeated running headers, footers and page numbers left in the text.
- Unmapped glyphs / 1,000
- Characters reported as U+FFFD because the PDF has no Unicode for them. Reported, never hidden: a higher number can mean the engine is honest about a problem others drop silently.
- Seconds
- Wall time for the document.
- Peak MB
- Peak memory.
Results: mean per engine
| Engine | Version | Licence | 4-gram recall | 4-gram precision | 8-gram recall | Merged words / 1,000 | Split words / 1,000 | Margin leak | Unmapped glyphs / 1,000 | Seconds | Peak MB | Failed pages |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| EpubCC engine (EPUB output) Text measured inside the EPUB the user downloads; time and memory include writing the EPUB. |
EpubCC engine, commit c953279 (2026-11-03) | Ours | 0.866 | 0.663 | 0.767 | 2.460 | 2.045 | 0.071 | 5.254 | 15.3 | 286 | 0 |
| EpubCC engine (plain text) The same layout engine before EPUB writing. |
EpubCC engine, commit c953279 (2026-11-03) | Ours | 0.865 | 0.657 | 0.765 | 2.419 | 2.100 | 0.988 | 5.185 | 3.6 | 101 | 0 |
| pdf_oxide · extract_text | pdf_oxide 0.3.78 | MIT OR Apache-2.0 | 0.842 | 0.636 | 0.726 | 3.985 | 0.286 | 0.887 | 0.525 | 5.7 | 206 | 0 |
| pdf_oxide · to_plain_text (XY-cut) | pdf_oxide 0.3.78 | MIT OR Apache-2.0 | 0.811 | 0.609 | 0.658 | 3.715 | 1.243 | 0.794 | 0.525 | 5.8 | 205 | 0 |
| pdf_oxide · Markdown | pdf_oxide 0.3.78 | MIT OR Apache-2.0 | 0.795 | 0.602 | 0.653 | 7.118 | 2.785 | 0.102 | 0.271 | 8.2 | 202 | 0 |
| PDFium · text().all() | pdfium-render 0.9.5, PDFium chromium/8125 | BSD-3-Clause / Apache-2.0 | 0.828 | 0.617 | 0.746 | 1.659 | 2.997 | 0.999 | 5.971 | 2.6 | 101 | 0 |
| OpenDataLoader PDF (local mode) Reference for heuristics; ruled out as an engine for its memory use. |
opendataloader-pdf 2.5.17 | Apache-2.0 | 0.827 | 0.625 | 0.749 | 2.067 | 0.835 | 0.010 | 0.019 | 32.2 | 1358 | 0 |
| pdftotext (Poppler) Measuring stick only: GPL software is never shipped in EpubCC. |
Poppler 25.09.1 | GPL | 0.841 | 0.629 | 0.743 | 1.970 | 1.840 | 0.046 | 5.652 | 3.9 | 36 | 0 |
Where other engines beat ours
Documents where another engine's 8-gram recall is more than 0.02 above our EPUB output. We publish these because they are what we work on next.
- Wikipedia (English): Alice's Adventures in Wonderland: pdf_oxide · extract_text 0.800 · EpubCC 0.761
- Wikipedia (Spanish): Don Quijote de la Mancha: pdf_oxide · extract_text 0.963 · EpubCC 0.941
- Wikipedia (French): Les Misérables: pdf_oxide · extract_text 0.903 · EpubCC 0.819
- Wikipedia (German): Johann Wolfgang von Goethe: pdf_oxide · extract_text 0.877 · EpubCC 0.845
- OLMo paper (arXiv 2402.00838): pdf_oxide · extract_text 0.854 · EpubCC 0.800
- Mistral 7B paper (arXiv 2310.06825): PDFium · text().all() 0.872 · EpubCC 0.760
- QLoRA paper (arXiv 2305.14314): OpenDataLoader PDF (local mode) 0.917 · EpubCC 0.786
- Calculus textbook typeset in LaTeX (Project Gutenberg #33283): OpenDataLoader PDF (local mode) 0.533 · EpubCC 0.487
- Alice's Adventures in Wonderland, 1867 printing (scan, Internet Archive): PDFium · text().all() 0.651 · EpubCC 0.513
Results per document (8-gram recall)
| Document | Pages | EpubCC engine (EPUB output) | EpubCC engine (plain text) | pdf_oxide · extract_text | pdf_oxide · to_plain_text (XY-cut) | pdf_oxide · Markdown | PDFium · text().all() | OpenDataLoader PDF (local mode) | pdftotext (Poppler) |
|---|---|---|---|---|---|---|---|---|---|
| Mistral 7B paper (arXiv 2310.06825) | 9 | 0.760 | 0.758 | 0.767 | 0.778 | 0.800 | 0.872 | 0.859 | 0.821 |
| Wikipedia (English): Alice's Adventures in Wonderland | 24 | 0.761 | 0.760 | 0.800 | 0.800 | 0.672 | 0.774 | 0.781 | 0.769 |
| Wikipedia (Spanish): Don Quijote de la Mancha | 50 | 0.941 | 0.941 | 0.963 | 0.963 | 0.857 | 0.945 | 0.954 | 0.944 |
| Wikipedia (French): Les Misérables | 22 | 0.819 | 0.820 | 0.903 | 0.903 | 0.751 | 0.884 | 0.891 | 0.846 |
| Wikipedia (German): Johann Wolfgang von Goethe | 86 | 0.845 | 0.848 | 0.877 | 0.877 | 0.758 | 0.854 | 0.862 | 0.853 |
| Wikipedia (Arabic): One Thousand and One Nights | 17 | 0.673 | 0.674 | 0.315 | 0.315 | 0.273 | 0.001 | 0.000 | 0.659 |
| Wikipedia (Japanese): The Tale of Genji | 67 | 0.963 | 0.963 | 0.970 | 0.970 | 0.797 | 0.952 | 0.942 | 0.965 |
| OLMo paper (arXiv 2402.00838) | 21 | 0.800 | 0.800 | 0.854 | 0.273 | 0.719 | 0.761 | 0.820 | 0.781 |
| QLoRA paper (arXiv 2305.14314) | 26 | 0.786 | 0.782 | 0.817 | 0.852 | 0.868 | 0.908 | 0.917 | 0.770 |
| Federal Register document 2026-22443 | 45 | 0.859 | 0.853 | 0.334 | 0.035 | 0.339 | 0.859 | 0.801 | 0.835 |
| The 9/11 Commission Report (GPO) | 585 | — | — | — | — | — | — | — | — |
| The Constitution of the United States (House Document 110-50) | 85 | — | — | — | — | — | — | — | — |
| Economic Report of the President 2024 | 487 | — | — | — | — | — | — | — | — |
| Calculus textbook typeset in LaTeX (Project Gutenberg #33283) | 292 | 0.487 | 0.487 | 0.479 | 0.496 | 0.481 | 0.494 | 0.533 | 0.511 |
| Alice's Adventures in Wonderland, 1867 printing (scan, Internet Archive) | 212 | 0.513 | 0.494 | 0.632 | 0.639 | 0.524 | 0.651 | 0.625 | 0.156 |
Documents and licences
| Document | Licence | Traits | Reference text |
|---|---|---|---|
| Mistral 7B paper (arXiv 2310.06825) | CC BY 4.0 | text PDF, one column, figures, academic paper, short | yes |
| Wikipedia (English): Alice's Adventures in Wonderland | CC BY-SA 4.0 (text); images under their own licences | text PDF, one column, images, English | yes |
| Wikipedia (Spanish): Don Quijote de la Mancha | CC BY-SA 4.0 (text); images under their own licences | text PDF, one column, images, Spanish | yes |
| Wikipedia (French): Les Misérables | CC BY-SA 4.0 (text); images under their own licences | text PDF, one column, French | yes |
| Wikipedia (German): Johann Wolfgang von Goethe | CC BY-SA 4.0 (text); images under their own licences | text PDF, one column, German | yes |
| Wikipedia (Arabic): One Thousand and One Nights | CC BY-SA 4.0 (text); images under their own licences | text PDF, one column, Arabic, right-to-left | yes |
| Wikipedia (Japanese): The Tale of Genji | CC BY-SA 4.0 (text); images under their own licences | text PDF, one column, Japanese, CJK | yes |
| OLMo paper (arXiv 2402.00838) | CC BY 4.0 | text PDF, two columns, tables, footnotes, figures, academic paper | yes |
| QLoRA paper (arXiv 2305.14314) | CC BY 4.0 | text PDF, one column, tables, footnotes, academic paper, long | yes |
| Federal Register document 2026-22443 | Public domain (work of the U.S. Government) | text PDF, three columns, repeated headers, footnotes | yes |
| The 9/11 Commission Report (GPO) | Public domain (work of the U.S. Government) | text PDF, book, chapters, repeated headers, endnotes, long, load test | no |
| The Constitution of the United States (House Document 110-50) | Public domain (work of the U.S. Government) | text PDF, book, repeated headers, table of contents | no |
| Economic Report of the President 2024 | Public domain (work of the U.S. Government) | text PDF, book, tables, charts, boxes, footnotes, long | no |
| Calculus textbook typeset in LaTeX (Project Gutenberg #33283) | Public domain in the U.S. (Project Gutenberg; its trademark only under its licence) | text PDF, book, chapters, repeated headers, footnotes, formulas | yes |
| Alice's Adventures in Wonderland, 1867 printing (scan, Internet Archive) | Public domain (1867; University of Toronto scan on the Internet Archive) | scanned, OCR layer, book, chapters, illustrations | yes |
Limits of these numbers
- The margin-leak metric is only reliable for engines that split lines like pdftotext -layout; it is not reliable for OpenDataLoader, pdf_oxide Markdown or pdftotext without -layout.
- Wikipedia references leave out tables and infoboxes, so precision is low for every engine alike on those documents.
- The 1867 Alice scan is compared with a different edition (Project Gutenberg, 1865): it is good for recall, not for exact precision. It is read through the PDF's own OCR layer; EpubCC does not do OCR yet.
- Text-quality averages cover only the 12 documents with an independent reference text (HTML, Gutenberg text or govinfo plain text of the same document, never another tool's output). The other 3 report time, memory, failed pages, unmapped glyphs and margin leak.
- These metrics measure extracted text. They do not measure chapter detection, footnote linking or how the EPUB looks on a reader; EPUBCheck validity is reported separately for our EPUB output.
- Calibre is not part of this run.
Data
- Licence: CC BY 4.0 (results data)
- Raw results (JSON)
- Corpus list (JSON)