Table Extraction

v0.6.7

Structured tables in PDFs, DOCX, and XLSX documents contain critical data — line items, amounts, dates, account codes — that was previously lost when tables were flattened into running text. DocIntel now extracts tables as first-class data and feeds them to the LLM as dedicated context windows.

The problem

Before v0.6.7:

• A 50-row invoice table becomes a wall of text: "Item 1 Widget A 100 $5.00 $500.00 Item 2 Gadget B 50 $10.00..."
• The LLM struggles to associate quantities with prices with descriptions
• Column relationships are lost entirely
• Many line items are missed or incorrectly attributed

How it works

Stage 3b: Table injection

After the format-specific parser produces a ParsedDocument, table injection runs as Stage 3b — between parsing (Stage 3) and classification (Stage 4):

Stage 3 · Parse → ParsedDocument (text + raw_tables + metadata)
    ↓
Stage 3b · Table injection                                    ← NEW
    ├── _tables_to_markdown() → markdown table strings
    └── _inject_table_windows() → dedicated LLM windows
    ↓
Stage 4 · Classify → extraction profile selected

Step 1: Markdown conversion

_tables_to_markdown() converts each table from the parser's raw_tables format (list of row arrays) into a proper markdown table:

| Description | Quantity | Unit Price | Total |
|---|---|---|---|
| Widget A | 100 | $5.00 | $500.00 |
| Widget B | 50 | $10.00 | $500.00 |
| Gadget C | 25 | $20.00 | $500.00 |

Step 2: Window injection

_inject_table_windows() creates dedicated LLM context windows from each markdown table:

• Each table becomes its own extraction window
• Large tables (exceeding the window token limit) are automatically split by row with header preservation — every split chunk gets the header row prepended
• Each window is tagged with is_table=True metadata so the pipeline can track table-sourced entities

Step 3: Extraction

Entity and relationship extraction runs on table windows alongside text windows. Because the table structure is preserved in markdown format, the LLM can:

• Associate values in the same row (Item → Quantity → Price → Total)
• Understand column semantics from headers
• Extract all line items, not just the first few

What gets extracted

Document type	Typical table content	Extracted entities
Invoice	Line items, amounts, tax	InvoiceLineItem, Amount, Date
Contract	Payment schedules, parties	MonetaryValue, Date, Party
Financial statement	GL accounts, balances	GLAccount, Amount, Transaction
Scientific paper	Results tables, benchmarks	Metric, Dataset, Finding
BIM schedule	Room schedules, quantities	IfcSpace, IfcElement, quantities

Example

Given an invoice PDF with a line items table:

// Before v0.6.7 — entities from flattened text
{
  "entities": [
    {"type": "Company", "name": "Acme Corp", "confidence": 0.92},
    {"type": "Amount", "name": "$1,500.00", "confidence": 0.78}
    // Line items missed entirely
  ]
}

// After v0.6.7 — entities from table windows
{
  "entities": [
    {"type": "Company", "name": "Acme Corp", "confidence": 0.74},
    {"type": "InvoiceLineItem", "name": "Widget A × 100", "confidence": 0.71},
    {"type": "InvoiceLineItem", "name": "Gadget B × 50", "confidence": 0.68},
    {"type": "Amount", "name": "$1,500.00", "confidence": 0.72},
    {"type": "Date", "name": "2026-04-01", "confidence": 0.69}
  ]
}

TIP

Confidence scores shown are calibrated (post-temperature scaling). Raw LLM scores would be ~0.10–0.15 higher.

Technical details

• Parser support: raw_tables populated by PDF, DOCX, and XLSX parsers
• Header detection: First row of each table is treated as the header
• Split threshold: Tables exceeding 2000 tokens are split; each chunk gets the header row
• 7 dedicated tests covering markdown conversion, window injection, splitting, and edge cases