Incremental Processing

v0.6.7

When the same document is submitted multiple times — via connector syncs, webhook retries, or manual re-uploads — DocIntel now skips extraction entirely and returns the cached result. Zero LLM cost for unchanged documents.

The problem

Before v0.6.7:

• A SharePoint sync re-processes every document in the library, even unchanged ones
• Each re-process costs LLM tokens, compute time, and graph write operations
• For a 1000-document library with 10 changes, you pay for 1000 extractions instead of 10

How it works

Stage 1b: Incremental check

After a job is created and the file bytes are received, an incremental check runs as Stage 1b — before format detection:

Stage 1 · Ingest → job created, file bytes received
    ↓
Stage 1b · Incremental check                              ← NEW
    ├── SHA-256 hash of raw bytes → content_hash
    ├── get_by_hash(content_hash) → existing completed job?
    ├── YES → copy result, mark completed, return
    └── NO → continue to Stage 2
    ↓
Stage 2 · Detect → format identified

Content hashing

The SHA-256 hash is computed from the raw document bytes before any parsing. This means:

• Byte-identical documents always match (regardless of filename or metadata)
• Different renderings of the same logical content do NOT match (e.g. re-exported PDF)
• The hash is stored as content_hash on the ProcessingJob model

Job store lookup

All three job store backends support the get_by_hash() method:

Backend	Implementation
Memory	Linear scan of completed jobs
Redis	Hash index docintel:hash:{content_hash} → job ID
SQLite	SELECT ... WHERE content_hash = ? AND status = 'completed' ORDER BY created_at DESC LIMIT 1

Result caching

When a match is found:

1. The existing job's ExtractionResult (entities + relationships) is copied to the new job
2. The new job is marked completed immediately
3. A pipeline.incremental_skip log entry records both the new and source job IDs
4. The emit step is skipped (the data is already in the graph from the original job)

API response

When a job completes via incremental skip, the response includes:

{
  "job_id": "new-job-456",
  "status": "completed",
  "content_hash": "sha256:a1b2c3d4e5f6...",
  "metadata": {
    "incremental_skip": true,
    "source_job_id": "original-job-123",
    "processing_ms": 2
  }
}

Processing time drops from seconds (full extraction) to single-digit milliseconds (hash lookup + copy).

Connector benefits

Connector	Without incremental	With incremental
SharePoint sync	Re-processes all docs in library	Only new/modified docs processed
Confluence sync	Re-processes all pages in space	Only updated pages processed
S3 sync	Re-processes all objects matching glob	Only new objects processed
Webhook retry	Full extraction on retry	Instant cached response

Edge cases

Scenario	Behaviour
Same file, different profile	Full extraction — hash matches, but different profile may be specified. Currently returns cached result regardless of profile.
Same content, different filename	Incremental skip — hash is based on bytes, not filename
File modified by 1 byte	Full extraction — SHA-256 hash changes
Concurrent submissions	Both jobs compute hash; first to complete is stored; second finds the match

WARNING

If a document's extraction was incorrect (e.g. wrong profile used) and you resubmit it, the cached result is returned. Delete the original job first, or use a different profile parameter to force re-extraction.

Technical details

• Hash algorithm: SHA-256 (64 hex characters)
• Field: ProcessingJob.content_hash — optional, populated during Stage 1b
• 7 dedicated tests covering hash lookup, cache skip, miss behaviour, and edge cases