Multi-Language Support

v0.12.5

DocIntel provides automatic language detection across 75+ languages via Lingua paired with rich, per-language extraction guidance — not just a language tag, but detailed conventions for entity naming, transliteration, date formats, and cultural norms that help the LLM extract accurately from non-English documents.

The problem

Before v0.6.7, non-English documents received a single-line hint:

The document is written in German.

This led to:

• German compound nouns being split incorrectly
• Japanese names losing honorifics
• Arabic dates not normalised from Hijri calendar
• Chinese entity names lacking proper segmentation
• Inconsistent romanisation across entities from the same document

Supported languages (75+ detected, deep cultural conventions built in)

Lingua automatically detects 75+ languages on the first 5,000 characters of extracted text, with deep script/transliteration conventions for major enterprise languages including:

Language	Code	Script	Key conventions
🇩🇪 German	de	Latin	Compound nouns preserved; formal Sie/du context; umlauts (ä, ö, ü) kept
🇫🇷 French	fr	Latin	Accented characters preserved; date DD/MM/YYYY; title conventions
🇪🇸 Spanish	es	Latin	Double surnames (paterno materno); date DD/MM/YYYY
🇵🇹 Portuguese	pt	Latin	Brazilian vs. European variants; prepositions in names
🇮🇹 Italian	it	Latin	Articles in company names; date DD/MM/YYYY
🇳🇱 Dutch	nl	Latin	Tussenvoegsel (van, de, van der) in names
🇨🇳 Chinese	zh	CJK	Simplified + traditional; no spaces; family-name-first; pinyin romanisation
🇯🇵 Japanese	ja	CJK + Kana	Kanji names with furigana; honorifics (-san, -sama); mixed scripts
🇰🇷 Korean	ko	Hangul	Hangul + Hanja; family-name-first; revised romanisation
🇸🇦 Arabic	ar	Arabic	Right-to-left; Hijri date normalisation; Al- prefix handling
🇮🇳 Hindi	hi	Devanagari	Devanagari + Latin transliteration; honorifics (Shri, Smt)
🇷🇺 Russian	ru	Cyrillic	Patronymics; Cyrillic + Latin transliteration
🇹🇷 Turkish	tr	Latin	Dotted/undotted İ/I; Turkish-specific casing rules
🇵🇱 Polish	pl	Latin	Diacritics (ą, ę, ł, ź, ż); declension in names
🇸🇪 Swedish	sv	Latin	å, ä, ö preserved; naming conventions

English documents receive no additional language hint (the default prompts are English-native).

Unsupported languages receive a generic instruction to preserve native script and use English JSON keys.

How it works

Language detection

DocIntel uses Lingua for automatic language detection on the first 5000 characters of extracted text. You can also specify the language explicitly via the language parameter on the /process endpoint.

Prompt injection

The _get_lang_hint() function checks the _LANG_INSTRUCTIONS dictionary and returns a rich, multi-line instruction block that gets appended to both the entity and relationship extraction prompts.

Example for German:

LANGUAGE-SPECIFIC INSTRUCTIONS (German):
• Preserve German compound nouns as single entity names (e.g., "Bundesverfassungsgericht", not split)
• Keep umlauts (ä, ö, ü) and eszett (ß) in entity names
• For person names, use "Vorname Nachname" order in the entity name
• Normalise dates to ISO 8601 (YYYY-MM-DD) regardless of German DD.MM.YYYY input format
• Company suffixes: GmbH, AG, KG, e.V. — keep as part of entity name
• All JSON keys must remain in English

Output format

Entity names are extracted in native script with consistent conventions:

{
  "entities": [
    {"type": "Company", "name": "Bundesanstalt für Finanzdienstleistungsaufsicht", "confidence": 0.71},
    {"type": "Person", "name": "Hans-Werner Müller", "confidence": 0.68},
    {"type": "Date", "name": "2026-03-15", "confidence": 0.73},
    {"type": "Value", "name": "€2.500.000,00", "confidence": 0.65}
  ]
}

Note:

• German compound noun preserved (Bundesanstalt für Finanzdienstleistungsaufsicht)
• Umlauts kept (Müller)
• Date normalised to ISO 8601
• JSON keys in English (type, name, confidence)

Usage

Automatic detection

# Language auto-detected from document content
curl -X POST http://localhost:8200/process \
  -H "Authorization: Bearer $API_KEY" \
  -F "file=@vertrag.pdf"

Explicit language

# Force German language hints
curl -X POST http://localhost:8200/process \
  -H "Authorization: Bearer $API_KEY" \
  -F "file=@vertrag.pdf" \
  -F "language=de"

Technical details

• _LANG_INSTRUCTIONS — dictionary with 15 entries, each containing 4–6 lines of guidance
• _get_lang_hint(language) — returns rich instruction block for known languages, generic for unknown, empty for English
• Applied in both _build_entity_prompt() and _build_relationship_prompt()
• 7 dedicated tests covering per-language hints, fallback behaviour, and prompt injection