UTF-8 Encoding Issues in API Response Text Fields

[pablospe]

Description

The Draftable API is returning text fields with UTF-8 encoding problems (mojibake) that require post-processing to be readable.

Problem Details

Issue: Text fields (leftText, rightText, text) in API responses contain improperly encoded UTF-8 characters, likely due to encoding/decoding issues in the text extraction pipeline.

Root Cause: Appears to be double-encoding, encoding detection errors, or improper handling of mixed character encodings from source PDFs.

Examples

Current API Response (garbled):

{
  "leftText": "caf\u00e9"  // Should be "café"
}

After ftfy.fix_text() (corrected):

{
  "leftText": "café"  // Properly decoded UTF-8
}

More severe example:

{
  "leftText": "áéíóú"  // Garbled accented characters
}

After ftfy.fix_text():

{
  "leftText": "áéíóú"  // Correct Spanish accented characters  
}

Impact

• Text appears garbled to end users
• Search functionality fails on corrupted text
• Document comparison accuracy is reduced
• Requires client-side post-processing with libraries like ftfy

Workaround

Using ftfy Python library to repair the encoding issues:

import ftfy

def fix_json_text_fields(response_data):
    """Fix UTF-8 encoding issues in Draftable API response."""
    target_fields = {"rightText", "text", "leftText"}
    stack = [response_data]
    
    while stack:
        current = stack.pop()
        if isinstance(current, dict):
            for key, value in current.items():
                if key in target_fields and isinstance(value, str):
                    current[key] = ftfy.fix_text(value)  # Fixes mojibake
                elif isinstance(value, (dict, list)):
                    stack.append(value)
        elif isinstance(current, list):
            for item in current:
                if isinstance(item, (dict, list)):
                    stack.append(item)

Suggested Fix

Please ensure proper UTF-8 handling in the API response pipeline:

1. Text extraction: Verify correct encoding detection from source PDFs
2. Internal processing: Maintain UTF-8 throughout the pipeline
3. API serialization: Ensure JSON output uses proper UTF-8 encoding
4. Consider using ftfy server-side to automatically fix common encoding issues

Environment

• Occurs with PDFs containing non-ASCII characters (accents, symbols, non-Latin scripts)
• Frequency: Common with multilingual documents
• All API endpoints returning text fields are affected

This significantly impacts international document processing. Server-side encoding fixes would eliminate the need for client-side workarounds.

[rychoo2]

Hello @pablospe ,

I was unable to reproduce the problem directly on the API endpoints. All the non-ASCII characters appear to be encoded correctly, see below console output from a sample comparison:

curl -X 'GET'   'https://api.draftable.com/v1/comparisons/vPwyZKAU/change-details'   -H 'accept: application/json'   -H 'Authorization: Token [REDACTED]'

{"changes":[{"kind":"replace","leftText":"café","rightText":"teść","leftRegion":{
[...]

Also appears to be correct using the below python snippet:

draftable_client = draftable.Client(account_id, auth_token, base_url)
comparisons = draftable_client.comparisons

change_details = comparisons.change_details("vPwyZKAU")

for change in change_details.changes:
    if change.kind != "match":
        print(f"change.kind: {change.kind}, leftText: {change.leftText}, rightText: {change.rightText}")

print(change_details.to_dict())

python example/test.py 

change.kind: replace, leftText: café, rightText: teść
{'changes': [{'kind': 'replace', 'leftText': 'café', 'rightText': 'teść', 'leftRegion': {'pageIndex': 0, 'rectangles': [{'left': 38.28, 'top': 18.78, 'right': 58.84, 'bottom': 27.41}]}, 'rightRegion': {'pageIndex': 0, 'rectangles': [{'left': 37.97, 'top': 18.78, 'right': 58.41, 'bottom': 27.41}]}, 'stylesInfo': {'leftStyles': [{'color': '#000000', 'font': 'Liberation Sans', 'emphasis': '', 'size': 11.4}], 'rightStyles': [{'color': '#000000', 'font': 'Liberation Sans', 'emphasis': '', 'size': 11.4}], 'leftStyleMap': ['0-3'], 'rightStyleMap': ['0-3']}, 'deletionMark': None}], 'summary': {'anyChanges': True}}

Could you provide me with a minimal snippet to reproduce the issue?

I suspect it is related to a default behaviour of python json.dump using ensure_ascii=True, as per the (docs(https://docs.python.org/3/library/json.html#basic-usage)):

ensure_ascii (bool) – If True (the default), the output is guaranteed to have all incoming non-ASCII characters escaped. If False, these characters will be outputted as-is.

So the text gets garbled here:

print(json.dumps(change_details.to_dict()))

{"changes": [{"kind": "replace", "leftText": "caf\u00e9", "rightText": "te\u015b\u0107" [...]

But it's fine here:

print(json.dumps(change_details.to_dict(), ensure_ascii=False))

{"changes": [{"kind": "replace", "leftText": "café", "rightText": "teść" [...]

Please let me know if this fixes your issue or if I am missing something. Feel free to provide a minimal reproducer.