Problems with unreadable characters

[renchix]

Description of the bug

I have found a bug when using doc[0].get_text('words') in case there is unreadable character present.

There is a table in the PDF, and whenever the "cell" contains this special character, the character is not read via get_text('words') method, and the next word gets wrong bbox values, having coordinates of that nonreadable character instead of its own coordinates.

In this example that strange symbol (vertical bar with x on top) is not read at all, and the -4.00 is assigned x0, y0, x1, y1 coordinates of that non-read character.

In cases where are several consecutive such characters then only \u200d (zero-length-joiner) is read (probably between such characters). In one case (se picture below) with four such characters, two \u200d's are read (as separate "words"), but probably because of multiple such characters, +200.00 gets correct coordinates.

However, in all cases with one such character in the "cell" coordinates get mixed up for the following word.
So far have noticed only with this particular character (it tends to come up several times in such files).

Using latest version of PyMuPDF.

P.S. Cannot share full file as it contains sensitive data (probably could try somehow to extract part of that PDF with problematic characters).

How to reproduce the bug

Have a PDF with wrong character.

PyMuPDF version

1.26.5

Operating system

Windows

Python version

3.11

[renchix]

Upon doing some research (at least then I am able to get similar symbol) it seems to be symbols VERTICAL LINE (\u007C) together with COMBINING X ABOVE (\u033D).
So probably that COMBINING X ABOVE is causing problems.

[JorjMcKie]

We cannot deal with unreproducible issue.
Please provide an example page / PDF.

[renchix]

error_example.pdf

Adding example file. Removed all irrelevant text.

get_text('dict') seems to perform better, as coordinates are kept correctly, and for this symbol merge correct number of \u200d (ZERO WIDTH JOINER) is shown
get_text('words') doesn't show all \u200d's and also subsequent words has wrong coordinates.

Example

import fitz
doc = fitz.open(file)
print(doc[0].get_text('words')) # two \u200d are output as text value
print(doc[0].get_text('dict')) # four \u200d are output as text value

print('---2nd page---')
print(doc[1].get_text('words')) # bbox values for text '-10.00' is wrong: 296.30767822265625, 383.546875, 481.40777587890625, 392.484375
print(doc[1].get_text('dict')) # correct bbox values for text '-10.00' - (458.727783203125, 383.546875, 481.40777587890625, 392.484375)

[JorjMcKie]

I am not convinced that there is a bug.
I have copy-pasted the relevant portions and inspected via Python. The result always looks like this:

Independent from the PDF viewer used.
0x200D is the 'ZERO WIDTH JOINER'. So this character itself has width=0. Here is what ChatGPT has to say:

🧩 Unicode Character: U+200D (Zero Width Joiner)
Code Point: U+200D

Block: General Punctuation (U+2000–U+206F)

Function: A format character used to control ligature formation and grapheme joining, especially in scripts like Devanagari, Arabic, and emoji sequences.

Bidirectional Class: Format (BN – Boundary Neutral)

It’s not rendered visibly and doesn’t carry semantic content, but it affects how adjacent characters are displayed.

Obviously we do see something though. AFAIK, this can only mean that there is a problem in the font's To-Unicode mapping.
BTW the base library also sees 0x200D and extracts it accordingly. Here is its output:file.zip

All I can do in "words" extraction is ignoring it ...

[renchix]

I am less concerned about the fact that the character itself is wrongly extracted (for correct extraction, though, it would need to be extracted as all three characters - 'vertical line' (\u007C), 'zero width joiner' (\u200d) and and 'combining x above' (\u033d).

The problem that I am concerned with, is that when using get_text('words') method, the word that follows this specific combined character is assigned wrong bbox coordinates - instead of its own coordinates, the coordinates of that strange character are used. As it is with word '-10.00' in the provided example.

So get_text('words') is outputting wrong coordinates and there seems to be problem with the specific method. Because get_text('dict') has different (and correct) coordinates for the same word: '-10.00'

[JorjMcKie]

Only one character 0x200D is there - not any of the others you mention. So they will never appear in any output.

The way "words" extraction works is scanning character by character, ignoring any white spaces (plus potentially delimiters).
It starts a word on encountering the first character outside the set of word delimiting characters and finishes the word upon encountering the next delimiter.
So currently character 0x200d - being not in the delimiters - will start a word ... and will immediately end it again in your example. This obviously causes confusion because the thus generated word has an empty rectangle.
I will probably fix this by ignoring this Unicode completely.

[JorjMcKie]

I have a fix that disallows a word to start with 0x200d.

I learned that I cannot generally treat this Unicode as a delimiter.

With this fix, your two example pages look like this:

and