TextWriter - wrong font being used

[jotaf98]

Describe the bug (mandatory)

TextWriter, when used repeatedly with 3 or more fonts, sometimes uses the wrong font, and prints out of bounds.

To Reproduce (mandatory)

Here's a script that prints the same text with 3 fonts, in multiple panels:

import fitz

font1 = fitz.Font('tiro')
font2 = fitz.Font('cour')
font3 = fitz.Font('helv')

(b, s) = (10, 150)  # border and panel size
text = 'abcde fghij klmno pqrst uvwxyz'

doc = fitz.open()
page = doc.newPage()

panel = fitz.Rect((b, b), (s, s))

for _ in range(15):
  wr = fitz.TextWriter(page.rect)
  rect = fitz.Rect(panel)

  for font in [font1, font2, font3]:
    wr.fillTextbox(rect=rect, text=text, fontsize=12, font=font)
    rect.y0 = wr.lastPoint.y

  wr.writeText(page)

  panel.x0 += s + b
  panel.x1 += s + b
  if panel.x1 >= page.rect.x1:
    panel.x0 = b
    panel.x1 = s + b
    panel.y0 += s + b
    panel.y1 += s + b

doc.save('test.pdf')

Expected behavior (optional)

The passed in fonts should be used in the specified order, and no text would be out of the specified rect.

Screenshots (optional)

See screenshot and attached PDF (tested with SumatraPDF and Foxit Reader).
Compare the first group of text (top-left), with the group below it; according to the code they should look the same but don't.

test.pdf

Your configuration (mandatory)

• Windows 10, 64 bits
• Python 3.7.3, 64 bits
• PyMuPDF 1.17.3, installed with pip (wheel)

[JorjMcKie]

Thanks for submitting this - issue confirmed.
There are two separate issues overlapping each other:

1. As documented, the new TextWriter feature has issues with character spacing for mono-spaced fonts. This seems to be the case for all such fonts, and I so far have encountered only one mono font handled correctly: consola.ttf. If you choose this one instead of Courier, you should be fine in that respect. I also have a little correction utiliy which makes other mono fonts also show correct layouts for a given PDF, if you are interested.
2. The second part is my bad: in order to determine the next available fontname looking like Fnnn, I am just increasing nnn, ignoring the fact that F9 > F10 (!!!) for example. This happens exactly in rectangle 4.

With these corrections and a little massage of the layout, the page looks like this:

This is the script producing it:

import fitz

font1 = fitz.Font("tiro")
font2 = fitz.Font(fontfile="C:/Windows/Fonts/consolab.ttf")
font3 = fitz.Font("helv")
fonts = (font1, font2, font3)
text = "abcde fghij klmno pqrst uvwxyz"
fsize = 15

doc = fitz.open()
page = doc.newPage()

# lay out a table of 15 rectangles covering the page
cols = 3
rows = 5
table_rect = page.rect + (72, 72, -72, -72)  # go away from page corners
cells = fitz.make_table(rect=table_rect, cols=cols, rows=rows)

for i in range(rows):
    for j in range(cols):
        wr = fitz.TextWriter(page.rect)
        pos = None  # first text part starts at top-left
        for font in fonts:
            print(pos)
            wr.fillTextbox(cells[i][j],  # each cell is a rectangle
                text=text, pos=pos, fontsize=fsize, font=font)
            pos = wr.lastPoint + (5, 0) # next text part starts here
        wr.writeText(page)

doc.save(__file__.replace(".py", ".pdf"))

[JorjMcKie]

BTW I just found out, that the Cascadia fonts also work correctly like Consolas does. This is a Microsoft font, which can be downloaded from here and which works on all OS. It is permissively licensed under the SIL Open Font license.

[JorjMcKie]

Here is another suitable mono spaced font: Fira Code from Mozilla.

[jotaf98]

Ah, that explains it!! That's great, thanks for looking into it.

The part relevant for my code is actually the second (looking for the wrong font name). I guess the monospace problem came up by accident.

[JorjMcKie]

I guess the monospace problem came up by accident.

Yes, as I mentioned: you have had two problems. Like my grandma always said: one can have lice and mice!
The mono space font issue does exist however. I am still investigating and I also have sent a bug report to MuPDF (Artifex) - without reaction so far after several weeks.
So I would recommend to not using Courier for mono text output.
However, as mentioned I have built a little repair utility, which digs its way through the generated text and corrects it. It will be part of the next PyMuPDF version. Actually a shame that it is needed (or I am too dumb to find the right trick).
It would be invoked like this fitz.repair_mono_font(page, font) where font is a fitz.Font with that misbehaviour. It does work for Courier and some handful of other problem fonts (DejaVu Mono, Noto Mono, Courier, ...).

I you need a solution with some urgency, I can send you a pre-version.

[jotaf98]

That's alright, perhaps a brief note (pointing to the repair function) in the docs would help, but otherwise I wouldn't fret too much if it's upstream.

It's not urgent as this is more of a hobby project, and I worked around it in a project-specific way.

Looking forward to the release that squashes one or two pests, when it's ready!

[JorjMcKie]

Just thought about your issue:
You can avoid the problem right away if you take TextWriter creation and writing to the page out of the loop!
That way, only one frontname reference per font will ever be created on the page and that "overflow" won't occur (here).
And of course there is some performance gain dealing with just textwriter instead of 15.

[JorjMcKie]

Resolved by v1.17.4, which is just uploading to PyPI.
Note:
I also found a way to automatically deal with the mono-spaced fonts: these should now all be correct without needing extra fiddling around with them.
This wasn't difficult once I realized that the misbehaving fonts do not specify character widths - or at least incorrect ones (like DejaVu Mono - this guy causedmy initial confusion). Mono fonts of course have exactly one character width. If this value is added to the font's object definition of the PDF, the world is all sunshine again ...

[jotaf98]

Thanks once again! I hate to re-open this, but even with the update the problem remains.

I don't mean that it doesn't fix the code I posted above -- it does! -- however a minor change brings back the bad behavior. Here it is:

import fitz

font1 = fitz.Font('tiro')
font2 = fitz.Font('cour')
font3 = fitz.Font('helv')

(b, s) = (10, 150)  # border and panel size
text = 'abcde fghij klmno pqrst uvwxyz'

doc = fitz.open()
page = doc.newPage()

panel = fitz.Rect((b, b), (s, s))

for i in range(15):
  wr = fitz.TextWriter(page.rect)
  rect = fitz.Rect(panel)

  wr.append(rect.bl, 'Testing', fontsize=14, font=[font1, font2, font3][i % 3])  # NEW BIT

  for font in [font1, font2, font3]:
    wr.fillTextbox(rect=rect, text=text, fontsize=12, font=font)
    rect.y0 = wr.lastPoint.y

  wr.writeText(page)

  panel.x0 += s + b
  panel.x1 += s + b
  if panel.x1 >= page.rect.x1:
    panel.x0 = b
    panel.x1 = s + b
    panel.y0 += s + b
    panel.y1 += s + b

doc.save('test.pdf')

Mixing fillTextBox and append (by adding the line I marked above) seems to result in the same bug showing up again.

As a separate quick question, you mentioned that reusing the TextWriter should be more performant; is there any way to do this but writing different text? (In these contrived examples, we write the same words over and over again, but in real applications the text is usually different.)

[JorjMcKie]

As a separate quick question, you mentioned that reusing the TextWriter should be more performant; is there any way to do this but writing different text? (In these contrived examples, we write the same words over and over again, but in real applications the text is usually different.)

Let me react to this one first:
If you take out text writer creation and writeText out of the loop, your example works correctly and is also more performant. It also should be no problem adding whatever (different) text in each loop.

doc = fitz.open()
page = doc.newPage()

panel = fitz.Rect((b, b), (s, s))
wr = fitz.TextWriter(page.rect)

for i in range(15):
    rect = fitz.Rect(panel)

    wr.append(
        rect.bl, "Testing", fontsize=14, font=[font1, font2, font3][i % 3]
    )  # NEW BIT

    for font in [font1, font2, font3]:
        wr.fillTextbox(rect=rect, text=text, fontsize=12, font=font)
        rect.y0 = wr.lastPoint.y

    panel.x0 += s + b
    panel.x1 += s + b
    if panel.x1 >= page.rect.x1:
        panel.x0 = b
        panel.x1 = s + b
        panel.y0 += s + b
        panel.y1 += s + b

wr.writeText(page)

doc.save("test.pdf")

This version only creates the 3 fonts actually used on the page. In your version, 3 fonts are created in every iteration, 45 in total (once my error has been corrected). Note that only the 45 font names are created, the binary font file content only once per font. So the file size difference is just about 2KB.

Haven't understood yet, why your original example fails ...