overlaying more than one pixmap breaks embedded pdf

[cbm755]

This seems to be a regression between 1.18.0 and 1.18.1.

I have an input pdf file which itself contains an embedded pdf "figure".

I overlay 1 pixmap on top. No problem. Overlay two or more and my embedded pdf figure is missing in the output.

• left-most is input.pdf. Note the "ID Box" with student number, name etc: this is the embedded PDF graphic.
• overlay "9" pixmap (called little.png in my example code):
  
• middle screenshot has "9" pixmap overlaid
• right screenshot has four "9"'s overlaid: note the "ID Box" is missing.

Your configuration (mandatory)

• Fedora 33
• Python 3.9
• PyMuPDF 1.18.1--1.18.6 all show this. 1.18.0 does not. All installed from pip.

Example code

import fitz

# note this input has "idBox2.pdf" embedded inside (generated with pdflatex)
input = fitz.open('input.pdf')

# new doc with first page of input
doc = fitz.open()
doc.insertPDF(input, from_page=0, to_page=0, start_at=-1)

# just boxes for 4 corners, probably not hugely relevant
page_width = doc[0].bound().width
page_height = doc[0].bound().height
rTL = fitz.Rect(15, 20, 85, 90)
rTR = fitz.Rect(page_width - 85, 20, page_width - 15, 90)
rBL = fitz.Rect(15, page_height - 90, 85, page_height - 20)
rBR = fitz.Rect(page_width - 85, page_height - 90, page_width - 15, page_height - 20)

im = fitz.Pixmap('little.png')

# As soon as we overlay 2 or more images, the idBox2.pdf graphic disappears!
doc[0].insertImage(rTL, pixmap=im, overlay=True)
doc[0].insertImage(rTR, pixmap=im, overlay=True)
doc[0].insertImage(rBR, pixmap=im, overlay=True)
doc[0].insertImage(rBL, pixmap=im, overlay=True)

doc.save('output.pdf')

[cbm755]

This is downstream https://gitlab.com/plom/plom/-/issues/1347

[JorjMcKie]

WTF ... my bad indeed. Did some unjustified assumptions on standard naming of embedded objects.
As a quick fix (which you presumably need!), do a page.clean_contents() before your insertions:

import fitz

# note this input has "idBox2.pdf" embedded inside (generated with pdflatex)
input = fitz.open("input.pdf")

# new doc with first page of input
doc = fitz.open()
doc.insertPDF(input, from_page=0, to_page=0, start_at=-1)
page = doc[0]
page.clean_contents()  # this will (also) rename the embedded PDF to "Fm1"
# just boxes for 4 corners, probably not hugely relevant
page_width = page.bound().width
page_height = page.bound().height
rTL = fitz.Rect(15, 20, 85, 90)
rTR = fitz.Rect(page_width - 85, 20, page_width - 15, 90)
rBL = fitz.Rect(15, page_height - 90, 85, page_height - 20)
rBR = fitz.Rect(page_width - 85, page_height - 90, page_width - 15, page_height - 20)

im = fitz.Pixmap("little.png")

# As soon as we overlay 2 or more images, the idBox2.pdf graphic disappears!
page.insertImage(rTL, pixmap=im, overlay=True)
page.insertImage(rTR, pixmap=im, overlay=True)
page.insertImage(rBR, pixmap=im, overlay=True)
page.insertImage(rBL, pixmap=im, overlay=True)

doc.save("output.pdf")

[JorjMcKie]

Another alternative: do not insert the template PDF page, but use page.showPDFpage(page.rect, input, 0) and then do you your inserts of images:

import fitz

# note this input has "idBox2.pdf" embedded inside (generated with pdflatex)
input = fitz.open("input.pdf")
rect = input[0].rect  # needed to make target page dimension

# new doc with first page of input
doc = fitz.open()
page = doc.newPage(width=rect.width, height=rect.height)
page.showPDFpage(rect, input, 0)  # fill new page with input

# just boxes for 4 corners, probably not hugely relevant
page_width = rect.width
page_height = rect.height
rTL = fitz.Rect(15, 20, 85, 90)
rTR = fitz.Rect(page_width - 85, 20, page_width - 15, 90)
rBL = fitz.Rect(15, page_height - 90, 85, page_height - 20)
rBR = fitz.Rect(page_width - 85, page_height - 90, page_width - 15, page_height - 20)

im = fitz.Pixmap("little.png")

# As soon as we overlay 2 or more images, the idBox2.pdf graphic disappears!
page.insertImage(rTL, pixmap=im, overlay=True)
page.insertImage(rTR, pixmap=im, overlay=True)
page.insertImage(rBR, pixmap=im, overlay=True)
page.insertImage(rBL, pixmap=im, overlay=True)

doc.save("output2.pdf")

No clean_contents() needed here.

[JorjMcKie]

Anyway, the error will be fixed in next version.

[cbm755]

I'll think about the workarounds: no hurry, we can just pin on 1.18.0 for now.

I dimly recall some (related?) weird behaviour if one includes pdf graphics using LaTeX's \scalebox, I'll see if I can make a minimal example.

[cbm755]

Unfortunately, I cannot reproduce the \scalebox/minipage thing. It was this:

• https://gitlab.com/plom/plom/-/issues/207
  And probably:
• https://gitlab.com/plom/plom/-/issues/125

The second one has a pdf file attached showing the result: https://gitlab.com/plom/plom/uploads/74aeb0076b59792efefe9b3a2db5a6c5/exam_0001.pdf

Unfortunately, I do not have the original input file, which is makes all this hard to confirm or reproduce!

Probably I will just close those issues, but just in case this rings a bell for you... In particular, I think \scalebox is uses some pdf trickery, something like making a another invisible page and including it via some reference.

[JorjMcKie]

Probably I will just close those issues, but just in case this rings a bell for you... In particular, I think \scalebox is uses some pdf trickery, something like making a another invisible page and including it via some reference.

Never heard of this before. Just followed the links you mentioned. It hurts to read comments like "fragility of pymupdf ..." and nobody ever bothered to open an issue!

I don't even understand where in all this PyMuPDF plays a role.

[JorjMcKie]

As per your issue, there are pre-release wheels for Linux and OSX in the respective branches of https://github.com/pymupdf/PyMuPDF-wheels repairing this.

[cbm755]

:( Yeah I apologize: it was a few years ago, we were rather new at this, it was a private repo at that time, and we were just pulling packages from Debian with no idea there was a vibrant maintained upstream.

Sorry for linking that here: that was thoughtless of me.

[cbm755]

The linux branch of https://github.com/pymupdf/PyMuPDF-wheels works great! Thanks!

Just FYI: I don't see the actual commits in this project (but quite possibly not looking in the right place).

[JorjMcKie]

Fixed by v1.18.7, currently being uploaded.