Support custom data sources

[bvitaliyg]

Currently, PdfiumAndroid, and subsequently, PdfiumAndroidKt support only 2 data sources: ByteArray and ParcelFileDescriptor. While this works for most of the cases, sometimes you might need to load the data from InputStream or remote file (for instance, using something like Apache VFS). There's no ParcelFileDescriptor in such cases, and loading the entire content to the RAM is very inefficient and could lead to OOMs.

I checked out the source, and it turns out, Pdfium does support custom data sources – and it's already used in the project:

PdfiumAndroidKt/pdfiumandroid/src/main/cpp/pdfiumandroid.cpp

Line 249 in c82f5c6

FPDF_DOCUMENT document = FPDF_LoadCustomDocument(&loader, cpassword);

So it could be possible to have a custom data source. All that needed to be done is creating an interface for representing the data source, adding corresponding newDocument() methods to PdfiumCore, and implement these methods in C++, using JNI to call callback methods from that interface.

Can this be implemented? I could try to make the PR if needed.

[johngray1965]

Hello, if you can try making a on PR, that would be nice. Just as long as it could be done without making changes the native libraries (the pdfium so's). I don't, generally, accept PR this requires changes the native binaries in the jniLibs directory. Just making changes to the code in this repository. Changes to the C++ code in the repository is fine (and would be necessary).

I just spent a few minutes researching what would be needed to do this, and looks fairly straight forward.

Any new API should have tests, and ideally be exposed by all 3 interfaces (the basic interface, the coroutines interface, and the arrow interface). I can implement the coroutine and arrow interfaces if need be.

I might implement it myself, I'm just not sure when. If you want make a PR, go ahead. Just leave a comment saying you're going work on it. I'll try to do the same so we aren't both working on it.

[johngray1965]

As much as I should be tending to other things, I will make quick pass at this now.

[johngray1965]

Here's the issue:

typedef struct {
  // File length, in bytes.
  unsigned long m_FileLen;

  // A function pointer for getting a block of data from a specific position.
  // Position is specified by byte offset from the beginning of the file.
  // The pointer to the buffer is never NULL and the size is never 0.
  // The position and size will never go out of range of the file length.
  // It may be possible for PDFium to call this function multiple times for
  // the same position.
  // Return value: should be non-zero if successful, zero for error.
  int (*m_GetBlock)(void* param,
                    unsigned long position,
                    unsigned char* pBuf,
                    unsigned long size);

  // A custom pointer for all implementation specific data.  This pointer will
  // be used as the first parameter to the m_GetBlock callback.
  void* m_Param;
} FPDF_FILEACCESS;

Notice in comment: It may be possible for PDFium to call this function multiple times for the same position.

I believe it needs random access. It not reading the PDF straight through in one pass.

I had same issue you having (I need to load a PDF where I had was an inputStream). I copied the file locally, and feed the local version in. It's not elegant. But it works.

[bvitaliyg]

Yep, it needs random access – and it jump pretty heavily TBF. However the implementation can also work with InputStreams, assuming you can recreate the stream and skip the bytes. I've already done it for my project and it seems to be working, just pretty slow – 71 seconds to load 18Mb PDF inside ZIP file :D Using Okio, it can be optimised up to 29 seconds.

In any event, it's a matter of underlying implementation. I think we can just provide an interface with 2 methods, something like:

val length: Long

fun read(position: Long, buffer: ByteArray, size: Int): Int

And then the users can implement it with anything they want – RandomAccess, streams hacks, etc.

I can write the tests as well, but I have no experience working with Arrow. Can open the PR today as I already have the code.

[johngray1965]

I asked Gemini, it thinks we can do it (though it's hardly reliably).

Okay, go head and do what you can do. And I can do the arrow portion (its trivial).

[johngray1965]

I was a little unsure if you were initially thinking a generic solution or something specifically for a inputStream. A generic solution is great.

[johngray1965]

Closing this now. It'll be available in the next release. I hope to fix #25 then push out a new release.

[bvitaliyg]

Thank you! Can I ask you to release it with AutoCloseable for PdfiumSource being supported? I can make another PR if needed.

[johngray1965]

Question for you. the JNI is returning native pointer to the document to the Java/Kotlin code, likewise for Pages and Text pages, and now the TextLinkis. Should they all be Global References?

[johngray1965]

Actually, I just looked it up. I believe they should all be Global References.

[bvitaliyg]

If I understood you correctly, when you're returning C/C++ pointer to store in Java/Kotlin code, the common practice to just to case this pointer to long and store in in JVM code as Long. Then, whenever you need to do something with this pointer (e.g. call a method or access a field), you just pass it as long argument to native method and cast it back to the pointer in JNI.

[bvitaliyg]

If you want to do the opposite (store Java pointer in C/C++ code), then you need to use the References. You can either use GlobalRef(), or you can use WeakGlobalRef(), which will be valid as long as Java code as live reference for the object. E.g. doing something like callMyNativeMethod(new JavaObject()) won't store valid reference to JavaObject with WeakGlobalRef, but if you put JavaObject to field and then null the field when it's no logner needed, it should be okay from what I remember.

[bvitaliyg]

You can have a look on PdfTextPage to see an example of storing C pointer in Java, and usually you don't need to do the opposite unless you need to provide Java callback for C code.

[johngray1965]

Ah, thank you. We are just casting the C++ pointers to longs. And passing them back to JVM land. So I think it's fine with what we currently have.