How to run a executorch model directly from memory instead of saving it as a disk file

[vikasbalaga]

📚 The doc issue

Hi,

I wanted to know if there is any ExecuTorch runtime API that can accept a *.pte model available in the memory (in some sort of a buffer format) and use it to do load and infer?

So far, I could only find a few which require the model to be passed as a disk file.

Suggest a potential alternative/fix

No response

[lucylq]

Hi @vikasbalaga, could you provide some more context on what you're trying to do?

There is a pybindings _load_for_executorch_from_buffer API that you could try:

executorch/extension/pybindings/pybindings.pyi

Lines 175 to 180 in bc0ad82

	def _load_for_executorch_from_buffer(
	buffer: bytes,
	enable_etdump: bool = False,
	debug_buffer_size: int = 0,
	program_verification: Verification = Verification.InternalConsistency,
	) -> ExecuTorchModule:

On the C++ side, there is also a BufferDataLoader:

executorch/extension/data_loader/buffer_data_loader.h

Line 27 in bc0ad82

class BufferDataLoader final : public executorch::runtime::DataLoader {

[vikasbalaga]

Hi @lucylq, thanks for your response.

My requirement is that, I am trying to encrypt my models for security and decrypt them on the fly in C++. So during the process, I don't want to save the decrypted models and I need to directly run inference from the model buffer in memory (which I will get after decryption).

So, in one line, I will have a model buffer of data type std::vector<unsigned char>, and are there any APIs which will directly take this buffer and perform inference, without needing the model as a disk file (*.pte)

[vikasbalaga]

Also, as I go through the class BufferDataLoader final : public executorch::runtime::DataLoader { , it seems to me that, it will be suitable for my requirement (can you confirm?).

If so, is there any documentation on how to use it, as I could only find info related to loading the model from file using "Module" class here

[lucylq]

Hi @vikasbalaga, thanks for the context!

Regarding how to use the BufferDataLoader, you can take a look at the tests here: https://github.com/pytorch/executorch/blob/main/extension/data_loader/test/buffer_data_loader_test.cpp

For usage outside of the Module API, you can try using the lower-level APIs, e.g. executor_runner: https://github.com/pytorch/executorch/blob/main/examples/portable/executor_runner/executor_runner.cpp#L161

The executor_runner example here also uses the FileDataLoader, but you can be replace it with the BufferDataLoader.

Module currently takes in a file. We could potentially add an overload that takes in a DataLoader as well. cc @shoumikhin

[shoumikhin]

@lucylq ptal at https://github.com/pytorch/executorch/blob/main/extension/module/module.h#L94

[lucylq]

Oh, thanks @shoumikhin! I missed that.

@vikasbalaga does that work for you?

[vikasbalaga]

@lucylq and @shoumikhin, thanks for your help and yes, the mentioned overloaded variant for Module is working?

Thanks for your support!

[vikasbalaga]

Hi again, I just have a small question regarding the "model setup time", when :

1. Model is loaded from a disk file using Module(const std::string& file_path, const std::string& data_map_path,...)
2. Model is loaded directly from memory using Module(std::unique_ptr<runtime::DataLoader> data_loader),,...)

I would expect that, the time taken for Option1 should be higher than Option2 (as in former case, the runtime needs to load from file system whereas in later case, it is already available in memory)

But the results I see are different, i.e. Option2 is taking more time for setup than Option1

Is this expected?
Thanks.

[lucylq]

Sorry for the delayed reply @vikasbalaga

I think you're right that Option1 should take a bit longer than Option2. Is the difference large, and how were you testing?

I know that we've found FileDataLoader to be faster than MmapDataLoader in the past, but I'm not sure about the BufferDataLoader.