How does the zero copy works ?

[Reef3rm4n]

Im prob missing something but im unsure of why it returns a byteArray ? id expect it only to place bytes somewhere using the callback

[chaokunyang]

Hi @Reef3rm4n , Fury can serialize object to MemoryBuffer too, you can wrap an offheap buffer into MemoryBuffer by org.apache.fury.memory.MemoryUtils#buffer(long, int) or org.apache.fury.memory.MemoryUtils#wrap(java.nio.ByteBuffer). For object graph serialization, Fury use callback to place binary aside, it's like pickle5 out-of-band zero-copy.

For row-format, fury can read object data without deserializing it

[StevenCurran]

hey @chaokunyang Sorry to bring this thread up again, but felt its applicable instead of creating a new thread. In your example for zero copy, could you elaborate a bit on what is happening here? https://fury.apache.org/docs/guide/java_object_graph_guide#zero-copy-serialization

What are the BufferObjects used for, and what about the MemoryBuffers list used a couple of lines later? I am just trying to get an idea of the flow of data here, and where it is done off heap at each line. Could a comment on the lines after the objects are created in lst help? Also looking at the source code, toBuffer, and creating BufferObjects are on heap from MemoryBuffer.newHeapBuffer(this.totalBytes()), so could you provide a full offheap version of this?

  public static void main(String[] args) {
	//3 objects allocated that we want to serialize off heap
    List<Object> list = Arrays.asList("str", new byte[1000], new int[100], new double[100]);
	//Off heap BufferObjects that are pre allocated, and can be reused as a serialization / deserialization memory location? 
    Collection<BufferObject> bufferObjects = new ArrayList<>();
	//Serialize the objects from list out of band (off heap) into each of the buffer objects? aka 0 allocation done and bufferObjects can be reused? This returns a byte[] though, so there is heap allocation happening? 
    byte[] bytes = fury.serialize(list, e -> !bufferObjects.add(e));
	//Then we take those objects and put them into a bytebuffer, also off heap? But return this list? Confused as to why
    List<MemoryBuffer> buffers = bufferObjects.stream()
      .map(BufferObject::toBuffer).collect(Collectors.toList());
	//Move these back on heap when using the deserialize, by using the pre allocated buffers?
    System.out.println(fury.deserialize(bytes, buffers));
  }
}

I have tried to put in comments as to what I believe, but perhaps it is not so clear to me what is happening here. Any guidance would be geatly appreciated.

Many thanks!

[chaokunyang]

Hi @Reef3rm4n , List<MemoryBuffer> buffers = bufferObjects.stream().map(BufferObject::toBuffer).collect(Collectors.toList()); are just an example. In your production case, you should not invoke such methods. You should invoke org.apache.fury.serializer.BufferObject#writeTo to write the data into the memory address you provided. For example, you can allocate a buffer from shared off heap memory, then you wrap that buffer into fury MemoryBuffer by MemoryUtils.buffer(long, int). Then you can invoke BufferObject#writeTo too write the data into shared memory directly.

[chaokunyang]

We are considering add writeTo(WriteableChannel) to BufferObject. THis will make it more easy to use