[Transfer Engine]Coro rpc communicator #795

JasonZhang517 · 2025-09-01T11:40:01Z

add a communicator including server and client pool
add pybind interface of the communicator
add test files

stmatengss · 2025-09-01T11:44:31Z

mooncake-integration/CMakeLists.txt

 pybind11_add_module(engine ${SOURCES} ${CACHE_ALLOCATOR_SOURCES}
    transfer_engine/transfer_engine_py.cpp 
+    ../mooncake-transfer-engine/src/transport/coro_rpc_connector/cororpc_interface.cpp
+    ../mooncake-transfer-engine/src/transport/coro_rpc_connector/cororpc_communicator.cpp


Why need to add these files here?

if we want to use coro_rpc_connector as a submodule of mooncake transfer engine, we have to compile it here

mooncake-transfer-engine/src/transport/coro_rpc_connector/cororpc_communicator.cpp

poor-circle · 2025-09-02T09:33:54Z

mooncake-transfer-engine/src/transport/coro_rpc_connector/cororpc_communicator.cpp

+    return false;
+}
+
+int CoroRPCCommunicator::sendData(const std::string& target_address,


sendData should be a wrapper of sendDataAsync, dont rewrite same code twice

use syncAwait(Lazy{}) as impl

mooncake-transfer-engine/src/transport/coro_rpc_connector/cororpc_communicator.cpp

mooncake-transfer-engine/src/transport/coro_rpc_connector/cororpc_interface.cpp

qicosmos · 2025-09-03T03:03:24Z

mooncake-transfer-engine/src/transport/coro_rpc_connector/cororpc_interface.cpp

+    auto future_ptr = std::make_shared<pybind11::object>(future_obj);
+    auto loop_ptr = std::make_shared<pybind11::object>(pybind11::reinterpret_borrow<pybind11::object>(loop));
+
+    auto task_func = std::make_shared<std::function<void()>>(


send_request or
auto lazy = [](int i) -> async_simple::coro::Lazy {
co_await c->send_data_async();
}

lazy(i).start({});

qicosmos · 2025-09-03T03:03:51Z

mooncake-transfer-engine/src/transport/coro_rpc_connector/cororpc_interface.cpp

+
+    auto task_func = std::make_shared<std::function<void()>>(
+        [communicator, target_addr, data_holder, future_ptr, loop_ptr]() {
+        int result = communicator->sendData(*target_addr, data_holder->data(), data_holder->size());


co_await communicator->sendDataAsync

qicosmos · 2025-09-03T03:08:14Z

mooncake-transfer-engine/src/transport/coro_rpc_connector/cororpc_interface.cpp

+    return future_obj;
+}
+
+int CoroRPCInterface::sendTensor(const std::string& target_address, pybind11::handle tensor) {


according async_send_tensor

mooncake-transfer-engine/src/transport/coro_rpc_connector/cororpc_interface.cpp

qicosmos · 2025-09-03T03:10:40Z

mooncake-transfer-engine/src/transport/coro_rpc_connector/cororpc_interface.cpp

+    return future_obj;
+}
+
+void CoroRPCInterface::setDataReceiveCallback(pybind11::function callback) {


use asyncio future

…ronous coroutine 3. change std::future to lazy return value

mooncake-transfer-engine/include/transport/coro_rpc_connector/cororpc_communicator.h

mooncake-transfer-engine/include/transport/coro_rpc_connector/cororpc_interface.h

mooncake-transfer-engine/src/transport/coro_rpc_connector/cororpc_communicator.cpp

mooncake-transfer-engine/src/transport/coro_rpc_connector/cororpc_interface.cpp

stmatengss · 2025-09-08T08:43:31Z

@JasonZhang517 Do you happen to have any updates for this PR?

mooncake-transfer-engine/tests/communicator_bandwidth_test.py

stmatengss · 2025-09-28T02:35:09Z

.github/workflows/ci.yml

      run: ${SCCACHE_PATH} --show-stats

    - name: Install dependencies
+      shell: bash


why move it here?

.gitignore

…_rpc_communicator

Copilot

Pull Request Overview

This PR adds support for CoRo RPC communication to the Mooncake transfer engine, enabling bandwidth testing between client and server using the yalantinglibs coro_rpc framework. The implementation provides both synchronous and asynchronous APIs for data and tensor transfer with Python bindings.

Key changes:

Added CoroRPCInterface and CoroRPCCommunicator classes for RPC communication
Integrated pybind11 for Python bindings with improved CMake configuration
Created a bandwidth test tool to measure communication performance

Reviewed Changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 17 comments.

Show a summary per file

File	Description
mooncake-transfer-engine/tests/communicator_bandwidth_test.py	New bandwidth test script for measuring RPC communication throughput
mooncake-transfer-engine/src/transport/coro_rpc_connector/cororpc_interface.cpp	Core RPC interface implementation with data/tensor transfer methods
mooncake-transfer-engine/src/transport/coro_rpc_connector/cororpc_communicator.cpp	RPC communicator implementation handling client pools and server operations
mooncake-transfer-engine/src/transport/coro_rpc_connector/CMakeLists.txt	Build configuration for coro_rpc_connector module
mooncake-transfer-engine/src/transport/CMakeLists.txt	Updated to include coro_rpc_connector subdirectory
mooncake-transfer-engine/src/CMakeLists.txt	Added pybind11 configuration and Python dependencies
mooncake-transfer-engine/include/transport/coro_rpc_connector/cororpc_interface.h	Header defining CoroRPCInterface API
mooncake-transfer-engine/include/transport/coro_rpc_connector/cororpc_communicator.h	Header defining CoroRPCCommunicator API
mooncake-integration/transfer_engine/transfer_engine_py.cpp	Python bindings for CoroRPCInterface
mooncake-integration/CMakeLists.txt	Updated to link coro_rpc_connector
CMakeLists.txt	Enhanced pybind11 discovery logic
.gitignore	Whitespace fix

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mooncake-integration/transfer_engine/transfer_engine_py.cpp

mooncake-transfer-engine/tests/communicator_bandwidth_test.py

mooncake-transfer-engine/src/transport/coro_rpc_connector/cororpc_interface.cpp

mooncake-transfer-engine/tests/communicator_bandwidth_test.py

Co-authored-by: Copilot <[email protected]>

Format function call parameters to conform to .clang-format style.

JasonZhang517 added 2 commits August 29, 2025 19:43

add basic implementation and python interfaces

e63fa88

add real communicator logic and interface

22e59a0

stmatengss reviewed Sep 1, 2025

View reviewed changes

JasonZhang517 added 8 commits September 1, 2025 19:47

add integration test

998a1c0

added tensor rebuild logic

f8cee4d

add tensor rebuild and tests

81cbfdd

removed useless lines

714ef16

removed blank lines

59c5105

turn single client creation to client pool

06a672f

merged conflicts

1c225b9

Merge remote-tracking branch 'upstream/main' into coro_rpc_communicator

1fc1ee0

poor-circle reviewed Sep 2, 2025

View reviewed changes

change std::future to async_simple coro lazy

f5deb6c

stmatengss requested a review from poor-circle September 2, 2025 12:16

qicosmos reviewed Sep 3, 2025

View reviewed changes

stmatengss requested a review from qicosmos September 4, 2025 03:45

JasonZhang517 added 6 commits September 4, 2025 11:50

1. removed some data copies 2. replace synchronous thread with asynch…

65ee0c8

…ronous coroutine 3. change std::future to lazy return value

1.added tests to ci 2. removed CLAUDE.md

8c061e3

add pybind installation in CI

aed64b1

fixed clang spelling

08146aa

fixed communicator spelling

584e386

fixed redundant functions in transfer engine py

e58e2c3

JasonZhang517 changed the title ~~Coro rpc communicator~~ [Transfer Engine]Coro rpc communicator Sep 4, 2025

fixed ci path

8953508

JasonZhang517 force-pushed the coro_rpc_communicator branch from 4162d03 to 8953508 Compare September 5, 2025 00:09

qicosmos reviewed Sep 5, 2025

View reviewed changes

JasonZhang517 added 3 commits September 8, 2025 15:23

fixed memory copy bugs, removed client pool creation

b2afec2

convert memorycopy to memoryview

fe165e9

set attachment for handleDataTransfer

a1d4c64

JasonZhang517 added 3 commits September 15, 2025 15:40

refactored client and server initialization methods

2b83a76

fixed server initialization pool size

ed76b8b

fixed interface pybind bugs

600db58

stmatengss reviewed Sep 16, 2025

View reviewed changes

mooncake-transfer-engine/tests/communicator_bandwidth_test.py Outdated Show resolved Hide resolved

fixed pybind

abd15a1

JasonZhang517 force-pushed the coro_rpc_communicator branch 2 times, most recently from 77b6863 to abd15a1 Compare September 23, 2025 07:50

JasonZhang517 added 5 commits September 23, 2025 16:05

update pybind11 parameters to -Dpybind11_DIR

20b1bc2

update cmakelists

fc7dde6

local py312 test complete

c790f21

adjused workflow

255a5a9

adjusted Cmakelists.txxt

61b03da

stmatengss reviewed Sep 28, 2025

View reviewed changes

restored some useless changes

2ac0105

stmatengss reviewed Sep 30, 2025

View reviewed changes

.gitignore Show resolved Hide resolved

JasonZhang517 added 4 commits October 10, 2025 20:37

remove Chinese comments

41e871f

remove blank lines

ba6bd7f

Merge remote-tracking branch 'origin/coro_rpc_communicator' into coro…

384e3b1

…_rpc_communicator

Merge branch 'main' into coro_rpc_communicator

2130bf0

stmatengss requested review from Copilot and removed request for poor-circle November 4, 2025 05:52

Copilot AI reviewed Nov 4, 2025

View reviewed changes

JasonZhang517 and others added 5 commits November 4, 2025 16:55

Apply suggestions from code review

479df8e

Co-authored-by: Copilot <[email protected]>

add initialization

164a368

style: apply clang-format to transfer_engine_py.cpp

bde6c37

Format function call parameters to conform to .clang-format style.

Merge branch 'main' into coro_rpc_communicator

9180b14

fixed cmakelists.txt

e260771

stmatengss added the high priority label Nov 17, 2025

JasonZhang517 added 2 commits November 21, 2025 15:35

Merge branch 'kvcache-ai:main' into coro_rpc_communicator

bdd86ec

Merge branch 'kvcache-ai:main' into coro_rpc_communicator

ae0e43b

[Transfer Engine]Coro rpc communicator #795

Are you sure you want to change the base?

[Transfer Engine]Coro rpc communicator #795

Uh oh!

Conversation

JasonZhang517 commented Sep 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

poor-circle Sep 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

stmatengss commented Sep 8, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

poor-circle Sep 2, 2025 •

edited

Loading