Skip to content

DataFusion Ray rewrite to connect stages with Arrow Flight Streaming #60

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 68 commits into from
Feb 13, 2025
Merged

DataFusion Ray rewrite to connect stages with Arrow Flight Streaming #60

Show file tree
Hide file tree
Changes from 58 commits
Commits
Show all changes
68 commits
Select commit Hold shift + click to select a range
e7917a2
wip
robtandy Dec 29, 2024
52a1258
wip
robtandy Dec 30, 2024
a651917
round trip to python and back working
robtandy Dec 31, 2024
e396c76
tips.py working
robtandy Jan 2, 2025
5714d1f
Merge pull request #1 from robtandy/rob.tandy/ray_shuffle
robtandy Jan 2, 2025
cffd046
workers die on tpch
robtandy Jan 3, 2025
86a0691
wip
robtandy Jan 4, 2025
e6e5f94
fixed race of multiple threads calling actor
robtandy Jan 5, 2025
acf9cb6
working but only for simple queries
robtandy Jan 10, 2025
493fa21
working with rewrite
robtandy Jan 12, 2025
8702412
Merge pull request #2 from robtandy/rob.tandy/ray_shuffle
robtandy Jan 12, 2025
4ad0ac1
distributed stages working
robtandy Jan 12, 2025
b46ebb5
Merge pull request #3 from robtandy/rob.tandy/ray_shuffle
robtandy Jan 12, 2025
62dcd0f
save point before arrow flight refactor
robtandy Jan 16, 2025
d68e05f
Merge pull request #4 from robtandy/rob.tandy/ray_shuffle
robtandy Jan 16, 2025
4174054
arrow flight wip
robtandy Jan 19, 2025
9b95dc3
arrow flight wip working for small example query
robtandy Jan 21, 2025
25fd5d7
arrow flight working with shadows
robtandy Jan 21, 2025
0002366
wip commit
robtandy Jan 22, 2025
c1abc25
flight working, still a bug in physical plan xfer
robtandy Jan 24, 2025
a29792e
wip
robtandy Jan 24, 2025
55ed9cf
add stats and do proper shutdown
robtandy Jan 25, 2025
0424b24
fix rounding issue with done fraction, and race condition calculating it
robtandy Jan 26, 2025
ad93bf6
compare with single node datafusion result for tpch
robtandy Jan 26, 2025
86a6ced
tidy up a bit of code and tpch benchmark
robtandy Jan 26, 2025
a90d7d1
add missing file after rename
robtandy Jan 26, 2025
7283d48
add register listing tables
robtandy Jan 26, 2025
561861d
fix listing table issue
robtandy Jan 26, 2025
7dd5497
timing for initial ray setup, fix bug in ray dataframe
robtandy Jan 26, 2025
587c03d
refactor to use an Exchange per Stage
robtandy Jan 27, 2025
304e22c
remove prints in codec
robtandy Jan 27, 2025
82bcb3f
parameterized number of exchangers
robtandy Jan 27, 2025
c41f76b
wip
robtandy Jan 27, 2025
699973f
tracking down deadlock WIP
robtandy Jan 28, 2025
c1ce07c
fixed nested lock deadlock
robtandy Jan 29, 2025
1c7f499
Merge pull request #5 from robtandy/rob.tandy/arrow-flight
robtandy Jan 29, 2025
27c00ab
randomly assign stages to exchanges
robtandy Jan 29, 2025
36aef70
Merge pull request #6 from robtandy/rob.tandy/arrow-flight
robtandy Jan 29, 2025
21fe59e
parameterize channel size
robtandy Jan 30, 2025
f04742c
do output partitions serially
robtandy Jan 30, 2025
ffac327
address SortExec and NestedLoopJoinExecs
robtandy Jan 30, 2025
73e3f92
make partition execution sequential optional
robtandy Jan 30, 2025
45cd0d1
wip
robtandy Jan 31, 2025
65f146d
wip add two utilities for large scale testing
robtandy Jan 31, 2025
aae8f86
probe for channel capacity
robtandy Jan 31, 2025
c88f147
wip add exchange method to inspect channel metadata
robtandy Jan 31, 2025
26c2724
wip fix exchange channel data bug
robtandy Jan 31, 2025
4c4dfc9
wip fix exchange channel data bug
robtandy Jan 31, 2025
83067e3
wip fix exchange channel data bug
robtandy Jan 31, 2025
3e8c079
add stage output of metrics
robtandy Feb 1, 2025
26f8c06
broken wip save
robtandy Feb 2, 2025
99da781
pull working roundtrip for tpch sf1 with shadows
robtandy Feb 3, 2025
f290ef0
add src/stage_service.rs I forgot
robtandy Feb 3, 2025
2ee7a78
add prefetch exec and some cleanup
robtandy Feb 3, 2025
7d18881
clean up for upstream PR
robtandy Feb 4, 2025
1154320
Merge pull request #7 from robtandy/rob.tandy/pull_arrow_flight
robtandy Feb 4, 2025
bae9329
change prints to log output. Set up proper logging
robtandy Feb 7, 2025
96398f1
Merge pull request #8 from robtandy/rob.tandy/pull_arrow_flight
robtandy Feb 7, 2025
e63c446
refactor to use configurable partitions per actor
robtandy Feb 9, 2025
07dd679
fix bug in splitting stages
robtandy Feb 10, 2025
09cf91e
fix bug in splitting stages II: return of fixing bug
robtandy Feb 10, 2025
d406242
capture settings in tpcbench results json
robtandy Feb 10, 2025
663ec64
fix bug in splitting stages III: this time its personal
robtandy Feb 11, 2025
0a7401d
hack until we upgrade to newest DataFusion to repair ParquetExec tabl…
robtandy Feb 11, 2025
b834e38
update docs
robtandy Feb 11, 2025
e8e0b36
Merge pull request #9 from robtandy/rob.tandy/pull_arrow_flight
robtandy Feb 11, 2025
af2d4c7
accommodate chagnes to asyncio.wait in python 3.11+
robtandy Feb 11, 2025
3248795
Merge pull request #10 from robtandy/rob.tandy/pull_arrow_flight
robtandy Feb 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 4 additions & 8 deletions .cargo/config.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,8 @@
[target.x86_64-apple-darwin]
rustflags = [
"-C", "link-arg=-undefined",
"-C", "link-arg=dynamic_lookup",
]
rustflags = ["-C", "link-arg=-undefined", "-C", "link-arg=dynamic_lookup"]

[target.aarch64-apple-darwin]
rustflags = [
"-C", "link-arg=-undefined",
"-C", "link-arg=dynamic_lookup",
]
rustflags = ["-C", "link-arg=-undefined", "-C", "link-arg=dynamic_lookup"]

[build]
rustflags = ["-C", "target-cpu=native"]
Loading
Loading