Skip to content

Introduce MAX_INLINE_VIEW_LEN constant for string/byte views #7719

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jun 25, 2025
Merged

Introduce MAX_INLINE_VIEW_LEN constant for string/byte views #7719

merged 2 commits into from
Jun 25, 2025
+44 −34

Conversation

@alamb
Copy link
Contributor

@alamb alamb commented Jun 20, 2025 •
edited
Loading

Which issue does this PR close?

As suggested by @Dandandan in #7650 (comment):

We probably should set this as constant somewhere and use it

Rationale for this change

Using a symbolic constant in the code rather than a hard coded constant makes it easier to:

  1. Understand what the value means
  2. Link / attach documentation to the constant to provide context

What changes are included in this PR?

  1. Introduce MAX_INLINE_VIEW_LEN constant for string/byte views
  2. Update code to use that instead of 12

Are there any user-facing changes?

A new constant

@alamb
Copy link
Contributor Author

alamb commented Jun 20, 2025

I will also run some benchmarks on this PR to make sure it doesn't mess with anything

@alamb
Copy link
Contributor Author

alamb commented Jun 20, 2025

🤖 ./gh_compare_arrow.sh Benchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/STRING_VIEW_LENGTH (1a38b68) to 1bed04c diff
BENCH_NAME=concatenate_kernel
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench concatenate_kernel
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_STRING_VIEW_LENGTH
Results will be posted here when complete

@alamb
Copy link
Contributor Author

alamb commented Jun 20, 2025

🤖: Benchmark completed

Details 

group                                                          alamb_STRING_VIEW_LENGTH               main
-----                                                          ------------------------               ----
concat 1024 arrays boolean 4                                   1.00     28.7±0.03µs        ? ?/sec    1.01     28.8±0.11µs        ? ?/sec
concat 1024 arrays i32 4                                       1.00     15.6±0.04µs        ? ?/sec    1.00     15.6±0.06µs        ? ?/sec
concat 1024 arrays str 4                                       1.01     56.6±0.31µs        ? ?/sec    1.00     55.8±0.41µs        ? ?/sec
concat boolean 1024                                            1.00    428.0±0.60ns        ? ?/sec    1.05    450.3±0.54ns        ? ?/sec
concat boolean 8192 over 100 arrays                            1.00     50.9±0.08µs        ? ?/sec    1.00     50.9±0.09µs        ? ?/sec
concat boolean nulls 1024                                      1.00    742.8±1.87ns        ? ?/sec    1.05    781.0±1.42ns        ? ?/sec
concat boolean nulls 8192 over 100 arrays                      1.00    109.4±0.21µs        ? ?/sec    1.00    109.6±0.18µs        ? ?/sec
concat fixed size lists                                        1.00   784.5±24.57µs        ? ?/sec    1.04   817.4±28.14µs        ? ?/sec
concat i32 1024                                                1.00    435.3±0.69ns        ? ?/sec    1.01    437.5±0.62ns        ? ?/sec
concat i32 8192 over 100 arrays                                1.03    212.7±7.58µs        ? ?/sec    1.00    206.3±3.42µs        ? ?/sec
concat i32 nulls 1024                                          1.00    758.6±1.03ns        ? ?/sec    1.01    767.0±2.07ns        ? ?/sec
concat i32 nulls 8192 over 100 arrays                          1.00    277.2±6.32µs        ? ?/sec    1.01    281.1±9.59µs        ? ?/sec
concat str 1024                                                1.00     13.3±1.08µs        ? ?/sec    1.03     13.7±1.05µs        ? ?/sec
concat str 8192 over 100 arrays                                1.00    105.3±0.69ms        ? ?/sec    1.00    105.8±0.86ms        ? ?/sec
concat str nulls 1024                                          1.07      6.7±0.66µs        ? ?/sec    1.00      6.2±0.51µs        ? ?/sec
concat str nulls 8192 over 100 arrays                          1.00     51.8±0.56ms        ? ?/sec    1.03     53.3±0.34ms        ? ?/sec
concat str_dict 1024                                           1.00      2.9±0.01µs        ? ?/sec    1.09      3.1±0.02µs        ? ?/sec
concat str_dict_sparse 1024                                    1.00      6.9±0.03µs        ? ?/sec    1.00      6.9±0.02µs        ? ?/sec
concat struct with int32 and dicts size=1024 count=2           1.00      6.5±0.03µs        ? ?/sec    1.05      6.8±0.10µs        ? ?/sec
concat utf8_view  max_str_len=128 null_density=0               1.00     77.6±0.37µs        ? ?/sec    1.00     77.8±0.34µs        ? ?/sec
concat utf8_view  max_str_len=128 null_density=0.2             1.04     87.6±0.33µs        ? ?/sec    1.00     84.4±0.36µs        ? ?/sec
concat utf8_view  max_str_len=20 null_density=0                1.05     81.4±0.43µs        ? ?/sec    1.00     77.7±0.52µs        ? ?/sec
concat utf8_view  max_str_len=20 null_density=0.2              1.04     87.2±0.15µs        ? ?/sec    1.00     84.0±0.47µs        ? ?/sec
concat utf8_view all_inline max_str_len=12 null_density=0      1.00     46.5±2.95µs        ? ?/sec    1.01     47.0±3.44µs        ? ?/sec
concat utf8_view all_inline max_str_len=12 null_density=0.2    1.04     55.8±3.70µs        ? ?/sec    1.00     53.8±4.05µs        ? ?/sec

@alamb
Copy link
Contributor Author

alamb commented Jun 20, 2025

🤖 ./gh_compare_arrow.sh Benchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/STRING_VIEW_LENGTH (1a38b68) to 1bed04c diff
BENCH_NAME=coalesce_kernels
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench coalesce_kernels
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_STRING_VIEW_LENGTH
Results will be posted here when complete

@alamb
Copy link
Contributor Author

alamb commented Jun 20, 2025

🤖: Benchmark completed

Details 

group                                                                                alamb_STRING_VIEW_LENGTH               main
-----                                                                                ------------------------               ----
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.001                               1.01    302.9±2.41ms        ? ?/sec    1.00    300.0±2.10ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.01                                1.00      9.1±0.10ms        ? ?/sec    1.01      9.2±0.11ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.1                                 1.02      4.3±0.14ms        ? ?/sec    1.00      4.2±0.09ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0, selectivity: 0.8                                 1.01      3.5±0.02ms        ? ?/sec    1.00      3.5±0.03ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.001                             1.01    297.5±2.49ms        ? ?/sec    1.00    294.5±2.38ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.01                              1.00     10.6±0.08ms        ? ?/sec    1.00     10.7±0.08ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.1                               1.00      4.7±0.12ms        ? ?/sec    1.00      4.7±0.11ms        ? ?/sec
filter: mixed_dict, 8192, nulls: 0.1, selectivity: 0.8                               1.00      4.7±0.02ms        ? ?/sec    1.00      4.7±0.02ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.001                               1.00     72.4±0.63ms        ? ?/sec    1.00     72.1±0.70ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.01                                1.01     12.7±0.17ms        ? ?/sec    1.00     12.7±0.19ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.1                                 1.00     10.2±0.38ms        ? ?/sec    1.00     10.2±0.33ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0, selectivity: 0.8                                 1.02      8.5±0.19ms        ? ?/sec    1.00      8.4±0.17ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.001                             1.00     94.9±0.59ms        ? ?/sec    1.00     94.4±0.70ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.01                              1.00     15.0±0.09ms        ? ?/sec    1.01     15.1±0.11ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.1                               1.00     10.0±0.36ms        ? ?/sec    1.00     10.0±0.30ms        ? ?/sec
filter: mixed_utf8, 8192, nulls: 0.1, selectivity: 0.8                               1.01      9.9±0.17ms        ? ?/sec    1.00      9.8±0.14ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.001      1.01     58.1±0.23ms        ? ?/sec    1.00     57.4±0.22ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.01       1.00      7.4±0.03ms        ? ?/sec    1.00      7.4±0.03ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.1        1.04      5.0±0.23ms        ? ?/sec    1.00      4.8±0.24ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0, selectivity: 0.8        1.01      3.2±0.04ms        ? ?/sec    1.00      3.2±0.03ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.001    1.01     78.9±0.36ms        ? ?/sec    1.00     78.3±0.30ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.01     1.01     10.6±0.03ms        ? ?/sec    1.00     10.5±0.04ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.1      1.03      5.7±0.26ms        ? ?/sec    1.00      5.5±0.24ms        ? ?/sec
filter: mixed_utf8view (max_string_len=128), 8192, nulls: 0.1, selectivity: 0.8      1.03      3.9±0.01ms        ? ?/sec    1.00      3.8±0.01ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.001       1.01     50.9±0.60ms        ? ?/sec    1.00     50.4±0.20ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.01        1.00      6.1±0.03ms        ? ?/sec    1.00      6.1±0.02ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.1         1.04      2.8±0.22ms        ? ?/sec    1.00      2.7±0.19ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0, selectivity: 0.8         1.00      2.3±0.01ms        ? ?/sec    1.01      2.3±0.01ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.001     1.01     69.6±0.35ms        ? ?/sec    1.00     69.0±0.28ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.01      1.01      9.3±0.04ms        ? ?/sec    1.00      9.2±0.02ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.1       1.06      3.7±0.20ms        ? ?/sec    1.00      3.5±0.05ms        ? ?/sec
filter: mixed_utf8view (max_string_len=20), 8192, nulls: 0.1, selectivity: 0.8       1.00      4.6±0.01ms        ? ?/sec    1.00      4.6±0.01ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.001                          1.00     62.3±0.13ms        ? ?/sec    1.00     62.1±0.19ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.01                           1.00      7.7±0.02ms        ? ?/sec    1.00      7.7±0.02ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.1                            1.04      4.8±0.25ms        ? ?/sec    1.00      4.6±0.21ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0, selectivity: 0.8                            1.02      3.8±0.02ms        ? ?/sec    1.00      3.7±0.04ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.001                        1.00     80.1±0.30ms        ? ?/sec    1.00     80.0±0.27ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.01                         1.01     11.3±0.04ms        ? ?/sec    1.00     11.2±0.05ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.1                          1.02      6.7±0.22ms        ? ?/sec    1.00      6.5±0.19ms        ? ?/sec
filter: single_utf8view, 8192, nulls: 0.1, selectivity: 0.8                          1.02      6.3±0.02ms        ? ?/sec    1.00      6.2±0.03ms        ? ?/sec

@alamb
Copy link
Contributor Author

alamb commented Jun 20, 2025

🤖 ./gh_compare_arrow.sh Benchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/STRING_VIEW_LENGTH (1a38b68) to 1bed04c diff
BENCH_NAME=filter_kernels
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench filter_kernels
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_STRING_VIEW_LENGTH
Results will be posted here when complete

@alamb
Copy link
Contributor Author

alamb commented Jun 20, 2025

🤖: Benchmark completed

Details 

group                                                                         alamb_STRING_VIEW_LENGTH               main
-----                                                                         ------------------------               ----
filter context decimal128 (kept 1/2)                                          1.00     42.5±4.74µs        ? ?/sec    1.21     51.3±7.56µs        ? ?/sec
filter context decimal128 high selectivity (kept 1023/1024)                   1.00     49.6±0.71µs        ? ?/sec    1.02     50.6±1.48µs        ? ?/sec
filter context decimal128 low selectivity (kept 1/1024)                       1.00    246.1±2.10ns        ? ?/sec    1.02    249.8±0.45ns        ? ?/sec
filter context f32 (kept 1/2)                                                 1.00     90.7±0.19µs        ? ?/sec    1.00     91.1±0.19µs        ? ?/sec
filter context f32 high selectivity (kept 1023/1024)                          1.00     12.9±0.43µs        ? ?/sec    1.00     12.9±0.51µs        ? ?/sec
filter context f32 low selectivity (kept 1/1024)                              1.00    473.4±0.40ns        ? ?/sec    1.05    499.3±1.20ns        ? ?/sec
filter context fsb with value length 20 (kept 1/2)                            1.00     71.0±0.27µs        ? ?/sec    1.00     71.0±0.15µs        ? ?/sec
filter context fsb with value length 20 high selectivity (kept 1023/1024)     1.00     71.0±0.31µs        ? ?/sec    1.00     71.0±0.11µs        ? ?/sec
filter context fsb with value length 20 low selectivity (kept 1/1024)         1.00     71.0±0.32µs        ? ?/sec    1.00     71.0±0.12µs        ? ?/sec
filter context fsb with value length 5 (kept 1/2)                             1.00     71.0±0.21µs        ? ?/sec    1.00     71.0±0.16µs        ? ?/sec
filter context fsb with value length 5 high selectivity (kept 1023/1024)      1.00     71.0±0.30µs        ? ?/sec    1.00     70.9±0.10µs        ? ?/sec
filter context fsb with value length 5 low selectivity (kept 1/1024)          1.00     70.9±0.21µs        ? ?/sec    1.00     70.9±0.09µs        ? ?/sec
filter context fsb with value length 50 (kept 1/2)                            1.00     71.0±0.27µs        ? ?/sec    1.00     70.9±0.07µs        ? ?/sec
filter context fsb with value length 50 high selectivity (kept 1023/1024)     1.00     71.0±0.28µs        ? ?/sec    1.00     70.9±0.08µs        ? ?/sec
filter context fsb with value length 50 low selectivity (kept 1/1024)         1.00     71.0±0.27µs        ? ?/sec    1.00     70.9±0.09µs        ? ?/sec
filter context i32 (kept 1/2)                                                 1.00     22.7±0.20µs        ? ?/sec    1.00     22.7±0.07µs        ? ?/sec
filter context i32 high selectivity (kept 1023/1024)                          1.02      6.3±0.39µs        ? ?/sec    1.00      6.2±0.28µs        ? ?/sec
filter context i32 low selectivity (kept 1/1024)                              1.00    248.7±1.53ns        ? ?/sec    1.02    253.5±0.38ns        ? ?/sec
filter context i32 w NULLs (kept 1/2)                                         1.00     94.2±0.54µs        ? ?/sec    1.00     94.4±0.62µs        ? ?/sec
filter context i32 w NULLs high selectivity (kept 1023/1024)                  1.00     13.0±0.52µs        ? ?/sec    1.02     13.3±0.42µs        ? ?/sec
filter context i32 w NULLs low selectivity (kept 1/1024)                      1.13    573.4±2.14ns        ? ?/sec    1.00    506.8±3.00ns        ? ?/sec
filter context mixed string view (kept 1/2)                                   1.02    117.0±7.11µs        ? ?/sec    1.00    114.5±6.72µs        ? ?/sec
filter context mixed string view high selectivity (kept 1023/1024)            1.00     56.9±1.35µs        ? ?/sec    1.04     58.9±1.07µs        ? ?/sec
filter context mixed string view low selectivity (kept 1/1024)                1.00    665.0±3.65ns        ? ?/sec    1.04    689.4±0.86ns        ? ?/sec
filter context short string view (kept 1/2)                                   1.01    116.1±6.93µs        ? ?/sec    1.00    115.5±6.10µs        ? ?/sec
filter context short string view high selectivity (kept 1023/1024)            1.02     57.0±1.91µs        ? ?/sec    1.00     55.6±1.65µs        ? ?/sec
filter context short string view low selectivity (kept 1/1024)                1.00    490.2±2.55ns        ? ?/sec    1.02    502.4±0.65ns        ? ?/sec
filter context string (kept 1/2)                                              1.00   597.3±12.64µs        ? ?/sec    1.00   594.6±12.94µs        ? ?/sec
filter context string dictionary (kept 1/2)                                   1.00     23.4±0.07µs        ? ?/sec    1.00     23.3±0.05µs        ? ?/sec
filter context string dictionary high selectivity (kept 1023/1024)            1.00      7.1±0.29µs        ? ?/sec    1.07      7.6±0.38µs        ? ?/sec
filter context string dictionary low selectivity (kept 1/1024)                1.01    835.2±5.37ns        ? ?/sec    1.00    823.9±1.78ns        ? ?/sec
filter context string dictionary w NULLs (kept 1/2)                           1.00     95.0±0.28µs        ? ?/sec    1.00     95.2±0.24µs        ? ?/sec
filter context string dictionary w NULLs high selectivity (kept 1023/1024)    1.00     13.6±0.46µs        ? ?/sec    1.01     13.7±0.36µs        ? ?/sec
filter context string dictionary w NULLs low selectivity (kept 1/1024)        1.02   1084.9±3.13ns        ? ?/sec    1.00   1068.4±2.21ns        ? ?/sec
filter context string high selectivity (kept 1023/1024)                       1.00   656.1±12.90µs        ? ?/sec    1.02   668.1±22.80µs        ? ?/sec
filter context string low selectivity (kept 1/1024)                           1.00   1089.8±9.66ns        ? ?/sec    1.00   1088.1±1.59ns        ? ?/sec
filter context u8 (kept 1/2)                                                  1.00     18.9±0.08µs        ? ?/sec    1.00     18.9±0.06µs        ? ?/sec
filter context u8 high selectivity (kept 1023/1024)                           1.04  1881.5±10.18ns        ? ?/sec    1.00  1805.8±11.72ns        ? ?/sec
filter context u8 low selectivity (kept 1/1024)                               1.00    244.0±0.95ns        ? ?/sec    1.01    246.6±2.85ns        ? ?/sec
filter context u8 w NULLs (kept 1/2)                                          1.00     90.1±0.41µs        ? ?/sec    1.00     90.2±0.65µs        ? ?/sec
filter context u8 w NULLs high selectivity (kept 1023/1024)                   1.00      8.1±0.04µs        ? ?/sec    1.01      8.1±0.06µs        ? ?/sec
filter context u8 w NULLs low selectivity (kept 1/1024)                       1.00    570.5±2.77ns        ? ?/sec    1.03    589.8±3.53ns        ? ?/sec
filter decimal128 (kept 1/2)                                                  1.01     97.0±0.51µs        ? ?/sec    1.00     96.4±0.29µs        ? ?/sec
filter decimal128 high selectivity (kept 1023/1024)                           1.01     52.8±1.20µs        ? ?/sec    1.00     52.2±1.77µs        ? ?/sec
filter decimal128 low selectivity (kept 1/1024)                               1.00      2.4±0.01µs        ? ?/sec    1.00      2.4±0.00µs        ? ?/sec
filter f32 (kept 1/2)                                                         1.00    198.8±0.26µs        ? ?/sec    1.00    199.1±0.32µs        ? ?/sec
filter fsb with value length 20 (kept 1/2)                                    1.00    152.2±1.10µs        ? ?/sec    1.00    151.7±0.67µs        ? ?/sec
filter fsb with value length 20 high selectivity (kept 1023/1024)             1.02     72.9±2.42µs        ? ?/sec    1.00     71.6±2.24µs        ? ?/sec
filter fsb with value length 20 low selectivity (kept 1/1024)                 1.00      2.6±0.02µs        ? ?/sec    1.03      2.6±0.01µs        ? ?/sec
filter fsb with value length 5 (kept 1/2)                                     1.00    152.9±0.29µs        ? ?/sec    1.00    152.6±0.23µs        ? ?/sec
filter fsb with value length 5 high selectivity (kept 1023/1024)              1.05     11.7±0.70µs        ? ?/sec    1.00     11.1±0.51µs        ? ?/sec
filter fsb with value length 5 low selectivity (kept 1/1024)                  1.00      2.5±0.02µs        ? ?/sec    1.00      2.5±0.01µs        ? ?/sec
filter fsb with value length 50 (kept 1/2)                                    1.06    191.2±9.06µs        ? ?/sec    1.00    180.6±2.74µs        ? ?/sec
filter fsb with value length 50 high selectivity (kept 1023/1024)             1.00    204.1±7.66µs        ? ?/sec    1.06   215.5±12.76µs        ? ?/sec
filter fsb with value length 50 low selectivity (kept 1/1024)                 1.00      2.6±0.01µs        ? ?/sec    1.01      2.6±0.01µs        ? ?/sec
filter i32 (kept 1/2)                                                         1.00     91.5±0.37µs        ? ?/sec    1.00     91.5±0.13µs        ? ?/sec
filter i32 high selectivity (kept 1023/1024)                                  1.00      8.8±0.57µs        ? ?/sec    1.01      8.9±0.46µs        ? ?/sec
filter i32 low selectivity (kept 1/1024)                                      1.01      2.5±0.02µs        ? ?/sec    1.00      2.5±0.01µs        ? ?/sec
filter optimize (kept 1/2)                                                    1.00     91.7±0.36µs        ? ?/sec    1.00     92.0±0.68µs        ? ?/sec
filter optimize high selectivity (kept 1023/1024)                             1.01      3.1±0.01µs        ? ?/sec    1.00      3.1±0.01µs        ? ?/sec
filter optimize low selectivity (kept 1/1024)                                 1.01      2.3±0.01µs        ? ?/sec    1.00      2.2±0.02µs        ? ?/sec
filter run array (kept 1/2)                                                   1.00    465.9±3.68µs        ? ?/sec    1.00    465.4±1.34µs        ? ?/sec
filter run array high selectivity (kept 1023/1024)                            1.00    446.1±1.86µs        ? ?/sec    1.00    446.0±1.76µs        ? ?/sec
filter run array low selectivity (kept 1/1024)                                1.00    332.3±0.89µs        ? ?/sec    1.00    333.0±0.69µs        ? ?/sec
filter single record batch                                                    1.00     94.9±0.44µs        ? ?/sec    1.00     95.0±0.23µs        ? ?/sec
filter u8 (kept 1/2)                                                          1.00     92.1±0.10µs        ? ?/sec    1.00     92.0±0.48µs        ? ?/sec
filter u8 high selectivity (kept 1023/1024)                                   1.01      4.0±0.03µs        ? ?/sec    1.00      4.0±0.04µs        ? ?/sec
filter u8 low selectivity (kept 1/1024)                                       1.00      2.4±0.02µs        ? ?/sec    1.02      2.4±0.01µs        ? ?/sec

@alamb
Copy link
Contributor Author

alamb commented Jun 20, 2025

🤖 ./gh_compare_arrow.sh Benchmark Script Running
Linux aal-dev 6.11.0-1015-gcp #15~24.04.1-Ubuntu SMP Thu Apr 24 20:41:05 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/STRING_VIEW_LENGTH (1a38b68) to 1bed04c diff
BENCH_NAME=arrow_reader
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench arrow_reader
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_STRING_VIEW_LENGTH
Results will be posted here when complete

@alamb
Copy link
Contributor Author

alamb commented Jun 20, 2025

🤖: Benchmark completed

Details 

group                                                                                                      alamb_STRING_VIEW_LENGTH               main
-----                                                                                                      ------------------------               ----
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, mandatory, no NULLs                           1.00   1338.0±2.48µs        ? ?/sec    1.00   1338.8±1.77µs        ? ?/sec
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, optional, half NULLs                          1.00   1346.2±4.27µs        ? ?/sec    1.00   1349.5±4.31µs        ? ?/sec
arrow_array_reader/BYTE_ARRAY/Decimal128Array/plain encoded, optional, no NULLs                            1.00   1346.8±7.50µs        ? ?/sec    1.00   1347.1±2.06µs        ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, mandatory, no NULLs                                     1.01    483.4±2.95µs        ? ?/sec    1.00    480.1±3.42µs        ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, optional, half NULLs                                    1.00    672.1±1.31µs        ? ?/sec    1.02    685.9±1.80µs        ? ?/sec
arrow_array_reader/BinaryArray/dictionary encoded, optional, no NULLs                                      1.00    481.2±3.89µs        ? ?/sec    1.02    489.8±4.14µs        ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, mandatory, no NULLs                                          1.00    606.7±3.13µs        ? ?/sec    1.01    615.4±1.77µs        ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, optional, half NULLs                                         1.00    778.1±3.22µs        ? ?/sec    1.00    779.3±2.41µs        ? ?/sec
arrow_array_reader/BinaryArray/plain encoded, optional, no NULLs                                           1.00    618.3±3.29µs        ? ?/sec    1.01    626.3±3.75µs        ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, mandatory, no NULLs                                 1.04    259.5±2.87µs        ? ?/sec    1.00    248.6±3.68µs        ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, optional, half NULLs                                1.04    275.0±0.70µs        ? ?/sec    1.00    265.3±0.40µs        ? ?/sec
arrow_array_reader/BinaryViewArray/dictionary encoded, optional, no NULLs                                  1.04    264.7±3.33µs        ? ?/sec    1.00    255.1±3.45µs        ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, mandatory, no NULLs                                      1.01    326.7±1.12µs        ? ?/sec    1.00    324.8±1.40µs        ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, mandatory, no NULLs, short string                        1.01    302.8±1.04µs        ? ?/sec    1.00    301.0±0.75µs        ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, optional, half NULLs                                     1.01    317.4±2.72µs        ? ?/sec    1.00    314.5±1.35µs        ? ?/sec
arrow_array_reader/BinaryViewArray/plain encoded, optional, no NULLs                                       1.00    336.4±1.54µs        ? ?/sec    1.00    334.9±2.18µs        ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs     1.00   1065.8±3.53µs        ? ?/sec    1.00   1063.9±3.05µs        ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, optional, half NULLs    1.01    944.8±7.91µs        ? ?/sec    1.00    939.5±3.11µs        ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/byte_stream_split encoded, optional, no NULLs      1.00   1075.1±3.34µs        ? ?/sec    1.00   1072.0±4.05µs        ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, mandatory, no NULLs                 1.00    442.8±3.49µs        ? ?/sec    1.00    442.2±3.81µs        ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, optional, half NULLs                1.00    629.0±1.61µs        ? ?/sec    1.01    634.5±3.18µs        ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Decimal128Array/plain encoded, optional, no NULLs                  1.01    452.2±4.56µs        ? ?/sec    1.00    449.1±3.24µs        ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, mandatory, no NULLs        1.00    153.1±0.29µs        ? ?/sec    1.00    153.1±0.34µs        ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, optional, half NULLs       1.00    296.1±3.09µs        ? ?/sec    1.00    297.2±0.52µs        ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/byte_stream_split encoded, optional, no NULLs         1.00    158.7±0.43µs        ? ?/sec    1.00    159.0±1.02µs        ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, mandatory, no NULLs                    1.00     76.8±0.21µs        ? ?/sec    1.00     76.6±0.53µs        ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, optional, half NULLs                   1.00    255.6±0.44µs        ? ?/sec    1.01    256.9±0.52µs        ? ?/sec
arrow_array_reader/FIXED_LEN_BYTE_ARRAY/Float16Array/plain encoded, optional, no NULLs                     1.00     81.1±0.18µs        ? ?/sec    1.00     81.3±0.16µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, mandatory, no NULLs                    1.00    683.9±1.35µs        ? ?/sec    1.00    686.1±1.47µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, optional, half NULLs                   1.00    558.6±2.24µs        ? ?/sec    1.00    557.8±7.02µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/byte_stream_split encoded, optional, no NULLs                     1.00    690.6±1.63µs        ? ?/sec    1.00    693.0±2.21µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, mandatory, no NULLs                                1.21     64.6±5.68µs        ? ?/sec    1.00     53.4±3.37µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, optional, half NULLs                               1.02    259.3±1.32µs        ? ?/sec    1.00    254.2±1.82µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(16)/plain encoded, optional, no NULLs                                 1.12     71.3±5.15µs        ? ?/sec    1.00     63.6±5.26µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, mandatory, no NULLs                     1.00     85.6±0.22µs        ? ?/sec    1.01     86.6±0.56µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, optional, half NULLs                    1.00    228.0±0.71µs        ? ?/sec    1.00    228.3±0.37µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/byte_stream_split encoded, optional, no NULLs                      1.00     91.4±0.19µs        ? ?/sec    1.00     91.4±0.54µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, mandatory, no NULLs                                 1.00      9.3±0.16µs        ? ?/sec    1.01      9.4±0.23µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, optional, half NULLs                                1.00    189.4±0.45µs        ? ?/sec    1.00    189.3±1.67µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(2)/plain encoded, optional, no NULLs                                  1.00     14.5±0.14µs        ? ?/sec    1.02     14.7±0.28µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, mandatory, no NULLs                     1.00    169.4±0.27µs        ? ?/sec    1.00    168.8±0.79µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, optional, half NULLs                    1.00    347.2±1.29µs        ? ?/sec    1.00    346.9±1.81µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/byte_stream_split encoded, optional, no NULLs                      1.00    175.8±0.46µs        ? ?/sec    1.00    175.4±0.44µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, mandatory, no NULLs                                 1.04     14.3±0.31µs        ? ?/sec    1.00     13.7±0.39µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, optional, half NULLs                                1.00    270.4±0.43µs        ? ?/sec    1.00    269.8±0.55µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(4)/plain encoded, optional, no NULLs                                  1.02     20.3±0.68µs        ? ?/sec    1.00     19.9±0.28µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, mandatory, no NULLs                     1.00    339.2±0.71µs        ? ?/sec    1.00    338.5±0.70µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, optional, half NULLs                    1.00    384.6±3.02µs        ? ?/sec    1.01    386.7±0.81µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/byte_stream_split encoded, optional, no NULLs                      1.00    346.3±4.08µs        ? ?/sec    1.00    345.9±0.89µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, mandatory, no NULLs                                 1.00     26.7±0.37µs        ? ?/sec    1.06     28.3±1.37µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, optional, half NULLs                                1.00    229.2±0.57µs        ? ?/sec    1.01    230.7±2.17µs        ? ?/sec
arrow_array_reader/FixedLenByteArray(8)/plain encoded, optional, no NULLs                                  1.00     33.0±0.26µs        ? ?/sec    1.07     35.4±1.11µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, mandatory, no NULLs                           1.03    123.0±0.35µs        ? ?/sec    1.00    119.7±0.24µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, half NULLs                          1.01    138.2±0.46µs        ? ?/sec    1.00    136.3±0.31µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed skip, optional, no NULLs                            1.02    125.8±0.38µs        ? ?/sec    1.00    122.9±0.30µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, mandatory, no NULLs                                1.00    174.2±0.58µs        ? ?/sec    1.00    173.4±0.21µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, optional, half NULLs                               1.00    232.8±0.87µs        ? ?/sec    1.00    232.7±0.64µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/binary packed, optional, no NULLs                                 1.00    179.5±1.07µs        ? ?/sec    1.00    179.0±0.40µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs                    1.00     76.1±0.25µs        ? ?/sec    1.04     79.1±0.35µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, optional, half NULLs                   1.00    178.7±1.05µs        ? ?/sec    1.01    181.0±0.95µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/byte_stream_split encoded, optional, no NULLs                     1.02     84.6±0.24µs        ? ?/sec    1.00     82.6±0.89µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, mandatory, no NULLs                           1.00    137.9±0.56µs        ? ?/sec    1.00    137.3±0.39µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, optional, half NULLs                          1.00    214.3±2.55µs        ? ?/sec    1.00    214.3±0.71µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/dictionary encoded, optional, no NULLs                            1.01    143.3±0.24µs        ? ?/sec    1.00    142.3±0.40µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, mandatory, no NULLs                                1.03     74.3±0.29µs        ? ?/sec    1.00     72.3±0.21µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, optional, half NULLs                               1.00    178.7±0.56µs        ? ?/sec    1.00    178.6±0.76µs        ? ?/sec
arrow_array_reader/INT32/Decimal128Array/plain encoded, optional, no NULLs                                 1.02     79.8±0.35µs        ? ?/sec    1.00     78.1±0.21µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, mandatory, no NULLs                           1.00    111.4±1.84µs        ? ?/sec    1.03    114.7±0.28µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, half NULLs                          1.00    139.1±0.43µs        ? ?/sec    1.01    140.1±4.52µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed skip, optional, no NULLs                            1.00    115.2±0.34µs        ? ?/sec    1.02    117.5±0.32µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, mandatory, no NULLs                                1.00    168.4±0.33µs        ? ?/sec    1.02    171.4±1.73µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, optional, half NULLs                               1.00    246.0±0.56µs        ? ?/sec    1.02    250.0±0.57µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/binary packed, optional, no NULLs                                 1.00    173.5±1.71µs        ? ?/sec    1.02    176.7±0.62µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, mandatory, no NULLs                    1.00    202.4±1.33µs        ? ?/sec    1.00    202.7±0.39µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, optional, half NULLs                   1.00    262.8±0.55µs        ? ?/sec    1.00    262.5±2.04µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/byte_stream_split encoded, optional, no NULLs                     1.00    208.5±0.49µs        ? ?/sec    1.00    208.1±0.42µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, mandatory, no NULLs                           1.02    154.2±0.29µs        ? ?/sec    1.00    151.7±0.28µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, optional, half NULLs                          1.00    232.8±0.56µs        ? ?/sec    1.00    233.1±0.54µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/dictionary encoded, optional, no NULLs                            1.01    157.9±0.58µs        ? ?/sec    1.00    157.1±0.37µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, mandatory, no NULLs                                1.00    106.0±0.94µs        ? ?/sec    1.03    109.0±1.28µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, optional, half NULLs                               1.00    205.5±0.87µs        ? ?/sec    1.04    213.7±1.24µs        ? ?/sec
arrow_array_reader/INT64/Decimal128Array/plain encoded, optional, no NULLs                                 1.00    116.7±1.69µs        ? ?/sec    1.02    119.1±1.33µs        ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, mandatory, no NULLs                                      1.00     95.6±0.92µs        ? ?/sec    1.03     98.1±0.20µs        ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, half NULLs                                     1.00    114.9±0.73µs        ? ?/sec    1.01    115.9±0.37µs        ? ?/sec
arrow_array_reader/Int16Array/binary packed skip, optional, no NULLs                                       1.00     98.4±0.17µs        ? ?/sec    1.03    100.9±0.17µs        ? ?/sec
arrow_array_reader/Int16Array/binary packed, mandatory, no NULLs                                           1.00    130.5±0.30µs        ? ?/sec    1.00    130.6±0.22µs        ? ?/sec
arrow_array_reader/Int16Array/binary packed, optional, half NULLs                                          1.00    190.7±0.58µs        ? ?/sec    1.00    191.4±0.41µs        ? ?/sec
arrow_array_reader/Int16Array/binary packed, optional, no NULLs                                            1.00    135.5±0.31µs        ? ?/sec    1.00    135.6±0.25µs        ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, mandatory, no NULLs                               1.05     44.7±0.17µs        ? ?/sec    1.00     42.8±0.07µs        ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, optional, half NULLs                              1.00    143.7±0.22µs        ? ?/sec    1.02    147.0±0.39µs        ? ?/sec
arrow_array_reader/Int16Array/byte_stream_split encoded, optional, no NULLs                                1.03     49.2±0.40µs        ? ?/sec    1.00     47.9±0.08µs        ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, mandatory, no NULLs                                      1.00    104.4±0.25µs        ? ?/sec    1.00    103.9±0.22µs        ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, optional, half NULLs                                     1.00    179.5±0.34µs        ? ?/sec    1.00    178.9±0.37µs        ? ?/sec
arrow_array_reader/Int16Array/dictionary encoded, optional, no NULLs                                       1.00    109.5±0.33µs        ? ?/sec    1.00    109.2±0.23µs        ? ?/sec
arrow_array_reader/Int16Array/plain encoded, mandatory, no NULLs                                           1.01     38.7±0.12µs        ? ?/sec    1.00     38.5±0.09µs        ? ?/sec
arrow_array_reader/Int16Array/plain encoded, optional, half NULLs                                          1.00    141.2±0.60µs        ? ?/sec    1.01    142.1±0.33µs        ? ?/sec
arrow_array_reader/Int16Array/plain encoded, optional, no NULLs                                            1.00     44.1±0.14µs        ? ?/sec    1.00     44.0±0.12µs        ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, mandatory, no NULLs                                      1.02     95.3±0.16µs        ? ?/sec    1.00     93.6±0.20µs        ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, half NULLs                                     1.00    109.7±0.36µs        ? ?/sec    1.00    109.4±0.21µs        ? ?/sec
arrow_array_reader/Int32Array/binary packed skip, optional, no NULLs                                       1.01     98.9±0.22µs        ? ?/sec    1.00     97.8±0.14µs        ? ?/sec
arrow_array_reader/Int32Array/binary packed, mandatory, no NULLs                                           1.00    120.0±0.17µs        ? ?/sec    1.02    122.7±0.27µs        ? ?/sec
arrow_array_reader/Int32Array/binary packed, optional, half NULLs                                          1.00    176.0±0.42µs        ? ?/sec    1.01    178.4±1.82µs        ? ?/sec
arrow_array_reader/Int32Array/binary packed, optional, no NULLs                                            1.00    125.2±0.20µs        ? ?/sec    1.02    127.7±0.21µs        ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, mandatory, no NULLs                               1.00     25.7±0.19µs        ? ?/sec    1.00     25.8±0.09µs        ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, optional, half NULLs                              1.00    125.5±0.48µs        ? ?/sec    1.01    127.1±0.34µs        ? ?/sec
arrow_array_reader/Int32Array/byte_stream_split encoded, optional, no NULLs                                1.01     30.7±0.44µs        ? ?/sec    1.00     30.5±0.07µs        ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, mandatory, no NULLs                                      1.01     84.9±0.11µs        ? ?/sec    1.00     83.8±0.20µs        ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, optional, half NULLs                                     1.00    155.8±0.39µs        ? ?/sec    1.03    160.3±0.26µs        ? ?/sec
arrow_array_reader/Int32Array/dictionary encoded, optional, no NULLs                                       1.00     89.0±0.23µs        ? ?/sec    1.00     89.2±0.17µs        ? ?/sec
arrow_array_reader/Int32Array/plain encoded, mandatory, no NULLs                                           1.00     18.1±0.27µs        ? ?/sec    1.01     18.3±0.54µs        ? ?/sec
arrow_array_reader/Int32Array/plain encoded, optional, half NULLs                                          1.00    121.3±0.24µs        ? ?/sec    1.00    121.7±0.40µs        ? ?/sec
arrow_array_reader/Int32Array/plain encoded, optional, no NULLs                                            1.00     24.8±0.78µs        ? ?/sec    1.03     25.6±0.64µs        ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, mandatory, no NULLs                                      1.00     85.2±0.36µs        ? ?/sec    1.03     87.6±0.35µs        ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, half NULLs                                     1.00    110.9±0.20µs        ? ?/sec    1.01    112.2±0.38µs        ? ?/sec
arrow_array_reader/Int64Array/binary packed skip, optional, no NULLs                                       1.00     88.2±0.20µs        ? ?/sec    1.01     89.4±0.26µs        ? ?/sec
arrow_array_reader/Int64Array/binary packed, mandatory, no NULLs                                           1.01    116.8±0.29µs        ? ?/sec    1.00    115.4±0.47µs        ? ?/sec
arrow_array_reader/Int64Array/binary packed, optional, half NULLs                                          1.00    185.9±1.17µs        ? ?/sec    1.01    187.1±0.44µs        ? ?/sec
arrow_array_reader/Int64Array/binary packed, optional, no NULLs                                            1.00    121.7±0.48µs        ? ?/sec    1.00    121.3±0.58µs        ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, mandatory, no NULLs                               1.00    147.9±0.37µs        ? ?/sec    1.01    150.0±0.35µs        ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, optional, half NULLs                              1.00    206.1±0.77µs        ? ?/sec    1.01    208.3±0.45µs        ? ?/sec
arrow_array_reader/Int64Array/byte_stream_split encoded, optional, no NULLs                                1.00    155.9±0.34µs        ? ?/sec    1.00    155.8±0.48µs        ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, mandatory, no NULLs                                      1.00     98.5±0.51µs        ? ?/sec    1.01     99.4±0.46µs        ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, optional, half NULLs                                     1.00    177.4±0.39µs        ? ?/sec    1.01    179.7±0.80µs        ? ?/sec
arrow_array_reader/Int64Array/dictionary encoded, optional, no NULLs                                       1.00    104.1±1.25µs        ? ?/sec    1.01    105.0±1.23µs        ? ?/sec
arrow_array_reader/Int64Array/plain encoded, mandatory, no NULLs                                           1.03     45.8±2.56µs        ? ?/sec    1.00     44.3±2.99µs        ? ?/sec
arrow_array_reader/Int64Array/plain encoded, optional, half NULLs                                          1.00    149.2±0.53µs        ? ?/sec    1.00    149.8±0.50µs        ? ?/sec
arrow_array_reader/Int64Array/plain encoded, optional, no NULLs                                            1.05     53.4±2.56µs        ? ?/sec    1.00     51.1±2.14µs        ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, mandatory, no NULLs                                       1.05     97.4±0.20µs        ? ?/sec    1.00     93.2±0.88µs        ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, half NULLs                                      1.03    114.2±1.13µs        ? ?/sec    1.00    111.0±0.13µs        ? ?/sec
arrow_array_reader/Int8Array/binary packed skip, optional, no NULLs                                        1.05    100.6±0.26µs        ? ?/sec    1.00     95.9±0.19µs        ? ?/sec
arrow_array_reader/Int8Array/binary packed, mandatory, no NULLs                                            1.08    132.7±0.36µs        ? ?/sec    1.00    122.7±0.27µs        ? ?/sec
arrow_array_reader/Int8Array/binary packed, optional, half NULLs                                           1.03    188.2±0.27µs        ? ?/sec    1.00    183.5±0.51µs        ? ?/sec
arrow_array_reader/Int8Array/binary packed, optional, no NULLs                                             1.08    137.9±2.47µs        ? ?/sec    1.00    127.6±0.33µs        ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, mandatory, no NULLs                                1.01     36.8±0.09µs        ? ?/sec    1.00     36.6±0.09µs        ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, optional, half NULLs                               1.01    138.1±0.25µs        ? ?/sec    1.00    136.6±1.09µs        ? ?/sec
arrow_array_reader/Int8Array/byte_stream_split encoded, optional, no NULLs                                 1.00     41.5±0.08µs        ? ?/sec    1.00     41.6±0.08µs        ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, mandatory, no NULLs                                       1.01     96.6±0.50µs        ? ?/sec    1.00     96.0±0.35µs        ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, optional, half NULLs                                      1.01    171.8±0.35µs        ? ?/sec    1.00    170.7±0.45µs        ? ?/sec
arrow_array_reader/Int8Array/dictionary encoded, optional, no NULLs                                        1.01    101.8±0.31µs        ? ?/sec    1.00    101.0±0.22µs        ? ?/sec
arrow_array_reader/Int8Array/plain encoded, mandatory, no NULLs                                            1.00     30.8±0.07µs        ? ?/sec    1.00     30.6±0.08µs        ? ?/sec
arrow_array_reader/Int8Array/plain encoded, optional, half NULLs                                           1.00    135.1±0.29µs        ? ?/sec    1.00    134.5±0.25µs        ? ?/sec
arrow_array_reader/Int8Array/plain encoded, optional, no NULLs                                             1.01     36.2±0.16µs        ? ?/sec    1.00     35.8±0.07µs        ? ?/sec
arrow_array_reader/ListArray/plain encoded optional strings half NULLs                                     1.00      7.1±0.04ms        ? ?/sec    1.00      7.1±0.02ms        ? ?/sec
arrow_array_reader/ListArray/plain encoded optional strings no NULLs                                       1.01     13.0±0.13ms        ? ?/sec    1.00     12.9±0.09ms        ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, mandatory, no NULLs                                     1.00    482.1±3.77µs        ? ?/sec    1.02    493.4±3.94µs        ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, optional, half NULLs                                    1.00    684.6±2.02µs        ? ?/sec    1.00    684.9±2.25µs        ? ?/sec
arrow_array_reader/StringArray/dictionary encoded, optional, no NULLs                                      1.00    490.8±3.56µs        ? ?/sec    1.01    494.7±3.70µs        ? ?/sec
arrow_array_reader/StringArray/plain encoded, mandatory, no NULLs                                          1.00    678.2±2.81µs        ? ?/sec    1.01    686.0±2.85µs        ? ?/sec
arrow_array_reader/StringArray/plain encoded, optional, half NULLs                                         1.00    816.5±3.81µs        ? ?/sec    1.01    824.7±2.88µs        ? ?/sec
arrow_array_reader/StringArray/plain encoded, optional, no NULLs                                           1.00    689.1±1.96µs        ? ?/sec    1.01    695.4±2.69µs        ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, mandatory, no NULLs                                1.00    314.4±0.87µs        ? ?/sec    1.04    325.4±0.63µs        ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, optional, half NULLs                               1.00    394.9±1.42µs        ? ?/sec    1.01    399.3±0.99µs        ? ?/sec
arrow_array_reader/StringDictionary/dictionary encoded, optional, no NULLs                                 1.00    320.2±0.78µs        ? ?/sec    1.01    324.0±0.93µs        ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, mandatory, no NULLs                                 1.00    251.4±3.42µs        ? ?/sec    1.00    250.9±3.50µs        ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, optional, half NULLs                                1.00    266.7±1.74µs        ? ?/sec    1.00    266.8±0.80µs        ? ?/sec
arrow_array_reader/StringViewArray/dictionary encoded, optional, no NULLs                                  1.01    258.4±2.95µs        ? ?/sec    1.00    256.9±3.02µs        ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, mandatory, no NULLs                                      1.01    405.4±0.86µs        ? ?/sec    1.00    401.9±1.20µs        ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, optional, half NULLs                                     1.02    354.4±1.35µs        ? ?/sec    1.00    349.0±1.24µs        ? ?/sec
arrow_array_reader/StringViewArray/plain encoded, optional, no NULLs                                       1.01    416.3±1.57µs        ? ?/sec    1.00    411.2±1.23µs        ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, mandatory, no NULLs                                     1.01    103.7±0.14µs        ? ?/sec    1.00    102.5±0.11µs        ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, half NULLs                                    1.00    119.7±0.17µs        ? ?/sec    1.00    119.3±0.39µs        ? ?/sec
arrow_array_reader/UInt16Array/binary packed skip, optional, no NULLs                                      1.00    105.9±0.19µs        ? ?/sec    1.00    105.6±0.20µs        ? ?/sec
arrow_array_reader/UInt16Array/binary packed, mandatory, no NULLs                                          1.00    140.5±0.68µs        ? ?/sec    1.00    140.5±0.56µs        ? ?/sec
arrow_array_reader/UInt16Array/binary packed, optional, half NULLs                                         1.00    196.9±0.39µs        ? ?/sec    1.00    197.4±0.49µs        ? ?/sec
arrow_array_reader/UInt16Array/binary packed, optional, no NULLs                                           1.00    145.5±0.22µs        ? ?/sec    1.00    145.6±1.62µs        ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, mandatory, no NULLs                              1.00     44.5±0.09µs        ? ?/sec    1.01     44.7±0.09µs        ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, optional, half NULLs                             1.00    145.1±0.37µs        ? ?/sec    1.00    145.0±0.33µs        ? ?/sec
arrow_array_reader/UInt16Array/byte_stream_split encoded, optional, no NULLs                               1.00     49.4±0.13µs        ? ?/sec    1.00     49.5±0.10µs        ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, mandatory, no NULLs                                     1.00    104.1±0.25µs        ? ?/sec    1.00    103.7±0.32µs        ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, optional, half NULLs                                    1.00    179.2±0.29µs        ? ?/sec    1.00    179.2±0.42µs        ? ?/sec
arrow_array_reader/UInt16Array/dictionary encoded, optional, no NULLs                                      1.00    109.2±0.13µs        ? ?/sec    1.00    109.2±0.22µs        ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, mandatory, no NULLs                                          1.00     38.7±0.15µs        ? ?/sec    1.00     38.6±0.09µs        ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, optional, half NULLs                                         1.01    141.6±1.00µs        ? ?/sec    1.00    140.8±0.44µs        ? ?/sec
arrow_array_reader/UInt16Array/plain encoded, optional, no NULLs                                           1.01     44.0±0.11µs        ? ?/sec    1.00     43.6±0.29µs        ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, mandatory, no NULLs                                     1.01     96.6±0.40µs        ? ?/sec    1.00     96.0±0.19µs        ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, half NULLs                                    1.01    111.5±0.79µs        ? ?/sec    1.00    109.9±0.23µs        ? ?/sec
arrow_array_reader/UInt32Array/binary packed skip, optional, no NULLs                                      1.03     99.7±0.75µs        ? ?/sec    1.00     97.2±0.22µs        ? ?/sec
arrow_array_reader/UInt32Array/binary packed, mandatory, no NULLs                                          1.00    121.2±0.21µs        ? ?/sec    1.00    121.5±0.26µs        ? ?/sec
arrow_array_reader/UInt32Array/binary packed, optional, half NULLs                                         1.00    178.7±0.98µs        ? ?/sec    1.00    179.6±0.39µs        ? ?/sec
arrow_array_reader/UInt32Array/binary packed, optional, no NULLs                                           1.00    126.6±0.37µs        ? ?/sec    1.00    127.2±0.35µs        ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, mandatory, no NULLs                              1.03     27.2±0.38µs        ? ?/sec    1.00     26.5±0.22µs        ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, optional, half NULLs                             1.00    126.5±0.35µs        ? ?/sec    1.00    127.0±0.31µs        ? ?/sec
arrow_array_reader/UInt32Array/byte_stream_split encoded, optional, no NULLs                               1.01     31.9±0.33µs        ? ?/sec    1.00     31.6±0.23µs        ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, mandatory, no NULLs                                     1.00     86.2±0.82µs        ? ?/sec    1.01     86.7±0.84µs        ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, optional, half NULLs                                    1.00    160.2±0.39µs        ? ?/sec    1.01    161.2±0.30µs        ? ?/sec
arrow_array_reader/UInt32Array/dictionary encoded, optional, no NULLs                                      1.00     91.4±0.94µs        ? ?/sec    1.01     92.0±0.30µs        ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, mandatory, no NULLs                                          1.03     21.2±0.83µs        ? ?/sec    1.00     20.6±0.24µs        ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, optional, half NULLs                                         1.00    124.1±0.33µs        ? ?/sec    1.01    124.8±0.50µs        ? ?/sec
arrow_array_reader/UInt32Array/plain encoded, optional, no NULLs                                           1.04     26.8±1.05µs        ? ?/sec    1.00     25.9±0.27µs        ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, mandatory, no NULLs                                     1.00     85.2±0.20µs        ? ?/sec    1.01     86.4±0.26µs        ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, half NULLs                                    1.00    111.6±0.35µs        ? ?/sec    1.01    112.2±0.41µs        ? ?/sec
arrow_array_reader/UInt64Array/binary packed skip, optional, no NULLs                                      1.00     88.8±0.44µs        ? ?/sec    1.01     89.6±0.29µs        ? ?/sec
arrow_array_reader/UInt64Array/binary packed, mandatory, no NULLs                                          1.01    117.0±0.33µs        ? ?/sec    1.00    115.5±0.50µs        ? ?/sec
arrow_array_reader/UInt64Array/binary packed, optional, half NULLs                                         1.04    194.3±0.71µs        ? ?/sec    1.00    187.3±0.51µs        ? ?/sec
arrow_array_reader/UInt64Array/binary packed, optional, no NULLs                                           1.00    121.7±0.42µs        ? ?/sec    1.00    121.1±0.62µs        ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, mandatory, no NULLs                              1.01    149.9±0.64µs        ? ?/sec    1.00    148.5±0.36µs        ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, optional, half NULLs                             1.01    208.1±0.44µs        ? ?/sec    1.00    206.0±0.38µs        ? ?/sec
arrow_array_reader/UInt64Array/byte_stream_split encoded, optional, no NULLs                               1.01    155.7±0.60µs        ? ?/sec    1.00    153.7±0.34µs        ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, mandatory, no NULLs                                     1.00     99.0±0.27µs        ? ?/sec    1.01     99.7±0.67µs        ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, optional, half NULLs                                    1.00    177.5±0.45µs        ? ?/sec    1.01    178.6±1.20µs        ? ?/sec
arrow_array_reader/UInt64Array/dictionary encoded, optional, no NULLs                                      1.00    104.1±0.32µs        ? ?/sec    1.01    105.1±0.49µs        ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, mandatory, no NULLs                                          1.00     42.9±1.86µs        ? ?/sec    1.10     47.3±2.42µs        ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, optional, half NULLs                                         1.01    151.6±0.77µs        ? ?/sec    1.00    149.9±0.92µs        ? ?/sec
arrow_array_reader/UInt64Array/plain encoded, optional, no NULLs                                           1.00     52.2±2.68µs        ? ?/sec    1.06     55.5±3.43µs        ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, mandatory, no NULLs                                      1.00    101.0±0.12µs        ? ?/sec    1.03    103.8±0.97µs        ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, half NULLs                                     1.00    114.3±0.13µs        ? ?/sec    1.02    116.8±0.25µs        ? ?/sec
arrow_array_reader/UInt8Array/binary packed skip, optional, no NULLs                                       1.00    101.6±0.91µs        ? ?/sec    1.05    106.9±1.02µs        ? ?/sec
arrow_array_reader/UInt8Array/binary packed, mandatory, no NULLs                                           1.00    133.1±1.01µs        ? ?/sec    1.07    142.5±0.71µs        ? ?/sec
arrow_array_reader/UInt8Array/binary packed, optional, half NULLs                                          1.00    188.4±0.28µs        ? ?/sec    1.03    193.2±0.45µs        ? ?/sec
arrow_array_reader/UInt8Array/binary packed, optional, no NULLs                                            1.00    138.6±0.45µs        ? ?/sec    1.07    148.1±0.27µs        ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, mandatory, no NULLs                               1.00     36.8±0.08µs        ? ?/sec    1.01     37.0±0.08µs        ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, optional, half NULLs                              1.00    137.8±0.43µs        ? ?/sec    1.01    139.2±0.41µs        ? ?/sec
arrow_array_reader/UInt8Array/byte_stream_split encoded, optional, no NULLs                                1.01     42.0±0.13µs        ? ?/sec    1.00     41.7±0.09µs        ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, mandatory, no NULLs                                      1.00     96.0±0.16µs        ? ?/sec    1.01     96.5±0.29µs        ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, optional, half NULLs                                     1.00    170.0±0.32µs        ? ?/sec    1.00    170.8±0.25µs        ? ?/sec
arrow_array_reader/UInt8Array/dictionary encoded, optional, no NULLs                                       1.00    101.5±0.26µs        ? ?/sec    1.00    101.7±0.16µs        ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, mandatory, no NULLs                                           1.00     30.9±0.10µs        ? ?/sec    1.00     30.9±0.17µs        ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, optional, half NULLs                                          1.00    133.6±0.32µs        ? ?/sec    1.00    134.1±0.47µs        ? ?/sec
arrow_array_reader/UInt8Array/plain encoded, optional, no NULLs                                            1.00     36.2±0.19µs        ? ?/sec    1.01     36.4±0.10µs        ? ?/sec

@Dandandan
Copy link
Contributor

Dandandan commented Jun 20, 2025

hah this shows the benchmarks are pretty stable when the compiled code is unchanged 😎

@alamb
Copy link
Contributor Author

alamb commented Jun 20, 2025

hah this shows the benchmarks are pretty stable when the compiled code is unchanged 😎

Indeed but there are still outliers like this (which shouldn't be impacted at all) 🤔

arrow_array_reader/Int8Array/binary packed, optional, no NULLs 1.08 137.9±2.47µs ? ?/sec 1.00 127.6±0.33µs ? ?/sec

@alamb alamb marked this pull request as ready for review June 20, 2025 19:32
@github-actions github-actions bot added the arrow Changes to the arrow crate label Jun 24, 2025
@alamb
Copy link
Contributor Author

alamb commented Jun 25, 2025

Thanks again @Dandandan for the review

@alamb alamb merged commit 10d9714 into apache:main Jun 25, 2025
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Dandandan Dandandan Dandandan approved these changes

Assignees

No one assigned

Labels

arrow Changes to the arrow crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@alamb @Dandandan