Improve parquet ListingTable speed with parquet metadata (short clickbench queries)

[alamb]

Is your feature request related to a problem or challenge?

I spent some time looking at the ClickBench results with DataFusion 40.0.0
#11567 (comment) (thanks @pmcgleenon 🙏 )

Specifically, I looked into how we could make some of the already fast queries on the the partitioned dataset faster. Unsurprisingly, for the really fast queries the query time is actually dominated by parquet metadata analysis and DataFusion statistics creation.

For example

ClickBench Q0

SELECT COUNT(*) FROM hits;

To reproduce, run:

cd datafusion
cargo run --release --bin dfbench -- clickbench --iterations 100 --path benchmarks/data/hits_partitioned  --query 0

I profiled this using Instruments. Here are some annotated screenshots

Some of my take aways are

1. a substantial amount of time is spent reading the parquet metadata twice
2. A substantial amount of time is spent managing the ScalarValues in statistics

Describe the solution you'd like

If would be cool to make these queries faster by reducing the per file metadata handling overhead (e.g. don't read the metadata more than once and figure out some way to make statistics handling more efficient)

Describe alternatives you've considered

Note this project isn't broken down into tasks yet

I think @Ted-Jiang did some work way back to cache parquet metaddata

Additional context

No response

[Rachelint]

Some ideas about solving

A substantial amount of time is spent managing the ScalarValues in statistics

Plan to try it today.

See one simple thing is, refactor the Statistics to:

pub struct StatisticsInner {
    /// The number of table rows.
    pub num_rows: Precision<usize>,
    /// Total bytes of the table rows.
    pub total_byte_size: Precision<usize>,
    /// Statistics on a column level. It contains a [`ColumnStatistics`] for
    /// each field in the schema of the table to which the [`Statistics`] refer.
    pub column_statistics: Vec<ColumnStatistics>,
}

pub struct Statistics {
   inner: Arc<StatisticsInner>,
}

And the clone of Arc is trivial.

[Rachelint]

take

[alamb]

That would be a very interesting experiment to try

[Rachelint]

Based on the detail profile about q0 in clickbench as following, maybe the optimization work can be divided into three parts:

• Reduce the cost about clone and drop of Statistics
• Maybe optimize the impl for get_statistics_with_limit(seems may tmp vectors exist, but not sure)
• Do cache for the result of object store list operation

Trying the first possible optimization now.

[alamb]

#11802 is very nice 👌 It would be fascinating to know what the flamegraph looks like after that PR (aka what are next highest bottleneck)

[Rachelint]

#11802 is very nice 👌 It would be fascinating to know what the flamegraph looks like after that PR (aka what are next highest bottleneck)

😄 I guess they will be the plan creation and object store list.

[alamb]

I don't think this one is actionable after we have merged

• [Epic] Enable parquet metadata cache by default #17000