perf(623): Use row counts in query optimization #677
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
joshua-spacetime
force-pushed
the
joshua/perf/623/table-size-metrics-index-join
branch
from
ce7e57b to
95c5eba
Compare
joshua-spacetime
force-pushed
the
joshua/test/index-to-inner-join
branch
from
3d88f58 to
9c22936
Compare
joshua-spacetime
force-pushed
the
joshua/perf/623/table-size-metrics-index-join
branch
from
95c5eba to
743439b
Compare
joshua-spacetime
force-pushed
the
joshua/test/index-to-inner-join
branch
2 times, most recently
from
285bf54 to
89ec242
Compare
joshua-spacetime
force-pushed
the
joshua/perf/623/table-size-metrics-index-join
branch
2 times, most recently
from
745efa7 to
1cd8d9d
Compare
| fn test_eval_incr_for_index_join() -> ResultTest<()> { | ||
| let (db, _) = make_test_db()?; | ||
| run_eval_incr_for_index_join(&db)?; | ||
| run_eval_incr_for_index_join(&db.with_row_count(Arc::new(|_| 5)))?; |
Collaborator
Author
There was a problem hiding this comment.
This new test case will run our incremental evaluation suite under the new query plan.
Comment on lines
+457
to
+460
| // If the size of the indexed table is sufficiently large, do not reorder. | ||
| Table::DbTable(DbTable { table_id, .. }) if row_count(table_id) > 1000 => self, | ||
| // If this is a delta table, we must reorder. | ||
| // If this is a sufficiently small physical table, we should reorder. |
Collaborator
Author
There was a problem hiding this comment.
Sufficiently small means 1000 rows or less.
joshua-spacetime
force-pushed
the
joshua/perf/623/table-size-metrics-index-join
branch
from
1cd8d9d to
d4ad20d
Compare
mamcx
reviewed
Dec 14, 2023
| Ok(db) | ||
| } | ||
|
|
||
| /// Returns an approximate row count for a particular table. |
Contributor
There was a problem hiding this comment.
There is already a RowCount object:
#[derive(Debug, Copy, Clone, PartialOrd, Ord, PartialEq, Eq)]
pub struct RowCount {
pub min: usize,
pub max: Option<usize>,
}on sats -> Relation
The trait RelOps use it.
Collaborator
Author
There was a problem hiding this comment.
I will add a TODO like you mentioned.
I wonder if we should move Relation out of sats, since it will need access to database statistics. Because right now RowCount only works on MemTables.
joshua-spacetime
force-pushed
the
joshua/perf/623/table-size-metrics-index-join
branch
from
d4ad20d to
a1b50d8
Compare
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
joshua-spacetime
force-pushed
the
joshua/perf/623/table-size-metrics-index-join
branch
from
a1b50d8 to
b8583e7
Compare
Closes #623. Before this patch query optimization was entirely syntax driven. Now that we keep table size metrics we can be somewhat data driven. This patch improves index joins, by using row counts to determine the index side and the probe side.
joshua-spacetime
force-pushed
the
joshua/perf/623/table-size-metrics-index-join
branch
from
b8583e7 to
8ee1346
Compare
joshua-spacetime
deleted the
joshua/perf/623/table-size-metrics-index-join
branch
joshua-spacetime
added a commit
that referenced
this pull request
Jan 12, 2024
Table size metrics were previously moved out of core. This was due to the query planner needing access to them. However that dependency was ultimately managed differently via #677.
github-merge-queue bot
pushed a commit
that referenced
this pull request
Jan 12, 2024
* chore: Remove unused metrics * chore: Move table size metrics back into core Table size metrics were previously moved out of core. This was due to the query planner needing access to them. However that dependency was ultimately managed differently via #677.
Closed
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #623.
Before this patch query optimization was entirely syntax driven. Now that we keep table size metrics we can be somewhat data driven.
This patch improves index joins,
by using row counts to determine the index side and the probe side.
Description of Changes
Please describe your change, mention any related tickets, and so on here.
API and ABI breaking changes
If this is an API or ABI breaking change, please apply the
corresponding GitHub label.
Expected complexity level and risk
How complicated do you think these changes are? Grade on a scale from 1 to 5,
where 1 is a trivial change, and 5 is a deep-reaching and complex change.
This complexity rating applies not only to the complexity apparent in the diff,
but also to its interactions with existing and future code.
If you answered more than a 2, explain what is complex about the PR,
and what other components it interacts with in potentially concerning ways.