Skip to content

Commit 8709a7c

Browse files
andrevandevenAndre van de Ven
authored and
tandonks
committed
Add Fetch Phase to Search Profile (opensearch-project#18664)
Signed-off-by: Andre van de Ven <[email protected]> Signed-off-by: Andre van de Ven <[email protected]> Co-authored-by: Andre van de Ven <[email protected]>
1 parent 1c3e2e3 commit 8709a7c

File tree

29 files changed

+2345
-36
lines changed

29 files changed

+2345
-36
lines changed

‎.idea/runConfigurations/Debug_OpenSearch.xml‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
3030
- Extend Approximation Framework to other numeric types ([#18530](https://github.com/opensearch-project/OpenSearch/issues/18530))
3131
- Add Semantic Version field type mapper and extensive unit tests([#18454](https://github.com/opensearch-project/OpenSearch/pull/18454))
3232
- Pass index settings to system ingest processor factories. ([#18708](https://github.com/opensearch-project/OpenSearch/pull/18708))
33+
- Add fetch phase profiling. ([#18664](https://github.com/opensearch-project/OpenSearch/pull/18664))
3334
- Include named queries from rescore contexts in matched_queries array ([#18697](https://github.com/opensearch-project/OpenSearch/pull/18697))
3435
- Add the configurable limit on rule cardinality ([#18663](https://github.com/opensearch-project/OpenSearch/pull/18663))
3536
- Disable approximation framework when dealing with multiple sorts ([#18763](https://github.com/opensearch-project/OpenSearch/pull/18763))

‎modules/lang-painless/src/yamlRestTest/resources/rest-api-spec/test/painless/130_script_fields_profile.yml‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
setup:
2+
- do:
3+
indices.create:
4+
index: test_fetch_profile
5+
body:
6+
settings:
7+
number_of_replicas: 0
8+
number_of_shards: 1
9+
mappings:
10+
properties:
11+
text_field:
12+
type: text
13+
fields:
14+
keyword:
15+
type: keyword
16+
numeric_field:
17+
type: integer
18+
date_field:
19+
type: date
20+
object_field:
21+
type: nested
22+
properties:
23+
nested_field:
24+
type: keyword
25+
stored_field:
26+
type: keyword
27+
store: true
28+
29+
- do:
30+
bulk:
31+
refresh: true
32+
index: test_fetch_profile
33+
body: |
34+
{ "index": {} }
35+
{ "text_field": "Hello world", "numeric_field": 42, "date_field": "2023-01-01", "object_field": { "nested_field": "nested value" }, "stored_field": "stored value" }
36+
{ "index": {} }
37+
{ "text_field": "Another document", "numeric_field": 100, "date_field": "2023-02-01", "object_field": { "nested_field": "another nested" }, "stored_field": "another stored" }
38+
{ "index": {} }
39+
{ "text_field": "Third document with more text", "numeric_field": 200, "date_field": "2023-03-01", "object_field": { "nested_field": "third nested" }, "stored_field": "third stored" }
40+
41+
42+
---
43+
"Script fields phase profiling":
44+
- skip:
45+
features: "contains"
46+
47+
- do:
48+
search:
49+
index: test_fetch_profile
50+
body:
51+
profile: true
52+
query:
53+
match_all: {}
54+
script_fields:
55+
my_field:
56+
script:
57+
lang: painless
58+
source: "doc['numeric_field'].value * 2"
59+
60+
- contains:
61+
profile.shards.0.fetch.0.children:
62+
type: "ScriptFieldsPhase"
63+
description: "ScriptFieldsPhase"
64+
- length: { profile.shards.0.fetch.0.children: 1 }
65+
66+
- is_true: profile.shards.0.fetch.0.children.0.breakdown.process
67+
- match: { profile.shards.0.fetch.0.children.0.breakdown.process_count: 3 }
68+
- is_true: profile.shards.0.fetch.0.children.0.breakdown.set_next_reader
69+
- match: { profile.shards.0.fetch.0.children.0.breakdown.set_next_reader_count: 1 }

‎qa/multi-cluster-search/src/test/java/org/opensearch/search/CCSDuelIT.java‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -831,10 +831,12 @@ private static Map<String, Object> responseToMap(SearchResponse response) throws
831831
Map<String, Object> responseMap = org.opensearch.common.xcontent.XContentHelper.convertToMap(bytesReference, false, MediaTypeRegistry.JSON).v2();
832832
assertNotNull(responseMap.put("took", -1));
833833
responseMap.remove("num_reduce_phases");
834-
Map<String, Object> profile = (Map<String, Object>)responseMap.get("profile");
834+
Map<String, Object> profile = (Map<String, Object>) responseMap.get("profile");
835835
if (profile != null) {
836-
List<Map<String, Object>> shards = (List <Map<String, Object>>)profile.get("shards");
836+
List<Map<String, Object>> shards = (List<Map<String, Object>>) profile.get("shards");
837837
for (Map<String, Object> shard : shards) {
838+
// Unconditionally remove the fetch profile to prevent flaky failures
839+
shard.remove("fetch");
838840
replaceProfileTime(shard);
839841
}
840842
}

‎rest-api-spec/src/main/resources/rest-api-spec/test/search.profile/10_fetch_phase.yml‎

Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,193 @@
1+
# Setup remains the same
2+
setup:
3+
- do:
4+
indices.create:
5+
index: test_fetch_profile
6+
body:
7+
settings:
8+
number_of_replicas: 0
9+
number_of_shards: 1
10+
mappings:
11+
properties:
12+
text_field:
13+
type: text
14+
fields:
15+
keyword:
16+
type: keyword
17+
numeric_field:
18+
type: integer
19+
date_field:
20+
type: date
21+
object_field:
22+
type: nested
23+
properties:
24+
nested_field:
25+
type: keyword
26+
stored_field:
27+
type: keyword
28+
store: true
29+
30+
- do:
31+
bulk:
32+
refresh: true
33+
index: test_fetch_profile
34+
body: |
35+
{ "index": {} }
36+
{ "text_field": "Hello world", "numeric_field": 42, "date_field": "2023-01-01", "object_field": { "nested_field": "nested value" }, "stored_field": "stored value" }
37+
{ "index": {} }
38+
{ "text_field": "Another document", "numeric_field": 100, "date_field": "2023-02-01", "object_field": { "nested_field": "another nested" }, "stored_field": "another stored" }
39+
{ "index": {} }
40+
{ "text_field": "Third document with more text", "numeric_field": 200, "date_field": "2023-03-01", "object_field": { "nested_field": "third nested" }, "stored_field": "third stored" }
41+
42+
---
43+
"Combined fetch sub-phases profiling":
44+
- skip:
45+
version: " - 3.1.99"
46+
reason: "Fetch phase profiling was introduced in 3.2.0"
47+
features: "contains"
48+
49+
- do:
50+
search:
51+
index: test_fetch_profile
52+
body:
53+
profile: true
54+
explain: true
55+
version: true
56+
seq_no_primary_term: true
57+
track_scores: true
58+
sort:
59+
- numeric_field: { order: asc }
60+
query:
61+
bool:
62+
must:
63+
- match_all: {}
64+
should:
65+
- nested:
66+
path: "object_field"
67+
query:
68+
wildcard:
69+
"object_field.nested_field": "nested*"
70+
inner_hits: {}
71+
- match:
72+
text_field:
73+
query: "document"
74+
_name: "my_query"
75+
docvalue_fields:
76+
- "numeric_field"
77+
fields:
78+
- "stored_field"
79+
highlight:
80+
fields:
81+
"object_field.nested_field": {}
82+
83+
# 1. Verify basic fetch profile structure
84+
- is_true: profile.shards.0.fetch.0
85+
- match: { profile.shards.0.fetch.0.type: "fetch" }
86+
- match: { profile.shards.0.fetch.0.description: "fetch" }
87+
- is_true: profile.shards.0.fetch.0.time_in_nanos
88+
89+
# 2. Verify detailed breakdown of the main fetch operation
90+
- is_true: profile.shards.0.fetch.0.breakdown
91+
- is_true: profile.shards.0.fetch.0.breakdown.load_stored_fields
92+
- match: { profile.shards.0.fetch.0.breakdown.load_stored_fields_count: 3 }
93+
- is_true: profile.shards.0.fetch.0.breakdown.load_source
94+
- match: { profile.shards.0.fetch.0.breakdown.load_source_count: 3 }
95+
- is_true: profile.shards.0.fetch.0.breakdown.get_next_reader
96+
- match: { profile.shards.0.fetch.0.breakdown.get_next_reader_count: 1}
97+
- is_true: profile.shards.0.fetch.0.breakdown.build_sub_phase_processors
98+
- match: { profile.shards.0.fetch.0.breakdown.build_sub_phase_processors_count: 1}
99+
- is_true: profile.shards.0.fetch.0.breakdown.create_stored_fields_visitor
100+
- match: { profile.shards.0.fetch.0.breakdown.create_stored_fields_visitor_count: 1}
101+
102+
# 3. Verify all expected fetch sub-phases are present as children
103+
- length: { profile.shards.0.fetch.0.children: 9 }
104+
- contains:
105+
profile.shards.0.fetch.0.children:
106+
type: "FetchSourcePhase"
107+
- contains:
108+
profile.shards.0.fetch.0.children:
109+
type: "ExplainPhase"
110+
- contains:
111+
profile.shards.0.fetch.0.children:
112+
type: "FetchDocValuesPhase"
113+
- contains:
114+
profile.shards.0.fetch.0.children:
115+
type: "FetchFieldsPhase"
116+
- contains:
117+
profile.shards.0.fetch.0.children:
118+
type: "FetchVersionPhase"
119+
- contains:
120+
profile.shards.0.fetch.0.children:
121+
type: "SeqNoPrimaryTermPhase"
122+
- contains:
123+
profile.shards.0.fetch.0.children:
124+
type: "MatchedQueriesPhase"
125+
- contains:
126+
profile.shards.0.fetch.0.children:
127+
type: "HighlightPhase"
128+
- contains:
129+
profile.shards.0.fetch.0.children:
130+
type: "FetchScorePhase"
131+
132+
---
133+
"No source or empty fetch profiling":
134+
- skip:
135+
version: " - 3.1.99"
136+
reason: "Fetch phase profiling was introduced in 3.2.0"
137+
138+
# Case 1: Test with _source: false, which removes FetchSourcePhase
139+
- do:
140+
search:
141+
index: test_fetch_profile
142+
body:
143+
profile: true
144+
query:
145+
match_all: {}
146+
_source: false
147+
docvalue_fields:
148+
- "numeric_field"
149+
150+
- is_true: profile.shards.0.fetch.0
151+
- length: { profile.shards.0.fetch.0.children: 1 }
152+
- match: { profile.shards.0.fetch.0.children.0.type: "FetchDocValuesPhase" }
153+
- match: { profile.shards.0.fetch.0.children.0.breakdown.process_count: 3 }
154+
155+
# Case 2: Test with size: 0, which results in an empty fetch profile
156+
- do:
157+
search:
158+
index: test_fetch_profile
159+
body:
160+
profile: true
161+
size: 0
162+
aggs:
163+
group_by_nested:
164+
terms:
165+
field: "object_field.nested_field"
166+
167+
- match: { profile.shards.0.fetch: [ ] }
168+
169+
---
170+
"Top-hits aggregation profiling":
171+
- skip:
172+
version: " - 3.1.99"
173+
reason: "Fetch phase profiling was introduced in 3.2.0"
174+
175+
- do:
176+
search:
177+
index: test_fetch_profile
178+
body:
179+
profile: true
180+
query:
181+
match:
182+
text_field: "document"
183+
aggs:
184+
top_hits_agg:
185+
top_hits:
186+
size: 1
187+
188+
# Verify that the profile contains a single fetch operation for the query
189+
- length: { profile.shards.0.fetch: 1 }
190+
- match: { profile.shards.0.fetch.0.type: "fetch" }
191+
- match: { profile.shards.0.fetch.0.description: "fetch" }
192+
- length: { profile.shards.0.fetch.0.children: 1 }
193+
- match: { profile.shards.0.fetch.0.children.0.type: "FetchSourcePhase" }

0 commit comments

Comments
 (0)