-
Notifications
You must be signed in to change notification settings - Fork 292
[doc] add documentation about tracing #6594
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,137 @@ | ||
+++ | ||
title = "Tracing" | ||
+++ | ||
|
||
Tracing is a powerful tool for observing system behavior across multiple components, making it especially | ||
useful for debugging and performance analysis in complex environments. | ||
|
||
By integrating OpenTelemetry (a standard that unifies OpenTracing and OpenCensus) and the Zipkin v2 protocol, | ||
XAPI enables efficient tracking and visualization of operations across internal and external systems. | ||
This facilitates detailed analysis and improves collaboration between teams. | ||
|
||
Tracing is commonly used in high-level applications such as web services. As a result, less widely-used or | ||
non-web-oriented languages may lack dedicated libraries for distributed tracing (An OCaml implementation | ||
has been developed specifically for XenAPI). | ||
|
||
# How tracing works in XAPI | ||
|
||
## Spans and Trace Context | ||
|
||
- A *span* is the core unit of a trace, representing a single operation with a defined start and end time. | ||
Spans can contain sub-spans that represent child tasks. This helps identify bottlenecks or areas that | ||
can be parallelized. | ||
- A span can contain several contextual elements such as *tags* (key-value pairs), | ||
*events* (time-based data), and *errors*. | ||
- The *TraceContext* HTTP standard defines how trace IDs and span contexts are propagated across systems, | ||
enabling full traceability of operations. | ||
|
||
This data enables the creation of relationships between tasks and supports visualizations such as | ||
architecture diagrams or execution flows. These help in identifying root causes of issues and bottlenecks, | ||
and also assist newcomers in onboarding to the project. | ||
|
||
## Configuration | ||
|
||
- To enable tracing, you need to create an *Observer* object in XAPI. This can be done using the *xe* CLI: | ||
```sh | ||
xe observer-create \ | ||
name-label=<name> \ | ||
enabled=true \ | ||
components=xapi,xenopsd \ | ||
``` | ||
- By default, if you don't specify `enabled=true`, the observer will be disabled. | ||
- To add an HTTP endpoint, make sure the server is up and running, then run: | ||
```sh | ||
xe observer-param-set uuid=<OBSERVER_UUID> endpoints=bugtool,http://<jaeger-ip>:9411/api/v2/spans | ||
``` | ||
If you specify an invalid or unreachable HTTP endpoint, the configuration will fail. | ||
- **components**: Specify which internal components (e.g., *xapi*, *xenopsd*) should be traced. | ||
Additional components are expected to be supported in future releases. An experimental *smapi* component | ||
is also available and requires additional configuration (explained below). | ||
|
||
- **endpoints**: The observer can collect traces locally in */var/log/dt* or forward them to external | ||
visualization tools such as [Jaeger](https://www.jaegertracing.io/). Currently, only HTTP/S endpoints | ||
are supported, and they require additional configuration steps (see next section). | ||
|
||
- To disable tracing you just need to set *enabled* to false: | ||
```sh | ||
xe observer-param-set uuid=<OBSERVER_UUID> enabled=false | ||
``` | ||
|
||
### Enabling smapi component | ||
|
||
- *smapi* component is currently considered experimental and is filtered by default. To enable it, you must | ||
explicitly configure the following in **xapi.conf**: | ||
```ini | ||
observer-experimental-components="" | ||
``` | ||
This tells XAPI that no components are considered experimental, thereby allowing *smapi* to be traced. | ||
A modification to **xapi.conf** requires a restart of the XAPI toolstack. | ||
|
||
### Enabling HTTP/S endpoints | ||
|
||
- By default HTTP and HTTPS endpoints are disabled. To enable them, add the following lines to **xapi.conf**: | ||
```ini | ||
observer-endpoint-http-enabled=true | ||
observer-endpoint-https-enabled=true | ||
``` | ||
gthvn1 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
As with enabling *smapi* component, modifying **xapi.conf** requires a restart of the XAPI toolstack. | ||
*Note*: HTTPS endpoint support is available but not tested and may not work. | ||
|
||
### Sending local trace to endpoint | ||
|
||
By default, traces are generated locally in the `/var/log/dt` directory. You can copy or forward | ||
these traces to another location or endpoint using the `xs-trace` tool. For example, if you have | ||
a *Jaeger* server running locally, you can run: | ||
|
||
```sh | ||
xs-trace /var/log/dt/ http://127.0.0.1:9411/api/v2/spans | ||
``` | ||
|
||
You will then be able to visualize the traces in Jaeger. | ||
|
||
### Tagging Trace Sessions for Easier Search | ||
|
||
#### Specific attributes | ||
To make trace logs easier to locate and analyze, it can be helpful to add custom attributes around the | ||
execution of specific commands. For example: | ||
|
||
```sh | ||
# xe observer-param-set uuid=<OBSERVER_UUID> attributes:custom.random=1234 | ||
# xe vm-start ... | ||
gthvn1 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
# xe observer-param-clear uuid=<OBSERVER_UUID> param-name=attributes param-key=custom.random | ||
``` | ||
|
||
This technique adds a temporary attribute, *custom.random=1234*, which will appear in the generated trace | ||
spans, making it easier to search for specific activity in trace visualisation tools. It may also be possible | ||
to achieve similar tagging using baggage parameters directly in individual *xe* commands, but this approach | ||
is currently undocumented. | ||
|
||
#### Baggage | ||
|
||
*Baggage*, contextual information that resides alongside the context, is supported. This means you can run | ||
the following command: | ||
|
||
```sh | ||
BAGGAGE="mybaggage=apples" xe vm-list | ||
``` | ||
|
||
You will be able to search for tags `mybaggage=apples`. | ||
|
||
#### Traceparent | ||
|
||
Another way to assist in trace searching is to use the `TRACEPARENT` HTTP header. It is an HTTP header field that | ||
identifies the incoming request. It has a [specific format](https://www.w3.org/TR/trace-context/#traceparent-header) | ||
and it is supported by **XAPI**. Once generated you can run command as: | ||
|
||
```sh | ||
TRACEPARENT="00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01" xe vm-list | ||
``` | ||
|
||
And you will be able to look for trace *4bf92f3577b34da6a3ce929d0e0e4736*. | ||
|
||
### Links | ||
|
||
- [Opentelemetry](https://opentelemetry.io/) | ||
- [Trace Context](https://www.w3.org/TR/trace-context/) | ||
- [Baggage](https://opentelemetry.io/docs/concepts/signals/baggage/) | ||
- [Ocaml opentelemetry module](https://ocaml.org/p/opentelemetry/latest) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.