Skip to content

Commit 1aec885

Browse files
authored
Add profilecli docs (#2631)
1 parent 782a1d6 commit 1aec885

File tree

1 file changed

+106
-39
lines changed

1 file changed

+106
-39
lines changed

docs/sources/configure-server/about-server-api.md

Lines changed: 106 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -7,18 +7,19 @@ weight: 20
77

88
# Pyroscope Server HTTP API Reference
99

10-
Pyroscope server exposes an HTTP API that can be used to query profiling data and to ingest profiling data from other sources.
10+
Pyroscope server exposes an HTTP API for querying profiling data and ingesting profiling data from other sources.
1111

1212
## Authentication
13-
Grafana Pyroscope does not come with any included authentication layer. Operators are expected to run an authenticating reverse proxy in front of your services.
1413

15-
Note that when using Pyroscope in multi-tenant mode, Pyroscope requires the HTTP header `X-Scope-OrgID` to be set to a string identifying the tenant; the responsibility of populating this value should be handled by the authenticating reverse proxy. Please refer to the [multi-tenancy documentation]({{< relref "./about-tenant-ids" >}}) for more information.
14+
Grafana Pyroscope does not include an authentication layer. Operators should use an authenticating reverse proxy for security.
15+
16+
In multi-tenant mode, Pyroscope requires the X-Scope-OrgID HTTP header set to a string identifying the tenant. This responsibility should be handled by the authenticating reverse proxy. For more information, refer to the [multi-tenancy documentation]({{< relref "./about-tenant-ids" >}}).
1617

1718
## Ingestion
1819

19-
Currently there's just one endpoint: `POST /ingest`. It takes profile data in request body and metadata as query params.
20+
There is one primary endpoint: POST /ingest. It accepts profile data in the request body and metadata as query parameters.
2021

21-
It takes multiple query parameters:
22+
The following query parameters are accepted:
2223

2324
| Name | Description | Notes |
2425
|:-------------------|:----------------------------------------|:--------------------------------|
@@ -31,27 +32,29 @@ It takes multiple query parameters:
3132
| `units` | name of the profiling data unit | optional (default is `samples` |
3233
| `aggregrationType` | type of aggregration to merge profiles | optional (default is `sum`) |
3334

34-
* `name` specifies application name. For example:
35+
36+
`name` specifies application name. For example:
3537
```
3638
my.awesome.app.cpu{env=staging,region=us-west-1}
3739
```
3840

39-
Request body contains the profiling data, and request header `Content-Type` may also be alongside `format` to determine profiling data format.
41+
The request body contains profiling data, and the Content-Type header may be used alongside format to determine the data format.
4042

4143
Some of the query parameters depend on the format of profiling data. Pyroscope currently supports three major ingestion formats.
4244

4345
### Text Formats
4446

45-
Most simple ingestion formats send a single type of profiling data (for example, `cpu` samples) and don't usually support any metadata (e.g labels) within the format to indicate which kind of data they are sending.
46-
In these cases, all the metadata is taken from the query parameters, and the format itself is defined by `format` query parameter.
47-
The formats that work this way are:
47+
These formats handle simple ingestion of profiling data, such as `cpu` samples, and typically don't support metadata (e.g., labels) within the format. All necessary metadata is derived from query parameters, and the format is specified by the `format` query parameter.
4848

49-
* `folded`. Some software also call this format `collapsed`. This is the default format. With this format you put one stacktrace per line with a number of samples you've captured for that particular stacktrace, for example:
49+
**Supported Formats:**
50+
51+
- **Folded**: Also known as `collapsed`, this is the default format. Each line contains a stacktrace followed by the sample count for that stacktrace. For example:
5052
```
5153
foo;bar 100
5254
foo;baz 200
5355
```
54-
* `lines` — This format is similar to `folded`, except there's no number of samples per stacktrace but a single line per sample, for example:
56+
57+
- **Lines**: Similar to `folded`, but it represents each sample as a separate line rather than aggregating samples per stacktrace. For example:
5558
```
5659
foo;bar
5760
foo;bar
@@ -61,55 +64,119 @@ foo;bar
6164

6265
### pprof format
6366

64-
`pprof` is a binary profiling data format popular in many languages, especially in the Go ecosystem.
67+
The `pprof` format is a widely used binary profiling data format, particularly prevalent in the Go ecosystem.
6568

66-
When this format is used, some of the query parameters behave slightly different:
67-
* `format` should be set to `pprof`.
68-
* `name` contains the _prefix_ of the application name. Since a single request may contain multiple profile types, the final application name is created concatenating this prefix and the profile type. For example, if you send cpu profiling data and set `name` to `my-app{}`, it will appear in pyroscope as `my-app.cpu{}`.
69-
* `units`, `aggregationType` and `sampleRate` are ignored, and the actual values depends on the profile types available in the data (see the next section, "Sample Type Configuration").
69+
When using this format, certain query parameters have specific behaviors:
70+
71+
- **format**: This should be set to `pprof`.
72+
- **name**: This parameter contains the _prefix_ of the application name. Since a single request might include multiple profile types, the complete application name is formed by concatenating this prefix with the profile type. For instance, if you send CPU profiling data and set `name` to `my-app{}`, it will be displayed in pyroscope as `my-app.cpu{}`.
73+
- **units**, **aggregationType**, and **sampleRate**: These parameters are ignored. The actual values are determined based on the profile types present in the data (refer to the "Sample Type Configuration" section for more details).
7074

7175
#### Sample Type Configuration
7276

73-
Out of the box Pyroscope server supports [default Go profile types](https://github.com/pyroscope-io/pyroscope/blob/main/pkg/storage/tree/pprof.go#L37-L75) (`cpu`, `inuse_objects`, `inuse_space`, `alloc_objects`, `alloc_space`). When working with other software that generates data in pprof format you may have to provide a custom sample type configuration for pyroscope to be able to parse the data properly.
77+
Pyroscope server inherently supports standard Go profile types such as `cpu`, `inuse_objects`, `inuse_space`, `alloc_objects`, and `alloc_space`. When dealing with software that generates data in pprof format, you may need to supply a custom sample type configuration for Pyroscope to interpret the data correctly.
7478

75-
To see an example python script to ingest a pprof file with custom sample type configuration, see [this file](https://github.com/grafana/pyroscope/tree/main/examples/api/ingest_pprof.py).
79+
For an example Python script to ingest a pprof file with a custom sample type configuration, see **[this Python script](https://github.com/grafana/pyroscope/tree/main/examples/api/ingest_pprof.py).**
7680

77-
In order to ingest pprof data with a custom sample type configuration, you have to make the following changes to your requests:
78-
* use an HTTP form (`multipart/form-data`) Content-Type.
79-
* send the profile data in a form file field called `profile`.
80-
* send the sample type configuration in a form file field called `sample_type_config`.
81+
To ingest pprof data with custom sample type configuration, modify your requests as follows:
82+
* Set Content-Type to `multipart/form-data`.
83+
* Upload the profile data in a form file field named `profile`.
84+
* Include the sample type configuration in a form file field named `sample_type_config`.
8185

82-
Sample type configuration is a JSON object that looks like this:
86+
A sample type configuration is a JSON object formatted like this:
8387

8488
```json
8589
{
86-
"inuse_space": { // "inuse_space" here is the internal pprof type name
90+
"inuse_space": {
8791
"units": "bytes",
8892
"aggregation": "average",
8993
"display-name": "inuse_space_bytes",
90-
"sampled": false,
94+
"sampled": false
95+
},
96+
"alloc_objects": {
97+
"units": "objects",
98+
"aggregation": "sum",
99+
"display-name": "alloc_objects_count",
100+
"sampled": true
101+
},
102+
"cpu": {
103+
"units": "samples",
104+
"aggregation": "sum",
105+
"display-name": "cpu_samples",
106+
"sampled": true
91107
},
92108
// pprof supports multiple profiles types in one file,
93109
// so there can be multiple of these objects
94110
}
95111
```
96112

113+
Explanation of sample type configuration fields:
114+
115+
- **units**
116+
- Supported values: `samples`, `objects`, `bytes`
117+
- Description: Changes the units displayed in the frontend. `samples` = CPU samples, `objects` = objects in RAM, `bytes` = bytes in RAM.
118+
- **display-name**
119+
- Supported values: Any string.
120+
- Description: This becomes a suffix of the app name, e.g., `my-app.inuse_space_bytes`.
121+
- **aggregation**
122+
- Supported values: `sum`, `average`.
123+
- Description: Alters how data is aggregated on the frontend. Use `sum` for data to be summed over time (e.g., CPU samples, memory allocations), and `average` for data to be averaged over time (e.g., memory in-use objects).
124+
- **sampled**
125+
- Supported values: `true`, `false`.
126+
- Description: Determines if the sample rate (specified in the pprof file) is considered. Set to `true` for sampled events (e.g., CPU samples), and `false` for memory profiles.
127+
128+
This configuration allows for customized visualization and analysis of various profile types within Pyroscope.
129+
130+
### Uploading a pprof File to Pyroscope Server Using profilecli
131+
132+
Using `profilecli`, you can easily upload your pprof files to the Pyroscope server. `profilecli` is a command-line utility that simplifies the process by allowing you to directly upload profiles with a single command. Below is a guide on how you can use `profilecli` for uploading a pprof file:
133+
134+
#### Prerequisites
135+
- Ensure you have `profilecli` installed on your system.
136+
- Have the pprof file you want to upload ready.
137+
138+
#### Steps to Upload
139+
140+
1. **Identify the pprof file and Pyroscope server details.**
141+
142+
- Path to your pprof file: `path/to/your/pprof-file.pprof`
143+
- Pyroscope server URL: If using Grafana Cloud, for example, `https://profiles-prod-001.grafana.net`. For local instances, it could be `http://localhost:4040`.
144+
145+
2. **Determine Authentication Details (if applicable).**
146+
147+
- If using cloud or authentication is enabled on your Pyroscope server, you will need the following:
148+
- Username: `<username>`
149+
- Password: `<password>`
150+
151+
3. **Specify any Extra Labels (Optional).**
152+
153+
- You can add additional labels to your uploaded profile using the `--extra-labels` flag.
154+
155+
4. **Construct and Execute the Upload Command.**
156+
157+
- Here's a basic command template:
158+
```
159+
./profilecli upload --url=<pyroscope_server_url> --username=<username> --password=<password> path/to/your/pprof-file.pprof
160+
```
161+
162+
- Modify the placeholders (`<pyroscope_server_url>`, `<username>`, `<password>`, `path/to/your/pprof-file.pprof`) with your actual values.
163+
164+
- Example command:
165+
```
166+
./profilecli upload --url=https://profiles-prod-001.grafana.net --username=my_username --password=my_password path/to/my_application_name.pprof
167+
```
168+
169+
5. **Check for Successful Upload.**
170+
171+
- After running the command, you should see a confirmation message indicating a successful upload. If there are any issues, `profilecli` will provide error messages to help you troubleshoot.
97172
98-
Here's a description of sample type configuration fields:
173+
#### Additional Tips
99174
100-
* `units`
101-
* Supported values: `samples`, `objects`, `bytes`
102-
* Description: This will change the units shown on the frontend. `samples` = cpu samples, `objects` = objects in RAM, `bytes` = bytes in RAM
103-
* `display-name`
104-
* Supported values: any string
105-
* Description: this becomes a suffix of app name, for example `my-app.inuse_space_bytes`
106-
* `aggregation`
107-
* Supported values: `sum`, `average`
108-
* Description: This changes how data is aggregated when shown on the frontend. use `sum` for data that you want to sum over time (e.g cpu samples, memory allocations), use `average` for data that you want to average over time (e.g. memory memory in-use objects)
109-
* `sampled`
110-
* Supported values: `true` / `false`
111-
* Description: This parameter defines if sample rate (specified in pprof file) is going to be taken into account or not. Set this to `true` for sampled events (e.g CPU samples), set it to `false` for memory profiles.
175+
- Use the `--verbose` flag if you need more detailed logs during the upload process.
176+
- The `--extra-labels` flag can be used multiple times to add several labels.
177+
- For help with `profilecli` commands, you can always use `./profilecli --help`.
112178
179+
Using `profilecli` streamlines the process of uploading profiles to Pyroscope, making it a convenient alternative to manual HTTP requests.
113180
114181
### JFR format
115182

0 commit comments

Comments
 (0)