[RFC]: Per-request metrics for the offline API.

[maxdebayser]

Motivation.

In V0 when calling LLM.generate() the metrics field of the RequestOutput object was set to a RequestMetrics object:

@dataclass
class RequestMetrics:
    """Metrics associated with a request.

    Attributes:
        arrival_time: The time when the request arrived.
        first_scheduled_time: The time when the request was first scheduled.
        first_token_time: The time when the first token was generated.
        time_in_queue: The time the request spent in the queue.
        finished_time: The time when the request was finished.
        scheduler_time: The time spent in the scheduler when this request was
                        being considered by the scheduler.
        model_forward_time: The time spent in the model forward pass when this
                            request was in the batch.
        model_execute_time: The time spent in the model execute function. This
                            will include model forward, block/sync across
                            workers, cpu-gpu sync time and sampling time.
    """

    arrival_time: float
    last_token_time: float
    first_scheduled_time: Optional[float]
    first_token_time: Optional[float]
    time_in_queue: Optional[float]
    finished_time: Optional[float] = None
    scheduler_time: Optional[float] = None
    model_forward_time: Optional[float] = None
    model_execute_time: Optional[float] = None

In V1 this was removed and the field was returned as None instead. Since the discussion on the rationale of this removal wasn't easy to find, PR #24947 added stats back, but now as a RequestStateStats object:

class RequestStateStats:
    """Stats that need to be tracked across delta updates."""

    num_generation_tokens: int = 0

    # This is an engine frontend timestamp (wall-clock)
    arrival_time: float = 0.0

    # These are engine core timestamps (monotonic)
    queued_ts: float = 0.0
    scheduled_ts: float = 0.0
    first_token_ts: float = 0.0
    last_token_ts: float = 0.0

    # first token latency
    first_token_latency: float = 0.0

This PR then prompted the discussion on whether this is a useful feature to support and what the best way to present these metrics is. The purpose of this RFC is to structure this discussion.

Proposed Change.

The expected results of this RFC are:

1. Hear from the community about the use cases for this feature
2. A decision on whether to support this issue
3. How to present the metrics, taking the nature of different kinds of timestamp into account https://docs.vllm.ai/en/latest/design/metrics.html#interval-calculations
4. If the offline API supports this feature, should the online API also support it?

Feedback Period.

2 weeks

CC List.

@markmc @robertgshaw2-redhat @huijjj @DarkLight1337 @njhill @frank-wei

Any Other Things.

Related PRs and issues:

• add(v1): RequestStatesStats to RequestOutput #24947
• [V1][Metrics] Add API for accessing in-memory Prometheus metrics #17010
• Pass down metrics regarding RequestMetrics #26210
• [Bug]: RequestMetrics object (accessed through output[0].metrics) is None #15394

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[maxdebayser]

My use case for this feature is to be able to compare the performance of different models when given the same prompt.

[simon-mo]

Yes I'm in favor, having the metrics will be very helpful in performance diagnostics.

[maxdebayser]

@simon-mo , are you also in favor of metrics for the online API?

[22quinn]

These metrics are very useful. We need per-engine-step, i.e. batch-level, metrics to measure inference performance in every RL step.

[huijjj]

I also find this feature highly useful. Although I made a few code modifications, I’ve used the functionality that existed in v0 to analyze how each request was scheduled and to study its relationship to performance. Being able to observe the internal timing and scheduling of requests within the engine provides valuable insight, and I wrote this blog post based on that experience. Although it focuses mainly on TRT-LLM, it still serves as a good example of how this kind of functionality can be applied and why it’s valuable.
Of course, similar observations can be approximated without such internal instrumentation. For instance, by sending requests through the streaming API while timestamping events on the client side. However, that approach involves many additional steps and makes it difficult to determine which tokens were scheduled in the same iteration, or when a request was first picked up by the scheduler. Information about the engine’s internal behavior remains quite limited in that case.
For that reason, I think having such a feature built into the engine is very helpful. Judging from the number of related issues and PRs, it seems that many others have felt a similar need. I’ve therefore added a minimal implementation of this functionality in my PR #24947.
I noticed a comment mentioning that the timestamps in my PR are not based on wall-clock time, so care should be taken when interpreting them. In practice, what matters most is the interval between timestamps, so this limitation doesn’t affect most analyses. Still, I think this issue is a good opportunity to better understand how this kind of feature is used and, if needed, consider switching to a wall-clock source such as time.perf_counter.
Finally, if allowed, I’d be happy to implement any follow-up changes that are decided in this issue. I’d love to have a productive discussion on how to make this feature more robust and useful.

[maxdebayser]

I think these are good points in favor of returning the metrics and they apply equally to the offline as well as the online API.

So maybe we should also think about the potential concerns:

1. Maintainability

• It could inadvertently expose implementation details that will change over time, forcing us to make breaking API changes
• It could inadvertently make it more difficult to refactor the code

2. Security

• Depending on the information we expose, such as if the prefix cache was used, we could reveal sensitive information (because now you can confirm that a given prompt was previously seen, but on the other hand you can use timing as a less precise way to obtain this information).

3. Complexity

• How do we make sure that the returned statistics are correct considering chunked prefill, prefix caching, KV-Cache offloading and disaggregated prefill?

4. Performance

• How much performance impact per request is acceptable?

(nice blog post, btw, @huijjj )

[alew3]

Should this be implemented together with OpenTelemetry support in V1? It hasn't been ported yet as per #23767