Skip to content

Conversation

@CodeBlanch
Copy link
Contributor

Changes

  • Reset ActivityTraceFlags.Recorded when creating activity if not requested by sampler(s) because it may be inherited from a parent

Details

[Ran into this while working on #6058.]

Scenario:

  • We have an incoming request with traceparent marked as Recorded.
  • OTel sampler decides to drop for whatever reason.
  • Because this is a root, even though a drop was requested, OTel SDK will still create a propagation-only span/activity.
  • Here is the issue: The created span/activity inherits recorded from the parent context and therefore has Recorded=true. A propagation-only span should have Recorded=false.

/cc @tarekgh @noahfalk @samsp-msft @cijothomas @alanwest

@dotnet-policy-service dotnet-policy-service bot added the community-contribution Indicates that the PR has been added by a community member label Jan 10, 2025
@tarekgh tarekgh added this to the 10.0.0 milestone Jan 10, 2025
@tarekgh
Copy link
Member

tarekgh commented Jan 10, 2025

            activity.ActivityTraceFlags = parentContext.TraceFlags;

should the fix done here instead?


Refers to: src/libraries/System.Diagnostics.DiagnosticSource/src/System/Diagnostics/Activity.cs:1236 in 5d2e54d. [](commit_id = 5d2e54d, deletion_comment = False)

@tarekgh tarekgh requested a review from noahfalk January 10, 2025 22:42
@tarekgh
Copy link
Member

tarekgh commented Jan 10, 2025

@CodeBlanch how likely this change can break anyone depending on the current behavior? I am asking to know if we need to file a breaking change for that.

@CodeBlanch
Copy link
Contributor Author

            activity.ActivityTraceFlags = parentContext.TraceFlags;

should the fix done here instead?

@tarekgh I thought about just removing that line. That would probably take care of it. The reason I went with the explicit clear is because right now ActivityTraceFlags only has the Recorded definition. But who knows in the future it may have other flags which actually should inherit from the parent. Kind of hard to predict 😄 Happy to this though just LMK.

@CodeBlanch
Copy link
Contributor Author

@CodeBlanch how likely this change can break anyone depending on the current behavior? I am asking to know if we need to file a breaking change for that.

Kind of hard to say.

Here are some thoughts...

The default sampler in OTel is ParentBased(AlwaysOn) which means first check the parent. If the parent is marked as Recorded (aka sampled) then respect that and create a recorded child. If there is no parent, create a recorded/sampled span (AlwaysOn).

So no one using the defaults will be impacted.

To be impacted users would have to switch the sampler or make a custom one which attempts to drop remote things. The bug is such that the drop isn't really respected. Recorded causes the spans to be exported (but they have IsAllDataRequested=false so they won't be populated). Impact is probably higher costs, maybe some screwy spans without data. What may be more interesting is that span, if it is propagated out of proc, will be transmitted as Recorded. So some other system downstream using ParentBased might decide to sample it fully essentially reversing the drop decision.

FWIW if this was being done in OTel SDK we would probably call it a breaking change to align behavior with spec.

@tarekgh
Copy link
Member

tarekgh commented Jan 11, 2025

I thought about just removing that line. That would probably take care of it. The reason I went with the explicit clear is because right now ActivityTraceFlags only has the Recorded definition. But who knows in the future it may have other flags which actually should inherit from the parent. Kind of hard to predict 😄 Happy to this though just LMK.

We don't have to remove the line if we are worried about the future adding to the flags. But we can just modify it to the following:

activity.ActivityTraceFlags = parentContext.TraceFlags & ~ActivityTraceFlags.Recorded;

@tarekgh
Copy link
Member

tarekgh commented Jan 11, 2025

FWIW if this was being done in OTel SDK we would probably call it a breaking change to align behavior with spec.

That makes me feel we need to file a breaking change to get this documented. @noahfalk may weigh on that too. Here is where you can file the breaking change https://github.com/dotnet/docs/issues/new?template=02-breaking-change.yml.

Copy link
Member

@tarekgh tarekgh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @CodeBlanch

@noahfalk
Copy link
Member

That makes me feel we need to file a breaking change to get this documented. @noahfalk may weigh on that too. Here is where you can file the breaking change https://github.com/dotnet/docs/issues/new?template=02-breaking-change.yml.

Yeah, I think a breaking change notice would be warranted here. For downlevel what is the plan? Just warn users that they won't be able to disable the recorded flag?

@tarekgh
Copy link
Member

tarekgh commented Jan 14, 2025

@CodeBlanch could you please file the breaking change in https://github.com/dotnet/docs/issues/new?template=02-breaking-change.yml? Thanks!

@CodeBlanch
Copy link
Contributor Author

@tarekgh Done, see link above ^

@tarekgh tarekgh added the breaking-change Issue or PR that represents a breaking API or functional change over a prerelease. label Jan 14, 2025
@dotnet-policy-service
Copy link
Contributor

Added needs-breaking-change-doc-created label because this PR has the breaking-change label.

When you commit this breaking change:

  1. Create and link to this PR and the issue a matching issue in the dotnet/docs repo using the breaking change documentation template, then remove this needs-breaking-change-doc-created label.
  2. Ask a committer to mail the .NET Breaking Change Notification DL.

Tagging @dotnet/compat for awareness of the breaking change.

@dotnet-policy-service dotnet-policy-service bot added the needs-breaking-change-doc-created Breaking changes need an issue opened with https://github.com/dotnet/docs/issues/new?template=dotnet label Jan 14, 2025
@tarekgh
Copy link
Member

tarekgh commented Jan 14, 2025

/ba-g the failure is known build timeout issue happening in other PRs too.

@tarekgh tarekgh merged commit 7132268 into dotnet:main Jan 14, 2025
80 of 82 checks passed
Comment on lines -1236 to +1238
activity.ActivityTraceFlags = parentContext.TraceFlags;
// Note: Don't inherit Recorded from parent as it is set below
// based on sampling decision
activity.ActivityTraceFlags = parentContext.TraceFlags & ~ActivityTraceFlags.Recorded;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this line exist at all? If W3C trace context introduces new flags, what is to say that inheriting the flags from the parent will be the correct thing to do for those new flags?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see this was already discussed... removing the line seems more correct to me...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In looking at the latest editor's draft of the spec, there is a new flag proposed: Random Trace ID Flag. My read of this flag is that it actually should be inherited from the parent, but there's no way to be certain this will be true for all new flags.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping this line is not hurting for now. Any newly introduced flag will need to be decided if it is inheritable from the parent or not anyway.

@ericstj
Copy link
Member

ericstj commented Oct 3, 2025

📋 Breaking Change Documentation Required

Create a breaking change issue with AI-generated content

Generated by Breaking Change Documentation Tool - 2025-10-03 13:49:29

@tarekgh
Copy link
Member

tarekgh commented Oct 3, 2025

@ericstj the breaking change for this is already submitted and finalized dotnet/docs#44282.

@tarekgh tarekgh removed the needs-breaking-change-doc-created Breaking changes need an issue opened with https://github.com/dotnet/docs/issues/new?template=dotnet label Oct 3, 2025
@ericstj
Copy link
Member

ericstj commented Oct 3, 2025

Thank you @tarekgh

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-System.Diagnostics.Activity breaking-change Issue or PR that represents a breaking API or functional change over a prerelease. community-contribution Indicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants