Skip to content

Add LLM environment detection telemetry data point with rule-based detection system and multiple environment support #50663

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Sep 18, 2025
Merged

Add LLM environment detection telemetry data point with rule-based detection system and multiple environment support #50663

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
09091ca
Initial plan
Copilot Sep 5, 2025
57b7837
Add LLM environment detection telemetry with CLAUDECODE support
Copilot Sep 5, 2025
dc5494e
Change LLM telemetry to use simpler property name and descriptive values
Copilot Sep 5, 2025
e4c52b4
Simplify LLM telemetry tests to only test essential scenarios
Copilot Sep 8, 2025
c242c38
Merge branch 'main' into copilot/fix-50641
baronfel Sep 12, 2025
92ec5d8
Implement rule-based environment detection system with nullable refer…
Copilot Sep 12, 2025
7736af5
Update src/Cli/dotnet/Telemetry/LLMEnvironmentDetectorForTelemetry.cs
baronfel Sep 17, 2025
984a512
Fix LLM detection syntax and add multiple environment support with te…
Copilot Sep 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions src/Cli/dotnet/Telemetry/ILLMEnvironmentDetector.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

#nullable disable

namespace Microsoft.DotNet.Cli.Telemetry;

internal interface ILLMEnvironmentDetector
{
string GetLLMEnvironment();
}
28 changes: 28 additions & 0 deletions src/Cli/dotnet/Telemetry/LLMEnvironmentDetectorForTelemetry.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
// Licensed to the .NET Foundation under one or more agreements.
// The .NET Foundation licenses this file to you under the MIT license.

#nullable disable

namespace Microsoft.DotNet.Cli.Telemetry;

internal class LLMEnvironmentDetectorForTelemetry : ILLMEnvironmentDetector
{
// Systems where the variable must be present and not-null
private static readonly string[] _claudeVariables = [
// Claude Code
"CLAUDECODE"
];

public string GetLLMEnvironment()
{
foreach (var variable in _claudeVariables)
{
if (!string.IsNullOrEmpty(Environment.GetEnvironmentVariable(variable)))
Copy link
Member

@MichaelSimons MichaelSimons Sep 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am sure this will be expanded over time to support numerous environments. Given that assumption, it might make sense to define rules that encapsule everything in a single place and support different conditions. Of course we can start simple (use a simple dictionary<rulename, envs> and build up as the specific requirements arise.

    internal static readonly IReadOnlyList<LLMEnvironmentRule> Rules = new[]
    {
        // Claude Code
        new LLMEnvironmentRule(
            name: "claude",
            anyOfEnvVars: new[] { "CLAUDECODE" }
        ),

        // Examples (commented out): enable when you know the correct markers
        // new LLMEnvironmentRule(
        //     name: "openai",
        //     anyOfEnvVars: new[] { "OPENAI_RUN_CONTEXT", "OPENAI_LOGDIR" }),
        //
        // new LLMEnvironmentRule(
        //     name: "copilot",
        //     allOfEnvVars: new[] { "GITHUB_COPILOT", "VSCODE_PID" }),
        //
        // new LLMEnvironmentRule(
        //     name: "my-llm",
        //     envVarEquals: new Dictionary<string,string> { ["MY_LLM_ACTIVE"] = "1" }),
    };

    public string GetLLMEnvironment()
    {
        foreach (var rule in Rules)
        {
            if (rule.IsMatch(Environment.GetEnvironmentVariable))
            {
                return rule.Name;
            }
        }

        return null;
    }

Copy link
Member

@baronfel baronfel Sep 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes total sense to me.

@copilot implement a design like this where we can define abstract rules for matching. Also convert the existing CI detection to use similar (not necessarily duplicated) logic.

Copy link
Member

@baronfel baronfel Sep 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot implement @MichaelSimons's feedback, and make sure that you don't disable nullability on the source code you create.

Copy link
Contributor Author

Copilot AI Sep 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented @MichaelSimons's feedback with a rule-based environment detection system. Created abstract EnvironmentDetectionRule classes that encapsulate matching logic in a single place with support for different conditions (boolean variables, all-present, any-present, and result-returning rules). Converted both CI and LLM detection to use this system while maintaining existing functionality. All source code uses the globally-enabled nullable reference types. Commit 92ec5d8

Copy link
Member

@richlander richlander Sep 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reminds me a bit of this old proposal: dotnet/designs#19.

A virtuous flow would look like this:

  • .NET SDK exposes an MCP that prompts the LLM and explains how to configure itself for optimal use.
  • LLM / assistant / agent sets ENVs (or using a .env file) with DOTNET_DO_GOOD_THINGS
  • No LLM-specifc ENVs need to be read.

It's possible we need to target the specific LLMs for the moment and that's fine. It's more a question of what the long-term plan is.

{
return "claude";
}
}

return null;
}
}
6 changes: 5 additions & 1 deletion src/Cli/dotnet/Telemetry/TelemetryCommonProperties.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,12 @@ internal class TelemetryCommonProperties(
Func<string> getDeviceId = null,
IDockerContainerDetector dockerContainerDetector = null,
IUserLevelCacheWriter userLevelCacheWriter = null,
ICIEnvironmentDetector ciEnvironmentDetector = null)
ICIEnvironmentDetector ciEnvironmentDetector = null,
ILLMEnvironmentDetector llmEnvironmentDetector = null)
{
private readonly IDockerContainerDetector _dockerContainerDetector = dockerContainerDetector ?? new DockerContainerDetectorForTelemetry();
private readonly ICIEnvironmentDetector _ciEnvironmentDetector = ciEnvironmentDetector ?? new CIEnvironmentDetectorForTelemetry();
private readonly ILLMEnvironmentDetector _llmEnvironmentDetector = llmEnvironmentDetector ?? new LLMEnvironmentDetectorForTelemetry();
private readonly Func<string> _getCurrentDirectory = getCurrentDirectory ?? Directory.GetCurrentDirectory;
private readonly Func<string, string> _hasher = hasher ?? Sha256Hasher.Hash;
private readonly Func<string> _getMACAddress = getMACAddress ?? MacAddressGetter.GetMacAddress;
Expand All @@ -46,6 +48,7 @@ internal class TelemetryCommonProperties(
private const string LibcVersion = "Libc Version";

private const string CI = "Continuous Integration";
private const string LLM = "llm";

private const string TelemetryProfileEnvironmentVariable = "DOTNET_CLI_TELEMETRY_PROFILE";
private const string CannotFindMacAddress = "Unknown";
Expand All @@ -66,6 +69,7 @@ public FrozenDictionary<string, string> GetTelemetryCommonProperties()
{TelemetryProfile, Environment.GetEnvironmentVariable(TelemetryProfileEnvironmentVariable)},
{DockerContainer, _userLevelCacheWriter.RunWithCache(IsDockerContainerCacheKey, () => _dockerContainerDetector.IsDockerContainer().ToString("G") )},
{CI, _ciEnvironmentDetector.IsCIEnvironment().ToString() },
{LLM, _llmEnvironmentDetector.GetLLMEnvironment() },
{CurrentPathHash, _hasher(_getCurrentDirectory())},
{MachineIdOld, _userLevelCacheWriter.RunWithCache(MachineIdCacheKey, GetMachineId)},
// we don't want to recalcuate a new id for every new SDK version. Reuse the same path across versions.
Expand Down
36 changes: 36 additions & 0 deletions test/dotnet.Tests/TelemetryCommonPropertiesTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,13 @@ public void TelemetryCommonPropertiesShouldContainLibcReleaseAndVersion()
}
}

[Fact]
public void TelemetryCommonPropertiesShouldReturnIsLLMDetection()
{
var unitUnderTest = new TelemetryCommonProperties(getMACAddress: () => null, userLevelCacheWriter: new NothingCache());
unitUnderTest.GetTelemetryCommonProperties()["llm"].Should().BeOneOf("claude", null);
}

[Theory]
[MemberData(nameof(CITelemetryTestCases))]
public void CanDetectCIStatusForEnvVars(Dictionary<string, string> envVars, bool expected)
Expand All @@ -184,6 +191,35 @@ public void CanDetectCIStatusForEnvVars(Dictionary<string, string> envVars, bool
}
}

[Theory]
[MemberData(nameof(LLMTelemetryTestCases))]
public void CanDetectLLMStatusForEnvVars(Dictionary<string, string> envVars, string expected)
{
try
{
foreach (var (key, value) in envVars)
{
Environment.SetEnvironmentVariable(key, value);
}
new LLMEnvironmentDetectorForTelemetry().GetLLMEnvironment().Should().Be(expected);
}
finally
{
foreach (var (key, value) in envVars)
{
Environment.SetEnvironmentVariable(key, null);
}
}
}

public static IEnumerable<object[]> LLMTelemetryTestCases => new List<object[]>{
new object[] { new Dictionary<string, string> { { "CLAUDECODE", "1" } }, "claude" },
new object[] { new Dictionary<string, string> { { "CLAUDECODE", "true" } }, "claude" },
new object[] { new Dictionary<string, string> { { "CLAUDECODE", "anything" } }, "claude" },
new object[] { new Dictionary<string, string> { { "CLAUDECODE", "" } }, null },
new object[] { new Dictionary<string, string> { { "SomethingElse", "hi" } }, null },
};

public static IEnumerable<object[]> CITelemetryTestCases => new List<object[]>{
new object[] { new Dictionary<string, string> { { "TF_BUILD", "true" } }, true },
new object[] { new Dictionary<string, string> { { "GITHUB_ACTIONS", "true" } }, true },
Expand Down
Loading