[Breaking change]: HttpClientFactory logging now redacts all header values by default

[CarnaViire]

Description

HttpClientFactory's default logging includes Trace level logs that outputs all the request and response headers. There is an existing API RedactLoggedHeaders to select which of these headers are sensitive, so their values should not be logged (the header names remain, but the values are substituted with *). Previously, if none of the headers were specified as sensitive with RedactLoggedHeaders, all headers were considered non-sensitive and were logged with all their values as-is. Starting with .NET 9 RC-1 (dotnet/runtime#106271), unless specified, all headers are instead considered sensitive and all the values would be redacted by default.

Version

.NET 9 RC 1

Previous behavior

If RedactLoggedHeaders is not called, all the headers are logged as-is. If RedactLoggedHeaders is called, the selected headers are redacted, other headers are logged as-is.

services.AddHttpClient("default", ...); // 1

services.AddHttpClient("redacted-predicate", ...) // 2
    .RedactLoggedHeaders(h => h.StartsWith("Auth") || h.StartsWith("X-"));

services.AddHttpClient("redacted-collection", ...) // 3
    .RedactLoggedHeaders(new[] { "Authorization", "X-Sensitive", });

(1) By default -- not redacted

trce: System.Net.Http.HttpClient.default.ClientHandler[102]
      Request Headers:
      Authorization: NTLM blob
      X-Sensitive: some, secret, values
      X-Other: some, other, values
      Cache-Control: no-cache

(2) Redacted with predicate

trce: System.Net.Http.HttpClient.redacted-predicate.ClientHandler[102]
      Request Headers:
      Authorization: *
      X-Sensitive: *
      X-Other: *
      Cache-Control: no-cache  

(3) Redacted with collection

trce: System.Net.Http.HttpClient.redacted-collection.ClientHandler[102]
      Request Headers:
      Authorization: *
      X-Sensitive: *
      X-Other: some, other, values
      Cache-Control: no-cache  

New behavior

If RedactLoggedHeaders is not called, all the headers are redacted. If RedactLoggedHeaders is called, the selected headers are redacted, other headers are logged as-is.

services.AddHttpClient("default", ...); // 1 <--- CHANGED

services.AddHttpClient("redacted-predicate", ...) // 2 <--- remained the same
    .RedactLoggedHeaders(h => h.StartsWith("Auth") || h.StartsWith("X-"));

services.AddHttpClient("redacted-collection", ...) // 3 <--- remained the same
    .RedactLoggedHeaders(new[] { "Authorization", "X-Sensitive", });

(1) By default -- CHANGED -- all is redacted

trce: System.Net.Http.HttpClient.default.ClientHandler[102]
      Request Headers:
      Authorization: *
      X-Sensitive: *
      X-Other: *
      Cache-Control: *

(2) Redacted with predicate -- remained the same

trce: System.Net.Http.HttpClient.redacted-predicate.ClientHandler[102]
      Request Headers:
      Authorization: *
      X-Sensitive: *
      X-Other: *
      Cache-Control: no-cache  

(3) Redacted with collection -- remained the same

trce: System.Net.Http.HttpClient.redacted-collection.ClientHandler[102]
      Request Headers:
      Authorization: *
      X-Sensitive: *
      X-Other: some, other, values
      Cache-Control: no-cache  

Type of breaking change

• Binary incompatible: Existing binaries might encounter a breaking change in behavior, such as failure to load or execute, and if so, require recompilation.
• Source incompatible: When recompiled using the new SDK or component or to target the new runtime, existing source code might require source changes to compile successfully.
• Behavioral change: Existing binaries might behave differently at run time.

Reason for change

Not specifying sensitive headers can lead to exposing sensitive information in logs. This change will make the logging safe by default.

Recommended action

It is recommended that all the users, if they wish to log the headers, should assess which headers can or cannot contain sensitive information and explicitly specify it using the RedactLoggedHeaders API. This can be done per-client, or globally for all clients by using the ConfigureHttpClientDefaults API.

Consider using an "allow-list" approach rather than a "block-list" approach.

If you strongly believe none of your headers can contain sensitive information, or for internal debug purposes, you can choose to disable the redaction completely by passing a delegate returning false.

private static readonly string[] SafeToLogHeaders = ....;
private static readonly string[] SuperSecretHeaders = ....;

// specify for all clients
services.ConfigureHttpClientDefaults(b =>
    b.RedactLoggedHeaders(h =>
        Array.IndexOf(SafeToLogHeaders, h) == -1)); // log values only for SafeToLogHeaders

// "globally" specified RedactLoggedHeaders can be overriden per-name
// NOTE: RedactLoggedHeaders completely replaces the previously specified check
services.AddHttpClient("override")
    .RedactLoggedHeaders(SuperSecretHeaders); // log values for everything except SuperSecretHeaders

// -OR-

// (dangerous) disable header value redaction for all clients
services.ConfigureHttpClientDefaults(b => b.RedactLoggedHeaders(_ => false));

Feature area

Extensions, Networking

Affected APIs

NuGet package: Microsoft.Extensions.Http
API: Microsoft.Extensions.DependencyInjection.AddHttpClient(...) -- all overloads -- in case RedactLoggedHeaders was not called on it (and Trace level logs are being collected).

---

Associated WorkItem - 297242