[Bug Report] Sometimes SendEventAsync fails with UnauthorizedAccess

[gregorybleiker]

Context

• OS, version, SKU and CPU architecture used: Windows 10 Desktop, Windows 11 Desktop
• **Application's .NET Target Framework: .NET 8
• Device: PC
• **SDK version used: 1.42.3

Description of the issue

We are using the iot sdk to send events to an iot hub. We are using certificate based authentication. We are seeing a lot of intermittent Unauthorized Access errors. Here is a stack trace:

Microsoft.Azure.Devices.Client.Exceptions.UnauthorizedException: error(condition:amqp:unauthorized-access,description:{"errorCode":401002,"message":"Unauthorized access","trackingId":"086E25554C674B1782FB73BA73B6B752-G2:-TimeStamp:2025-07-01T12:59:54.653313718Z","timestampUtc":"2025-07-01T12:59:54.653313718Z","info":null})
---> Microsoft.Azure.Amqp.AmqpException: {"errorCode":401002,"message":"Unauthorized access","trackingId":"086E25554C674B1782FB73BA73B6B752-G2:-TimeStamp:2025-07-01T12:59:54.653313718Z","timestampUtc":"2025-07-01T12:59:54.653313718Z","info":null}
   at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.OpenAsyncResult.End(IAsyncResult result)
   at Microsoft.Azure.Amqp.AmqpObject.EndOpen(IAsyncResult result)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpIotSession.OpenSendingAmqpLinkAsync(IDeviceIdentity deviceIdentity, AmqpSession amqpSession, Nullable`1 senderSettleMode, Nullable`1 receiverSettleMode, String deviceTemplate, String moduleTemplate, String linkSuffix, String correlationId, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpIotSession.OpenSendingAmqpLinkAsync(IDeviceIdentity deviceIdentity, AmqpSession amqpSession, Nullable`1 senderSettleMode, Nullable`1 receiverSettleMode, String deviceTemplate, String moduleTemplate, String linkSuffix, String correlationId, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpIotSession.OpenTelemetrySenderLinkAsync(IDeviceIdentity deviceIdentity, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpUnit.EnsureSessionIsOpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpUnit.EnsureSessionIsOpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpUnit.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler.OpenAsync(CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.<>c__DisplayClass27_0.<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler.ExecuteWithErrorHandlingAsync[T](Func`1 asyncOperation)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.OpenInternalAsync(Boolean withRetry, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.EnsureOpenedAsync(Boolean withRetry, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.<>c__DisplayClass22_0.<<SendEventAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Client.TransientFaultHandling.RetryPolicy.<>c__DisplayClass34_0.<<RunWithRetryAsync>g__TaskWrapper|0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Devices.Client.TransientFaultHandling.RetryPolicy.RunWithRetryAsync[T](Func`1 taskFunc, ShouldRetry shouldRetry, Func`2 isTransient, Action`3 onRetrying, Boolean fastFirstRetry, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.TransientFaultHandling.RetryPolicy.RunWithRetryAsync(Func`1 taskFunc, ShouldRetry shouldRetry, Func`2 isTransient, Action`3 onRetrying, Boolean fastFirstRetry, CancellationToken cancellationToken)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler.SendEventAsync(Message message, CancellationToken cancellationToken)

On the IoT hub we see

ResourceType
IOTHUBS
OperationName
deviceConnect
ResultType
401002
ResultDescription
Unauthorized access
Level
Error
DurationMs
0
SourceSystem
Azure
connectionId_g
0171242d-8387-4402-bebb-d26cb41b8b38
RelatedActivityId_g
4efcfc26-de8a-4193-adf7-6515b23cc4eb
identity_s
LSUOKFFUWLPQ
properties_s
{"statusCode":"401","errorMessage":"Unauthorized access","deviceId":"LSUOKFFUWLPQ","trackingId":"1BD5E33A0CE848F68202D439E5252E5C-G2:-TimeStamp:2025-07-01T13:00:54.682360032Z","authType":null,"protocol":"Amqp","sdkVersion":".NET/1.42.3+2a97dd06e82cb67c5be8a7c7d189bef8048a3a86 (.NET 8.0.17; Microsoft Windows 10.0.19045 WindowsProduct:0x00000030; X64; {3C59FEBE-1D13-4709-84A3-B5A31FF22988})","maskedIpAddress":"146.185.0.XXX"}
location_s
WestEurope
Type
AzureDiagnostics

Code sample exhibiting the issue

// in init code
        var certWithPw = new X509Certificate2(certificate, "<pw>"); 
        var auth = new DeviceAuthenticationWithX509Certificate(proxyId, certWithPw);
        hubClient= DeviceClient.Create(hub, auth, TransportType.Amqp);

        // we have our own retry mechanism, so we don't want to use build-in retry
        hubClient.SetRetryPolicy(new NoRetry());
...
// in message handler (factory defined elsewhere)

        using var hubMessage = MessageFactory.CreateMessage(rawData);
        await hubClient.SendEventAsync(hubMessage, ct);

[gha-zund]

Hi! I'm a co-worker of gregorybleiker. Some additional information:

• the device is not disabled at IoT-Hub device management
• the certificate is not expired
• certificate renewal ist not the cause of this

It could be, that the device successfully sends data and then, seems like out of a sudden, this exception occurs. After restarting the PC, data can be sent again successfully. Could this be a hint to the cause?

As a workaround we already tried to automatically:

• immediate retry
• retry with backoff
• dispose and recreate DeviceClient before retry
  All of that did not resolve the problem.

We cannot restart the process the application runs in due to this exception due to requirements of the business logic of this application.

Is there a way to check for clock skew against IoT-Hub? To have evidence that the exception is not because of a time sync problem.

Any other ideas what the cause could be?