Cannot resolve scoped service 'KubeOps.Operator.Controller.IEventQueue`1[MyOperator.Entities.V1DemoEntity]' from root provider

[hemantkathuria]

Describe the bug
Not able to debug the sample operator in .Net 6.0. Getting the below exception.

To Reproduce

1. Created the sample project via template
2. Project is targeting KubeOps 6.5.3 and .Net 6.0
3. Did dotnet run install
4. Did dotnet run and got the below error

Startup Leader Elector for operator "myoperator". Unhandled exception. System.InvalidOperationException: Cannot resolve scoped service 'KubeOps.Operator.Controller.IEventQueue1[MyOperator.Entities.V1DemoEntity]' from root provider.
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteValidator.ValidateResolution(Type serviceType, IServiceScope scope, IServiceScope rootScope)
at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.ConstructorMatcher.CreateInstance(IServiceProvider provider)
at Microsoft.Extensions.DependencyInjection.ActivatorUtilities.CreateInstance(IServiceProvider provider, Type instanceType, Object[] parameters)
at KubeOps.Operator.Builder.OperatorBuilder.<>c__DisplayClass18_0.b__3(ControllerRegistration r)
at System.Linq.Enumerable.SelectEnumerableIterator2.MoveNext() at System.Collections.Generic.List1.InsertRange(Int32 index, IEnumerable1 collection) at System.Collections.Generic.List1.AddRange(IEnumerable1 collection) at KubeOps.Operator.Controller.ResourceControllerManager.StartAsync(CancellationToken cancellationToken) at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at KubeOps.Operator.Commands.RunOperator.OnExecuteAsync() at McMaster.Extensions.CommandLineUtils.Conventions.ExecuteMethodConvention.InvokeAsync(MethodInfo method, Object instance, Object[] arguments) at McMaster.Extensions.CommandLineUtils.Conventions.ExecuteMethodConvention.OnExecute(ConventionContext context, CancellationToken cancellationToken) at McMaster.Extensions.CommandLineUtils.Conventions.ExecuteMethodConvention.<>c__DisplayClass0_0.<<Apply>b__0>d.MoveNext() --- End of stack trace from previous location --- at McMaster.Extensions.CommandLineUtils.CommandLineApplication.ExecuteAsync(String[] args, CancellationToken cancellationToken)

[Silvenga]

@hemantkathuria I think this is fixed via #437 - can you try and downgrade to v6.5.2? I'm not sure what version v6.5.3 was based on, since there's no tag in this repo. I'm wondering if something got mixed up on release.

[frabe1579]

Same problem here.
I've tried v6.5.2, v6.5.3, v6.5.1, all gives me same error.

Any other suggestion?

[buehler]

I'm still kind of baffled why there is a version 6.5.3, but no tag in github...

[ewassef]

Same here, not fixed with latest version on nuget as of june 15

[frabe1579]

The latest functioning version seems to be 6.3.1

[hemantkathuria]

Yes, at least the above error goes away, If I downgrade to 6.3.1. Since I am doing this for the first time, facing few more issues.

1. I did "dotnet run --install" and it installed only the CRD but no container. Once done, I did F5 in VS and the above error don't show up. Do we need to have container running in local K8s for debugging? I assumed, yes and pushed the same to my registery. Is it right understanding or there is a way to debug against local container? - With the container from my registry, I am able to debug. Assuming this is the only way :)

2. My console is showing this error constantly. I have not made any changes to DemoEntity.

I have noticed that the below error occurs until I have created a new custom resource for demo entity type. Once the custom resource is created, this error goes away. Also, I can confirm that no reconcile is getting fired in my democontroller and have no control to catch this error, I believe this needs to be catch in kubeops sdk?

fail: KubeOps.Operator.Kubernetes.ResourceWatcher[0] There was an error while watching the resource "MyOperator.Entities.V1DemoEntity". Microsoft.Rest.SerializationException: Unable to deserialize the response. ---> System.Text.Json.JsonException: The input does not contain any JSON tokens. Expected the input to start with a valid JSON token, when isFinalBlock is true. Path: $ | LineNumber: 0 | BytePositionInLine: 0. ---> System.Text.Json.JsonReaderException: The input does not contain any JSON tokens. Expected the input to start with a valid JSON token, when isFinalBlock is true. LineNumber: 0 | BytePositionInLine: 0. at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan1 bytes)
at System.Text.Json.Utf8JsonReader.Read()
at System.Text.Json.Serialization.JsonConverter1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state) --- End of inner exception stack trace --- at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, JsonReaderException ex) at System.Text.Json.Serialization.JsonConverter1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonConverter jsonConverter, Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonReaderState& readerState, Boolean isFinalBlock, ReadOnlySpan1 buffer, JsonSerializerOptions options, ReadStack& state, JsonConverter converterBase) at System.Text.Json.JsonSerializer.ContinueDeserialize[TValue](ReadBufferState& bufferState, JsonReaderState& jsonReaderState, ReadStack& readStack, JsonConverter converter, JsonSerializerOptions options) at System.Text.Json.JsonSerializer.ReadAll[TValue](Stream utf8Json, JsonTypeInfo jsonTypeInfo) at System.Text.Json.JsonSerializer.ReadAllUsingOptions[TValue](Stream utf8Json, Type returnType, JsonSerializerOptions options) at System.Text.Json.JsonSerializer.Deserialize[TValue](Stream utf8Json, JsonSerializerOptions options) at k8s.Kubernetes.CreateResultAsync[T](HttpRequestMessage httpRequest, HttpResponseMessage httpResponse, Nullable1 watch, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at k8s.Kubernetes.CreateResultAsync[T](HttpRequestMessage httpRequest, HttpResponseMessage httpResponse, Nullable1 watch, CancellationToken cancellationToken) at k8s.Kubernetes.ListClusterCustomObjectWithHttpMessagesAsync(String group, String version, String plural, Nullable1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable1 limit, String resourceVersion, String resourceVersionMatch, Nullable1 timeoutSeconds, Nullable1 watch, Nullable1 pretty, IDictionary2 customHeaders, CancellationToken cancellationToken) at k8s.WatcherExt.<>c__DisplayClass1_02.<b__0>d.MoveNext()
--- End of stack trace from previous location ---
at k8s.Watcher1.<>c.<CreateWatchEventEnumerator>b__21_1[TR](Task1 t)
at System.Threading.Tasks.ContinuationResultTaskFromResultTask2.InnerInvoke() at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj) at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at k8s.Watcher1.CreateWatchEventEnumerator(Func1 streamReaderCreator, Action1 onError, CancellationToken cancellationToken)+MoveNext()
at k8s.Watcher1.CreateWatchEventEnumerator(Func1 streamReaderCreator, Action1 onError, CancellationToken cancellationToken)+System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult() at k8s.Watcher1.WatcherLoop(CancellationToken cancellationToken)
at k8s.Watcher1.WatcherLoop(CancellationToken cancellationToken)

[rcm-steve-horne]

FWIW it looks like replacing the below line: https://github.com/buehler/dotnet-operator-sdk/blob/704c99c52e6c1d5f0a8bfe08b5f0438e6b92cd00/src/KubeOps/Operator/Builder/OperatorBuilder.cs#L180 with Services.BuildServiceProvider() seems to fix this for me; the service provider that's passed in to the delegate doesn't have IEventQueue registered.

[erin-allison]

Don't do this! Building a new service provider for each controller is not a solution. It will create a disjointed dependency tree and multiple extra instances of a handful of services.

[rcm-steve-horne]

Noted, will look for another solution.

[erin-allison]

I can't say easily based off what's in here - the one stack trace is poorly formatted to the point it's difficult to read and I have been too busy as of late to dig into this one.

[rcm-steve-horne]

Can't see any way to format nicely in SO, have attached my example as a text file in case that's any better.

KubeOps-450-StackTrace.txt

[rcm-steve-horne]

So https://stackoverflow.com/questions/55089755/cannot-resolve-scoped-service suggests we should be creating a service scope if we want to inject something with a scoped lifetime? In the meantime I've got this working locally by disabling scope validation (not proposed as a fix but might be useful to someone until this is fixed properly):

builder.Host.UseDefaultServiceProvider(i => i.ValidateScopes = false);

[Silvenga]

Ugh, I dislike the default DependencyInjection container, its errors aren't the most useful. Using AutoFac, I'm not seeing any issues (https://github.com/Contrast-Security-OSS/agent-operator/blob/master/src/Contrast.K8s.AgentOperator/Startup.cs#L61) - pinned at a version close to latest.

Hopefully I'll have time a bit later to repo this. I wonder if the assemblies got mixed up. The error looks correct.

ResourceControllerManager is added as a hosted service, which defaults to singleton scope. It's requesting the activation of IEventQueue which is bound in request scope via the activation of the transient scoped IManagedResourceController (which any good DI container should provide in their error messages).

In AutoFac, activating a request scoped service outside of a request scope creates a binding to the singleton scope by default. The default DependencyInjection resolves this ambiguity the other direction, by breaking.

The correct fix is by making the whole dependency graph either not-request-scoped, or request-scoped (since making a default request scope is only an option in the more advanced DI containers it seems). Since the ResourceControllerManager is a hosted controller in singleton scope, effectively all IManagedResourceController are in transitive singleton scope (which includes IEventQueue). So either the ResourceControllerManager has to create a fake request scope that's bound in transitive singleton scope, or IEventQueue should be transitive/singleton scope.