New-AzKubernetesExtension - installing Flux fails with error "Failed to perform resource identity operation"

[janegilring]

Description

I am trying to install the Flux extension on an AKS cluster using the following command:

New-AzKubernetesExtension -ClusterName $resourceName -ClusterType ManagedCluster -Name flux -ResourceGroupName $Env:resourceGroup -ExtensionType microsoft.flux -IdentityType 'SystemAssigned'

This fails with an identity-related error as shown below.

Installing the extension using Azure CLI works fine:

 az k8s-extension create --name "flux" --extension-type "microsoft.flux" --cluster-name $resourceName --resource-group $Env:resourceGroup --cluster-type $ClusterType 

Issue script & Debug output

PS C:\Windows\system32> New-AzKubernetesExtension -ClusterName $resourceName -ClusterType ManagedCluster -Name flux -ResourceGroupName $Env:resourceGroup -ExtensionType microsoft.flux -IdentityType 'SystemAssigned'
Az.KubernetesConfiguration.internal\New-AzKubernetesExtension : Identity operation for resource 
'/subscriptions/16471a83-9151-456e-bbb1-463027bed604/resourceGroups/aks-rg/providers/Microsoft.ContainerService/managedClusters/Ag-AKS-Staging/providers/Microsoft.KubernetesConfiguration/extensions/flux' failed with 
error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{"error":{"code":"BadRequest","message":""}}'.'.
At C:\Program Files\WindowsPowerShell\Modules\Az.KubernetesConfiguration\0.7.0\custom\New-AzKubernetesExtension.ps1:223 char:9
+         Az.KubernetesConfiguration.internal\New-AzKubernetesExtension ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: ({ SubscriptionI...gned"
  }
} }:<>f__AnonymousType4`7) [New-AzKubernetesExtension_CreateExpanded], Exception
    + FullyQualifiedErrorId : FailedIdentityOperation,Microsoft.Azure.PowerShell.Cmdlets.KubernetesConfiguration.Cmdlets.NewAzKubernetesExtension_CreateExpanded 

# If trying without specifying -IdentityType, the following is returned:

 PS C:\Windows\system32> New-AzKubernetesExtension -ClusterName $resourceName -ClusterType $ClusterType -Name flux -ResourceGroupName $Env:resourceGroup -ExtensionType microsoft.flux
New-AzKubernetesExtension : The value of the parameter -IdentityType must be 'SystemAssigned'.
At line:1 char:1
+ New-AzKubernetesExtension -ClusterName $resourceName -ClusterType $Cl ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,New-AzKubernetesExtension

Environment data

$PSVersionTable

Name                           Value                                                                                                                                                                                                                                                                                                                                                                                                                                               
----                           -----                                                                                                                                                                                                                                                                                                                                                                                                                                               
PSVersion                      5.1.20348.1850                                                                                                                                                                                                                                                                                                                                                                                                                                      
PSEdition                      Desktop                                                                                                                                                                                                                                                                                                                                                                                                                                             
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                                                                                                                                                                                                                                                                                                                                                                             
BuildVersion                   10.0.20348.1850                                                                                                                                                                                                                                                                                                                                                                                                                                     
CLRVersion                     4.0.30319.42000                                                                                                                                                                                                                                                                                                                                                                                                                                     
WSManStackVersion              3.0                                                                                                                                                                                                                                                                                                                                                                                                                                                 
PSRemotingProtocolVersion      2.3                                                                                                                                                                                                                                                                                                                                                                                                                                                 
SerializationVersion           1.1.0.1

Module versions

Get-Module Az*

ModuleType Version    Name                                ExportedCommands                                                                                                                                                                                                                                                                                                                                                                                                         
---------- -------    ----                                ----------------                                                                                                                                                                                                                                                                                                                                                                                                         
Script     2.12.5     Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault...}                                                                                                                                                                                                                                                                                                                                                 
Script     0.7.0      Az.KubernetesConfiguration          {Get-AzKubernetesConfigFluxOperationStatus, Get-AzKubernetesConfiguration, Get-AzKubernetesConfigurationFlux, Get-AzKubernetesExtension...}                                                                                                                                                                                                                                                                              
Script     6.8.0      Az.Resources                        {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment...}

Error output

Resolve-AzError


   HistoryId: 32


Message        : [FailedIdentityOperation] : Identity operation for resource '/subscriptions/16471a83-9151-456e-bbb1-463027bed604/resourceGroups/aks-rg/providers/Microsoft.ContainerService/managedClusters/Ag-AKS-Staging/providers/Microsoft.KubernetesConfiguration/extensions/flux' failed with error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{"error":{"code":"BadRequest","message":""}}'.'.
StackTrace     : 
Exception      : System.Exception
InvocationInfo : {New-AzKubernetesExtension_CreateExpanded}
Line           :         Az.KubernetesConfiguration.internal\New-AzKubernetesExtension @PSBoundParameters
                 
Position       : At C:\Program Files\WindowsPowerShell\Modules\Az.KubernetesConfiguration\0.7.0\custom\New-AzKubernetesExtension.ps1:223 char:9
                 +         Az.KubernetesConfiguration.internal\New-AzKubernetesExtension ...
                 +         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 32

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @NarayanThiru.

Issue Details

---

Description

I am trying to install the Flux extension on an AKS cluster using the following command:

New-AzKubernetesExtension -ClusterName $resourceName -ClusterType ManagedCluster -Name flux -ResourceGroupName $Env:resourceGroup -ExtensionType microsoft.flux -IdentityType 'SystemAssigned'

This fails with an identity-related error as shown below.

Installing the extension using Azure CLI works fine:

 az k8s-extension create --name "flux" --extension-type "microsoft.flux" --cluster-name $resourceName --resource-group $Env:resourceGroup --cluster-type $ClusterType 

Issue script & Debug output

PS C:\Windows\system32> New-AzKubernetesExtension -ClusterName $resourceName -ClusterType ManagedCluster -Name flux -ResourceGroupName $Env:resourceGroup -ExtensionType microsoft.flux -IdentityType 'SystemAssigned'
Az.KubernetesConfiguration.internal\New-AzKubernetesExtension : Identity operation for resource 
'/subscriptions/16471a83-9151-456e-bbb1-463027bed604/resourceGroups/aks-rg/providers/Microsoft.ContainerService/managedClusters/Ag-AKS-Staging/providers/Microsoft.KubernetesConfiguration/extensions/flux' failed with 
error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{"error":{"code":"BadRequest","message":""}}'.'.
At C:\Program Files\WindowsPowerShell\Modules\Az.KubernetesConfiguration\0.7.0\custom\New-AzKubernetesExtension.ps1:223 char:9
+         Az.KubernetesConfiguration.internal\New-AzKubernetesExtension ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: ({ SubscriptionI...gned"
  }
} }:<>f__AnonymousType4`7) [New-AzKubernetesExtension_CreateExpanded], Exception
    + FullyQualifiedErrorId : FailedIdentityOperation,Microsoft.Azure.PowerShell.Cmdlets.KubernetesConfiguration.Cmdlets.NewAzKubernetesExtension_CreateExpanded 

# If trying without specifying -IdentityType, the following is returned:

 PS C:\Windows\system32> New-AzKubernetesExtension -ClusterName $resourceName -ClusterType $ClusterType -Name flux -ResourceGroupName $Env:resourceGroup -ExtensionType microsoft.flux
New-AzKubernetesExtension : The value of the parameter -IdentityType must be 'SystemAssigned'.
At line:1 char:1
+ New-AzKubernetesExtension -ClusterName $resourceName -ClusterType $Cl ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,New-AzKubernetesExtension

Environment data

$PSVersionTable

Name                           Value                                                                                                                                                                                                                                                                                                                                                                                                                                               
----                           -----                                                                                                                                                                                                                                                                                                                                                                                                                                               
PSVersion                      5.1.20348.1850                                                                                                                                                                                                                                                                                                                                                                                                                                      
PSEdition                      Desktop                                                                                                                                                                                                                                                                                                                                                                                                                                             
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}                                                                                                                                                                                                                                                                                                                                                                                                                             
BuildVersion                   10.0.20348.1850                                                                                                                                                                                                                                                                                                                                                                                                                                     
CLRVersion                     4.0.30319.42000                                                                                                                                                                                                                                                                                                                                                                                                                                     
WSManStackVersion              3.0                                                                                                                                                                                                                                                                                                                                                                                                                                                 
PSRemotingProtocolVersion      2.3                                                                                                                                                                                                                                                                                                                                                                                                                                                 
SerializationVersion           1.1.0.1

Module versions

Get-Module Az*

ModuleType Version    Name                                ExportedCommands                                                                                                                                                                                                                                                                                                                                                                                                         
---------- -------    ----                                ----------------                                                                                                                                                                                                                                                                                                                                                                                                         
Script     2.12.5     Az.Accounts                         {Add-AzEnvironment, Clear-AzConfig, Clear-AzContext, Clear-AzDefault...}                                                                                                                                                                                                                                                                                                                                                 
Script     0.7.0      Az.KubernetesConfiguration          {Get-AzKubernetesConfigFluxOperationStatus, Get-AzKubernetesConfiguration, Get-AzKubernetesConfigurationFlux, Get-AzKubernetesExtension...}                                                                                                                                                                                                                                                                              
Script     6.8.0      Az.Resources                        {Export-AzResourceGroup, Export-AzTemplateSpec, Get-AzDenyAssignment, Get-AzDeployment...}

Error output

Resolve-AzError


   HistoryId: 32


Message        : [FailedIdentityOperation] : Identity operation for resource '/subscriptions/16471a83-9151-456e-bbb1-463027bed604/resourceGroups/aks-rg/providers/Microsoft.ContainerService/managedClusters/Ag-AKS-Staging/providers/Microsoft.KubernetesConfiguration/extensions/flux' failed with error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{"error":{"code":"BadRequest","message":""}}'.'.
StackTrace     : 
Exception      : System.Exception
InvocationInfo : {New-AzKubernetesExtension_CreateExpanded}
Line           :         Az.KubernetesConfiguration.internal\New-AzKubernetesExtension @PSBoundParameters
                 
Position       : At C:\Program Files\WindowsPowerShell\Modules\Az.KubernetesConfiguration\0.7.0\custom\New-AzKubernetesExtension.ps1:223 char:9
                 +         Az.KubernetesConfiguration.internal\New-AzKubernetesExtension ...
                 +         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
HistoryId      : 32

Author:	janegilring
Assignees:	-
Labels:	Service Attention, bug, Kubernetes Configuration
Milestone:	-

[bavneetsingh16]

I am currently collaborating with the PowerShell team to resolve the issue. I will update this thread as soon as I have more information.

[Alex-wdy]

@lijinpei2008 Can you also pay attention to this issue? Can it be fixed?

[lijinpei2008]

Okay, I'm trying to fix it.

[lijinpei2008]

First of all, this error is expected.
This is because if the IdentityType value SystemAssigned is not provided when creating the Flux resource, the New-AzKubernetesExtension command will run for a long time and eventually throw a timeout error.

Also, creating a Flux resource with the command New-AzKubernetesExtension will be mutually exclusive with the resource created with the command New-AzKubernetesConfiguration.

Finally, when creating a Flux resource, you need to provide the parameter -AutoUpgradeMinorVersion and the parameter -ReleaseNamespace.

Here is the example:

• Create Kubernetes service
• Create Kubernetes - Azure Arc
• Run command: New-AzKubernetesExtension -ClusterName azpstest_cluster_arc -ClusterType ConnectedClusters -Name flux -ResourceGroupName azps_test_group -ExtensionType microsoft.flux -AutoUpgradeMinorVersion -ReleaseNamespace flux-system -IdentityType 'SystemAssigned'

Help docs link:
https://learn.microsoft.com/en-us/powershell/module/az.kubernetesconfiguration/new-azkubernetesextension?view=azps-11.5.0

[bavneetsingh16]

@lijinpei2008, the ‘IdentityType’ is specifically applicable for ‘connectedClusters’. However, in this context, we are dealing with an AKS cluster, also known as ‘managedClusters’. For this cluster type, it is not required to provide the ‘identity’ parameter. This is because AKS autonomously generates the identity for its extensions. The 'IdentityType' param should be mandatory only for connectedClusters but not in the case of managedClusters.

[lijinpei2008]

Thanks @bavneetsingh16 for the explanation, after several attempts I found what you described to be correct.
I will fix this issue in the near future.