bug: unable to deploy version 1.22 on AWS eks; errors thrown by pulumi

[qdzlug]

Describe the bug
When trying to run on the most recent version of EKS (1.22) the following errors are thrown:

16:02:26  
16:02:26  #=============================================================================#
16:02:26  #                _     __        __  ____      _____   _  __  ____            #
16:02:26  #               / \    \ \      / / / ___|    | ____| | |/ / / ___|           #
16:02:26  #              / _ \    \ \ /\ / /  \___ \    |  _|   | ' /  \___ \           #
16:02:26  #             / ___ \    \ V  V /    ___) |   | |___  | . \   ___) |          #
16:02:26  #            /_/   \_\    \_/\_/    |____/    |_____| |_|\_\ |____/           #
16:02:26  #                                                                             #
16:02:26  #=============================================================================#
16:02:26  
16:02:26  
16:02:27  Previewing update (marajenkaws3)
16:02:27  
16:02:27  View Live: https://app.pulumi.com/qdzlug/aws-eks/marajenkaws3/previews/9c2cbcf0-eac9-43af-b220-b98c9cb39f9e
16:02:27  
16:02:27  
16:02:28      pulumi:pulumi:Stack aws-eks-marajenkaws3  aws **** profile
16:02:28   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 create aws **** profile
16:02:29   +  aws:iam:Role ec2-nodegroup-iam-role create 
16:02:29   +  aws:iam:Role eks-iam-role create 
16:02:29   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 create read pulumi:pulumi:StackReference qdzlug/aws-vpc/marajenkaws3
16:02:29   +  aws:iam:RolePolicyAttachment eks-workernode-policy-attachment create 
16:02:29   +  aws:iam:RolePolicyAttachment ec2-container-ro-policy-attachment create 
16:02:29   +  aws:iam:RolePolicyAttachment eks-cni-policy-attachment create 
16:02:29   +  aws:iam:InstanceProfile node-group-profile-aws-eks-marajenkaws3 create 
16:02:29   +  aws:iam:RolePolicyAttachment eks-service-policy-attachment create 
16:02:29   +  aws:iam:RolePolicyAttachment eks-cluster-policy-attachment create 
16:02:29   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 create read pulumi:pulumi:StackReference qdzlug/aws-vpc/marajenkaws3
16:02:29   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 create vpc id: vpc-05315b2f4bfb9acce
16:02:31   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 create public subnets: ['subnet-085c683366982fc83', 'subnet-09e52459a6598217d', 'subnet-0f1483979c35c3fe1', 'subnet-0a7c801352bda906c']
16:02:31   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 create public subnets: ['subnet-027f5d25d0b5cdc03', 'subnet-038d95e76e9cdaa9b', 'subnet-0d056c799b48ee6c4', 'subnet-0e5d8659ed51f17ef']
16:02:33   +  eks:index:Cluster aws-eks-marajenkaws3 create 
16:02:35   +  eks:index:ServiceRole aws-eks-marajenkaws3-instanceRole create 
16:02:35   +  aws:ec2:SecurityGroup aws-eks-marajenkaws3-eksClusterSecurityGroup create 
16:02:35   +  eks:index:RandomSuffix aws-eks-marajenkaws3-cfnStackName create 
16:02:35   +  aws:iam:Role aws-eks-marajenkaws3-instanceRole-role create 
16:02:35   +  aws:eks:Cluster aws-eks-marajenkaws3-eksCluster create 
16:02:35   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksClusterInternetEgressRule create 
16:02:35   +  pulumi:providers:kubernetes aws-eks-marajenkaws3-eks-k8s create 
16:02:35   +  eks:index:VpcCni aws-eks-marajenkaws3-vpc-cni create 
16:02:35   +  aws:ec2:SecurityGroup aws-eks-marajenkaws3-nodeSecurityGroup create 
16:02:35   +  aws:iam:RolePolicyAttachment aws-eks-marajenkaws3-instanceRole-3eb088f2 create 
16:02:35   +  aws:iam:RolePolicyAttachment aws-eks-marajenkaws3-instanceRole-03516f97 create 
16:02:35   +  aws:iam:RolePolicyAttachment aws-eks-marajenkaws3-instanceRole-e1b295bd create 
16:02:35   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksExtApiServerClusterIngressRule create 
16:02:35   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksNodeIngressRule create 
16:02:35   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksNodeInternetEgressRule create 
16:02:35   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksClusterIngressRule create 
16:02:35   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksNodeClusterIngressRule create 
16:02:35   +  kubernetes:core/v1:ConfigMap aws-eks-marajenkaws3-nodeAccess create 
16:02:35   +  aws:iam:InstanceProfile aws-eks-marajenkaws3-instanceProfile create 
16:02:35   +  aws:ec2:LaunchConfiguration aws-eks-marajenkaws3-nodeLaunchConfiguration create 
16:02:35   +  aws:cloudformation:Stack aws-eks-marajenkaws3-nodes create 
16:02:35   +  pulumi:providers:kubernetes aws-eks-marajenkaws3-provider create 
16:02:36   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 create 4 messages
16:02:36   
16:02:36  Diagnostics:
16:02:36    pulumi:pulumi:Stack (aws-eks-marajenkaws3):
16:02:36      aws **** profile
16:02:36      vpc id: vpc-05315b2f4bfb9acce
16:02:36      public subnets: ['subnet-085c683366982fc83', 'subnet-09e52459a6598217d', 'subnet-0f1483979c35c3fe1', 'subnet-0a7c801352bda906c']
16:02:36      public subnets: ['subnet-027f5d25d0b5cdc03', 'subnet-038d95e76e9cdaa9b', 'subnet-0d056c799b48ee6c4', 'subnet-0e5d8659ed51f17ef']
16:02:36   
16:02:36  
16:02:36  Updating (marajenkaws3)
16:02:36  
16:02:36  View Live: https://app.pulumi.com/qdzlug/aws-eks/marajenkaws3/updates/1
16:02:36  
16:02:37  
16:02:37      pulumi:pulumi:Stack aws-eks-marajenkaws3  aws **** profile
16:02:38   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating aws **** profile
16:02:39   +  aws:iam:Role ec2-nodegroup-iam-role creating 
16:02:39   +  aws:iam:Role eks-iam-role creating 
16:02:39   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating read pulumi:pulumi:StackReference qdzlug/aws-vpc/marajenkaws3
16:02:39   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating read pulumi:pulumi:StackReference qdzlug/aws-vpc/marajenkaws3
16:02:40   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating vpc id: vpc-05315b2f4bfb9acce
16:02:41   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating public subnets: ['subnet-085c683366982fc83', 'subnet-09e52459a6598217d', 'subnet-0f1483979c35c3fe1', 'subnet-0a7c801352bda906c']
16:02:41   +  aws:iam:Role ec2-nodegroup-iam-role created 
16:02:41   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating public subnets: ['subnet-027f5d25d0b5cdc03', 'subnet-038d95e76e9cdaa9b', 'subnet-0d056c799b48ee6c4', 'subnet-0e5d8659ed51f17ef']
16:02:41   +  aws:iam:RolePolicyAttachment eks-workernode-policy-attachment creating 
16:02:41   +  aws:iam:RolePolicyAttachment ec2-container-ro-policy-attachment creating 
16:02:41   +  aws:iam:RolePolicyAttachment eks-cni-policy-attachment creating 
16:02:41   +  aws:iam:InstanceProfile node-group-profile-aws-eks-marajenkaws3 creating 
16:02:41   +  aws:iam:Role eks-iam-role created 
16:02:41   +  aws:iam:RolePolicyAttachment eks-service-policy-attachment creating 
16:02:41   +  aws:iam:RolePolicyAttachment eks-cluster-policy-attachment creating 
16:02:41   +  aws:iam:RolePolicyAttachment eks-workernode-policy-attachment created 
16:02:42   +  aws:iam:RolePolicyAttachment ec2-container-ro-policy-attachment created 
16:02:42   +  aws:iam:RolePolicyAttachment eks-cni-policy-attachment created 
16:02:42   +  aws:iam:RolePolicyAttachment eks-service-policy-attachment created 
16:02:42   +  aws:iam:RolePolicyAttachment eks-cluster-policy-attachment created 
16:02:42   +  aws:iam:InstanceProfile node-group-profile-aws-eks-marajenkaws3 created 
16:02:44   +  eks:index:Cluster aws-eks-marajenkaws3 creating 
16:02:45   +  eks:index:ServiceRole aws-eks-marajenkaws3-instanceRole creating 
16:02:46   +  aws:ec2:SecurityGroup aws-eks-marajenkaws3-eksClusterSecurityGroup creating 
16:02:46   +  aws:iam:Role aws-eks-marajenkaws3-instanceRole-role creating 
16:02:46   +  eks:index:RandomSuffix aws-eks-marajenkaws3-cfnStackName creating 
16:02:46   +  eks:index:RandomSuffix aws-eks-marajenkaws3-cfnStackName created 
16:02:48   +  aws:iam:Role aws-eks-marajenkaws3-instanceRole-role created 
16:02:48   +  aws:iam:RolePolicyAttachment aws-eks-marajenkaws3-instanceRole-3eb088f2 creating 
16:02:48   +  aws:iam:RolePolicyAttachment aws-eks-marajenkaws3-instanceRole-e1b295bd creating 
16:02:48   +  aws:iam:RolePolicyAttachment aws-eks-marajenkaws3-instanceRole-03516f97 creating 
16:02:48   +  aws:ec2:SecurityGroup aws-eks-marajenkaws3-eksClusterSecurityGroup created 
16:02:48   +  aws:eks:Cluster aws-eks-marajenkaws3-eksCluster creating 
16:02:48   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksClusterInternetEgressRule creating 
16:02:49   +  aws:iam:RolePolicyAttachment aws-eks-marajenkaws3-instanceRole-3eb088f2 created 
16:02:49   +  aws:iam:RolePolicyAttachment aws-eks-marajenkaws3-instanceRole-e1b295bd created 
16:02:49   +  aws:iam:RolePolicyAttachment aws-eks-marajenkaws3-instanceRole-03516f97 created 
16:02:49   +  aws:iam:InstanceProfile aws-eks-marajenkaws3-instanceProfile creating 
16:02:49   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksClusterInternetEgressRule created 
16:02:50   +  aws:iam:InstanceProfile aws-eks-marajenkaws3-instanceProfile created 
16:12:18  @ Updating...............................
16:12:18   +  aws:eks:Cluster aws-eks-marajenkaws3-eksCluster created 
16:12:18   +  aws:ec2:SecurityGroup aws-eks-marajenkaws3-nodeSecurityGroup creating 
16:12:18   +  aws:eks:Cluster aws-eks-marajenkaws3-eksCluster created Cluster is ready
16:12:18   +  eks:index:VpcCni aws-eks-marajenkaws3-vpc-cni creating 
16:12:18   +  pulumi:providers:kubernetes aws-eks-marajenkaws3-eks-k8s creating 
16:12:18   +  pulumi:providers:kubernetes aws-eks-marajenkaws3-eks-k8s created 
16:12:18  @ Updating....
16:12:18   +  kubernetes:core/v1:ConfigMap aws-eks-marajenkaws3-nodeAccess creating 
16:12:18   +  kubernetes:core/v1:ConfigMap aws-eks-marajenkaws3-nodeAccess creating 
16:12:18   +  kubernetes:core/v1:ConfigMap aws-eks-marajenkaws3-nodeAccess created 
16:12:18   +  aws:ec2:SecurityGroup aws-eks-marajenkaws3-nodeSecurityGroup created 
16:12:18   +  eks:index:VpcCni aws-eks-marajenkaws3-vpc-cni creating error: Command failed: kubectl apply -f /tmp/tmp-23547dNCAQqk1wCDq.tmp
16:12:18   +  eks:index:VpcCni aws-eks-marajenkaws3-vpc-cni **creating failed** error: Command failed: kubectl apply -f /tmp/tmp-23547dNCAQqk1wCDq.tmp
16:12:18   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key: beta.kubernetes.io/os is deprecated since v1.14; use "kubernetes.io/os" instead
16:12:18   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[1].key: beta.kubernetes.io/arch is deprecated since v1.14; use "kubernetes.io/arch" instead
16:12:18   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating error: unable to recognize "/tmp/tmp-23547dNCAQqk1wCDq.tmp": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
16:12:18   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksExtApiServerClusterIngressRule creating 
16:12:18   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksNodeIngressRule creating 
16:12:18   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksNodeInternetEgressRule creating 
16:12:18   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksClusterIngressRule creating 
16:12:18   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksNodeClusterIngressRule creating 
16:12:18   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksExtApiServerClusterIngressRule created 
16:12:19   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksClusterIngressRule created 
16:12:19   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksNodeIngressRule created 
16:12:19   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksNodeInternetEgressRule created 
16:12:20   +  aws:ec2:SecurityGroupRule aws-eks-marajenkaws3-eksNodeClusterIngressRule created 
16:12:20   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating error: update failed
16:12:20   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 creating error: Resource monitor has terminated, shutting down
16:12:20   +  eks:index:Cluster aws-eks-marajenkaws3 created 
16:12:20   +  pulumi:pulumi:Stack aws-eks-marajenkaws3 **creating failed** 2 errors; 7 messages
16:12:20   
16:12:20  Diagnostics:
16:12:20    eks:index:VpcCni (aws-eks-marajenkaws3-vpc-cni):
16:12:20      error: Command failed: kubectl apply -f /tmp/tmp-23547dNCAQqk1wCDq.tmp
16:12:20      Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key: beta.kubernetes.io/os is deprecated since v1.14; use "kubernetes.io/os" instead
16:12:20      Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[1].key: beta.kubernetes.io/arch is deprecated since v1.14; use "kubernetes.io/arch" instead
16:12:20      error: unable to recognize "/tmp/tmp-23547dNCAQqk1wCDq.tmp": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
16:12:20   
16:12:20    pulumi:pulumi:Stack (aws-eks-marajenkaws3):
16:12:20      aws **** profile
16:12:20      vpc id: vpc-05315b2f4bfb9acce
16:12:20      public subnets: ['subnet-085c683366982fc83', 'subnet-09e52459a6598217d', 'subnet-0f1483979c35c3fe1', 'subnet-0a7c801352bda906c']
16:12:20      public subnets: ['subnet-027f5d25d0b5cdc03', 'subnet-038d95e76e9cdaa9b', 'subnet-0d056c799b48ee6c4', 'subnet-0e5d8659ed51f17ef']
16:12:20      error: update failed
16:12:20      error: Resource monitor has terminated, shutting down
16:12:20   
16:12:20      Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key: beta.kubernetes.io/os is deprecated since v1.14; use "kubernetes.io/os" instead
16:12:20      Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[1].key: beta.kubernetes.io/arch is deprecated since v1.14; use "kubernetes.io/arch" instead
16:12:20      error: unable to recognize "/tmp/tmp-23547dNCAQqk1wCDq.tmp": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
16:12:20   
16:12:20  Resources:
16:12:20      + 28 created
16:12:20  
16:12:20  Duration: 9m44s

The workaround is to downgrade to EKS 1.21

To Reproduce
Attempt to deploy version 1.22 of EKS.

Expected behavior
Should deploy and run normally

Your environment
Current python modules:

awscli~=1.22.100
grpcio==1.43.0
fart~=0.1.5
lolcat~=1.4
nodeenv~=1.6.0
passlib~=1.7.4
pulumi-aws>=4.37.5
pulumi-docker==3.1.0
pulumi-eks>=0.37.1
pulumi-kubernetes==3.18.2
pycryptodome~=3.14.0
PyYAML~=5.4.1
requests~=2.27.1
setuptools==62.1.0
setuptools-git-versioning==1.9.2
wheel==0.37.1
yamlreader==3.0.4
pulumi-digitalocean==4.12.0
pulumi-linode==3.7.1
linode-cli~=5.17.2
pulumi~=3.30.0

Additional context
None.

[qdzlug]

This seemed to be tied into the issue noted in #139 - however, even with the kubectl fix discussed in that issue this version still fails.

[agpenton]

Any update on this issue? I'm stuck in the version upgrade because of this error.

[qdzlug]

#167 will close this issue; fix is to go to 1.23.x (but not 1.24.x as that shows other errors #151 and related)