bug: Issues with versions and AWS deployments

[qdzlug]

Describe the bug
Recently, AWS started to blow up with errors of this nature:

[2022-05-04T17:23:43.848Z] Diagnostics:

[2022-05-04T17:23:43.848Z]   pulumi:pulumi:Stack (aws-eks-marajenkaws20):

[2022-05-04T17:23:43.848Z]     aws **** profile

[2022-05-04T17:23:43.848Z]     vpc id: vpc-0a20475856e1a54b1

[2022-05-04T17:23:43.848Z]     public subnets: ['subnet-030061ff2e5160de0', 'subnet-031ca75432c49deb8', 'subnet-00f157263bb9722b8', 'subnet-0cefe955aafdc0dda']

[2022-05-04T17:23:43.848Z]     public subnets: ['subnet-0e40d375a97c077e7', 'subnet-035238ff2d6ff9b20', 'subnet-01d50932155b662b6', 'subnet-03942ed91feac5b17']

[2022-05-04T17:23:43.848Z]     error: update failed

[2022-05-04T17:23:43.848Z]     error: Resource monitor has terminated, shutting down

[2022-05-04T17:23:43.848Z]  

[2022-05-04T17:23:43.848Z]     error: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"

[2022-05-04T17:23:43.848Z]  

[2022-05-04T17:23:43.848Z]   eks:index:VpcCni (aws-eks-marajenkaws20-vpc-cni):

[2022-05-04T17:23:43.848Z]     error: Command failed: kubectl apply -f /tmp/tmp-24836aEhoNz3T9Dk0.tmp

When checking into this, the problem seemed to be around the versioning of the API, as per this StackOverflow post: https://stackoverflow.com/questions/71318743/kubectl-versions-error-exec-plugin-is-configured-to-use-api-version-client-auth

This was happening at the same time as #134 so the initial check was to determine what versions of the modules were used with the last known good run of the AWS code. The first fix tried was lock down the versions used to those that were from the last known working run. This still failed.

This lead to some investigation into how Pulumi interact with kubectl, specifically this link:

With the kubeconfig available, Pulumi communicates with the API Server using the official Kubernetes client-go library, just like kubectl does.

This led us down the path to determine that the startup script pulls the current stable release of kubectl, which is then used by Pulumi. This only happens if no kubectl has not already been downloaded; if it has been downloaded it is left as-is.

To Reproduce
Steps to reproduce the behavior:

1. Deploy to AWS (this only happens with AWS)
2. Process will fail with EKS build.

Expected behavior
Should work.

Your environment

• n/a

Additional context
None.

[qdzlug]

Version 1.24 seems to be the breaking change for EKS 1.21; regressing to 1.23.6 seems to fix the problem. Testing is ongoing.

[qdzlug]

The 1.24 issue is the same that we are seeing in #151 and friends