Automatically generate a dynamic client or a fargateprofiles client with a runtime scheme

[jicowan]

Is your feature request related to a problem?
I want to be able to create a CRD object programmatically.

Describe the solution you'd like
I had to create a dynamic client in Go to create an instance of the CRD. I could have used runtime scheme too. I'd like for these clients to be included in the package so I don't have to create them.

Example 1:

client, err := dynamic.NewForConfig(config)
j, err := json.Marshal(fargateInstance)
if err != nil {
    panic(err)
}
u := &unstructured.Unstructured{}
u.UnmarshalJSON(j)

// create GVR. This should exist somewhere already
gvr := v1alpha1.GroupVersion.WithResource("fargateprofiles")

n, err := client.Resource(gvr).Namespace("default").Create(context.TODO(), u, metav1.CreateOptions{})
if err != nil {
    panic(err)
}
fmt.Println(n)

Example 2:

v1alpha1.AddToScheme(scheme.Scheme)

config.ContentConfig.GroupVersion = &v1alpha1.GroupVersion
config.APIPath = "/apis"
config.NegotiatedSerializer = serializer.NewCodecFactory(scheme.Scheme)
fargateClient, err := rest.UnversionedRESTClientFor(config)
if err != nil {
    panic(err)
}
...
result := v1alpha1.FargateProfile{}
body, err := json.Marshal(fargateInstance)
res := fargateClient.Post().Resource("fargateprofiles").Namespace("default").Body(body).Do(context.TODO())
fmt.Println(res.Error())
fmt.Println(result)

Example 3:

obj, err := runtime.DefaultUnstructuredConverter.ToUnstructured(fargateInstance)
if err != nil {
    panic(err)
}
u := &unstructured.Unstructured{Object: obj}

// create GVR. This should exist somewhere already
gvr := v1alpha1.GroupVersion.WithResource("fargateprofiles")
result := &v1alpha1.FargateProfile{}
n, err := client.Resource(gvr).Namespace("default").Create(context.TODO(), u, metav1.CreateOptions{})
if err != nil {
    panic(err)
}
runtime.DefaultUnstructuredConverter.FromUnstructured(n.Object, result)
fmt.Println(result)

Describe alternatives you've considered
Using a different language like Python, https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/CustomObjectsApi.md

[vijtrip2]

I had to create a dynamic client in Go to create an instance of the CRD

If I understand correctly, you created a fargateprofile resource dynamically and wondering why ACK does not provide this CRD?

I think the short answer is that ACK does not support Fargate controller yet. Here is the list of services currently supported/proposed for ACK. https://aws-controllers-k8s.github.io/community/docs/community/services/

Please let me know if I did not understand your concern correctly.

---

I want to be able to create a CRD object programmatically.

ACK controllers are not the correct place for this. ACK controllers comes with a defined set of CRDs and controller that manages those CRDs. Can you please help me understand how this problem relates to ACK controllers.

[jicowan]

@vijtrip2 I want to use the CRDs that ACK creates to programmatically create instances of those CRDs. In this particular instance, I am writing an admission controller to automatically create a Fargate profile when a namespace with a specific annotation is created. I want to use the FargateProfile from the eks-controller to create the profile. I am asking that ACK bundle a client with the controller that makes this easier. Here's the complete code that I'm currently using:

package main

import (
	"context"
	"github.com/aws-controllers-k8s/eks-controller/apis/v1alpha1"
	"github.com/aws/aws-sdk-go/aws"
	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
	"k8s.io/apimachinery/pkg/runtime"

	//"encoding/json"
	"flag"
	"fmt"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	//"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
	"k8s.io/apimachinery/pkg/runtime/schema"
	"k8s.io/client-go/dynamic"
	"k8s.io/client-go/tools/clientcmd"
	"k8s.io/client-go/util/homedir"
	"path/filepath"
)

func main() {
	var kubeconfig *string
	if home := homedir.HomeDir(); home != "" {
		kubeconfig = flag.String("kubeconfig", filepath.Join(home, ".kube", "config"), "(optional) absolute path to the kubeconfig file")
	} else {
		kubeconfig = flag.String("kubeconfig", "", "absolute path to the kubeconfig file")
	}
	flag.Parse()

	// use the current context in kubeconfig
	config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)
	if err != nil {
		panic(err.Error())
	}

	profileSelector := &v1alpha1.FargateProfileSelector{
		Namespace: aws.String("bar"),
	}
	fgSpec := v1alpha1.FargateProfileSpec{
		ClusterName:         aws.String("bottlerocket"),
		Name:                aws.String("fargate-bar"),
		PodExecutionRoleARN: aws.String("arn:aws:iam::820537372947:role/AmazonEKSFargatePodExecutionRole"),
		Selectors: []*v1alpha1.FargateProfileSelector{profileSelector},
		Subnets: aws.StringSlice([]string{"subnet-0b3633ef148863c83"}),
	}

	fargateInstance := &v1alpha1.FargateProfile{
		TypeMeta:   metav1.TypeMeta{
			Kind: "FargateProfile",
			APIVersion: "eks.services.k8s.aws/v1alpha1",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name: "fargate-bar",
		},
		Spec:       fgSpec,
		Status:     v1alpha1.FargateProfileStatus{},
	}

	client, err := dynamic.NewForConfig(config)
	output, err := client.Resource(schema.GroupVersionResource{
		Group:    "eks.services.k8s.aws",
		Version:  "v1alpha1",
		Resource: "fargateprofiles",
	}).Namespace("default").Get(context.TODO(),"foo", metav1.GetOptions{})
	if err != nil {
		fmt.Println(err)
	}
	fmt.Println(output)

	obj, err := runtime.DefaultUnstructuredConverter.ToUnstructured(fargateInstance)
	if err != nil {
		panic(err)
	}
	u := &unstructured.Unstructured{Object: obj}

	// create GVR. This should exist somewhere already
	gvr := v1alpha1.GroupVersion.WithResource("fargateprofiles")
	result := &v1alpha1.FargateProfile{}
	n, err := client.Resource(gvr).Namespace("default").Create(context.TODO(), u, metav1.CreateOptions{})
	if err != nil {
		panic(err)
	}
	runtime.DefaultUnstructuredConverter.FromUnstructured(n.Object, result)
	fmt.Println(result)
}

[RedbackThomson]

I've tried to do lots of digging through documentation to find best practices for creating custom resources. Some documentation still references typed client sets, which I believe is an outdated idea (lmk if I'm wrong?). However, for the dynamic client sets, the use of the UnstructuredConverter seems a little scary to me. Do you have a preferred method - perhaps a link to some existing code sample that could serve as the prime example?

[jicowan]

See https://stackoverflow.com/questions/53341727/how-to-submit-generic-runtime-object-to-kubernetes-api-using-client-go/53359468#53359468. What don't you like about the UnstructedConverter?

[RedbackThomson]

Okay. I've just done a bit of messing with it myself to see if I could find a cleaner way, but I think your 3 examples are a comprehensive set of methods. It appears the k8s.io/client-go/dynamic is the current best-practice way to support programmatic CRUD of custom resources. This article gives a really good overview of how to best use the library.

To support this, the only blocker I see on our end is a missing representation of the GroupVersionResource? Anything else you'd like to call out? Happy to include this as a simple change to all resources.

[jicowan]

Yes, I think adding GroupVersionRresouce will simplify things @RedbackThomson. Thanks for looking into this.

[RedbackThomson]

@jicowan Take a look at aws-controllers-k8s/code-generator#212 . Let me know if that works for you.

[eddiezane]

Some documentation still references typed client sets, which I believe is an outdated idea (lmk if I'm wrong?).

My expectation when working with a CRD backed by a Go type/controller is that there is a generated client (and usually informers + listers). code-generator/client-gen/kubebuilder would normally do this for free but it looks like we're using a different approach for code gen. I haven't dug in but It should still be possible to generate once the type exists.

[jaypipes]

Some documentation still references typed client sets, which I believe is an outdated idea (lmk if I'm wrong?).

My expectation when working with a CRD backed by a Go type/controller is that there is a generated client (and usually informers + listers). code-generator/client-gen/kubebuilder would normally do this for free but it looks like we're using a different approach for code gen. I haven't dug in but It should still be possible to generate once the type exists.

The typed clientsets are merely a porcelain API versus the plumbing API that the dynamic client represents. In other words, they are just syntactic sugar.

I didn't generate typed clientsets for our ACK controller CRDs originally because I wanted to keep things simple (since we were generating a ton of controller implementations). I still don't think it's necessary to generate typed clientsets for all our CRDs but I suppose I could revisit the idea if this is something that lots of users would need.