Plugins

[mumoshu]

Would it be useful/possible to introduce "kube-aws plugins" to define sets of configurations and resources(settings, k8s manifests, required iam policies, etc) used to extend kube-aws, without complicating the core of kube-aws?

For example, I guess with the feature, #507 can be implemented as a plugin composed of:

Design 1

• plugins/enabled/
  • <unique name of an instance of plugin e.g. cluster-dump>/
    • controller
      • manifests/
        • cluster-dump.yaml.tmpl with the content of cluster-dump.yaml
      • iam-policies/
        • allow-puts-to-s3-path.json.tmpl with the content of the iam policy required by cluster-dump

Intentions:

• The directory is named plugins/enabled/<unique name of an instance of plugin> so that
  • we can later introduce plugins/available/<unique name of a plugin> for defining a plugin=a template of instances of plugin
• The controller directory is there to place controller node specific files and settings

Design 2

• plugins/available/
  • <unique name of a plugin e.g. cluster-dump>/
    • controller
      • manifests/
        • cluster-dump.yaml.tmpl
      • iam-policies/
        • allow-puts-to-s3-path.json.tmpl
• cluster.yaml
  • With the new configuration key to allow enabling and customizing plugins

    clusterName: ***
    externalDNSName: ***
    kmsKeyARN: ***
    *snip*
    plugins:
      clusterDump:
      - pluginName: <unique name of a plugin e.g. cluster-dump>
          s3Prefix: <customs3prefix_to_where_cluster_is_dumped>
      rescheduler:
      - pluginName: rescheduler

and we bundle default plugins into kube-aws binaries, but putting files to plugins/available would allow customizing a default plugin.

---

@c-knowles I can't locate the exact link to it but if I remember correctly, we've discussed before about introducing "plugins" to make kube-aws extendable by users?

[jeremyd]

I think that yes, many plugins of this style are simply needing to add to the cloud-config units or files section, with this PR this can be done in a generic way. #510 ;

There is the need for additional IAM configuration in the PR for S3 dumps. Perhaps IAM is the other configuration option that could be implemented generically.

[cknowles]

@mumoshu I think plugins or addons have been discussed in a few places. The ones I can think of:

• Addons discussion for rescheduler
• Hooks for snippets of CF
• Custom IAM polices
• Restricted SSH access

Many kubernetes addons or even some application kube-aws users deploy require them to edit various aspects of the cluster. It could be edited IAM, adding RBAC policies, adding CF resources or various other things. Ultimately as we know, we need to decide how many of these we wish to actively support as top tier features. There's ways to do all of the above with @redbaron's git workflow without changing kube-aws at all. So one option is to only support that method.

I've always thought plugins would be good so we get a bit of both worlds - 1) we stop having to encapsulate each and every feature/addon into kube-aws and 2) we still allow users to more easily connect their additions in, not just well know addons like we support but anything they wish.

I would go as far as saying some things currently implemented in kube-aws as top tier features should actually be removed and just use the plugin system. For example, the rescheduler and kube2iam probably belong there.

If we go the plugin way, I think we should provide a contrib folder with plugins that are at least validated at a basic level.

As for the designs, I'm not sure I fully understand the intentions of 1 but I think I prefer 2.

• Do we wish to add as well as manifests and IAM policies? Good to start simple though.
• I was also thinking that even "core" aspects should be plugins that are enabled by default, maybe even impossible to switch off. We could try at first to separate out something like RBAC or the autoscaler and have them use the exact same plugin mechanism. I've used a few systems which are entirely pluggable like Serverless or Metalsmith and a lot of the power comes from the built in features using the same plugin mechanism.

[mumoshu]

@jeremyd @c-knowles Thanks for feedbacks 👍

Regarding the IAM configuration part of the imaginary plugin system, would you like the iam-policies directory in the above examples?

[cknowles]

Seems a reasonable start to me but also happy if we start with a single use case.

[mumoshu]

I'm extending the design 2 of the plugin system like shown below - @c-knowles @jeremyd Would you like it? Which part do you need for the first release of the "plugin system" feature?

• <kube-aws root>/builtin/plugins/

  • <unique name of a plugin e.g. cluster-dump>/

    • plugin.yaml

      # plugin.yaml defines "parameters" correspond to `arguments` passed via top-level/controller's/worker's `pluginConfgurations.<pluginName>` in cluster.yaml
      controller:
        parameters:
          myVar: 
            type: string
      worker:
        parameters:
          myVar:
            type: string          

    • worker/ worker specific customizations

      • The same as controller/ but its manifests/ directory

    • controller/ controller specific customizations

      • node/ controller node specific customizations
        • manifests/ k8s manifests persisted under /srv/kubernetes/manifests to be deployed to k8s via kubectl apply -f
          • cluster-dump.yaml.tmpl
        • static-pods/ k8s static pods persistend under /etc/kubernetes/manifests to be deployed via kubelet --pod-manifest-path
          • your-static-pod-spec.yaml.tmpl
      • cfn-resources/
        • your-single-required-cfn-resource.json.tmpl
        • iam-policies/ iam policies added to the managed IAMRoleController
          • allow-puts-to-s3-path.json.tmpl

• your work tree/

  • adhoc-plugins/
    • <unique name of your adhoc plugin e.g. myCustomizations>/
  • cluster.yaml

    • With the new configuration key to allow enabling and customizing plugins

      clusterName: ***
      externalDNSName: ***
      kmsKeyARN: ***
      *snip*
      
      # All the plugins needs to be enabled under the `enablePlugins` key in a cluster.yaml
      enabledPlugins:
      #- <unique name of a plugin e.g. cluster-dump in lowerCamelCase>:
      #  myKeyDefinedInThePluginYaml: foo
      - clusterDump:
        s3Prefix: <customs3prefix_to_where_cluster_is_dumped>
      - rescheduler:
        enabled: true
      
      # If you'd like to customize settings not for the whole cluster but just for controller nodes...
      controller:
        pluginConfigurations:
          clusterDump: #Only a plugin enabled under enablePlugins can be configured
            dumpInterval: 1h
            keptRevisions: 3
      
      # If you'd like to customize settings passed to a plugin per node pool...
      worker:
        nodePools:
        - name: pool1
          pluginConfigurations:
            nameOfPlugin:
              myVar: foo
        - name: pool2
          pluginConfigurations:
            nameOfPlugin:
              myVar: bar

[mumoshu]

Thanks @c-knowles,

Seems a reasonable start to me but also happy if we start with a single use case.

Anything specific which seems missing/redundant for now?
For example, the original Design 2 seemed necessary and sufficient for rescheduler for me. Your thoughts?

[cknowles]

The rescheduler is a very simple case in that it just needs a way to deploy a manifest file to /srv/kubernetes/manifests/ of the controllers. Above you asked about iam-policies which I think we'll need for another use cases but not if we first target rescheduler. Are we also targeting the export tool or something else with first attempt?

On the structure, could the built in worker plugin structure mirror the controller one? I think it would be better if worker/ and controller/node/manifests/ is the same. Also, will we have other builtin files? The /builtin/plugins/ seems to imply we plan on adding something else? (it's different to adhoc-plugins style)

[mumoshu]

Ah, your are correct, iam-policies is too much for the rescheduler plugin.
I agree that we should start with a single use-case.
I just want to design the structures of plugins before we start so that we don't need to change the plugin interface a lot later.

On the structure, could the built in worker plugin structure mirror the controller one?

Yes

I think it would be better if worker/ and controller/node/manifests/ is the same.

As manifests are deployed cluster-wide by kubectl apply -f on controller nodes as of today, I assumed it would not make sense to place it under worker/ hence to run the command in worker nodes.

Also, will we have other builtin files? The /builtin/plugins/ seems to imply we plan on adding something else?

Nothing in particular but just wanted not to name it like builtin-plugins(not a valid golang package name AFAIK) or builtinplugin(I want to avoid this one as long as plugin is unique enough as the golang package name)

(it's different to adhoc-plugins style)

I assumed that we just have a different naming convention and structure between a work tree(managed by users) and the kube-aws project's file tree(managed by developers).
For example, there has been no hyphens in directory names inside the project tree.
I also assumed that if we make some plugins "builtin", it will be bundled into kube-aws binaries so making directory names valid go package names would make a life a bit easier 😄

[cknowles]

I see ok, I hadn't realised you intended for the built in files to be a go package I thought it was just some template files we'd be bundling. If it makes live easier to use a valid go package name then I'm all for it.

[cknowles]

@mumoshu @camilb I was thinking about the Dex support from #568 and various other places we've added support for addons like kube2iam. I was wondering if some items like Dex can almost be covered by Helm and it seems like that's mostly true.

So, my idea is to allow the plugin system to bootstrap Helm charts generically then all we would need to enable for Dex judging from the changelog is some hooks to take the few --oidc flags in the controller setup. That could also be coded generically, to allow Helm Chart values to be passed into cluster bootstrap. i.e. just like we are allowing a folder with plugin values to be specified we also also Helm values.yml files to be specified for each chart that kube-aws would preinstall.

I'm not saying we do it in the Dex PR as I'm really looking forward to that feature! However, I'd like to get your thoughts on the idea? Using Helm as the installation mechanism has a few advantages including:

1. Ability to wait for a deploy to start up fully is built into Helm including PV mounts
2. As each addons gets more and more complex, we don't need to add the complexity directly to kube-aws
3. Opens us up to the Helm and Helm Charts community so more likely that each Chart gets the attention it deserves, (see Update kubelet deprecated flags #543 and Update kube-dns to latest (v1.14.1) #542 for the sorts of things I mean)
4. Should mostly have the ability to upgrade the Chart in place although for Dex that assumes the --oidc flags don't change

[redbaron]

So, my idea is to allow the plugin system to bootstrap Helm charts generically

+1

[mumoshu]

Opens us up to the Helm and Helm Charts community so more likely that each Chart gets the attention it deserves

+1

Btw, so - can we deploy tiller by default in all the clusters created by kube-aws? (which I personally like to 😄

[cknowles]

I would say yes for the default to install Tiller. We could make it optional and error out if any Charts have been added but Tiller is off. Or we could install it if and only if any Charts are being bootstrapped.

[camilb]

@c-knowles My first idea was only to enable the flags in the API server and deploy dex using helm or kubectl. I agree with you to use helmand reduce the complexity in kube-aws.

[cknowles]

Great, I'm going to have a think about exact design here and propose something.