add networking test config files #536
Conversation
tests/tekton-resources/tasks/generators/clusterloader/load-networking.yaml
Outdated
Show resolved
Hide resolved
tests/tekton-resources/tasks/generators/clusterloader/load-networking.yaml
Outdated
Show resolved
Hide resolved
| action: start | ||
| {{if $ENABLE_NETWORK_POLICY_ENFORCEMENT_LATENCY_TEST}} | ||
| - module: | ||
| path: modules/network-policy/net-policy-enforcement-latency.yaml |
There was a problem hiding this comment.
where are these modules ?
There was a problem hiding this comment.
these NP modules are all from upstream: https://github.com/kubernetes/perf-tests/tree/master/clusterloader2/testing/load/modules/network-policy
There was a problem hiding this comment.
I see that you are copying this config in task here - https://github.com/awslabs/kubernetes-iteration-toolkit/pull/536/files#diff-fc65d141840f1a98569538f9a751871bc06280084dec75e8e1777f47f429814dR175 to cl2 load test dir.
Could you add the comment explicitly here saying you are copying this file to relative path under cl2 load test dir, because by default this won't work if your config file is sitting in some other directory.
There was a problem hiding this comment.
added a comment.
The test needs to access modules under the cl2 folder, so copying the config over cl2 is cleaner.
| name: test-svc-deployment | ||
| namespace: test-svc | ||
| spec: | ||
| replicas: 5000 |
There was a problem hiding this comment.
do you want to make this configurable ?
There was a problem hiding this comment.
We don't have to block on this for this PR if you want, but you may want to keep this configurable to test at different scales.
There was a problem hiding this comment.
Yeah I would prefer to take this as a TODO. I'm thinking of 2 options:
- Migrate to use daemonset from deployment, enabling us to test endpoints across all nodes (where number of endpoints = node count)
- Implement a configurable setup with m services, each containing k endpoints, where both parameters are configurable
I'd like to evaluate these 2 options further with an actual cluster that we will use for all the testing, and decide a better option (less time costly, but stress test kp)
| EOF | ||
| cat $(workspaces.source.path)/perf-tests/clusterloader2/pkg/prometheus/manifests/exporters/kube-state-metrics/deployment.yaml | ||
|
|
||
| # # TODO: Remove this once we fix https://github.com/kubernetes/kubernetes/issues/126578 or find a better way to work around it. |
There was a problem hiding this comment.
not sure i get this ?
There was a problem hiding this comment.
I'm not sure if we still need this, looks like the issue with endpoint controller has been fixed upstream. Removing coredns service monitor will cause Prometheus stop scrapping coredns metrics. so I commented out
tests/tekton-resources/tasks/generators/clusterloader/load-networking.yaml
Outdated
Show resolved
Hide resolved
tests/tekton-resources/tasks/generators/clusterloader/load-networking.yaml
Outdated
Show resolved
Hide resolved
|
|
||
| # create the service backed by 5k pods to test kubeproxy network programming performance | ||
| # we can tune the scale of pods later | ||
| kubectl apply -f $(workspaces.source.path)/perf-tests/clusterloader2/testing/load/test-svc.yaml |
There was a problem hiding this comment.
why are you creating the workload even before test is kicked off ?
There was a problem hiding this comment.
It's better to create the service with endpoints before the clusterloader binary runs since clusterloader only collects kp metrics.
The svc creation itself will trigger kp to sync the network programming rules, and generate the latency metrics (the metrics measures the time gap b/w endpoints creation timestamp and the time when kp done it's job, so it's better to create before cl collects the metrics).
3 participants
Issue #, if available:
Description of changes:
Onboard networking components scale test
Test:
Tested via internal pipeline for 1k node. link; test results
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.