[Help Wanted] Add Test Suite Toolchain

[ntindle]

We need to add a test suite toolchain for code stability

• Tests run in Pipeline
• Integration for installing plugins
• Integration tests for each plugin

[BaseInfinity]

Hi there I may be interested in helping out but would like to get a little more details on expected outcomes =)

Like are we looking for a suite of Unit + Integration tests?

Does the test-suite have to test the plugins itself or does the suite just have to test that they are loaded correctly?

What about first party plugins? Are those expected to have more in depth tests as well?

Thanks!

[ntindle]

Right now, the test suite should run the existing tests so it can be put into automation.

Ideally each plugin would have their own unit tests and we’d have a comprehensive suite of integration tests but that is a longer term goal

[ntindle]

The expected outcomes for today are:
Create a workflow that clones the repo, installs dependencies, runs pytest, reports the results and uploads it to codecov.

Most of that can be referenced from the ci pipeline for the main repo

[BaseInfinity]

Okay great! I didn't realize there were tests in place already. I'm pretty comfortable with GH Actions and ive setup these types of workflows before at my day job so I'll take a peek at the ci pipeline for the main repo and see if this is something I can commit to =)

[BaseInfinity]

ive been super pre-occupied catching up with the tech and playing with all the plugins, I should have a chance to look over this on the weekend

[h4ck3rk3y]

Hey @ntindle I created #123 that adds the CI from the main repo here.

I can see the GH actions run fine on my fork h4ck3rk3y#3 but I think it hides them for me in the PR as I am not admin.

Would love to help with the integration testing piece! What are you thinking with that? What about spinning up AutoGPT, loading the plugin giving it some prompts and seeing if it responds as expected? Or is it enough to test that the latest image continues to load with the plugin?

[BaseInfinity]

The power of open source :)!

[BaseInfinity]

@h4ck3rk3y: Would love to help with the integration testing piece! What are you thinking with that? What about spinning up AutoGPT, loading the plugin giving it some prompts and seeing if it responds as expected? Or is it enough to test that the latest image continues to load with the plugin?

I was thinking similar lines. Spin up autogpt with plugins and an assert there are no errors on start

Then maybe start adding basic "smoke" tests that runs an example command from each plugin to make sure the plug-in isn't completely borked?

Over time I suspect the more intricate unit/integration test cases come from the authors of the plugins themselves?

[h4ck3rk3y]

Hey @BaseInfinity @ntindle to keep the conversation going -

Here's an example of a nightly run that checks whether the plugin loads as expected; it tests it for three different plugins (but all of them have the same zip)

https://app.circleci.com/pipelines/github/kurtosis-tech/autogpt-package/72/workflows/eac40ed0-17a5-4849-9bf4-1df6258f05b4

Here's the code for it https://github.com/kurtosis-tech/autogpt-package/blob/main/.circleci/config.yml#L41-L69

Would you be open to a similar GH action on this repo?

[ntindle]

Yes, in GHA and we need a time to talk about the install method. Really would love cross (all) platform support though. Is it on the roadmap?

[h4ck3rk3y]

You were very prompt to respond!

Hey @ntindle Kurtosis runs on Linux, MacOS and Windows(via WSL)! Does that work for you? Or do you want native Windows support? We have it on the roadmap but haven't prioritized it yet as we haven't seen much demand for a Windows native client.

Is Windows support a blocker for a GitHub Action that does the integration test?

Here are the installation instructions.