Caching automl instances between `pytest` invocations as opt-in instead of `opt-out`

[eddiebergman]

With PR #1417, automl instances are pickled on the first time they are requested and the saved between runs. This was to allow for rapid local testing, using real trained automl instances and not have to retrain for every test. However, in the unpickling process, if changes are done to the AutoML class for example, where an attribute changes, the unpickled instance will not reflect these changes.

The caching process use to be default with an "opt-out" feature

# Will train the automl instances the first time they are requested
# then each use within this invocation will be cached
pytest

# Will use the same cached instances, meaning no training
pytest

# Will delete the cache and start as it did in the first invocation
pytest --cache-clear

However, due to this being a potential confusing issue, we should make it an "opt-in" feature,
perhaps with a flag like --with-cache

# Will train the automl instances the first time they are requested and cache them for all tests
# in this invocation
pytest

# Will train the automl instances the first time they are requested and cache them
# ... deleting the old ones
# ... and then use the new ones for all tests in this invocation
pytest

# Will use the cached items from last run
pytest --with-cache

To prevent having to pass the argument everytime, for example when testing feature that can use such cached items safely, there are two options (or more) to consider:

• Use pyproject.toml to just include it when running pytest
  • Has a danger of being committed in, but only if someone is aware of the possibility and it is not picked up in Review
• Have a marker file like .with-cache which can be safely put in .gitignore.
  • Has a danger of not being noticed and having cached item issues