-
Notifications
You must be signed in to change notification settings - Fork 40
Description
Recently, I've had a lot of issues with the run_tests
tool in Gemini CLI where the LLM interprets the "isError" field to mean that the test tool failed to run, rather than that tests have failed. It basically keeps trying to re-run "run_tests" because it thinks there is an error in parsing the test output. It makes the run_tests
tool useless.
I think that a non-zero exit code for things like dart test
, dart format
, etc. shouldn't be interpreted as a "failure" of the tool. The LLM doesn't get to see the "failureReason", so it can't tell the difference between missing roots or some other failure and a failed test.
I think that the isError
should only be set if the tool actually failed to run.
The output of "run_tests" would normally be expected to include test failures. We use the error exit code to inform CI and Unix tools that there were test failures, but even then they can't tell the difference between a path error and a test failure.
It's unlikely that the LLM will interpret a tool that doesn't have isError
set as having all passing tests in the presence of test output showing failures.
As a data point, the analyze_files
tool doesn't set isError
when there are analyzer errors, and the LLM can interpret that perfectly well.