Skip to content

Add CI specific evals #250

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Dec 1, 2024
Merged

Add CI specific evals #250

merged 6 commits into from
Dec 1, 2024

Conversation

seanmcguire12
Copy link
Member

why

We need to be able to rely on a subset of evals that should be at or around 100% before merging.

what changed

  • added a list of CI specific evals in ci.yml
  • added a check to make sure the overall exactMatch result of these evals is above 90%

notes

  • this list will grow with time, this is just a starting point of evals that are presently known to be reliably passing

Copy link

changeset-bot bot commented Nov 30, 2024

🦋 Changeset detected

Latest commit: 9c053cf

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@browserbasehq/stagehand Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@seanmcguire12 seanmcguire12 added this to the Evaluation milestone Nov 30, 2024
@seanmcguire12 seanmcguire12 merged commit 5886620 into main Dec 1, 2024
1 check passed
@github-actions github-actions bot mentioned this pull request Dec 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants