Skip to content

Agent2pwnbench: Comparing Models and CAI Agents in Lab Challenges of PortSwigger Web Academy. #275

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1,007 commits into from
Closed

Agent2pwnbench: Comparing Models and CAI Agents in Lab Challenges of PortSwigger Web Academy. #275

wants to merge 1,007 commits into from

Conversation

cristobalvch
Copy link
Contributor

modifications that I have done:

  • Add env variables.
  • Add json format (same as CAIbench with name, difficulty, techniques, instructions, challenges, source).
  • Remove hardcoded prompts + add our agents prompts.
  • Update readme of benchmarks

luijait and others added 30 commits May 22, 2025 08:58
@luijait
Robust pricing
cb5057e
@luijait23
Merge branch 'stream_pricing' into '0.4.0'
717b9be 
Robust pricing

See merge request aliasrobotics/alias_research/cai!190
@luijait
Pricing streaming
5728087
@luijait
local model fix
4541e3f
@vmayoral @luijait
Fixup replay so that it doesn't crash if no content
4741669 
Signed-off-by: Víctor Mayoral Vilches <[email protected]>
@luijait
Support pricing in alias0
0287ba0
@luijait
Robust pricing
cc754d4
@luijait
add info
ae802b1
@luijait
fix
b167a29
@luijait
cker virtualization for ctf_inside false
5075bb7
@luijait
remove devcontainer
26bb3ed
@luijait
Add qdrant
9ae8171
@luijait
Add qdrant
1f93493
@luijait
Improve prompting
f6eda51
@luijait23
Merge branch 'pentestperf' into '0.4.0'
7ded87a 
Integrate Pentestperf in v0.4.0

See merge request aliasrobotics/alias_research/cai!179
@luijait
Async sessions output fixed
6967077
@luijait23
Merge branch 'sessions_async' into '0.4.0'
37beb84 
Async sessions output fixed

See merge request aliasrobotics/alias_research/cai!192
@luijait
Add support for claude 4
0c89793
@luijait
Add support for claude 4
9220937
@luijait23
Merge branch 'sessions_async' into '0.4.0'
08dbc79 
Add support for claude 4

See merge request aliasrobotics/alias_research/cai!193
@luijait
Cache ephemeral for claude
a096a77
@luijait
Add caches
54e9d15
@vmayoral
Merge branch 'claud4' into '0.4.0'
8287bfe 
Claud4

See merge request aliasrobotics/alias_research/cai!194
@luijait
typo
384e844
@luijait
Fix requirement
e9f7b50
@luijait23
Merge branch '0.4.0-valid' into '0.4.0'
29b1fc8 
0.4.0 valid

See merge request aliasrobotics/alias_research/cai!195
@luijait
Improve claude thinking
834a42b
@vmayoral
Add missing alias model cost aspects
60b1902 
Signed-off-by: Víctor Mayoral Vilches <[email protected]>
@luijait
Pricing via JSON
1ab700c
@luijait
Fix stream plus reason
1212c01
cris and others added 20 commits September 3, 2025 12:10
add updated readme for prompt-bench project
1fc7c44
add function to create log folder
c9c2c88
update readme project structure
e139441
remove readme on utils folder
5a32b8d
fix spelling mistak in readme
8c1d03d
change nam of the project
3b70098
change name of project in readme folder structure
49eb4c1
@vmayoral
Add sola and minor correction
577f9dc 
Signed-off-by: Víctor Mayoral Vilches <[email protected]>
@vmayoral
Address aliasrobotics#262
5376aaf 
Signed-off-by: Víctor Mayoral Vilches <[email protected]>
@joleeee
Fix link in README.md for tools section (aliasrobotics#272)
5878c65
@aFrogThatExists
Updated README.md to add note on installing python3.12-venv (aliasro…
3d8f347 
…botics#271)

* Update README.md to add note on installing python3.12-venv

* Updated README.md to move the note to FAQ
@top9th
Fix Google Gemini OpenAI compatibility issues (aliasrobotics#268)
a065d4f 
Gemini is ready to use with this configuration.
```
OPENAI_API_KEY="gemini_key"
OPENAI_BASE_URL="gemini_openai_compatible_url"
CAI_MODEL=openai/gemini-2.5-pro
```
@IamYipi
docs: Azure OpenAI update; create 'Providers' section in mkdocs.yml a…
cb9e506 
…nd specify configuration for CAI to work correctly (aliasrobotics#264) (aliasrobotics#273)

docs: Azure OpenAI update; create 'Providers' section in mkdocs.yml and specify configuration for CAI to work correctly (aliasrobotics#264)
@cristobalvch
Merge branch 'aliasrobotics:main' into main
e466d64
add new benchmark Agent2PwnBench
f2058d9
@vmayoral
Add irregular
f2e39df 
Signed-off-by: Víctor Mayoral Vilches <[email protected]>
@vmayoral
Include new humanoids papers
39fcea6 
Signed-off-by: Víctor Mayoral Vilches <[email protected]>
@vmayoral
Update README again
71ef1e1 
Signed-off-by: Víctor Mayoral Vilches <[email protected]>
@luijait
Add support for Chrome MCP (aliasrobotics#279)
efd12d3
@rohitaryal
Fix typo (aliasrobotics#283)
9a5a067
@Mery-Sanz Mery-Sanz self-requested a review September 29, 2025 06:11
Mery-Sanz
Mery-Sanz approved these changes Sep 30, 2025
View reviewed changes
vmayoral pushed a commit that referenced this pull request Oct 14, 2025
@rm-openai
fixed from agents.voice import SingleAgentVoiceWorkflow, VoicePipelin… (
b52fea0 
#275)

…e in  quickstart.md

from agents.voice import SingleAgentVoiceWorkflow, VoicePipeline,

remove extra "," from the first line
@vmayoral vmayoral closed this Oct 14, 2025
vmayoral pushed a commit that referenced this pull request Oct 14, 2025
@rm-openai
fixed from agents.voice import SingleAgentVoiceWorkflow, VoicePipelin… (
59e6834 
#275)

…e in  quickstart.md

from agents.voice import SingleAgentVoiceWorkflow, VoicePipeline,

remove extra "," from the first line
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Mery-Sanz Mery-Sanz Mery-Sanz approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.