CAI does not perform pentest

[w4cky]

Description

I launched CAI against my own test website, which contains a POST form vulnerable to SQL injection (the form echoes back the SELECT statement for easier testing). I connected CAI to a local model deepseek-r1-distill-qwen-14b-abliterated-v2 via LM Studio.
However, CAI did not perform any SQL injection attempts or any meaningful attacks.

Expected Behavior

CAI should recognize the SQL injection opportunity and try at least some injection payloads, or provide relevant attack strategies.

Actual Behavior

CAI does not execute SQL injection payloads and produces no useful attack attempts, making it ineffective in this scenario.

Steps to Reproduce

Run CAI against a test website with a POST form vulnerable to SQL injection (echoing the query).
Connect CAI to LM Studio with the model deepseek-r1-distill-qwen-14b-abliterated-v2.
Observe CAI’s behavior.

Questions
Is this expected behavior when using local models, or a bug in CAI?
When I used the OpenAI API, the model refused to perform the pentest saying it was not ethical. How can CAI be used with OpenAI models in such cases?

Logs in file

cai.txt

[vmayoral]

Had a look at this @w4cky, log seems invalid and can't be reproduced with cai-* tooling.
What version of CAI are you using. Can you facilitate the correct .jsonl file and not a processed version?

On top of the above, the model you're using is (respectfully) not good for pentesting (deepseek-r1-distill-qwen-14b-abliterated-v2). I doubt it's invoking the right tools and reasoning appropriately. You should leverage good models. Refer to our paper for what's goodl

[SoyGema]

Hello @w4cky I might try to change the model in CAI via /model deepseek/deepseek-reasoner and / or make the test with Deepseek V3 as the paper suggests. Happy hacking!

[w4cky]

thx @vmayoral and @SoyGema for answers

@vmayoral logs below:
cai_c892109a-e846-4a92-a554-88831614fe77_20250904_201446_michal_darwin_24.6.0_178_215_207_96.jsonl.txt
`

[w4cky]

@SoyGema this model is ok mradermacher/DeepSeek-V2-Lite-Chat-Uncensored-Unbiased-Reasoner-GGUF ?
i use LM studio
https://huggingface.co/nicoboss/DeepSeek-V2-Lite-Chat-Uncensored-Unbiased-Reasoner

[SoyGema]

Hey under this context you might want to try huggingface_api_key and use
your model as model name ( sorry this is confusing, you are naming a model and then linking another one ) -nicoboss and mradermacher are different providers-

https://docs.litellm.ai/docs/providers/huggingface

FYI , and as Victor suggest, I would recommend section 3.2 from the paper to see which models have been tested with CAI.
I´m assuming that you heve your .env file config correctly and your agent configuration as well redteam_agent . Maybe starting with a empirically tested agent might help on get (at least at first ) things to work
thank you for your follow up . Hope we can help!

[SoyGema]

@w4cky Hey ! I had the opportunity to dig into the log.
It seems that is a missconfiguration issue . This appears Shodan API key (SHODAN_API_KEY) must be set in environment variables
Have you configured it in your .env file?

[SoyGema]

Also you are using polish and english . Depending on the model I would recommend to stick to one language for user prompt. Hope this helps!

[SoyGema]

Hey @w4cky how is it going ? Have you had the opportunity to look at this or try somethings from here?
Let me know if I can help you with anything!

[w4cky]

@SoyGema I downloaded Claude and used it, but it doesn't work. Why does he need a key to Shodan? I'll tell him to create a domain in LAN.

How to force CAI to work with OpenAI? Because now OpenAI is censoring and refuses to execute commands. How to get around this?

[vmayoral]

@SoyGema I downloaded Claude and used it, but it doesn't work. Why does he need a key to Shodan? I'll tell him to create a domain in LAN.

Not needed. Some agents may work with it if available, but doesn't stop the agent if not available, Shodan will simply not work and it'll find another way around.

How to force CAI to work with OpenAI? Because now OpenAI is censoring and refuses to execute commands. How to get around this?

That's not CAI's issue, that's the model you're trying to use. Models from OpenAI and Anthropic are tremendously censored for security purposes.

[w4cky]

That's not CAI's issue, that's the model you're trying to use. Models from OpenAI and Anthropic are tremendously censored for security purposes.

@vmayoral thx. Is there an online model that is uncensored and that I can use with cai? My m4 MacBook Pro with 48GB of RAM can't handle the local Claude.

[vmayoral]

@w4cky we're getting pretty decent results with alias0 but unfortunately it's not publicly available for individuals. Reach out to me though (research at aliasrobotics.com) and let's see if we can do for your use case.

[vmayoral]

Closing here in the meantime as the original issue was addressed (otherwise, re-open).