How to use Seleniumbase as a replacement for UC ?

[MacMarde]

It seems from what I can read in the Issues that UC it is not bypassing CF anymore for weeks.
I also read that Seleniumbase might still work (for some reason) and is also using UC.
So I started reading the docs and the Issue threads for both projects on Github.

It seems like there are several approaches how to use Seleniumbase with UC=true and I am not sure which might be the best for my needs or if it will work at all.
The reason is I do have a larger project realized with UC.
Now my goal is to be undetectable again using Seleniumbase with UC=true with as less code changes as possible.

What I am doing now with UC and want to achieve with Seleniumbase is:

• Using multithreading to run several chrome instances, each instance separated from the others and each instance using its own profile-folder.
• Bypassing CF captcha (nowsecure.nl) and be undetectable in every situation.
• Using chrome_options, for example chrome_options.add_argument() but also change the profile, which only seems to be possible using baseCase and overwrite the get_new_driver function. See How to add options #1221.

I am not sure if it will be possible and which is the best approach to realize it with less code changes.

[mdmintz]

Here are some existing examples that use --uc mode:

• https://github.com/seleniumbase/SeleniumBase/blob/master/examples/verify_undetected.py
• https://github.com/seleniumbase/SeleniumBase/blob/master/examples/uc_cdp_events.py
• https://github.com/seleniumbase/SeleniumBase/blob/master/examples/raw_uc_mode.py

Here's a raw driver example:

from seleniumbase import Driver
import time

driver = Driver(uc=True)
driver.get("https://nowsecure.nl/#relax")
time.sleep(6)
driver.quit()

---

SeleniumBase UC Mode can be activated by adding --uc as a pytest command-line option. For the formats that support it, using uc=True is another option. If you overwrite the get_new_driver function, then UC Mode will no longer work (because then all the UC Mode code is gone). Using a custom Chrome profile or user_data_dir will likely also break UC Mode, because the default one is specially configured to appear as a regular human-controlled browser profile. The default options are the ones that are specially configured to work best with UC Mode when using that mode.

---

Multi-threading is possible, but you need to use -n8 (8 threads for example) with pytest, as it has the pytest-xdist plugin that is specially configured to support multi-threading. More info on multi-threaded UC Mode: #2006 (comment)

Here's a sample file that uses @pytest.mark.parametrize() to turn one test into four tests when run with pytest:

import pytest

@pytest.mark.parametrize("", [[]] * 4)
def test_multi_threaded(sb):
    sb.open("https://nowsecure.nl/#relax")
    try:
        sb.assert_text("OH YEAH, you passed!", "h1", timeout=5.25)
        sb.post_message("Selenium wasn't detected!", duration=2.8)
        sb._print("\n Success! Website did not detect Selenium! ")
    except Exception:
        sb.fail('Selenium was detected! Try using: "pytest --uc"')

Here's the output when running that file with pytest --uc -n4:

pytest test_multi_uc.py --uc -n4
============================ test session starts =============================
platform darwin -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0
rootdir: ~/github/SeleniumBase/examples
configfile: pytest.ini
plugins: html-2.0.1, rerunfailures-12.0, cov-4.1.0, metadata-3.0.0, ordering-0.6, xdist-3.3.1, seleniumbase-4.17.13
4 workers [4 items]     

 Success! Website did not detect Selenium! 

 Success! Website did not detect Selenium! 
..
 Success! Website did not detect Selenium! 
.
 Success! Website did not detect Selenium! 
.
============================= 4 passed in 9.38s ==============================

[oculunto]

I understand, so it would be possible to add an arg to Driver that disables/enables patching when i want (I don't need to patch the same driver multiple times in a short time frame, only when a new version is available) ? I will look into it, thank you and thank you for your work !

[mdmintz]

Where did you get the "5" seconds from? When I ran UC Mode tests with pytest multithreading, it only spaced out browser launches by slightly less than 1 second if the driver was already patched. If it wasn't already patched, then there was a 3-second spacing for patching the first one, then back to slightly less than 1 second after that between launches. Tested on the latest version.

[oculunto]

It's not exactly 5s, but when opening many drivers with a 2s delay between each, some get stuck somewhere (I have to ctrl+C), I am also using proxy and user dir for each. Maybe I am doing something wrong somewhere, because I just adapted it from undetected-chrome.

[mdmintz]

There are several differences now between undetected-chromedriver and SeleniumBase UC Mode.
For the best multi-threading experience with UC Mode, use it together with pytest multithreading.

[oculunto]

Understood, thank you!

[MacMarde]

@mdmintz
Why is UC working in SB but the UC directly from the UC github project is not?
Aren't you guys working together and share information?

[mdmintz]

There is some information sharing. Eg. I shared information on how to fix headless mode:

• Issue with --headless with Chrome V110 (Python) ultrafunkamsterdam/undetected-chromedriver#1046 (comment)
• Issue with --headless with Chrome V110 (Python) ultrafunkamsterdam/undetected-chromedriver#1046 (comment)

Then I received a mention for it:

https://github.com/ultrafunkamsterdam/undetected-chromedriver/blob/cea80717c5a3d95ccf5c40e6e38081d5454ec7a5/README.md?plain=1#L65

And SeleniumBase UC Mode is based on undetected-chromedriver, so there are many similarities.

I've also suggested other ideas there, such as how to prevent browser/driver mismatch errors, etc.

Our projects do evolve separately though, so there will be some differences in how undetected-chromedriver and seleniumbase UC Mode work. I'm also one of the maintainers of Selenium, so I have some deep insights on how some things work with selenium: https://github.com/SeleniumHQ/selenium/pulls?q=is%3Apr+author%3Amdmintz+is%3Aclosed+

In summary, the differences matter. The results will be different between different frameworks.

[MacMarde]

Thats cool.
But I guess you do not know why UC from ultrafunkamsterdam isn't working?

[mdmintz]

That might be a better question for the maintainer of that repo.

But if you have questions specifically related to SeleniumBase UC Mode, then this is the place to ask.

[MacMarde]

Ok thank you. I need to do some tests first.

[MacMarde]

@mdmintz
I have tried using custom user_data_dir with Driver() and it is not working.
I can not pass CF captcha.

from seleniumbase import Driver
chrome_obj = Driver(uc=True, user_data_dir=user_data_dir)

So as far as I did understood you there is no way to use SB with UC and user_data_dir and still be undetectable?

[mdmintz]

There's no problem if the user_data_dir was created by a test running in UC Mode. If you had an existing user_data_dir that was made from regular Chrome, and you tried using it with UC Mode, then there would be a data mismatch, and you would get detected.

[MacMarde]

@mdmintz
Well I am a little bit confused now, because I DO use the user_data_dir created by the test running in UC Mode.

So first thing I tried for now is just using Driver(uc=True) without any user_data_dir specified.
Astonishingly it also stopped working. I am 100% sure it was working 2 hours before, see my post here.

So maybe UC just stopped working?

Does it also work when I enter www.nowsecure.nl manually in the browser adress field, instead of using driver.get(https://www.nowsecure.nl)?
This is the only difference I can think of.

[MacMarde]

@mdmintz
I have tested it myself and yes it depends if you enter https://www.nowsecure.nl manually or if you use driver.get(https://www.nowsecure.nl).
In the first case it does not work and in the second case it is working.

[mdmintz]

@MacMarde What version of SeleniumBase are you using?
And are you using uc=True ?

[MacMarde]

@mdmintz
Seleniumbase 4.17.14
using driver = Driver(uc=True)