Roadmap: Remote/headless desktops

[asweigart]

This issue tracks using PyAutoGUI on remote machines or headless machines. This is an advanced feature, and I personally am not planning to add it in the near future but I welcome anyone who would want to take a crack at it. Ideally, we'd have solutions for Windows, Mac, and Linux.

[david-guenault]

Seems that some french guy found a workaround (related to stackoverflow again). At least for mouse and image recognition. Just tested and it work like a charm
http://fredtantini.free.fr/blog/index.php?article58/automatiser-des-actions-avec-selenium-pyautogui-et-xvfb

[daluu]

Some related notes:

• w/o xvfb kind of thing on Windows, or using VNC, I doubt there is much to workaround issues with remote desktop (RDP) issues with PyAutoGUI (e.g. locked desktop, exited/minimized RDP session). See my old blog post for reference: https://autumnator.wordpress.com/2011/11/18/how-to-run-remote-headless-gui-automation/

• for remote execution of PyAutoGUI (but not headless per se) and better Selenium API integration, someone should consider a port of my https://github.com/daluu/autopydriverserver for PyAutoGUI. Perhaps I'll consider it when I have some time.

[alphaCTzo7G]

@daluu .. Pyautogui seems to be target towards platform-neutral application-neutral application..

Are you suggesting, that Pyautogui detect whether the application is a web-browser.. if so use the utopydriverserver?

I actually run Pyautogui on a remote WIndows server (VM) and it work just as well as on my desktop. I had to modify some registry settings: https://support.microsoft.com/en-us/help/324737/how-to-turn-on-automatic-logon-in-windows

so it automatically logs in everytime..

[daluu]

@alphaCTzo7G what I meant for my 2nd bullet point in previous comment was that for better integration of Pyautogui commands/code with Selenium commands/code say in a single test case or framework, to:

1. start up a RemoteWebDriver server instance of Pyautogui (this requires someone to create the Pyautogui equivalent or port of my AutoPyDriverServer, I haven't gotten around to it myself. Could name this version as PyautoguiDriverServer)
2. instantiate a (Web)driver instance of Pyautogui (which connects to server in step 1)
3. wherever you need to execute Pyautogui in Selenium test, you just call the webdriver instance of Pyautogui and issue the right commands (desktop GUI and web GUI tend to have same UI operations, so the WebDriver API maps nicely to desktop GUI operations).

This approach makes it simpler to integrate different tooling using same APIs (e.g. WebDriver), since Pyautogui I believe at last check doesn't have a remote execution interface, so you have to use SSH/telnet/PSExec.exe to call Pyautogui on a remote machine). That makes it more code & different code for the test writer. Normalizing it with WebDriver API with a server interface to Pyautogui makes things simpler and uniform to the test writer.

As for my 1st bullet point, that was just to highlight caveats to be aware of when executing any (desktop) UI automation on a remote machine. Not that it can't be done, like the adjustment you had to make as a workaround.

[doverradio]

Get it done please!

Getting blocked due to Selenium detection and knowing this could use the non-webdriver browser would be great.

[nandish017]

@asweigart I have a python script that establishes ssh connection to a remote machine and executes python script, normal operations work fine, but pyautogui functions are not working (click, locateonscreen etc) when i run locally it works fine. Also after establishing ssh connection i see screen resolution is (1024768) where as original screen resolution of remote machine is (19801080).
Can you please help me with that?
Os- Windows 10
Python version- 3.8.2

Not exactly related to this particular issue (headless mode), but I was able to implement the solution described by @david-guenault above. (Docker or plain Python on Linux.)

https://github.com/claudiobizzotto/pyautogui-with-selenium

[geojaya]

i have python script on remote Windows 10 pc Python version 3.97x, where I am logged to the desktop through rdp. When I run locally, it works fine on remote pc , locateonscreen and screenshot from pyautogui - works fine . I tried two options : 1. scheduled the python with a bat file in Windows task scheduler. I enabled the task - while I am able to get size() of the screen but screenshot does not work. Please note that I am logged in via rdp. (2) option - I tried the same psexec (from Sysinternals). here again I am getting size() function result but erroring on screenshot . Appreciate if anyone can share any references where it worked - or may be another approach