Lesson 48:

Controlling the Mouse with Python

Python can be used to control the keyboard and mouse, which allows us to automate any program that uses these as inputs.

Graphical User Interface (GUI) Automation is particularly useful for repetative clicking or keyboard entry. The program's own module will probably deliver better programmatic performance, but GUI automation is more broadly applicable.

We will be using the pyautogui module. The documentation is available here.

You can follow instructions on this page to install the necessary packages. (I had particular trouble installing the pyobjc dependency, so if you have similar issues in your environment, it might be useful to install it from source using Mercurial. You may also need the pillow module.)



In [1]:

    
import pyautogui

This lesson will control all the mouse controlling functions in this module.

The module treats the screen as cartesian coordinates of pixels, with x referencing a point on the horizontal line, and y referencing a point on the vertical line.

We can examine the available screen size using the size() function, with variables returned as an (x,y) pair.



In [5]:

    
pyautogui.size()









    Out[5]:





(1680, 1050)

We can store this tuple in two variables, width and height:



In [6]:

    
width, height = pyautogui.size()

Similarly, the position() function returns the current position of the mouse cursor.



In [7]:

    
pyautogui.position()









    Out[7]:





(289, 843)

The left and right most corners are 1 less than the maximum.



In [8]:

    
# left corner
print(pyautogui.position())









    Out[8]:





(0, 1049)



In [9]:

    
# right corner
print(pyautogui.position())

The first function to control the mouse is the moveTo() function, which moves the mouse immediately to an absolute location.



In [12]:

    
pyautogui.moveTo(10,10)

We can pass the duration parameter to this function slow down this process to simulate human activity over the duration defined.



In [14]:

    
pyautogui.moveTo(10,10, duration=2)

We can also use the moveRel() function to move the mouse relative to a certain position.



In [15]:

    
pyautogui.moveRel(200, 0, duration=2)

We can also pass in y coordinates to move up or down, but we have to use negative values to move 'up' in a relative way.



In [18]:

    
pyautogui.moveRel(0, -100, duration=1.5)

Now that we have mastered movement, we can now use click() functions to interact with objects.



In [34]:

    
# Find the 'Help' button in the top Jupyter Navigation
# helpCoordinates = pyautogui.position()

helpCoordinates = (637, 126)

# Click at those coordinates
pyautogui.click(helpCoordinates)

We can use functions like rightClick(), doubleClick(), or middleClick() for similar behavior. We can even run these functions without any coordinates, which will click at any particular location.

We also have dragRel() and dragTo() functions, which can be used to click and drag; used here to draw in a paint program.

An important thing to note is that during automation, the script will control your mouse and keyboard, which may affect your ability to interact with the computer (like what happened in Fantasia).

To avoid this pyautogui has a built in fail safe, which checks if the mouse is in coordinate (0,0); the top left of the screen. If it is, the script terminates. Commands take a tenth of a second between commands to run, so any indicator in that period will raise an error.



In [35]:

    
pyautogui.moveRel(0, -100, duration=1.5)









    



---------------------------------------------------------------------------
FailSafeException                         Traceback (most recent call last)
<ipython-input-35-d3785f96b921> in <module>()
----> 1 pyautogui.moveRel(0, -100, duration=1.5)

/usr/local/lib/python3.5/site-packages/pyautogui/__init__.py in moveRel(xOffset, yOffset, duration, tween, pause, _pause)
    644     xOffset, yOffset = _unpackXY(xOffset, yOffset)
    645 
--> 646     _failSafeCheck()
    647 
    648     _mouseMoveDrag('move', None, None, xOffset, yOffset, duration, tween)

/usr/local/lib/python3.5/site-packages/pyautogui/__init__.py in _failSafeCheck()
   1007 def _failSafeCheck():
   1008     if FAILSAFE and position() == (0, 0):
-> 1009         raise FailSafeException('PyAutoGUI fail-safe triggered from mouse moving to upper-left corner. To disable this fail-safe, set pyautogui.FAILSAFE to False.')
   1010 
   1011 

FailSafeException: PyAutoGUI fail-safe triggered from mouse moving to upper-left corner. To disable this fail-safe, set pyautogui.FAILSAFE to False.

To make pyautogui useful, you need to know the coordinates on the screen at any given time, and interact with them. The module contains a useful sub program, called displayMousePosition, which can be run via the terminal to track the position of the real-time mouse at any given time (it also returns RGB values.)



In [ ]:

    
# Not run here, just keeps spitting out print functions. Much more useful in the terminal.
pyautogui.displayMousePosition()

Recap

Controlling the mouse and keyboard is called GUI automation.
The pyautogui module has many functions to control the mouse and keyboard.
The pyautogui.size() function returns the current screen resolution.
The pyautogui.position() returns the current mouse position, as a tuple of two integers.
The pyautogui.moveTo() function moves the mouse instantly to a cartesian (x,y) coordinate on the string.
The pyautogui.moveRel() function moves the mouse to a point relative to the current position.
Both of these functions take a duration paramater to slow the mouse transition.
The pyautogui.click(), pyautogui.doubleClick(), pyautogui.rightClick() and pyautogui.middleClick() all clcik the mouse buttons.
The pyautogui.dragTo() and pyautogui.dragRel() functions will move the mouse while holding down the mouse button.
If your program gets out of control, activate the failsafe by quickly moving the cursor to the top left of the screen.