Headless Chrome with Python and Selenium

In version 59, Google Chrome acquired the option to run headlessly! Here is what works for me (Ubuntu: 14.04, Chrome version: 60.0.3112.78, chromedriver: 2.31) :

from selenium import webdriver

def get_page_html(url, headless=False, screen_shot_path=None):
    options = webdriver.ChromeOptions()    
    if headless:
        options.add_argument('--headless')
    options.add_argument('--disable-gpu')
    selenium = webdriver.Chrome("/usr/local/share/chromedriver", chrome_options=options)
    self.selenium.set_window_size(1800, 1000)  # w,h
    
    selenium.get(url)
    page_html = selenium.page_source
    if screen_shot_path:
        selenium.save_screenshot(screen_shot_path)

    selenium.quit()
    return page_html

Make sure your version of chromedriver is up-to-date. Also, notice that the options.add_argument() expects the arguments to be as if you were running google-chrome from the command line (e.g. prefix with –).

Django, Selenium Headless Tests Fail

Running headless selenium tests is great. Except when the tests succeed when headed, then fail when headless. When that happens it’s hard not to let your imagination run wild with crazy javascript thoughts.

There are many ways to run headless. The method I use is described in this post. In a sense this is not headless because a normal browser is running; it’s just running on a hidden display. This makes the headless fails all the more perplexing.

In this case, the problem was the headless browser window was a different size compared to the headed version. I use the chosen widget a lot. It’s a fancy combo of an HTML select widget and auto-complete. Clicking on this widget creates a pop-up drop down. If the needed choice is outside of the browser, selenium auto scrolls the page so that the choice is visible. When it decides to scroll depends on the browser window dimensions.

In many cases, this auto scroll would no create problems. In my case, I also have a nav-bar locked to the top of the screen. Sometimes the desired choice would end up under the nav-bar after auto-scroll. When selenium moved to select the choice, it would get the nav bar instead. The form value would not be set and the form would fail. Although it did not happen to me, I supposed it’s possible selenium could have ended up on a different page. That could make some strange errors.

The first part of the fix is to explicitly set the browser window so the headed and headless versions are the same. Something like this:

if getattr(settings, 'HEADLESS_TESTS', False):
    self.vdisplay = Display(visible=False, size=(1600, 1000))
    self.vdisplay.start()
self.selenium = webdriver.Firefox()
self.selenium.set_window_size(1500, 900)

If all the problems go away, you are done. If not, then hopefully the headed tests and headless tests will now fail in the same way, making debugging easier.

In my case, I added code to scroll the page if the item was under the nav bar. Here is the code.