Chrome 59: headless browser testing with Selenium WebDriver

By Zhimin Zhan

Chrome 59 is released this week. One key new feature that will interest test engineers is: headless mode. 'Headless' in this context means a Chrome browser without graphic user interface. Frankly, as I wrote in my books, I am not a big fan of using headless browser simulation, such as HtmlUnit and PhantomJS, in test automation. So far, I seem to be right, HtmlUnit is rarely seen nowadays and PhantomJS maintainer stepped down (as a result, the project future is in doubt). However, Chrome might bring headless testing back to real use, simply because it is the most popular web browser and the reputation of the team behind.

Before I show the selenium webdriver test scripts driving headless Chrome, please not that 'Headless mode' only works on Mac and Linux at the moment. Don't worry, Window Support is coming.

I will use Selenium WebDriver in Ruby, to change to use 'headless mode' is easy: adding a '--headless' switch as below.

driver = Selenium::WebDriver.for :chrome:switches => ["--headless"]  
driver.get("http://sandbox.clinicwise.net")
expect(driver.title).to eq("Wise Clinic")
driver.quit

For some Chrome builds, '--disable-gpu' might also be needed.

:switches => ["--headless", "--disable-gpu"

Now let's see it in action: YouTube Video: Run Selenium tests in Chrome Headless

For above simple test, besides not being able to resize browser window (which is fair given there is no browser window in headless), it works fine and slightly faster (6.7 vs 5.4 seconds).

The next question is, how stable and faster will be in real life test automation projects (i.e, most complex UI)? So I decided to test with my own project ClinicWise, which has 575 UI tests. First, I give a run against a typical test (booking appointment), the test running in headless mode failed quickly: returning "Selenium::WebDriver::Error::ElementNotVisibleError".

when trying to clicking a menu.

driver.find_element(:id, "menu_calendar_link").click

As a result, my experiment ended quickly. However, it could be due to the version of chromedriver. The current version 2.29 officially supports Chrome v56-58.

In summary, I am happy to see the headless support in Chrome. I already used it to automate some tasks. For example, I have a set of automation script to generate Ebooks on Leanpub, now with Chrome headless, the scripts are running in 'stealth' mode, less distraction.

Will I use it for test automation? My answer is NO for the reasons below:

  1. Only marginally faster. For simple benchmark tests I conducted, headless probably gain 10-20% in raw execution. I much prefer running selenium tests in parallel in Continous Testing Server like BuildWise. Here is a screenshot of a recent build report, saving 78.2% percent execution time with 6 build agents running tests in parallel.

2. Not reliable, however, I believe it will go better in future versions.

3. Unable to run window-size related tests. For example, selenium tests for responsive UI involve resizing browser windows.

The test script can be found in my book: Selenium WebDriver Recipes in Ruby.