Chrome 59: headless browser testing with Selenium WebDriver

By Zhimin Zhan.

Chrome 59 is released this week. One key new feature that will interest test engineers is: headless mode. 'Headless' in this context means a Chrome browser without graphic user interface. Frankly, as I wrote in my books, I am not a big fan of using headless browser simulation, such as HtmlUnit and PhantomJS, in test automation. So far, I seem to be right, HtmlUnit is rarely seen nowadays and PhantomJS maintainer stepped down (as a result, the project future is in doubt). However, Chrome might bring headless testing back to real use, simply because it is the most popular web browser and the reputation of the team behind.

Before I show the selenium webdriver test scripts driving headless Chrome, please not that 'Headless mode' only works on Mac and Linux at the moment. Don't worry, Window Support is coming.

I will use Selenium WebDriver in Ruby, to change to use 'headless mode' is easy: adding a '--headless' switch as below.

driver = Selenium::WebDriver.for :chrome, :switches => ["--headless"]  
driver.get("http://sandbox.clinicwise.net")
expect(driver.title).to eq("Wise Clinic")
driver.quit

For some Chrome builds, '--disable-gpu' might also be needed.

:switches => ["--headless", "--disable-gpu"] 

Now let's see it in action: YouTube Video: Run Selenium tests in Chrome Headless

For above simple test, besides not being able to resize browser window (which is fair given there is no browser window in headless), it works fine and slightly faster (6.7 vs 5.4 seconds).

The next question is, how stable and faster will be in real life test automation projects (i.e, most complex UI)? So I decided to test with my own project ClinicWise, which has 575 UI tests. First, I give a run against a typical test (booking appointment), the test running in headless mode failed quickly: returning "Selenium::WebDriver::Error::ElementNotVisibleError".

when trying to clicking a menu.

driver.find_element(:id, "menu_calendar_link").click

As a result, my experiment ended quickly. However, it could be due to the version of chromedriver. The current version 2.29 officially supports Chrome v56-58.

In summary, I am happy to see the headless support in Chrome. I already used it to automate some tasks. For example, I have a set of automation script to generate Ebooks on Leanpub, now with Chrome headless, the scripts are running in 'stealth' mode, less distraction.

Will I use it for test automation? My answer is NO for the reasons below:

  1. Only marginally faster. For simple benchmark tests I conducted, headless probably gain 10-20% in raw execution. I much prefer running selenium tests in parallel in Continous Testing Server like BuildWise. Here is a screenshot of a recent build report, saving 78.2% percent execution time with 6 build agents running tests in parallel.

2. Not reliable, however, I believe it will go better in future versions.

3. Unable to run window-size related tests. For example, selenium tests for responsive UI involve resizing browser windows.

The test script can be found in my book: Selenium WebDriver Recipes in Ruby.

Read more

Complimentary TestWise and BuildWise licenses for referring ClinicWise

By Zhimin Zhan.
If you have followed AgileWay's toolset, my blog or books, you probably know about ClinicWise, a web-based clinic management software. The development and maintenance of ClinicWise heavily relies on the Test Automation and Continuoous Integration process using TestWise and BuildWise. Furthermore, we have a public Sandbox ClinicWise site, which serves two purposes:
  • We provide free test script solutions for TestWise owners who replicate the test they are finding to be a challenge on Sitewise
  • A modern, fast and production-quality (more importantly, with database reset) target site for automation testers to practise writing scripts
On the other hand, I understand testers/developers may want to learn/adopt test automation with TestWise and BuildWise, but are not in a position to purchase. While TestWise and BuildWise Agent have free functional versions, it is recommended to go for the professional edition to take the talked-a-lot-but-rarely-implemented test automation challenge.
Here is a proposal we came up with for the new year: If you like ClinicWise and recommend it to the clinics you know (physio, dental , ..., etc) and they sign up officially, AgileWay will issue one complimentary BuildWise Agent or TestWise license. The more referrals, the more licenses. The details are still in the works, but it is active from today. So you might help your friend to use an efficient, easy-to-use, reliable and affordable web-based practice management system, and do yourself a favour to get TestWise and BuildWise.

Read more

Is it safe to upgrade to Selenium WebDriver 3? Yes!

By Zhimin Zhan.

Selenium WebDriver 3 was released three days ago! Since there is 5 years between versions 2 and 3, a lot changes are expected in this major release. For teams with existing test suite, here comes the question:

  • Will there will be a lot of work to upgrade existing test scripts to Selenium WebDriver 3?

My short answer: Selenium WebDriver 3 is a drop-in replacement for Selenium WebDriver 2.53. As I am a test engineer, my statement is backed by testing. I executed my test suite for ClinicWise : 520 user story level Selenium WebDriver test scripts in Ruby.

Before the testing, I upgraded Chrome to the latest version 2.54 and BuildWise Agent1.7 with latest Selenium-WebDriver 3.0 gem (the latest chromedriver 2.24 is embedded in BuildWise Agent).

I triggered a build to run all tests (not changed) in BuildWise, which distributed these 520 tests to 6 BuildWise Agents to run them in parallel. Here is the result:

All tests pass. In terms of performance, I couldn't compare precisely because test scripts were assigned to different build machines. Overall, it seemed slightly faster.

For my another test suite for SiteWise (the content management system I developed), I did find an issue: a test failure on file uploading in a modal dialog within SummerNote editor.

 Failure/Error: insert_image_modal_page.select_file(test_file("green_lamp.jpg"))

  Selenium::WebDriver::Error::UnknownError:
    unknown error: cannot determine loading status
    from unknown error: missing 9aea000394714d2fbb20850021f6204f2256b9cf),platform=Windows NT 6.1.7601 SP1 x86_64)
     # ./vendor/source/selenium-webdriver-3.0.0/lib/selenium/webdriver/remote/response.rb:69:in `assert_ok'
     # ./vendor/source/selenium-webdriver-3.0.0/lib/selenium/webdriver/remote/response.rb:32:in `initialize'

However, I believe this is an issue with the chromedriver with Chrome 54. The test passed with Chrome 53.

My suggestion: before upgrading to Selenium WebDriver, make sure test execution of your test suite is reliable (it is easier to say than getting it done, I know) and lock down the version of Browser and WebDriver (such as chromedriver). Then upgrade.

Read more

Selenium WebDriver 3 is out

By Zhimin Zhan.

Five years after Selenium 2, finally, Selenium WebDriver 3 is released. For testwise users who want to be an early adopter of Selenium WebDriver 3, We release the following TestWise beta versions supporting latest Selenium WebDriver:

Selenium with Firefox 48+ (GeckoDriver) is still problematic. If you got the error "Could not find Firefox binary (os=windows).", set the Firfox binary path specificly

Selenium::WebDriver::Firefox::Binary.path = "C:/Program Files/Mozilla Firefox/firefox.exe"
driver = Selenium::WebDriver.for(:firefox)

Read more

TestWise Community Ed. switch to Chrome browser

By Zhimin Zhan.

There is no built-in support for Selenium WebDriver from Firefox 47+,  you need GeckoDriver. Installation of GeckoDriver is the same as ChromeDriver.  We received several enquires on TestWise no longer works with Firefox 47+.   The solution is to download the working version of GeckoDriver and put it in the PATH. 

GeckoDriver is still in beta, and we had issues with one version works on Firefox 47 but not 48. To avoid the hassles for TestWiser Community Edition users, We switched the default (and only) browser to Chrome. This makes sense too, as Chrome is the most used browser and the convenience of built-in support for Firefox no longer exists. 

Download TestWise Community Edition.

Read more

TestWise pricing changes

By Zhimin Zhan.

TestWise, the next-generation functional testing tool, now starts from US $20 per month (and $15/month for renewals). Get your own TestWise today!

Product License Price
TestWise Personal License

US $240 first year

US $180 on 2nd year onwards 		Bonus one eBook
TestWise Floating License

US $480 first year

US $360 on 2nd year onwards 		Bonus two eBooks

You may choose your own eBook at the support site: https://agileway.net

Read more

Why David Thomas says "Agile is Dead"

By Zhimin Zhan.

David Thomas, one of Agile Manifesto author, publishd this article Time to kill Agile. He suggested: "So I think it is time to retire the word 'Agile.'". In this recent interview, David Thomas, says This is all to do with the fact that "agile" has become meaningless."

This surely upsets many agile coaches, who makes money with the word 'Agile'. There are surely a few of them are really good and know what they are talking about, but most are not. Let's put it in test, try to answer questions below:

  • How often do your project release your software?
  • How do you honestly think customers' confidence on the product?
  • How do you do regression testing?

The objective of Agile, if my still use this word, is to release software frequently to product with high quality. I think the reason that Agile word lost the meaning due to it does not quantify the "frequently" and "high quality".

  • High Quality: every release must pass full automated regression tests
  • Frequently: multiple times a day to production, yes, in a matter of hours.

Sound impossible? Facebook release twices a day, is Facebook doing Agile? Comparing to Facebook, how are you doing Agile? Have a look at the Facebook's test lab:

.

Read more

© 2006 - 2024 AgileWay Pty Ltd. Powered by SiteWise CMS