Introduction

BuildWise is a Continuous Testing Server. To put it simply, BuildWise helps software teams to run automated UI/API tests and release the quality software faster.

Continuous Testing is a joint term of another two popular terms in software development: Continuous Integration and Automated Testing. Unfortunately, despite of being talked a lot, many software professional have fuzzy views of CI and test automations, rarely see them in action with good outcome. For example, we commonly heard: “We have a test automation team, but we don’t what they do, our testing is performed by manual testers”, “Our CI server does build and deployment only, no, it does not run tests. Some unit tests were run before, but hard to get all pass, it is mostly broken now and no one cares”.

If you are interested in why, click the links to view the excerpt of the upcoming book Continuous Web App and API Testing.

BuildWise is a Continuous Testing solution for small to medium companies to adopt continuous testing to achieve real Agile/DevOps: release high quality software at great efficiency. BuildWise server is 100% open-source (in Ruby) and free.

Few people have seen real UI Test Automation

Test Automation is the foundation of Agile, with which few will argue. Many architects/test managers/software engineers in test talk about test automation constantly. However, few people have ever seen real UI test automation. Try answering these questions:

  1. Do automated functional tests cover most features (or user stories) of your application?
  2. How many automated tests do your team have?
  3. Is automated test execution visible to the team?
  4. Do you run ALL automated UI tests against every build candidate?
  5. How often do your run the automated regression test suite? weekly, or 5 times a day?
  6. What do your teams do for test failures?
  7. Does every release candidate (available to customers to check) have to pass the automated regression suite?

These are simple questions, yet very few can answer them well. The reason is simple: UI test automation is extremely hard. Don’t just take my word for it, let’s see what renowned experts say:

  • “UI Test Automation Tools are Snake Oil” - Michael Feathers (author of ‘Working Effectively with Legacy Code’)

    The following scenario has happened over and over again: I visited a team and asked about their testing situation. We talked about unit tests, exploratory testing, the works, etc. Then, I asked about automated end-to-end testing. They pointed at a machine in the corner. That poor machine has had an installation of some highly-priced per seat testing tool (or an open source one), and the chair in front of it was empty. We walked over, swept the dust away from the keyboard, and loaded up the tool. Then, we glanced through their set of test scripts and tried to run them. The system failed a couple of times and then they gave me that sheepish grin and said “we tried.” I said, “don’t worry, everyone has.”

  • “For 95% of all software applications, automating the GUI is a waste of time.” - Alan Page, author of “How We Test Software at Microsoft”.

    Alan further added, “For the record, I typed 99% above first, then chickened out. I may change my mind again.” 7 years later, in an interview in 2015, Alan still mostly thought “I think it’s really hard to get right”.

  • “Automated testing through the GUI is intuitive, seductive, and almost always wrong!” - Robert C. Martin, 2009-09-30, Coauthor of the ‘Agile Manifesto’

Real UI test automation is that software teams run full regression automated tests many times a day. If they all pass, software is ready to ship to production. Some might find it hard to believe that all AgileWay’s web applications are developed and maintained this way, see the case study of ClinicWise below.

UI Testing: Executing Selenium WebDriver tests in Chrome

Further readings:

Few run UI tests in traditional CI servers

Ever since Martin Fowler published the famous “Continuous Integration” article in 2006, CI has been one of most talked topic in software world. Aligning with Marin’s definition of a CI process: “fully automated and reproducible build, including testing, that runs many times a day”, I believe a typical CI build will perform the following core tasks:

  1. Build software project, such as compiling code and packaging software
  2. Run Programmer Test (Unit Tests)
  3. Run full regression functional (UI/API) tests. If pass, release to production.

We can find the use of Continuous Integration servers (such as Jenkins, Bamboo, TeamCity, …) in many so-called ‘agile’ projects. However, most software teams have never reached Step 3. Functional testing is mostly the same: test manually. Without automated regression tests, the whole CI promise (release high quality software faster) falls short. Ironically, the first sentence of the original CI article is: “Software development is full of best practices which are often talked about but seem to be rarely done”, 12 years later, despite of the high hype, it is still largely true.

There are mainly two technical reasons that have contributed to this situation:

  1. Testers lack of knowledge to develop maintainable automated UI test scripts and maintain them efficiently.
  2. Long feedback loop from UI test execution.


Test Automation Camel - excerpt from Practical Web Test Automation: two humps for achieving UI test automation

Why traditional CI servers not fit for running UI tests?

The test execution features in traditional CI servers are for unit testing, which is quite different from UI test execution. Comparing to unit tests, UI tests are

  • far more fragile, changing one simple button text on one page might break a large number of UI tests.

  • much slower in execution, and there are very limited ways that we can speed up UI test execution.

  • more sensitive to external environments, such as regular checking Chromedriver’s version adjacent Chrome browser, the machine’s time zone settings, etc.

  • fragile due to dynamic nature of web applications, the wide use of JavaScripts and AJAX (which make web apps more dynamic) bring challenges……….

Besides, it often requires a team effort to detect UI test failures from a build,. Firstly, testers need to find the repeating steps and verify them, then communicate with programmers, business analysts (for clarification) or even the manager. With more tests added to the automated regression test suite, testers will find the effort of maintaining the test suite grows exponentially if the process is not super efficient,.

In summary, executing UI tests in CI is difficult, and much more challenging than expected. This is why we see failed attempts everywhere. For example, so called ‘DevOps architects’ managed to get a task configured to run a batch Ui tests. With the nondeterministic number of failures growing every day, they eventually gave up.

Some might have doubts: Is this Agile? How could we be fast if we have to stop to address defects from testing? Please refer to the “Toyota Factory story” in the classic Agile book Extreme Programming Explained from Kent Beck: the father of Agile and the first author of Agile Manifesto. If there are still doubts for the real meaning of CI, please refer to Martin Fowler’s original “Continuous Integration” paper. As a matter of fact, if you step out of he current bad practices of Agile, CI or DevOps, you will see it quite obvious and clear: without end-to-end comprehensive testing, you will pay big price for buggy releases; without the foundation (quality through test automation), Agile simply is not possbile.

Still not convinced? Have a look at How Facebook does CI?, a great presentation by Facebook engineer at F8 conference. Facebook “top engineers” built the SandCastle CI infrastructure to ensure ‘Facebook is released twice a day’(and keeping up this pace is at the heart of Facebook culture). Obviously, you can get Sandcastle, even Facebook makes Sandcastle publicly available, which is extremely unlikely, it will be too complex for most software teams. Luckily, we have BuildWise.

Further readings:

Set up and run Selenium Tests in BuildWise in 3 mins

For many of us, running UI tests in a server is a new concept. The below video shows that it can be done in around 3 minutes.

Case study - ClinicWise

ClinicWise is a modern web-based clinic management system. It is developed solely by me in my spare time (initially for my brother’s dental clinic, later became commercially available). The Continuous Testing process with BuildWise has made the development of this high quality and extreme high efficiency application possible.

You may check the clone of our Build server at http://buildwise.itest2.com.

Over 500,000 test executions over 4 years

The testing process prevents many many critical defects, quite some could have ended the project if not detected.

Over 600 comprehensive automated UI test cases

The number of automated UI tests increases gradually along with the development. Experienced test automation engineers will understand this is not just a simple increment. With every new features (or change request) implemented, besides developing new tests, all previous tests need to pass as well ( i.e. regression).

Reduce build time and increase stability with parallel test execution

This below chart shows test executions over 4.5 years (near 1900 builds).

Another message from the chart is the approach is scalable. With the number of test grows, the chance of getting a green build remains relatively stable.

© 2006 - 2024 AgileWay Pty Ltd. Powered by SiteWise CMS