I have been reading about how to structure my automation scripts for checking
scenarios, and this is what I have learnt.
The Test Pyramid
(Source: Test Pyramid)
First of all, we should know the test pyramid concept that was developed by Mike Cohn, and described in his book Succeeding with Agile.
In a nutshell, the concept is about having many more low-level unit tests than high-level end-to-end UI tests. UI tests are brittle, expensive to write, and time consuming to run. So, the automation should lean much more towards unit tests.
UI layer refers to the end-to-end checks using something like Selenium or Sahi. The Service layer refers to checks that act through a service layer of an application. (Eg. checking through an API layer)
And the final base of checks is the low-level unit tests.
Guidelines for structuring automated tests
#1 Structure – Prioritize creation of unit tests to cover most test scenarios over
any other types of tests. Unit tests should cover the bulk of your testing strategy.
Keep UI tests to a minimum, only covering critical happy flow scenarios & failures. Avoid making assertion on validation error message in UI level. It should just verify the error case by virtue of redirect to same page. All validation scenarios are already covered with messages in unit tests
#2 Speed – Team should collectively decide the acceptable time for running the test suite, define it for all test types – unit, integration and acceptance.
Create mocks for any external services that tests might be dependent on.
#4 Debugging – When it fails, it should point broken code.
Self Testing Code
(Source: Self Testing Code)
Every object should be able to test itself. Write your code in such a way that typing a command should make the whole software system do a self-test.
Also, practice Test Driven Development while working.
This command will be run every time code is committed and pushed to Git develop branch. By running the test suite frequently, at least several times a day, you’re able to detect such bugs soon after they are introduced, so you can just look in the recent changes, which makes it much easier to find them.
Build a testing culture where developers are naturally thinking about writing code and tests together.
When Should a Test Be Automated?
Do not over-automate. Only some tests should be automated.
Decision process would use following questions:
1. If I automate this test, what manual tests will I lose (not be able to execute in the time given)? How many bugs might I lose with them? What will be their severity?
2. An automated test has a finite lifetime, during which it must recoup that additional cost. Is this test likely to die sooner or later? What events are likely to end it?
When deciding whether to automate a test, you must estimate how many code changes it will survive. When the test breaks after code change, if it hasn’t repaid the automation effort by that point, you would have been better off leaving it as a
Create product-specific test libraries. After that many user interface changes will
require no changes to tests, only to the test library, thus, saving a lot of effort.
3. During its lifetime, how likely is this test to find additional bugs (beyond whatever bugs it found the first time it ran)? How does this uncertain benefit balance against the cost of automation?
If those questions don’t suffice for a decision, other minor considerations might tip the balance.
Note – The tests you need are often called task-driven tests, use-case tests, or scenario tests. Because scenario tests favor common tasks, they find the bugs most users will find.
Some of the secondary considerations
* Humans can notice bugs that automation ignores, especially GUI issues.
* But, while humans are good at noticing oddities, they’re bad at painstaking or precise checking of results.
* As humans can’t be precise about inputs, repeated runs of a manual test are often slightly different tests, which might lead to discovery of a support code bug.
For example, people make mistakes, back out, and retry inputs, thus sometimes stumbling across interactions between error-handling code and the code under test.
* Because test automation takes time, you often won’t report the first bugs back to the programmer as soon as you could in manual testing.
* Automated tests, if written well, can be run in sequence, and the ordering can vary from day to day
* An automated test might not pay for itself until next release. A manual test will find any bugs it finds this release. Bugs found now might be worth more than bugs found next release.
Within your web app’s UI there are areas that your tests interact with. A Page Object simply models these as objects within the test code. This reduces the amount of duplicated code and means that if the UI changes, the fix need only be applied in one place.
Some other articles I read that could be useful –