Code structure for automated checks


I have been reading about how to structure my automation scripts for checking
scenarios, and this is what I have learnt.
 
The Test Pyramid
(Source: Test Pyramid)
First of all, we should know the test pyramid concept that was developed by Mike Cohn, and described in his book Succeeding with Agile.
In a nutshell, the concept is about having many more low-level unit tests than high-level end-to-end UI tests. UI tests are brittle, expensive to write, and time consuming to run. So, the automation should lean much more towards unit tests.
 test-pyramid
UI layer refers to the end-to-end checks using something like Selenium or Sahi. The Service layer refers to checks that act through a service layer of an application. (Eg. checking through an API layer)
And the final base of checks is the low-level unit tests.
 
Guidelines for structuring automated tests
#1 Structure – Prioritize creation of unit tests to cover most test scenarios over
any other types of tests. Unit tests should cover the bulk of your testing strategy.
Keep UI tests to a minimum, only covering critical happy flow scenarios & failures. Avoid making assertion on validation error message in UI level. It should just verify the error case by virtue of redirect to same page. All validation scenarios are already covered with messages in unit tests
#2 Speed – Team should collectively decide the acceptable time for running the test suite, define it for all test types – unit, integration and acceptance.
#3 Robustness – Avoid testing JavaScript behavior and scenarios heavily using
Selenium based acceptance tests. Instead, use Jasmine-based pure JavaScript unit testing, which are more robust and fast.
Create mocks for any external services that tests might be dependent on.
#4 Debugging – When it fails, it should point broken code.
 
Self Testing Code
Every object should be able to test itself. Write your code in such a way that typing a command should make the whole software system do a self-test.
Also, practice Test Driven Development while working.
This command will be run every time code is committed and pushed to Git develop branch. By running the test suite frequently, at least several times a day, you’re able to detect such bugs soon after they are introduced, so you can just look in the recent changes, which makes it much easier to find them.
Build a testing culture where developers are naturally thinking about writing code and tests together.
 
When Should a Test Be Automated?
(Source: Automate)
Do not over-automate. Only some tests should be automated.
Decision process would use following questions:
1. If I automate this test, what manual tests will I lose (not be able to execute in the time given)? How many bugs might I lose with them? What will be their severity?
2. An automated test has a finite lifetime, during which it must recoup that additional cost. Is this test likely to die sooner or later? What events are likely to end it?
When deciding whether to automate a test, you must estimate how many code changes it will survive. When the test breaks after code change, if it hasn’t repaid the automation effort by that point, you would have been better off leaving it as a
manual test
Create product-specific test libraries. After that many user interface changes will
require no changes to tests, only to the test library, thus, saving a lot of effort.
3. During its lifetime, how likely is this test to find additional bugs (beyond whatever bugs it found the first time it ran)? How does this uncertain benefit balance against the cost of automation?
If those questions don’t suffice for a decision, other minor considerations might tip the balance.
 
Note – The tests you need are often called task-driven tests, use-case tests, or scenario tests. Because scenario tests favor common tasks, they find the bugs most users will find.
Some of the secondary considerations
* Humans can notice bugs that automation ignores, especially GUI issues.
* But, while humans are good at noticing oddities, they’re bad at painstaking or precise checking of results.
* As humans can’t be precise about inputs, repeated runs of a manual test are often slightly different tests, which might lead to discovery of a support code bug.
For example, people make mistakes, back out, and retry inputs, thus sometimes stumbling across interactions between error-handling code and the code under test.
* Because test automation takes time, you often won’t report the first bugs back to the programmer as soon as you could in manual testing.
* Automated tests, if written well, can be run in sequence, and the ordering can vary from day to day
* An automated test might not pay for itself until next release. A manual test will find any bugs it finds this release. Bugs found now might be worth more than bugs found next release.
 
Page objects
Within your web app’s UI there are areas that your tests interact with. A Page Object simply models these as objects within the test code. This reduces the amount of duplicated code and means that if the UI changes, the fix need only be applied in one place.
Some other articles I read that could be useful –

Trying to be Agile


dilbert user stories

In my current company, we are still in the process of learning how to become Agile. It is a phase of confusion on how to overcome obstacles like the one depicted in the above Dilbert comic strip.

So, I have been reading and reading. I also picked up a book on “Agile Testing” by Lisa Crispin and Janet Gregory, and I am going through it with my testing team.

The questions that I need answered are –

1. People are resistant to change. How do we inspire them to adopt Agile behaviour without forcing it down their throats?

2. How do you handle someone who comes late in daily huddle almost every morning? Is that person supposed to be thrown out of the team?

3. How do you make the whole team feel responsible for quality when the programmers have made it a habit to leave all testing for the testers?

4. In our teams here, we have programmers specialized in particular sections of the code. Now, if we have a high priority task in one section (worth one or more sprints), and one lower priority task in another section. Tasks are taken up to give the full high priority section to the particular resource. The other resource is asked to work on the lower priority. What is the correct way? Can we improve?

5. We are struggling with the attitude that scrum is only for projects that do not get critical, urgent requirements frequently. How do we change this?

Maybe, as we finish the book, we will get answers to the above questions. Till then, I would love to hear any thoughts that you would like to share about them.

Book Reading: The One Minute Manager Builds High Performing Teams


I have an awesome team of high performers. While I am proud of them, I want them to achieve more and be even better, and for that I have to learn how to be a good leader and a great teacher. So, I have been trying to get hold of books that would help me support my team. This book was already in my dad’s book shelf, so I decided to read this first.

This book is written by Kenneth Blanchard along with Donald Carew and Eunice Parisi-Carew.

The book was good (not awesome maybe, but good). I would rate it 3 out 5 stars. I did get to learn something new that I did not know about before.

I came across the ‘Situational Leadership’ concept while reading the book. The situational leadership theory is a leadership theory originally developed by Paul Hersey, professor and author of the book ‘Situational Leader’, and Ken Blanchard, leadership guru and author of ‘The One Minute Manager’.

What Situational Leadership means is that there is no single best style of leadership. The leader needs to change his/her style of leadership according to the current needs of the individual or team he/she is leading.

The book talked about how we can use ‘Situational Leadership® II’ to help our teams become highly productive teams. Leaders need to provide varying combination of directive and supportive behaviours, according to the development stage the team is in. Different teams in different stages were described along with their leaders to help relate the theory better.

I also came across the concept of PERFORM. High performing teams show the characteristics defined under the acronym of PERFORM (There is a nice blog post regarding this here). The leader’s work is to help the team achieve these characteristics.

Perform Venn diagramTo be an effective group leader, or member, one needs to become an effective observer and a participant at the same time. While observing a group’s process, you need to be observing

– Communication & participation: who talks to whom? who is left out? who talks most often?
– Decision making: how the group goes about making a decision
– Conflict: how is conflict handled?
– Leadership: who is influencing whom?
– Goals and roles: is the team clear about these?
– Group norms: which norms (rules governing group’s behaviour) are most obvious in the group?
– Problem solving: how does the group solve problems?
– Climate/tone: feeling or tone of the group – how pleasant it seems.

Once you get the hang of being a participant observer, it will be easy to diagnose the group’s functioning and development stage. Accordingly, the team leader adjusts the leadership style, and works towards empowering the team.

The four stages through which a group goes through are –

Stage 1: Orientation (Leadership style = Directing)
Stage 2: Dissatisfaction (Leadership style = Coaching)
Stage 3: Resolution (Leadership style = Supporting)
Stage 4: Production (Leadership style = Delegating)

A team leader’s most important function is to help the group move through the stages of development.

More notes from the book –

1. When in doubt about a group’s stage, start with a directive style

2. Try to move quickly from directive to coaching style (but in a gradual manner – step by step), and begin to encourage members to share their ideas and opinions.

3. You will never, never, never have an empowered, self-directed team unless the manager is willing to share control

4. Your job as a manager is to help people and teams develop so they have competence and commitment and the ability to share in making decisions.

5. Groups may regress to a previous stage – when groups gain, lose, or change members, when task changes or if a major event occurs which disrupts group functioning. (move back one leadership style at a time until you solve the problem)

6. The words “Manager” and “Educator” are synonymous.

Why I am signing the petition to stop ISO 29119


There is a movement being started to stop ISO 29119. If you are even a wee bit serious about your profession, you should be knowing about it. Here’s all about it. Read it if you want to know more, or just skip to the end and sign the petitions.

James Christie gave a presentation in CAST 2014 on Standards – promoting quality or restricting competition? He talked about how some organizations are trying to enforce testing standards for their own economic gain, while victimizing the software testing profession. He also stated how the standards promote excessive documentation over actual testing, thus affecting the quality of testing by taking away time from the process. At a time, when we are trying to implement Agile, and moving towards exploratory testing, heavy documentation and rigid processes make no sense. And, to force it down the throats of everyone in this profession should not be tolerated.

As Fiona Charles puts it “People on Agile projects will struggle with the conflicting demands of their projects and the standards.”

There is a reason why I have not taken the certification. Why should I pay so much money for something that won’t add any value to my work, except a bunch of words I might never need to use? I know this for a fact that being certified does not say anything about the tester’s expertise or potential. Most current certifications, for a fact, can be easily passed by rote learning. But, would that help? Rote learning never helps. How much do you remember of the history lessons from your high school? You did pass the exam.

I have known many testers who are certified, but do not have a clue about testing, and are low performers. I have also known many testers, who are awesome at their work, have high potential, are high performers, and are not certified. That is why I ignore the certification status of people I interview, and instead focus on finding out myself about how much they are interested in testing, and how well they know their craft.

This is what Karen Johnson says about wrong hiring based on blind belief in certifications:
“For several years when I was working as a test manager, I interviewed and hired numerous people. Once during an interview, a candidate pulled out a certificate and held it up for me, telling me he should be hired instantly due to his certification. I asked that he put the certificate away and talk with me. After discussion, it was my assessment that the candidate had little working understanding of how to test a product. Perhaps he had memorized material or had someone else take the exam for him – whatever had been the case; there was no evidence that the candidate would be able to perform well. As this event took place some years ago, I will refrain from trying to recall more specifics.
Given this same scenario, if I did not have a testing background but needed to hire someone, perhaps I would have been convinced based on the certificate that the candidate was equipped for the job.

If a company is hiring testers and the person or persons interviewing do not understand testing (which is often the case with HR), I do not believe that hiring person is qualified to make a decision – or even qualified to establish a pool of candidates for others to interview. This is where certification is dangerous – at first glance, it sounds good, it seems like it should provide some assurance that a person is qualified. But as I have found, having a certificate does not provide evidence of a person’s knowledge or skill.”

If you read my blog, you will know I am not against learning. In fact, I am very passionate about learning. I keep reading books and blogs to learn more about my profession. But, I want to learn out of curiosity and interest, not out of fear. That is what this standard tries to instill – the fear of losing your job or validity as a software tester if you have not paid the bucks to get certified by rote.

If the standard is accepted, companies will start believing that testers without a certification are not good, which is far from the truth. They will start over-depending on tester certifications to judge the quality of a tester.

Iain McCowatt says “Standards in manufacturing make sense: the variability between two different widgets of the same type should be minimal, so acting in the same way each time a widget is produced is desirable. This does not apply to services, where demand is highly variable, or indeed in software, where every instance of demand is unique.

Attempting to act in a standardized manner in the face of variable demand is an act of insanity: it’s akin to being asked to solve a number of different problems yet merrily reciting the same answer over and over. Sometimes you’ll be right, sometimes wrong, sometimes you’ll score a partial hit. In this way, applying the processes and techniques of ISO 29119 will result in effort being expended on activities that do nothing to aid the cause of testing.”

Karen also explains this in detail in her blog where she says that the knowledge required by an e-commerce software tester, a BI software tester, and a Medical device tester are vastly different, and can not be covered by a generic testing certification. The testers would instead do better to learn more about their domain, and to get domain-specific knowledge.

ISO 29119 claims that it is “an internationally-agreed set of standards for software testing”. But, it is not “internationally-agreed”. In fact, many influential software testers, and the experts that I look up to, are against it.

Karen Johnson – My Thoughts on Testing Certifications
Fiona Charles – Why I oppose adoption of ISO 29119
Iain McCowatt – Stop 29119
Keith Klain – The Petition to Stop ISO 29119
Michael Bolton – Rising Against the Rent-Seekers
James Bach – How not to standardize software testing

If you agree with the above view point, and are interested in saving our profession from vandalism, please sign the below shared documents:

The Professional Tester’s Manifesto

The Petition to ISO

Other articles of note –
ISO 29119 Debate
On ISO 29119 content

Book Summary: Don’t make me think by Steve Krug


I just finished reading this book. This is a very good book on usability. Everyone who is interested in improving their website, or in being a success in the Web domain, should read this.

I wanted to write a summary for myself for the times when I am not having this book with me and want to recap the main points discussed, or for the time when I am in a hurry and just want a summary to go through before starting my work. So, here it goes..
(Please read the book for detailed information and explanations. It will be totally worth it. Don’t expect to know everything there is in the book by reading the highlights below.)

1. Usability is mostly just common sense, but not necessarily obvious until after someone has pointed it out to you.

2. First law of Usability: “Don’t make me think”
* Get rid of uncommon names/text
* links/buttons should be obviously clickable
* Pages should be self evident, etc.

3. How we use the web
* we don’t read pages, we scan them
* We don’t make optimal choices. We satisfice (choose first reasonable option)
* we don’t figure out how things work. We muddle through

4. Design pages for scanning, not reading
* Clear visual hierarchy
* Conventions are your friends
* Break pages into clearly defined areas
* Make it obvious what’s clickable
* Keep noise down to a dull roar

5. It doesn’t matter how many times i have to click, as long as each click is a mindless, unambiguous choice.

6. Get rid of half the words on each page, then get rid of half of what’s left

7. Web navigation.
Following elements should be presented as per conventions:
* Site ID
* Sections
* Utilities
* Subsections
* “You are here” indicator
* Page name
* Local navigation (Things at current level)
* Small text version (links in footer)

8. Web navigation.
* Persistent navigation including 5 elements – Site ID, “home” link, Search option, Utilities, Sections

9. To check if a page is well designed – Choose a page, keep at arms length , and answer following questions as quickly as possible
* What site is this? (Site ID)
* What page am I on? (Page name)
* What are major sections of this site? (Sections)
* What are my options at this level? (Local navigation)
* Where am I in the scheme of things? (Bread crumbs)
* How can I search?

10. Home page should answer the following 4 questions, at a glance, correctly and unambiguously –
* What is this?
* What do they have here?
* What can I do here?
* Why should I be here – and not somewhere else?
* Where do I start?

11. Add a good tagline to make it clearer

12. Web design arguments within team are a waste of time. Instead, conduct usability tests by calling outside people for opinion

13. Ideal number of users for each round of testing is three, at most four

14. How to conduct usability testing – two kinds of testing – “get it” testing, “key task” testing

15. For doing usability testing by yourself, find more advice in chapter “Usability testing: The movie” that is in the first edition of this book

16. Web site should be behaving like a Mensch. Each problem user encounters on the site lowers the user’s level of goodwill reservoir

17. Things that diminish good will
* Hiding information that I want
* Punishing me for not doing things your way
* Asking me for information you don’t really need
* Faux sincerity, and disingenuous attempts to convince me that you care about me
* Putting sizzle in my way – Flash intros, animations, etc
* You site looks amateurish

18. Things that increase goodwill
* Know the main things that people want to do on your site and make them obvious and easy
* Tell me what I want to know
* Save me steps wherever you can
* Put effort into it
* Know what questions I am likely to have, and answer them
* Provide me with creature comforts like printer friendly pages
* Make it easy to recover from errors
* When in doubt, apologize

19. Accessibility is part of Usability
5 things you can do right now:
* Fix usability problems that confuse everyone
* Read an article. Highly recommended article – “Guidelines for accessible and usable Web sites: Observing users who work with screen readers” by Mary Theofanos and Janice Redish
* Read a book
* Start using cascading style sheets
* Make sure you have implemented the most common things
– Add appropriate alt text to images
– Make your forms readable with screen readers. Use labels for fields
– Create a “skip to main content” link at the beginning of the page
– Make all content accessible by keyboard
– Don’t use java script without a good reason
– Use client-side (not server-side) image maps

Issue in auto suggestions when multiple words of the same record start similarly


similar multiple words in a record

This one adds to my list of test cases for auto suggestion functionality. It really depends on the implementation whether this test applies or not, but it is still good to add to your check list.

In our case, auto suggestions are displayed using keywords mapped to the record. There is one keyword for each word in a name.

Example: “New Delhi” should show up as a suggestion when either “New” or “Delhi” is typed in.
So, there are two records in database – one with “New” as a keyword, and one with “Delhi” as a keyword.

But, what happens when words in a single name start similarly.

For example, there is a city named “Tarn Taran”. Both words in the name start with “Tar”.
When I type in “Tar”, both keywords “Tarn” and “Taran” will get matched, and if it’s not handled in the code, Auto-suggest list will display two records for “Tarn Taran”.

Improving tester-programmer relationship


testing tipsHere is something to read for today. I found this book, written by Ajay Balamurugadas, while browsing through Google results, and thought I should share it here too.

It has a lot of tips that will help you become a better tester (but only if you implement what you read🙂 ).

Books on testing

 

Edit: Removed book’s specific link. Instead, I am sharing the download page for all 6 books written by the author