Testing Strategy


dilbert_strategy

Here’s the testing strategy given by G.J. Myers in “The Art of Software Testing – 2nd Edition”.

This is a very good strategy and worth following at all times.

1. If the specification contains combinations of input conditions, start with cause-effect graphing.

2. In any event, use boundary-value analysis.

The boundary-value analysis yields a set of supplemental test conditions, but, many or all of these can be incorporated into the cause-effect tests.

3. Identify the valid and invalid equivalence classes for the input and output, and supplement the test cases identified above if necessary.

4. Use the error-guessing technique to add additional test cases.

5. Examine the program’s logic with regard to the set of test cases.

Use the decision-coverage, condition-coverage, decision/condition-coverage, or multiple-condition-coverage criterion (the last being the most complete).

If the coverage criterion has not been met by the test cases identified in the prior four steps, and if meeting the criterion is not impossible (i.e., certain combinations of conditions may be impossible to create because of the nature of the program), add sufficient test cases to cause the criterion to be satisfied.

Black box testing with Cause-Effect Graphs


cause-effect-graph-nodesI’ve been reading “The Art of Software Testing – 2nd Edition” by G.J. Myers. The book introduced me to a concept called Cause-Effect Graphs.

This provoked me to research further about it, and to think how I could implement it into my work.

Below is the summary of what I learnt. Unfortunately, I can not share an example of how I implemented it into my work due to concerns of confidentiality. But I’ll share links with other examples for further understanding of this concept.

Cause-Effect Graphing (CEG) is basically a Black-box testing technique that is used to create test cases according to the specifications provided. (—It is a type of Requirements-Based Testing, also known as Dependency modelling)

CEG can not be used in all scenarios. It can only be used in cases where the test output depends on a combination of test inputs.

For eg, you’re showing your users a set of results based on which one of the 2 options – A or B – they chose while requesting data. The type of output (results) here depend only on one test input (that could either be option A or option B). So, CEG is not possible here .

But, if the results depend on a combination of the selected option (A or B), the current date, the user’s nationality, and the user’s language, CEG could be applied because the output depends upon a combination of these inputs.

What CEG does is that it helps capture the relationships between specific combinations of inputs (causes) and outputs (effects).

—The causes and effects are represented as the  nodes of the graph.
I’m listing down a few advantages of CEG below:
  • —Unlike Boundary Value Analysis and Equivalence Partitioning, Cause-Effect Graphing takes into account input combinations
  • —Helps in selecting, in a systematic way, a set of high-yield test cases out of an astronomical set of input combinations.
  • —As a beneficial side-effect, it also points out incompleteness and ambiguities in the specifications.

Notations used – Nodes

Nodes can have either of 2 states:            0 (absent) or 1 (present)

cause-effect-graph-symbols

Notations used – Constraints

Constraints define the relationship between a set of inputs

  • —E => c1 & c2 can not be 1 simultaneouslycause-effect-graph-constraints
  • —I => c1, c2, c3 can not be 0 simultaneously
  • —O => One, and only one,

of c1 & c2 must be 1

  • —R => For c1 to be 1, c2

must be 1

Notations used – “Masks” Constraint

If effect e1 is 1, effect e2 is forced to 0 cause-effect-graph-masks-constraint

Now, that I have introduced you to the building blocks of CEG, let’s go through the method:

  1. Divide the specifications into workable pieces. Trying to work with the whole specs all at once would make the process very, very complicated.
  2. Identify cause & effects in the specifications.Cause = A distinct input condition or equivalence class of input conditionsEffect = An output condition or a system transformation
  3. Analyze the semantic content of the specification & transform it into a Boolean graph linking the cause and effects   (Cause-Effect Graph)
  4. Annotate the graph with constraints describing combinations of causes and/or effects that are impossible due to syntactic/environmental constraints
  5. Convert graph into a limited-entry decision table by methodically tracing state conditions in the graph
      1. Select an effect to be the present state (=1)
      2. From the graph, find all combinations of causes (subject to constraints) that will set this effect to 1.
      3. Create a column in the decision table for each combination of causes.
      4. For each combination determine the states of all other effects and place these in each column
    considerations used when tracing cause effect graph
    CEG trace 1
    CEG trace 2
    The final table will look something like this:
    CEG table
  6. Each column in graph represents a test case. So, the last step would be to derive test cases from the table by converting each column to a separate test case.

To keep the number of test cases to a minimal set of high-yield test cases we can assimilate Boundary-value and Equivalence class in the test cases just derived.

I will also list the testing strategy given in the book when i write my next post.

Below are some links with examples of how CEG can be used to create tests. I hope this post was informative for you.

Chapter on CEG from VTU Learning

Good Postal Regulation example

Importance of Boundary Value analysis while testing in Database


A few days back, we found a critical issue in production.

The issue that occurred was as follows:

We were given a functionality to test for uploading images for a selected car. There was only a Browse button, and an Upload button in the page.

We had to test if the image name is getting saved in the format

(make year-car make-car model-car version-unique ID.jpg)

We tested it for a number of different cars and functionality was working fine.

But, we missed out one thing.

There was a stored procedure created for saving these image URLs to the database (it was not shared with us. We normally do black box testing.)

Database column being used to save was of size 100 characters, but the stored procedure was storing image URLs in a variable of length 50 characters

Now, the Image names and URLs were getting saved properly for most cars.

But, for cars where Image URLs got greater than 50 chars in length, data got truncated while saving in the database. And this resulted in the images not getting displayed in production.

I was left baffled at first about how I could have caught this bug before release, without having had a look at the SP. How could I have found that variable size was less than what it was supposed to be?

Then it struck me. This is how i am going to work on preventing such defects in the future-

When given a page to test, I will also list out the DB tables that are affected.

For all the columns affected, I will note the size of the column and try to insert values that tend towards the maximum length allowed in the column.

If I had tried this earlier, by selecting longer car names in my effort to reach a 100 char length, I would have found that the maximum length of a possible input into the column was around 76. And in that case I would have found the bug.

I urge you to put this test case as a top priority test case as data truncation is a serious issue. The final goal is to confirm that both the below given conditions are always true:

1. The maximum length of data that a user can input <= The size of the variable in stored procedure that is used to store the value

2.  The size of the variable in stored procedure <= The size of the related Database column