Code structure for automated checks


I have been reading about how to structure my automation scripts for checking
scenarios, and this is what I have learnt.
 
The Test Pyramid
(Source: Test Pyramid)
First of all, we should know the test pyramid concept that was developed by Mike Cohn, and described in his book Succeeding with Agile.
In a nutshell, the concept is about having many more low-level unit tests than high-level end-to-end UI tests. UI tests are brittle, expensive to write, and time consuming to run. So, the automation should lean much more towards unit tests.
 test-pyramid
UI layer refers to the end-to-end checks using something like Selenium or Sahi. The Service layer refers to checks that act through a service layer of an application. (Eg. checking through an API layer)
And the final base of checks is the low-level unit tests.
 
Guidelines for structuring automated tests
#1 Structure – Prioritize creation of unit tests to cover most test scenarios over
any other types of tests. Unit tests should cover the bulk of your testing strategy.
Keep UI tests to a minimum, only covering critical happy flow scenarios & failures. Avoid making assertion on validation error message in UI level. It should just verify the error case by virtue of redirect to same page. All validation scenarios are already covered with messages in unit tests
#2 Speed – Team should collectively decide the acceptable time for running the test suite, define it for all test types – unit, integration and acceptance.
#3 Robustness – Avoid testing JavaScript behavior and scenarios heavily using
Selenium based acceptance tests. Instead, use Jasmine-based pure JavaScript unit testing, which are more robust and fast.
Create mocks for any external services that tests might be dependent on.
#4 Debugging – When it fails, it should point broken code.
 
Self Testing Code
Every object should be able to test itself. Write your code in such a way that typing a command should make the whole software system do a self-test.
Also, practice Test Driven Development while working.
This command will be run every time code is committed and pushed to Git develop branch. By running the test suite frequently, at least several times a day, you’re able to detect such bugs soon after they are introduced, so you can just look in the recent changes, which makes it much easier to find them.
Build a testing culture where developers are naturally thinking about writing code and tests together.
 
When Should a Test Be Automated?
(Source: Automate)
Do not over-automate. Only some tests should be automated.
Decision process would use following questions:
1. If I automate this test, what manual tests will I lose (not be able to execute in the time given)? How many bugs might I lose with them? What will be their severity?
2. An automated test has a finite lifetime, during which it must recoup that additional cost. Is this test likely to die sooner or later? What events are likely to end it?
When deciding whether to automate a test, you must estimate how many code changes it will survive. When the test breaks after code change, if it hasn’t repaid the automation effort by that point, you would have been better off leaving it as a
manual test
Create product-specific test libraries. After that many user interface changes will
require no changes to tests, only to the test library, thus, saving a lot of effort.
3. During its lifetime, how likely is this test to find additional bugs (beyond whatever bugs it found the first time it ran)? How does this uncertain benefit balance against the cost of automation?
If those questions don’t suffice for a decision, other minor considerations might tip the balance.
 
Note – The tests you need are often called task-driven tests, use-case tests, or scenario tests. Because scenario tests favor common tasks, they find the bugs most users will find.
Some of the secondary considerations
* Humans can notice bugs that automation ignores, especially GUI issues.
* But, while humans are good at noticing oddities, they’re bad at painstaking or precise checking of results.
* As humans can’t be precise about inputs, repeated runs of a manual test are often slightly different tests, which might lead to discovery of a support code bug.
For example, people make mistakes, back out, and retry inputs, thus sometimes stumbling across interactions between error-handling code and the code under test.
* Because test automation takes time, you often won’t report the first bugs back to the programmer as soon as you could in manual testing.
* Automated tests, if written well, can be run in sequence, and the ordering can vary from day to day
* An automated test might not pay for itself until next release. A manual test will find any bugs it finds this release. Bugs found now might be worth more than bugs found next release.
 
Page objects
Within your web app’s UI there are areas that your tests interact with. A Page Object simply models these as objects within the test code. This reduces the amount of duplicated code and means that if the UI changes, the fix need only be applied in one place.
Some other articles I read that could be useful –

‘Appium’ – presentation video from GTAC 2013


I came across this post with a really cool video on Appium (an open-source mobile app automation test tool that integrates with Selenium).

Xnovo

Hey,

Guess what, GTAC (Google Test Automation Conference) which held its place on April 23-24 in NY this year – had a Live Stream.

I’ve managed to watch some and some of them even recorded 🙂 However guys said that they will upload videos later in May – I’ve decided to share some with you now.

gtac2013

View original post 52 more words

Aside

Test Automation – Selenium webdriver with python


This is a long pending post. I had done the installation months back, and then forgotten the steps. I needed to re-install to be able to write down the below steps, and was not getting the time. Now, finally, I have done an installation again, and this time I am noting down the steps.

The below method is very simple. Contrary to what most of the Google search results have you believe, it is not necessary to install easy_install or pip to install the selenium python bindings.

So, first I will just introduce you to Selenium WebDriver (part of Selenium 2). It is a library for automating your test cases across browsers, and it can be used from a variety of language bindings. I prefer Python because I love the language and its no-nonsense format. So, I will be installing the Python language bindings along with the Selenium Framework.

The Selenium webdriver allows you to programmatically drive a browser and interact with web elements. It uses the browser’s native methods instead of javascript injection, and should be preferred over Selenium RC

Now, let us go through the installation steps:

1. Install Python in your machine

– Go to http://python.org/download/
– Download the latest MSI installer for Python 2.7 (32-bit)
– Run the installer

2. Set your PATH
– Go to your system’s environment variables, and add ‘C:\Python27’ to your PATH

(this will enable you to invoke ‘python.exe’ from any command line)

3. Go to https://pypi.python.org/pypi/selenium and download the .tar.gz file (the Selenium Python Client Driver)

4. Extract files from the .tar.gz file

5. Go through the extracted files. You will find a selenium-2.xx.x.tar file in the dist folder. (x denotes version number in the file name)

Extract files from this .tar file. Verify that the extracted folder contains a setup.py file

6.  Open the cmd Command prompt, and navigate to the location where setup.py is located (the folder from step 6):

 >> cd <location of the setup.py file for selenium bindings>

7. Enter the following command into the command prompt:

 >> python setup.py install

A “selenium” directory gets added in the C:\Python27\Lib folder.

Selenium is now installed in your machine with Python bindings.

The following code should work now:

#!/usr/bin/env python

from selenium import webdriver

browser = webdriver.Firefox()
browser.get("https://amybughunter.wordpress.com/")

This should open a Firefox browser session and navigate to https://amybughunter.wordpress.com/

A very useful doc to understand the Selenium Webdriver API for python is http://selenium-python.readthedocs.org/en/latest/api.html. It has helped me a lot while creating test scripts.

Following is a simple functional test in Python, using Selenium WebDriver and the unittest framework:

#!/usr/bin/env python

import unittest
from selenium import webdriver

class TestUdacityHomepage(unittest.TestCase):

    def setUp(self):
        self.browser = webdriver.Firefox()

    def testTitle(self):
        browser = self.browser
        browser.get('https://www.udacity.com/')
        title = browser.title
        self.assertIn('Udacity', title)

    def tearDown(self):
        self.browser.quit()

if __name__ == '__main__':
     unittest.main()

Output should be:

>>> 
.
----------------------------------------------------------------------
Ran 1 test in 15.478s

OK
Exit code:  False

A simple way to migrate Redmine?


Redmine_login_pageFor migration (or DB update) of my Redmine instance, I have performed the steps from my previous article twice by now.

But, this time, I spent 2 days of annoyance by never-before-seen errors from the rake command, while performing the exact same steps. That’s when I gave up trying to research the millions of errors on Google, and decided to adopt a new way.

This time, I performed the below given steps:

1. I installed the exact same version of the Bitnami Stack that I had earlier (with the Installation directory also having the same name and path as before),

2. Stopped the Stack Services of the current instance through the shortcut.

3. Then, I replaced the contents (all the folders & the files) of the entire installation directory with the ones in my previous Stack directory (I had the whole installation directory of my previous stack saved as backup)

4. Executed the following commands –

cd <installdir>

serviceinstall.bat INSTALL

5. Restarted the Stack services through shortcut

And that was it! It all seemed to work fine.

According to the article in Bitnami’s blog, this seems to be a valid way to do it. But, I’m still not sure why any one would perform all the other steps if it were that simple.

What do you think?

Updating Redmine data when you have no backup.sql file


In reference to my previous post in the blog (https://amybughunter.wordpress.com/2013/01/22/migrating-your-redmine-instance-to-another-machine/), I am writing this post to describe how I went about updating the database of my new Redmine instance without the backup file.

I had just copied the whole old Redmine folder as an error occurred while creating the database backup file.
Now, to update the new Redmine instance, you need an SQL backup file as detailed in my previous post. And I only had my old MySQL folder at hand.
I could not find much online for this scenario, and I hope this post will, someday, help someone who is lost about how to go about it.
So, this is how I went about it.

1. Here, I will assume you have copied the whole of your old Bitnami Redmine Stack to a safe location in your machine.
That is, you have directly performed steps 3 and 4 from the previous post, without performing steps 1, and 2.
2. Install a New Bitnami Redmine Stack as detailed in step 5 of previous post.
3. Replace the MySQL folder of this Stack with that of the Old one.
4. Now, you can create the database backup file “redmine_backup.sql” as detailed in the previous post.
5. Perform all the steps (from previous post) from the beginning for the migration of the Stack. The files, plugins will be taken from the old Stack’s backup that you saved in the step 1 here.

Migrating your Redmine instance to another machine


The following will be very useful information if you ever need to migrate your Redmine instance (along with existing data) to a different machine.

Some very useful links are:

This article is mostly a summary of the above mentioned articles, with some additional steps that I had to perform.

1. First you need to create backup of your current instance

  • Redmine services should be in started state
  • Verify that <bitnami>\mysql\my.ini has correct  user/password/port in [mysqladmin] group and that, if using non-standard port, it is set in the [mysqldump] group with a port=# setting. (Password should be same as that defined for admin user)
  • Open “use_redmine” console, and run the following commands:
1. scripts\setenv.bat
2. cd mysql\bin
3. mysqldump -u root -p bitnami_redmine > redmine_backup.sql

You will be asked to enter a password. Enter the password from my.ini (The same password that is for the admin user)

  • The database backup file ‘redmine_backup.sql’ will get created in the mysql\bin folder. Copy it into another location as the database  backup.

2. Also, keep a copy of all the files from <Installdir>\apps\redmine\files. This folder contains all the files that have been uploaded to Redmine by User as attachments.

In case you had installed any plugins, copy their folders from <Installdir>\apps\redmine\vendor\plugins location.

3. Stop and uninstall previous Bitnami Redmine Stack services. Then, Uninstall the entire existing Bitnami Redmine stack

4. As a Backup, copy/move the <Installdir> to another location, and remove from the previous location (to avoid confusion in future). I deleted the whole folder from my C:\Program Files location ( after copying it to a folder named “Redmine Backup”)

5. Install the new Bitnami Redmine. You need to install the new Bitnami Redmine stack in NEW location (i.e. You are not supposed to overwrite the previous instance. An entirely new folder should get created).

As I had already removed files of previous instance from the ‘Program Files’ location, I did not need to worry about this much. But, still I changed the name of the instance by removing the spaces in between. So, the new instance got installed in “C:\Program Files\BitnamiRedmine”

Other details you have to take care of in this step are:

  1. Download the new Redmine Stack for your platform.
  2. Keep username/password same as that for the admin user in previous instance. These will be used when restoring database.
  3. Do not enter email address, otherwise you may get “Email has already been taken” if you attempt to change account settings of a Redmine user that has same email address!
  4. Setup email settings now or later; old settings are in <old_bitnami>\apps\redmine\config\email.yml
  5. Allow redmine to start
  6. Confirm Redmine works (visit http://localhost:<apache_server_port>/redmine). You will get your <apache_server_port> value from the properties.ini file in the root folder.

6. Make sure you have downloaded all gems required for the plugins you want to install.

Then, install the gems in the machine using “gem install” command.

If internet is not available, install the required gems locally by first downloading them from http://rubygems.org/gems, and then running the following command from inside the folder that contains the .gem files

           gem install --local <gem_name>

7. Open a NEW “use_redmine” console for the NEW redmine and run the following commands (Points 7 to 13) (Make sure the stack services are in a started state at this point):

  1. scripts\setenv.bat
  2. cd mysql\bin

8. Create proper database schema to receive old database.

cmd> mysql -u root -p
Enter password: ********   <password will be same as that for admin user>
Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 10
Server version: 5.0.83-community-log MySQL Community Edition (GPL)
Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.
mysql> show grants;
+---------------------------------------------------------------------------------------------------------------+
| Grants for root@localhost                                |
+---------------------------------------------------------------------------------------------------------------+
| GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' IDENTIFIED BY PASSWORD '426f44267cd6981a' WITH GRANT OPTION |
+---------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
mysql> drop database bitnami_redmine;
Query OK, 47 rows affected (0.28 sec)
mysql> create database bitnami_redmine;
Query OK, 1 row affected (0.00 sec)
mysql> GRANT ALL PRIVILEGES ON bitnami_redmine.* TO 'root'@'localhost' IDENTIFIED 
BY PASSWORD '<see below note to know what to enter here as BITNAMI_USER_PASSWORD>';
Query OK, 0 rows affected (0.00 sec)
mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)
mysql> exit;
Bye

Note – 1) The BITNAMI_USER_PASSWORD should match the String after ‘IDENTIFIED BY PASSWORD’ when the command ‘show grants’ is run in the previous step.
2) Some useful in below link regarding mysql user creation:
http://stackoverflow.com/questions/9529651/mysql-user-creation-script
9. Restore old Database into new

mysql -u root -p bitnami_redmine < redmine_backup.sql

If the redmine_backup file is not available in the ‘mysql\bin’ folder where you are executing the query, the absolute address of the file should be specified (in double quotes) instead of just the file name. Otherwise, you can just give the name of the file (as shown above), and the command will work

10. Merge settings from old (C:\Program Files\BitNami Redmine Stack\apps\redmine\config) into new (C:\Program Files\BitNamiRedmine\apps\redmine\config). This involves three files:

  • settings.yml: I found that no lines had changed; only new lines added so left the new version untouched.
  • database.yml: left the new version untouched; note: the username is bitnami, rather than the one you selected at installation time; the password is randomly generated by installation tool, it is not a hash.
  • email.yml: if you didn’t setup at installation time, copy file from old stack.

11. Copy the plugins you wish installed into the <Bitnami>\apps\redmine\vendor\plugins location.

12. Migrate the data imported to the new schema:
$ cd <installdir>\apps\redmine\vendor\plugins
$ rake db:migrate RAILS_ENV=production
$ rake db:migrate:upgrade_plugin_migrations RAILS_ENV=production
$ rake db:migrate_plugins RAILS_ENV=production

13. Clear caches:

$ rake tmp:cache:clear
$ rake tmp:sessions:clear

14. Copy all files from previous backup of <Installdir>\apps\redmine\files to the new <Installdir>\apps\redmine\files folder

15. Start new stack

16. Verify that your New Redmine is working fine, and that the data looks ok : issues, wiki, attachments, projects hierarchy, users

Once again, I would like to thank the author of http://schollii2.wordpress.com. His article was a great help to me, and most of the steps I have written above are lines from his article.

I hope my article will help someone…Took me hours to gather all this information.

Defect Tracking tools


When I was deciding upon which open-source defect tracking tool to use in our company, I mainly looked into Bugzilla, Asana, and Redmine.

Asana was already being used by the development team for task management. But, till I joined, there hadn’t been much of a separate process for logging bugs. Whatever issues the developers found were logged as ‘tasks to be done’ in Asana.

Till then there was not much of a ‘bombarding’ of identified issues, because there wasn’t a testing team till then.

Asana is very good as a task management tool, but it lacked some features I consider important for a defect tracking tool – like the consolidated issue lists/summary, and reports.

So, I went through the demo versions for both Bugzilla and Redmine.

Bugzilla is a good tool, but I think Redmine, being a cross-over of Bugzilla and JIRA, is a better choice.

Below is a comparison chart I made to compare the tools.

(Backgroud=Green  implies the preferred choice according to the feature discussed in the row)

Redmine vs Asana vs Bugzilla

Comparison chart – Redmine vs Asana vs Bugzilla

So, finally I chose Redmine. And I am really glad. Installation is very easy if you choose to install it through Bitnami Redmine Stack. The stack does all the configuration and installation for you.

It makes it so easy to go through all raised issues, and to extract reports according to whatever parameters I want.

For example, Whenever a new tester joins, I extract the latest list of all the issues ever raised for a particular project from Redmine and ask him/her to go through all of them. This would help him/her understand the product better, in addition to preventing the logging of duplicate bugs in the future.

And if I want to see only those issues raised by a particular tester in my team, I can always go to the Issue search list and add a parameter “Author = <author_name>” in the search criteria.

Also, there are many Redmine plugins available online to customize your Redmine instance however you want it.

I would go ahead and recommend Redmine to everyone who’s deciding on a defect tracking tool. It will surely make your life easier.