Code structure for automated checks


I have been reading about how to structure my automation scripts for checking
scenarios, and this is what I have learnt.
 
The Test Pyramid
(Source: Test Pyramid)
First of all, we should know the test pyramid concept that was developed by Mike Cohn, and described in his book Succeeding with Agile.
In a nutshell, the concept is about having many more low-level unit tests than high-level end-to-end UI tests. UI tests are brittle, expensive to write, and time consuming to run. So, the automation should lean much more towards unit tests.
 test-pyramid
UI layer refers to the end-to-end checks using something like Selenium or Sahi. The Service layer refers to checks that act through a service layer of an application. (Eg. checking through an API layer)
And the final base of checks is the low-level unit tests.
 
Guidelines for structuring automated tests
#1 Structure – Prioritize creation of unit tests to cover most test scenarios over
any other types of tests. Unit tests should cover the bulk of your testing strategy.
Keep UI tests to a minimum, only covering critical happy flow scenarios & failures. Avoid making assertion on validation error message in UI level. It should just verify the error case by virtue of redirect to same page. All validation scenarios are already covered with messages in unit tests
#2 Speed – Team should collectively decide the acceptable time for running the test suite, define it for all test types – unit, integration and acceptance.
#3 Robustness – Avoid testing JavaScript behavior and scenarios heavily using
Selenium based acceptance tests. Instead, use Jasmine-based pure JavaScript unit testing, which are more robust and fast.
Create mocks for any external services that tests might be dependent on.
#4 Debugging – When it fails, it should point broken code.
 
Self Testing Code
Every object should be able to test itself. Write your code in such a way that typing a command should make the whole software system do a self-test.
Also, practice Test Driven Development while working.
This command will be run every time code is committed and pushed to Git develop branch. By running the test suite frequently, at least several times a day, you’re able to detect such bugs soon after they are introduced, so you can just look in the recent changes, which makes it much easier to find them.
Build a testing culture where developers are naturally thinking about writing code and tests together.
 
When Should a Test Be Automated?
(Source: Automate)
Do not over-automate. Only some tests should be automated.
Decision process would use following questions:
1. If I automate this test, what manual tests will I lose (not be able to execute in the time given)? How many bugs might I lose with them? What will be their severity?
2. An automated test has a finite lifetime, during which it must recoup that additional cost. Is this test likely to die sooner or later? What events are likely to end it?
When deciding whether to automate a test, you must estimate how many code changes it will survive. When the test breaks after code change, if it hasn’t repaid the automation effort by that point, you would have been better off leaving it as a
manual test
Create product-specific test libraries. After that many user interface changes will
require no changes to tests, only to the test library, thus, saving a lot of effort.
3. During its lifetime, how likely is this test to find additional bugs (beyond whatever bugs it found the first time it ran)? How does this uncertain benefit balance against the cost of automation?
If those questions don’t suffice for a decision, other minor considerations might tip the balance.
 
Note – The tests you need are often called task-driven tests, use-case tests, or scenario tests. Because scenario tests favor common tasks, they find the bugs most users will find.
Some of the secondary considerations
* Humans can notice bugs that automation ignores, especially GUI issues.
* But, while humans are good at noticing oddities, they’re bad at painstaking or precise checking of results.
* As humans can’t be precise about inputs, repeated runs of a manual test are often slightly different tests, which might lead to discovery of a support code bug.
For example, people make mistakes, back out, and retry inputs, thus sometimes stumbling across interactions between error-handling code and the code under test.
* Because test automation takes time, you often won’t report the first bugs back to the programmer as soon as you could in manual testing.
* Automated tests, if written well, can be run in sequence, and the ordering can vary from day to day
* An automated test might not pay for itself until next release. A manual test will find any bugs it finds this release. Bugs found now might be worth more than bugs found next release.
 
Page objects
Within your web app’s UI there are areas that your tests interact with. A Page Object simply models these as objects within the test code. This reduces the amount of duplicated code and means that if the UI changes, the fix need only be applied in one place.
Some other articles I read that could be useful –
Advertisements

Check broken links and handle multiple windows through Selenium webdriver in Python


webdriver - handle multiple windows

Today I am going to post the code of a function I wrote to check broken links in a web page.

You can learn two things by going through it –

First, how to check broken links (I know that’s obvious :D).
Second, how to handle multiple windows in selenium webdriver + python. There are three useful functions for this –

1. webdriver.window_handles 
 => returns the handles of all windows within the 
    current webdriver session
2. webdriver.current_window_handle
 => returns the handle of the current window 
     (the window currently in focus)
3. webdriver.switch_to_window(window_name) 
 => switches focus to the window having specified 
    window_name or window_handle 
    (we can pass the window_handle instead of window_name
        as a parameter to this function)

To check broken links, my code navigates to the link given, and checks whether user lands up in the same page as expected.
While checking links on a page, I also find some links that, when clicked, open a page in a separate window. Most common example of such links are the links to social media sites on a page.

Eg. In the below code, I am testing the home page of “http://www.carwale.com/“.
This page has 4 links – the icons at the bottom of the page for facebook, youtube, google plus, and twitter – that when clicked, open the respective pages in a new window. Here, the section that handles multiple windows will come into play.

from selenium import webdriver
browser = webdriver.Firefox()
home_page = "http://www.carwale.com/"

def check_page_broken_links(self, url):
# Sample usage:
#     check_page_broken_links(self,"http://www.carwale.com/") 
#         will return empty list if all links in the page work fine
#         else it will return list of all the broken links 
#                                    (either link text, or link href)
#   Will check for -  i) "Page Not found" error
#                     ii) Redirects

    try:
        failed = []
        self.implicitly_wait(5)
        self.get(url)
        number_of_links = len(self.find_elements_by_tag_name('a'))

        for i in range(number_of_links):
            # Save current browser window handle
            initial_window = self.current_window_handle 
            ## print "initial_window_handle:      ", initial_window

            link = self.find_elements_by_tag_name('a')[i]
            link_address = link.get_attribute("href")
            link_name = link.text
            print "link checked: ",i,": ",link_name,": ",link_address

            if ((link_address is not None) 
                and ("google" not in link_address) 
                and ("mailto" not in link_address) 
                and is_link_element_displayed(self,element=link) is True):
                  link.click() # link clicked
                  open_windows = self.window_handles
                  ## print "window_handles:      ", open_windows

                  # Navigate to the browser window where 
                  #               latest page was opened
                  self.switch_to_window(open_windows[-1])
                  ## print "current_window_handle:"
                  ## print self.current_window_handle
                  time.sleep(5)
                  print "defined: ",link_address
                  print "current: ", self.current_url

                  if (link_address[-1] == "#" 
                       and self.current_url in 
                        [link_address, link_address[:-1],
                                      link_address[:-2],link_address+'/']):
                          # A "#" at the end means user 
                          # will stay on the same page.(Valid scenario) 
                          pass  
                  elif (self.current_url not in 
                        [link_address,
                         home_page + link_address[1:]]): 
                        # if user lands up in a page different 
                        #                    from that intended 
                        if link_name:
                          failed.append(link_name)
                        else:
                          failed.append(link_address)

                  if len(self.window_handles) > 1:  
                          # close newly opened window
                          self.close()

                  # Switch to main browser window
                  self.switch_to_window(open_windows[0]) 

            self.get(url)
    except Exception, e: 
           return ['Exception occurred while checking',e]
    return failed

# call defined function to check broken links in carwale.com home page
print check_page_broken_links(browser,"http://www.carwale.com/")

If there are any broken links in the URL you passed to the function, they will be printed out in a list at the end of program execution.

‘Appium’ – presentation video from GTAC 2013


I came across this post with a really cool video on Appium (an open-source mobile app automation test tool that integrates with Selenium).

Xnovo

Hey,

Guess what, GTAC (Google Test Automation Conference) which held its place on April 23-24 in NY this year – had a Live Stream.

I’ve managed to watch some and some of them even recorded 🙂 However guys said that they will upload videos later in May – I’ve decided to share some with you now.

gtac2013

View original post 52 more words

Aside

Test Automation – Selenium webdriver with python


This is a long pending post. I had done the installation months back, and then forgotten the steps. I needed to re-install to be able to write down the below steps, and was not getting the time. Now, finally, I have done an installation again, and this time I am noting down the steps.

The below method is very simple. Contrary to what most of the Google search results have you believe, it is not necessary to install easy_install or pip to install the selenium python bindings.

So, first I will just introduce you to Selenium WebDriver (part of Selenium 2). It is a library for automating your test cases across browsers, and it can be used from a variety of language bindings. I prefer Python because I love the language and its no-nonsense format. So, I will be installing the Python language bindings along with the Selenium Framework.

The Selenium webdriver allows you to programmatically drive a browser and interact with web elements. It uses the browser’s native methods instead of javascript injection, and should be preferred over Selenium RC

Now, let us go through the installation steps:

1. Install Python in your machine

– Go to http://python.org/download/
– Download the latest MSI installer for Python 2.7 (32-bit)
– Run the installer

2. Set your PATH
– Go to your system’s environment variables, and add ‘C:\Python27’ to your PATH

(this will enable you to invoke ‘python.exe’ from any command line)

3. Go to https://pypi.python.org/pypi/selenium and download the .tar.gz file (the Selenium Python Client Driver)

4. Extract files from the .tar.gz file

5. Go through the extracted files. You will find a selenium-2.xx.x.tar file in the dist folder. (x denotes version number in the file name)

Extract files from this .tar file. Verify that the extracted folder contains a setup.py file

6.  Open the cmd Command prompt, and navigate to the location where setup.py is located (the folder from step 6):

 >> cd <location of the setup.py file for selenium bindings>

7. Enter the following command into the command prompt:

 >> python setup.py install

A “selenium” directory gets added in the C:\Python27\Lib folder.

Selenium is now installed in your machine with Python bindings.

The following code should work now:

#!/usr/bin/env python

from selenium import webdriver

browser = webdriver.Firefox()
browser.get("https://amybughunter.wordpress.com/")

This should open a Firefox browser session and navigate to https://amybughunter.wordpress.com/

A very useful doc to understand the Selenium Webdriver API for python is http://selenium-python.readthedocs.org/en/latest/api.html. It has helped me a lot while creating test scripts.

Following is a simple functional test in Python, using Selenium WebDriver and the unittest framework:

#!/usr/bin/env python

import unittest
from selenium import webdriver

class TestUdacityHomepage(unittest.TestCase):

    def setUp(self):
        self.browser = webdriver.Firefox()

    def testTitle(self):
        browser = self.browser
        browser.get('https://www.udacity.com/')
        title = browser.title
        self.assertIn('Udacity', title)

    def tearDown(self):
        self.browser.quit()

if __name__ == '__main__':
     unittest.main()

Output should be:

>>> 
.
----------------------------------------------------------------------
Ran 1 test in 15.478s

OK
Exit code:  False