Load Testing

How to Load Test your Application 

Load testing is a kind of performance testing which determines a system's performance under real-life load conditions. This testing helps determine how the application behaves when multiple users access it simultaneously.

This testing usually identifies - The maximum operating capacity of an application Determine whether current infrastructure is sufficient to run the application Sustainability of application with respect to peak user load Number of concurrent users that an application can support, and scalability to allow more users to access it. It is a type of non-functional testing. Load testing is commonly used for the Client/Server, Web based applications - both Intranet and Internet.

Need of Load Testing:

Some extremely popular sites have suffered serious downtimes when they get massive traffic volumes. E-commerce websites invest heavily in advertising campaigns, but not in Load Testing to ensure optimal system performance, when that marketing brings in traffic.
Consider the following examples

• Popular toy store Toysrus.com, could not handle the increased traffic generated by their advertising campaign resulting in loss of both marketing dollars, and potential toy sales.
• An Airline website was not able to handle 10000+ users during a festival offer.
• Encyclopedia Britannica declared free access to their online database as a promotional offer. They were not able to keep up with the onslaught of traffic for weeks.

Many sites suffer delayed load times when they encounter heavy traffic. Few Facts - Most users  click away after 8 seconds delay in loading a page $ 4.4 Billion Lost annually  due to poor performance

 

Why Load Testing ?

• Load testing gives confidence in the system & its reliability and performance.
• Load Testing helps identify the bottlenecks in the system under heavy user stress scenarios before they happen in a production environment.
• Load testing gives excellent protection against poor performance and accommodates complementary strategies for performance management and monitoring of a production environment.

 

Goals of Load Testing:

Loading testing identifies the following problems before moving the application to market or Production:

• Response time for each transaction
• Performance of System components under various loads
• Performance of  Database components under different loads
• Network delay between the client and the server
• Software design issues
• Server configuration issues like Web server, application server, database server etc.
• Hardware limitation issues like CPU maximization, memory limitations, network bottleneck, etc.

Load testing will determine whether system needs to be fine-tuned or modification of hardware and software is required to improve performance.

Environment needs to be setup before starting the load testing:

Hardware Platform	Software Configuration
Server Machines Processors Memory Disk Storage Load Machines configuration Network configuration	Operating System Server Software

 

Prerequisites of load testing:

The chief metric for load testing is response time. Before you begin load testing , you must determine -

• Whether the response time is already measured and compared - Quantitative
• Whether the response time is applicable to the business process - Relevant
• Whether the response time is justifiable - Realistic
• Whether the response time is achievable - Achievable
• Whether the response time is measurable using a tool or stopwatch - Measurable

 

Strategies of load Testing:

There are many numbers of ways to perform load testing. Following are a few load testing strategies-

Manual Load Testing: This is one of strategies to execute load testing, but it does not produce repeatable results, cannot provide measurable levels of stress on an application and is an impossible process to coordinate. In house developed load testing tools: An organization, which realizes the importance of load testing, may build their own tools to execute load tests. Open source load testing tools: There are several load testing tools available as open source that are free of charge. They may not be as sophisticated as their paid counterparts, but if you are on a budget, they are the best choice. Enterprise class load testing tools: They usually come with capture/playback facility. They support a large number of protocols. They can simulate an exceptionally large number of users.

 

Load Testing Process:

The load testing process can be briefly described as below -

1. Create a dedicated test environment for load testing
2. Determine the following
3. Load Test Scenarios
4. Determine load testing transactions for an application
   • Prepare Data for each transaction
   • Number of Users accessing the system need to be predicted
   • Determine connection speeds. Some users may be connected via leased lines while others may use dial-up
   • Determine different browsers and operating systems used by the users 
   • Configuration of all the servers like web, application and DB Servers
5. Test Scenario execution and monitoring . Collecting various metrics
6. Analyze the results. Make recommendations
7. Fin-tune the System
8. Re-test

 

Guidelines for load testing:

Load testing should be planned once the application becomes functionally stable. Large number of unique data should be ready in the data pool Number of users should be decided for each scenario or scripts Avoid creation of detailed logs to conserve the disk IO space Try to avoid downloading of images in the site Consistency of response time over elapsed period should be logged & the same should be compared with various test runs

 

Difference between Load and Stress testing:

Load Testing	Stress Testing
Load testing identifies the bottlenecks  in the system under various workloads and checks how the system reacts when the load is gradually increased	Stress testing determines the breaking point of the system to reveal the maximum point after which it breaks.

 

Difference between Functional and Load Testing:

Functional Testing	Load Testing
Results of functional tests are easily predictable as we have proper steps and preconditions defined	Results of load tests are unpredictable
Results of functional tests vary slightly	Load test results vary drastically
Frequency of executing functional testing will be high	Frequency of executing load testing will be low
Results of functional tests are dependent on the test data	Load testing depends on the number of users.

 

Load Testing Tools:

Tools recommended for load testing are: Loadrunner Web Load Astra Load Test Radview's Web Load Studio, Rational Site Load Silk Performer Loadrunner and Web Load are the popular tools. Their features are listed below -

Load Runner:

Load runner is HP tool  used to test the applications under normal and peak load conditions. Load runner generates load by creating virtual users that emulate network traffic. It  simulates real time usage like a production environment and gives graphical results.

Web load:

Web load supports HTTP1.0 and 1.1, including cookies, proxies, SSL, TSL, client certificates, authentications, persistent connections and chunked transfer coding. During execution, Webload gathers results at a per-client, per-transaction and per-instance level from the computers that are generating the load.

 

Advantages and disadvantages of Load testing:

Following are the advantages of Load testing:

• Performance bottlenecks identification before production
• Improves the scalability of the system
• Minimize  risk related to system down time
• Reduced costs of failure
• Increase customer satisfaction

Disadvantages of Load testing:

• Need programming knowledge to use load testing tools.
• Tools can be expensive as pricing depends on the number of virtual users supported.

 

Conclusion:

Load testing typically improves performance bottlenecks, scalability and stability of the application before it is available for production. This testing helps to identify the maximum operating capacity of applications as well as system bottlenecks.

 

Adhoc Testing

Adhoc Testing

Adhoc testing is an informal testing type with an aim to break the system. This testing is usually an unplanned activity . It does not follow any test design techniques to create test cases. In fact is does not create test cases altogether! This testing is primarily performed if the knowledge of testers in the system under test is very high. Testers randomly test the application without any test cases or any business requirement document.
 

 

Adhoc Testing does not follow any structured way of testing and it is randomly done on any part of application. Main aim of this testing is to find defects by random checking. Adhoc testing can be achieved with the testing technique called Error Guessing. Error guessing can be done by the people having enough experience on the system to "guess" the most likely source of errors.

This testing requires no documentation/ planning /process to be followed. Since this testing aims at finding defects through random approach, without any documentation, defects will not be mapped to test cases. Hence, sometimes, it is very difficult to reproduce the defects as there are no test-steps or requirements mapped to it.

When execute Adhoc Testing?

Adhoc testing can be performed when there is limited time to do elaborative testing. Usually adhoc testing is performed after the formal test execution. And if time permits, adhoc testing can be done on the system).Adhoc testing will be effective only if the tester is knowledgeable of the System Under Test.

Types of adhoc testing

There are different types of Adhoc testing and they are listed as below:

Buddy Testing	Two buddies mutually work on identifying defects in the same module. Mostly one buddy will be from development team and another person will be from testing team. Buddy testing helps the testers develop better test cases and development team can also make design changes early. This testing usually happens after unit testing completion.
Pair testing	Two testers are assigned modules, share ideas and work on the same machines to find defects. One person can execute the tests and another person can take notes on the findings. Roles of the persons can be a tester and scriber during testing. Comparison Buddy and Pair Testing Buddy testing is combination of unit and system testing together with developers and testers but Pair testing is done only with the testers with different knowledge levels.(Experienced and non-experienced to share their ideas and views)
Monkey Testing	Randomly test the product or application without test cases with a goal to break the system.

Best practices of Adhoc testing

Following best pratices can ensure effective Adhoc Testing.

 

Good business knowledge

Testers should have good knowledge of the business and clear understanding of the requirements- Detailed knowledge of the end to end business process will help find defects easily. Experienced testers find more defects as they are better at error guessing.

Test Key Modules

Key business modules should be identified and targeted for adhoc testing.. Business critical modules should be tested first to gain confidence on the quality of the system.

Record Defects

All defects need to be recorded or written in a notepad . Defects must be assigned to developers for fixing. For each valid defect ,corresponding test cases must be written & must be added to planned test cases.
These defect findings should be made as lesson learned and these should be reflected in our next system while we are planning for test cases.

Conclusion:

The advantage of Adhoc testing is to check for the completeness of testing and find more defects then  planned testing. The defect catching test cases are added as additional test cases to the planned test cases.
Adhoc Testing saves lot of time as it doesn't require elaborate test planning , documentation and test case design.

 

Agile Methodology

Agile Model and Methodologies: Guide for Developers and Testers

To understand the concept of agile testing, first let's understand-

 

What is Agile?

AGILE is a methodology that promotes continuous iteration of development and testing throughout the software development life cycle of the project. Both development and testing activities are concurrent unlike the Waterfall model
I hope we got an idea of Agile!!! Now, we can step on to Agile Testing.

The agile software development emphasizes on four core values.

1. Individual and team interactions over processes and tools
2. Working software over comprehensive documentation
3. Customer collaboration over contract negotiation
4. Responding to change over following a plan

 

Agile versus Waterfall Method

Agile and Waterfall model are two different methods for software development process. Though they are different in their approach, both methods are useful at times, depending on the requirement and the type of the project.

Agile Model	Waterfall Model
Agile method proposes incremental and iterative approach to software design	Development of the software flows sequentially from start point to end point.
The agile process is broken into individual models that designers work on	The design process is not broken into an individual models
The customer has early and frequent opportunities to look at the product and make decision and changes to the project	The customer can only see the product at the end of the project
Agile model is considered unstructured compared to the waterfall model	Waterfall model are more secure because they are so plan oriented
Small projects can be implemented very quickly. For large projects, it is difficult to estimate the development time.	All sorts of project can be estimated and completed.
Error can be fixed in the middle of the project.	Only at the end, the whole product is tested. If the requirement error is found or any changes have to be made, the project has to start from the beginning
Development process is iterative, and the project is executed in short (2-4) weeks iterations. Planning is very less.	The development process is phased, and the phase is much bigger than iteration. Every phase ends with the detailed description of the next phase.
Documentation attends less priority than software development	Documentation is a top priority and can even use for training staff and upgrade the software with another team
Every iteration has its own testing phase. It allows implementing regression testing every time new functions or logic are released.	Only after the development phase, the testing phase is executed because separate parts are not fully functional.
In agile testing when an iteration end, shippable features of the product is delivered to the customer. New features are usable right after shipment. It is useful when you have good contact with customers.	All features developed are delivered at once after the long implementation phase.
Testers and developers work together	Testers work separately from developers
At the end of every sprint, user acceptance is performed	User acceptance is performed at the end of the project.
It requires close communication with developers and together analyse requirements and planning	Developer does not involve in requirement and planning process. Usually, time delays between tests and coding

 

Methodologies of Agile Testing

There are various methods present in agile testing, and those are listed below:

 

Scrum

SCRUM is an agile development method which concentrates specifically on how to manage tasks within a team based development environment. Basically, Scrum is derived from activity that occurs during a rugby match. Scrum believes in empowering the development team and advocates working in small teams (say- 7 to 9 members). It consists of three roles, and their responsibilities are explained as follows:

• Scrum Master
  • Master is responsible for setting up the team, sprint meeting and removes obstacles to progress
• Product owner
  • The Product Owner creates product backlog, prioritizes the backlog and is responsible for the delivery of the functionality at each iteration
• Scrum Team
  • Team manages its own work and organizes the work to complete the sprint or cycle

 

Product Backlog

This is a repository where requirements are tracked with details on the no of requirements to be completed for each release. It should be maintained and prioritized by product owner, and it should be distributed to the scrum team. Team can also request for a new requirement addition or modification or deletion

 

Scrum Practices

Practices are described in detailed:

 

Process flow of Scrum:

Process flow of scrum testing is as follows:

• Each iteration of a scrum is known as Sprint
• Product backlog is a list where all details are entered to get end product
• During each Sprint, top items of Product backlog are selected and turned into Sprint backlog
• Team works on the defined sprint backlog
• Team checks for the daily work
• At the end of the sprint, team delivers product functionality

 

eXtreme Programming (XP)

Extreme Programming technique is very helpful when there is constantly changing demands or requirements from the customers or when they are not sure about the functionality of the system. It advocates frequent "releases" of the product in short development cycles, which inherently improves the productivity of the system and also introduces a checkpoint where any customer requirements can be easily implemented. The XP develops software keeping customer in the target.

Business requirements are gathered in terms of stories. All those stories are stored in a place called the parking lot.
In this type of methodology, releases are based on the shorter cycles called Iterations with span of 14 days time period. Each iteration includes phases like coding, unit testing and system testing where at each phase some minor or major functionality will be built in the application.

 

Phases of eXtreme programming:

There are 6 phases available in Agile XP method, and those are explained as follows:

Planning

• Identification of stakeholders and sponsors
• Infrastructure Requirements
• Security related information and gathering
• Service Level Agreements and its conditions

Analysis

• Capturing of Stories in Parking lot
• Prioritize stories in Parking lot
• Scrubbing of stories for estimation
• Define Iteration SPAN(Time)
• Resource planning for both Development and QA teams

Design

• Break down of tasks
• Test Scenario preparation for each task
• Regression Automation Framework

Execution

• Coding
• Unit Testing
• Execution of Manual test scenarios
• Defect Report generation
• Conversion of Manual to Automation regression test cases
• Mid Iteration review
• End of Iteration review

Wrapping

• Small Releases
• Regression Testing
• Demos and reviews
• Develop new stories based on the need
• Process Improvements based on end of iteration review comments

Closure

• Pilot Launch
• Training
• Production Launch
• SLA Guarantee assurance
• Review SOA strategy
• Production Support

There are two storyboards available to track the work on a daily basis, and those are listed below for reference.

• Story Cardboard
  • This is a traditional way of collecting all the stories in a board in the form of stick notes to track daily XP activities. As this manual activity involves more effort and time, it is better to switch to an online form.
• Online Storyboard
  • Online tool Storyboard can be used to store the stories. Several teams can use it for different purposes.

 

Crystal Methodologies

Crystal Methodology is based on three concepts

1. Chartering: Various activities involved in this phase are creating a development team, performing a preliminary feasibility analysis, developing an initial plan and fine-tuning the development methodology
2. Cyclic delivery: The main development phase consists of two or more delivery cycles, during which the
   1. Team updates and refines the release plan
   2. Implements a subset of the requirements through one or more program test integrate iterations
   3. Integrated product is delivered to real users
   4. Review of the project plan and adopted development methodology
3. Wrap Up: The activities performed in this phase are deployment into the user environment, post- deployment reviews and reflections are performed.

 

Dynamic Software Development Method (DSDM)

DSDM is a Rapid Application Development (RAD) approach to software development and provides an agile project delivery framework. The important aspect of DSDM is that the users are required to be involved actively, and the teams are given the power to make decisions. Frequent delivery of product becomes the active focus with DSDM. The techniques used in DSDM are

1. Time Boxing
2. MoSCoW Rules
3. Prototyping

The DSDM project consists of 7 phases

1. Pre-project
2. Feasibility Study
3. Business Study
4. Functional Model Iteration
5. Design and build Iteration
6. Implementation
7. Post-project

 

Feature Driven Development (FDD)

This method is focused around "designing & building" features. Unlike other agile methods, FDD describes very specific and short phases of work that has to be accomplished separately per feature. It includes domain walkthrough, design inspection, promote to build, code inspection and design. FDD develops product keeping following things in the target

1. Domain object Modeling
2. Development by feature
3. Component/ Class Ownership
4. Feature Teams
5. Inspections
6. Configuration Management
7. Regular Builds
8. Visibility of progress and results

 

Lean Software Development

Lean software development method is based on the principle "Just in time production". It aims at increasing speed of software development and decreasing cost. Lean development can be summarized in seven steps.

1. Eliminating Waste
2. Amplifying learning
3. Defer commitment (deciding as late as possible)
4. Early delivery
5. Empowering the team
6. Building Integrity
7. Optimize the whole

 

Kanban

Kanban originally emerged from Japanese word that means, a card containing all the information needed> to be done on the product at each stage along its path to completion. This framework or method is quite adopted in software testing method especially in agile testing.

Difference between Scrum and Kanban

Scrum	Kanban
In scrum technique, test must be broken down so that they can be completed within one sprint	No particular item size is prescribed
Prescribes a prioritized product backlog	Prioritization is optional
Scrum team commits to a particular amount of work for the iteration	Commitment is optional
Burndown chart is prescribed	No particular item size is prescribed
Between each sprint, a scrum board is reset	A Kanban board is persistent. It limits the number of items in workflow state
It cannot add items to ongoing iteration	It can add items whenever capacity is available
WIP limited indirectly	WIP limited directly
Timeboxed iterations prescribed	Timeboxed iterations optional

 

Agile metrics:

Metrics that can be collected for effective usage of Agile is:

• Drag Factor
  • Effort in hours which do not contribute to sprint goal
  • Drag factor can be improved by reducing number of shared resources, reducing the amount of non-contributing work
  • New estimates can be increased by percentage of drag factor -New estimate = (Old estimate+drag factor)
• Velocity
  • Amount of backlog converted to shippable functionality of sprint
• No of Unit Tests added
• Time taken to complete daily build
• Bugs detected in an iteration or in previous iterations
• Production defect leakage

Unit Testing

Unit Testing: Full Guide

Unit testing of software applications is done during the development (coding) of an application.
The objective of unit testing is to isolate a section of code and verify its correctness. In procedural programming a unit may be an individual function or procedure
The goal of unit testing is to isolate each part of the program and show that the individual parts are correct. Unit testing is usually performed by the developer.

Why do Unit Testing? Why it is important?

Sometimes software developers attempt to save time by doing minimal unit testing. This is a myth because skimping on unit testing leads to higher defect fixing costs during system testing, integration testing and even beta testing after the application is completed. Proper unit testing done during the development stage saves both time and money in the end.

Building unit Test Cases

Unit testing is commonly automated, but may still be performed manually. The IEEE does not favor one over the other. A manual approach to unit testing may employ a step-by-step instructional document.
Under the automated approach-

• A developer could write another section of code in the application just to test the function. They would later comment out and finally remove the test code when the application is done.
• They could also isolate the function to test it more rigorously. This is a more thorough unit testing practice that involves copy and pasting the function to its own testing environment to other than its natural environment. Isolating the code helps in revealing unnecessary dependencies between the code being tested and other units or data spaces in the product. These dependencies can then be eliminated.

 
A coder may use a Unit Test Framework to develop automated test cases. Using an automation framework, the developer codes criteria into the test to verify the correctness of the unit. During execution of the test cases, the framework logs those that fail any criterion. Many frameworks will also automatically flag and report in a summary these failed test cases. Depending upon the severity of a failure, the framework may halt subsequent testing.
 

 Mock Objects

Unit testing relies on mock objects being created to test sections of code that are not yet part of a complete application. Mock objects fill in for the missing parts of the program. For example, you might have a function that needs variables or objects that are not created yet. In unit testing, those will be accounted for in the form of mock objects created solely for the purpose of the unit testing done on that section of code.
 

Unit Testing Tools

There are several automated tools available to assist with unit testing. We will provide a few examples below:

• Rational Software - Rational Software by IBM has a unit test feature known as "Rational Test Realtime". The software contains a complete range of testing tools for much more than just unit testing. It is used for Ada, Java, C and C++. It creates unit tests by reverse engineering the software.  Operating systems it supports include Windows, Linux, Solaris, HP-UX and AIX. Go to http://www-01.ibm.com/software/rational/ to learn more.
• JavaScript Assertion Unit- Also known as jsAsserUnit, this Freeware JavaScript unit testing tool can be used on any platform that supports JavaScript. It is available at http://jsassertunit.sourceforge.net/docs/index.html
• CUT - CUT is a Freeware unit test tool for C, C++ and Objective C. It is great for embedded software testing frameworks and desktop applications on Linux and Windows operating systems. Learn more at sourceforge.net by going to http://sourceforge.net/projects/cut/.
• Dotunit - Dotunit is a .net framework Freeware unit testing tool. Part of Junit on the Microsoft .net framework, Dotunit is used for automating unit testing on windows systems. This is another tool from sourceforge.net, so look for it at: http://dotunit.sourceforge.net/

 
Those are just a few of the available unit testing tools. There are lots more, especially for C languages and Java, but you are sure to find a unit testing tool for your programming needs regardless of the language you use.

Extreme Programming & Unit Testing

Unit testing in Extreme Programming involves the extensive use of testing frameworks. A unit test framework is used in order to create automated unit tests. Unit testing frameworks are not unique to extreme programming, but they are essential to it. Below we look at some of what extreme programming brings to the world of unit testing:

• Rely heavily on testing frameworks
• Tests are written before the code
• All classes in the applications are tested
• Quick and easy integration is made possible

 

 Unit Testing Myth

It requires time and I am always overscheduled

My code is rock solid! I do not need unit tests.

Myths by their very nature are false assumptions.  These assumptions lead to a vicious cycle as follows -
 
 
Truth is

Unit testing increase the speed of development.

Programmers think that integration testing will catch all errors and do not unit test. Once units are integrated, very simple errors which could have very easily found and fixed in unit tested take very long time to be traced and fixed.
 

Unit Testing benefits

• Developers looking to learn what functionality is provided by a unit and how to use it can look at the unit tests to gain a basic understanding of the unit API.
• Unit testing allows the programmer to refactor code at a later date, and make sure the module still works correctly (i.e. Regression testing). The procedure is to write test cases for all functions and methods so that whenever a change causes a fault, it can be quickly identified and fixed.
• Due to the modular nature of the unit testing, we can tests parts of project without waiting for others to be completed.

 

Unit Testing Limitations

• Unit testing can't be expected to catch every error in a program. It is not possible to evaluate all execution paths even in the most trivial programs
• Unit testing by its very nature focuses on a unit of code. Hence it can't catch integration errors or broad system level errors.

 
It's recommended unit testing be used in conjunction with other testing activities.
 

Unit testing best practices

• Unit Test cases should be independent. In case of any enhancements or change in requirements, unit test cases should not be affected.
• Test only one code at a time.
• Follow clear and consistent naming conventions for your unit tests
• In case of change in code in any module, ensure there is a corresponding unit test case for the module and the module passes the tests before changing the implementation
• Bugs identified during unit testing must be fixed before proceeding to the next phase in SDLC
• Adopt a "test as your code" approach. The more code you write without testing the more paths you have to check for errors.

 Summary
As you can see, there can be a lot involved in unit testing. It can be complex or rather simple depending on the application being tested and the testing strategies, tools and philosophies used. Unit testing is always necessary on some level. That is a certainty.

Automation Testing

Why Automated Testing?

  Automated testing is important due to following reasons:

• Manual Testing of all work flows, all fields , all negative scenarios is time and cost consuming
• It is difficult to test for multi lingual sites manually
• Automation does not require Human intervention. You can run automated test unattended (overnight)
• Automation increases  speed of test execution
• Automation helps increase  Test Coverage
• Manual Testing can become boring and hence error prone.

 

Which Test Cases to Automate?

Test cases to be automated can be selected using the following criterion to increase the automation ROI

• High Risk - Business Critical test cases
• Test cases that are executed repeatedly
• Test Cases that are very tedious or difficult to perform manually
• Test Cases which are time consuming

The following category of test cases are not suitable for automation:

• Test Cases that are newly designed and not executed manually  atleast once
• Test Cases for which the requirements are changing frequently
• Test cases which are executed on ad-hoc basis.

Automation Process

Following steps are followed in an Automation Process

 

Test tool selection

Test Tool selection largely depends on the technology the Application Under Test is built on. For instance QTP does not support Informatica.  So QTP cannot be used for testing Informatica applications. It's a good idea to conduct Proof of Concept of Tool on AUT

Define the scope of Automation

Scope of automation is the area of your Application Under Test which will be automated. Following points help determine scope:

• Feature that are important for the business
• Scenarios which have large amount of data
• Common functionalities across applications
• Technical feasibility
• Extent to which business components are reused
• Complexity of test cases
• Ability to use the same test cases for cross browser testing

 

Planning, Design and Development 

During this phase you create  Automation strategy & plan, which contains following details-

• Automation tools selected
• Framework design and its features
• In-Scope and Out-of-scope items of automation
• Automation test bed preparation
• Schedule and Timeline of scripting and execution
• Deliverables of automation testing

 

Test Execution

Automation Scripts are executed during this phase. The scripts need input test data before there are set to run. Once executed they provide detailed test reports.
Execution can be performed using the automation tool directly or through the Test Management tool which will invoke the automation tool.
Example: Quality center is the Test Management tool which in turn it will invoke QTP for execution of automation scripts. Scripts can be executed in a single machine or a group of  machines. The execution can be done during night , to save time.

Maintenance:

As new functionalities are added to the System Under Test with successive cycles, Automation Scripts need to be added, reviewed and maintained for each release cycle. Maintenance becomes necessary to improve effectiveness of Automation Scripts. 
Automation tools
Following are the most popular test tools :


 QTP : HP's  Quick Test Professional ( now known as HP Functional Test) is the market leader in Functional Testing Tool. The tool supports plethora of environments including SAP , Java , Delphi amongst others. QTP can be used  in conjunction with Quality Center which is a comprehensive Test  Management Tool.  know  is light tool which can be recommended for web or client/server applications.
Rational Robot: It  is an IBM tool used to automate regression, functional and configuration tests for client server, e-commerce as well as  ERP applications. It can be used with Rational Test Manager which aided in
 Test Management Activities
Selenium: Its an open source Web Automation Tool. It supports all types of web browsers. Despite being open source its actively developed and supported

How to Choose an Automation Tool? 

Selecting the right tool can be a tricky task. Following criterion will help you select the best tool for your requirement-

• Environment Support
• Ease of use
• Testing of Database
• Object identification
• Image Testing
• Error Recovery Testing
• Object Mapping
• Scripting Language Used
• Support for various types of test - including functional, test management, mobile, etc...
• Support for multiple testing frameworks
• Easy to debug the automation software scripts
• Ability to recognize objects in any environment
• Extensive test reports and results
• Minimize training cost of selected tools 

Tool selection is one of biggest challenges to be tackled before going for automation. First, Identify the requirements, explore various tools and its capabilities, set the expectation from the tool and go for a Proof Of Concept.
Framework in Automation 
A framework is  set of automation guidelines  which help in

• Maintaining consistency of Testing
• Improves test structuring
• Minimum usage of code
• Less Maintenance of code
• Improve re-usability
• Non Technical testers can be involved in code
• Training period of using the tool can be reduced
• Involves Data wherever appropriate 

There are four types of framework used in software automation testing:

1. Data Driven Automation Framework
2. Keyword Driven Automation Framework
3. Modular Automation Framework
4. Hybrid Automation Framework

 

Automation Best Practices: 

To get maximum ROI of automation, observe the following

• Scope of Automation needs to be determined in detail before the start of the project. This sets expectations from Automation right.
• Select the right automation tool: A tool must not be selected based on its popularity but it's fit to the automation requirements.
• Choose appropriate framework
• Scripting Standards- Standards have to be followed while writing the scripts for Automation .Some of them are-
  • Create uniform scripts, comments and indentation of the code
  • Adequate Exception handling - How error is handled on system  failure or unexpected behavior of the application.
  • User defined messages should be coded or standardized for Error Logging for testers to understand.
• Measure metrics- Success of automation cannot be determined by   comparing the manual effort with the automation effort but by also capturing the following metrics.
  • Percent of defects found
  • Time required for automation testing for each and every release cycle
  • Minimal Time taken for release   
  • Customer satisfaction Index
  • Productivity improvement

The above guidelines if observed can greatly help in making your automation successful.

 

Benefits of automated testing

 Following are benefits of automated testing:

70% faster than the manual testing Wider test coverage of application features Reliable in results Ensure Consistency Saves Time and Cost Improves accuracy Human Intervention is not required while execution Increases Efficiency Better speed in executing tests Re-usable test scripts Test Frequently and thoroughly More  cycle of execution can be achieved through automation  Early time to market

 Conclusion
Right selection of automation tool, testing process and team, are important players for automation to be successful. Manual and automation methods go hand-in hand for successful testing.

Softwere Testing

>> Overview

Testing is the process of evaluating a system or its component(s) with the intent to find whether it satisfies the specified requirements or not. In simple words, testing is executing a system in order to identify any gaps, errors, or missing requirements in contrary to the actual requirements.
According to ANSI/IEEE 1059 standard, Testing can be defined as - A process of analyzing a software item to detect the differences between existing and required conditions (that is defects/errors/bugs) and to evaluate the features of the software item.

Who does Testing?

It depends on the process and the associated stakeholders of the project(s). In the IT industry, large companies have a team with responsibilities to evaluate the developed software in context of the given requirements. Moreover, developers also conduct testing which is called Unit Testing. In most cases, the following professionals are involved in testing a system within their respective capacities:

• Software Tester
• Software Developer
• Project Lead/Manager
• End User

Different companies have different designations for people who test the software on the basis of their experience and knowledge such as Software Tester, Software Quality Assurance Engineer, QA Analyst, etc.

It is not possible to test the software at any time during its cycle. The next two sections state when testing should be started and when to end it during the SDLC.

 

When to Start Testing?

An early start to testing reduces the cost and time to rework and produce error-free software that is delivered to the client. However in Software Development Life Cycle (SDLC), testing can be started from the Requirements Gathering phase and continued till the deployment of the software. It also depends on the development model that is being used. For example, in the Waterfall model, formal testing is conducted in the testing phase; but in the incremental model, testing is performed at the end of every increment/iteration and the whole application is tested at the end.

Testing is done in different forms at every phase of SDLC:

• During the requirement gathering phase, the analysis and verification of requirements are also considered as testing.
• Reviewing the design in the design phase with the intent to improve the design is also considered as testing.
• Testing performed by a developer on completion of the code is also categorized as testing.

 

When to Stop Testing?

It is difficult to determine when to stop testing, as testing is a never-ending process and no one can claim that a software is 100% tested. The following aspects are to be considered for stopping the testing process:

• Testing Deadlines
• Completion of test case execution
• Completion of functional and code coverage to a certain point
• Bug rate falls below a certain level and no high-priority bugs are identified
• Management decision

   

>> Types Of Testing

Manual Testing:

Manual testing includes testing a software manually, i.e., without using any automated tool or any script. In this type, the tester takes over the role of an end-user and tests the software to identify any unexpected behavior or bug. There are different stages for manual testing such as unit testing, integration testing, system testing, and user acceptance testing.

Testers use test plans, test cases, or test scenarios to test a software to ensure the completeness of testing. Manual testing also includes exploratory testing, as testers explore the software to identify errors in it.

 

Automation Testing:

Automation testing, which is also known as Test Automation, is when the tester writes scripts and uses another software to test the product. This process involves automation of a manual process. Automation Testing is used to re-run the test scenarios that were performed manually, quickly, and repeatedly.

Apart from regression testing, automation testing is also used to test the application from load, performance, and stress point of view. It increases the test coverage, improves accuracy, and saves time and money in comparison to manual testing.

 

What is Automate?

It is not possible to automate everything in a software. The areas at which a user can make transactions such as the login form or registration forms, any area where large number of users can access the software simultaneously should be automated.

Furthermore, all GUI items, connections with databases, field validations, etc. can be efficiently tested by automating the manual process.

 

When to Automate?

Test Automation should be used by considering the following aspects of a software:

• Large and critical projects
• Projects that require testing the same areas frequently
• Requirements not changing frequently
• Accessing the application for load and performance with many virtual users
• Stable software with respect to manual testing
• Availability of time

 

How to Automate?

Automation is done by using a supportive computer language like VB scripting and an automated software application. There are many tools available that can be used to write automation scripts. Before mentioning the tools, let us identify the process that can be used to automate the testing process:

• Identifying areas within a software for automation
• Selection of appropriate tool for test automation
• Writing test scripts
• Development of test suits
• Execution of scripts
• Create result reports
• Identify any potential bug or performance issues

 

Software Testing Tools:

The following tools can be used for automation testing:

• HP Quick Test Professional
• Selenium
• IBM Rational Functional Tester
• SilkTest
• TestComplete
• Testing Anywhere
• WinRunner
• LaodRunner
• Visual Studio Test Professional
• WATIR

		>> Methods Of Testing Black-Box Testing: The technique of testing without having any knowledge of the interior workings of the application is called black-box testing. The tester is oblivious to the system architecture and does not have access to the source code. Typically, while performing a black-box test, a tester will interact with the system's user interface by providing inputs and examining outputs without knowing how and where the inputs are worked upon. The following table lists the advantages and disadvantages of black-box testing. Advantages Disadvantages Well suited and efficient for large code segments. Code access is not required. Clearly separates user's perspective from the developer's perspective through visibly defined roles. Large numbers of moderately skilled testers can test the application with no knowledge of implementation, programming language, or operating systems. Limited coverage, since only a selected number of test scenarios is actually performed. Inefficient testing, due to the fact that the tester only has limited knowledge about an application. Blind coverage, since the tester cannot target specific code segments or error-prone areas. The test cases are difficult to design. White-Box Testing: White-box testing is the detailed investigation of internal logic and structure of the code. White-box testing is also called glass testing or open-box testing. In order to perform white-box testing on an application, a tester needs to know the internal workings of the code. The tester needs to have a look inside the source code and find out which unit/chunk of the code is behaving inappropriately. The following table lists the advantages and disadvantages of white-box testing. Advantages Disadvantages As the tester has knowledge of the source code, it becomes very easy to find out which type of data can help in testing the application effectively. It helps in optimizing the code. Extra lines of code can be removed which can bring in hidden defects. Due to the tester's knowledge about the code, maximum coverage is attained during test scenario writing. Due to the fact that a skilled tester is needed to perform white-box testing, the costs are increased. Sometimes it is impossible to look into every nook and corner to find out hidden errors that may create problems, as many paths will go untested. It is difficult to maintain white-box testing, as it requires specialized tools like code analyzers and debugging tools.   Grey-Box Testing: Grey-box testing is a technique to test the application with having a limited knowledge of the internal workings of an application. In software testing, the phrase the more you know, the better carries a lot of weight while testing an application. Mastering the domain of a system always gives the tester an edge over someone with limited domain knowledge. Unlike black-box testing, where the tester only tests the application's user interface; in grey-box testing, the tester has access to design documents and the database. Having this knowledge, a tester can prepare better test data and test scenarios while making a test plan. Advantages Disadvantages Offers combined benefits of black-box and white-box testing wherever possible. Grey box testers don't rely on the source code; instead they rely on interface definition and functional specifications. Based on the limited information available, a grey-box tester can design excellent test scenarios especially around communication protocols and data type handling. The test is done from the point of view of the user and not the designer. Since the access to source code is not available, the ability to go over the code and test coverage is limited. The tests can be redundant if the software designer has already run a test case. Testing every possible input stream is unrealistic because it would take an unreasonable amount of time; therefore, many program paths will go untested.   A Comparison of Testing Methods: The following table lists the points that differentiate black-box testing, grey-box testing, and white-box testing. Black-Box Testing Grey-Box Testing White-Box Testing The internal workings of an application need not be known. The tester has limited knowledge of the internal workings of the application. Tester has full knowledge of the internal workings of the application. Also known as closed-box testing, data-driven testing, or functional testing. Also known as translucent testing, as the tester has limited knowledge of the insides of the application. Also known as clear-box testing, structural testing, or code-based testing. Performed by end-users and also by testers and developers. Performed by end-users and also by testers and developers. Normally done by testers and developers. Testing is based on external expectations - Internal behavior of the application is unknown. Testing is done on the basis of high-level database diagrams and data flow diagrams. Internal workings are fully known and the tester can design test data accordingly. It is exhaustive and the least time-consuming. Partly time-consuming and exhaustive. The most exhaustive and time-consuming type of testing. Not suited for algorithm testing. Not suited for algorithm testing. Suited for algorithm testing. This can only be done by trial-and-error method. Data domains and internal boundaries can be tested, if known. Data domains and internal boundaries can be better tested.   >> Levels Of Testing There are different levels during the process of testing. In this chapter, a brief description is provided about these levels. Levels of testing include different methodologies that can be used while conducting software testing. The main levels of software testing are: Functional Testing Non-functional Testing    Functional Testing:- This is a type of black-box testing that is based on the specifications of the software that is to be tested. The application is tested by providing input and then the results are examined that need to conform to the functionality it was intended for. Functional testing of a software is conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements. There are five steps that are involved while testing an application for functionality. Steps Description I The determination of the functionality that the intended application is meant to perform. II The creation of test data based on the specifications of the application. III The output based on the test data and the specifications of the application. IV The writing of test scenarios and the execution of test cases. V The comparison of actual and expected results based on the executed test cases. An effective testing practice will see the above steps applied to the testing policies of every organization and hence it will make sure that the organization maintains the strictest of standards when it comes to software quality. Unit Testing: This type of testing is performed by developers before the setup is handed over to the testing team to formally execute the test cases. Unit testing is performed by the respective developers on the individual units of source code assigned areas. The developers use test data that is different from the test data of the quality assurance team. The goal of unit testing is to isolate each part of the program and show that individual parts are correct in terms of requirements and functionality. Limitations of Unit Testing: Testing cannot catch each and every bug in an application. It is impossible to evaluate every execution path in every software application. The same is the case with unit testing. There is a limit to the number of scenarios and test data that a developer can use to verify a source code. After having exhausted all the options, there is no choice but to stop unit testing and merge the code segment with other units. Integration Testing: Integration testing is defined as the testing of combined parts of an application to determine if they function correctly. Integration testing can be done in two ways: Bottom-up integration testing and Top-down integration testing. S.N. Integration Testing Method 1 Bottom-up integration This testing begins with unit testing, followed by tests of progressively higher-level combinations of units called modules or builds. 2 Top-down integration In this testing, the highest-level modules are tested first and progressively, lower-level modules are tested thereafter. In a comprehensive software development environment, bottom-up testing is usually done first, followed by top-down testing. The process concludes with multiple tests of the complete application, preferably in scenarios designed to mimic actual situations. System Testing: System testing tests the system as a whole. Once all the components are integrated, the application as a whole is tested rigorously to see that it meets the specified Quality Standards. This type of testing is performed by a specialized testing team. System testing is important because of the following reasons: System testing is the first step in the Software Development Life Cycle, where the application is tested as a whole. The application is tested thoroughly to verify that it meets the functional and technical specifications. The application is tested in an environment that is very close to the production environment where the application will be deployed. System testing enables us to test, verify, and validate both the business requirements as well as the application architecture.    Regression Testing: Whenever a change in a software application is made, it is quite possible that other areas within the application have been affected by this change. Regression testing is performed to verify that a fixed bug hasn't resulted in another functionality or business rule violation. The intent of regression testing is to ensure that a change, such as a bug fix should not result in another fault being uncovered in the application. Regression testing is important because of the following reasons: Minimize the gaps in testing when an application with changes made has to be tested. Testing the new changes to verify that the changes made did not affect any other area of the application. Mitigates risks when regression testing is performed on the application. Test coverage is increased without compromising timelines. Increase speed to market the product.   Acceptance Testing: This is arguably the most important type of testing, as it is conducted by the Quality Assurance Team who will gauge whether the application meets the intended specifications and satisfies the client’s requirement. The QA team will have a set of pre-written scenarios and test cases that will be used to test the application. More ideas will be shared about the application and more tests can be performed on it to gauge its accuracy and the reasons why the project was initiated. Acceptance tests are not only intended to point out simple spelling mistakes, cosmetic errors, or interface gaps, but also to point out any bugs in the application that will result in system crashes or major errors in the application. By performing acceptance tests on an application, the testing team will deduce how the application will perform in production. There are also legal and contractual requirements for acceptance of the system.   Alpha Testing: This test is the first stage of testing and will be performed amongst the teams (developer and QA teams). Unit testing, integration testing and system testing when combined together is known as alpha testing. During this phase, the following aspects will be tested in the application: Spelling Mistakes Broken Links Cloudy Directions The Application will be tested on machines with the lowest specification to test loading times and any latency problems.    Beta Testing: This test is performed after alpha testing has been successfully performed. In beta testing, a sample of the intended audience tests the application. Beta testing is also known as pre-release testing. Beta test versions of software are ideally distributed to a wide audience on the Web, partly to give the program a "real-world" test and partly to provide a preview of the next release. In this phase, the audience will be testing the following: Users will install, run the application and send their feedback to the project team. Typographical errors, confusing application flow, and even crashes. Getting the feedback, the project team can fix the problems before releasing the software to the actual users. The more issues you fix that solve real user problems, the higher the quality of your application will be. Having a higher-quality application when you release it to the general public will increase customer satisfaction. Non-Functional Testing:- This section is based upon testing an application from its non-functional attributes. Non-functional testing involves testing a software from the requirements which are nonfunctional in nature but important such as performance, security, user interface, etc. Some of the important and commonly used non-functional testing types are discussed below.  Performance Testing: It is mostly used to identify any bottlenecks or performance issues rather than finding bugs in a software. There are different causes that contribute in lowering the performance of a software: Network delay Client-side processing Database transaction processing Load balancing between servers Data rendering Performance testing is considered as one of the important and mandatory testing type in terms of the following aspects: Speed (i.e. Response Time, data rendering and accessing) Capacity Stability Scalability Performance testing can be either qualitative or quantitative and can be divided into different sub-types such as Load testing and Stress testing.   Load Testing: It is a process of testing the behavior of a software by applying maximum load in terms of software accessing and manipulating large input data. It can be done at both normal and peak load conditions. This type of testing identifies the maximum capacity of software and its behavior at peak time. Most of the time, load testing is performed with the help of automated tools such as Load Runner, AppLoader, IBM Rational Performance Tester, Apache JMeter, Silk Performer, Visual Studio Load Test, etc. Virtual users (VUsers) are defined in the automated testing tool and the script is executed to verify the load testing for the software. The number of users can be increased or decreased concurrently or incrementally based upon the requirements.   Stress Testing: Stress testing includes testing the behavior of a software under abnormal conditions. For example, it may include taking away some resources or applying a load beyond the actual load limit. The aim of stress testing is to test the software by applying the load to the system and taking over the resources used by the software to identify the breaking point. This testing can be performed by testing different scenarios such as: Shutdown or restart of network ports randomly Turning the database on or off Running different processes that consume resources such as CPU, memory, server, etc.   Usability Testing: Usability testing is a black-box technique and is used to identify any error(s) and improvements in the software by observing the users through their usage and operation. According to Nielsen, usability can be defined in terms of five factors, i.e. efficiency of use, learn-ability, memory-ability, errors/safety, and satisfaction. According to him, the usability of a product will be good and the system is usable if it possesses the above factors. Nigel Bevan and Macleod considered that usability is the quality requirement that can be measured as the outcome of interactions with a computer system. This requirement can be fulfilled and the end-user will be satisfied if the intended goals are achieved effectively with the use of proper resources. Molich in 2000 stated that a user-friendly system should fulfill the following five goals, i.e., easy to Learn, easy to remember, efficient to use, satisfactory to use, and easy to understand. In addition to the different definitions of usability, there are some standards and quality models and methods that define usability in the form of attributes and sub-attributes such as ISO-9126, ISO-9241-11, ISO-13407, and IEEE std.610.12, etc.   UI vs Usability Testing: UI testing involves testing the Graphical User Interface of the Software. UI testing ensures that the GUI functions according to the requirements and tested in terms of color, alignment, size, and other properties. On the other hand, usability testing ensures a good and user-friendly GUI that can be easily handled. UI testing can be considered as a sub-part of usability testing.   Security Testing: Security testing involves testing a software in order to identify any flaws and gaps from security and vulnerability point of view. Listed below are the main aspects that security testing should ensure: Confidentiality Integrity Authentication Availability Authorization Non-repudiation Software is secure against known and unknown vulnerabilities Software data is secure Software is according to all security regulations Input checking and validation SQL insertion attacks Injection flaws Session management issues Cross-site scripting attacks Buffer overflows vulnerabilities Directory traversal attacks   Portability Testing: Portability testing includes testing a software with the aim to ensure its reusability and that it can be moved from another software as well. Following are the strategies that can be used for portability testing: Transferring an installed software from one computer to another. Building executable (.exe) to run the software on different platforms. Portability testing can be considered as one of the sub-parts of system testing, as this testing type includes overall testing of a software with respect to its usage over different environments. Computer hardware, operating systems, and browsers are the major focus of portability testing. Some of the pre-conditions for portability testing are as follows: Software should be designed and coded, keeping in mind the portability requirements. Unit testing has been performed on the associated components. Integration testing has been performed. Test environment has been established.    >> Documentaion Of Testing Testing documentation involves the documentation of artifacts that should be developed before or during the testing of Software. Documentation for software testing helps in estimating the testing effort required, test coverage, requirement tracking/tracing, etc. This section describes some of the commonly used documented artifacts related to software testing such as: Test Plan Test Scenario Test Case Traceability Matrix   Test Plan: A test plan outlines the strategy that will be used to test an application, the resources that will be used, the test environment in which testing will be performed, and the limitations of the testing and the schedule of testing activities. Typically the Quality Assurance Team Lead will be responsible for writing a Test Plan. A test plan includes the following: Introduction to the Test Plan document Assumptions while testing the application List of test cases included in testing the application List of features to be tested What sort of approach to use while testing the software List of deliverables that need to be tested The resources allocated for testing the application Any risks involved during the testing process A schedule of tasks and milestones to be achieved   Test Scenario: It is a one line statement that notifies what area in the application will be tested. Test scenarios are used to ensure that all process flows are tested from end to end. A particular area of an application can have as little as one test scenario to a few hundred scenarios depending on the magnitude and complexity of the application. The terms 'test scenario' and 'test cases' are used interchangeably, however a test scenario has several steps, whereas a test case has a single step. Viewed from this perspective, test scenarios are test cases, but they include several test cases and the sequence that they should be executed. Apart from this, each test is dependent on the output from the previous test.   Test Case: Test cases involve a set of steps, conditions, and inputs that can be used while performing testing tasks. The main intent of this activity is to ensure whether a software passes or fails in terms of its functionality and other aspects. There are many types of test cases such as functional, negative, error, logical test cases, physical test cases, UI test cases, etc. Furthermore, test cases are written to keep track of the testing coverage of a software. Generally, there are no formal templates that can be used during test case writing. However, the following components are always available and included in every test case: Test case ID Product module Product version Revision history Purpose Assumptions Pre-conditions Steps Expected outcome Actual outcome Post-conditions Many test cases can be derived from a single test scenario. In addition, sometimes multiple test cases are written for a single software which are collectively known as test suites.   Traceability Matrix: Traceability Matrix (also known as Requirement Traceability Matrix - RTM) is a table that is used to trace the requirements during the Software Development Life Cycle. It can be used for forward tracing (i.e. from Requirements to Design or Coding) or backward (i.e. from Coding to Requirements). There are many user-defined templates for RTM. Each requirement in the RTM document is linked with its associated test case so that testing can be done as per the mentioned requirements. Furthermore, Bug ID is also included and linked with its associated requirements and test case. The main goals for this matrix are: Make sure the software is developed as per the mentioned requirements. Helps in finding the root cause of any bug. Helps in tracing the developed documents during different phases of SDLC.   >> Estimation Techniques Of Testing Estimating the efforts required for testing is one of the major and important tasks in SDLC. Correct estimation helps in testing the software with maximum coverage. This section describes some of the techniques that can be useful in estimating the efforts required for testing. Functional Point Analysis: This method is based on the analysis of functional user requirements of the software with the following categories: Outputs Inquiries Inputs Internal files External files   Test Point Analysis: This estimation process is used for function point analysis for black-box or acceptance testing. The main elements of this method are: Size, Productivity, Strategy, Interfacing, Complexity, and Uniformity. Mark-II Method: It is an estimation method used for analyzing and measuring the estimation based on end-user’s functional view. The procedure for Mark-II method is as follows: Determine the viewpoint Purpose and type of count Define the boundary of count Identify the logical transactions Identify and categorize data entity types Count the input data element types Count the functional size   Miscellaneous: You can use other popular estimation techniques such as: Delphi Technique Analogy Based Estimation Test Case Enumeration Based Estimation Task (Activity) based Estimation IFPUG method