UNIT II System and Acceptance Testing

SYSTEM TESTING OVERVIEW

 The testing conducted on the complete integrated products and solutions to evaluate system compliance
with specified requirements on functional and non-functional aspects is called system testing.
 System testing is defined as a testing phase conducted on the complete integrated system, to evaluate the
system compliance with its specified requirements.
 It is done after unit, component, and integration testing phase.
 A system is a complete set of integrated components that together deliver product functionality and features.
A system can also be defined as a set of hardware, software, and other parts that together provide product
features and solutions.
 System testing is the only phase of testing which tests the both functional and non-functional aspects of the
product.
system brings in different testing types (also called quality factors),
some of which are as follows.
1. Performance/Load testing: To evaluate the time taken or response time of the system to perform its required
functions in comparison with different versions of same product(s) or different competitive products(s) is called
performance testing.
2. Scalability testing: A testing that requires enormous amount of resource to find out the maximum capability
of the system parameters is called scalability testing.
3. Reliability testing: To evaluate the ability of the system or an independent component of the system to
perform its required functions repeatedly for a specified period of time is called reliability testing.
4. Stress testing: Evaluating a system beyond the limits of the specified requirements or system resources (such
as disk space, memory, processor utilization) to ensure the system does not break down unexpectedly is called
stress testing.
5. Interoperability testing: This testing is done to ensure that two or more products can exchange information,
use the information and work closely.
6. Localization testing: Testing conducted to verify that the localized product works in different languages is
called localization testing.

WHY IS SYSTEM TESTING DONE?

To summarize, system testing is done for the following reasons.
1. Provide independent perspective in testing
2. Bring in customer perspective in testing
3. Provide a “fresh pair of eyes” to discover defects not found earlier by testing
4. Test product behavior in a holistic, complete, and realistic environment
5. Test the both functional and non- functional aspects of the product
6. Build confidence in the product
7. Analyze and reduce the risk of releasing the product
8. Ensure all requirements are met and ready the product for acceptance testing.

FUNCTIONAL VERSUS NON-FUNCTIONAL TESTING

Functional testing involves testing a product’s functionality and features. Non-functional testing involves
testing the product’s quality factors. System testing comprises both functional and nonfunctional test
verification.

GCC BCA V SEM (2020) SOFTWARE PRACTICES AND TESTING Prof: Vaibhav Ambekar
FUNCTIONAL SYSTEM TESTING
Functional testing is performed at different phases and the focus is on product level features. As functional
testing is performed at various testing phases, there are two obvious problems. One is duplication and other
one is gray area.
Duplication refers to the same tests being performed multiple times and gray area refers to the certain test
being missed out in all the phases.
Gray areas in testing happen due to lack of product knowledge, lack of knowledge of customer usage, and lack
of co-ordination across test teams.
Some of the common Functional Testing techniques are given below.
1. Design/architecture verification
2. Business vertical testing
3. Deployment testing
4. Beta testing
5. Certification, standards, and testing for compliance.
Design/ architecture verification
In this method of functional testing, the test cases are developed and checked against the design and architecture
to see whether they are actual product-level test cases. Comparing this with integration testing, the test cases for
integration testing are created by looking at interfaces whereas system level test cases are created first and
verified with design and architecture to check whether they are product-level or component-level test cases.
Some of the guidelines used to reject test cases for system functional testing include the following.
1. Is this focusing on code logic, data structures, and unit of the product?(if yes, then it
belongs to unit testing.)
2. Is this specified in design and architecture specification for integration testing? (if yes,
then it belongs to integration testing.)
3. Is this specified in the functional specification of any component? (if yes, then it belongs
to component testing.)
4. Is it focusing on product implementation but not visible to customer?(this is focusing on
implementation-to be covered in unit/component/integration testing.)
5. Is it the right mix of customer usage and product implementation?(customer usage is a
prerequisite for system testing.)

Business vertical testing

General purpose products like workflow automation system can be used by different businesses
and services. Using and testing the product for different business verticals such as insurance,
banking, asset management, and so on, and verifying the business operations and usage, is called
“business vertical testing”.
GCC BCA V SEM (2020) SOFTWARE PRACTICES AND TESTING Prof: Vaibhav Ambekar
For this type of testing, the procedure in the product is altered to suit the process in the business. For example,
in loan processing, the loan is approved first by the officer and then sent to a clerk. In claim processing, the
claim is first worked out by a clerk and then sent to an officer for approval. User objects such as clerk and
officer are created by the product and associated with the operations. It is important that the product understands
the business processes and includes customization as a feature so that different business vertical can
use the product.

Another important aspect is called terminology. To explain this concept let us take the example
of e-mail. An e-mail sent in the insurance context may be called a claim whereas when an e-mail
is sent in a loan-processing system, it is called a loan application. The users would be familiar
with this terminology rather than the generic terminology of “e-mail.”

Yet another aspect involved in business vertical testing is syndication: Solution integrators, service providers
pay a license free to a product organization and sell the products and solutions using their name and image. In
this case the product name, company name, technology names, and copyrights may belong to the latter parties
or associations.
Business vertical testing can be done in two ways-simulation and replication.
 In simulation of a vertical test, the customer or the tester assumes requirements and the business flow is
tested.
In replication, customer data and process is obtained and the product is completely customized,
tested, and the customized product as it was tested is released to the customer.

Deployment testing
System testing is the final phase before product delivery. By this time the prospective customers and their
configuration would be known and in some cases the products would have been committed for sale. Type of
deployment testing that happens in a product development company to ensure that customer deployment
requirements are met is called offsite deployment.

Deployment testing is also conducted after the release of the product by the utilizing the resources and setup
available in customer‟ locations. This is a combined effort by the product development organization and the
organization trying to use the product. This is called onsite deployment. Onsite deployment testing is done at
two stages.
In the first stage(stage1), actual data from the live system is taken and similar machines and configurations
are mirrored, and the operations from the users are rerun on the mirrored deployment machine. This gives an
idea whether the enhanced or similar product can perform the existing functionality without affecting the user.
In the second stage (stage2), after a successful first stage, the mirrored system is made a live system that runs
the new product. Regular backups are taken and alternative methods are used to record the incremental
transactions from the time mirrored system became live the recorded that was used in the first stage can also be
used here.

Beta Testing
• A product may be rejected by a customer even after delivery due to following reasons:
– Not meeting the implicit requirements
– Failure to reflect the requirement changes in the product.
– Not resolving the ambiguity in the requirement.
– Understanding of the requirement may be correct, but implementation may be wrong.
– Lack of usability and documentation
• To reduce the risk of such rejection, periodic feedback is obtained on the product.

GCC BCA V SEM (2020) SOFTWARE PRACTICES AND TESTING Prof: Vaibhav Ambekar
One of the mechanisms used is sending the product that is under test to the customer and receiving the
feedback. This is called beta testing.
This testing is performed by the customer and helped by the product development organization. During the
entire duration of beta testing, there are various activities that are planned and executed according to a
specific schedule. This is called a beta program.
Some of the activities involved in the beta program are as follows.
• Collecting the list of customers and their beta testing requirements.
• Schedule Beta program & inform the customers. End date should be reasonably before the release
date so that defects can be fixed.
• Sending documents & training the customers about the product.
• Sending the product for Beta testing.
• Collecting the feedback periodically.
• Responding to customer feedback with fixes & document changes.
• Analyzing and concluding whether the beta program met the exit criteria.
• Communicate the progress and action items to customers and formally closing the beta program.

Certification, Standards and Testing for Compliance

A product needs to be certified with the popular hardware, operating system, database, and other
infrastructure pieces. This is called certification testing.
The sales of a product depend on whether it was certified with the popular system or not.
The product has to work with the popular system, as the customer would have already invested heavily on
those.
 Not only should the product co-exist and run with the current versions of these popular system, but the
product organization should also document a commitment (in the form of a roadmap) to continue to work
with the future versions of the popular system.

Standard Testing:
There are many standards for each technology area and the product may need to conform to those standards.
This is very important as adhering to these standards makes the product interact easily with other products.
Some of the standards are evolved by the open community and published as public domain standards(for
example, open LDAP standard).
Testing the product to ensure that these standards are properly implemented is called as Testing for
Standards. Tools associated with those open standards can be used free of cost to verify the standards
implementation.

Compliance Testing:
There are many contractual and legal requirements for a product. Failing to meet these may result in business
loss and bring legal action against the origanization and its senior management.
The following are some examples of compliance testing.
Compliance to FDA This act by the food and drug administration requires that adequate testing be done for
products such as cosmetics, drugs, and medical sciences.
508 accessibility guidelines This accessibility set of guidelines requires the product to meet some
requirements for its physically challenged users. These guidelines insist that the product should be as accessible
to physically challenged people as it is to people without those disabilities.

6.4 Non-functional testing

The process followed by non-functional testing is similar to that of functional testing but differs from the
aspects of complexity, knowledge requirement, effort needed, and number of times the test cases are repeated.

GCC BCA V SEM (2020) SOFTWARE PRACTICES AND TESTING Prof: Vaibhav Ambekar
Compared to functional testing, non-functional testing is less repeated as it involves more effort, time &
resources.

Challenges
6.5.1 Setting up the configuration
Setting up a configuration is a challenge for the following reasons.
1. Given the high diversity of environments and variety of customers, it is very difficult to predict the type of
environment that will be used commonly by the customers.
2. Testing a product with different permutations and combinations of configurations may not prove effective
since the same combination of environment may not be used by the customer and testing for several
combinations involves effort and time.
3. The cost involved in setting up such environments is quite high.
4. The people may not have the skills to set up the environment.
6. It is difficult to predict the exact type and nature of data that the customer may use.

Coming up with entry/exit criteria

Coming up with entry and exit criteria is another critical factor in non-functional testing. Meeting the entry
criteria is the responsibility of the previous test phase.
Balancing key resources
This section intends to discuss the concepts non-functional testing with respect to four key resources-CPU,
disk, memory, and network. The four resources are related to each other and we need to completely understand
their relationship to implement the strategy for nonfunctional testing.

Scalability testing
The objective of scalability testing is to find out the maximum capability of the product Parameters. As the
exercise involves finding the maximum, the resources that are needed for this kind of testing are normally very
high.
For example, one of the scalability test case could be finding out how many client machines can
simultaneously log in to the server to perform some operations.
Hence a high-end configuration is selected and the scalability parameter is increased
step by step to reach the maximum capability.
• It Helps to identify major bottlenecks in a product.
• After the test, collected data is analyzed & appropriate actions are taken.
• Needs fine analysis to find out where is the problem & what exactly is the corrective action.
• Demand on the resources increases exponentially, scalability parameter increases & reaches
saturation-max capability.
• Takes care of product bottleneck & resource up gradation.
• Whenever tuning is performed, recommended values of the parameters is documented, called a
“sizing guide” .
• Requires experience & detailed study of the product.
• Impact caused by fixing scalability defects is taken care by reliability testing.

In scalability testing, the demand on resources tends to grow exponentially when the scalability parameter is
increased. The scalability reaches a saturation point beyond which it cannot be improved. This is called the
maximum capability of a scalability parameter. Even though resources may be available, product limitation may
not allow scalability. This is called a product bottleneck.

GCC BCA V SEM (2020) SOFTWARE PRACTICES AND TESTING Prof: Vaibhav Ambekar
Reliability testing
Reliability testing is done to evaluate the product’s ability to perform its required functions under stated
conditions for a specified period of time or for a large number of iterations.
Examples: queering a database continuously for 48 hours and performing login operations 10,000 times.
The reliability of a product should not be confused with reliability testing.
Producing a reliable product requires sound techniques, good discipline, robust processes, and strong
management, this product reliability is achieved by focusing on the following activities.
1. Defined engineering processes Software reliability can be achieved by following clearly
defined processes.
2. Review of work products at each stage At the end of each stage of the product
development life cycle, the work products produced are reviewed. This ensures early
detection of and their fixes as soon as they are introduced.
3. Change management procedures Many errors percolate to the product due to improper
impact analysis of changes made to the product.
4. Review of testing converges Allocating time for the different phases and type of testing can
help in catching errors as and when the product is being developed, rather than after the
product is developed.
5. Ongoing monitoring of the product Once the product has been delivered, it is analyzed
proactively for any possibly missed error in this case the process as well as the product is
fixed for missed defects. This prevents the same type of defects from reappearing.

Reliability testing on the other hand refers to testing the product for a continuous period of time.Performing
good reliability testing does not ensure a reliable product on its own, reliability testing only delivers a
“reliability tested product” but not a reliable product.

In figure 6.4, it can be seen that the product is very close to meeting the reliability criteria towards the end.
Figure 6.4(a) suggests that the progress towards meeting the criteria is smooth,whereas in figure 6.4 (b), the
transition contains spikes. These spikes indicate that defects in the product for the reliability tests go up an
down periodically. This may be a pointer to indicate that the defect fixes are creating new defects in the system.
Analyzing the spikes and taking action to avoid the spikes, both from the process and from the product
perspective.

There are different ways of expressing reliability defects in charts.

1. Mean time between failures is the average time elapsed from between successive product
failures.
2. Failure rate is provided as a function that gives the number of failures occurring per unit
time (the graphs above depict this measurement).
3. Mean time to discover the next k faults is the measure used to product the average length
of time unit the next k faults are encountered.
GCC BCA V SEM (2020) SOFTWARE PRACTICES AND TESTING Prof: Vaibhav Ambekar
To summarize, a “reliability tested product” will have the following characteristics.
1. No errors or very few errors from repeated transactions.
2. Zero downtime.
3. Optimum utilization of resources.
4. Consistent performance and response time of the product for repeated transactions for
a specified time duration.
5. No side-effects after the repeated transactions are executed.

Stress testing
Stress testing is done to evaluate a system beyond the limits of specified requirements or
resources, to ensure that system does not break.
Stress testing is done to find out if the product‟s behavior degrades under extreme conditions and when it is
denied the necessary resources.
Stress testing help in understanding how the system can behave under extreme (insufficient memory,
inadequate hardware) and realistic situatations.
To know the conditions under which these tests fail so that the max limits, in terms of simultaneous users,
search criteria, large no of transactions is known.
• Though, tools are available for such tests, it is recommended to repeat the tests without tools.
• In these tests, the load is increased through various means (clients, transactions etc) till ans beyond
the resources are completely utilized.
• The product reaches a „Stress point‟ when some of the transactions start failing.
• At this point, the load is slightly reduced to see if the product recovers & the failure rate decreases.
This process is repeated to see consistency in product‟s behavior.
• The time required for the product to quickly recover from the failure is represented by MTTR (Mean
Time To Recover).
• MTTR may be different for different operations.
• Several iterations of tests for different operations are conducted around the Stress point & MTTR is
calculated from mean.
• The tests performed to calculate Stress point need to be closer to real life scenario.

The following guidelines can be used to select the tests for stress testing.
1. Repetitive tests: Executing repeated tests ensures that at all times the code works as expected.
2. Concurrency: Concurrent tests ensure that the code is exercised in multiple paths and
simultaneously.
3. Magnitude: This refers to the amount of load to be applied to the product to stress the system.
4. Random variation :Stress testing depends on increasing/ decreasing variable load. Tests that
stress the system with random inputs (in terms of number of users, size of data).

Interoperability testing
Interoperability testing is done to ensure the two or more products can exchange information, use
information, and work properly together.
System can be interoperable unidirectional (the exchange of information is done way) or bidirectional
(exchange of information in both ways). For example, the text available in a text
editor can be exported into a Microsoft word application using the “insert->File” option. But a
picture available in Microsoft word cannot be exported into text editor. This represents one-way
interoperability. The two-way interoperability is represented by exchange of information
between email management (Microsoft outlook) and Microsoft word, where information can be
cut and pasted on both directions.

GCC BCA V SEM (2020) SOFTWARE PRACTICES AND TESTING Prof: Vaibhav Ambekar
 Interoperability can not be achieved unless two or more products are designed for exchanging
information.

It is more important in context of Internet. It is essential for more & more products to be interoperable so
that they can communicate with all OS, browsers, development tools, compilers & so on.

The following are some guidelines that help in improving interoperability.

1. Consistency of information flow across system when an input is provided to the product, it
should be understood consistently by all systems.

2. Changes to data representation as per the system requirements

3. Correlated interchange of messages and receiving appropriate responses When one system
sends an input in the form of a message, the next system is in the waiting mode or listening mode to information
exchange, there could be clashes, wrong response, deadlocks, or delays in communication.
4. Communication and messages When a message is passed on from a system A to B, if any and
the message is lost the product should be tested to check how it responds to such erroneous messages. The
product must not crash or hang. It should give useful error message to the user.

Acceptance testing
Definition: Acceptance testing is done by the customers or representatives of the customer to
check whether the product is ready for use in the real life environment.
The customer defines a set of test cases that will be executed to qualify and accept the product.
Acceptance tests are written to execute near real-life scenarios. Apart from verifying the functional
requirements, acceptance tests are run to verify the non-functional aspects of the system also.

Acceptance criteria

Acceptance criteria-product acceptance During the requirements phase, each requirement

is associated with acceptance criteria. It is possible that one or more requirements may be
mapped to form acceptance criteria (for example, all high priority requirements should pass
100%). Whenever there are changes to requirements, the acceptance criteria are accordingly
modified and maintained.
Acceptance criteria-procedure acceptance: Acceptance criteria can be defined based on the
procedures followed for delivery. An example of procedure acceptance could be documentation and release
media. Some examples of acceptance criteria of this nature are as follows.
1. User, administration and troubleshooting documentation should be part of the release.
2. Along with binary code, the source codes of the product with build scripts to be delivered in a CD.
3. A minimum of 20 employees are trained on the product usage prior to deployment.

Acceptance criteria-service level agreements Service level agreements (SLA) can become part of
acceptance criteria. Service level agreements are generally part of a contract
signed by the customer and product organization.
For example, time limits to resolve those defects can be mentioned part of SLA such as
_ All major defect that come up during first three months of deployment need to be fixed
free of cost;
_ Downtime of the implemented system should be less than 0.1%;
_ All major defects are to fixed within 48 hours of reporting.

GCC BCA V SEM (2020) SOFTWARE PRACTICES AND TESTING Prof: Vaibhav Ambekar
Selecting test cases for acceptance testing

1. End-to-end functionality verification: It ensures that all business transactions are tested with real life
scenarios.
2. Domain tests :Test cases that reflect business domain knowledge are included.
3. User scenario tests Acceptance tests reflect the real-life user scenario verification. As a
result, test cases that portray them are include.
4. Basic sanity tests Tests that verify the basic existing behavior of the product are included. These tests ensure
that the system performs the basic operations that it was intended to do.
5. New functionality When the product undergoes modifications of changes, the acceptance test cases focus on
verifying the new features.
6. A few non-functional tests Some non-functional tests are include and executed as part of acceptance testing
to double-check that the non-functional aspects of the product meet the expectations.
7. Tests pertaining to legal obligations and service level agreements Tests that are
written to check if the product complies with certain legal obligations and SLAs are included in the acceptance
test criteria.
8. Acceptance test data Test cases that make use of customer real-life data are included for acceptance testing.

GCC BCA V SEM (2020) SOFTWARE PRACTICES AND TESTING Prof: Vaibhav Ambekar