Software Testing-Challenges and Strategies 8/12/2004

Software Testing – Challenges

& Strategies
ht
r ig
p y
Mr. Arunkumar Khannur

Co
Managing Director- QSIT
Quality Solutions for Information Technology Pvt Ltd
A-706 & 707, 7th Floor, Mittal Towers,

I T
M.G.Road, Bangalore-560 001

QS
Email: [email protected]
URL: www.qsitglobal.com
Ph: +91-80-25585386/51134334

The Challenges we cover today…

z Challenge # 1 – What to Test…
z Challenge # 2 – How Much to Test…
z Challenge # 3 – When to Start Testing…
ht
ir g
z Challenge # 4 – Integration Testing

y
z Challenge # 5 –– Only Test Team follows
process
p
Co
z Challenge # 6 – Intelligent Testing
z

I T
Challenge # 7 – Test Execution Management

QS
2

[email protected] 1
Software Testing-Challenges and Strategies 8/12/2004

Challenge #1: What to Test…

z Deciding on What to test is becoming more and
more more strategic.
z WHY??!!
ht
r ig
y
Likability

p
Co
Non-
Functionality
Functionality
Performance Test

T
Usability

I
Feature Tests
Scalability Test

QS
Scenario Test Platform Test

Inter-operability Test

Reliability Test

Challenge #1: What to Test…(Contd.)

z Functional Requirements – more or less clear –
what the system does and why part
z But
ht
ir g
– Performance Requirements – what are the target #
z Transactions/second
z Download times

p y
Co
– Scalability Requirements
– Inter-operability – with which version of platforms,

I T
products, utilities and so on….

QS
4

[email protected] 2
Software Testing-Challenges and Strategies 8/12/2004

Challenge #1: What to Test…(Contd.)

z Implicit Requirements that come up later in the
cycle
– I assume we use JVM x.xx.xx…..

ht
ig
– I assume we are using double encryption algorithms

r
z There are certain requirements for which coding
happens after testing!
p y
Co
– Support Red Hat Linux x.xx.x
Support Solaris y.yy.yy

T
–

I
z No need to code anything for these – just test, if we get any

QS
defects, we will fix

Challenge #1: What to Test…(Contd.)

z In summary
– All requirements are not clear
–

ht
For all requirements we do not have inputs

ir g
– For different requirements we should talk to different
stakeholders
–

p y
For those requirements there is no coding, we have

Co
no clue what is happening till we get a defect!
– ……..

I T
QS
6

[email protected] 3
Software Testing-Challenges and Strategies 8/12/2004

Strategy #1a What to Test…

z Be in the “game”
– Be stakeholder of requirements
–

ht
Do some Benchmarks and provide “data” for

ig
performance, scalability requirements

r
– Be prepared to do R&D
z Be “proactive” where required
p y
Co
– Make effective use of “scope” section in your test
plan
–

I T
Communicate details of the plan & strategy in

QS
nutshells rather than full fledged document mails
– Be involved in the project all the time (if not whole of
test team, may be the lead/manager)

Strategy #1b What to Test…

z Have suggestions and solutions apart from
questions and issues – after all Testing is one of

t
the “stakeholder”

h
ir g
z Take initiative and get things done! – rather than
being in the receiving end – move out of the

p y
mode - I test whatever is “stated” – be a relay

Co
partner

I T
QS
8

[email protected] 4
Software Testing-Challenges and Strategies 8/12/2004

Strategy #1 What to Test…Summary

z Visualize Requirements using models
z Models are to understand the context
z Express the “perceived” behavior
ht
ig
z Any for of the model can be used
z
y
Be consistent within project
r
p
Co
z Picture Vs Words
z Mental Models can also be a tool
z
I T
Key is to “gain” product knowledge

QS
9

Challenge #2: How much to Test…

z Deciding on How much to test is becoming
more and more more tricky.
z WHY??!!
ht
ir g
– Increased Abstraction, Reusability
Computer, Content, Communication, Connectivity

y
–

p
– Global Computing, Heterogeneous Resources

Co
– 24/7 Uptime Requirements

I T
QS
10

[email protected] 5
Software Testing-Challenges and Strategies 8/12/2004

Strategy #2 How much to Test…

z Analyze Specified Requirements to understand
– Test Factors
– Testability

ht
ig
– Product Success Criteria

yr
p
T Co
I
QS
11

ht
ir g
p y
T Co
I
QS
12

[email protected] 6
Software Testing-Challenges and Strategies 8/12/2004

Challenge #3: When to Start Testing…

z Who will tell us that its ideal time to start?
z Or Should we decide, if yes who will “certify”
you can start now?
ht
ig
z When will we meet the “Entry Criteria” 100%?
z

yr
What is the guarantee that we needn’t stop if we

p
start now?

Co
z Who will ensure smooth sailing?

I T
z How do we know that this “work around” is

QS
acceptable for now?
z By when we fix all defects, so that we can start
all over again?

13

Strategy #3 When to Start Testing…

z Have a broad Entry Criteria for “formal” start
– Make sure that we don’t end up repeating things…

t
z Start as early as possible in case resource usage is

h
moderately flexible (triple constraints theory)
z
ir g
For entry criteria start tracking the same while we

y
are nearing to “code complete” milestone

p
Co
z Have the ownership of entry criteria – make sure
that we get what we want, rather then wait and see

I T
what we “got”!

QS
14

[email protected] 7
Software Testing-Challenges and Strategies 8/12/2004

Challenge # 4 – Integration Testing

z We all know definitions!
– When it comes to practice especially in an advanced

t
technology environment – this is difficult to strategize

h
ig
z If I do integration testing should I do

r
“component” or “Module” testing?
z

p y
How do I scope between IT and ST?

Co
z Does the architecture permit a top-down or
bottom-up integration?

I T
QS
z Who will own the “integration” so that we can
test it?

15

Strategy #4 – Integration Testing

z Decide upfront whether IT is required, feasible or NOT –
most of the time we do this as a routine in the process

t
than looking for benefit – all products may not need IT!!!

h
z Have a high level Strategy for Unit, Component,

ir g
Integration and System Testing scope

y
z Train your testing team in the related technology – since

p
integration involves a lot of white-box stuff

Co
z Make the project to decide – who, when and how will
“integrate” the product – so that we can test the same

I T
(ideally integration would be a stepwise approach and

QS
each step can be tested separately)

16

[email protected] 8
Software Testing-Challenges and Strategies 8/12/2004

Challenge # 5 – Only Test Team follows process

z We ask for Requirements specification & review
them
z We ask for Design & review them
ht
ig
z We seek review comments closure
z

yr
We write Test Plans and get them reviewed
z
p
We write TC’s and get them reviewed

Co
z We observe “entry criteria”
z

I T
We observe “exit criteria”

QS
17

Challenge # 5 – Only Test Team follows process

(contd.)
z Does this sound crazy?
z Are we the only ones bothered about processes
(spare that poor SQA!!)
ht
ir g
z Do only we need documents – how will they
code – how will they make changes??
z
p y
People are sidelining us saying “you in testing

Co
know only to crib”……

T
z Should we also stop following process?

I
QS
18

[email protected] 9
Software Testing-Challenges and Strategies 8/12/2004

Strategy # 5 – Only Test Team follows process

z This is more of a “mindset” – we should do
things the way we want to, because we believe

t
in them

h
ig
z More of team building
z Awareness building
yr
p
z Bringing the feel that all part of the Project team

Co
– differentiation of we or they should be stopped

T
z Make sure that we go by setting expectations

I
QS
with our peers, other functions and so on…
z Needs some “cultural” orientation…..

19

Challenge # 6 – Intelligent Testing

z Should we be repeating all TC’s in all Cycles – release
after release? What is the basis for TC selection for

t
“re-testing”?

h
z How do we decide on regression test strategy – what is

ir g
the basis?

y
z How effective our regression testing would be?

p
z How much time we may need to regress defects?

Co
z Does it have dependency on the development team?

T
z Who can tell me what to test and what not while

I
regressing defects?

QS
z How do we equip ourselves on deciding TC’s for
regression?

20

[email protected] 10
Software Testing-Challenges and Strategies 8/12/2004

Strategy # 6 – Intelligent Testing

z Have Regression test strategy for each
feature/functionality – this would also depend on the

t
relationship across features – if possible develop a

h
‘n’by’n’ relationship matrix for all features

ig
z Mark Regression TC’s for every feature – irrespective of

r
defects we found in the earlier rounds
z

p y
Make sure that remaining TC’s selection for regression

Co
test happen just before we start the same – depends on
the “defect fix” details – look for possible areas that

I T
might have got impacted – get a developer or designer

QS
support while doing this
z Take intermittent developer builds of the product and
run through some TC’s – decide on the priority for these
for formal cycles based on result
21

Challenge # 7 – Test Execution Management

z We get a build and start the cycle, within 2 days
we get another!! – set up takes half day off all

t
my test engineers!!!

h
ir g
z TC’s passed one build fails in another!
z When we report a defect, we get a reply “try with

p y
this new build” – no details on the fix!

Co
z The more the #of defects, the more the time we

T
need to complete testing

I
QS
z Testing is the last activity – hence all previous
delays get forgotten – onus of the release slip is
getting thrust upon test teams!!!

22

[email protected] 11
Software Testing-Challenges and Strategies 8/12/2004

Challenge # 7 – Test Execution Management (contd.)

z We have no clue when all the defects would get
fixed, so that we can regress!
z

ht
For non-functional testing we can’t change

ig
builds frequently also we need more time to

r
complete scaling up the product and data
z

p y
We find a lot of non-reproducible defects – we

Co
spend a lot of time in this attempt (reproduction)

T
z We find a lot of defects which gets fixed only in

I
the “next” release

QS
z Some time we get orders “roll back” to previous
build and continue testing!!

23

Strategy # 7 – Test Execution Management

z Have a definite plan for build “transition” – make sure
that all your resources doesn’t move from one build to

t
another same day – provide cooking time for each build

h
to be rolled out to entire test set-up

ir g
z Make sure that development provides the exact tasks or
check-in’s that have gone into every new build – with

p y
details which help us in identification of change

Co
z Follow a statistical defect prediction model (preferably
internal) to predict defects – plan the regression cycle

I T
based on the prediction

QS
z Make sure that when the “baton” is passed to you the
relevant delay is accounted – also demark the “bug
fixing” cycles between your test cycles

24

[email protected] 12
Software Testing-Challenges and Strategies 8/12/2004

Strategy # 7 – Test Execution Management (contd.)

z Analyze the defects from the past – educate the
testing team to recognize valid defects and

t
ignore others

h
ig
z Have a criteria for build reproduction – beyond

r
which the defect to be kept under observation –

p y
during slack time we try to reproduce them

Co
z Have a set of minimum details to be assessed
while reporting a defect – this could be even

I T
feature specific – get this list done by Dev team

QS
z Make sure the Fix Next Release is decided
based on an agreeable criteria to you as well –
stakeholder concept!!
25

Other New Challenges

z Application Integrations
z New Technologies
z Open source models
ht
ir g
z Re-use Models

y
z “Knowledge-able” users – a blessing in
disguise??
p
T Co
I
QS
26

[email protected] 13
Software Testing-Challenges and Strategies 8/12/2004

ht
r ig
p y
T Co
I
QS
Summary

27

New Tester’s Sources…..

z Requirements z User Stories
z
+
Functional Specifications z Customer Scenarios

t
z Designs z Defect Database

h
ir g
z Product Descriptions z Benchmark Reports
z Use Cases z Research Reports

p y z Beta Feedbacks

Co
z Customer Support DB
z Developer Communities

I T z International Forums

QS
28

[email protected] 14
Software Testing-Challenges and Strategies 8/12/2004

New Tester’s Career Path/Options

t
Test Engineer

h
r ig
Automation Expert Platform Specialist

Use Case Specialist

p y Environment

Co
Specialist

Test Lead

Test Architect

I T Performance Analyst

Human Factors
Specialist
QS Test Manager
Domain
Specialist

29

New Tester’s Skill sets

z Equivalent to Developers
z Need to know the Programming Skills
z Automation Skills
ht
ir g
z Understand the “User”

y
z Intuitive

p
Co
z Creative
z Know/Appreciate “Systems Thinking”
z ……….
I T
QS
30

[email protected] 15
Software Testing-Challenges and Strategies 8/12/2004

Your Questions Please……..

ht
r ig
p y
T Co
I
QS
31

ht
ir g
p y
T Co
I
QS
THANK YOU

32

[email protected] 16