QAGuild Weekly Article

"+title+"

Mon, 02 Nov 2009 00:39:32 -0800

Dr Mike Bartley
Test and Verification Solutions Ltd (TVS)

How to Build and Maintain a Successful Outsourced, Offshored Testing Partnership

Somebody recently asked me to review a course on �Building a Great Team� - where is the section on outsourcing? I asked. In my opinion, outsourcing should always be a consideration when trying to maximise the performance of a team. In this article I will consider this in the context of software testing. I will mainly consider software testing that is both offshored and outsourced1, although it should also be applicable to software development and to companies that just use outsourcing or offshoring. The article is based the successful application of outsourcing at ClearSpeed Technologies plc (which I�ve described elsewhere -see [1]) and in advising other companies on how to outsource their software development and testing.

Outsourcing = is the use of components or labor from external suppliers.
Offshoring = the relocation of some of a company's production, services or jobs overseas.

The objectives for offshoring are evolving. Initially companies looked for cost savings. Then companies looked externally for extra resource (for example during the Y2K crisis). Companies are now also looking for strategic benefits and better defined business impacts from their offshoring relationships. My aim in this article is to demonstrate how such benefits and impacts can be achieved with minimum risk.

To attain the strategic benefits you seek, you must first identify them. Let�s consider the benefits claimed for offshoring by managers of technology SMEs (Small/Medium Enterprises) in [2]:

To accelerate the maturation of new products: �The internal skills of rapidly developing products that meet market needs is augmented by offshored teams who have the patience, precision and perseverance to evolve young products into mature ones� (Jajiv Dholakia, Cenzic Inc.)
Agility and cost savings: �on average we are able to deliver projects at a much faster rate and save at least 30% cost to our clients� (Jing Liu, CEO, Enter Suite)
Better software through additional resources: �The additional talent and resource that we can carry allows us to develop better software than our competition can afford in the same period� (Scott Allan, Symbol technologies Inc.)
Concentrate on core competence: �Businesses should focus on clients and IP which satisfies their need. Everything else should be outsourced to those whose core competency is that business� (Bill Widmer, President and CEO, g8solutions)
Ability to undertake one-off projects: �we were entering a virtually untested market with a new product. The US team focused on the current roadmap while I was able to develop the new product at a rapid speed and at a low cost� (Gopan Madathil, Founder, TechCoire)

The strategic goals must also be agreed by a suitable forum. When I started on the offshoring route at ClearSpeed, I (as the Quality Manager) agreed the following prioritised objectives with the CFO and Engineering Manager.

To improve company focus.

To improve company resource flexibility and by doing so, reduce time to market.

Quality improvement.

Cost variablization2.

Cost reduction.

The above goals were shared with the offshore supplier. This gave us a common understanding which helped the day-to-day management of the projects. We also derived metrics to measure our progress against the goals. For example, we used Defect Detection Percentage (DDP) to measure the improvement in our testing (see [3] for an explanation of DDP). The time to market and the cost of testing relative to development were also tracked.

The next major decision regards on your road to offshoring is to decide what type or stage of testing to offshore. There are a number of considerations to take into account.

Amount of documentation: In order to test your software your offshore supplier will need to understand it. This will ultimately require good documentation.

Domain knowledge: How much domain knowledge will the tester need and how easy will it be to find or train testers with that knowledge?

Complexity of the environment: How complex is the test environment and how easy will it be for the offshore test organisation to re-create it?

Development process: Testing within the V model lends itself well to offshoring. Offshoring of agile development less so as it is based on nimble teams with good communication and rapid change.

Stability: How stable is the software that you are asking the supplier to test.

Level of interaction: To what extent will the offshored test team need to interact with the development team.

Level of independence: NASA uses an independent V&V team to improve the quality of their software (see [7]). An offshore test team allows you independence along three dimensions: managerial, financial and technical.

Taking a customer perspective: An offshore team can be used as a �friendly� customer, helping to identify all the problems that your real customers would otherwise find.

IP protection: This is always a major concern in outsourcing and the concern seems to deepen with offshoring.

Sign-off criteria and incoming acceptance: You will need to sign-off the work performed by your supplier through some sort of incoming Quality Assurance (yes � testing the tester!).

Now consider how the above affect your decision on what to outsource:

Unit testing:

This will probably require you to release source code which may lead to IP protection issues.

Unit code may not initially be stable and may not be well documented.

The offshore organisation can act as an independent V&V team.

The test environment for unit testing is not usually over-complex and the domain knowledge requirements tend to be lower.

Sign-off could be defined through structural coverage goals.

As an example, I successfully offshored the unit testing of a new library of memory management functions. The functions were developed using agile techniques in close collaboration with marketing and two lead customers and, once the specification was stable, we offshored the unit testing. We had an existing, trusted relationship with an offshore partner who first produced a test specification and then tests against that specification. Finally, they topped up their tests to ensure our structural coverage goals were met. Our incoming signoff consisted of signing off the test specification, reviewing a selection of actual tests against that specification and re-running the tests to check the coverage.

System testing:

In my experience by the time you reach system testing the software is more stable and the chances of some documentation are improved.

If you are following the V-model of development then the appropriate documentation may have been available for a while so that the offshore organisation could have already written test specifications reviewed by you.

The test environment and domain knowledge required can be complex.

The level of interaction should be reduced compared to earlier stages of testing.

The offshore team can perform the testing independently and can take a customer perspective in their testing.

IP protection issues can be avoided by releasing the binaries rather than source.

Testing the tester is harder as an objective measure of system test quality is more difficult - but it is not impossible. A system test specification and reviews of the test scripts for example. The test results should be also made available for audit.

Some of my most successful offshored testing has been in system testing at ClearSpeed. We had an automated test environment which gave very high unit and integration test coverage, and some system testing. We then offshored all of the system testing that was impossible or not cost-effective to automate. The offshore team was encouraged to act as a customer with no prior assumptions and not to use any work-a-rounds offered by �helpful� members of the development team!

There are numerous other types and phases of testing where the above considerations can be applied but space does not allow for here. For example: specialist testing (such as security testing) where the outsourcing arguments are very strong but expertise rather than price will be most important; compliance testing (where you are required to demonstrate conformance to a particular standard) where there is a strong argument for outsourcing to specialists in that standard; release or acceptance testing has similar arguments to system testing discussed above. And the list goes on�

You also need to identify your preferred outsourcing model. [5] describes three types of service deal related to the scope and complexity of the offshoring contract and your strategic objectives.

Efficiency: These deals tend to focus on cost efficiency, improved resource agility or greater organisational focus on core skills. The contract terms concentrate on service levels. So, for example, you may be outsourcing unit testing currently performed by developers to free them up for additional projects and the service level might be around test specification, structural coverage and time to completion.
Enhancement: These deals are more to do with process improvement. For example, you may outsource to a specialist test company to improve your security testing.
Transformation: These deals aim to improve your competitiveness through innovation in the relationship. So, for example your goal may be to reduce time to market or improve the quality of your software (through reduced DDP for example). Your outsource supplier would need to act in a consultancy role as well as a service supplier.
Obviously the above three imply an increasingly complex relationship with increasing levels of trust. [5] goes on to consider the appropriate commercial models for the deal and this is summarised in Table 1.

Table 1: Commercial models, service deals and risk

Commercial model	Description and discussion of risk;	Recipient Risk	Provider Risk
Cost-plus pricing	The recipient pays the providers costs plus a percentage. The provider is guaranteed cost recovery. The recipient is not guaranteed service levels.	Medium	Low
	Appropriate for efficiency deals.
Fee-for-service pricing	Price is based on the amount and/or quality of the work actually delivered. The provider�s costs are not covered unless service levels are met. The recipient costs are not entirely predictable and service levels are not guaranteed. Appropriate for efficiency and enhancement deals.	Medium	Medium
Fixed price	Defines a specific service level and a price for achieving it. This is higher risk for the provider. The recipient has lower risk as prices are capped. Appropriate for efficiency deals.	Low3	High
Shared-risk/reward pricing	This involves a flat rate with additional payments for achieving specified outcomes. Appropriate for enhancement and transformation deals.	High	High
Business outcome achievement	The provider only receives payment for achievement of specified business outcomes. The receiver has no guarantee that expected outcomes will be achieved. Appropriate for transformation deals.	Medium	High

You also need to consider your outsource location: on-shore, near-shore or offshore. This decision should be made in the context of your chosen service deal. I would also recommend considering multiple suppliers. In my experience this brings a number of advantages.

Using suppliers with different types of skills.

Using one supplier for an efficiency deal and another for an enhancement or transformation deal.

Allowing the suppliers to competitively tender for new work (rather than becoming increasingly dependent on a single supplier).

Now that you understand what you want to offshore and your preferred commercial model, the next consideration is your readiness for offshoring. A number of attempts at offshoring fail because an organisation has bad existing internal development practices: offshoring your chaos will just generate additional chaos! So, how do you measure your readiness? There are formal measurements such as CMMi (see [4]). Table 2 is taken from [8] and shows the national differences in CMMi take-up. It is also interesting to note from the table the differences in maturity from the countries more likely to be supplier to those more likely to be provider. In my experience providers tend to go for CMMi for a mixture of technical, management and marketing purposes.

Table 2: Number of Appraisals and Maturity Levels Reported to the SEI by Country (where total appraisals is above 20)

Country	Total App- raisals	Maturity Level Reported
Country	Total App- raisals	1	2	3	4	5
Argentina	47		31	10	2	3
Australia	29	1	8	4	2	4
Brazil	79		37	31	1	8
Canada	43		10	18	5	3
China	465	1	103	293	18	34
Egypt	27		12	11	2	2
France	112	4	67	34	1	2
Germany	51	7	27	7	1	1
India	323	1	11	127	22	151
Japan	220	16	64	88	13	15
Korea, Republic Of	107	1	31	48	11	7
Malaysia	42		15	24		3
Mexico	39	1	18	13	3	4
Spain	75	1	49	21	1	3
Taiwan	88		60	25		2
UK	71	3	36	24	1	2
US	1034	25	365	347	21	114

Most organisations will not want to gate their offshoring on such formal appraisals. In my experience, the minimum set of processes that your organisation will require to outsource software testing is:

Requirements management

Change tracking and management

Project planning, monitoring and control

Configuration management

Build, test and release automation

Validation and verification processes

You now need to select your partner. There is no magic in the selection process:

Identify potential suppliers.

Define your selection criteria.

Create a short-list.

Perform an in-depth appraisal of your short-list (including a visit if possible)

Contract negotiation.

Perform a pilot project with your selected supplier (or suppliers)

The major issues in the above list are identifying appropriate selection criteria and exchanging contracts. Below are some example selection criteria

People: Skills profile; Academic background of staff; Staff training statistics.

Retention: Staff attrition rates

Process: Process maturity; CMMi and other process maturity measures attained. You should consider how well matched the relative process maturity between your two organisations are. For example, do the outsource organisation processes require deliverables and maturity on your side that you cannot deliver on?

Quality: Quality processes employed; how well do they match your quality processes?

Project and risk management: What management processes do they follow? How well do they fit with your preferred management methodology?

Company profile: Financial stability; historic financial data strategic goals for the company.)

Technology: Domain knowledge/experience of the technologies employed by your company.

Cost: What are the typical pricing models? How are changes managed from a cost point-of-view? Recent cost change history.

If you are offshoring then your selection criteria will need to reflect circumstances in the providers� country. For example, if the destination is India then staff turnover or retention rate should be a major consideration.
Regarding contractual negotiations, I have found it helpful to have two documents: a Master Services Agreement (MSA) and a Service Level Agreement (SLA). The MSA covers the legal, financial and contractual issues (such as confidentiality, ownership of deliverables, assignment of rights, indemnification, warranties, liability, changes in personnel, subcontracting, termination, penalties for not meeting agreed service and delivery targets, etc.). The Service Level Agreement (SLA) should cover the service and support goals and is strongly affected by the type of service deal that you have (efficiency, enhancement or transformation). The MSA and SLA are complex legal documents which I will not attempt to cover in detail in this article.

You are finally ready to execute your offshored software testing project and this is where it starts to becomes difficult! If you have followed the process I have described above then your chances of your success are significantly improved. However, any offshored development is inherently risky. First there are the problems of managing a distributed team (communication issues, knowledge management, change management, etc). Then, according to Ralph Kliem in [7], in offshoring there are also �the unique challenges posed by geographical, cultural, and other differences�. There is not enough space in this article to go into details of how to manage offshored projects but I will highlight the main items to consider. Firstly, you need to distinguish management and governance. Management is about responsibility for specific decisions; Governance covers decision-making processes: the who, the how and the when. It is important to define the governance rules to allow your offshore team to know when they are empowered to make decisions and how to make good decisions.

Good governance involves at least the following

Established standard decision-making procedures.

Well-understood processes for handling exceptions.

Decisions made quickly at the correct level against well-defined criteria

Decisions get recorded.

And good governance is helped by the following

Having agreed strategic objectives.

Formal communication methods which record project status and decisions.

Controls & records to ensure governance procedures are followed.

Regular meeting of the stakeholder group.

Good management means the usual good software project management practices plus

Communication

Who can communicate with whom? And how often?

Formal vs. informal communication.

How does the communication get recorded?

What happens on a miscommunication?

Knowledge management

Getting up the initial learning curve and staying there through staffing changes

On-going knowledge management

What is the process for asking a question?

What is the process for recording an answer?

Audit trails

What trails do you need? Decision making, Execution history, Timesheets?

You need to save this data quickly and cheaply but what about access time?

How strict does your audit need to be? Are their any legal requirements?

What happens on a miscommunication?

Well-defined incoming QA procedures

In summary, there are many benefits to offshoring your software testing, especially if you aim to build a competence based rather than transactional outsourcing relationship. However, the risks are also high and in the above I have laid out a process to improve your chances of success.

Agree the strategic goals in an appropriate forum and share them with your provider.

Decide what type or stage of testing to offshore.

Identify your preferred outsourcing model and location.

Assess your readiness for outsourcing.

Select your partner or partners.

References

�How to Boost your Productivity through Outsourcing�, Mike Bartley, Software and Systems Quality Conference, London, Sept 2008

�Happy About� Outsourcing�, Mitchell Levy, January 2005

Identify your preferred outsourcing model and location.

�How to measure test effectiveness using DDP (Defect Detection Percentage)�, by Dorothy Graham et al, Grove Consultants (available from http://www.dorothygraham.co.uk/downloads/generalPdfs/DDP_Tutorial.pdf)

�Multisourcing: Moving Beyond Outsourcing to Achieve Growth and Agility�, Linda Cohen and Allie Young, Gartner, Inc.

�Managing the Risks Of Offshore IT Development Projects�, Ralph Kliem, Information Systems Management, 2004, Vol. 21 Issue 3

�The Role of Independent V&V in Upstream Development Processes�, Steve Easterbrook, NASA/WVU SW Research Lab (http://www.cs.toronto.edu/~sme/papers/1996/NASA-IVV-96-015.pdf )

�Process Maturity Profile�, SEI, 2008 (www.sei.cmu.edu/appraisal-program/profile/pdf/CMMI/2008MarCMMI.pdf)

About the author

Dr Mike Bartley gained a PhD in Logic at Bristol University and an MSc in Software Engineering and MBA with the Open University. More recently he has added ISEB Practitioner Testing Qualifications in both Test Management and Test Analysis. Mike has been involved in software and hardware development for 20 years, including outsourcing, and regularly writes articles and delivers conference papers. He recently established his own consultancy to help companies in product QA and outsourcing.

"+title+"

Mon, 26 Oct 2009 05:42:37 -0700

R. Rajan

CHANGE LEADERS

INTRODUCTION:

Successful change implementation should be one of the key objectives of all champions in any organization. It is not easy to reach the �Satisfied State of Successful Change Deployment� many times, as it is more of perception based and does not have very clear measure. Many changes start with a grand ribbon-cutting and disappear at the very early phase of the change management cycle. Gartner 2008 survey says; almost one in three project implementation fails. The reasons for a project failure could be many but the ultimate ownership is with the Top Management. The approach should be always �Top-Down�. This article will provide insight in to a common frame work for successful Change implementation. It is very important that more emphasis should be given to people for any change implementation as the benefit of the change must result in people using a much better systems that provides value to all the stake holders, with Customer being the center of the nucleus.

Why Change implementation fails?

Let us look at a very simple example of a change that we want to bring about in our daily life. The change is that we need to walk for 25 minutes daily for five days in a week and lose 150 calories of energy, to keep you fit. How would you go about �Creating the Sense of urgency�, which will make you to implement this change?

One will accept this change, fundamentally after getting answers to lot of questions. Why should we walk and strain our self? What will happen if I do not walk? Will the walking be instrumental in achieving all my goals? You can answer this question in multiple ways. A large collection of data showing percentage of people dies in heart attacks with varying age group might be required to accept that it�s worth taking efforts to walk. The need of reducing the fat by burning calories stored in the body must be triggered psychologically. You can have summary of data on people under obese category and difficulties they experience, to convince that this objective is very important.

You can ask a doctor to explain the need of walking daily. These are some of the ways of creating sense of urgency. You will have to prepare your system to take up this change for implementation. Remember that this is only a starting. You have not accomplished anything yet.

Few of the reasons for failure in implementing Changes, felt by many organizations are:

Failing to create a sense of urgency.

Allowing too much of complacency.

Failing to create a powerful guiding team.

Absence of a strong vision.

Not understanding the basic principle of communication.

Not able to create a short-term win.

Change Deployment Frame work:

With almost two decades of experience in change implementation across different industries and after conducting an extensive research on the published information available in internet, I was able to conclude that managers who are supposed to play �Change Leaders� role, do not follow a structured framework for change deployment. The term change deployment refers to any change undertaken to improve the existing state with the objective of delivering excellence. You can think of implementing a ERP system implementation, payroll system implementation, HR system implementation, VOIP communication system, Border Gateway protocol implementation, Lean six-sigma implementation, Marketing system implementation, Travel process enhancements, Testing process productivity improvement, Reuse management etc.,: You need to have a frame work which must help the organization to implement change.

I was inspired by the book �Leading Change� written by John P.Kotter for developing a comprehensive frame work for change implementation which can facilitate creating Change Leaders for contributing a very high value to the organizations they work with by achieving a very high success rate. The following are the pre-requisites for adapting this frame work.

Rule 1: Change is a Top- Down approach and Top Management needs to get involved.
Rule 2: Top management must create a very highly empowered team and believe that this team can win.
Rule 3: Managers need to understand that people are the most important asset of the organization and without involving them, change cannot be implemented.
Rule 4: Managers are responsible to create a Sense of urgency and sustain it until the change is disseminated in to the culture.
Rule 5: Managers should know that people will accept the change only when you show them why you
need it. To create an acceptance of the people, you need to follow a cycle �SEE-FEEL-CHANGE�.
Rule 6: As change leaders, you should be able to articulate a one minute vision of the Change.
Rule 7: The change should not be pursued further, if a short-term win cannot be created and all the stake holders accept it unanimously.

The successful change deployment framework is the one which will facilitate a very high level of people � process interaction. As an organization lot of time needs to be spent for preparing people for change and conduct marketing campaigns inside the organization to spread the need for change. We need to keep talking to people as the only thing that will insinuate in to their minds is the �Repetition�. Managers need to create a visual picture of the where do want to reach and what financial benefits will be achieved, after the change is implemented. To reach the end state of a change implementation, which is touching the heart of people so that the change becomes a daily routine, a lot of training and talking to people are required.

The 8 Stage Change Implementation Frame work

The frame work proposed in this article has eight stages as illustrated in the picture titled �Change Leaders Frame Work�.

Stage 1: Creating Sense of Urgency:

Managers are responsible for Pushing up the sense of urgency levels always to ensure that the change is implemented very effectively. The following are some of the ways by which the sense of urgency is pushed-up.

Create a crisis by explaining and detailing the financial loss.

Eliminate obvious examples of excess.

Set targets so high that cannot be met by conducting business-as-usual.

Evolve broader measures of business performance (no functional level performance goals measures).

Expose financial performance to more employees.

Insist people to talk regularly to unsatisfied customers, suppliers & disgruntled stake holders.

Bombard people with more information on feature opportunities and the current inability to pursue.

Sources of complacency

Stage 2: Create a Guiding Coalition:

It is the responsibility of Top Management to form a guiding coalition with appropriate positional power, empowerment to take decisions for implementing the change successfully.

Four key characteristics for effective guiding coalition.

Position Power
Expertise
Credibility
Leadership

Qualities to avoid

Egos & Snakes (people who can create enough mistrust to kill team work)

Building Coalition that can make change happen can be done through a very structured three step process.

Step 1.Find the right people:

With strong position power, broad expertise & high credibility.
With leadership & management skills.

Step 2. Create Trust:

Through carefully planned off-site events
With lots of talks and joint activities.

Step 3. Develop a common goal:

Sensible to the head
Appealing to the heart.

Stage 3: Developing a vision & Strategy:

This is an important stage wherein the Change Leader creates a visual picture of what would be the future, in financial terms after the implementation of the changes and document the vision in explicit terms which can be articulated within one minute by any team member. The visual picture is very important as it can reinforce quite strongly in to the minds of people and the buy-in becomes an easy task at a later point of time. The Vision thus articulated will meet the following characteristics.

Imaginable Desirable Feasible

Focused Flexible Communicable

The Change Leader then creates a strategy as shown in picture 3 �Role of a Leader and Management� (The Logic of how the vision could be achieved) and Budgets (plans converted in to financial projections and goals).

Role of a Leader and Management

Stage 4: Communicating the change vision:

Communicating the vision is an important stage as the change leaders will have to ensure that the vision reaches across a wider cross section of people, more importantly it creates an impact in the minds of people. We need to conduct campaigns within the organization and explain the importance of change with clear financial terms. Communication will be very successful only when it is repeated multiple times.

KEY ELEMENTS IN EFFECTIVE COMMUNICATION:

Simplicity: (do not use jargons)

Metaphor, analogy & Example: A verbal picture is worth than a thousand words.

Multiple forums.

Repetition: (Ideas sink in deeply only after it is heard many times).

Leadership by example

Give-and-take (Two way communication is always more powerful).

Stage 5: Empowering Employees for Broad based Action:

This is the phase where in the obstacles are dealt effectively that block action, especially disempowering bosses, lack of information, the wrong performance measurement, reward system and lack of self- confidence.

WHAT WORKS

Finding individuals with change experience who can bolster people�s self-confidence with

Recognition and reward systems that inspire promote optimism and build self-confidence.

Feed back that can help people make better vision-related decisions.

Re-tooling� disempowering managers by giving them new job that clearly show the need

WHAT DOES NOT WORK

Ignoring bosses who seriously disempowering their subordinates.

Trying to remove all the barriers at once.

Giving in to your own pessimism and fears.

Stage 6: Generating a Short-term win:

In this phase, the change leader will produce sufficient short-term wins, sufficiently fast, to energize the change helpers, enlighten the pessimists, defuse the cynics and build momentum for the effort.

The role of a short-term win:

Creating a short-term win is extremely important and any change effort without this will not sail through smoothly. The change leaders need to understand that the �short-term win� will facilitate buy-in from those who oppose the change.

The key roles of a short-term win are:

Reinforcement for the effort needed.

Opportunity to relax for few seconds & celebrate.

It is a test of vision against concrete conditions.

Will make blockers difficult.

Will retain the essential support of the bosses

Managers must be able to create a short term win which will be:

Visible.

Unambiguous.

Clearly related to the change effort.

The change that does not go through this face will be forced to be withdrawn.

Stage 7: Consolidate Gains and Produce More Change:

This phase will bring in more momentum to the change effort, using the short-term win gained in the previous phase. Since a visible short-term win was shown to many stake holders, now people who opposed for the change will also fall in your favor as you will get tremendous support from management. Every own will understand and know now that this change effort will bring in lot of benefits and visibility, thus will start providing their support whole-heartedly.

It is the time for you to take decision by removing all unwanted interfaces that will prevent you speeding up the change implementation process. Remember that you might be required to remove people who are not inclined to put their efforts to implement the change being pursued. You will add additional resources/budgets to ensure that the speed of implementation is stepped-up gradually.

As a change leader, you need to play a key role here by pushing-up the urgency levels in this phase so that the team does not become complacent.

Stage 8: Anchoring New Change in to the Culture:

This phase is a standardization phase and usually comes at last in any change implementation. The new process need to go and sink in to the minds of people so that it becomes a routine. To make this happen, the change leader needs to do a lot of talking to people, create a visual story about this change and benefits, conducting quizzes and workshops, designing a very strong and robust feedback mechanism for swift action and changing the people who are relentless.

Precautions:

Cultural factors come last, not first: Alterations in norms comes at the end of the transformation process.

Depends on results: New methods usually sink in to culture only after they work and superior to old method.

Requires a lot of talk: Without verbal instruction, people will be reluctant to new methods.

May involve turnover: Sometimes the only way to change culture is to change people.

Makes decision on succession is crucial: The old culture will reassert, if promotion processes are not changed to be compatible with the new practices.

Conclusion:

Organization need to mentor and develop more leaders to effective contribute to the bottom line and take the organization forward. Managers need to come forward without waiting for instructions and volunteer change initiatives, lead them successfully. The frame work for change deployment recommended in this article will work only if all the stages are meticulously followed, without skipping any stage.

About the author

R. Rajan M.S, MBA, MSC, CISA, SIX SIGMA BLACK BELT, LA ISO 9001, 27001 & QS 9000, LA & BCM. With 19 years of experience in Change Leadership Role from a wider cross section of industries such as Software Testing, IT Infra structure, IT Networking, Information Security, BPO, Portal and Manufacturing. Expertise includes Process Design, MBWA, BPR, Lean Six sigma, Process Automation, IT Network design, Information Security implementation, ISO 27001 Policy deployment, PCI Practices deployment, IS Audits, ISO 9001 Implementation and QS 9000 automotive standard implementation.

"+title+"

Sun, 18 Oct 2009 23:21:24 -0700

Soon Hui
Esteem Innovation

Why Developers should Do Testing

It suddenly occurs to my mind that asking developers to play tester's role once in a while is a good idea. Here are whys:

Enable the developers to gain system knowledge

Improve the developers writing skills

Improve software quality.

Make the developers better software engineer

Enable the developers to gain system knowledge

Typically on a large team, individual developers are each responsible for a component of the system. Testers, on the other hand, need to be at least rudimentary familiar with the whole system. Letting the developers test the system will get them to familiarize with the whole system, enabling them to see the big picture.

Improve the developers writing skills

Developers hate writing. Ask them to fill out form hurts them. Ask them to write report disgust them. Being a developer myself, I saw a lot of developers who have a hard time explaining his ideas in words.

But testers need to be reasonably good at writing, because the testers must be able to

Write down the steps to reproduce the problem

Explain what happened and what should happen.

Make sure that other people can follow their bug report

If the developers come up with a bug report that no one else can understand, they would be asked to clarify their meaning, and this actually trains them to be a better writer next time. Repeat this a few times you will find that their ability to write improves.

Instead of sending the developers to communication courses, ask them to fill in bug reports!

Improve software quality

The whole point of testing is to improve software quality. Testers do find bugs, but after finding bugs for so long their bug count will start to decrease because the software starts to get immune to their way of testing.

Getting the developers to do testing will provide a fresh approach to testing, and hence uncover more bugs. It is also a form of dogfooding. The more the developers use the software, the more they feel the pain of end users. In this way they can design better features, better UI next time. The software quality improves as a result.

Make the developers better software engineer!

Testing can develop a developer. A developer's main job is to construct, whereas a tester's main job is to destruct. Different mentality, which is why sometimes a good tester may not be a good developer, and vice versa. Developers have been concerning about constructing, constructing, constructing the code all the time. Sometimes this makes them oblivious to the edge cases and the holes in their program. After all, the developers are paid to make sure that their code is matched to the specification. But what about things that are not found in specs? What about the situations that no one has thought of? Good testers must be critical minded, or else they would miss a lot of edge cases. Won't it be good if the developers can be critical minded when constructing their programs instead of throwing their half-baked code over the wall for the testers to test?

Not only can that, testing job help developers to realize the value of unit testing. Get the developers to wear the tester's hat, and ask them to make sure that a particular brittle feature is always working, let them do the mundane job of testing a feature over and over again, until they realize that automated testing is the only way to go. Sounds like a good way to promote unit testing (although I haven't try it).

About the author

Soon Hui is a .Net developer working in Esteem Innovation, a software company that creates desktop applications to help structural engineers in drafting, analysis, design and detailing

"+title+"

Mon, 12 Oct 2009 01:19:39 -0700

Waseem Ahmed
Standens Ltd

Learn �Proactive Approach� Which helps us in saving costs

In previous edition, we discussed about Lean Quality Manufacturing.

When I started writing this article, I thought to myself that I am not a novelist or a good story teller but all I knew was, it�s a good opportunity to convey you some of my thoughts through this article. Finally, I decided to write something that I have learnt in my career and always believe on it �Proactive Approach� which always helps us in saving costs. You can / may use some thoughts and ideas from this article and apply the theme to improve your processes and systems and save Standen�s money.

Transformation of Reactive thinking to Proactive thinking is as challenging as to bring revolution in a state / territory. I have a strong belief that deeper we get into thinking positively; more opportunities we will discover for improvements. In this issue I will show you some examples of reactive and proactive thinking (R- Reactive and P- Proactive)

R - PFMEA (Process Failure Mode Effect Analysis) is only focused as a document required for customer PPAP.
P - PFMEA is a risk assessment tool to improve manufacturing processes.
R � Lack of importance to voice of customer.
P - Every customer complaint is taken as an area for improvement. Focus is not only detection but also prevention of such problem recurrence in future. Decisions are now made using multiple brains from cross functional team
R - Doing things because the procedure asked for it
P - Review the procedures and align it with your current practices and eliminate non value added (NVA) tasks. (NVA activities are those tasks which are not adding any value to customer / paid by the customer)
R - Written procedures are just like lengthy story books and novels. Why, because, we want to document every single action that we perform in our procedures
P - Procedures should be user friendly and not to be written more than 1 or 2 pages otherwise, people will loose their interest in the subject. Flow chart is a good example of shrinking the matter of procedures
R - ISO / TS Auditors are taken as police officers and their focus is to find as many problems as they could.
P - Audits are not about fishing non-conformances but a method to check the compliance and effective implementation of Management Systems audits using process approach
R - Every thing is recorded on paper because it�s required by the auditors
P - Again, if it�s not adding any value, don�t just do it because it�s required by auditors. �Work with the system not for the system�
R - Every single activity of manufacturing process needs verification and approval by QA
P � Process capability studies should be done and their control charts to be monitored frequently. Once we have good process capability (Cpk >1.33), we can reduce verification / inspection activities and their frequencies. Until we achieve the desired situation QA has to play a vital proactive contribution in respect to finding issues, QA should work with production to resolve the issue not just recording it in a piece of paper and think the job is done.
R - Don�t worry about that customer because we don�t build lots for them
P - Customers have equal values. Their voice is important to their suppliers and most importantly, during the bad phase of economic crises these small customers become a mean of survival for us
R - This is not my job, go and talk to manufacturing
P - �Can Do Attitude� and positive team works resolves most of company�s issues
R � We have goals and objectives because it is required by ISO / TS standard.
P - Goals and objectives are the performance measurement of company�s progress. How do I know that my department is doing a good job? Answer: Through the performance measurements. These objectives are synchronized with company�s direction
R - Keep every single record for indefinite time period just because it�s a ISO / TS requirement
P - ISO / TS requirement is to identify the records you want to store and define the retention time for their disposal. Otherwise, if you keep it storing for indefinite period of time, then, very soon, you need more space for record storage
R - Found the Quality problem but still sending those parts to customer. Why, because customer will complain if they catch this issue. We will respond to such complaint that�s not a big deal.
P - If you find a problem in-house, don�t ship it to customer and fix the problem then and there. Also check the inventory to make sure it�s all good otherwise re-work on them
R - Do in-coming inspections for raw material and all components to verify if they are meeting our specs and requirements.
P - Trust your suppliers and develop strong relationships with them. Increase frequency of supplier assessments instead of doing full time incoming inspections. During production, operators But if we find any issues with supplier products we will charge them back with warranty claims. Let them be responsible for their products.
R - We need more inspectors for inspection activities
P - Inspectors / operators are human beings and they have potential to make mistakes. Rely on your process performance and make it robust. Apply every possible method to make it error proof for defects.
R � Long meetings indicate that we are busy
P - Make the best use of people�s time. Stay focused on the subject. Team facilitators to take control of such meetings and don�t let people go out of agenda. Time management is MUST to follow
R - Always complaining about less manpower.
P - LEAP (Lean Enterprise Assessment Program) is an effective tool to make the best use of current manpower and shop floor space. LEAP contributes to ROI and sales.
R - Based on experience, people jump to conclusion too fast for customer complaints
P - Using 8D �Problem Solving Activity� we eliminate the root causes, take corrective actions and actions to prevent its recurrence. Prevention always includes any possible mistake proofing activities

About the author

Waseem Ahmed has more than 12 years experience in Quality Engineering Arena. Waseem is presently working in Standens Ltd, a well known name in leaf spring manufacturers in North America as their QA Manager. His qualification includes ; Bachelors in Mechanical, ISO / TS Lead Auditor, ISO-14001 Lead Auditor, ASQ SSGB, ASQ CQA.

"+title+"

Thu, 01 Oct 2009 01:45:11 -0700

Waseem Ahmed
Standens Ltd

Layered Process Audit (LPA).

In this edition, we will discuss about Layered Process Audit (LPA).

Fundamental to improving process control, is verifying that critical process elements are compliant with requirements on an ongoing basis. During the past few years several automotive parts suppliers have implemented Layered Process Audits (LPA). LPAs have been mandated by Chrysler and strongly recommended by General Motors and Ford. Many OEMs and Tier 1 suppliers are considering directing their suppliers to implement Layered Process Audits. LPA is one of the most powerful strategies to take a good supplier and make them better; or to take a great supplier and keep their quality indices improving. Certainly, improvement in customer quality levels is important to the supplier; but for the supplier itself, this benefit is just the tip of the iceberg.

�Improvement in quality levels is important, but those benefits are just the tip of the iceberg.�

Layered Process Auditing is an ongoing chain of simple verification checks, which through observation, evaluation and conversations at the work stations, assure that key work steps are being performed properly.

LPAs for any given work station are performed by different layers of management and various staff on a set schedule. This ensures that each process is viewed by many sets of eyes and all levels of management. LPA helps eliminating human error and ensure that parts are produced right the first time. Since the checks are repeated daily, weekly, monthly and quarterly by all layers of management, it�s likely that process errors will be found sooner. LPA proactively minimizes process variation and the result will be evident in process, product and financial improvements, e.g., defective parts per million, control charts, productivity, overall equipment effectiveness, scrap and rework cost etc. I am confident that Standens will embrace LPAs and will find that the payback will be very significant.

LPA facilitates two-way communication between management and floor personnel. These interactions strengthen trust and demonstrate shared interest in work being done right. How can we meet the requirement of zero-defects? Simply with �zero� shipment of non-conforming products. How can we prevent shipping bad parts to customers? Need to develop a culture where every person is working towards �right the first time, every time.�

LPA is never being counted as a detection control but at the same time error proofing is not always shielded from variation and failure. These devices can be misaligned, damaged, mis -calibrated or even turned off. Also, human error can undo almost any system or safeguard.

Initially QA always voluntarily take ownership for LPA implementation and execution but delegates ownership to Manufacturing once it�s implemented across the shop-floor.

The focus in LPA is checking items related to known problems and cause factors linked to high risk problems. These are items that vary over time and lead to undesirable consequences. There might be some exceptions, but most LPA checks look at �inputs� to the process, that is: equipment settings, condition of tooling, craftsmanship and work sequence. Like other process audits, LPAs verify the details of how the process is performed.

LPA usually takes 10 to 15 minutes to complete an audit. Intent of LPA is to fix issues immediately.

�Who would you rather find an error � your supervisor or your customer?�

LPA show respect for operators by giving them feedback � feedback that they are complying or not complying. LPAs aren�t designed to catch workers making errors. The 15 minute window per shift allocated to LPA is way too small for that. What really changes employee behavior is when they do things right and are recognized for it. People do what gets measured; and employees respect what you inspect.

Improving processes will improve product Quality. Improving product Quality improves customer satisfaction. When customer starts building their confidence and good relationship with us, it will open new opportunities for new businesses.

Note: Next operation is line 1 for LPA implementation.

"+title+"

Sun, 20 Sep 2009 23:08:32 -0700

Pentest Limited

Database Security; why expensive tools may not be the whole answer

The world of Information Security is a fickle one, dominated by tools with fantastic capabilities or at least that what it says on the box. So it�s not surprising that when shifts occur within this industry, the tools and their associated claims are not far behind. The current shift is a move to the data, more specifically to the database and it is in this area that we see a rapidly growing selection of security tools. Most organisations have been aware for some time that the business risk posed by internal threats to sensitive data and supporting systems, in particular those posed by application and technical staff who have access to the most critical and sensitive corporate data, have been largely ignored.

The promise of compliance creates a compelling case for many of these tools (which vendor�s brochure does not mention SOX, PCI, HIPAA, etc.?). This is all the more understandable in the financial or any other highly regulated industry sector where reputations, bonuses and even careers can be lost by one failed audit.

The problem is that no matter how carefully you review, assess and then select a security tool, it is only part of the picture and if you don�t have the whole picture or if the tool does not easily fit into the picture you could be left with some significant gaps. Worst of all, you may not know what the gaps are or even that you have them.

So, are we saying that security tools are not fulfilling their purpose whilst being yet another drain on your ever decreasing budget? Absolutely not! The right tool is an invaluable part of most solutions and where they can deliver a measureable benefit within a defined budget and clear timescales they can prove to be a real life saver. The point is, where and when do they deliver this benefit.

The best way to answer this question is to decide where your security priorities lie. If intrusion detection, access reporting or break-glass access is your biggest priority then security tools will clearly go a very long way in assisting your understanding and mitigating risk, but this is rarely the case in a high compliance environment. Most organisations are comfortable with the proactive prevention of unauthorised access to data and of course the inevitable audits that follow. These security considerations call for at least as many procedural and organisational changes as technical changes and in these case tools will often satisfy no more than a fraction of your requirements.

Now that we have covered the bad news, its time for some optimism. Maintaining appropriate controls may seem a daunting task in light of the above but you should remember that you have already invested in the vast majority of the resources you need to secure your organisation, namely your existing service support and delivery organisation who already have the skills to provide the majority of the security controls you need. All that is needed is the framework that will satisfy the auditors and protect your data and systems. Could it be any easier?

The first step is deciding how to build your security framework. This is not as difficult as it might seem if you treat it like any other IT problem by using technology to meet the business requirement.

In this case the business requirement is defined by best security practice or your particular compliance rules. You can then assess the requirement and deliver the required standards as you would any other technical issue you have. A common approach to this is to employ experienced 3rd party consultancy or specialist security advisory services to help you build the framework always remembering that you are unlikely to be facing a situation that they have not encountered before.

So after all of those questions are answered, how do I know when security tools can help me? The simple answer is that once you have mapped all of your requirements and decided which of them you can satisfy, without security tools the requirements that are left drive your selection.

You can then use the following typical business cases for the use of 3rd party tools to refine your requirements. If any business case is a good fit with a requirement it becomes a further candidate for the use of a security tool:

The tool Delivers a Greater level of compliance

The tool Achieves segregation of duties via automation

The tool Delivers a better ROI case than a custom tool, process or procedure

The tool Delivers other significant benefits such as Management Information or Decision Support

The tool can be implemented within a required timescale that is not likely to be met by a custom tool or procedure

From the above business cases it should be clear that database security tools can provide a great many benefits and will have a role to deliver until the database vendors provide this functionality as standard (many are already striving to deliver these features but usually as add-on products such as Oracle�s Data Vault and Audit Vault).

The next step is to take the requirements identified above and ask Security tool vendors to demonstrate how their product can be used to fulfil those requirements, thereby removing some of the pain from selection process.

The key to meeting your security requirements is to define and balance them against your ability to define and deliver the security or compliance framework with your internal resources and then recognise the opportunities that the security tools offer to close the gap. This approach provides the following specific benefits:

Ensures the best possible use of tools reducing wasted licence, support and implementation costs

Uses the security requirements to define the correct scope and use of security tools

Aids in the tool selection process

Demonstrates the business case for any product expenditure

Assists in justification and negotiations at licence purchase and renewal time

Database Security Tools provide a specific and valuable benefit in database security but the hardest question may be why. Obviously �why� will define your priorities and we seem to have covered this but the last and most lingering question may unfortunately be out of your control. Do you secure for compliance or protection? If they were synonymous we would all have a single set of compliances rules and we clearly do not. The use of the correct policies aided by appropriate use of tools should deliver against your requirements but how can you define a requirement that equally satisfies the technically expert IT staff and the risk focussed auditor. Experience tells us that it is never that easy.

About the author

Pentest Limited is one of the foremost providers of IT security and penetration testing services in the UK and has a worldwide reputation for excellence. Each Pentest consultant has a deep and specialised knowledge in their respective fields, allowing Pentest to match your requirements as, when and where necessary. Pentest's specialist fields include database security, application testing and security, network security and wireless security.

"+title+"

Tue, 08 Sep 2009 23:56:13 -0700

Yaxiong Lin

Model-Based Testing and Its Tools (part 3 of 3)

In Part 1, I gave a brief introduction of MBT and different aspects in MBT. In Part 2, I reviewed and compared several MBT tools and discussed the architecture of TestOptimal and its modeling capability. In this final part, I will describe in more detailed about other features and capabilities TestOptimal supports.

State-of-the-Art Test Sequence Generators

TestOptimal offers 5 different test sequencers, they are:

randomWalk - based on Markov process to randomly traverse the model until a desired coverage is reached,

greedyWalk - similar to randomWalk except that it prefers the un-traversed transitions during the test sequence generation,

optimalSequence - generates the optimal test sequence to traverse all transitions in the model,

mCaseSerial - generates the test sequences to traverse all transitions and/or states in mCase in the order specified, and

mCaseOptimal - generates the optimal test sequence to traverse all transitions and/or states in mCase.

The transitions in mCase do not need to be adjacent to each other or in any order, TestOptimal automatically finds the appropriate transitions from the model to fill in the gap to ensure the complete test sequence is consecutive and valid.

Select different sequencer to achieve different effects to meet various testing needs. For example. you may want to use randomWalk to let it run until it finds a defect, or you may use one of mCase sequencers to test certain part of the model which has been recently changed without re-running the complete system testing. The ability to slice-and-dice the model to test different scenarios on the fly can help you reproduce the defect much quicker.

With the selected sequencer, TestOptimal can also generate a sequence graph from the model which can be used as a visual tool to validate the model.er.

Test Automation with XML Scripting

With the model created, the next step is to attach testing script to each transition to instruct TestOptimal test automation component what to do when the transition is traversed. This is done with mScrpt - an XML based scripting. Of course, you can also write java code as well.

The testing scripts are then executed through Selenium or HtmlUnit. Selenium tests your application in real browser (Internet Explorer, Firefox, Safari, etc.) while HtmlUnit tests your web application in a simulated browser with faster performance.

MBT Execution, Debugging and Monitoring

There are 3 ways to execute your model: dryRun, debug and run. DryRun just executes the model without test automation while debug and run do. With debug, you can set breakpoints at runtime and step through transitions and mScripts.

TestOptimal provides an extensive monitoring on MBT execution, including current coverage of the model, number of detects found and details of defects, number of traversals on each transition, average response time of each transition, estimated completion time, etc.

The execution results can be stored for comparison. TestOptimal uses the stored execution results to evaluate the performance of current execution as model is being executed.

You can assign a severity level to each defect and use the severity level to categorize the defects or to use it as the stop criteria.

Load and Stress Testing

With the built-in test automation, TestOptimal is capable of stress and load test your application by running the model concurrently on multiple sessions. With mCase, you can easily create a bunch scenarios and execute them against your application. As a result your application is exposed to a realistic simulation of production environment.

Licensing Editions

TestOptimal offers 5 editions to meet specific needs: BasicMBT, JavaMBT, WebMBT, Enterprise and Community. BasicMBT provides basic MBT modeling and test generation while JavaMBT and WebMBT include all functionalities in BasicMBT plus test automation. Enterprise edition offers advanced features like stress/load testing and custom plugin.

TestOptimal also offers Community edition with limited functionality free to MBT community for educational and non-commercial use. It can be downloaded at here..

I hope you are as convinced as I am on the benefits MBT can bring to the software testing [2] [3] [7] and these benefits can not be realized without a strong MBT community. If you need more practical how-to guidance, check out 6-part series tutorial Model-Based Testing HOWTO.

Tools :

Conformiq

MaTeLo

mbt.tigris.org

ModelJUnit

NModel

Reactis

Smartesting (formerly Leirios)

SpecExplorer

TestOptimal

About the author

Yaxiong Lin has over 20 years of experience in system modeling and software design, development and testing. He was the chief architect of several mission critical enterprise systems.He is the owner of open source project openOptima and is the chief architect of TestOptimal.

"+title+"

Mon, 31 Aug 2009 01:56:48 -0700

Yaxiong Lin

Model-Based Testing and Its Tools (part 2 of 3)

In Part 1, I gave a brief introduction of MBT and talked about different aspects in MBT. In this part, I will cover some of these tools and present a comparison matrix. To help you appreciate how far the tool can go to help you in your MBT project, I will focus on just one of these tools which I architected. If you need more detailed information about other tools, you may check the references section at the end of this article.

MBT Tools

There are quite a few tools and projects developed over the years. Unfortunately many of them have been either inactive or been morphed into other projects. I will only cover the active tools and projects here as of date of this article.

Conformiq - MBT modeling, test sequence generation with scripts generation in different languages.

MaTeLo - a statistical MBT test sequence generation using random Walk.

mbt.tigris.org - offers test sequence generation from imported models in graphML.

ModelJUnit - a Java library that extends JUnit to support MBT. It is used to unit test java classes using MBT. The model is created with java classes inheriting from the framework classes.

NModel - MBT modeling and test sequence generation using c#.

Reactis - offers random walk test sequence generation.

Smartesting (formerly Leirios) - offers MBT test sequence generation.

SpecExplorer - by Microsoft, a framework to develop MBT models in Spec# and Asml Test sequence generation using randomWalk and Chinese Postman algorithm.

TestOptimal - supports full MBT process with MBT modeling, 5 different test sequence generation algorithms, and test automation for web and java applications with plug-in architecture to support other types of applications.

MBT Tool Comparison Matrix
MBT Tool (alphabetically)	Licensing	Modeling	Sequencers	Test Automation
Conformiq	commercial	yes	yes	no
MaTeLo	commercial	no	yes	no
mbt.tigris.org	open source	no	randomWalk, A_star	no
ModelJUnit	open source	java	yes	java
NModel	open source	c#	yes	c#
Reactis	commercial	yes	randomWalk	no
Smartesting	commercial	no	yes	no
SpecExplorer	free	Spec#, Asml	randomWalk, CPP*	yes
TestOptimal	commercial, free Community Edition	yes	randomWalk, CPP*, mCase	yes

* Chinese Postman Problem algorithm.

Before you choose a tool for your MBT project, it is very important to know how the tool will fit into your development process and understand what the tool can and cannot do. There is nothing more frustrating than trying to make a tool do what it is not designed for.

In the following sections, I will discuss in greater details on one of these tools: TestOptimal to help you understand what to look for in the MBT tools.

TestOptimal Architecture

TestOptimal is built with a typical web client/server architecture. At its heart is the TestOptimal Server which can be deployed on Windows or Linux. You would usually run TestOptimal Server in your local machine while you are developing and when completed you deploy and execute the model on a dedicated server.

TestOptimal consists of modeling, test generation, test automation, execution and debugging, and statistical analysis of the execution results. It currently supports test automation for web and java applications but its flexible architecture will allow it to expand to other types of applications like Adobe Flex, and .NET applications.

You can access TestOptimal Server with a web browser from any location or use one of many supported interfaces to integrate TestOptimal with your IDE and existing testing framework / tools as illustrated in the architecture diagram above.

Supporting Full MBT Process

TestOptimal was built from ground up to support full MBT process starting at the modeling through test generation to test automation and execution result analysis. With TestOptimal, you can easily go back and forth between modeling, test sequence generation and test automation. Based on the result analysis, you can make changes to the model and immediately execute the model on your application right within the TestOptimal browser.

Modeling with TestOptimal

The model is created in the simple tree-view like model editor. It supports super state which is a state containing child states. Transitions are placed under the state. Each transition may be defined as optional or required with minimum number of times it must be traversed, a useful feature to devise a Data-Drive Testing. The model can then be viewed in a graph.

You may also import models in several formats: graphML, graphXML, SCXML, and XMI from Eclipse/RAD.

In Part 3, I will cover TestOptimal's 5 test sequence generators and its built-in test automation.

"+title+"

Tue, 25 Aug 2009 03:09:57 -0700

Yaxiong Lin

Model-Based Testing and Its Tools (part 1 of 3)

Model-Based Testing (MBT) is the newest development in the software testing industry. In this 3-part series, I will give a brief overview of MBT and provide comparisons of different tools from the perspective of UI interface testing with the intent to increase the awareness of MBT in the software testing community.

A Different Approach to Software Testing

Model-Based Testing (MBT) approaches software testing from a brand-new perspective that is quite different from the traditional software testing [1]. MBT is a methodology that is based on science with a clear objective to achieve the best test coverage of the software. The traditional "record / replay" and manually crafting test cases often do not provide sufficient coverage and are labor intensive to create resulting in a system that is very difficult to maintain especially in today's agile development process.

MBT provides an attractive alternative to address these challenges.

MBT Process

There are 5 distinctive phases in MBT process: modeling, test case (sequence) generation, test automation, test execution and test result analysis [2]. In practice this is often an iterative process. For example during test case generation or test automation phases, you might need to make changes to the model.

Having a tool that supports this iterative process is imperative for a successful MBT implementation.

MBT Modeling

The first phase in MBT process is modeling, an activity to create a behavior model from the requirements and specifications for the Application Under Test(AUT). The model is described with a Finite State Machine (FSM) diagram or State Diagram in UML

Using a web application as an example, a node (state) in the diagram usually represents a web page and the arrow (transition) represents a button or link which leads to another web page when clicked

In some cases it may be important to distinguish a web page under different context in which case you would use a collection of states to represent different contexts in a page. Flash movie model is one of such example.

When displayed in graph, the model becomes a visual representation of the requirements and specifications which is much easier to understand and validate and can be used as an effective communication tool.

Test Sequence Generation

The ability to generate test sequence from the application model is the single most important aspect in MBT. All MBT tools are equipped with at least one test sequence generator

Behind these test sequence generators there are different mathematical algorithms. These algorithms are the brains of test sequence generation. There are various types of algorithms ranging from as simple as random walk derived from Markov process to a fairly complex algorithm like the algorithm to solveChinese Postman Problem (aka Route Inspection Problem).

Often times you may want to test just certain parts of AUT or repeat certain parts of the model for Data-Driven Testing as a example. A good test sequence generator should allow users to change configurations to meet different testing needs.

Test Automation Integration

With the test sequence generated from the model, the next step is to turn them into executable. A typical test sequence may contain hundreds of thousands of test steps. Clearly you do not want to do this by hand.

At a minimum each MBT tool should generate the test script in certain format which can then be executed on a test automation tool. In fact this is what most of MBT tools do.

The problem with this approach is that no matter how good the vendor claims their tool can generate the script to run on someone else's tool, it is inevitable that you will end up making little change here and there to get it working on your test automation tool. As soon as you do that they become high maintenance artifacts.

There are MBT tools which offer built-in test automation giving you a nice seamless integration from modeling to test automation. This is certainly the tool you want for your project.

As we all know, a good tool enhances the process and enables us to become more productive. It is even more so with MBT. Before choosing an MBT tool, you will want to know how it will integrate with your development process and existing tool set. You will also want to have a good understanding of its capabilities and limitations.

In Part 2, I will review some of the tools and give a comparison matrix on them.

"+title+"

Fri, 21 Aug 2009 06:32:06 -0700

Soon Hui
Esteem Innovation

How to Evangelize Unit Testing

You can't force others to practice unit testing, especially if they are working on legacy applications. They will have a lot of excuses and true enough, doing unit testing on legacy code that have not been designed for testability is hard. But not doing any automated testing is tantamount to not paying your credit card bill. Sooner or later your debt will eat you up and you will go bankrupt. Or not, if you are lucky, you can pass your debt to either your descendants or successors and let them worry about the problem.

But sometimes you don't have the luxury (i.e., passing the mess around), worse still, you may need to anticipate the messy code problem because you may have to do the cleanup. Or you are such a purist that you can't see the whole application walking around without automated testing covering its ass, then my experience below maybe helpful to you.

Risk based testing is getting popular since it is aimed at optimization of time and resources available for testing without comprising on the quality of testing. However, most approaches for Risk Based Testing today, suffer from the following drawbacks:

First, a little background

We did our application in C#. Since the language is fairly modern, so it must follow that the software development style should be fairly modern, right? Unfortunately, no.

We didn't have unit testing until fairly late into the development. And so, I was in charge of brining the unit testing practice to the company because I was a fierce proponent of it.

One thing I can tell is that it's never easy to convince the developers to adopt new things unless you can show tangible benefits. But there is a chicken and egg problem . Since the benefits of unit testing is only apparent when you work it on long term, how can you prove to them that unit testing is a good thing? Unless you can take over their projects and do the test cases then it's a different story. But on a large scale commercial application this is simply not possible. Unit testing is a bit like religion; you can't taste the goodness unless and until you become a convert.

One may want to show that the code with unit testing coverage has the lowest bug count. But this number is not easy to get. And even if you can do that, so what? Using bug count to evaluate developers or to promote unit testing is no better than using bug count to evaluate testers. If you have very few bugs in your code, it could have meant that your code is still not tested, or that your work is so easy that there isn't much bugs lurking around, or that you are already a superior coder for your program is much better than the guy seating next to you, regardless of whether you do the unit tests or not.

The only way I can think of is to create integration test cases.

Well, it's not exactly integration tests. It's a hybrid between the integration tests and the unit tests, leveraging on the existing NUnitframework.

Sounds complicated?

It's not, they are just NUnit test cases that test large modules with different components interacting together. Since those test cases were not testing individual methods, so they were not exactly unit testing. Low level methods and classes were just too tightly coupled to be tested individually. So a compromise had to be made. Luckily we still had some high level design and low coupling between different modules, so creating NUnit test cases on module level were still feasible.

Since the developers hadn't bought in the idea, so I created the test cases for them.

This must be done carefully. The developers usually didn't like other people creating test cases for them because they would feel like they were not trusted. A political revolt might result if one did this briskly. Besides that, the code was also not structured for maintainability so you might need to refactor the code first before you could write tests. Generally, the developers' willingness to let you touch their code is proportional to the code quality. So the worse the code is, the more need there is to create tests for them. But the worse the code is, the harder it is-- either technically or politically--to test them.

This is a vicious cycle.

What I did was I appealed to the boss to lobby for automated regression tests. Luckily he fully understood the problem and gave me a go-ahead sign. I also reasoned to the developers and asked for their permission to

Refactor the code
Create test cases to test their code

At that point of time, the developers would normally agree to let you did the work, regardless of how grudging they were at the beginning. After all, they needed not to do extra work and since the boss had agreed on it, so there was no reason to complain.

After receiving the green lights to create integration tests, I understood that I must did this correctly, on the first attempt. There would be no second chance if I couldn't show the tangible benefits within a short time frame. Everyone was impatient to see the results. If they couldn't see the results in two months time, all the good will and enthusiasm would drop off exponentially and to pickup unit testing next time would be significantly harder.

Which was why it was necessary to go for the biggest bang for buck.

The next step I did, I strategized and look for the area that was most error-prone and easiest to create tests on. You wouldn't normally get both, but in my case, I was lucky to find the weakest spot and started to create tests on it.

And being the person who didn't write the code, one might not know what to test; after all, unit testing was supposed to be done by the developers who knew their code inside out.

So to work around the problem, I went through the bug tracker to look for relevant bugs. Since all cases had ( or were supposed to have) the reproduction steps, the observed outcome and the expected outcome, I knew how to reproduce the problem, what were the correct and incorrect results. And so, I knew what to test and how to test. Problem solved.

So, as test cases accumulated, the ability for the entire test suit to find bugs increased gradually. Seeing that the tests did catch bugs, the attitude towards unit testing changed gradually as well. After sometime I passed the whole test suit to the respective developers and asked them to maintain it themselves. Now they were willing to own the test suit.

I think there is an important point here. To help the developers to see the virtue of good software design, it is often necessary to let them realize the benefits of unit testing first before anything else. Too often the developers who write bad code will be reluctant to cleanup their design mess under various pretexts. But once their buy in unit testing, they will naturally design their code to be testable and will pickup good software design practices along the way.

No, we are still far from Test Driven Development (TDD), but at least the developers were more receptive to unit testing and started to create their own tests. Well, occasionally they might still forget to do it, but at that point, a little reminder could solve the problem. My personal experience with unit testing was that it was infectious; once I started doing it in one area of my code, I would like to apply it onto other areas. And with time I learn how to write test first, let the test drives the design!

I believe that gradually every developer would start to write test cases themselves automatically, without reminder once they got onto it.

"+title+"

Mon, 10 Aug 2009 09:45:21 -0700

Keshava Moorthy
InvenTest

Model for evaluating value at risk for Risk Based Testing

Abstract: Delivering high quality reliable software on time and within budget is the main focus of software testing. The quality of the software delivered is decided based on the test coverage, approach taken to discover the defects and effective risk evaluation.

The major amount of rework done in testing phase is due to poor risk evaluation and risk management. The roles of development team, testing team and support teams are crucial in evaluating the risk and the magnitude of the risk in testing. The risks which are identified while testing is in progress consume 30-40% additional overall project effort and cost. Though testing is one kind of risk mitigation for the overall project lifecycle, there are lots of areas in testing where the risks are involved. Important areas can either be functions or functional groups, or properties such as performance, capacity, security etc. The result of this risk analysis is a list of functions and properties or combination of both that need attention.

Lack of precise definition of risk. Risks are often confused with issues

Subjectivity in assessing the impact of risk, without any real, relevant quantification of impact

The dynamic nature of risk, in terms of differing impact and probability of occurrence of the risk factor, over different stages of the project lifecycle, is not captured

This paper focuses on the deriving a model for risk evaluation and risk based testing, based on a simple value at risk model, addressing limitations of the traditional models.

Introduction

Software testing in many organizations is key and critical to delivering success in the current business environment. Because of the complexity of software or systems, the testing done on them is comprehensive but not exhaustive. Most of the test management fails to handle the risks on time due to lack of knowledge in identifying the risks and a risk based /modeled approach for testing.

Risk can be broadly defined as the degree of uncertainty about the project meeting its goals, and is a function of probability of occurrence of the defined undesirable event and the magnitude of severity. A risk can arise almost anywhere and have a significant impact on the test schedules and quality. So it is important that testing is done in operational frame work gives the ability to mitigate the risks early in the test life cycle.

Testing can reduce the probability of occurrence of certain risks by ensuring a wide coverage of test risk management wide across all possible risks that may occur due to development, testing approach, tools or test environment. Risk mitigation and compliance reduces costs, ensures quality, ensures timeliness and protects reputation, all of which are key factors to remain competitive

The higher and more critical risks are practically visualized in reality either very close to the integration testing, system testing and performance testing phases, or when these phases are in progress.

�Forward�[1] risks are those associated with the system operation and which involves the consequences of the software failure. But there are �Backward risks �[1] which are more of development life cycle originated risks.

Risk based testing is aimed at the optimization of the time and resources available for testing without comprising on the quality of work. In standard approaches of testing all aspects are given equal priority. In the later phases of the software life cycle such as the System Testing phase there is a need to ensure completion testing in a short span of time. In such situations standard approaches of testing are not effective as it is difficult to take decisions about the aspects that have to be tested.

Risk based testing has a major advantage of being the most suitable kind of testing for the System testing phase. The risk based testing procedure prioritizes risks and hence the testing effort is focused which leads to completion of testing in a lesser time span. Risk based testing also indicates the risk distribution across the application which is helpful in taking decisions about the release probability of the software application.

To carryout this important factor is to understand the cost of each identified risk. The value at risk model for software testing will provide access to efficiently manage the testing risks..

Why Risk based testing?

Though risk based testing is contrasting from the traditional approach, it has got numerous benefits to the project in terms of cumulative improvements in quality. Risk based testing ensures that the applications are delivered to meet business and time requirements. This also helps in optimizing the testing resources in a better manner. Risk based testing allows the software organizations to manage and minimize the risks by tracking and addressing the risks ahead.

Typical points of failure in new application development and testing effort are:

Meet the timelines

Meet the budget

Application logic

Keep up with business changes

To realize and implement business cases as expected

Measuring the reliability of the application

Value addition to the application due to testing

Risk based testing outcome plays a major role in making a final decision to Go live or not, objectively. This is because, risk based testing gives the access to understand the status of the identified risks and depth of testing done. The more testing is done, the less value it adds. Most organizations find it comfortable to invent workaround solutions rather than addressing the risk. But these work around solutions in reality cost a lot compared to addressing the risk itself.

The traditional risk model for testing

The traditional risk model has two elements

1) The probability of a fault being present
2) The impact of a defect /fault in the corresponding function if it occurs during normal operation

RE(f) = P(f) * I(f)

Where RE(f) is the risk exposure function of f, P(f) is the probability of a fault occurring in function f and I(f) is the impact when a fault is executed in a function f in operational mode.

The cost of software failure is very high. More number of defects or problems and defects can contribute to the software cost exponentially. To address this it might need a rigorous risk management to comprehensively mitigate and make contingency plans..

�Heuristic reasoning is a reasoning not regarded as final and strict , but as provisional and Plausible only, whose purpose is to discover the solution for the present problem � written by George Polya , Greek Philosopher.

Using this approach, one need to identify the situations and related impacts by repeatedly asking �what could go wrong here?�. The three factors that can be considered are Vulnerabilities, threats and Victims.

Begin with a set of potential risks and match them to the details of the situation. Basically the risks can be identified under three categories viz., Quality criteria, Generic risks and risk catalogs.

Using following steps one can analyze the risks

Select a component or a function you want to analyze

Determine the severity of the concern

Gather more information or inputs on the concern

Visit each risk area on each list and determine its importance in the situation

Record unknowns

In the traditional model, the risk exposure is seldom a quantified value but more a subjective value like low, medium high, based on the corresponding probabilities and impact.

Value at Risk Model

Value at Risk model is an extension of the traditional model, to of qualify, quantify and mitigate the risks in software testing. It follows a 4-step process

Risk identification

Risk description

Risk estimation

Risk mitigation

Risk identification

Risk identification process provides the access to the projects/software developments exposure to uncertainties that can occur during testing. This needs sound knowledge of the entire application (software) and the other systems with which the application interacts/interfaces and the deployment environment. This also requires a sound knowledge of performance requirements and business user�s perspectives. Test risk identification can be done systematically and methodically to ensure that the all the risks are identified and all the activities linked to the risks are defined.

Developing a risk identification checklist :

Generic risks

Complex: anything disproportionately large, intricate, or convoluted

New: anything that has no history in the product

Changed: anything that has been tampered with or �improved�

Upstream Dependency: anything whose failure will cause cascading failure in the rest of the system

Downstream Dependency: anything that is especially sensitive to failures in the rest of the system

Critical: anything whose failure could cause substantial damage

Precise: anything that must meet its requirements exactly

Popular: anything that will be used a lot

Strategic: anything that has special importance to your business, such as a feature that sets you apart from the competition

Third-party: anything used in the product, but developed outside the project

Distributed: anything spread out in time or space, yet whose elements must work together

Buggy: anything known to have a lot of problems

Recent Failure: anything with a recent history of failure

Generic risks

Capability: Can it perform the required functions?

Reliability: Will it work well and resist failure in all required situations?

Usability: How easy is it for a real user to use the product?

Performance: How speedy and responsive is it?

Installability: How easily can it be installed onto its target platform?

Compatibility: How well does it work with external components and configurations?

Supportability: How economical will it be to provide support to users of the product?

Testability: How effectively can the product be tested?

Maintainability: How economical will it be to build, fix, or enhance the product?

Portability: How economical will it be to port or reuse the technology elsewhere?

Localizability: How economical will it be to publish the product in another language?

Risk description

Risk description is a method of displaying risks in a structured format. A well designed structure is necessary to ensure comprehensive risk identification, description and assessment process. By considering the consequence and probability of each of the risks set out in the table, it should be possible to prioritize the risks that would require further analysis

#	Item	Description
1	Risk name
2	Scope	Description of the event,size,dependencies
3	Nature/ Area of impact	Quality,compliance,technical etc
4	Probability	Probability of occurrence
5	Severity	Severity of the risk(critical/High/Low)
6	Risk tolerance	Potential impacted areas and losses(gains)
7	Risk mitigation/control mechanism	Primary means of managing the risks, identification of protocols and other requirements/dependencies to control the risk
8	Policy development	Identify the group( testing /development) to develop the policies, suggestions

Risk Estimation

Identified risks can be quantified in terms of the magnitude of the impact and the probability of occurrence. In the traditional model, they are typically represented in the following tabular format:

Probability

Rating	Description	Indicators
High	Likely / can occur more frequently	Has occurred recently few times
Medium	Likely / can occur less frequently	Has occurred some time back once/twice
Low	Not likely to occur in the near future	Unlikely. No past experience

Impact

Rating	Description
High	Significant impact on delivery/user/stake holder
Medium	Less impact on the delivery/user/stake holder
Low	Negligible or no impact on the delivery/user/stake holder

The next step is to calculate the magnitude of the risk by developing the Probability /Impact matrix.

Probability	Impact	RE (Risk Exposure)
High	High	High
High	Medium	High
High	Low	Medium
Medium	High	High
Medium	Medium	Medium
Medium	Low	Low
Low	High	Medium

This helps to understand and make decisions about the risks. This is called as risk profiling or risk estimation.

Risk Estimation in the Value at Risk Model

Where Value at Risk model extends the traditional model is in terms of quantifying high/medium/lows into just a single number � the loss associated to a given probability. Organizations can further balance the cost of testing from the benefits derived through statistical analysis of the derived metrics.

Value at risk has three components and they are

Approximate cost of the risk

Time over which the risk has to be monitored and measured

Confidence level

Following factors play important role in identifying value at risk

Complexity of the software:
The complexity of the software plays a major role in detriming overall cost of the risk. Examples include complex control logic, data flow, dependency on the subsystems etc. This means the one needs to consider different aspects of complexity and problem areas. The amount and types of testing required depends on the complexity of the software

Changed /Changing areas:
Frequent changes to the software have a significant impact on testing and overall quality of the software.

Areas with many defects before (Historical experience) :
Defect fixes may potentially cause some other defects. The modules that had most defects are likely to have more defects in the future.

Impact of number of people involved: If the testing involves more people the overhead for communication, retesting and downtime

Impact of time pressures Time pressure makes people to take shortcuts and workarounds. This also may lead to overtime work and will result in poor quality job done by the resources

Geographical usage People working on the same project in different geographical locations potentially have communication gaps. The cost contributed to this may be considerable

Impact on business Due to system downtimes , unstable system or more iterations of the software releases , the cost to the business may be very high

By considering and factoring the above items along with the other identified risks to determine the value at risk will make a significant difference in managing the risk.

Sl number	Factors	Impact Value (in USD)
1	Complexity of the software	5000
2	Changed /Changing areas	3000
3	Areas with many defects before	10000
4	Impact of number of people involved	2000
5	Geographical usage	1000
6	Impact of time pressures	1000
7	Impact on business	10000

Total cost of a risk related to testing is = 32000 USD

Obtain Value at Risk table

Sl no	Risk	Value at Risk	Risk qualification(Qualified /not qualified)
1	Unidentified business requirement	32000 USD	Qualified

By calculating the Value at risk for each of the identified risks, the overall cost of the risk in the testing area will be

CRe = ? Value at risk of R1+ Value of risk at R2 + ��..Value of risk at Rn

But in reality this is not true as some part of the Value at risk for each factors may be embedded in other factor by extrapolating the risk factors.

Sl number	Factors	Cost(in USD)
1	Complexity of the software	1000
2	Changed /Changing areas	Absorbed in complexity
3	Areas with many defects before	7000
4	Impact of number of people involved	2000
5	Geographical usage;	Absorbed in impact of business
6	Impact of time pressures	Absorbed in impact of business
7	Impact on business	10000

Now the actual value at risk = 11000 USD

The final (optional) step is to capture the dynamic nature of risk by plotting a VAR distribution. There are three methods of calculating VAR distribution: the historical method, the variance-covariance method and the Monte Carlo simulation. The historical method takes into account past data for a risk factor, re-organizes actual historical impact, putting them in order from worst to best. It then assumes that history will repeat itself, from a risk perspective. The Variance-Covariance method assumes that impact is normally distributed. In other words, it requires that we estimate only two factors - an expected (or average) return and a standard deviation - which allow us to plot a normal distribution curve. The Monte Carlo Simulation method involves developing a model for future impact and running multiple hypothetical trials through the model.

Based on the confidence level, which would be different at different stages of the project, one can calculate the overall value at risk and answer the key question � �what is the worst loss that I can expect during a specified time period with a certain confidence level?�

Ilustrative plot of probabilities and VAR

Risk Mitigation:

Projection of value of risk may provide the organization the visibility to the near future

Generation of the re-evaluated risk watch list based on the execution results of the risk based testing cycle. This watch list will also indicate the regression risks.

Projection of the risk lowering factors and product release probability.

Case study

One of the world�s largest Insurance companies Scenario : This organization has four applications which are interfaced and these applications are being used in different geographies. These applications are financial transaction critical applications. These applications were in the market for more than 10 years . The organization had recently re-platformed the application in J2EE technology.

Risks Identified in testing area :

Shared UAT environment between all four applications

Unavailability of performance test environment

Technology upgrade was not done in the test environment

Unavailability of strategy for end to end testing

Unavailability of test data

Poor test coverage

The above factors are important because it will lead to obvious software failure and rework. The values obtained after normalization given as below

Sl number	Factors	Cost of risk(in USD)
1	Shared UAT environment	2.5 million
2	Unavailability of performance test environment	1.6 million
3	Technology upgrade was not done in the test environment	0.5 million
4	Unavailability of strategy for end to end testing	1 million
5	Unavailability of test data	0.25 million
6	Poor Test coverage	4 million

This helped the management to come up with a mitigation plan based on the cost factors.

The benefits derived :

Early mitigation of all the risks effectively

Good Time and space for test preparation based on the risk factors

The cost to business ( impact on business) is reduced drastically

No fear of reputation loss in the market due to extensive test coverage

Summary

Testing in a situation where management cuts both budget and time is a bad game. We have to endure and survive this game and turn it into a success. The general methodology for handling this situation is not to test everything in bits, but to concentrate on high risk areas and the worst impacted areas. The main objectives should be to return the product as fast as possible to the developers with a list of critical defects or deficiencies as possible and to make sure that best testing with good out put is achieved with in the dead line. So to do effective test risk management involves

Anticipate the risks

Asses the impact and plan to handle

Calculate the value at risk in the testing area

Use control /mitigation mechanisms to catch the ocuurance of risks

Proactively handle the risk whenever it triggers

By doing the points discussed above a quality product can be delivered to any customer with minimum pain

Reference:

Redmill 2004: Redmill F: Exploring risk based test and its implications .

Technical report CMU/SEI-93-TR-6,ESC-TR-93-183, Taxonomy based risk identification ,1993

Specification based regression testing measurement with Risk analysis by yanping chen , October 2002

Heuristic risk- based testing by James Bach. Software quality engineering Magazine,11/99

Joachim Karlsson & Kevin Ryan � A cost �Value approach for prioritizing the requirements� IEEE software,Sept 1997

James Bach � Risk based Testing �,STQE 6/1999, www.stquemagazine.com

Evaluation of value at risk Models using historical data ,FRBNY economic policy review /Apr 1996

About the author

Keshava Moorthy Chandrashekar is the Founder of InvenTest Limited. He worked as Presales Consultant at Wipro Technologies and Senior Engg QA at Arcot R&D designs. He completed his education from Manipal Academy of Higher Education.

"+title+"

Mon, 03 Aug 2009 02:24:47 -0700

Soon Hui
Esteem Innovation

Automated GUI Testing � Challenges!

Automated GUI Testing is always a pain in neck. There are many things that one can talk against it. But this Article will concern itself with the accidental difficulties of the automated GUI test.

Before I start my rant I would like to make a difference between Automated GUI Testing and unit testing. There are a lot of differences between the two. Unit test tests against a component; its scope is often limited to class libraries. But GUI testing is more like a system testing, whereby the whole application is tested as a black box. Unit test does not involve UI actions, whereas GUI testing involves simulation of keystrokes, mouse clicks or keyboard actions. And the most important of all, unit test is usually written by the same developers who write the logic code, whereas UI scripts are often written by other developers-testers. As such, the unit test code is usually written in the same languages, with the same tools as the production code, whereas the UI scripts are more often coded using different IDEs using different languages. And that, contributes to the first problem of the UI tests.

Tools do play a huge part in programming, because a good tool can easily reduce the accidental difficulties of it and increase programmer's productivity. There are numerous automated UI tools available on the market (such as QTP, TestComplete, QA Wizard etc), and a mature one should come with a scripting editor. For someone who is accustomed to Visual Studio or Eclipse, the scripting editor seems awfully inadequate. Living with an editor without refactoring, syntax highlighting and numerous other features is just painful.

The second problem with automated UI test is, it is slow to run-- compare to unit test. Unit test is fast, because you just have to load the relevant libraries and test them from your IDE. But for the UI test, you got to

Compile the whole application
Launch it
Launch your Scripting IDE
Attach the application to the IDE
Start testing

So many steps! With unit test we can talk about letting the test drives the development, but we can't do that with UI test. Every time we make changes to our code, we can run all the unit test cases before we check-in. But due to the slowness nature of UI test, we can only run it after we have a daily build.

A good IDE eases maintenance effort, since most programming is all about maintenance, a serious software development effort must always take it into account. But the poor state of scripting tool complicates the task:

My scripting code is not properly formatted, the braces are not properly aligned and the if-else statement does not lie at the same column location, any shortcut keys to format the text? No.
I need to debug my test scripts, but the debuggers are primitive and hard to use.

The list just goes on and on. Your automated test tools may be different from mine but I am sure that they are not as advanced as the standard programming tools. And that's a serious problem because it makes writing automated test scripts expensive.

There are a lot of primitive actions that the developers-testers have to write. Actions such as clicking and dragging objects are not wholly supported. Modern computer languages (such as C# and Java) abstract away these low level details so that the developers can concentrate on higher level things, but apparently the scripting technologies are not that advanced yet.

Capture/Replay Tool to save the day?

You may ask, why all the trouble? Don't we have a fanciful capture/replay tool built into almost all of the automated test tools? There is no need to write scripts, you click a start recording button, do everything you want and press a stop recording button. Viola! You have a test case.

But these scripts are very fragile, and unmaintainable. The capture/replay tool is just like any other programming tool that is used to write program. It can never be as intelligent as human. Anyone using it to generate test scripts will have a hard time living with them.

And most of the time, the capture/replay tools cannot intelligently identify the tested objects. All they can do is to blindly record your action in terms of screen coordinates. Thus if your menus or icons change place, the tests break. In all fairness, some test tool vendors (such as TestComplete) can identify common objects to a certain degree by linking those objects with their names. But still, this capability is very limited.

Finally, there is a lack of interoperability and standards among different test tool vendors. This means that the scripts I write using Test Tool A cannot be run on Test Tool B. Although the tool vendors love to lock-in their customers for obvious short term reasons, but in fact it simply discourages people from practicing automated UI testing, resulting in a loss of potential customers and hindering the process made in removing the accidental difficulties of automated UI test.

None of the problems outline above are insolvable. All it takes to solve these problems is to have a more powerful scripting editor. There are other essential difficulties with automated UI tests too. These are the ones which come�s to mind quickly!

"+title+"

Thu, 30 Jul 2009 04:25:42 -0700

Himanshu Joshi

Challenges faced while testing!

Every now and then I hear people saying that we don�t have enough time for testing or our estimates have gone wrong due to some resource issues, however we can resolve these things by doing risk analysis, we need to identify the areas where testing should be focused.

Since it�s rarely possible to test every possible aspect of an application, every possible combination of events, every dependency, or everything that could go wrong, risk analysis is appropriate to most software development projects.

This requires judgment skills, common sense, and experience. (If warranted, formal methods are also available.) Considerations can include:

- Which functionality is most important to the project�s intended purpose?
- Which functionality is most visible to the user?
- Which functionality has the largest safety impact?
- Which functionality has the largest financial impact on users?
- Which aspects of the application are most important to the customer?
- Which aspects of the application can be tested early in the development cycle?
- Which parts of the code are most complex, and thus most subject to errors?
- Which parts of the application were developed in rush or panic mode?
- Which aspects of similar/related previous projects caused problems?
- Which aspects of similar/related previous projects had large maintenance expenses?
- Which parts of the requirements and design are unclear or poorly thought out?
- What do the developers think are the highest-risk aspects of the application?
- What kinds of problems would cause the worst publicity?
- What kinds of problems would cause the most customer service complaints?
- What kinds of tests could easily cover multiple functionalities?
- Which tests will have the best high-risk-coverage to time-required ratio?

Not enough time for thorough testing?

This is a cry frequently heard as deadlines approach.
You can hear most of the managers screaming about this issue�.
However there could be a number of answers:

1. The testers were not able to complete testing due to a new release being loaded.
2. The bug was not in an earlier release (reload that earlier release and see).
3. The bug could not be tested for earlier because some part of the release did not work and inhibited
the test�s ability to �see� the bug.
4. The bug was in some part of the system not originally planned for the release for which a test has only just been written.
5. The bug was found while running some other test.
6. The bug was in a part of a system which was not the focus of testing.
7. The bug would have been found eventually, but the tester hadn�t run the test (which would have found it) yet.
8. And yes, maybe if we�d been more thorough we�d have found that bug earlier.

It�s always good to keep in place a corrective action in place so that the impact of the issue can be minimized and the stake holders and the client/s do not lose faith on you and your team.

Enjoy and let me know your thoughts on this.

About the author

Mr. Himanshu Joshi has 7 years of proven experience in software testing that involved Comprehensive testing of System, Client/Server and Web applications in both windows as well as UNIX environment. Extensive exposure in Different testing methodologies, different testing techniques, rich experience in Build and Release of applications at QA end as well as Prod end.

"+title+"

Sun, 19 Jul 2009 22:55:48 -0700

Steve Miller
Pragmatic Software

Benefits of Keyword Driven Testing for Test Automation

Most software companies have considered automated testing and many have fully automated their regression test cases in an effort to reduce manual effort needed to test new builds of their software. Many companies that have been successful with automation attribute it to keyword driven testing techniques that reduce the time spent creating test cases. This newsletter addresses why companies consider automation and best practices for ensuring that time spent automating test cases provide a return on investment.

Why Automate Your Test Cases?

Many companies run their regression test cases manually, so when does it make sense to begin automating your regression test cases? It makes sense to automate your test cases when you can no longer run the regression test cases on each build created. For example, if you are doing daily or weekly builds of your code to quality assurance and you cannot quickly run your regression test cases with each build, it is time to consider automating them. Automating your test cases provides these benefits:

Quicker Releases � By having your regression test cases run automatically, your software quality team can concentrate on testing new features of your software and less time regressing existing features.
Higher quality releases � Your software releases will have fewer bugs and require less customer support because they will be of higher quality.
Happier Customers � Your customers will be happier and more willing to serve as testimonials for future prospects.

Why Does Automation Fail?

Many companies are thinking about test case automation or have experimented with it in the past. Some companies that experimented with it eventually abandoned it because of these reasons:

Poor Understanding of Test Automation � Many companies see automation as a silver bullet that will allow them to quickly automate every test scenario quickly and allow them to abandon manual testing and reduce staff. In reality, automated testing is designed to quicken the running of regression test cases but it is not a substitute for manual testing. A quality-oriented software team will see the value of utilizing both automated and manual test cases to ensure great test coverage and higher quality releases.
Improper Tester Education � Automation requires a tester to learn the testing tool. Companies that purchase the tool but not any training will eventually abandon the tool because the testers are not equipped to use the tool.
Lure of Record / Playback � Many companies think that they can quickly get up and running with automation by simply recording their screen actions and playing them back. While record and playback will create scripts that they can use as a starting point for your automation, it does take time to update the scripts to be more re-usable and it takes scripting language knowledge to do this.

What is Keyword Driven Testing?

Most automated tools require the test engineer to understand a scripting language (VB Script, Java Script, etc.) to write their automated test cases. Most tools have the ability to create the scripts using record and playback, but this does not always write the most efficient scripting code and is not as re-usable and maintainable. Since many testers do not have deep scripting skills, it is imperative that your automated testing tool has a way to create Keyword Driven Tests.

Keyword Driven Testing is a way to define automated test cases without the need for scripting skills. It allows a tester (or even a subject matter expert) to create automated tests by describing each step of the automation. For example, if you are automating the login process of your application, your user will access your application, type in their user-id and password and press a button to login. Traditionally, testers would do this by writing VB Script that will navigate to your application, identify each object on the screen (user-id, password and login button), then write script to enter in the user-id, password and to press the login button. With keyword driven testing, the tester does not need to understand the scripting language to make this happen, they can simply describe the event (navigate to your application, enter in "abc" for the user-id, enter in "xxx" for the password, press the Login button when done). As you can imagine, this is a much simpler approach to automated testing than scripting.

How have Successful Companies Implemented Automation?

Successful companies understand the enormous benefits of automated testing and have implemented strategies to ensure that they receive the maximum return on investment with their test efforts. Below are the secrets to becoming successful with test automation:

Start Smart � Automation efforts are similar to software development efforts -- it takes upfront thought and a good test design architecture to ensure that your automated test cases will be re-usable and easy to maintain. Before jumping into automation, ensure that your automated tool has the ability to maximize your efforts with keyword driven testing -- as this will ensure that you can easily re-use your automated test cases.
Start Small � Start your automation efforts on an established project that already has a good set of manual regression test cases written. Take those manual regression test cases and develop your automated test cases that will replace them. Once this is done, you will find that you can automatically run those automated test cases each day of builds and with minimal efforts, freeing your team up to do more manual exploratory test cases. Once you have this established for a single project, move on to other projects and expand your effort.
Blend Automated and Manual Tests � Not all test cases should be automated. Some test cases require a human eye to ensure that screen cosmetics are appropriate, that data is reasonable, etc. Spend time automating test cases that do not require this type of human contact and continue to use manual test cases when appropriate. A good application lifecycle tool should allow you to track both automated and manual test cases.
Keep track of Metrics � Keep track of how many automated and manual test cases are run for each build, how many pass, how many fail, etc. Track how many additional automated test cases your team can write with each release so that you can determine average time needed to develop automated test cases in the future. Track how many defects are found post-production to determine if your automation efforts are paying dividends. A good application lifecycle tool should allow you to track metrics for both automated and manual test cases.
Build on your Successes � Once you have successfully implemented automated testing on a single project, roll it out to more projects. Help other teams in your organization learn the benefits of automation and help them do it right.
Schedule Your Automation Efforts - A good application lifecycle tool will allow you to organize your automated test cases into "test sets" that allow you to run test cases in a specific order. It should also allow you to schedule those test sets to run at specific intervals (nightly, weekly, monthly, etc.).

How can I learn more about Test Automation?

There are a number of automated and manual test management solutions on the market, as long as your selected solution allows you to follow the best practices of this newsletter, you can easily begin making the transition to automation and begin receiving a return on investment from your efforts.

About the author

Steve Miller is President and CEO of Pragmatic Software, responsible for formulating strategic goals for the company, and leading corporate initiatives. Pragmatic Software flagship product is Software Planner, an award winning application lifecycle management (ALM) tool that manages all phases of the software development life cycle.

"+title+"

Sun, 12 Jul 2009 23:09:15 -0700

Himanshu Joshi

Test Metrics and the Process!

A Metric is a quantitative measure of the degree to which a system, component or process possesses a given attribute. Software metrics are measures that are used to quantify the software, software development resources and software development process. A metric is defined to be the name of a mathematical function used to measure some attribute of a product or process. The actual numerical value produced by a metric is a measure.

For example, cyclomatic complexity is a metric; when applied to program code, the number yielded by the formula is the cyclomatic complexity measure.

* Management metrics, which assist in the management of the software development process.

* Quality metrics, which are predictors or indicators of the product qualities.

Metrics related to software error detection (�Testing�) in the broad sense, grouped into the following categories:

General metrics that may be captured and analyzed throughout the product life cycle

Software Requirements metrics, which may give early warning of quality problems in requirements specifications

Software Design metrics, which may be used to assess the status of software designs;

Code metrics reveal properties of the program source code;

Test metrics can be used to control the testing process, to assess its effectiveness, and to set improvement targets;

Software Installation metrics, which are applicable during the installation process;

Software Operation and Maintenance metrics, including those used in providing software product support.

Test Metrics

The following are the metrics collected in the testing process.

1. Defect age.
Defect age is the time from when a defect is introduced to when it is detected (or fixed). Assign the numbers 1 through 6 to each of the software development activities from software requirements to software operation and maintenance. The defect age is computed as shown.

(Activity Detected � Activity Introduced)

Average Defect Age = ��

Number of Defects

2. Defect response time
This measure is the time between when a defect is detected to when it is fixed or closed.

3. Defect cost ($ d)
The cost of a defect may be computed as:

$ d = (cost to analyze the defect) + (cost to fix it)
+ (cost of failures already incurred due to it)

4. Defect removal efficiency (DRE)
The DRE is the percentage of defects that have been removed during an activity, computed with the equation below. The DRE can also be computed for each software development activity and plotted on a bar graph to show the relative defect removal efficiencies for each activity. Or, the DRE may be computed for a specific task or technique (e.g., design inspection, code walkthrough, unit test, 6 month operation, etc.).
[SQE]
Number Defects Removed
DRE = �� * 100
Number Defects at Start of Process

5. Mean time to failure (MTTF)
Gives an estimate of the mean time to the next failure, by accurately recording failure times t i , the elapsed time between the ith and the (i-1)st failures, and computing the average of all the failure times. This metric is the basic parameter required by most software reliability models. High values imply good reliability.

MMTF should be corrected by a weighted scheme similar to that used for computing Fault density (see under Test Metrics).

6. Fault density (FD)
This measure is computed by dividing the number of faults by the size (usually in KLOC, thousands of lines of code).

Hope these are useful.

QAGuild Weekly Article

"+title+"

How to Build and Maintain a Successful Outsourced, Offshored Testing Partnership

References

"+title+"

CHANGE LEADERS

INTRODUCTION:

Why Change implementation fails?

Change Deployment Frame work:

The 8 Stage Change Implementation Frame work

Stage 1: Creating Sense of Urgency:

Stage 2: Create a Guiding Coalition:

Stage 3: Developing a vision & Strategy:

Stage 4: Communicating the change vision:

Stage 5: Empowering Employees for Broad based Action:

Stage 6: Generating a Short-term win:

The role of a short-term win:

Stage 7: Consolidate Gains and Produce More Change:

Stage 8: Anchoring New Change in to the Culture:

Precautions:

Conclusion:

"+title+"

Why Developers should Do Testing

Enable the developers to gain system knowledge

Improve the developers writing skills

Improve software quality

Make the developers better software engineer!

"+title+"

Learn �Proactive Approach� Which helps us in saving costs

"+title+"

Layered Process Audit (LPA).

"+title+"

Database Security; why expensive tools may not be the whole answer

"+title+"

Model-Based Testing and Its Tools (part 3 of 3)

State-of-the-Art Test Sequence Generators

Test Automation with XML Scripting

MBT Execution, Debugging and Monitoring

Load and Stress Testing

Licensing Editions

Further Readings

Tools :

"+title+"

Model-Based Testing and Its Tools (part 2 of 3)

MBT Tools

TestOptimal Architecture

Supporting Full MBT Process

Modeling with TestOptimal

"+title+"

Model-Based Testing and Its Tools (part 1 of 3)

A Different Approach to Software Testing

MBT Process

MBT Modeling

Test Sequence Generation

Test Automation Integration

"+title+"

How to Evangelize Unit Testing

First, a little background

"+title+"

Model for evaluating value at risk for Risk Based Testing

Introduction

Why Risk based testing?

The traditional risk model for testing

Value at Risk Model

Risk identification

Risk description

Risk Estimation

Probability

Impact

Risk Estimation in the Value at Risk Model

Obtain Value at Risk table

Risk Mitigation:

Case study

Risks Identified in testing area :

Summary

Reference:

"+title+"

Automated GUI Testing � Challenges!

Capture/Replay Tool to save the day?

"+title+"