Sharon Niemi, Practice Director of SQA, talks about how the right combination of predictive and reactive metrics can help you build a measurement portfolio that improves product quality and release consistency. You’ll learn how to build a measurement system that incorporates leading and lagging indicators to improve your team’s consistency in delivering quality products on time and within budget. Near the end, Jeff Amfahr, Director of Product management at Seapine Software, demonstrates how Seapine’s TestTrack solution for product development processes makes capturing and reporting on these metrics possible.

Answered Questions 

Q: As someone relatively new to QA, would it be beneficial to take a six sigma course or should more experience be gained prior to six sigma?

A: Six sigma is really a fundamental approach; I would definitely look to see if the organization was looking to implement six sigma practices. If so it would definitely be worth your while. I would go out and see what the approach to measurement is and then make your decision on what training you need

Q: What do you think of technical debt metrics, used as a software quality metric during development, as a predictive measure?

A: It is a very useful one for our customers. As your technical debt grows, and as that backlog of issues grows, it’s inevitable that you will be in a constant catch up game. We look at the growth of the backlog over time as well as make-up of that growth in terms of open feature requests, issues, and architecture feedback from the development team. Those are all things that we look at as a critical metric to view overall quality.

Q: Is one person or group expected to claim ownership of the measurements, or is it a shared responsibility between multiple people to update measurement controls?

A: I think you need one person assigned to the measurement system itself. That person would then identify the measurement techniques that are going to be used, the roles that will be required, and so forth. The ownership of the measures themselves would be placed on the manager who is responsible for the process. So there would be one owner of the measurement system but multiple owners of the metrics. The business lines would probably have someone own the requirement metrics and the stability and the rate changes. Then, the test manager who would be responsible for the defects, and the departments would be responsible for providing the measures around capabilities. To reiterate, the measurement system has one owner but the metrics should have shared ownership.

Q: Can you use TestTrack to predict level of effort on future requirement changes? What do you input to the tool and what do you get out?

A: You can configure TestTrack to require users to enter an estimate of effort when they create test cases. You can also require users to enter how long they’ve been working on a test, or how much work is left to do. When they complete a test, you can require them to enter how much time was spent on it.

To use that information for predictive metrics, we look at some charts and some of the metrics run on the variance column I talked about. That shows us how we are doing relative to our estimate and what our variance is. Our customers use that variance data in two ways.

Initially, that variance gives insight into where estimations have been way off and management can use that information to train the team and individuals on better estimation techniques. Second, once you have variance over a few development cycles or sprints, you can start to buffer estimates with common variance numbers. So if one part of the project seems to consistently under-estimate the effort by X hours or points, adding that into the final estimates can get your schedule more consistent.

Q: Is there a way to organize a metric review to get to actionable outcomes?

A: Yes, you need to have weekly or monthly meetings to review your portfolio of measures and what is being presented. Structure the meetings so you review predictive measures first and outcome measures second. If an outcome measure isn’t meeting the specified goals, ask the owner of the measure to discuss what they are doing to affect the results of the measure next week/month. Makes this process part of your measurement system, building it in as part of the ongoing measurement review and meeting discussion.

Q: Which TestTrack tool was used for the demo? Is it a reporting tool? Is it based on TestTrack Pro, TestTrack TCM, TestTrack RM, or all three?

A: We used all three tools in the demo. If you’re not familiar with TestTrack, the tools have a unified interface.

Q: How do you get time in a particular state in an item/defect workflow?

A: That was a calculated custom field, which were introduced in TestTrack a couple releases back. You can now set up custom fields that are calculations based on other information; so things like time in state were all calculations based on information that I would otherwise display back to the user.

Dr. David Vogel of Intertech Engineering runs down his classification of software risk control measures, and provides specific examples on when each type of risk control is best used.

1. Inherently safe design—global design requirements or constraints that render the potential hazard or harm all but impossible.
2. Preventative controls—requirements and constraints that help to prevent a hazardous or harmful situation from materializing.
3. Corrective actions—”detect and correct” risk control measures take a corrective action when a hazardous situation is detected.
4. Mitigate—the severity of harm resulting from a hazard is reduced, but not necessarily eliminated.
5. Soft controls—labeling, training, and operator instructions

This is a short excerpt from our recent Leveraging Traceability in Your Risk Management Strategy webinar. Click the link to watch the entire webinar.

Agile Point of View

I work remotely, as does many Agile team members. In this article, Shane Hastie and Johanna Rothman talk about how to get the most of out a distributed Agile team.

While most Agile testing has or is becoming automated, that doesn’t mean that all automated testing is Agile, says Ole Lensmar.

Ben Linders talks about different approaches to grooming our backlog lists.

Testing Philosophy

Yes, estimates are lousy, and they are going to keep being lousy, says Dan Milstein—and he invokes Kahneman to tell us why that’s so.

I don’t know much about the Miagi-Do school of software testing, but Matt Heusser is helping to educate me.

Finally, a higher level perspective and voice of reason in the increasingly heated debate on testing certification. Leave it to testing dean Cem Kaner to put the entire discussion into perspective.

I’ve started presenting on the topic of exploratory testing, with a focus on how to transition between it and scripted testing without losing important information. But I’m always trying to learn more about exploratory testing, and in this article Petter Mattsson offers one of the best explanations I’ve seen.

Jonathan Kohl answers the question we should all be asking – whether we create value with our testing.

It’s all about storytelling, says Jonathan Kohl.

Interesting Read

I’m not a big fan of the “anyone can learn to code” movement. That said, here are ten places where, well, anyone can learn to code. Forgive me.

Andy Lester provides us with an abridged list of bad interview questions for technical jobs, and how you might go about answering them.

Hala Saleh reminds me that the way to progress in our lives is to embrace discomfort. This is really important, and we don’t do nearly enough of it.

One of the most amazing biases described by both Ariely and Kahneman is the anchoring bias. In an experiment, Kahneman asks subjects to spin a “wheel of fortune” that is designed to stop on one of two different numbers. He then asks these subjects how many countries are there on the African continent. The number they spun on the wheel of fortune greatly influenced their resulting guess. It turns out that we can become anchored by a random value prior to making other decisions that concern numbers.

In economics, Ariely notes that initially setting the price of a product at one level serves to anchor the consumer to that price. If the price then goes down, we automatically believe that the lower price represents a better deal, irrespective of the profit made on the product or of our own utility in owning the product. Apple has practiced this technique well, first setting the price of the initial iPhone high, then lowering it within weeks. Potential buyers became anchored to accept the higher price, and thought they were getting a great deal at the lower price.

In testing, the anchoring bias can leave us mentally transfixed on our numbers, whether they are defect count, find and fix rate, tests passed, or other metric. An initial value, early on in a project, can serve to provide us with an anchor by which we evaluate subsequent data. This isn’t necessarily a bad thing, except that we may have gone into the project with expectations as to what those values need to be in order to meet the project goals.

One of the best ways to overcome the anchoring bias in testing is to share those metrics with other stakeholders. By sharing data as broadly as possible, we start getting different perspectives and questions about the underlying meaning of the data. The questions can prompt the team to start a discussion of the underlying meaning of the metrics.

Another way of overcoming this bias is to define your expectations prior to the beginning of the project. Those expectations may be based on an assessment of the required quality, and what metric values directly correlate to that level of quality. By anchoring to numbers based in logic and design, we can train our biases to come to the right conclusions.

In general the anchoring bias tells us to not depend upon metrics so absolutely that we fail to consider their meaning in context. We need to encourage ourselves and our team members to think deeply about the meaning of metrics in the context of the actual software.

Seapine’s TestTrack provides the ability to capture and report on many possible metrics, with predefined and easily customized reports that make it simple to share those measurements. And thanks to linking between project artifacts, we can delve behind the numbers to better understand their impact on the quality of our project.

Agile Point of View

Ian Mitchell talks about whether or not estimation should be an integral part of Agile development processes. This is a great overview on how to begin to estimate the amount of work required by user stories.

Cameron Laird says that managing technical debt during your Agile development effort will result in applications that are more business-ready.

Testing Philosophy

There has been a heated debate in the testing community on the reality and value of best practices. This was started by Rex Black, who has been espousing the value of certain best practices. While others have taken to Twitter to object, James Christie has written down his thoughts on the fallacy of best practices in a blog post. Christie follows that up with a second post on whether best practices actually provide protection for those seeking to build and deploy quality software.

Further, the debate on best practices is really an outgrowth of an even more heated discussion by the same participants on the value behind testing certification, most especially that offered by the ISTQB. While James Bach has been leading the charge against ISTQB certification on Twitter, Keith Klain has written a blog post outlining his objections to the claims made by the organization about the value of certified testers.

The EuroSTAR conference has done a great job of summarizing the debate on certifications, citing multiple blog posts. It also notes that the time has come to separate the entire topic of certification from that of the value of testing courses.

Scott Barber notes in this video he recorded at STPCon in April that business executives see software testing as an expensive practice that doesn’t contribute to business goals. He looks at ways that testers can be better contributors to the business. If you watch one professional video on YouTube a year, this should be the one.

Huib Schoots is looking for nominations for the top 10 best books for testers. Make your recommendations here.

I’ve enrolled in another MOOC, this one on software testing. Check it out on Udacity to see if it might meet your needs.

Seapine View

Seapine is sponsoring a free webinar with Software Quality Associates (SQA) to show how testing teams can dramatically improve productivity by adding predictive metrics to your measurement portfolio. This will be held on May 16, and is available for sign-up now. Read more about it here.

Interesting Read

I confess that I don’t have anywhere near enough of a physics background to understand the concept of a quantum Internet, but it looks like US scientists at the Los Alamos Labs have created and are using just such a network designed for completely secure communications. Will this concept be available for general use in the near future?

The ability to lead and the ability to communicate with influence are two of the most valuable skills that test professionals at all levels need to perform their roles effectively.  Test skills are important, but softer skills such as negotiating and leadership are starting to take center stage at all levels.  Several of the presentations in the Leadership Perspectives for Testers Track at STPCon focused leadership and influence.

I took part in one of the eight-minute “speed-geeking” sessions, where speakers communicate a topic to impromptu groups of attendees around a table. Matt Heusser and I addressed aspects of communication and influence.  Matt used the facts surrounding the Star Wars’ Deathstar destruction to reach a completely different conclusion than the Star Wars movies; he concluded that it was an inside job. Using this analogy, he challenged attendees to use the facts about test results to communicate the true picture, rather than the picture the organization wanted to see.  In Why Didn’t I Find that Bug?, I showed how testers miss bugs and misinterpret test results based on their biases and preconceived notions regarding the application under test, requirements, the developers and the goals of the project.

In her presentation, Become an Influential Tester, Jean Fraser examined many aspects of influence including how it can be positive or negative, that it comes with responsibility and that leadership is influence.

Pete Walen, in Stepping Up to Leadership: Test Leadership Lessons from Harry Potter, discussed the meaning of leadership and how we can all be leaders regardless of our position within our organization. He suggested that testers are leaders if they have not only information for which team members look to, but also, a vision and a roadmap to get to the next level.  He provided ways of becoming leaders including self-examination and evaluation, formal training, following your curiosity, writing and speaking about your experiences in testing, and networking and building professional relationships. Pete is an honest and forthright speaker who gets left a powerful impression on his audience in espousing testing leadership.

Lynn McKee took leadership and influence a step further in her keynote by describing a process for leading and influencing change within organizations. While Lynn gave an inspiring talk, she didn’t draw upon any of her own personal experiences to add strength and validity to her words.

In my joint presentation, Looking For Love In All The Wrong Places: Achieving Quality Without Organizational Commitment, testing manager Gerie Owen and I showed how to develop and advocate a vision of quality to obtain senior management commitment. We stressed that the proposal must be in synch with the organizational values, for example; implementing an automated regression test suite where speed to market is business critical. Then we provided ideas for creating quality with little time, money or resources.

In the current business climate where testing is often automated or executed offshore, leadership and influence skills are becoming increasing critical for quality assurance professionals. QA managers and team leads must effectively interpret the test results and articulate risk to the project stakeholders. As information technology project management evolves and we move from waterfall to agile methodologies, testers need be vocal, influence and constantly demonstrate their value. And it will be our ability to lead and influence that will keep quality assurance professionals at the leading edge of change.

According to Peter, exploratory testing can be a path to structured testing activities. By creating test cases and recording your explorations, and automatically submitting defects during that process, you can incorporate your explorations seamlessly into a larger structured testing process.

This session provides actionable ways to use exploratory testing as a jumping off point for creating manual test cases, automating those test cases, and submitting and tracking defects found as a result of exploratory testing. Peter emphasizes through example how exploratory testing can become an integral part of the full testing process, and concludes with several actionable takeaways that enable testers to immediately use exploratory testing as an entry to automation and defect tracking.

Register for From Exploratory Testing to Test Automation today, and be sure to check out the other great sessions at the STP Online Summit for Test Automation: Anatomy, Architecture & Approaches.

A custom calculated field can be created as a flag using a simple formula that first finds the event count, and if the event has happened at least once compares the most recent event with the event that occurred just before that. This flag can then be used in automation rules to move items through the workflow or notify users.  The output type for this field is numeric (no decimal places) as it will only have the value of 0 or 1. Since the field will only be accessed by automation rules you will want to modify the Security groups to hide the field for both Add and Edit actions.

Once you have created the field create a filter that restricts the value of the custom field to “1”. Then, use that filter in an automation rule with the trigger based on the event being added to the item. For example, a rule  to notify the assigned users via email that a critical defect date has changed would look like this:

[Notify on Date Change] applies to [defects] passing filter [Date Changed] -- when [Estimate event is added]

Send email using template [Defect Changed] to [all assigned users]

The formula below can be copied and pasted directly in the formula field.  Modify the event and field name as needed.

//has Completion Date changed?

var x=Item.Events.count("Estimate");

var chg;

//no events so set to 0

if (x==0)

    chg=0;

//first event means date has changed

if (x==1)

    chg=1;

//multiple events

if (x>1)

    {

    //get previous event index (index is 0-based, last event is x-1, the event before that is x-2)

    x=x-2;

    date1=Item.Events.at(x,"Estimate").fieldValue("Completion Date")

    date2=Item.Events.last("Estimate").fieldValue("Completion Date")

    //if the last date is equal to the previous date then it has not changed (0)

    if ((date2-date1) == 0)

        chg=0;

        //else it has changed (1)

     else

        chg=1;

    }

//return the current value of chg

chg;

If you’re like the majority of QA professionals, defining and capturing the correct data metrics can seem like an overwhelming task. What are the best metrics to help manage quality proactively? What are the benefits of investing time into developing your measurement system? Why should you measure at all?

Your data is not your master—it is a tool for continuous improvement that, if leveraged correctly, can improve performance, communicate progress, and improve your overall development process. The key is to determine which metrics provide the richest source of information, then leverage that for maximum value.

Join SQA’s Practice Director for Lifecycle Optimization, Sharon Niemi, in a collaborative webinar with Seapine Software for Use the Windshield, Not the Mirror: Predictive Metrics that Drive Successful Product Releases. In this informative webinar, Sharon will guide you through a case study of one company’s experience learning how to develop a mature measurement system.

Sharon will explain the process from beginning to end, covering topics such as:

• What measurement questions should be asked
• 7 steps to capturing data
• The lifecycle of a metric
• The difference between can and should—just because you can measure it doesn’t mean you should
• How to analyze the data, and the link between cause and effect
• How to use the data to create a line of sight

To conclude, there will be a brief demonstration on how TestTrack makes capturing data easier, providing built-in metrics visibility throughout the product development process.

The 60-minute webinar will be held on May 16 at 1:00 p.m. ET. Learn more and register.

Agile Point of View

Amr Elssamadisy notes that context is everything. There is no silver bullet; even the best Agile techniques can fail without the right people and culture.

Far more teams have attempted to make the transition to Agile than will succeed. Johanna Rothman tells us why Agile is not for everyone.

Testing Philosophy

Fiona Charles always tells it like it is. Read an interview with her on her thoughts on how to deliver bad news as a tester.

QA Heaven offers up his first weekly newsletter on testing and quality issues around the world.

Real World Testing talks about how to abuse testing metrics in the pursuit of quality.

Iain McCowatt takes once again takes up the topic of testing versus checking, and in doing so invokes Ben Kelly’s outstanding presentation and posts on The Testing Dead.

Seapine View

I’ve started detailing some of the biases in testing that I explored in my Moneyball presentations and writings. The first two of those posts can be found here and here.

Dave Borcherding writes about my presentation at STPCon on April 25^th.

Interesting Read

How should we reward our team? Not just with salary, says Lisa Vaas. Salary is just the ante for showing up, not for doing great work.