Training Mentors
Typically delivered internally this type of training is usually based in the real world. It involves having your trainer work side by side with the students in a live environment. The aim is to have the mentor(s) help staff by walking them through the system as they work normally. The big advantage here is that the students learning is reinforced by working in a familiar environment. The down side is that the effective mentor to student ratio needs to be quite low (e.g. one mentor to 1 or 2 students) for this to be effective. This of course can add to the cost or time scales if you have the recommended low ratio. Alternativley if you have many students to one mentor the teaching tends to get diluted.

Training Delivered Internally
Here the company implementing the test management system uses their own internal trainers to teach the students. The big advantage of this is that the internal trainer usually knows the companies internal processes and systems very well. In this way he or she can help cover scenarios that are important to company with depth and authority.

Usually to implement this approach an external trainer will train the internal trainer. The down side to this tends to be that many companies don’t have a dedicated internal training team. So they pass the job onto someone in the organisation that has little experience training. It also relies on the internal trainer understanding the test management tool well.

Training Delivered Externally
In this scenario the vendor supplying the test management tool offers a training package along side the purchase of the software. Here you can expect the vendor to have experienced trainers that understand the product inside out. Conversely they are unlikely to understand all of the companies internal processes and systems. As such they are restricted in the extent to which they can relate the training to the companies real business practices.

Remote Training
Many enterprise level solutions these days will come with some sort of computer based training package. This may be standalone video tutorials or training guides. Clearly the down side to this is that it relies on the student putting aside time to study and learn. Difficult when you have a day job demanding your time.

A good compromise on the remote training front can be delivery of training remotely by an external trainer. This type of training is usually carried out with tools like Go To Meeting or WebEx. Here the external trainer can share their desk top and talk directly with the students over a conference call. This approach means students have to dedicate time to the training. Of course they also get a specialist training who’s very familiar with the product. Without face to face interaction it’s unwise to allow each session to go over 2 hours. Students’ attention soon starts to drift without that direct interaction with the trainer.

Summary
There are many training options available beyond the usual “get an external instructor” solution. All of them have their benefits and disadvantages. So the option you choose will usually be dictated by a number of factors. Firstly do you have the resources in-house to undertake the training to a decent standard. Secondly how crucial is it for the students to learn about the test management system in “your” business context. If you have little resources and the context isn’t important then remote web based training may be the best solution. If you have the resources and context is important then your own internal trainer may be the way to go. Whichever option you go for it’s extremely important that you never underestimate the importance training has on the uptake of you test management system.

So when we see failures, like the three examples that follow, it’s difficult not to feel a large degree of empathy for the some of the software testers. These people are likely to be bearing the brunt of the failure. I’ve worked in financial companies where failures cost millions in lost revenue. I’ve seen testers fired on the spot in the witch hunt that follows. I’ve seen, miraculously, testing budgets doubled after such a failure. I’ve seen boards of companies suddenly understand why testing is so important. No doubt these three companies are currently following that same cycle.

1. US financial conglomerate fined millions for covering up bug

A fund that used computer models to work out its trading approach was found to have a significant bug. A bug that resulted in investors loosing millions of dollars. When questioning these losses investors were fobbed off with explanations around market volatility. The real reason was a software defect in the trading algorithm. Whilst it is alleged that employees found the issue and acted to resolve it the company alleges that they tried to hide the issue. What ever the background the company was fined heavily by the SEC (Securities and Exchange Commission).

2. System issues result in ATM downtime for one of Japans largest bank

Issues on the ATM network for one of Japans largest banks resulted in a complete outage of the network. With thousands of machines out of action for nearly a day customers were left having to withdraw from branches only. This was compounded with failures to make salary payments and a million unprocessed payments. In conjunction Internet banking facilities were offline for three days.

3. Australian ATM defect gives customers extra money

With 40 ATMs giving out significant sums of money by mistake this Australian bank’s customers thought they had hit the jackpot. Apparently with the machines operating in a standby mode customers could withdraw funds without being prevented by any account limits. With this issue lasting more than 5 hours large queues formed with customers withdrawing funds well past limits set on their accounts.

There’s a predictable pattern when high profile failures like this happen. Blame the test team. They blame poor process (like the test management systems in place), lack of resources and poor requirement definition. Senior management wake up to the importance of the QA function. Budgets double to ensure it doesn’t happen again. When the dust settles budgets get cut as part of a company wide efficiency drive. Failures happen due to lack of resources/commitment to the QA process. The cycle starts all over again.

If these examples highlight one thing it’s the complexity of integrated systems. No longer are we testing a single system in isolation but we need to be testing massively integrated systems that all rely on this interconnectedness to function correctly. Some may argue that this complexity is going beyond our ability to comprehend and test effectively. Perhaps we’re at a point where no amount of well thought out test management process and software testing skill is going to prevent these types of failures?

With 3 months of use under our belts our impression is that SmartBear have been listening to their client base. They have restructured the way testcases are managed but retained the core traceability and visibility features. Essentially it’s still easy to use but delivers a far richer feature set for the QA team.

So here are some of the key features from this release that stood out for us:

Releases

You can now define releases in terms of versions, builds, iterations (or using which ever nomenclature best suite your setups). With releases defined you can then link all other test management artifacts back to the release. Where this delivers big benefits is with traceability reports for a release. You can now quickly run a report to see which requirements have been implemented for a release, which defects are impacting the release and which sets are being executed. This does make it really easy to see at a glance exactly where you are up to with a release.

Test Management

The whole approach to testing in QAComplete has been restructured. No longer do you have a ‘Test Cases’ area but a whole new ‘Test Management’ area. This delivers a test library, configurations, sets and an execution area. This brings QAComplete right into the same space as HP’s Quality Center product with a similar feature line up. Not only that but also the capability to export your data from QC and import directly into QAComplete.

Library

The new library feature allows you to store all your testcases in one area of QAComplete. These can then be reused within sets. Tests are now version controlled and contain steps (a major omission in previous versions which is now rectified). The steps feature is easy to use. You can enter steps and expected results in much the same way as you would in a spreadsheet. In addition you can quickly reuse steps from other cases by dragging and dropping from other testcases in the library.

Configurations

Test management execution runs are now tracked against configurations. This makes it easy to trace runs against say different browser types. You can structure your configurations into folders, add custom fields and link to other artifacts like defects.

Test Sets

All of this comes together in the new test sets area. A set contains a collection of cases from the library. The test management set is then executed against a configuration and a release. So for example you might run your regression set against, release version 1 and XP/IE8 configuration. Each testcase within the set is then executed in turn, where each step within the case is given a pass or fail result. The first core concept to grasp here being that a set is linked to the releases it will be executed against and the configurations the tests will be run on (see below).

From here when you run the set you are given the chance to specifiy exactly which release and configuration you will run this against. Crucially here you can specifiy multiple combinations at the same time. So you could for example run against version 1 and version 2 of your product at the same time. Or you could run against config A and config B at the same time.

Execution

The new run and execution engine in QAComplete makes running the testcases very straight forward. Once you’ve specified the release and config to run against you can step through each testcase in the set and mark the individual steps as either pass or fail. Time taken to execute is tracked along with percentage complete.

Whilst each test case is given an individual status, the set is also given an status. So when you complete the set you can either pause the run, or end the run as a pass or fail.

In summary this is quite a prescriptive workflow that has been implemented within QAComplete’s test management engine. It’s either an approach that will work well for your process or it won’t. Having said that it’s an approach that has worked well within other popular test management tools for years now, so it’s well proven and well worth trying out.

You’ll see in this video that there are two core approaches to exporting. The first is to export directly from within the GUI to CSV format. This involves selecting the data you wish to export (usually by defining filters to select the test cases you need) and then selecting one of the two following options…

1. Export visible: using the ‘select fields’ feature you first select the fields you want to display on the listings page. It’s this data that is then exported when you select export visible data

2. Export all: regardless of the fields displayed or selected all data associated with the test case is exported.

Note that in QAComplete this export option only supports CSV format. If you need this in excel then just open the CSV file in Excel and then save again in Excel format.

The second option involves creating a test management report and then exporting the report data. QAComplete comes with various test management reports from which you can then export the underlying data. Note that you can only export the information that the reports suck out of QAComplete. If you don’t have a report format that covers the information you need then you can’t export the information you need. Having said that you can create your own reports with exactly the data you require. Once the information is displayed in a report you can then send this to CSV, Excel, Pdf, Word or XML formatted files. See the video to understand how this is achieved.

In short every QA team has differing requirements for exporting data. The key part is to make sure the test management tool you select supports an extensive range of formats so that you have the flexibility long term to produce the exported data your team is likely to need.

On a fairly regular basis we get asked for a specific test management report that will show the aggregated results from multiple runs. The idea behind this is that ultimately you can show that every testcase was run and that every one passed.

So for example on test cycle 1 you may get…

my_test_case_A:  failed
my_test_case_B:  passed
my_test_case_C:  passed

Then on cycle 2 you get….

my_test_case_A:   passed

The aggregated test management information would then show the following:

my_test_case_A:  passed
my_test_case_B:  passed
my_test_case_C:  passed

So we’re looking for the latest result for each testcase and showing that result as if all the tests have been run in one go. Very misleading when you consider it from that perspective.

There is a certain level of attraction to this test management information. It’s one of those reports that can be categorised in the “that gives us and our customer a nice cosy feeling that everything has passed and the product is ready for release” reports.  From the QA engineers point of view it’s nice to show a list of results to your customer that’s simple and lists every case as passed. Customers see this as a nice to have too. However, If I was the client, receiving this, I would be horrified. Results reporting in this way hide key information, and misleads on a number of levels.

Firstly, it demonstrates that little, or no, attention has been paid to regression testing. Assuming you are just re-running failures then you’re probably not re-running tests that passed first time round. If you’ve got to do a rerun then that implies you’ve made changes to the application. If you’ve made changes to the application then you need to be regression testing. As a customer that receives such a management report all I can say is that I’ve got absolutely no visibility of any regression testing that’s taken place. I’m going to start questioning the worth of your current test management tool at this point.

Secondly it entices the QA team to aim for a goal of 100% of tests completed with 100% passes. This leads to a kind of “Oh how lovely, we have a perfect application to release” syndrome. The team should not be aiming for a goal of 100% passes they should be aiming to find defects. It’s too easy to think testing is complete when you’re inadvertently enticed into producing the perfect report rather than a perfect approach to finding bugs.

Thirdly, it implies that little or no attention is being given to version control of the application under test. Such a report demonstrates that all testcases have been run against “the” application rather than some run again version x of the application and others run against version y of the application. Your product might be under version control but you’re not recording tests against the version they were run against. Without this test management information you have no traceability and no repeatability within your process.

I suspect that most requests for such a report come either from an inexperienced QA team or the end client. A client that’s looking to see information conveyed in this way ought to be educated by the QA team. A team that’s providing information to the client in this way ought to be improving their test management process as a matter of urgency.

This is not a criticism of using people from the business who have no experience of using such applications. It’s more that the people driving the project hope to maximize productivity by giving the users a tool that they can become productive with quickly. This aim should be approached with caution though. There is a trade off here. Many simple to use test management tools are easy to use for a reason. They lack the features and capabilities that enable a team to manage and track the process effectively.

For example if you don’t need the complexity that’s usually associated with traceability between requirements, defects and testcases then that’s fine. If you need to know that a certain requirement has been tested fully and is covered by x and y testcases then you have to accept some complexity in the tool. That complexity is there because the process is complex.

That complexity will take time for a team with little experience to pick up. It’s getting the balance right between your requirements for your tool (e.g. we need requirements to testcase traceability) and accepting that these needs are likely to result in a steeper learning curve for your team. Don’t delude yourself though. Do you need your test management tool to deliver certain complex features? Well don’t assume that your business team will learn the tool in one 30 minute play around session. Your team may well need training.

This is the crux of the point. Many popular test management tools aren’t complex just for the sake of it (although it could be argued that some vendors throw so many features in to help sell the product that they are complex just for the sake of it). Many applications are complex because they are helping you track a process which is complex.

You may be looking to track which testcase covers which requirement, which release the check is run against and in turn which defects are raised. This sounds simple in principal. Once you have 5 business people running a couple of hundred tests a day against several different builds of the software over the course of a few weeks….. well we all know that tracking this gets complicated very quickly. This is why test management tools were invented in the first place.

In conclusion simplicity should not be confused with usability. As Albert Einstein said “Everything should be made as simple as possible, but not simpler.”  Some of the complexity you find in test management tools is there for a reason. It’s there because you need to track information relating to your process that is inherently complex. Make sure when you are looking at test management tools you are focusing on usability and not simplicity.

On the surface a simple question. Dig a little deeper though and this question is more interesting than it would at first appear. For example what stops a test manager becoming a project manager? Or from the other direction what attributes would a project manager need to bring to the test management role in order to be successful? Why does a test manager end up focusing on the testing aspects? Why do very few test managers move across into the project management arena?

The fact that very few people seem to migrate between the roles would suggest that there are significant differences between the roles. I would argue that three key points differentiate the two roles:

1. The PM has more responsibilities relating to the budgets and making sure the project comes in on budget. Whilst the TM does have to think about costs and budgets this is not usually the primary focus.

2. The PM has the responsibility of making sure the project comes in on time. If the quality isn’t right the TM can be found actively working to make sure the project doesn’t come in on time.

3. The TM has the responsibility of making sure the quality and features of the deliverable are going to meet the customers needs. So not so much focus on cost or time scales more the features and quality.

The analogy of a ships pilot was the best point put forward. Whilst big tankers and cruise liners have a captain they always make use of a pilot to bring the tanker into port. The captain knows the tanker inside out and is responsible for getting from point A to point B safely. Now the pilot may not know this ship inside out but he or she knows the port that they are docking in back to front. It’s this experience that makes the pilot invaluable for successfully bringing the ship into the port and birthing on the quay.

Well it’s the same argument for the test manager. He or she is responsible for bringing the software product to the point where it meets an acceptable level of quality. Whilst the PM is responsible for getting the product in the vicinity of the port it’s the TM that is responsible for overseeing the balance between quality, features and time as we birth the ship (or rather release the software).

Whilst this analogy breaks down a little bit in places (i.e. the test manager doesn’t just join the project in the last few weeks of the project to help out) it does help us understand some of the important aspects of our roles. That is, the importance and judgement of the test manager grows as we get closer to release. And whilst the PM concerns him or her self with the high level picture of the project (mainly the cost and schedule), it’s the test manager that ultimately focuses on ‘what we deliver’.

It’s this difference in focus that demands different skill sets and different points of focus between the two different roles. Maybe it’s even more clear cut than this. Perhaps it’s that the project manager is concerned with the product and everything else. Whereas the test manager is concerned just with the product that is delivered. Or is that too simplistic?

Like I said this test management forum does throw up some interesting points for discussion.

You may even consider the price more important than the technology. Even so you’re still missing one very important factor. Probably the most important factor. That factor influences the success of your project like no other. That factor is the choice you make on who implements the product.

Most companies have requirements to integrate their test management solution into their existing process and environment. This means configuring the tool to match their workflow. And integrating with other tools like defect tracking and source code control tools. This is no simple matter for established QA teams.

Configuring the work flow can cause problems in it’s own right. Often it’s the first time a company sits down and actually thinks through their workflow. Couple this with a new test management tool (which you’re still learning) and you’ve got lots of opportunities to get it wrong.

Once you’ve defined the work flow incorrectly it can be difficult to put right. For example a good vendor will likely take you down the ‘less is more’ path. It’s easier to start out with fewer configurable fields and add more later. It’s more difficult to start out with lots of configurable fields, tracking every last bit of info you think you need. Then when you come to remove the unwanted fields later you find they’re locked in to some aspect of our work flow. The temptation to add more custom fields and extra work flow steps is always high. For someone in your team who’s been left to setup the tool with little experience you’ll end up tracking everything. An experienced implementation vendor will keep things lean and efficient for you right from the start.

Integration with existing tools the other big item to consider. Rarely these days will a test management tool sit in isolation. Often they will need to integrate with existing defect management applications, possibly requirements tracking applications and maybe even with source code control tools. Add to that integration with automation environments and you have a situation which no out of the box solution is going to work with. An implementation vendor will have seen this and managed this type of setup many times. The time they save getting this setup will out weigh the cost of bringing on board help many times over.

A good decision made on choosing the technology can very quickly be undone by choosing the wrong implementation vendor. Test management applications no longer live in a vacuum where you are setting up the process from scratch. They have to mould to your existing  process and integrate with existing applications. The way to extract maximum benefit from your test management tool is to choose a supplier that understands this. So separate the selection of the technology from the selection of the implementation partner.

Innovations in Test Automation - putting the case forward for IT innovations that deliver cost savings through the implementation of automation. Covering the cost challenges normally associated with implementation this session aims to identify the benefits of looking beyond just the price of purchasing the tools.

The Agile Pretence - in this session the speaker is looking to challenge that often spoken statement that ‘we’re agile’. Are we really agile? Is this phrase being slightly overused these days to describe development environments that aren’t really quite as agile as they’d like to think?

Performance by design - for most of us performance usually only becomes an issue once we’ve gone live with a new product. In this session the speaker puts forward the case for designing in performance right from the start. Building on this the aim is to discuss how the QA team should approach test management from this perspective when we take it right from the start of the application design.

User Acceptance Expertise in Contract Negotiation - examples and reasons why the QA team should get involved right up front during the negotiation of new contracts. Most testers have a wealth of real life experience that can help immensely whilst negotiating and scoping out projects.

For those interested in attending this stimulating forum on test management you can find out more here…

Test Management Forum 26th October

• Quality Center is costing you too much?
• You’ve outgrown TestLink?
• Excel and Word are not scaling up for you?

If this is the case then take a look at the latest 9.7 release of QAComplete.

      Test Management with QAComplete

The latest release of QAComplete now delivers…

Dynamic test-step management
Create test-steps with easy to drag and drop features. Build your cases with the simplicity of Excel yet keep the traceability that Excel can never deliver.

An easy to reuse library
Hundreds, thousand or even hundreds of thousands of testcases. Either way this solution scales up like TestLink and Excel could never hope to.

Traceability from all artifacts to releases
Need to check the current status of your release? One single test management report shows you cases written, runs, requirements covered and defects raised. It doesn’t get easier than this.

Execution runs that are tagged against configurations you define
One of the biggest headaches for QA teams is usually tracking which testcases have been run against which configurations. You no longer have to worry about this. QAComplete worries about it for you.

The ability to migrate form HP Quality Center
Had enough of the renewal costs for your Quality Center licenses? Yet you need a similar feature set to QC along with a the ability to migrate? We’ve go the solution for you.

All this adds up to give you greater understanding of how requirements, defects and testcases relate. As a result you’ll deliver higher quality products.

In short QAComplete now gives you a comparable test management solution to HP Quality Center at one-fifth of the cost. Want to make a good investment and see a quick return? Then trial QAComplete now….

Test Management Trial