Software Testing Zone

Lessons Learned from “How I Fixed” Posts!

2009-09-16T10:27:00.000-07:00

Yesterday during an online conversation, one of my blog readers expressed her concern: “Debasis, recently I have seen you posting 2 “How I Fixed” posts in quick succession. I wonder why you are deviating from your mission, considering “Software Testing Zone” is primarily a testing related blog”! I replied back quickly: “Well, why not? And no, I don’t think that I’m deviating from my mission to grow STZ as a testing-centered blog”!

If you are a regular reader of “Software testing Zone” and also wondering the same [why I have started writing these “How I Fixed” posts (first “IE 7 Operation Aborted Error” and now “MyWebSearch Spyware Removal”)], here is why…

I believe that technical investigation is one of the major responsibilities that we are often required to perform as testers. Personally, I love this aspect of testing more than anything else and I take pride in describing my role as a tester to be a “technical investigator” who helps the project stakeholders by providing important project related information that in turn guides the stakeholders in taking critical decisions. While trying to get rid of (fix) the above problems I think I did quite a bit of technical investigation to get around the problem and learned the following lessons that, in my opinion, can be helpful for a tester in her work:

a) How to maintain a cool head while being confronted with tough problems/situations.

b) How to narrow down the problem and get to the root of it. This particular skill can help testers while trying to narrow down the reproducibility criteria for hard to reproduce bugs.

c) How to engage in critical problem solving.

d) How to explore free information that is available via search engines and how to use them to your advantage.

e) How to use your observation skills to spot any subtle clues that can lead you to the error triggering condition of the problem.

f) How to use your critical thinking skills to pinpoint the problem.

g) How to use your lateral thinking skills to identify multiple possible causes of the problem.

I think each of the above lessons can also help someone in becoming a better tester. As testers, often we are required to do some research to learn more about a bug and do some investigation to narrow down it's reproducibility factors. As we all know, the shorter and crispier are the steps to reproduce certain bug, easier it is for the programmers to reproduce it and hence the chance of the bug getting fixed gets higher. Good technical investigation skills can also empower a tester to quickly nail down a critical defect that may be too subtle for a mediocre tester to spot and recognize. So overall, I think while trying to *fix* the above problems in my real life I learned important testing lessons.

Moreover as Dr. Cem Kaner says, “The best software tester isn’t the one who finds the most bugs in the software or who embarrasses the most programmers. The best tester is the one who gets the most bugs fixed.” (~The Bug Advocacy) I thought, why not move one step ahead and fix the problems whenever we have the opportunity to do the same? I completely understand that as testers we seldom get the chance to fix the problems (bugs, defects, issues) that we find in the software. But while testing (problem solving) in real life I did recognize opportunities where I had the chance to hunt down the problem and at the same time try and fix it myself. Thus my recent “How I Fixed” posts took birth. At any rate, I like to solve problem that has the potential to tickle my gray matter and problems like the above proved to be worth trying. And I thought that it would be appropriate to record my approach in the form of blog posts as the documentary of the episodes!

Do you feel that this is against the mission of a blog that is supposed to be testing-oriented? Do you think that I should stop writing these posts? Do you think that I am deviating from my goals as a tester? I would like to hear you out.

Happy Testing…

Confusing LinkedIn Status Update Screen when the Session is timed-out! Is this a Bug?

2009-09-14T05:38:00.000-07:00

As if the number of post categories on Software Testing Zone were few, I am introducing yet another series of posts, which I have decided to call as “Is this a Bug?” In this series I am going to post a number of software anomalies that look suspicious and buggy to me. This series will be a kind of repository where I am going to dump all the suspected software defects and bugs that I encounter in my day to day life. When I post them under this category it essentially means that these are the kind of software failures that are ambiguous and one can’t be very sure if something indeed is a software failure or it is just the way it was designed to work! I would leave the final decision (whether this is a Bug or not) up to my readers and will just post here my observations. I think this is going to be fun.

So here goes the first post of the “Is this a Bug?” series:

1. Login to your LinkedIn A/C (yes, you need a LinkedIn profile to be able to verify this).

2. Update your LinkedIn status. You can find the corresponding text box to enter and update your status on the home page, on the left side bar and elsewhere. “Your status was updated” message should be displayed by LinkedIn.

3. Now simulate a scenario of network failure.

[Hint: unplug your modem and connect it back OR disconnect the connection and connect again OR simply logout of LinkedIn -> hit the browser Back button -> Page Refresh (F5) OR use your own ingenious method to simulate a network failure.]

I had a REAL network failure and when I connected back to the internet I saw this rather confusing LinkedIn screen (please refer the screenshot) that displayed on one hand that *my* status was updated and on the other hand asked me to Join/Sign-In to LinkedIn. This is same as asking a guest at your door to come in and take a seat in your living room, offering her with coffee, chatting with her for a while and then asking her "do I know you?"!

[click on the image for an enlarged version]

What do you think? Is this a Bug? Feel free to share your opinion by commenting below.

P.S. If you had experienced such a bug?/not-a-bug? situation, feel free to Contact Me with the details (if possible screenshots) and I will try to post it on Software Testing Zone with your name.

P.P.S. This “Is this a Bug?” series is inspired by Ben Simo’s Is There A Problem Here? Blog. If you liked this post I recommend you to visit Ben’s Blog as well. Chances are very high that you would like it.

Happy Testing…

How I Fixed my Hijacked Firefox from MyWebSearch Spyware!

2009-08-26T11:31:00.000-07:00

3 days back I was browsing a popular website and I spotted “Zwinky” (which is supposed to turn a photograph into a 3D image)! It did look harmless and hence I installed it willing to give it a try. If you read me regularly, then you might probably know that I prefer Firefox over any other browser. I had used Firefox while downloading and installing “Zwinky”. But my excitement was short-lived when I spotted a new web search tool bar (MyWebSearch) that was added to my Firefox toolbar without my permission during the installation. I uninstalled this toolbar and thought it was the end of it; but soon I was going to be proved wrong!

Sometimes later, while trying to lookup something in Google, I directly typed the search query into the Firefox location/address bar. Usually, it should have displayed Google search results or a direct website based on Google’s "I'm feeling Lucky" algorithm. But this time, I got this instead:

It was super-annoying because I couldn’t search with Google, my primary search engine and Firefox, my primary web browser. It was clear that my dear browser (Firefox) was hijacked by MyWebSearch, which was causing browser redirection to their site. What the hell? I went through all of the Firefox preferences (Tools --> Options) to see if I could change back the default search engine to Google but I couldn't find any such option! Searching in Google for “MyWebSearch” gave me loads of information about this spyware. Yes, I would call it a spyware because it collects and stores information about the web pages you view, the data you enter in online forms and search fields, the "clicks" you make, the IP address, URL and country of the sites you visit, your IP address, information about your browser and operating system, and the products you purchase online while using the service. Instantly I did a scan of my PC using “Spybot – Search & Destroy”. It did find some instances of “MyWebSearch” and claimed to clean it too. But when I started Firefox, I saw it remained hijacked! Damn! :(

"MyWebSearch" Spyware Removal - Getting rid of Firefox/Google redirect Hijack:

The most irritating thing about a spyware is that it can manage to hide in your system and thus hard to be cleaned/removed. And “MyWebSearch” appeared to be quite good at it. I did the following things trying to hunt it down:

1. I checked again in “Add & Remove Programs” list. It wasn’t there.

2. I did a manual search in the “C:\program files”. I didn’t find any suspicious folder here as well. I expanded my search to whole “C:\”, without any luck.

3. Now I opened the “Registry Editor” (Start --> Run --> regedit). I did a search (Ctrl+F) for “MyWebSearch” and found 3 registry entries. I deleted them after making sure that they were the ones I was looking for. To make sure I was not missing any more registry keys, I did a search for “search” and this gave me some more entries. Out of these most were genuine Windows registry keys. But I found 2 of them were pointing to “MyWebSearch” entries; so I deleted them as well.

WARNING! If you are doing this, please be very careful while deleting a registry entry. Accidental deletion of a genuine entry may result in corrupted Windows that can only be fixed via reinstalling Windows.

After deleting the relevant “MyWebSearch” entries from the registry I was almost sure that this time it was finally removed. I started Firefox and oops; I was wrong! It was still hiding somewhere and hijacking my search results everytime I tried to do a quick Google Search via Firefox location/address bar. I searched on Web in hopes of finding out a MyWebSearch removal tool. But most (all) of them described how to get rid of the toolbar, which I had removed already. I could hardly find any info that could help in getting back my hijacked Firefox. I tried HijackThis (a free spyware removal tool by Trend Micro) too. But it was unable to sniff out “MyWebSearch” in its scan result.

I was beginning to get frustrated at this point and suddenly another “test idea” came across my mind. I went to the configuration mode of Firefox by typing “about:config” on the location bar. But searching for “defaultSearch” in the filter bar, gave me “Google” as the default engine! Damn. Where did they hide the redirect hijack configuration then?

Baffled, I now keyed in “myweb” in the filter box and here it was. It showed me the entries where the user setting was modified to hijack the browser, without my permission.

I right-clicked both the entries, choose “Reset” and restarted Firefox.

Hurray! And now the search result is back to Google. I am glad that this nasty hijack episode is finally over for me. Sorry Firefox. You had to spend 3 days in hostage situation due to my stupidity (in deciding to try out a malicious program like “Zwinky”). If you are facing a similar situation of browser hijack and looking for a way out, feel free to try my above steps and let me know if it helped.

Happy Testing…

Interviewing a Testing Expert - Phil Kirkham

2009-08-01T10:16:00.000-07:00

I am back again with yet another interesting Interview with a Testing Expert; this time with Phil Kirkham (from UK) who is a Programmer turned Tester turned Test Consultant at Acutest. He writes about testing on his blog at Expected Results! He is a Moderator of Software Testing Club and Testing Reflections as well. I felt it would be exciting to interview him and when I approached him he was kind enough to honor my interest in taking his interview. Here is what Phil has to say:

Debasis: What led you to become a software tester? And what was the topmost reason that attracted you to the field of testing?

Phil: I was working as a programmer but the s/w we were sending out was of really poor quality and the customers were getting more and more irate. My boss knew I had a knack for breaking things and debugging so asked if I would help out with the testing effort. At the same time I was also thinking about a career change after 20 years of programming, I'd got a few career change books ( such as 'What Colour is Your Parachute' ) and realised that my personality traits and tester traits were a very good match. I'd also read a few testing books by then and started to realise that testing was more than just banging away on a keyboard.

Debasis: Did you try testing anything other than software before diving into software testing?

Phil: Funnily enough, when I left university I was on a graduate training scheme at a large company and was moved around various departments. One of the departments was testing a sea-mine clearance system which meant using a large device to simulate the noise of a ship.

Debasis: Tell me 5 unknown/least-known facts about you.

Phil: (i) I used to coach a girls soccer team. After that, managing anything else is easy.

(ii) I won a slogan competition run by a coffee company and the prize was a £10K holiday which was spent on 2 weeks in the Seychelles.

(iii) My first money making scheme was finding lost golf balls at Royal Birkdale golf course. That was a bit like finding bugs - you knew there were some hiding and you got to know the areas where they could be found.

(iv) I have a scar on the back of my right hand after I put it into an oven when I was 3 - unsupervised exploratory testing can be dangerous! :)

(v) My first website was set up over 13 years ago - all about footy and the girls teams I was coaching and my daughter was playing for. Some pages still exist [http://www.geocities.com/Colosseum/3562/wkham.html]. Doing this site taught me a lot about building a community, lessons I've tried to put into practice with the Software Testing Club.

Debasis: What was the hardest challenge that you faced in your career as a tester?

Phil: Trying to convince management to take testing seriously and that being 'agile' did not mean shipping out a program after testers had give it a "quick once over". That and trying to get a tester I worked with to read a testing book.

Debasis: Tell me about the most satisfying moment in your testing career.

Phil: Sitting in a meeting near the end of a project and hearing someone say that the testing effort seemed so much more professional now that I was involved. And someone adding onto the end of that that they wish I could be cloned. Also in an annual review having it noted that "Phil's testing efforts saved the project from disaster".

Debasis: Tell me of any situation when you had wished you were NOT a tester!

Phil: The end of year pay review when the CEO decided that because I was now a tester I shouldn’t get as big a raise as the programmers (despite the comments being made in the answer above).

Debasis: Has the profession (testing) ever affected your personal life? If yes, how?

Phil: Couple of ways. I can sometimes embarrass my wife by trying to break things when out in public - such as the self-serve machines in the supermarket.

And when I was trying to change the culture of a company from a chaos culture where testing meant bashing keyboards to somewhere where testing was given some thought I found I couldn't switch off after 5:30. The work/life balance was way off.

Debasis: What do you think as the most essential skills that make a great tester?

Phil: Desire and ability to learn and a passion for testing.

Debasis: How do you see software testing as a career, let’s say after a decade? What would be the biggest challenges for the field and what would be the biggest advancements?

Phil: I had a blog post called 'The 50 years old test' that showed that in some ways not much has changed. There's the techy challenge of keeping up with all the advances that allow programmers to create apps with a few lines of code and the challenge to get management to understand what proper testing can give. I'd like to see testing moving up the chain so it takes place much earlier and I'd love it if we got to the stage where we were never finding and reporting simple boundary value bugs.

Debasis: What single thing would you want to tell every newbie who is struggling in the early stage of building software testing career?

Phil: Keep going! I found it hard to break into the field [no ISEB certificate, being a programmer rather than a tester (implies I'd be too expensive)] and I'd been at the same company for 20 years. Then a company came along that wanted people who were passionate about testing and I was on my way. Thank you Acutest.

Debasis: Is there anything else that you would like to say?

Phil: I learnt a lot from the online community and got a lot of support. I'm trying to give some of it back and would encourage everyone else reading this to do the same.

Thanks Phil for taking your time and answering my questions. It is really interesting to hear how Phil was a Programmer and became a Tester, how NOT having a Certification (ISEB, ISTQB etc) didn’t stop him from advancing in his career as a tester and how his “Passion for Testing”, more than anything else, helped him in becoming a Testing Expert. I hope all of you enjoyed this interview as mush as I did. Any thoughts?

Happy Testing…

How I Fixed "IE 7 cannot open the Internet site - Operation aborted" error!

2009-06-18T01:25:00.000-07:00

It all started from a reader’s email this morning. Duane Cantera reported: “When I try to open this web page I get an error message in internet explorer and the operation aborts.”

Emails like these are strange. They can send a blog owner into panic mode much quicker than an earthquake warning can possibly do; I was no exception. Before I knew, I had already opened Internet Explorer 7 and was opening the URL. The page started loading, as my eyes remained glued to the monitor screen and for a moment my heart raced like a speeding bullet. And boom! IE 7 crashed with “Internet Explorer cannot open the Internet site. Operation aborted.” message [here is a screenshot]. Clicking the OK button displayed "The page cannot be displayed" instead, thus restricting anybody from opening the blog URL! This was really shocking as well as unexpected for me. Shocking, because my blog was crashing in Internet Explorer [around 35-45% of my visitors use IE, mind it]. And unexpected, because I had NOT touched my blog template (HTML code base) in months, thanks to my current hectic schedules. And still here I was. All I could remember, my blog was working fine in IE 7, the last time I had checked. Hence, I was spellbound regarding what must have caused this. Instinctively, I did a Sitemeter and Google Analytics check and saw a 40% decline in my blog’s recent page visits. Damn! Must have been due to this IE crash.

But when all this was happening, the tester inside me was telling me not to panic and be rational. Quickly, I did a Google search for the above error message in IE and came up with some informative references. It looked like; it was an old bug in IE and could happen if your web page had many JavaScript code snippets (which I had many). But, still I had not edited my blog template in months. So the chance of this being a regression bug was almost impossible, in my case. Few more minutes of Googling and I came to know that this error has been reported and is now a "Known Issue for Blogger".

» I tried to follow Blogger’s recommendation to remove the “Follower’s widget” but in vain. I could not get rid of the crash.
» The more I searched in Google, the more suggestions I saw; to move all your JavaScript tags below the body tag, adding defer="defer" to your JavaScript tags… I tried them all, but still in vain.
» Even I saw suggestions to redirect the readers to Mozilla page to download Firefox. I considered this option for a while but felt that it would be too intruding for the readers.
» I searched for a patch from Microsoft so that I could redirect my readers with IE 7 there. I could not get hold of one. Instead, I came across a suggestion from Microsoft to upgrade to IE 8 instead.

I was getting tired and frustrated by now and decided to get back to my blog template code and see if I could find out the cause that was causing this mysterious crash. All I knew at this point was that this was happening due to a bug in IE 7 that caused errors while parsing the JavaScript. I didn’t have the luxury of loosing 40% of my blog readers just because Microsoft IE 7 had a bug in it and was having trouble parsing my blog template. As far as I knew, IE was able to parse my blog template till recently. As I had not made any changes myself, it had to be some third party script that must have changed and was causing this error out of the blue.

Initial investigation/testing into this issue suggested that:

» This was a problem with ONLY Internet Explorer 7 (and earlier versions) and not in Firefox, Chrome, Opera or any other.
» My blog’s homepage was opening fine in IE 7, but not the individual archived "post pages”. This gave me a test idea. I tried opening the label pages. And wow! They opened fine in IE 7.

This showed that it must be some JavaScript code snippet that was there on my “post pages” but was not on the “home”, “label” or “search” pages and was causing IE 7 to crash. I do use conditional display of some JavaScript widgets in my blog [using b:if cond='data:blog.pageType == "item"' tag]. But before scanning them, something else struck me as the possible culprit that differentiated my blog “post page” and rest pages. It was the “Comment Form”! Few months back, I had switched to the “Embedded Comment Form”. It had been working fine after an initial glitch (even in IE 7). But, it was also the one thing that stood out as the difference (as far as JavaScript code was concerned) between my “post pages” and rest pages. So, as a test, I changed my comment settings from “embedded comment form” to “pop-up”. And hurray! When I opened my “post pages” in IE 7, I no more saw any crash/errors. So I conjecture that Google/Blogger must have made some server side changes that affected it’s JavaScript code snippet that handles the “embedded comment form” and in turn it was crashing IE 7. I personally liked the embedded comment form and found it more user-friendly than the pop-up version. But for now, I have to change back and wait until either Blogger fixes it’s JavaScript OR Microsoft fixes the bug in IE 7 (the chance is low, as they have a newer non-crashing version in form of IE 8). So, I am hoping that Blogger fixes their embedded comment form code soon so that I (and many bloggers like me) can again use it without crashing our blogs in IE 7!

I must admit, it was a great and thrilling experience. Being a tester, it was frustrating to see my own blog crashing like a pack of cards in Internet Explorer. Though, the problem was with a problematic parsing error on the part of IE and a possible JavaScript error at Blogger’s end, ultimately I was loosing visitors to my blog. And it really feels great to have fixed the problem at my end in just 6 hours [Microsoft is yet to fix it in IE7 even after 1 year, and Blogger is yet to come up with a solution, mind it :)]! If you are a blogger and experiencing this problem, try if this post can help you fix it. And thanks a ton to Duane, for alerting me about this error.

Update1! Looks like the fix was not good enough to support IE 8! This is irrespective of Microsoft's claim that IE 8 has the fix that should take care (handle parse errors more gracefully by hiding them to the status bar and allow the user to continue browsing the web page) of the issue in IE 7. Thanks to Mistah Bonzai for alerting me about it. I have upgraded to IE 8 now and I could reproduce the "Aborted Operation" error, even though I was on Windows XP. Damn! Now what? Well, I don't know! Let me investigate more into this and I will update this post as soon as I have new information.

Update2! Looks like Blogger (read Google) and IE (read Microsoft) have signed a secret "no-offense; let's be friends" agreement! Out of curiosity, I reverted back to the Embedded Comment form recently and now IE seems to behave nicely without any "Operation aborted" error. I have confirmed this in IE 6, 7 and 8. And all 3 of them work fine with Blogger's Embedded Comment form now. I am hoping that this is a permanent fix. I will keep testing this compatibility from time to time and update this post in case I find anything unpleasant.

Happy Testing…

Interviewing a Testing Expert - Karen Johnson

2009-05-26T09:24:00.000-07:00

I understand that it has been a while since I have posted anything new on “Software Testing Zone”. This was partly because of the hectic schedule through which life has been going through recently for me. Anyway, I think I have settled down for now and hence I took liberty to come back with yet another interesting Interview with a Testing Expert; this time with Karen Johnson from karennjohnson.com. Karen is an independent software test consultant. She views software testing as an intellectual challenge and believes in the context-driven school of testing. She has over 17 years' of experience in software testing and software test management. She is a frequent speaker at software testing conferences and is an active participant in several software testing workshops. She's published articles in software testing publications as well. I must thank Karen for honoring my interest in taking her interview. Here is what Karen has to say:

Debasis: What led you to become a software tester? And what was the topmost reason that attracted you to the field of testing?

Karen: I fell into software testing, it was an unplanned career. I majored in journalism in college and worked as a technical writer at a software company. After a few years as a tech writer, I was getting burnt out. I was quickly becoming more interested in testing software. I had a conversation with the president of the company I was working at and asked if I could switch roles and become the company's tester. He agreed and said we could give it a try for six months. That was back in 1992 and I haven't left software testing since.

Debasis: Did you try testing anything other than software before diving into software testing?

Karen: I didn't try testing other things before I started testing software but now I feel like I could test anything. I could test the construction of a chair, I could test the functionality of an airplane door and I could keep writing a list of things that could be tested. I feel like I could find flaws in anything. And every year more and more so, I don't feel I'd have to be a chair designer to test a chair or an avionics person to test an airplane door. General systems thinking makes more sense to me now. I've learned I don't have to know every nuance about a technology to be able to break it down, dissect it and find flaws.

Debasis: Tell me 5 unknown/least-known facts about you.

Karen: 1. My dad was an elementary school principal and my mom was a kindergarten teacher. In our family, we talked about school and what we learned every night. Our homework was reviewed. While the kids were doing homework, our parents were usually reading themselves or helping one of us. I grew up believing this was what most people's weeknights looked like.

2. I grew up in Boston. I spent some amount of time every summer down Cape Cod. I have a deep attachment to the beach and the water. I'm not much of a swimmer and I don't know a thing about sailing but being near the water is important to me. I feel better near the water. I live less than 1 mile from Lake Michigan and walk the beach frequently even in the winter.

3. Over the years, I’ve believed that if I wanted to be a good writer, I should push myself to write in different genres. I worked as a news reporter. In college I published some poetry. I even tried writing greeting cards. I wrote a puzzle that was published in a children’s magazine as recently as a couple of years ago. I feel similar with software testing; if I can test software then I feel I should be able to work with different types of software and software for different industries.

4. I enjoy traveling. I have tried several times to find work in Europe and would like the opportunity to move there for a year or more.

5. My daughter is the most important person to me. I think she's fascinating. I have tremendous respect and faith in her. Having a daughter changed my entire life.

Debasis: What was the hardest challenge that you found in your career as a tester?

Karen: Not having a technical background, I took only one computer class in college and frankly, I didn't enjoy the class in the least.

Not having a strong math background, I'm amazed at how often I think having a stronger math background would have/could help me.

Not having taken a single business course, I wasn't interested. All I wanted to do was read and write and preferably be left alone to do both. I was painfully shy as a kid, loved reading and wanted to be a writer for a living.

I got over being shy, most of the time; I’m still an introvert and a loner to a great extent. I write pretty frequently now. I write various items for clients, some articles and I’m currently working on a couple of longer writing pieces – hopefully you’ll see those published this year.

I see myself as an outlier of sorts. I don't have the more typical academic background of a software tester but that's ok. I think I'm proof that if you want to pursue something it may be that sincere interest and motivation matter more than other factors.

Debasis: Tell me about the most satisfying moment in your testing career.

Karen: This is probably the hardest question you have on your list for me. I've had many satisfying moments.

On a day to day situation, I look for a moment on every project when I know I've helped the team. I'm not always sure at the start of a project how I will provide the most value, it’s not always directly software testing. For example this past winter on a project, my most valuable contribution on one project was running a UAT session. For a variety of reasons, the UAT and the audience for the UAT had the project stakeholders anxious. I wrote materials and ran the kick-off session. It went very well. If I don't provide value on a project, I feel awful.

Outside of my more typical day to day life with project work, my most satisfying moments have been invitations to other countries. In 2008 being asked to present in Sweden and Spain was wonderful. The opportunity to present in two countries in the course of two weeks felt like an award of sorts. In 2009 I'll be traveling to Australia and New Zealand. I'm pretty thrilled about having that opportunity. I enjoy meeting other people and being exposed to different cultures.

Debasis: What do you think as the most essential skills that make a great tester?

Karen: I have to laugh, one skill – I’m not sure about that. I often see multiple factors come together in a blended way – you’ll notice in all my answers that I struggle to name one factor and instead see multiple factors.

Curiosity: If you're not engaged in what you're doing and you don't continue to ask questions, then I don't think software testing is the right match for a person. When I say asking questions - it doesn't always mean walking down the hall and asking someone else a question. It easily can mean you're asking questions in your mind and seeking answers for yourself.

Tenacity: Tenacity comes into play when you work with a team that might not be used to having a tester or doesn't want a tester around and the team maybe less inclined to help you. Stand your ground.

Caring: And finally, if you don't care about software testing and find your work interesting - then find something else to do. I feel like it’s a big world with so many possible options, find something you love to do. As recently as last week I had a day when I thought I can't believe I get paid to do this - I wish everyone could feel that way about their work at least some of the time.

Debasis: What single thing would you want to tell every newbie who is struggling in the early stage of building software testing career?

Karen: If you decide you want something don't give up. Figure out what you're strengths are and focus on those. If you're pretty good at coding, learn test automation. If you're pretty good with people, look to a test lead or management role. Not all testers are the same. Everyone has different strengths. Be an honest critic of what you're good at, know what you want to do and then steer your career in those directions.

There are some aspects of software testing, I think every tester has to handle - one aspect is change. Learn not to be intimidated to learn new things, technology moves at a rapid rate. Will you be able to move with it?

Another aspect is continually working around other people who are more technical than you might be - I think of several system architects that I've worked with who simply know so much more about whatever technology than I do. Learn not to be intimidated by people who know more than you may know.

And a third aspect is stress. Testing is at the end of the cycle and it is often inevitable that the time compression factor will add stress to your work. Learn to fight for more time and learn to cope when you might not get the time you feel you need. Being able to manage stress is essential.

Debasis: Is there anything else that you would like to say?

Karen: Thanks for asking me for this interview. I like when the software testing world feels like people I know and not an anonymous field. I think between blogs, forums and other outlets, many of us have gotten to know each other - at least a little bit. I like when the community feels connected – even when we disagree and argue about ideas and approaches.

I hope all of you enjoyed the interview as much as I did. Let me once again thank Karen for allowing me to take the interview and for sharing her experience, views and ideas with us. Feel free to comment and let me know if there is any particular Testing Expert whom you would like to see me interviewing. Check back soon for more interesting interviews.

Happy Testing…

Software Testing Diplomacy: A Tester’s Guide on How to Deal with Programmers!

2009-04-15T03:10:00.000-07:00

When I started my career as a software tester, it hardly took me any time to witness the ongoing rivalry between the programmers and the testers. When you are young in your software testing career and join a new work place, things are much like a prison. You notice people are divided into groups and you are pulled into a group even before you realize what is going on! This is probably so common in software development teams that people take it for granted! To make matters worse I have even seen some managers who encourage such practice of antagonism! Unfortunately, they believe that such enmity between the programmers and testers can help them squeeze out more value out of the team. But ironically, I am yet to experience a case where this was true!

After spending all these years in software testing field, I have heard lot of stories involving such friction between the testing and development team. I am not sure what started this enmity and how all this began. But under such circumstances of serious hostility it might often be very difficult to maintain a positive attitude and deliver your 100%. However, instead of blaming others for the situation let's search for solutions that might help during such situations:

Deliver the bitter medicine in a sugar-coated capsule:

Most of the times, as testers we are the bearer of bad news. We will have to tell the programmers that their baby (code) is ugly. And clearly, no mother will ever want to hear such a statement [irrespective of how ugly her baby really is]. Hence, I can understand how difficult it can be for the programmers to accept the fact that their code is defective and someone else has detected it. To deliver such a bad news, you will have to do so tactfully and professionally! Hence, testers may have to be diplomatic when confronting a programmer with a fundamental goof. Diplomacy, tact and a smile on the face - all can work to the tester’s advantage and are essential in preserving a harmonious relationship with the programmers.

Be unbiased:

While communicating, treat programmers and testers alike (e.g. refer them as "we" instead of "you". Make the programmers feel that they belong to us and together we are a team.). Restrain from any blame game. The programmers should not feel that they are being targeted for anything. I have seen that if the programmers are once convinced about the importance and the complexity of our job, they start respecting our craft. I often tell my programmers that because of them I earn my bread and butter and because of me their jobs are saved from getting sacked. It is essentially a symbiotic relationship between a tester and a programmer. And once the programmers realize this fact, gaining their cooperation becomes easy.

Maintain a sweet tongue:

Make sure that you don’t make the programmer embarrassed while reporting a defect in her code. They are also human beings who are prone to errors. It is quite natural for them to make mistakes. Do realize this and stop making your bug reports weapons of embarrassment for your programmer, every time you report a bug. To earn respect, start showing them respect. Above all else, watch your temper and leave aside your ego. As the old proverb warns us, "You catch more flies with honey than you do with vinegar." A little courtesy can go a long way towards building successful and long-lasting relationships. A balanced combination of humility and humor can be fruitful.

How do you deal with your programmers? How do you manage when things become hostile? What do you do when you are dealing with a programmer who considers you as her enemy? Share your stories and ideas via commenting.

Happy Testing…

Interviewing a Testing Expert - Jonathan Kohl from Kohl Concepts [Part-2]

2009-03-23T02:44:00.000-07:00

Friends, I am back with the final part of the interview with Jonathan Kohl, co founder of Kohl Concepts and a world-renowned expert in Software Testing. In case, you missed out the first part, I would strongly suggest you to read it first before proceeding to read this one. So here we go:

Debasis: Tell me of any situation when you had wished you were NOT a tester!

Jonathan: Testing can be political. The wishes of the technical staff and executives sometimes don't map to the reality of what they are delivering. That means you can be the least popular person on the team, and it can be discouraging in the moment.

When you challenge someone's personal faith or political views, they get upset. If you challenge their favorite technology or methodology, they behave similarly. Confronting that requires a lot of courage, integrity, ethics and a thick skin. They respect you for it in the long run, but it’s tough to take in the short run. I've learned over the years that if I am tempted not to say something, that means I should say it, but with tact, diplomacy and empathy. It's served my career well, but it's tough in the moment to challenge something that is distracting us from providing value. Anytime someone has said: “don’t raise that important issue, it will be a career-limiting move” I have listened to my conscience, spoken up, faced initial resistance, but over the long term, they were always career-catapulting moves.

Debasis: Has the profession (testing) ever affected your personal life? If yes, how?

Jonathan: I think about testing everything. Sometimes it's hard to take time off and not analyze all sorts of systems and look for weak points. Now that we can work from home, or work on side projects and things like that much more easily with the Internet, it's hard to turn work off and relax sometimes. The '55 Pontiac and car club are good outlets, and when I have the time for it, so is performing music.

Sometimes my friends don’t want a thorough analysis on their ideas, they just want me to empathize, so once in a while they ask me to turn off my tester brain.

Debasis: What do you think as the most essential skills that make a great tester?

Jonathan: Mostly, skills related to investigation and communication. You can develop those just about anywhere. Most testers have something in their background that they learned some basic investigative skills from, such as scientific experiments at school, or reading detective novels. Sometimes making that connection to your testing work can be a real eye opener. Good investigators see things that others miss, and are able to find problems and report them clearly. There are many different skills testers draw on to do that, and that depends on their background and training. It doesn’t hurt to learn some technical skills so you can help track down issues more quickly and communicate with programmers and other technical people in terms they understand.

Good investigators have solid general knowledge. They know what new technologies are up and coming, and they have a deeper knowledge of the systems they are working with. What are the inherent problems with the technology? They have a catalog of problem areas, and remember things from their own experience to draw on as they look for potential problem areas. They also have a deep knowledge of the users of the software. Who are our users? What problems are they trying to solve with our software? What are their goals with our software? They also have a good understanding of the team, and the equipment available for testing. They also have an understanding of a lot of different testing techniques, and how to identify risks or opportunities, and recommend techniques to look into those areas more. They have at least a basic technical understanding of the software and the environment in which it operates. All of that helps them strategize, plan and generate a lot of testing ideas that leads to important information about the product they are testing.

It's not a skill, but having ethics and integrity is also important as a tester. Stakeholders need to be able to trust us to tell them the truth. When we violate that trust (even by giving in to their pressure in the short term) we lose our effectiveness.

You also need an interest in discovering problems. Sometimes testing can be tedious, and if you are creative, you can turn any situation into something interesting for you. Without that interest in finding and reporting problems, it’s going to be difficult to enjoy testing and want to develop skills.

Debasis: Tell me about the most fascinating bug that you have encountered in your entire testing career.

Jonathan: Several intermittent bugs have been fascinating. One of the most difficult to track down was due to a social issue. Every release, one of our biggest clients would file a bug report. We would try to repeat it, and just couldn't get a repeatable case. Our software could dial home with a stack trace on error, so we asked the clients to send it in. The programmers looked at the code and tried to make it as robust as possible, then we'd release it, still get the same bug report from the client. The team sent me to spend the day with one of the client's users to get more information, and I was shocked at what I learned. It turned out that the software would crash on startup, but the user blamed herself for that error because they had learned a workaround (and hadn't told us about it) and she forgot to go through those workaround steps. After she went through the workaround, a few minutes later the software crashed and she showed me the familiar stack trace. I got her to give us the information from the first crash, which is really where the bug was located. The stack trace after that was a red herring because the system was in an unstable state, and could fail any number of ways. I learned to pay a lot more attention to the users and the social context they use our software in, and to incorporate that into my testing. That led me to look into usability testing and user scenarios instead of just looking at the requirements and functional aspects of the application.

Debasis: What single thing would you want to tell every newbie who is struggling in the early stage of building software testing career?

Jonathan: Concentrate on developing your skills as a software tester. There are a lot of distractions from skill development, whether it is to pass some multiple choice certification exam, learning about the latest fad software development methodology. We also talk a lot about analysis and planning more than test execution. (There is value to all of those activities, but not at the expense of test execution skill development.)

Certifications, tools, methodologies and practices change over time, but demand for good testing skills remains constant. For example, if you are testing on an Extreme Programming team, the novelty of the unique environment and context wears off, and you need to deliver skilled testing. Learning the methods, values, shared language and rituals only gets you so far. Your team is going to expect you to add value as a tester. It’s kind of like extreme ironing. Once they get over the novelty of you ironing in an unusual context, your customers will expect nicely pressed shirts. If you focus on improving your test execution and reporting skills, you will find more interesting information, and will become more valuable to the organizations you serve. Good testing and communication skills transcend methodologies and tools, and good testers can provide value anywhere if they have the skills.

Some things to consider: Develop your investigative and reporting skills. Be a software investigator, and practice your test idea generation. Use your background to leverage your investigative side. If you have a programming background, use that. If you don’t look at other areas you developed skills as an investigator. I’ve worked with wonderful testers with backgrounds from accounting, writing, studying history, programming, systems administration, investigative journalism and others. Most people have some sort of investigative experience in their background. Find yours, and apply it to software testing. Failing that, read detective stories and watch investigative programs on TV and try to apply some of the techniques to testing.

Here is how to practice being a software investigator: Look at simple systems around you, pinpoint their weaknesses and imagine how they might fail. Apply that to software testing. Bounce the ideas off of colleagues, and take their criticism of the ideas seriously. Be self-critical – are you finding important problems? Is your feedback helping the team prevent defects? Do the problems you log get fixed? If not, why? Also, work on your bug and status reports. Ask for help from programmers on how to write better reports, and ask them what kinds of bugs they appreciate you finding. Work with them to learn more diagnostic and investigative reporting skills.

I thank Jonathan for taking time to answer these questions and for sharing his thoughts and insights with my readers and me. It was great to know him in a better way via these questions. I hope that you (my readers) enjoyed this interview as much as I did. Let me know how you felt about the interview, so that I will plan more such interviews on Software Testing Zone in future.

Happy Testing…

Interviewing a Testing Expert - Jonathan Kohl from Kohl Concepts [Part-1]

2009-03-17T02:20:00.000-07:00

I am back with the second installment of the “Interviewing a Testing Expert” series. This time I am going to present an interview with Jonathan Kohl, co-founder of Kohl Concepts, who was kind enough to spare me some time for this interview. Based in Calgary, Alberta, Canada, Jonathan is a software testing consultant, author, and speaker in the software industry. Jonathan writes about and speaks on software testing. He draws on technical, philosophical and business concepts in his work. Jonathan's pragmatic approach focuses on getting down to the nuts and bolts of problem solving, encouraging organization-wide collaboration. Since this interview grew bit lengthier, for the sake of easy readability I have decided to present it in 2 parts. So, here is what Jonathan has to say:

Debasis: What led you to become a software tester? And what was the topmost reason that attracted you to the field of testing?

Jonathan: I started in testing when a small software company offered me an intern position in a QA department. They tended to start people with no professional software experience in QA. That way you could get an overall view of the organization, products, tools and methodologies and learn the basics. If you wanted to move on to programming, support, marketing, etc. then you had a good basic understanding from your QA experience. I enjoyed testing because I saw it as "philosophy of software". I had taken a lot of philosophy and logic courses as an undergrad, and I enjoyed the investigative, experimental approach of problem solving in testing, making a good case for bug fixes in bug reports, and having to make my test strategy, approach and decisions defensible to the rest of the team.

I later moved on to technical writing and then became a full time programmer, but I missed the variation of skills and activities in testing, so I went back into a testing role. My topmost reason for staying is the software investigation angle of my work. It’s challenging and fun to help track down and solve important problems that help teams succeed. I am still attracted to the variety of skills I use: investigative, analysis, observation, programming, writing, psychology… the list goes on. I can make testing into whatever I need to help discover and report important information that stakeholders need to make decisions.

Debasis: Did you try testing anything other than software before diving into software testing?

Jonathan: I grew up in a family where critical thinking and coming up for different explanations of why things are the way they are were rewarded. My father was a teacher, and is a brilliant critical thinker. He is able to quickly work through a massive amount of data, synthesize it, come to conclusions and make his thinking process defensible. We have debated many issues over the years, and it took me a long time to be able to spar with him. That background was really valuable as a tester, because he taught me to look for things other people might miss, would point out inconsistencies or flaws in my arguments, and he also taught me a lot about strategic thinking. From that, I went on to study a lot of logic and philosophy in university, and the critical thinking and argumentation analysis was vital to transitioning to thinking about software systems from a testing perspective.

Other than that, I used to take apart everything I owned and tried to put it back together. I needed to know why things worked the way they did so I could understand them. As a software tester, I decompose or reverse engineer products all the time so that I can find areas of potential weakness and I exploit those with tests.

Also, as an undergrad, I took part in experiment design (qualitative research), as well as a participant (helping with quantitative research.) In Biology, my experiments always went horribly awry and I frequently skewed the results in a sample set, much to the chagrin of the grad student who was overseeing us. They didn't like my results because they didn't map to their expectations (a problem with the hypothetical deductive model that scientific research hinges on) so they discouraged me from doing any more work for them. In testing, having experiments go horribly awry leads to fabulous information that stakeholders value, so I get paid to have failed experiments and my audience likes that information. I suppose my failed Biology experiments contributed to my career in testing.

Debasis: Tell me 5 unknown/least-known facts about you.

Jonathan: 1. I own, drive and work on a 1955 Pontiac custom. It is done in the style that car customizers and hot rodders used in the late '50s and early '60s. I belong to a car club with a local group of car and motorcycle builders who all drive and work on hot rods or customs. Some are original hot rodders from the '50s, right up to me, the youngest and least experienced.

2. I was involved heavily in music for most of my life, until quite recently. From the age of 4 when my parents put me on stage to sing a solo, to saxophone in junior high band, singing classical music in a high school chamber choir that traveled around to interesting places, playing guitar or bass and singing in alternative bands through university, to dabbling in my spare time after graduating. Music has always been a big part of my life and I have learned that a lot of that experience has shaped my approach to software projects.

3. I have an entrepreneurial streak. As a kid, I started money-making ventures from roadside lemon-aid stands to a tiny museum. None of them made much money, but they were a lot of fun. As I got older, I worked in sales part-time. I sold anything from shoes, to clothes, to house wares, and later, I even sold web-based software to help pay for university. I also helped start a software company in university, and now as an independent consultant I work with startup companies and advise founders and executives and help them with their business plans, strategic vision, product or service vision, and as they develop pitches to investors. It takes an enormous amount of effort to start something from nothing, and I have huge respect for software entrepreneurs. It irritates me when we techies make fun of management, or sale. They are hard jobs to do well and can come with enormous pressure and responsibility.

4. I'm a trained manager with experience from corporate training and undergrad work while in university. I started management training on the job as an assistant at the age of 18, during a recession a few years ago. Many employees reporting to me were former entrepreneurs or former senior managers from firms that had laid them off, and they were working part-time to help make ends meet. To lead people who were old enough to be my parents took a special combination of skill and diplomacy. I had to earn their respect by being willing to do any job I asked them to do, to work hard, and to not be a dictator. I learned something important from them - they were willing to work and do whatever it took to do right by their ethics and their families. They weren't distracted by titles or status; they just did whatever needed to be done. They were successful prior to the last big recession, survived it, and were even more successful afterwards. Those early lessons had a huge impact on my career.

5. I've been writing for most of my life, and I love it. Now I write on technical topics, but I plan on branching out into fiction eventually.

Debasis: What was the hardest challenge that you found in your career as a tester?

Jonathan: Overcoming narrow thinking about testing. Early in my career, I worked for small startups, and they demanded that all employees provide value. They didn't really care how we did that as testers, as long as we provided them with the information they needed. Some of them were using iterative, incremental lifecycles and practices back in the '90s, and we had to be creative and solve problems differently than traditional testing texts recommended. As I moved on, I found that some testing groups were less concerned about results than they were in some adhering to some process ideal. Back then, people were more concerned about heavyweight processes. More recently, being Agile-compliant is often more important than providing good results. (By the way, I’ve been involved on a lot of Agile projects over the past several years, some successful, others not.)

I don't understand why we have so many extreme views in software that have so much emotion and blind faith wrapped up in them. I value sustainable results, and so do most stakeholders I work with. On most real-world projects, there is a mix of practices: Agile and traditional, exploratory and scripted testing, manual and automation. It bothers me that we have to be one thing or another, or be at one extreme and denigrate other ideas, tools and practices. Why not broaden thinking and try for a mix of tools and practices in testing? Thankfully, there has been a softening of views and more people are open to trying whatever it is that can help them create value, rather than being pure adherents to some methodology or tool ideal.

That has been interesting and rewarding to see. Even five years ago exploratory testing was often misunderstood and discouraged. Now, people are interested in trying out practices to learn how to do it better and add it to their process and tool mix. The same thing goes for open source testing tools, and different software development methodologies. The fusion of ideas, and the mashups of tools, processes, practices and methodologies different teams use to create value over time is fun to watch.

Debasis: Tell me about the most satisfying moment in your testing career.

Jonathan: When I first realized my work mapped directly to our company being able to provide more value to our customers and end users. Now, I get satisfaction when I help track down high impact intermittent bugs that not only help the team create value for their users, but also it takes a weight of pressure off the team. As a trainer of software testers, I get enormous satisfaction when I see a tester learn and grow as I help them develop their skills, and learn how to learn the craft of software testing. It's fun to have a manager ask me what on earth I did to help this junior tester rocket to a bug finding and bug-preventing star in such a short time. Of course, I did very little but encourage an interested tester figure out how to tap into their own skill set and potential.

I hope all of you enjoyed the interview as much as I did. I will be posting the final part of this interview soon. So stay tuned.

Happy Testing...

Software Testing Interviews in India - Importance of definitions for a Tester!

2009-02-26T04:44:00.000-08:00

Last Sunday I was having a conversation with Ajit, a tester friend who works for a reputed Software MNC. He has been in testing field for more than 4 years now with experience in some cool domains. He is in his current organization since 2 years and now, much to his discomfort his company has started sacking employees citing the goddamn recession as an excuse. So obviously Ajit feels a sense of job insecurity despite his good performance in the organization all these years. People with such experience will tell you, it is never easy to concentrate on your job when you are insecure about it and always in fear of a pink slip landing on your desk bringing the news of misfortune. And it would not be long before you would find yourself searching for a new job (even though that would not mean any better sense of job security). May be, it is human nature to change nest when they feel a storm might be on it’s way. Obviously, it is no secret that Ajit has also started job hunting and has started appearing job interviews. If you have started to wonder if this post is going to talk about the current recession and it’s impact on people in IT industry, then I am sorry for disappointing you. I am no economics expert, and hence I don’t think I can take a look into the global slow down and suggest any solution.

The idea of this post came from a question that Ajit raised during our conversation. He has been attending some interviews recently and has spotted an interesting tendency among the interviewers, i.e. “they tend to ask lot of can-you-define-xyz-testing kind of questions”! Ajit was curious why they are so much interested in the definition of some testing buzzword, whereas they should be more interested in judging the level of competence of the tester being interviewed, by asking her some practical questions.

I can understand how stupid it can be to ask such “define this testing terminology” kind of questions. The stupidity would be more obvious if we take a look at the following reasons:

1. No testing terminology can be defined in a single unanimous way. Any particular term may mean differently to 2 different testers, testing organizations, schools of testing, testing gurus depending upon their own understanding of that particular testing terminology. Then how can the interviewer ask the candidate to define something and expects that their definitions match?

2. A single underlying testing principle might be called differently (different testing terminologies) by a different group of testers depending on their organizational practice, education, school of thought etc. Then how can the interviewer ask the candidate to define something and expects that their definitions match?

3. Lets say, I have memorized the definition of a testing buzzword and I am lucky! My definition (luckily/coincidentally/by chance) matches with the definition that my interviewer had memorized from his testing institute days. So suddenly, I look more competent as compared to other candidates whose definitions had not (due to bad luck) matched with the interviewer. But what does this prove? Does this prove that I have applied this understanding in practice? Does this prove that I can apply my understanding of that buzzword in real life testing? Does this prove that I am a better software tester than the earlier candidates who could not define it in a way that maps to the understanding of the interviewer? If not, then I wonder what makes the interviewer think that asking such definitions is a good way to judge how skilled the candidate being interviewed is!

I am not saying that knowing the basic fundamentals (read as theoretical testing) is a waste of time. I agree, learning the fundaments of testing is equally important to start a career in testing. But wait, doesn’t that hold good for any other professions too? Isn’t having theoretical knowledge necessary for any other profession? Then why is the fuss? The problem seems to arise when people on the interviewer's chair start imagining that whatever they know is the ultimate truth about testing and all the rest (read as candidates) must also know it. This problem becomes even worse when the interviewers start to think that testing is just all about textbook definitions and fundamentals (types of testing, levels of testing, testing life cycle, verification vs. validation, smoke vs. sanity testing, test automation, testing certifications, and so on).

Stories like Ajit’s often make me wonder if software-testing definitions are so important for becoming a skilled tester that they must be asked in interviews! Is it an absolute necessity to know how something is called (definition) to be able to actually do it? I think the answer lies with the Mother Nature. An average human child starts to smile, cry, sleep, play, crawl, stand and even walk much before it actually learns to SPEAK (define?)! If a child can do so many things even before knowing how they are called and described (defined), then why should it be so mandatory for a tester to define something in order to apply it in practical testing? I have come across many exceptionally good testers who are great at testing even though they often don’t realize that there must be a term to describe what/how they perform testing. At the same time I have come across many testers who are like the bookworms of a testing bible (they know every nook and corner of the textbooks and know each and every word in it), but when it comes to apply that knowledge into testing, they often fail miserably.

Then why do we give so much importance to testing definitions in interviews? Is it wise to ask these questions just because we were asked the same in our interviews? Is it a good way to judge whether a candidate for testing is skillful by asking some testing definitions [as if there is no better way to judge a tester’s competency]! Let me hear what you think.

Happy Testing…

How to test on a tight testing schedule?

2009-02-06T06:43:00.000-08:00

If you have spent some time in the field of testing, then you must have faced situations where you were asked by your test manager to test the application on the fly and deliver your report in XYZ days [replace XYZ with as few days as you can]! In case, you have not faced such a situation yet, then either you are unbelievably lucky or with all due respect, you have not worked on sufficient projects. But either way, it won’t be long before you encounter such a situation in your career as a software test engineer.

Contrary to popular belief/myth, proper skilled testing requires lot of planning, effort and work, and hence a substantial amount of time. But unfortunately, when projects get delayed, the time planned for testing invariably gets the hit while squeezing the schedule. What? The development team is running 2 months behind the schedule? No problem. Time for the magic trick! Squeeze the testing schedule by 2 months and presto! Congratulations, we have got ourselves back on the project schedule. Cool, isn't it? Well, probably not!

If you have started to presume that this post is going to criticize how project management stinks, then thankfully, you are mistaken. Because, as a tester I believe that a part of my job is to act like a “problem solver”. So instead of whining about how the rescheduled testing time frame keeps killing the poor tester, I would rather concentrate on finding a way to deal with such a situation. When you are hit by such a squeezed testing schedule, first thing that you could do to help yourself deal with it, is to “accept it”. This kind of things keep happening to everybody who works on a software project. Once we accept it as a part and parcel of our profession, dealing with it would suddenly start looking easier.

When facing a short time frame available for testing, you have to make the best use of the time and resources available. Starting testing with an assumption that “we can’t test everything, no matter what”, can really help. Even from an economic stand point it does not make any sense to spend lot of time and energy testing areas of the application where the chances of having bugs are low [this we can fairly tell based on our previous testing experience]. Also identifying areas where the impact would be negligible [based on expected user behavior] even if bugs were found is a good strategy while starting testing. As a rule of thumb determining what to test first and in which sequence, so that you spend the limited time testing areas that really matter, is an important decision that requires certain amount of analysis, intuition, and experience. Start doing a risk analysis to identify functions with the highest risk [thus most important and need highest attention] and functions that would be used most by the end user. Having a checklist to remind you of key areas that you would not want to miss certainly helps. Here is a checklist that I often use when I have much less time than I would have wanted to bargain for testing an application:

» Functionalities that are often used by the users. Start by asking yourself, “Which functionality is most visible to the user”.
» Functionalities those are most important to the project’s intended purpose.
» The most risky areas of the application with the largest safety impact. Areas, which if broken can bring down the entire application to it’s knees. [Talking to the developers for suggestions on the same is probably a good idea here].
» The areas of the application with the largest financial impact on the users (and hence on the project stakeholders).
» Newly added functionalities. Often they are the least tested ones and hence the dirtiest.
» Complex functionalities those are easy to be misunderstood (and hence misinterpreted). Look for parts of the code that are most complex, and thus most prone to errors.
» Functionalities that are based on parts of the requirements and design that are unclear or poorly thought out.
» Functionalities that are developed using challenging new technology, tools, architecture.
» Functionalities that are developed in rush or panic mode.
» Functionalities that demand a consistent level of performance.
» Functionalities that reflect complex business logic.
» Functionalities that require interfacing with external systems (e.g. third party shrinkwrapped software). These are often classic areas to look for integration bugs.
» Functionalities developed under extreme time pressure.
» Functionalities that had recent updates or bug fixes.
» Functionalities developed by many programmers at the same time.
» Functionalities those are most important to the project stakeholders.
» Identify related functionalities of similar/related previous projects that caused problems (in terms of user reported bugs). Correlate them to the current application and try to use it to your advantage.
» Identify related functionalities of similar/related previous projects that had large maintenance expenses. Correlate them to the current application and try to use it to your advantage.
» Identify functionalities, which if gone wrong could result in bad publicity.
» Identify functionalities, which could cause most customer support complaints.
» Devise tests that could cover multiple functionalities/features at the same time.
» Devise tests that could cover high-risk-coverage at the minimum time.

This is clearly not the exclusive list of guidelines/checklist to test under a tight testing schedule. But still, it covers quite a lot of important areas that usually needs attention. Being a context driven tester, I am well aware of the fact that using this checklist may or may not help a tester who is trying to test an application on a jam-packed schedule. However, it has served me quite well whenever I was in need of it.

What do you do when faced with such a situation of a tight testing schedule? How do you react when you suddenly find yourself being stripped off with some valuable testing time at the very last minute of a project deadline? How do you readjust your testing strategy to cope with it? Do you have any such checklist that you follow? I would be delighted to hear your ideas.

Happy Testing…

New Year Gift for Software Testing Zone Readers/Subscribers!

2009-01-22T02:35:00.000-08:00

I do realize that I am yet to write something new on STZ (Software Testing Zone) in the Year 2009. Just when I was trying to come up with something new and exciting for my readers and my loyal subscribers, Flavius Saracut, the Marketing Manager of backup software Titan Backup, contacted me with a fascinating offer for Software Testing Zone readers. He offered the readers and subscribers with the full version of Titan Backup 1.5 for FREE! I felt it would be a great idea to present my readers with this small yet powerful utility software as a symbol of my gratefulness. And you know what? You don’t even have to be a subscriber of this blog [though I would appreciate it so much] to grab this gift!

Here is my New Year Gift to all my dear Readers and Subscribers:
Titan Backup is a Complete Data Backup Solution. Its a paid software and costs $39.95 for a single license. But now you can get a FULL license of Titan Backup 1.5 version absolutely FREE here on Software Testing Zone. Yes, you heard it right. Titan Backup is giving a FULL license to all Software Testing Zone readers/subscribers absolutely FREE! So no need of worrying about the genuineness, this gift is fully legal. Consider this as a small gift from me to you for reading my write-ups all these years. Hope you would find this useful.

Step 1: You can download Titan Backup 1.5 from here: Download Titan Backup 1.5. [You might have to wait a while after clicking this link before you finally get the binary installer file to save]

Step 2: After installing, launch Titan Backup 1.5. Now click Enter Key. Copy and paste this following Serial key [meant exclusively for Software Testing Zone readers] to activate Titan Backup 1.5:

000020-ACM8KK-1YXHMT-JZT4D3-YHVFMK-XJTU01-4UXJUH-NY492A

You should see the following confirmation: “Key is valid and has been stored”. Press OK.

Step 3: Enter your registration details and press Register & Activate. Now Titan Backup will activate the registration. You should see the following confirmation: “Program successfully activated”.

Step 4 (Optional): Should you at any point of time, decide to upgrade to Titan Backup 2.3, please visit this Special Offer Page. A special 50% discount is offered to Software Testing Zone readers/subscribers, in case you want to upgrade to the latest version. Introduce the following Coupon Code in the Special Offer Page: NEOB-BBN8

Hope you like this small gift from me. In case you liked this free gift, feel free to continue reading me (Software Testing Zone) and consider subscribing to the blog. :) This is just the beginning of 2009 at Software Testing Zone. Stay tuned for lot more interesting articles and interviews on software testing in near future.

Happy Testing…

Best of STZ (Software Testing Zone) in year 2008!

2008-12-29T01:08:00.000-08:00

As the end of year 2008 draws near, I have decided to bid it goodbye with a compilation of 10 top posts on STZ (Software Testing Zone) in last 12 months (according to me and reader's feedback and comment). If you are new at STZ, then you will find this post a good place to quickly scan this blog. If you are a regular reader, then probably this post may help you to read some posts that you may have missed earlier. So here are the top 10 best posts on STZ in the year 2008:

1. Software Testing and Cricket - an Analogy! Howzat! - This post tries to analyze and come up with 10 similarities between Software Testing and the sports Cricket! If you are an avid cricket fan then this is a must read for you. If you are not, then also I guess it would be equally fascinating for any tester. I am aware of at least 2 of my blog readers who had chosen this post as their presentation topic and were highly appreciated by their audience. Whether you are a Cricket fan or not, I believe this is going to be an interesting read for you.

2. Confusing Terminologies in Software Testing - A Case Study! - This post tries to explore the existence of confusing terminologies in the field of testing and tries to find out a way to deal with it. I find it so amazing, the way testers use different terms to mean the same underlying testing principle and the way testers use different ways to define a single testing terminology. What might be the solution to handle such confusion? Read in this case study.

3. Programming Skills and Testers! - Knowledge of programming and effectiveness of a tester - is there a connection? Can a good programmer make a good tester? Can a bad programmer make a good tester? Can a non-programmer make a good tester? Can you think of contexts when knowledge of programming can enhance your testing? Can you think of contexts when knowledge of programming can hamper your testing? Can you think of contexts when ignorance of programming can enhance your testing? Can you think of contexts when ignorance of programming can hamper your testing? Find out the answers in this blog post.

4. Repeatability of Tests - A necessary Evil! - “If you cannot repeat what you are testing then you are NOT testing properly! Think of regression testing. What will happen if your regression tests are exploratory by nature and are non-repeatable”? Do you also believe the same? Then may be its time to rethink! Don't miss to read Michael Bolton's insightful opinion in the comment section of this post.

5. Priority Report! - Still confused with bug severity and bug priority? May be the short testing story in this post would help you bust the myth. Check it out for some realistic examples describing bug severity and priority in different combination (high severity and high priority bug, low severity and high priority bug, high severity and low priority bug, low severity and low priority bug).

6. Can your kid beat you in testing? - Check out this post to find out 10 reasons why I think kids could be good testers and 10 reasons why I think kids could be terrible at testing software!

7. Does SW Testing add value to the Project? - Does your programmer think that testing does not add any value to the project? Does he think that the testers are a bunch of crazy sadistic people who find it fun to find defects in his code? Does your programmer think that testing is a dumb worthless job? Then may be it would be a good idea to point him to this blog post!

8. Can test automation tools replace human testers? - Are you a tester who does not use much of test automation? Are you afraid that soon the testing tools are going to replace human testers? Do you fear that the automation robots are going to take over the world of human testers? Then, take a look at this post for a sigh of relief.

9. Interviewing a Testing Expert - Alan Page from Microsoft! - Alan Page is one of Microsoft's first Test Architects and currently chairs the test group in Microsoft's Engineering Excellence team. Among other things, this means that Alan teaches testers how to be better testers, and he has designed and teaches courses for Microsoft's most senior testers and test managers. Beyond teaching, Alan creates and updates technical courses, working with test teams across Microsoft to help them reach their goals. He has been a tester for over 15 years, and has been at Microsoft since 1995. In this interview Alan shares his professional journey to become Microsoft's one of the top Test Architects.

10. Top 10 reasons why there are Bugs in Software! - Ever wondered, why there are bugs in the software that we test? Ever wondered, how errors creep into the code? Then this is a post you must not miss.

Hope you enjoy reading the above posts. Wish you all a very Happy and Prosperous NEW Year.

Happy Testing...

Criticizing the Software that we Test!

2008-12-23T05:23:00.000-08:00

A couple of days back I was contacted by a reader (Sombit Chakraborty) with the following question:

Hello Debasis, I am a regular viewer of your blog. In (and through) your blog I have got (and still gaining) immense knowledge about the field Software Testing. Your work is highly appreciable. Coming to the point I want to ask you something. Do you think a Tester needs to have a critical approach to the work he/she does? i.e. critically judging the task/build at hand. By nature I am a self-proclaimed critic (few others agree). And my job as a tester is only making it more evident. Need your valuable inputs.

I felt that this question was worth spending some time (brainstorming). Some readers have asked me this question few times before. So I decided to take Sombit’s question and try coming up with a blog post as my answer.

Being critical while testing. Is it an absolute necessity?

Among others, verification and validation are often considered as some of the responsibilities of a tester. While validating a functionality/feature/program, a tester is left with at least a couple of approaches to start with.

1. To validate whether the program functions as intended.
2. To validate whether the program does not function as intended.

Going by the first approach is important to see if the application performs as it was designed to. But testing whether the application fails to function as per the intended design is equally important too. And usually it is the second approach that digs out more unexpected bugs out of an application. Often people in testing refer the first approach as “positive testing” and the second as “negative testing”. [However, the notion of positive and negative testing is debatable. And I would like to leave that topic for another blog post]

Experts in testing say that our craft is similar to that of a criminal detective. We as testers, do technical investigations to reveal important project related information and deliver the same to the stakeholders of the project in order to help them take any critical decision about the project. A good criminal investigator believes in the saying: "The suspect is guilty until proven innocent". And I believe a good tester also should believe the same. Unless and until we test and verify that a particular application works as it is supposed to be working, we must not believe it’s innocence (that it should be working without any ambiguity). Being skeptical can help us in nailing down bugs that could not have been caught if we had started with an over-optimistic mindset! Once we start with a mindset that the AUT (Application Under Test) works as intended, we risk falling into a trap that can create illusion (inattentional blindness) in our mind. Such illusions may distract us when we come across real bugs in the software while testing. Our earlier assumption (that it must be working fine) may restrict us from identifying the bugs while they happen in front of our eyes. So YES, I do feel that we must be critical in our approach to testing unless of course the context demands us to act otherwise. A tester should be critical while testing the software (or anything else). Being critical helps us with a greater sense of cautiousness and attention. And as we all know, an attentive mind is more capable of identifying a defect than a mind that has already presumed that there are no possible defects here! How many times have you heard an alarm going off in your brain (sounding "he must be kidding!"), every time a programmer had told you that "I have confidence in my code that it wouldn't break!"? The more skeptical you are while testing, the more you must have heard such alarm!

Can there be contexts where a tester must leave aside his skepticism while testing?

These are few contexts that come into my mind where a tester should focus more on the positive side of the things (how the software works) rather than spending energy on criticizing it!

» During the last stage of a scheduled project deadline when the testing mission is to do a quick sanity testing of all major and commonly used functionalities and verify if they are functioning as intended.

» During the initial phase of integration of different modules where the main goal is to test and see whether those independent modules couple with each other seamlessly and still continue to work as a common integrated program.

» Can you think of more such scenarios where a tester must leave aside the critic inside him and focus just on the positive sides of testing? Then feel free to mention them as comment below.

Few points to remember while dealing with the critic inside us:

Being a skeptical tester that I am, I have already faced many awkward situations where the critic inside me had managed to overpower me. However, I have learned that sometimes it is good to identify contexts where we must control the critic inside us. Here are few pointers:

1. Once Mahatma Gandhi had said: "hate the disease, not the patient". Likewise, criticize the defect in the software, not the programmer. This would save you lot of friction with the development team (which is an unfortunate yet common scenario in many organizations; a bitter relationship between the testers and the programmers).

2. Learn and practice how to control your critic mind while dealing with human beings. Human beings are much more sensitive than a mere software. Moreover, a human being possesses emotions, which the software usually lacks! So it is important to control one's critic part while dealing with humans. Try to practice - "no testing at home". [I owe this wisdom to James Bach (via Ben Simo)]

3. If you find the above too difficult to practice and "if you fail to resist the urge to test and no testing at home is almost impossible, then at least there must be no bug reporting at home". [I owe this wisdom to Ben Simo]

4. Try to direct the criticism towards yourself once in a while. Practicing self-criticism is a very powerful skill of a good tester. Self-criticism helps us to evaluate ourselves and identify the areas of our weaknesses before someone else points it out. As Jerry Weinberg once said: "if you can't think of at least 3 reasons how and why your proposed solution for some problem can’t fail then you have not understood the problem well enough!" [While I am sure that Jerry is the original author of this quote, someone please point me to the actual source so that I can link there. I could not find out the online source of this statement]

Coming back to the need of being critical while evaluating the software that we test, what do you think about it? Do you think that a tester must remain critical while testing? Or do you feel it is absolutely unnecessary and an extra burden on the shoulders of the tester? Share your views.

Merry Christmas and Wish you all a very Prosperous and Happy New Year. Happy Testing...

Top 10 reasons why there are Bugs/Defects in Software!

2008-12-15T01:05:00.000-08:00

Here is a question that has made many testers wonder on several occasions - “Why are there bugs/defects in the software? Why/How do errors creep into the software and make it buggy?” I realize the paradoxical nature of this question. i.e. If there won’t be any errors in software then there won’t be any need of testing. But again, as long as software development exists as a field of business, there would be errors in them. To me, if something is *bug free* it translates as: it is either too trivial or non-existent! The increasing complexity of software would make sure that there are defects in them. But the question remains: what are those factors that cause defects in software? Well, here are 10 reasons (not in any particular order) why I think bugs creep into the software that we develop/test:

1. Human factor: It is because human beings develop software. It is often said that “to err is human, to forgive divine”. Human beings are not perfect. They are prone to make mistakes. As human beings develop software, it would be foolish to expect the software to be perfect and without any defects in it! Hence there are errors in software. Ironically, we are yet to discover any other non-human agent who could develop software any better than human beings. So we continue to rely on the human intelligence to develop software thus allowing chances of errors in it.

2. Communication failure: Another common reason for software defects can be miscommunication, lack of communication or erroneous communication during software development! The communication failure can happen at different levels (requirement gathering stage, requirement interpretation/documentation stage, requirement-to-implementation translation stage etc.). Imagine a case where the requirements are vague or incomplete. This could lead to a situation where the programmers would have to deal with problems that are not clearly understood, thus leading to errors. Another scenario of problem with communication may arise when a programmer tries to modify code developed by another programmer.

3. Unrealistic development timeframe: Let’s face it. More often than not software are developed under crazy release schedules, with limited/insufficient resources and with unrealistic project deadlines. So it is probable that compromises are made in requirement/design to meet delivery schedules. Sometimes the programmers are not given enough time to design, develop or test their code before handing it over to the testing team. Late design changes can require last minute code changes, which are likely to introduce errors.

4. Poor design logic: In this era of complex software systems development, sometimes the software is so complicated that it requires some level of R&D and brainstorming to reach a reliable solution. Lack of patience and an urge to complete it as quickly as possible may lead to errors. Misapplication of technology (components, products, techniques), desire/temptation to use the easiest way to implement solution, lack of proper understanding of the technical feasibility before designing the architecture all can invite errors. Unfortunately, it is not that the people are not smart; it is just that they often don't-have-time/are-not-allowed to think!

5. Poor coding practices: Sometimes errors are slipped into the code due to simply bad coding. Bad coding practices such as inefficient or missing error/exception handling, lack of proper validations (datatypes, field ranges, boundary conditions, memory overflows etc.) may lead to introduction of errors in the code. In addition to this some programmers might be working with poor tools, faulty compilers, debuggers, profilers, validators etc. making it almost inevitable to invite errors and making it too difficult to debug them!

6. Lack of version control: If as a tester you keep encountering lots of occasion of regression bugs that keep showing up at regular intervals, then it is about time to check the version control system (if at all any). Concurrent version systems help in keeping track of all work and all changes in a set of code base. Complete lack of a version control system to safeguard the frequently changing code base is a sure fire way to get lots of regression errors. Even if a version control system (e.g. Visual SourceSafe) is in place, errors might still slip into the final builds if the programmers fail to make sure that the most recent version of each module are linked when a new version is being built to be tested.

7. Buggy third-party tools: Quite often during software development we require many third-party tools, which in turn are software and may contain some bugs in them. These tools could be tools that aid in the programming (e.g. class libraries, shared DLLs, compilers, HTML editors, debuggers etc.) or some third-party shrink-wrapped plug-ins/add-ons used to save time (like a shopping cart plug-in, a map navigation API, a third party client for 24X7 tech support etc.). A bug in such tools may in turn cause bugs in the software that is being developed.

8. Lack of skilled testing: No tester would want to accept it but let’s face it; poor testing do take place across organizations. There can be shortcomings in the testing process that are followed. Lack of seriousness for testing, scarcity of skilled testing, testing activity conducted without much importance given to it etc. continues to remain major threats to the craft of software testing. Give your team some time to introspect and I won’t be too surprised if you find it in your own testing team! While you might argue that poor testing do not *introduce errors* in software, actually they do! Poor testing do leave the software in a buggy state. Moreover, in this era of agile software development poor unit tests (e.g. in TDD) may result in poor coding and hence escalate the risk of errors.

9. Last minute changes: Changes that are made to requirement, infrastructure, tools, platform can be dangerous, especially if are being made at the 11th hours of a project release. Actions like database migration, making your software compatible across a variety of OS/browsers can be complex things and if done in a hurry due to a last minute change in the requirement may cause errors in the application. These kind of late changes may result in last minute code changes, which are likely to introduce errors.

Considering that this post has been talking about possible cause of errors, defects and bugs in software, did you notice any error in this post! Did you notice that I have listed out only 9 reasons as against the promised 10 in the blog post title? Well, it is a deliberate error. Why don't you come up with the 10th (may be 11th, 12th... as well) reason why there are defects in software? Feel free to let me (and other readers) hear your reason(s) by commenting below.

Happy Testing…

Interviewing a Testing Expert - Alan Page from Microsoft!

2008-12-02T00:25:00.000-08:00

As per my long time plan finally I am starting a series of "Interviewing a Testing Expert" posts on my blog. In this series I would be interviewing some renowned software testers from around the globe. In this inauguratory post of this series I am going to present an interview with Alan Page from Microsoft, who was kind enough to honor me with this opportunity to interview him.

Alan Page is one of Microsoft's first Test Architects and currently chairs the test group in Microsoft's Engineering Excellence team. Among other things, Alan teaches testers how to be better testers, and he has designed and teaches courses for Microsoft's most senior testers and test managers. Beyond teaching, Alan creates and updates technical courses, working with test teams across Microsoft to help them reach their goals, and writes on his blog and elsewhere. He has been a tester for over 15 years, and has been at Microsoft since 1995. Here is what Alan has to say:

Debasis: What led you to become a software tester? What was the top reason that attracted you to the field of testing?

Alan: About 15 years ago, I got a job at a company that made music software. I was hired for tech support, but on my first day at work I found out that I was also going to be a tester (as well as the network administrator). I learned as I went for a while, but don't think I was really figured out what a tester did until a few years later.

I love problem solving – I like taking big open-ended questions (like "does this software work") and figuring out how to break it down and answer it. I've also discovered that with testing, there is so much more to learn – it's a relatively new job in the world of software, and I think there's a long way to go to get it where it needs to be.

Debasis: Did you try testing anything other than software before diving into software testing?

Alan: In a way, I test everything. It seems that I'm always challenging and questioning. My first job out of university was a school teacher. I taught band (mostly beginning band and jazz band) for 4 years. Music and teaching are both loves of mine (partially why I like teaching so much in my current job at Microsoft). After teaching, I went to graduate school to get a Masters degree in music composition. I used computers extensively both for writing my thesis (a Symphony), and for supporting papers. This was 1992, so I had to learn a lot about computers just to keep things working. When I graduated, I wasn't sure what I wanted to do. For a while, I taught private music lessons and worked as a bike messenger. I was thinking about getting back into teaching when I came across a tech support job at a company that made music software. I found out on my first day that I was also a software tester (and network administrator). I thought that would be something fun to do for a while. 15 years later, I'm still doing it and enjoying it more every day.

Debasis: Tell me 5 unknown/least-known facts about you.

Alan: I already told you a few of them above – that I have a Masters degree in music composition, and that my job previous to testing was as a bike messenger, and that before that I taught school. Here are some more:

1. I'm a bit of an oenophile. I have a modest collection of about 175 bottles, but I'm getting some more storage so I can expand a bit.

2. My favorite musician is singer-songwriter John Wesley Harding. Bonus fact: Wes played a set at my 40th birthday party!

3. I'm one of the rare Microsofties who grew up in the shadow of Redmond. My family moved to the Seattle area when I was 3, and other than college and 6 cold months in Wisconsin, I've lived in the Seattle area my entire life.

4. I speak a bit of Japanese and French. Not coincidentally, those are two countries I'd love to live in someday.

5. My first name is Donald. I've gone by my middle name for my whole life.

Debasis: What was the hardest challenge you found getting going as a tester?

Alan: Funny as it sounds, simply knowing what to do was probably the hardest thing. Like many testers, my first exposure to testing was having a piece of software handed to me and being told, "test this". Over time, we testers build up our toolbox of techniques, but a lot of us have to start from scratch. I think that even today that we could be doing a lot more to help new testers understand their role and common techniques, approaches, or patterns that can be used in different situations.

Debasis: Tell me about the most satisfying moment in your testing career.

Alan: Depends on the definition of "satisfied." One story that comes to mind is a situation many years ago, I was working with an exec who insisted that testing "owned" quality. I explained, as you'd expect, that test can measure quality, but everyone had to have a stake in quality in order to get a quality product. He kind of blew me off and said, "fine – but when push comes to shove, test is on the line for quality". I know how to choose my battles, so I continued to go about my work. The satisfying moment happened several months year later. This exec sent mail to all of his top level managers + me saying that he had a revelation and finally got it - that development, testers and program management – from management to front line workers all owned quality. It certainly didn't solve all of our problems, but it was nice to see the mindset change.

Debasis: Tell me of any situation when you had wished you were NOT a tester!

Alan: I can't think of such a time!

Debasis: Has the profession (testing) ever affected your personal life?

Alan: Well, my wife was a tester on the Windows 98 team at Microsoft. We originally met in 1997 or so, and she later went on to work for a vendor that made Japanese computer software. We met again almost 7 years ago (she was onsite at Microsoft), and were married 6 years ago. She's still a good tester, but stays at home with our kids these days.

Debasis: What do you think are the most essential skills that make a great tester?

Alan: Problem solving skills and critical thinking skills are huge. You need to be able to work with ambiguity and break down a big problem into solvable parts. You need to have a passion for learning and be able to understand what you're testing from top to bottom when necessary. You need the ability to see what you are testing from the 10,000 foot view as well as the micro level depending on the situation – then use the best tool/approach for the given situation.

Although it's not a skill, you should never get to a stage where you feel like you have everything figured out. As soon as you let yourself become stagnant, you're done. If you're not getting better – and actively working at getting better at testing and advancing the science and craft of testing, your value diminishes quickly.

Debasis: Tell me about the most fascinating bug that you have encountered in your entire testing career.

Alan: They're all fascinating in their own way. When I was on the Windows 95 team, one of my responsibilities was testing networking functionality on Japanese systems. On Win9x (and 16 bit versions of DOS and Windows), Japanese Kanji characters were two bytes long. The first byte was in a specified range – indicating to any program (correctly) parsing the text that the next character was the second half of a double-byte character, and that the two characters should be considered to be one character. As you can imagine, there were tons of problems found testing these characters - everything from general failures to cursors that could appear in the middle of a character (both of these scenarios being quite common). Also interesting was that the "trail byte" – the second half of the character could be a character that normally couldn't be in a file name. One such character that was a valid trail byte was the 0x5C character – commonly referred to as the backslash ('\'). As you can imagine, characters with this trail byte were prime test collateral for network testing. Among the bugs I discovered was one, where given the right circumstances, copying characters with this character as the last character in the filename to a Japanese Windows NT system caused the NT machine to crash. I remember reproducing it 4 or 5 times in a row to make sure it was really happening. This was cool to me at the time for two reasons. One was that crashing an NT machine was supposed to be hard (I suppose it was). It was also fun to find a bug in a shipping product. It was fixed shortly after, but I don't think a customer ever hit this particular issue.

Debasis: How do you see software testing as a career, lets say after a decade? What would be the biggest challenges for the field and what would be the biggest advancements?

Alan: Software is going to get more complex, while at the same time, people are going to expect more from it. If you think back 10 years, Windows 98 was shipping and my parents were just getting around to buying their first computer. 10 years from now, software will be accomplishing things we only dream of, and it will be running, making millions of decisions for us 24 hours a day. Software will be part of everything from traffic lights to home electronics and control systems, cars grocery carts, books, phones…you name it, and all of these devices – running software that allows these systems to interact in a safe and predictable manner. Furthermore, as software moves off of the desktop and into devices that we interact with, our expectations will go up. It's sad how we, as computer users, allow software to fail by creating workarounds or rebooting. We need to get software to a point where we just expect it to work and it does.

The big challenge is coming up with a way to make sure we design quality into the products in the first place and where bugs are the exception. Today, "finding bugs" is too easy. In the future, I want to get to a stage where testers finding bugs is a rare event that causes extreme concern rather than being an "everyday occurrence".

Debasis: What single thing would you tell every newbie who is struggling in the early stage of building software testing career?

Alan: You can't ignore the fundamentals, but you need to find a way to balance theoretical knowledge of testing with a practical, effective approach. I think many testers bury themselves in textbooks or courses on testing and attempt to apply what they've learned in possibly inappropriate ways. Of course, many others discard the fundamentals completely and rely primarily on guesswork to learn the craft. Success in testing, as in many careers, depends on balance. Learn the fundamentals, and learn where they do and don't work. Experiment and learn. Challenge yourself and show success.

Debasis: Is there anything else you would like to say?

Alan: Thanks for giving me the opportunity to answer these questions. Feel free to throw out answers that you may not like :}

Honestly it was great to know Alan better via this interview. It made me realize that I had not known him enough even after reading him for so many years until this interview [and this explains why I have NOT edited a single word from his answers]. Once again, thanks a lot Alan for taking your time and honoring my readers and me with this interview.

Happy Testing...

Problem with Blogger’s Embedded Comment Form! A Regression Bug?

2008-11-04T03:05:00.000-08:00

If you have noticed recently, for past few weeks I am using Blogger’s Embedded Comment form in place of the pop-up version. This is done via combination of Blogger Draft's new experimental feature and a simple hack to embed the form beneath every post. This new embedded comment form had been working nicely till now and as I checked an hour back, to my horror I noticed that the "Comment as:" drop down is empty and does not show any options to choose from. At first look I thought it was due to some accidental change in my own template during recent code refactoring. But when I checked around the blogosphere I came to notice that other blogs using this feature are also showing this problem. Moreover even the Blogger in Draft itself shows the same problem!

Though it comes as a relief to know that this bug is not something in my blog alone, still it is a terrifying scenario for any blogger like me who likes to see readers leaving comment on their blog post. Hope the Blogger team has already spotted this minor looking yet critical (from a functionality stand point) bug and must be working on it to deliver a quick fix.
Here is a screenshot of the bug below:

I have tried it in different browsers like IE 6,7, Mozilla Firefox 3 and Google Chrome. And unfortunately the bug shows up irrespective of your browser and/or screen resolution. Moreover, in IE I am getting this script error that is pointing to a possible problem at Blogger's side:

I am waiting for a quick fix from the Blogger team. In case they fail to deliver a quick enough fix, I would probably revert back to the earlier less user-friendly but at least functional comment form for now. In the mean time if you wish to leave me a comment, feel free to use my Contact Me form.

Update1! Blogger has not yet fixed this bug. Looks like Blogger is having problem in replacing the Blog ID and the Post ID while retrieving data from it's server via this (http://www.blogger.com/comment-iframe.g?blogID=xyz&postID=abc) URL. Usually Blogger replaces the "xyz" with the publisher's Blog ID and the "abc" with the Post ID. In fact for some blogs, it is actually replacing the data correctly and importing the javascript file (www.blogger.com/jsbin/1858126393-cmt.js) successfully. But for others (like mine) it is failing to do so resulting in the empty "Comment as:" drop down box. [NOTE: Thanks to 'Kimberly Salem' for pointing this to me.]

However, as a temporary workaround I have reverted back to the earlier pop-up version of Comment form so that you (the readers) will be able to leave comments. I am tracking this Blogger bug and I am already with contact with the Blogger team. Hopefully I would be among the firsts who would be intimated in case there is a fix. It would be my pleasure to update you all when ever there is a fix. Stay tuned for more updates...

Update2! Looks like this issue has been mysteriously fixed now. I have reverted back to embedded comment form in my blog and it is working. I have seen it working in other blogs too. Though Blogger team is yet to announce this fix officially, it has started working again. If you had removed embedded comment form in your blog then revert back to it now and see if it is working!

Happy Testing…

Can test automation tools replace the human testers?

2008-10-22T05:00:00.000-07:00

Vincent: “I keep hearing a lot about test automation tools. In one of those SQA Forums I heard of companies firing their entire manual testing team citing the reason as a shift to automation. That makes me uncomfortable as I am still working mostly in manual areas of testing. Are manual testers going to be obsolete in near future due to test automation?”

Jolly: “I have been working as a test engineer for past 2 years and now I want to move into test automation to avoid being singled out due to lack of tool knowlege. I find lot of tools in the market like QTP, Rational Robot, Silk Test, LoadRunner, JMeter etc. I am confused where to start. Can you help?”

As you must have already figured out, these are couple of FAQs from my mailbox. But before I can start writing anything on this delicate topic, let me start with asking myself few related questions aloud:

1. Can automation tools ever *replace* human testers?
2. Can testing tools catch defects/bugs *automatically*?
3. Can tools ever develop enough (artificial) intelligence to think like a human tester?
4. Talking about scripted regression test tools (like QTP, PTQ, WinRunner, LoseRunner, LoadRunner, UnloadRunner, JMeter, KMeter, Rational Robot, Irrational Robot blah blah blah), which operate at the GUI (not unit) level. Is their valuation/pricing/license cost fair enough to provide you a healthy ROI (Return On Investment)?
5. Is it a question of Manual Testing Vs. Automated, or something much more complex and intricate?

Before we can come to a conclusion and start answering these questions let me come up with some facts about the contenders here: automation tools and human testers. I am a human being not a robot. So I will prefer starting with the human testers. Here we go.

What is manual testing? I think the term “manual” means, “involving or using human effort, skill, power, energy, etc.” Going by this understanding, any testing that requires the human intervention can be considered as manual testing. Let us see the areas where human testers can fair better than the automation tools:

Manual Testing Pros:
1. A human can think. She has the ability to think on her own and make decisions. Though decision-making is possible in case of computers (automation tools), they are yet to reach the level of (artificial) intelligence to beat a human being when it comes to decision-making.

2. A human tester can explore new applications using her intuition and self-learning abilities. Even if she has no experience of testing anything similar before, if she is skillful she can be able to quickly learn, explore and test the application. An automation tool might not!

3. A human tester can think out side of the pre decided test boundaries. In my knowledge, more defects are unearthed using out of the box exploratory testing approach than running automation scripts. Automation tests can only find you bugs for which they were written. They can’t find you any NEW bug. If you want to find new bugs, then you need to either find those yourself manually (that’s right, by a human tester), or write NEW automation scripts to find them.

4. A human tester needs no less spoon-feeding as compared to a tool. Computers are dumb, and so are automation tools. You change a variable/object name in a line of code of your AUT (Application Under Test) and the tool would struggle to figure out where it has gone! More over tools need step-by-step instructions to perform a specific testing task. They (tools) are poor at understanding natural human language. Imagine asking a human tester to test a “User Help Manual”. What if we are to automate this test? Can a tool test a human readable help manual efficiently? In case, you happen to know of a way to automate this process, kindly let me know. I am in search of such a tool.

5. Human testers rock when it comes to Usability testing where automation tools fail miserably. Tools are poor in judging what is more usable from a human point of view like screen alignments, appearance of windows, smoothness of object alignments, color combinations, ease of usability, an entertaining user experience etc. After all we develop software to be used by human beings, not computers! So who would be a better candidate to judge the usability of such an application? You already know the answer.

6. Human beings are good examples of being adaptable. We could adapt to the changing environment and survived when big creatures like Wooly Mammoth and the Dinosaurs perished and became extinct due to their lack of adaptability. Human testers can adapt and learn from past experience. But (automation) tool may fail to learn on it’s own. The entire suite of test automation scripts may soon start failing if a simple thing like an object name has been changed/renamed in recent code refactoring. A tool cannot remember and learn from past experiences to adjust it accordingly. Human testers can.

7. Considering the huge cost of License fees of the so-called GUI automation tools, human testers are still cheaper. More often than not you might run into a project where the ROI (Return On Investment) for such a tool would be in negative as compared to when the tests were done manually by human testers.

I think that is enough of pro-manual and anti-automation rant. Before you (wrongly) classify me as an ardent opposer of test automation, let me clarify that I am in fact a great supporter of test automation, if done rightly! If you have read my earlier post, then you might already know that I love to test in agile environment like TDD and as you know no agile development can be imagined without test automation. What I oppose is the idea of automating tests at the GUI level using commercial tools that eat up a large chunk of your testing budget. IMHO, this is just waste of money on something that could have been avoided if proper testing (including automation) was done at the unit level, in the first place. If you must automate your tests at the GUI level, then find out a way to develop and use your own test driver. This way you can save on a lot of testing budget, which could be spent on other areas of testing. e.g In our organization we use our own in-house “GUI Test Driver” to perform automated GUI tests, which uses XML and XSLT and runs on top of Nunit framework. But at any rate, I am a clear supporter of test automation if done at the unit level. And of course there are areas of testing (like stress testing, performance testing, penetration testing, link testing, API testing etc) where the need of automating your tests becomes an absolute necessity.

Test Automation Pros:
1. Computers work much faster than a human tester and are less prone to get confused while doing multi-tasking (doing several tasks at the same time, switching between them). Computers never care to spend time on attending a phone call or attending a review meeting/presentation, unlike a human tester. This can result in much more productivity as compared to a human tester, of course under certain contexts.

2. Computers are great at mathematical calculations. Computer's memory is much more accurate than a human brain, it’s capacity to remember stuffs is much more than a human and it’s capacity to retrieve data from stored memory is much faster and more accurate as compared to human brain. This accuracy of computation can be exploited via test automation, especially in tests that involve high volume of computation work.

3. Computers never skip any hours of testing. People get tired, people get distracted, but computers are great at repetitive tasks humans are not very good at. Computers need no rest, shows no sign of fatigue, never gets bored with repetitive work, can work on weekends, holidays and even night shifts without complaining at all (of course until they encounter a break down/malfunctioning). This power of computers can be used to our advantage via test automation.

4. Computers never complain about salary hikes, number of paid leaves, holidays. Nor do they ask for change of project if it starts to get boring working on the same project for a long time. Tools lack emotions and it can prove to be a good thing when looked from an employee satisfaction point of view. [Remember, an organization spends on a tool. Hence the tool is also like an employee working for the organization]

5. At certain contexts, test automation can be cost effective than manual testing. Paying for the development of a tool and a small group of testers to run it may be cheaper at contexts, as compared to maintaining a whole big group of testers testing everything manually.

6. When it comes to load test, obviously automation is an absolute requirement. Gone are the days when you summoned 100 employees into a big lab, asked each of them to hit the Enter key at the same time and hoped that they hit it exactly at the same point of time (millisecond level)! In testing we need accuracy of actions. And for scenarios like load and performance testing automation tests can offer us such precision.

7. Going for test automation can be a good choice for regression testing nightly builds. If you have a set of tests that you need to test periodically over a long period of time and if you are confident that little or no major change would be made in areas covered by those tests, then you might opt to automate such tests to save you some precious time.

Having said all this let me summarize my opinion. Coming to the core question of “can automation tools ever *replace* human testers”, I honestly don’t think that it is going to happen ever. Well, not at least in next few centuries, not before there are no more software to develop and they are *completely* tested or not before human beings become so lazy that they even stop moving around [watched WALL-E?]. Until then, I think it would be impossible for manual testing to be phased out. There will always be a requirement for *skilled* manual testers. You can't automate everything and you would always need testers with skill and experience, people who can think as they test. Automation needs a human tester to write the test scripts. Technology is yet to reach a stage when tools can themselves write tests on their own. And again automation may only speed up execution but what about the analysis of failed tests! You still need an analyst (a human tester) for analyzing a failed automation test. Furthermore, what an end user does can be compared with repeated testing and retesting of the application. And those tests are *mostly* MANUAL (unless we are talking about a piece of application like an API). So if you wish to mimic end user’s (potential) behavior then manual testing is vital for your test coverage. Automated tests mostly run on a constricted set of parameters. There is usually no guarantee that the situation at the end user’s will fall neatly into this constricted set. There is usually no way to predict such differences (in an automated way at least). It takes a (human) tester's imagination and creativity to cover these situations. So I believe, “Automation testing tools can never replace human testers”.

Does this make automation look like a poor contender? Not necessarily, I would imagine. As we have seen above, test automation has some Pros that can be used along with manual testing to derive a higher productivity and efficiency. The key is probably to have the right balance. Modern day cricket is all about having all-rounders. And IMHO, so is modern day testing field. I find it pleasing to discover myself a born optimist. Though people might say that too much of optimism is not good for testing, I however believe that optimism still retains the key to success. Let me try and combine the Pros of both class of testers (human and computers) and see if we can bring out their best:

1. Automate tests that involve repetitive, tedious and high level of computation work.
2. Let human testers do tasks that are highly intellectual and needs analysis.
3. Use automation for areas that are least prone to change and yet require frequent testing. This way you can save some energy on tests that need to be tested periodically.
4. As much possible, let humans do UI testing.
5. Use automation for areas like regression, load, link (broken link testing), API testing, first level agile (unit) testing.
6. Don’t automate if you know the UI is going to have a revamp. The slightest changes in UI can cause automation to fail. When you know the UI has been settled down, excluding any fixes to bugs, that's probably the time to automate.
7. Realize and understand that not all tests can/should be automated. It will initially take longer to develop effective automation scripts than to do manual testing. Knowing what and when to automate is critical.
8. Test automation is a supplement to help/boost the overall testing. Automation can neither suffice on it’s own nor it can replace other kinds of testing.

I feel that manual and automation testing should be allowed to marry to live happily ever after and to give an offspring of new hybrid tests. Why not take advantage of both of their Pros and create a win-win situation for us rather than fighting over manual OR automated battle? What do you think? Do you think automation tools are going to prove weapons of mass destruction for human testers? Do you think tools can never take over the place of a human being? Let out your opinions via commenting.

Happy Testing…

Does software testing add value to the project?

2008-10-09T04:59:00.000-07:00

Last weekend I was invited to give a presentation on TDD (Test Driven Development) to a bunch of techies who largely constituted programmers, a couple of IT managers and some testers. At first I had wished if the audience had a major population of software testers, but as the presentation progressed I began to realize that probably I was wrong in my wish! Want to know what made me think otherwise? Read on…

But before I can start writing this post, let me tell you that this post is basically for software programmers who think:

1. Software Testing does not do any good to the project!
2. What is the point in spending project resources on testing activity if doing so can not guarantee a “100% Bug-Free Pakka Product” [please don’t ask me for the meaning of that paraphrase. I am simply quoting a programmer from my recent presentation]
3. Software Testers are a bunch of crazy people who find it amusing to find defects in the code.
4. Testing is a dumb job. What’s the big fuss? Anybody can do it!
5. Software Testing is a worth less process. It does not add any value to the project!

Having said that, in this post I am not going to start again the traditional Programmer VS. Tester fight. I am a Tester and I am more concerned about Testing. Hence I will try to confine this post to busting the myths of a typical Programmer about the necessity and importance of Software Testing in a Software Development scenario rather than trying to prove how programming sucks and testing rocks!

Coming back to my recent presentation, when I was trying to point out the fact that “even if a tested product can be considered as robust and reliably (known) bug-free, it may not necessarily mean that it is a High Quality Product”, a programmer friend from the audience came up with the following view point. Here is a truncated version of the excerpt of the discussion:

Mr. Programmer: If a product cannot be guaranteed of high quality even after testing, then does it make any sense to spend time, energy and effort on testing it?

Me: Well, dear friend to answer this question first of all I will need to know your understanding of “high quality”. Quality is a multi-dimensional aspect. As Jerry Weinberg describes it - “Quality is value to some person(s) who matters”! In case of Software Development, testing alone cannot guarantee high quality of a product. Think of a video game that is free from any known bugs and is robust and stable enough but unfortunately it fails to engage the curiosity of the players (users). Would you still call it as a quality video game simply because it does not contain any visible defects and is quite robust? At any rate, Software Testing is not a salesperson job. As testers we test software. We don’t sell television sets where we can (and we are supposed to) give you guarantee for the product that we are trying to sell. As a matter of fact, even television companies these days offer warranty, which comes with a long list of “if you do this you void your warranty” terms listed under Conditions Apply* column. Then how can someone expect a tester to offer guarantee for a Product, which he has not even developed himself? Isn’t it unfair?

Mr. Programmer: Taking about fairness, even after years of release of Windows XP, still Microsoft keeps releasing Service Packs to patch defects that were shipped along with the original Product. If there can be bugs in a piece of application even after testing then where is the point in testing it in the first place?

Me: That is an interesting question, thanks. Before answering your question let me tell you some facts. In an average MNC the Tester:Programmer ratio hovers somewhere around 1:3. That means for every 3 developers there is one tester to test their coded program. In case of Microsoft the ratio is said to be 1:1 and in NASA it is known to be a whopping 3:1. As you can clearly see, NASA spends obviously lot of effort on testing its projects. So apparently their projects must be the most robust among all. Unfortunately still we see major space mission failures like Space Shuttle Columbia disaster from time to time. Does this mean that the testers of NASA are doing nothing? Why don’t we look at it at from a different perspective? Microsoft had to release 3 Service Packs for Windows XP. It is quite possible that in case of lack of enough testing effort the number of Service Packs could have been even 30! NASA ends up with a space shuttle mishap once in every decade (on an average). In case of lack of enough testing it could well have been 10 accidents per year!

Mr. Programmer: Okay that sounds like a plausible explanation. But how would you defend yourself in case a Product that was tested by your testing team is reported with user found defects? They are your (tester’s) defects, which have slipped into production, right?

Me: Well, I don’t think that I need to defend my testing team here for any reason. That’s because the way I see it, it is not the tester’s defect rather it is the programmer’s! Because it was the programmer who had made an error in his code at some point of time and in turn had given birth to the defect. How can you even think of blaming a tester for a defect that was created by you (a programmer) in the first place? Having said this, I do agree that there are moral obligations for such kinds of defects that slip through the testing phase. It’s not that I don’t feel guilty for have missed such a defect before it managed to reach the end user. I DO feel guilty about it. Every responsible tester does. But I honestly feel that the programmer should also feel equally guilty (if not less) for such defects. It is not a blame game. Instead of accusing each other for the mistake made, we must feel responsible for the mistake made and more importantly take corrective actions so as to avoid such mistakes from happening in future.

Mr. Programmer: You are right. But my point is if testing can not make sure that the released Product is not bug-free then where is the point in spending resources (time, person hours, money) on testing it before releasing it? How does testing add value to a Product development?

Me: I am glad that you asked this question, thanks. Talking about value addition let me ask you a question first. Why do you taste your dish before serving it to your guests? Does tasting a prepared dish add value to the preparation? Can tasting make the dish better? I don’t want to argue whether tasting the dish adds value to the preparation or not, but I know it gives us an idea of the preparation. It gives us information about the preparation (another pinch of salt needed, the dish is bit spicy for the taste of your guest, it could have been better if you had left it on the stove for another 5 minutes etc). These information can help you in making the preparation better. So does tasting the dish add value to its preparation? It’s up to you to decide. Similarly does testing add value to the project? The answer can be Yes or NO depending on your way of looking at it. It does add value if you consider information gathering as a part of value addition. It does not add value if you think that only coding the application is all about value addition. However, testing does provide important value related information about the Application Under Test (AUT) to the stakeholders of the project. And this investigation and information gathering is as much important as the development of the application. That is why it becomes important to spend resources on testing.

Either my replies were satisfactory to Mr. Programmer or he was tired of fighting back with more “we don’t need testing” arguments, but after this I did not hear more questions from him till the end of the presentation. I genuinely thank him for bringing such questions to my attention. At least by doing so he gave me a hint of idea about how a programmer feels about testing. I have no hard feelings for him. He felt that testing is a useless job may be because he had not got any chance to be made aware of the need of testing in Software Development. I hope I was able to instill at least a little bit of awareness and respect for testing in his heart. I hope I was able to make him realize the fact that just because we testers catch defects in your codes, we are not your enemies. We simply do it to save you from the embarrassment before someone else (a senior programmer, a manager, an end user, the client) catches it!

Coincidentally, at present there is a discussion (Risk-Reward, Output-based pricing model for Testing services - Is our community ready?) going on in software testing yahoo group (software-testing@yahoogroups.com) where I saw Michael Bolton responding with an excellent reply to a similar question. I am quoting him below:

Thinking more expansively, if you consider the project as the sum of the product AND everything we know about it, then testers do add value by adding to the pool of knowledge. - Michael Bolton

I don’t think I could have described it more brilliantly. As Michael points out, testers do add value to the project by providing important mission-critical information to the stakeholders. What do you think? Do you think testers add value to the project? Or they don’t? Let me hear your viewpoint by leaving your comment.

Happy Testing…

For those who might be wondering how my presentation about TDD looked like here is the slideshow:

Can your kid beat you in testing?

2008-09-04T07:40:00.001-07:00

Recently I was conversing with Chris (he is a tester friend from Poland) on Skype and he told me about his recent post on kids (his son) as the best testers. I must confess that was how I got the idea for this post. Thanks Chris for giving me a topic to write something on. Although I have read before on quite a few occasions on similar topics (using kids as software testers) this time I decided to try and brainstorm on this and see what comes out!

Coming back to Chris’s story, I am not sure if I would want to call children as the “best” testers in the world. As a Context-Driven thinker/tester myself, I am little shy of the word "best"! To me, the definition of best can vary depending on the given context. What looks and works out as best for me under one particular context MIGHT or MIGHT NOT work as the best for someone else in a different context! So calling children as the “best” testers in the world is something that I feel would be little too dangerous to declare. However, children could have some of the great qualities (skills) of a good tester.

Why I think kids could be good testers:
1. Curiosity: One of the things that I find interesting in kids (from a tester's point of view) is their curiosity for everything that they come across with! You might have noticed kids asking “why” to explore about new things. By asking why they show their curiosity to develop further knowledge of the world. I often wonder "why" as adults we tend to forget to ask "why" as often as we did as kids! IMHO, any good tester should learn to ask why because this is one such question, which is capable of unearthing a whole lot of information. I find kids more spontaneous and instinctive in asking “why” and showing their curiosity for stuffs, thus making them good candidates for testers!
2. No fear of the unknown: Pass a kid an unknown toy (any unknown object). Chances are great that she would surprise you with her curiosity to explore it instead of fearing it. As we grow up our fear of the unknown also grows up, which sometimes can affect our testing in an adverse way. A good tester should not fear the unknown (new technology, new domain, new testing tool, new test team and so on); rather leaving aside inhibition should go ahead, explore and learn about it.
3. Lack of guilt after breaking something: You must have observed that kids seldom feel guilty after they break something. It’s their innocence that restricts them from realizing that they should not have broken it. This is very important for a tester. If a tester starts feeling guilty for breaking the software (in fact finding out places where the software is broken) her testing will be in jeopardy. It would be like a doctor who fears hurting a patient by giving injection! The hurt due to the injection might be painful, but the doctor must understand that it is temporary and far less lethal than the disease that might risk the life of the patient without the injection!
4. Lack of common sense: Having common sense might be a good thing. But at times too much of common sense could lead to expectations and expectations in turn might lead to inattentional blindness. Imagine asking a kid to fill up a user registration form that contains those typical fields like First Name, Last Name, Address, Date of Birth, Email Address and so on. A grown up adult has the common sense to use only alphabets in a Name, to use comma and space between Address field parameters, to use valid Email format (e.g. myname@mail.com), knows a year has 12 months and a month can have 28-31 days. But a kid if asked to fill up such a form might end up entering all kinds of garbage. Thus the kid would be (albeit inadvertently) testing if the application does have proper validation in place to detect and avoid such invalid inputs.
5. Don’t require manuals/documents: You hand over a kid a new toy and she would not ask you for the User manual to learn how to operate it. Instead she would snatch it from you and would start playing with it to figure out how this thing works! How often do you hear a tester saying something like this: “I can’t start testing without SRS/BRS/FRS/XYZRS”! If it were a kid, instead of expecting base documents to start testing (as if those requirement documents were written by god himself and they cover every detailed requirement of the client accurately) she would have started exploring (testing) right away!
6. Don’t require test cases: Kids could be far more instruction independent than an adult. They don’t require to be told how they should do certain task. In fact even if they were told to do something in some “specific” way, more often than not they would deviate from that “specific” way and would try to see other possible ways. This could be a very desirable character of a good tester!
7. Never worry if somebody would think them as idiots: Kids never bother if someone would think them as idiots if they do something silly. A part of a tester’s job is to perform all those things that a normal human being would not want to do (hence can be termed as “silly” by such normal human beings). But mostly these are the “silly” actions that uncover hidden defects in the software.
8. Don’t have to worry about deadlines: Ever asked your kid to finish brushing in 10 minutes? Then you know what I am trying to talk about here. Sometimes deadlines of tasks can affect our productivity in an adverse way. But kids could be invincible to the pressure arising due to such deadlines, thanks to their god-gifted care-freeness!
9. Ignorance could be bliss: Kids can be good examples of this proverb! What you don't know can't hurt you! Sometimes no knowledge about something could prove to be a better thing than having partial (and often misleading) knowledge about it. As adults as our boundary of knowledge grows the chance of their dilution also increases. But in case of kids their lack of knowledge could help them in getting rid of any prejudices/false-assumptions like an adult tester!
10. Vulnerable to emotional distress: Kids are emotionally more fragile and susceptible to emotional disturbances like anger, frustration, agony, sorrow etc. Thus they are better candidates for identifying problems that have potential to cause user dissatisfaction. An adult tester might not be able to identify a problem with the login process of an online game site that takes minutes altogether to allow you in. But kids would be in a better position to spot such anomalies, thanks to their sensitiveness to emotional distress (in this case frustration caused by a slow login process)!

Why I think kids could be terrible testers:
1. Inability to identify problems: Kids often lack the ability to identify problems even after uncovering them. Only triggering problems is not all about testing. One needs to identify them. Kids often lack such skills. As a tester if you are not able to identify the problem, then it does not do any good to the testing effort! Even if a tester identifies the problem she should be able to reproduce the problem reliably to help the programmer figure it out. Kids might not be as good as a skilled tester in this matter.
2. Inability to report errors: Even if a tester identifies the problem if she lacks the ability to report them appropriately, then it’s as good as nothing! Kids might not be the best testers to report errors efficiently.
3. Poor decision-makers: Kids often lack decision-making abilities. They might lack the wisdom to take well-judged decisions like an adult skilled tester.
4. We need testers; not a demolition army: A tester’s job is to find places in the software where it is broken! Think of a toy and compare it with a software (AUT). Given to a kid, chances are more that the kid would break it while trying to explore it rather than finding if the some part of the toy was broken at some invisible portions during manufacturing! If the same happens in a testing project it might turn out to be a testing nightmare for the test manager!
5. Too much of questions could backfire: Questioning skill is no doubt an essential weapon in a tester’s arsenal to dig out problems in the AUT. But questioning is a skill. Over doing it might prove unproductive. Asking the right questions at the right time is something that differentiates a skilled tester from a novice. A kid might lack such skill.
6. Enough Testing, slowdown cowboy: A kid might not realize where to stop testing! Knowing when to stop testing is as much important in testing as knowing when to start testing. But in case of kids, they might lack the maturity to understand it.
7. Might shoot at their own legs: Being adventurous is a good thing for a tester. But stepping into traps and danger zones is something a good tester should always try to avoid. But in case of kids, they might lack the wisdom to differentiate between right and wrong!
8. Could ignorance be bliss? Ignorance might be bliss at certain contexts. But ignorance of what you are doing (testing) could be termed as stupidity! If a tester does not realize “why” she is testing something (“what”), then it makes no point in the testing effort made. The same could happen with kids as testers.
9. Too much of emotion could spoil testing: Emotions have the power to help a seasoned tester to detect problems in the AUT. But too much of emotion might distract a tester from her work. Testers are like detectives who perform technical investigation on the Product that they are testing. Imagine a situation where the detective herself is too emotional and breaks down upon seeing a disturbingly severed subject on a crime scene! It could prove disastrous for the success of the crime investigation. Detectives are meant to be strong. As testers we do face many situations in our professional career, which can be termed as disturbing and hostile! But a good tester knows how to keep her head cool and faces such situations tactically and negotiates a strategic way out of it. Kids might lack such diplomatic skills.
10. Because they are kids: Yes you heard it right! They are kids. They lack the maturity of the adults. And good testing requires a level of maturity that might be hard to find in a kid! May be someone was right when he said - “Testing is no kid’s play”!

Well, wait a minute before you start complaining that this article does not prove any point! At one hand, I am trying to say that kids might have certain traits that are helpful for a software tester. And at the other hand, I am trying to counter my own argument by listing down points on how kids might result in horribly bad testers. Then where is the point of this post? Well, I would request you to look closely. In this post I have tried to identify and explore the possibility of using a small kid to test software. The idea might or might not work [I am yet to experiment it with my kid and see the results after I get married :)]. However, it appears that even though kids do have certain desirable qualities of a good tester, the lack of maturity in their thinking ability kind of offsets all those goodies! But still, it would be interesting to watch a kid testing something while an experienced and skilled tester watches from the back. In fact, there are people who have tried it already and their findings were certainly exciting. You can read more about their stories at the end of this post under further reading section. But before you head for those articles saying goodbye to this post, how about taking a couple of minutes and sharing your thoughts via commenting!

Happy Testing…

Further Reading:
What software testers can learn from children - by Scott Barber
8-year-olds should test my code - by Michael Schidlowsky
Are you smarter than a 3rd grader? - by Ben Simo

Priority Report!

2008-08-27T21:29:00.000-07:00

Dolly: The more I read on the net about Bug Severity and Bug Priority the more I am getting confused. Everyone out there seems to have his own way of describing it and it leaves me even more confused. Can you help?

Victor: Why do we have two fields in a defect tracker to fill in - Severity and Priority? Are not they all the same? I have noticed that when a defect is of high severity usually it is also assigned with a top priority and vice versa. If they mean the same, why do we have 2 different names and 2 different fields in the defect tracker? Why do we need these two different parameters (severity and priority) for a defect? Can't we do only with one?

Krishna Prasad: I want to know about Priority and Severity of software bugs. Please clarify my doubt with one example of each for the following:

1. Low Severity and Low Priority

2. Low Severity and High Priority

3. High Severity and High Priority

4. High Severity and Low Priority

Once again I am back with few FAQs from my mailbox. And this time they are regarding the age-old question of Bug Severity VS. Bug Priority. I can imagine how confusing it can become for someone new to the S/W testing field. But before I can try to offer any kind of solution to dismiss this confusion, kindly allow me to present you with a small story!

One afternoon at a theater:
One fine Saturday morning when Ashley (the female protagonist of my story) is still enjoying the warmth of her bed, much to her annoyance, the cell phone rings breaking the silence of the room; as if yelling to make her realize that its 8AM already and the world out side her hostel room is up and on its feet. Lazily she reaches out for her cell phone and as she looks at the screen a smile flashes across her face, even without her knowledge. It’s her BF on the phone. He asks if she can bunk her weekend MBA class and watch the noon show of the latest blockbuster in the theater near to her hostel! Ashley thinks for a moment and nods in affirmative with the typical unforgettable blushing smile flashing across her face. :) Everything seems to happen as per the plan [telling lies to her roomies, bunking the important class, reaching the multiplex in time; something that does not happen quite often :)...] until, to her horror and surprise she finds out in the theater that the person sitting behind her is none other than her own cousin who also stays in the city! Oops…

Moral of the story:
How would you describe the situation (in terms of severity)? Critical, Major, Normal, Minor... Well, assuming that Ashley’s cousin is unaware of the love story I would imagine that she would probably consider this as a Critical situation. Won’t she? Now considering that we have a critical situation (bug/defect) at hand... How fast she would want to fix this up (either by taking her cousin on a Lunch and begging her not to tell about it to anybody or by simply trying to get even with her and threaten her saying that if she opens her mouth about it she would doom her by telling about the cousin's own secret love story to others) describes the priority! The more desperate Ashley is to fix this situation up, the higher the priority! Isn't it? :)

Bug Severity:
WordWeb dictionary defines the word “Severity” as - Used of the degree of something undesirable, something hard to endure, extreme plainness. Severity of a defect/bug tells us how undesirable the defect is. For example, a bug that causes the program to crash and terminate would be considered as high severity, while a minor spelling error might be of low severity. In our organization we use a 7-point scale to rate the severity of a bug:

Blocker: This type of bug blocks development and/or testing work. Blocks users from completing the task the function was created to facilitate.

Critical: The software crashes, hangs, or causes you to lose data. E.g. Crashes, loss of data, severe memory leak.

Major: Major loss (or lack) of function. Users cannot do what they NEED to do, with no workarounds.

Minor: Minor loss (or lack) of function. Users cannot do what they WANT to do or a major problem where an easy workaround exists.

Trivial: Cosmetic problem like misspelled words or misaligned text. No loss of function.

Enhancement: Request for new feature or enhancement.

Unprioritized: The reporter has no opinion on the criticality of the issue.

Though this can vary from organization to organization, the overall practice may remain similar. You are free to adopt any terminology that suits you as long as they describe the impact of the bugs accurately and unambiguously.

Bug Priority:
WordWeb dictionary defines the word “Priority” as - Status established in order of importance or urgency. Priority of a defect/bug tells us how soon it is required to fix the problem. Priority reflects a business decision as to how soon that bug should be fixed. Priority of the bug determines what gets fixed next and what does not. The priority of a bug can be decided from either the Project management point of view or from the user’s point of view. A bug that is of high priority from the Project management point of view may or may not be of high priority if assessed from the user’s point of view. Check with your users; their prioritizations may surprise you! Priority of a bug can be classified as:

P1 – FIX the bug ASAP before the release, preferably in the current development iteration itself.
P2 – The Bug can be fixed in the successive iteration. But the fix must be done before the next major release of the Product.

P3 – The bug can be left to be fixed in the subsequent releases.

P4 – There is no hurry. This bug can be fixed as and when time permits.

Bug Severity Vs. Bug Priority:
The severity of a bug does not necessarily translate into the urgency to fix it. A severe bug that crashes the application only once on a Feb 29th (leap year) for 0.0001% of the users is lower priority than a mishandled error condition resulting in the need to re-enter a portion of the input for every user every time. Many bugs cause crashes, but aren't fixed because the crash is very infrequent or on a version/platform/feature low on the vendor's support list. Corner-case crashes, crashes that are dependent on a rare combination of sequence of events (trigger to set off the crash) fall under such classification of low priority bugs. Once I was testing the Parser of my application. And the Parser used to crash when the parameters were: 'MaxNulls' = '999999999' and 'MaxPrefixes' = '999999999'. This was a catastrophic crash, which was causing the Program to terminate. However, it took almost 2 years to get this crash fixed as it was rated with a low Priority (despite of its high severity) for obvious reason! :)

ATM Machine Example!
Let’s take the example of an ATM machine to further understand the different possible combinations of bug severity and priority:

Low Severity and High Priority Bug: Suppose you are testing an ATM machine and you notice that the welcome screen displays a misspelled Bank name! (e.g. in place of “Welcome to ICICI Bank” it rather displays “Welcome to UCICI Bank”. This is quite possible, considering the fact that the letters “U” and “I” share close neighborhood in a typical QWERTY type keyboard). This might be a cosmetic error and would not hamper the functioning of the ATM machine in any possible way; still this could prove to be a real pain in the eyes for a customer. Considering that the spelling error is in a frequently used part of the program, it might give an overall bad impression that could hurt the Bank’s reputation. These kinds of bugs are real annoyance for the customers and no Bank would want to loose revenue for a silly error like this one. So though this bug is of low severity, its priority would be high for obvious reasons. It is quite interesting that often, low severity cosmetic errors like these get a high priority.

High Severity and Low Priority Bug: Think of a bug that cause the ATM machine to black out if an user tries to use an expired ATM card and try it for 13 consecutive times (even after the machine keeps rejecting it)! Sounds like a critical crash? Well, it might be a critical bug as far as its severity is concerned, but don’t be too surprised if you see a low Priority being assigned to it. Reason? I think its obvious!

High Severity and High Priority Bug: These kinds of bugs are easier to imagine. Any kind of bug that results in some major catastrophic failure (crashes, system hang, memory corruptions, data loss, functionalities that does not work and so on) and that occurs in areas of the application that are frequently used can be classified under such types of high severity and high priority bugs. Think of scenarios where the ATM machine - does not detect a valid card even after entering correct PIN number, hangs if tried with an invalid card and/or PIN number, blocks the card after just 1 unsuccessful entry of PIN (instead of 3 wrong entries), does not dispense money even if there is sufficient cash in the a/c and in the ATM machine, dispenses incorrect amount to the user, does not forget the previous session even after the transaction is complete (in case of ATM machines where you have to swipe your card instead of inserting it), does not return the card even after the transaction is complete and so on. As you can imagine all of these cases of bugs can fall under high severity and high priority type.

Low Severity and Low Priority Bug: These are often low impact low on urgency-meter bugs that are quite harmless and can be fixed without a hurry. These kind of bugs do not harm the application in a disastrous way and the chance of an end user/customer being annoyed by it is also very less. Think of a scenario where the ATM machine does not append a title (Mr/Miss/Mrs) to the name of the customer in the welcome screen. This would not affect the business in a drastic way. And there might not be many customers who would mind it if their name is displayed without a title.

Hope I was able to make the distinction (Bug Severity Vs. Bug Priority) clear through the examples. Now here is a small testing exercise for you. Why don’t you try and present few more examples (at least one from each combination viz. high severity and low priority bugs, low severity and high priority bugs and so on) taking the ATM machine as example?

Happy Testing…

Repeatability of Tests - A necessary Evil!

2008-08-21T07:52:00.000-07:00

“If you cannot repeat what you are testing then you are NOT testing properly! Think of regression testing. What will happen if your regression tests are exploratory by nature and are non-repeatable”? A tester friend of mine argued strongly, trying to give himself ground to continue his fight against my advocacy for exploratory testing!

This made me cogitate. Not that he had succeeded in proving that Exploratory Testing (ET) sucks! But I had to start thinking if repeatability (read reusability) of a test is really all that bad! I guess “repeatable test” VS “non-repeatable test” argument is just another face of the “Scripted testing” VS “Exploratory testing” debate. Here in this post, I am not going to continue the same debate trying to prove which one is better and which one is not. Rather I would like to think more on the repeatability aspect of a test!

Some questions to start with can be:
1) Can ALL the tests be made repeatable?
2) Should ALL the tests be made repeatable?
3) By making a test repeatable are we not (may be inadvertently) making them too predictable?
4) Can repeatable tests discover NEW problems (defects/bugs) in the system?
5) Blindly trying to make each and every test repeatable. Is it worth the effort and expense?
6) Why should we repeat just a set of tests (test cases, test scripts whatever), when we can utilize the same time exploring much more tests that may uncover unknown defects!
7) When we say a test is repeatable, is the test really repeatable in all its senses? Can someone guarantee that the test runs exactly in the same way (environment, concurrently running processes and applications, exactly same DLLs loaded at that moment, machine condition, configuration settings, and so forth) as it had run the last time around?
8) Do repeatable tests guarantee reproducible defects/bugs?
9) A repeatable test can make sure there is no re-occurrence of an earlier defect. Ahh well! But how long? Will these so-called repeatable tests not loose value over a period of time (after a number of iterations)?
10) Is software testing all about repeatability?

The argument in support of making your tests repeatable may hold good to some extent, in certain cases like Regression testing or for that matter Performance testing. But is there any point of trying hard to make ALL the tests repeatable? Here are few interesting quotes from some honorable Testing Gurus/Experts:

Highly repeatable testing can actually minimize the chance of discovering all the important problems, for the same reason that stepping in someone else’s footprints minimizes the chance of being blown up by a land mine.
- James Bach, Test Automation Snake Oil, 1996
- Brian Marick's talk Classic Testing Mistakes

By repeating tests we actually make sure we are avoiding other possible defects (remember minefield analogy? Stepping on someone’s footstep actually makes sure we may avoid stepping on a live landmine) thus minimizing our chance of discovering new defects!

Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.
- "Pesticide Paradox", Boris Beizer, in his book Software Testing Techniques, Second Edition, 1990

"Pesticide paradox" compares software defects with that of pests! Imagine a context when a farmer applies certain pesticide to get rid of insects from his crop. There is every possibility that there will be some insects that will survive the pesticide. If he keeps applying the same pesticide, the insects eventually may build up resistance and the pesticide would no longer work! "Pesticide paradox" describes the problem that a regression test series gets less and less powerful as you use it over and over again. When a set of repeatable tests is run for a period of time they tend to pass more often than failing. A test is valuable when it fails and the failure uncovers a bug. Running a set of tests that seldom fail and have least chance of uncovering defects sounds like a bad idea. Isn’t it?

Having said this, does this mean repeatable tests are waste of time? May be not!
1) Think of Performance tests that a tester may need to run over and over again for days in and days out.
2) Think of Benchmark tests.
3) Think of Build Verification Tests/Sanity Tests that need to be run every time you have a new build!
4) Think of Regression tests.
5) Think of a scenario where you need to run certain tests that are important in nature. The tests that verify some very critical functionalities of the application and these tests must be run periodically to make sure those functionalities continue to work without issues.

So, it appears that having some repeatable tests can be helpful for your test project along with those tests that are non-repeatable. As always, it certainly seems to be context dependant and depends on your testing mission/goal! What do you think? Do you think ALL (what that can mean!) tests MUST be repeatable, as if we are not software testers rather some robotic human beings trying to repeat some algorithm already decided in advance! Repeatable tests! Are they the need of the hour or necessary evil? Your thoughts please.

Happy Testing…

Further Reading:
Reasons to Repeat Tests - By James Bach
Repeatability is Overrated - By Elisabeth Hendrickson

What to test? Pareto analysis, High-risk areas, User feedbacks…

2008-06-04T04:15:00.000-07:00

Recently during a team discussion with my Manager, he came up with a question – “What is the best use of our testing resources? How should time be allocated? Which areas should we test more often?” Luckily for me, he clearly understands that it is not possible to test everything (let me tell you, not many managers understand this simple fact). He also knows how from an economic stand point, it does not make sense to spend lots of time testing areas of the application where the chances of having defects/bugs are low, or even if defects/bugs are there the impact of those defects would be low as far as the end users are concerned. So he was curious to explore the kind of strategy we should follow while deciding a roadmap to test our upcoming yearly Corporate Release. This might sound like a simple question, but in real environment one can imagine how complex it can get to formulate a testing strategy to test a release.

Well, testing is not all about hitting the keyboard and clicking the mouse until you stumble upon some hidden defects in the application. Finding corner-case defects that nobody in real environment would ever come across might not make much sense. Similarly finding defects, which are trivial and have least chance to trouble a real-time user can eat away your precious testing time. At the same time, missing important defects could be vital for the reputation of your application after release and also for your own credibility as a tester. Keeping all these things in mind, often it becomes bit tricky to adapt a testing strategy that answers all these possible loopholes and yet helps the testing team to perform adequate testing on schedule.

Coming back to my Manager’s question, he had a suggestion for his own question - to focus most time on high-risk areas of the software and less time on the areas of less risk. In other words, test deeply and frequently in areas of high risk, and spread the rest of the testing effort broadly and less frequently over the areas of less risk. But now the question was how to decide which are the high-risk areas and which are of low risk. Some more brainstorming, and we came up with these points to start with:

1. Pareto Analysis
2. Try and identify High-risk areas
3. Analyze User feedbacks

Pareto Analysis:
As most of you might already know, Pareto analysis is a statistical technique in decision-making that is used for selection of a limited number of tasks that produce significant overall effect. It uses the Pareto principle (80-20 rule) - the idea that for many events, 80% of the effects come from 20% of the causes. Or in terms of quality improvement, a large majority of problems (80%) are produced by a few key causes (20%). [Wikipedia]

In case of our Product, we use JIRA for issue tracking (bugs, user stories, improvements etc). And we decided to filter out last 5 years (coinciding with the first beta release of our Product) JIRA bugs (with Resolutions ranging from Unresolved, Completed, Fixed, Insufficient Details, Cannot Reproduce) and perform a Pareto Analysis on the filtered bugs to see if a pattern exists. Though personally I don’t believe much on statistics, this time the findings were interesting (if not shocking)! In our case, the Pareto Analysis showed that 80% of the bugs that were reported in last 5 years were present in areas constituting just 25% (well, little more than 20% though) of the whole application. Now we had at least something to start considering as the high-risk areas in our Product.

Try and identify High-risk areas:
Risk analysis can be used to decide where testing should be focused. As it is next to impossible to test every possible aspect of an application, every possible combination of events, every dependency, or everything that could go wrong, risk analysis becomes necessary (if not mandatory) for most contexts of software development. This might require analytical/judgment skills, ability to think like an end-user, common sense, and past experience. Some of the considerations can include:

» Functions those are often used by the users.
» Functionalities those are most visible to the user.
» Complex functionalities.
» Functions that have a lot of code refactoring, updates or bug fixes.
» Functions that require a consistent level of performance.
» Functions that use new technologies.
» Functions that require interfacing with external systems.
» Functions that use third party plug-ins/tools.
» Functions developed by more programmers at the same time.
» New functionalities.
» Functions developed under extreme time pressure.
» Functions those are most important to the stakeholders.
» Functions that reflect complex business logic.
» Functionalities that have large financial impact (e.g. shopping cart, online transactions, payment gateways etc).
» Functionalities that have large safety impact (e.g. health care products, life critical systems, software to be used in automobile/airlines, navigational gadgets etc).
» Functionalities that have large security impact (e.g. login forms, database transactions, authentication and authorization criteria etc).
» Talk to your developers and try to identify parts of the code that are most complex, and thus most prone to errors.
» Similar/related functionalities with bad history of defects.

Analyze User feedbacks:
User feedbacks often provide good enough idea about the areas in which the end users have most chance of finding defects/problems. If you have a Product support team for your software, keep your eyes and ears open to the help desk call transcripts. By doing so you can have an overall idea of the risk areas that your end user is prone to hit upon! You can seek help of your Business Analyst to get some ideas on the areas that she thinks as high-risk zone.

Once you have a clear picture of the high-risk drive-slow zones of your software, you would be in a far better position to plan out a strategy for your Testing efforts. Test deeply and frequently in areas of high risk, and spread the rest of the testing effort broadly and less frequently over the areas of less risk. However, I am curious to hear your stories. How do you decide the high-risk zones in your software? How do you allocate testing effort (and time) for different areas/components? Share your ideas via commenting.

Happy Testing…

Programming Skills and Testers!

2008-04-03T20:47:00.000-07:00

Am I a Tester or a Programmer? Who am I?

Abhijit Navindgikar: “I am having one question regarding software testing. Currently I am working as a software tester. I am having 3 years of experience in manual testing and having the basic knowledge of C/C++. Is it necessary for me to learn new technologies like C# .NET for future prospects in testing? Is it necessary for the tester to have the programming skills also? (Till know I didn’t feel that tester need to know the programming but let me know your views). Also will it be possible for a black box tester to shift his career to white box testing? If yes what steps need to follow to start the same?”

Lakshmi: “Recently I was browsing some of the testing sites and came to know that "No knowledge on programming language is a hindrance to a tester’s career". Is that correct? Really, programming language knowledge is that much necessary for a tester? I am not able to digest this. Please help me.”

John: “My test manager feels that all testers must have decent programming skills. He is so much obsessed with his belief that I fear he may go ahead and fire testers who are not so good in programming (even though they are quite good at testing). Is there a way to change his mindset without hurting his ego?”

These are excerpts of few emails from my inbox. These are kind of questions that always have made me to think. Every time I think on these questions, some more questions start popping up in my mind. Questions like:

1. Knowledge of programming and effectiveness of a tester – is there a connection?
2. Can a good programmer make a good tester?
3. Can a bad programmer make a good tester?
4. Can a non-programmer make a good tester?
5. Can I think of contexts when knowledge of programming can enhance my testing?
6. Can I think of contexts when knowledge of programming can hamper my testing?
7. Can I think of contexts when ignorance of programming can enhance my testing?
8. Can I think of contexts when ignorance of programming can hamper my testing?
9. The list of questions continues…

I have seen quite a lot of such “Do testers need programming knowledge” kind of debates, especially since Microsoft started distinguishing its testers as Software Test Engineers (STE) and Software Design Engineers in Test (SDET)! And this debate gathered momentum as Microsoft started shifting weightage from STEs to SDETs while hiring (and firing) test engineers! However, I am not going to extend the STE Vs. SDET debate in this particular post of mine. Rather I will try to focus on the need of programming knowledge for a tester.

To me the single sentence answer for the question “Do I need programming skills to excel as a tester” can be – “It depends”. It depends on your particular context, the type and complexity of the AUT [Application Under Test] and more importantly your testing mission. Blindly hiring testers who can code may prove to be a bad idea, especially if you are neglecting your testing mission while taking such a biased decision!

Having said that, there can be contexts where a tester who can code, can be an asset for the test project. Think of scenarios where testers need to automate GUI tests. Even if the tester might be using some so-called record-and-playback kind of tool (WinRunner, QTP, IBM Rational Robot blah blah...), knowledge of programming, can be an added advantage. That can allow a tester to tweak his tests at script level to make them more powerful and flexible! e.g. The tester need not re-record his tests each time a object name for a recorded test object is changed (may be due to recent code refactoring). He can simply go and change that particular object name from the script. As Danny Faught talks in this interview, "Test automation requires programming skills. Plain and simple, no way around it. No tool can get around that."

In cases where the organization (or the client) can not afford acquiring license for costly automation tools, the programming skill (in most cases knowledge in some scripting languages like Perl, Python, Ruby, JavaScript and markup languages like XML, XSLT) can help the tester in setting up a home-grown test driver framework to cut down the license cost of Commercial tools and at the same time can result in a customized tool that fits better for your testing requirement. Forget about test automation, testers with programming skills may detect defects earlier in development cycle and may also be able to pin point the cause for the defect (provided that the tester is aware of the underlying code and the system architecture). This may also help the tester in finding other areas in code where there can be similar errors. A tester who has a development background can also participate in code reviews, can debug problems, perform unit testing and identify patterns in the code that are error prone. If you are a tester, knowledge in programming can sometimes earn you greater credibility among the programmers.

But does this mean a tester without coding skills is useless? Of course not! To say Manual Testing (Sapient Testing, the James Bach way) is losing its value, in my opinion, is too early to jump into any conclusion. With user's expectations increasingly higher, it would be foolish to lower its value. How many testers do unit testing in this agile era of software development (where the programmer has to write unit tests for his own code snippet)? A good tester is a good tester for his ability to test, not for his ability to code! After all, a tester gets paid to test, not to code. Although coding background can help in certain contexts to test better, that should not be over-generalized! Test automation can never replace manual testing. I think, the same can be safely said about programming skills of a tester. A tester with coding skills can not replace a tester without coding skills. If asked to test an application, both of them will find different sets of defects. Think of user experience/interface related defects for example. Or for that matter, think of scenarios where you don’t have enough time to test. Would you sit down to do code review and perform a cause and effect analysis using your programming skills or would you rather exploratory test it right away?

If I were a test manager, I would rather hire testers both with and without coding skills. Testing is all about flexibility! Each and every member of the testing team can’t be expected to have equal skill sets and equal areas of expertise. Testing as a craft is evolving into different specializations and it must be understood that each kind has its own importance. So it always helps to have a testing team comprising of a variety of skill sets. Testers with different specializations (with or without coding knowledge) are not mutually exclusive rather they should compliment each other. What do you think?

Happy Testing...

Confusing Terminologies in Software Testing - A Case Study!

2008-02-20T07:05:00.000-08:00

Couple of days before, I received an email from a blog reader (Lakshmi NSMV) asking me the following query:

What is Build Verification Test (BVT)? What are the things that are taken into consideration while performing BVT? Is it the tester’s responsibility to perform BVT?

At first glance, it did look like yet another testing question that is often asked in software testing interviews (as if there is no other better question to judge the skill sets and competency level of the candidate)! If you have attended quite a few number of interviews for the post of a software tester, then chances are high that you might have already come across such buzzwords like BVT, Smoke Testing, Sanity Testing, Build Acceptance Testing and so on.

Well, I could have given a pretty straightforward reply to Lakhsmi’s question via email describing my understanding of Build Verification Test (BVT), how I define it and how I approach it. But I doubted if that could have helped her in any way. After all that would have been *my* definition of BVT and could have differed from others. Curiosity [to hear how others understand and define Build Verification Test (BVT)] got the best of me and I decided to contact few testers and seek for their definition of BVT. Here is the result:

Me: How do you understand (define) BVT (Build Verification Testing)?

Ashok: As per my experience BVT is - when we get a build to test, we will be verifying whether it is suitable for further testing or not. Using sanity testing and without any formal test cases we will be going through here and there to find any show stoppers. If we are not able to accept the build we will discard this build, switch over to the earlier build (provided there is one) and continue testing the earlier build.

Kiruba: According to my knowledge, after the build has been released, we will do the retesting and regression testing. It is build verification testing.

Santhosh: BVT is verification of the build by a Release Engineer. This BVT focuses on the main functionalities of the application. BVT is also called as Build acceptance test. Any build that fails the build verification test is rejected, and testing continues on the previous build.

John: This test is the first test, which we conduct as soon as we get the build from development team. Here we check/verify whether the build is usable, understandable, stable or not. Mainly we concentrate on whether the build is working or not and usable or not etc.; means verifying the build.

Ravi: BVT is nothing but Sanity Testing where we verify the final build to test its sanity or cleanliness.

After reaching this point, I had started questioning my decision to ask other testers for their understanding of BVT. Because, my own understanding of BVT was already diluted and on the verge of shattering as I had started getting confused hearing all these variations in the definition of a terminology as common as Build Verification Testing (BVT). As I was wondering whom to contact next who could come to my rescue and help me come out of such confusion, I spotted Mr. Shrini Kulkarni to be online on Google Talk! Thanks to him for instantly responding to my IM. Here is an excerpt from our conversation.

Me: Hi, hope I am not disturbing you. I wanted to hear your way of understanding (own definition) for Build Verification Testing.

Shrinik: BVT... Let me think...

Shrinik: It is a cursory method checking that few basic functions are working fine and the build is ready for more thorough testing. Typically BVT should not run for more than 30 mins to 1 hr and it is typically automated. You might confuse this with smoke test, sanity test or build acceptance test. Technically all mean the same or I can spin my own story to create my own definition.

Smoke Testing = Build Verification Testing (BVT) = Sanity Testing = Build Acceptance Testing (BAT)

More than the names it is important to understand the underlying principle. Here the underlying principle is "cursory checking" before detailed testing. You might decide to call it as "Debu's" testing. :)

Me: This is one of the reasons why sometimes I wish testing world were free from buzzwords, terminologies and abbreviations! Thanks a lot for guiding me on this.

Shrinik: It (testing world) cannot be (free from misleading terminologies and abbreviations)! We can only keep a constant eye on terminologies.

Me: You are right.

Shrinik: As Michael Bolton says: “People might use different words and mean same thing or people using same word might mean different things”. Hence, it always helps to clarify. James (Bach) some times says when confronted with a new word: "I might be doing the same thing as this word. But not by that name"!

Me: That is an interesting way to deal with terminologies that are not-so-familiar to us. Once again thanks a ton for responding to my query.

I think as long as there would be testing there would be terminologies to confuse testers. Having our own definition for a software testing term is just fine, as long as the definition reflects our own understanding of the term under question and as long as it fits well to our context and work environment. The actual problem seems to start when people start taking their own definitions as the universal definitions and start expecting others to understand things in the same way as they understand it. Asking such buzzwords in a job interview and expecting the candidate to reply in a way that exactly matches our own definition is, in my opinion, inappropriate, unreasonable and inhuman to some extent! How can we be so obtuse?

How to deal with Confusing Terminologies in Testing?

A tester’s questioning skills can come in handy while dealing with terms that are either new or confusing to him. As Shrini rightfully pointed out above, seeking for clarification is the first step to get rid of such confusion. Testing world is full of traps. And a good tester always tries to clear traps by asking relevant questions capable of revealing important information. Same strategy can also be applied while dealing with an alien term in testing as you hear it for the first time. When confronted with such a new word, ask what the person actually means when he uses such word/term/para-phase/abbreviation. And the reply could surprise you! You might discover that you already might be following the same testing principle, but might be using a different name to denote it. Asking for clarification, can demystify most of those dangerous looking terminologies and in turn can help you add some new words to your testing vocabulary! As a bonus, it can help a tester while communicating with different circumstances like:

a) attending a job interview.
b) discussion with a tester who belongs to a different testing community/country/organization.
c) communicating with different stakeholders (developers, clients, managers etc) of the project who use different terms to mean the same testing principle or vice versa.
d) communicating with fellow testers.

How do you deal with the confusion arising from misleading testing terminologies? What do you do when you are presented with a term that is new to you? I am curious to hear you. Voice out your ideas by Commenting!

Happy Testing…