Testlabs.com

iPhone Development Basics - Memory Management Part 5

2010-05-19T08:00:00.001-07:00

This week is the final video from Mark Johnson on iPhone memory management. In case you missed any of the previous videos, here's a list of the articles we've posted with Mark's first 4 memory management videos:

iPhone Development Basics - Memory Management Part 1
iPhone Development Basics - Memory Management Part 2
iPhone Development Basics - Memory Management Part 3
iPhone Development Basics - Memory Management Part 4

As in the previous articles, I've listed some suggested questions you might want to ask your development team in regards to memory management. Today's video offers specific situations when an app might crash because of memory problems. It would be particularly useful to ask your developers why this could happen - it may help them avoid these situations.

Suggested questions for Part 5:

(4:21min) What does it mean to call dealloc to the super class?
(4:41min) What is a leaking object?
(5:29min) Supposedly there are two types of memory bugs - releasing an object that is already deleted and forgetting to send a release. Why does the first one cause a crash and the second one just a memory leak?
(7:26min) What does it mean that there is only around 12MB of memory for an app before memory is increased?

Top 10 Tips For Testing iPhone Applications

2010-05-19T08:00:00.000-07:00

This article started out as a Top 5 list but quickly grew as we researched all the tips and techniques we use at RTL for iPhone application testing. Most of these tips came from tracking down memory-related problems, which often resulted in defects that were very difficult to capture.

1. Accurately report available memory. Many of the non-reproducible bugs you run into when testing iPhone apps are related to memory problems. It's critical that you know and report available free memory before launching an application. In all likelihood, the reproducibility of a crashing iPhone app bug is related to low memory conditions. Consequently, a crashing defect may disappear when there's plenty of free memory. In a previous article (Using Memory Sweep for iPhone App Testing) we described a tool that can be used to determine free memory.

2. Provide 'crash reporter' logs with your defect reports. Each time an iPhone application crashes, a .crash file is created on your iPhone. You can retrieve this file when you synch your iPhone with iTunes. Here's a link that describes where those files are stored: iPhone Crash Logs

3. Spy on the app from the console. iPhone apps will report application and system level warnings to the console. You can view these warnings in real-time using Apple's iPhone Configuration Tool. By knowing what's being reported when interacting with an app can help you refine the steps you need to reproduce tricky (and memory related) problems.

4. Test under low memory conditions. This relates to #1 above. You'll be able to tease additional crashing bugs if you force free memory to a very low level, e.g. < 2MB, before proceeding with your tests. One way to do this is to open several Safari windows before you start your testing.

5. Screenshots, screenshots, screenshots. Nothing makes a UI bug stand out for a developer than when you send screenshots. And with the iPhone's built-in screen capture, there's no excuse not to do this.

6. Provide useful defect characterization information. Developers always like to have help in their debugging process, and useful defect characterization helps them narrow down the root cause of a bug. If a crash happens under low memory conditions (see #1 and #4 above), then try it under conditions where there's lots of memory available, e.g. >40MB. If a problem occurs under iPhone OS 2.2.x, then try it under 3.x.

7. Create connectivity problems. If you're testing an iPhone app that depends on internet connectivity, then test for degraded or unavailable connectivity. It's easy to make connectivity unavailable by simply turning on Airplane Mode. To degrade connectivity, especially on Edge or 3G, employ some sort of metallic "shield" on top of your iPhone.

8. Boundary test data input. Wherever an iPhone app asks for text input, you have an opportunity to find a bug. My favorite technique for this is to copy a huge amount of text, then paste it into each text field. You'd be surprised at how this trips up some apps.

Additionally, we’ve been finding that application errors are generating when entering the following characters into text fields: !@#$%^&*()_. (Note: Holding down letters (A, E, I, O, etc) or symbols ($, !, &, etc) on the onscreen keyboard generates a keyboard popup that includes localized and 2-byte characters. These should also be entered into text fields.)

9. Gather up UDIDs (unique device identifiers) early. This is a simple logistics task but seems to be one that becomes critical as the first build approaches. And it's a hassle for the dev team to add new UDIDs and create new provisioning files as each new person wants to install an application during development. Get the UDIDs of all known devices that will be used during testing and set a cut-off date for the addition of any new devices. Check out Find UDID with a single click. You can also connect all your iPhones and iPods touches to your computer and use the iPhone Configuration Tool to collect UDIDs.

10. Employ background applications. But the iPhone can only have one application running at a time, right? That's true for those of us that develop and test applications, but not for Apple. Applications that continue running in the background on the iPhone are Safari, iPod and Mail. And what about reminders and push notifications? These "interrupters" can affect the behavior of an application under test.

Also, since iPhones and iPod touches are devices that users buy primarily to use as a phone or a music player, it’s important to test that an app can gracefully handle situations where the user receives a call or plays music from their music library (iTunes) while the app is running. We’ve seen issues here where apps aren’t smoothly multi-functioning in these areas.

Automated Web Services Testing - An Interview With Matt Krapivner of SmartPilot Software

2010-03-22T08:00:00.000-07:00

As more and more companies face decisions about automating their testing, it's important to understand some of the pitfalls of automation and explore methods that avoid those pitfalls. Likewise, we need to find mature open source testing tools that help bend the test automation curve in the right direction. In previous articles, we've written about model-based and keyword-driven testing, each promising to reduce test automation script maintenance costs while increasing effectiveness (coverage). As I've searched for answers in this areas, I've sought examples of how others have approached test automation as a way of benchmarking good ideas. Recently, I've had the good fortune to meet a practitioner who has pursued new and more effective approaches to test automation. His name is Matt Krapivner and his consultancy business has provided a very impressive test automation framework to a local Web 2.0 company.

We interviewed Matt recently and talked to him about the new approach he's using with his current client and how it's a step up from traditional test automation practices.

Q: Matt, you started out with a Computer Science academic background and since have been working in the area of Quality Engineering. What got you interested in that area versus software development and programming?

Ray, my first quality engineering engagement was actually as a developer on a quality engineering team. I believe some companies call this Software Engineer in Testing, but in a nutshell the team was developing testing software. I started as an individual contributor, but ended up managing the team of 7 engineers locally plus another 7 offshore. I saw QE as an oft-undervalued field with an opportunity to make a great impact on the perception of the company’s products in the marketplace. It is also quite fascinating because at different times you have to wear multiple hats, such as a developer at one time, a tough customer advocate at another, sometimes a product marketing/management hat, and so on. The opportunities for improving the product and making an impact are there, it’s just that many companies often overlook QE and do not get the deserved output from their teams, which in turn impacts their perception of QE overall.

There are two different approaches to testing: Quality Assurance and Quality Engineering. In my opinion, those are very different. One will give you an assessment of the state of the system AFTER that system is put together. The other will help you make sure that your system is ENGINEERED with the highest-quality possible. Let me give your readers an analogy. Say you are building a retaining wall and 2 years after it’s built the wall comes down in a mudslide. Using the traditional QA approach, you would state “the wall fell down because the pressure was too large”. If I was using QE methodologies, I would hire an engineer to build the wall according to the building code to begin with, to make sure that mudslides will not affect it. That’s a fundamental difference, because the further you get in the product lifecycle, the more expensive it will get to fix the mistakes. In my retaining wall example (which I actually just finished building in the backyard), I made a mistake of not putting in the drainage behind it. It cost me more than the initial total construction cost to make an opening in the wall, add the pipe, and close the wall. If I did what was necessary from the beginning, it would only add 5-10% to the construction cost. Instead, by trying to save a few dollars and rushing the construction crew I ended up spending double the amount allocated initially. The same can be said about software development – the longer you wait to find and address an issue, the more you will end up paying to correct the mistakes.

Q: Can you tell us about your first test automation project and what you learned from it?

My first automation engagement was at Palmsource, where I managed a team which was responsible for design and development of an automated test harness for the Palm OS API. In a way, our team played a role of 3rd-party developers early on in the software development lifecyle (SDLC). This was quite fascinating as we got to develop software using a cutting-edge OS before everyone else did. We also received a lot of visibility in the company since as “early adopters” we were able to make informed suggestions for features, and avert a few blunders along the way before they hit the market. As we drove internal adoption, our team started receiving praises from other groups which used our automated tests to validate their changes, small and large. Very quickly, the automated tests were used as a means of certifying customers’ devices for deployment field. This meant that no device could ship with Palm OS or bear the Palm OS logo until it successfully passed our automated tests.

The interesting observation I have made since then in most companies I have worked with, is that very often the testing software ends up being more complex than the actual application under test. There are multiple reasons for that, from the need to cover and verify various edge cases, to just being able to confirm the results. In some cases, especially when you have a hardware/software interaction, this is not trivial at all. One also has to learn where to draw the “line of trust”. Let me explain. Assume that you are testing a PDA application and you need to verify that certain text appeared in a certain position on the screen. While it sounds trivial, verifying the correct result often is anything but. There are typically several solutions. We can call an internal API to query it for the text at certain coordinates, or a text in a specific field. But can you “trust” this API? What if it doesn’t exist to begin with? What if the API itself is buggy? In that case, your test will be meaningless. Another option is to ask your developers to give you a custom “hook” into the display buffer, but most likely this request will not be on top of their priority list. Also, there is a chance it may introduce another point of failure. Yet another option is using a software simulator, if one exists. We actually had to write our own software simulator for testing purposes at one of the companies where I ran the QE group. The point is that there are often many ways of verifying the information in your application under test, and the “easiest” choice may not necessarily be the right one. The challenge then becomes to persuade upper management to invest time and money into building a test system, where an immediate ROI may not necessarily be realized, but rather will result in intangible benefits such as improved product quality, customer satisfaction, etc.

Q: I understand that you have recently provided test automation development services for several local companies. What tools have you used and what was your assessment of the long term maintenance of the frameworks and test scripts you've delivered?

It’s true that there are many test automation systems, and there is no “perfect choice” that would work anywhere. Over the course of the last few years, I have designed a house-grown test automation system; used a hybrid house-grown and commercial tool (Automated QA’s TestComplete); used a commercial tool in combination with open source ones (Squish and Selenium); and finally developed what I believe to be the future of automated testing – a system based on FitNesse.

I will not dive deeply into the benefits and challenges of using commercial tools, there are countless articles and books written on the topic. However, I would like to mention a few points from my experience as a manager/director of QE teams, and as a consultant working on implementing these systems. When selecting an automation tool, one of the main things a company should look at is whether or not a specific tool will work with their application. If it will not, all other benefits become a moot point. Another significant factor is price. In some cases, it will run into tens of thousands of dollars for the tool alone. As a QE director, I had to restrain my teams quite often because the price of the tool was just prohibitive given our budget. Most often though, there is no technical reason to select those tools anyway as there often is a less-expensive alternative. With any commercial tool though, a person responsible for test automation will have to know at least a scripting language, such as Perl or Python. Ideally, that person will be able to program in a higher-level language as well, such as C++ or Java. As a result, your automation system will most likely be built fairly well. The major limiting factor to the tool’s adoption then becomes its scalability, or how quickly can new tests be developed to ensure adequate test coverage. It is, of course, less of a factor for large companies with deep pockets as these companies can most likely just organically grow the test automation team. For startups and smaller companies though, this becomes a major hurdle and automation efforts sometimes may not succeed, or may not provide the desired ROI.

At my last consulting engagement, my client approached me with a similar problem and suggested that I implement a system based on FitNesse. FitNesse is a Wiki-based system where tests are written in natural English language in the form of a Wiki table. Underneath, there is a “fixture” (in Fit-speak) which does the “translation” between English and the underlying tool. That’s right, we can still plug in any test automation tool with FitNesse (well, as long as it exposes some API). With this system, the automated tests now can be written by anybody in the company, including manual testers, product management folks, even a VP of Engineering. It is also easy to answer questions such as “what’s our testing coverage for feature XYZ”, “why did the latest ABC test failed”, and so on. Granted, there still needs to be someone to integrate FitNesse with a testing tool of choice, but this approach provides companies with the much-desired scalability of their testing efforts and a tool with no price tag and ample documentation.

I have integrated FitNesse with Selenium Grid for my client’s web application, and with CruiseControl for continuous build integration. We can now simultaneously test over 30 various targets (different OS/browser combinations), and get results within 15-20 minutes for a test run that spans the entire product in 10 languages. It is easy for anyone now to add to the automated test collection.

Q: For those of us that have tried to bring more automation to testing, we've each run into the problem of "brittle" automated test cases, as well as the need to generate more test cases early on in the development cycle. How has FitNesse solved this problem?

FitNesse is actually a great tool for this, since it allows non-programmers to write test cases which are immediately automated by definition. Most likely, the product managers in this case will not even have to change their logic since they will write the test cases in the form of a “use case”, using more or less the same English language to define a feature. So when the feature is defined, it is put into FItNesse. When the feature is implemented, the test case can be executed to provide immediate response to the team. This becomes even less labor-intensive when a continuous build integration system is utilized. In fact, it is so easy for developers to utilize it to achieve the ever-elusive target of “test-driven development”, that there are no reasons for them NOT to use it…

Q: Given what you've said about FitNesse, particularly the ability for non-programmers to write test cases using a Wiki, how easy is it for someone to produce a test case?

As easy as it is for someone to edit a Wiki page. The tests literally look like:

|user clicks button|Submit|
|page reloads in less than|30|seconds|
|page contains text|Hi Ray|
…
If a person can write a manual test case, they can write a test case in Wiki. If a person can write a use case in Word, they can write a test case in Wiki. You get the idea.

Q: What would you say is the most compelling reason to adopt FitNesse and what challenges are there for organizations to take this path?

The most compelling reason for companies to adopt FitNesse is its ability to provide so much valuable output with so little effort. Granted, it is a relatively new tool and sometimes you will stumble but in my experience the hurdles one will face with FitNesse are not any greater than with any other commercial test automation tool. In fact, the major challenge will likely be organizational in nature as it will require a slight change in the way people operate and contribute to the testing efforts. A FitNesse endeavor is much more likely to be successful if folks from multiple departments contribute.

Q: How extensible is FitNesse, i.e. what if my web service uses a lot of AJAX or components that don't work well with Selenium?

As long as Selenium can handle it, so can FitNesse. Remember that FitNesse operates with “fixtures”, which provide translation between English (the language of FitNesse) and Selenium API.

I chose to use Java API for Selenium, because FitNesse is written in Java and it is technically easier to use Java for the fixture. I was comfortable with Java, so I chose not to change something I did not have to. Any language can be used though, and if your automation tool of choice can only speak Python, for example, then you simply write your fixture in Python.

AJAX is an interesting case though, and the limitations one will face will have nothing to do with FitNesse, but rather with Selenium’s ability to handle AJAX. In my case, we handle AJAX by a combination of standard Selenium APIs, custom Selenium APIs that I wrote specifically for my client’s application, and custom hooks that developers have provided on the server-side. However, all of this is hidden from the tester since their tests are written in plain English. That’s the beauty of FitNesse – it really hides implementation details from a tester who is only concerned with, for example, whether or not a text appeared in the right spot on the page. We provide a custom action for the testers, and they simply do not and should not care how we implement this action behind the scenes.

Q: What would you recommend to those that want to adopt FitNesse for their test automation projects and where can they go for help in getting a FitNesse framework up and running at their company?

First of all, find a test automation tool that you would use without FitNesse. Then, find out if this tool has an API that you can call from FitNesse. Some tools do, and some do not. Some tool companies will be willing to open up their API for you, and some may even be willing to write a sample fixture for you that you would be able to extend. Some may prohibit you from calling their tools from anything besides their own proprietary interface. When in doubt, just ask the tool vendor.

As for FitNesse, there is plenty of easily-digestible documentation that comes along with the distribution of FitNesse. If you search online, you may even find ready-to-use “fixtures” for the most popular open source automation tools, such as Selenium for example. Those fixtures will give you a very good start.

You can reach Matt Krapivner by email at matt@smartpilotsoft.com.

Google Chrome Not Ready For Prime Time Compatibility Testing Yet (Still)

2010-01-12T08:00:00.003-08:00

This Just In: CNET reports that Chrome has overtaken Safari in the latest usage stats (Chrome edges out Safari in browser usage). What exactly does this mean to developers and testers?
I think it's an interesting statistic but not significant (yet). Why? For one thing, Chrome doesn't ship as a standard browser on any system (yet). And until it does, a browser like Safari, which is the browser of choice for each new Mac user (until their friends somehow con them into using FireFox), will remain one of the two browsers (the other being Firefox) you must test on for the Mac OS. For a second reason, read the article below we posted last week.

December 9, 2009 8:00 AM PST
Even though Chrome has gained a 3.93% market share (as compared to Safari at 4.36%, IE at 63.62% and Firefox at 24.72%), it's not ready to be added to compatibility testing matrices yet. There are two reasons for this (besides still being in beta):

It doesn't run on Mac OS 10.4.x (only on Leopard)
Features are not equally supported on all platforms, in particular there's a lack of support for extensions on the Mac

So what's the big deal here? Well for starters, by not supporting the next-to-the-next-to-the-last version of Mac OS X (as opposed to support for the 7+ year old Windows XP) , the adoption by Mac users will be limited to those that don't like Safari and Firefox on their Leopard machines, which might as well be no adoption at all.

Furthermore, because extensions are not supported on the Mac (and Linux) versions of Chrome, testing on Chrome is only worthwhile on Windows. That makes Chrome a single-platform browser bet when it comes to compatibility testing.

With such limited prospects for use on the Mac platform, it's not worth QA testing Chrome on the Mac, and making a case for testing it on Windows alone will be hard to make given the bigger numbers for IE and Firefox.

Chrome doesn't warrant a line on the browser compatibility testing matrix yet.

Understanding The Differences Between Testers And Developers

2010-01-12T08:00:00.002-08:00

Are testers from Venus and developers from Mars? In an ACM article titled An Exploratory Research Study on Interpersonal Conflict between Developers and Testers in Software Development, the authors look at the difference between testers and developers.
As stated in the article, the goal is to better understand the differences between thse two actors in order to produce better software. The authors' call to action is based on two reasons:

Because of a trend toward more agile software development, testers are in contact with developers earlier and more often
Conflict can have negative consequences not only in relation to the end product but also in relation to the job satisfaction of both developers and tester

This paper points to studies that clearly separate testers from developers in terms of their goals: developers seek to maximize efficiency while testers are all about effectiveness. Developers seek to do get their work done with the least effort while testers seek the highest quality. Those are clearly different goals that can easily create conflict if each group sub-optimizes. Unless upper management reconciles these different goals and helps align them to reach the broader objective of the company, failure may ensue.

In addition to understanding the goals of each actor, managers need to understand and work with the differences between testers in terms of what makes them good at what they do. The authors point to a list of the twelve traits that make good testers and developers (Pettichord, 2000):

If these are truly the traits that make for good testers and developers, then you can see why those bug review meetings tend to be a little tense at times, especially when one group or the other has to justify fixing or not fixing a bug. Unless management understands the differences between the respective goals and attributes of developers and testers, and fails to effectively manage the natural conflict, the risk of failing will always be high.

So how do you effectively manage the conflict? I suggest taking the list above and adding another column with notes on how to address the conflict when it arises. For example, what do you say when a tester points out a bug (what's observed) and the developer say it's a feature (how it's designed)? Do you blow it off and agree with the developer or do you take the issue on as something that needs to be resolved either as an issue with the design or the documentation? By thinking through examples for each conflicting pair of attributes, and writing down examples and responses, you'll be able to better manage the conflict.

You can read the entire article here.

How Much Is Your Offshore/Outsource Testing Defect Find/Fix Cycle Costing You?

2010-01-12T08:00:00.001-08:00

Update: Linda G. Hayes, who is the CTO of Worksoft, Inc., and the founder of three software companies including AutoTester, the first PC-based test automation tool, wrote an interesting article on this subject at StickyMinds.com. She wrote about the "promise of the same value proposition" and outlined some key factors for success or failure with offshoring. One of the more interesting, and relevant comments (at least as it pertains to this article), was on "Time to Value". Linda said:

Offshore resources are never a quick fix, especially for testing. A 2005 report from AMR Research found that it took between fourteen months and three years before the offshore testers had sufficient familiarity to be effective in finding the root cause of problems. Others have found that the lack of domain knowledge resulted in spurious defects being reported that actually increased development overhead due to review and response times. In fact, one organization identified that as many as 33 percent of reported issues were traceable to tester error.

There are many reasons to involve external testing services. Sometimes companies need the expertise of an interoperability lab, other times they are attempting to reduce testing staff costs. Whatever the reason, one cost that should be calculated based on a known process is the Find/Fix Cycle Cost for defects. It's important to evaluate these costs not just based on the hourly rate of the offshore/outsource testing service provider, but on the total cost involved in finding and fixing defects. Ultimately, this cost involves all QA and developer resource costs.

The find/fix cycle represents the time required to find and fix one defect and you should know what that cost is to your company. In order to calculate this cost, you need to collect information along the way. I found a nice source for this calculation, as well as a number of other interesting testing topics, at Software Quality Consulting, Inc. Food For Thought. Steve Rakitin does a very nice job in describing this cost in the table below.

Estimate the find/fix cycle Cost
T1 is time required to run a test that finds a potential defect	T1
T2 is time required to report a defect	T2
T3 is time required to determine that this is a defect	T3
T4 is time required to determine if the defect should be fixed	T4
T5 is time required to fix the defect	T5
T6 is time required to include fix in a build	T6
T7 is time required to run regression tests to determine if defect is really fixed and close out defect report	T7
Total time required…	TT = (T1 + … + T7)
LC represents fully loaded labor cost ($/hour) for engineering time (development and QA)	LC
C represents cost to find and fix one defect	*C = LC TT**
TC represents the total cost to find and fix the defects found and fixed in the last release (F1)	*TC = C F1**

As you can see from this table, the "round trip" cost of a defect incorporates 7 different time elements that add up pretty quickly, especially if you have any inherent time delays such as dealing with remote test teams, external test staff turnovers, re-requests for defect characterization information, etc. And the final cost is not just that of the test team, but also incorporates the dev team costs.

My guess would be that T1 and T5 would be the largest time elements, while times T3 and T4, which require more verbal communication, could very well match T1 and T5, depending on the teams involved. It would be interesting, and most likely revealing, to measure these times and factor them into the costs associated with each project's QA and dev team composition.

USB 3.0 Adapters Available Now

2010-01-12T08:00:00.000-08:00

Update: USB 3.0 finally arrives on consumber laptops. CES: USB 3.0 arrives in HP laptop: Yes, it's fast.

November 25, 2009 8:00AM PDT

We may not have readily available computer systems with USB 3.0 yet, but there are USB 3.0 host cards and cables available now for compatibility testing of USB 3.0 supported devices. Several motherboard manufacturers have announced offerings with USB 3.0, but until we see them in mainstream computer systems, it will be hard to justify their use for testing USB 3.0 supported devices. Third party USB 3.0 adapters, on the other hand, are available to consumers and are a perfectly reasonable addition for your compatibility test matrices.

The adapters and cables listed below are available for purchase now:

USB 3.0 Super High Speed 2-Port PCI Express Card for Windows 7 and Vista

ExpressCard USB 3.0 2-Ports Hub

USB 3.0 Hi-Speed PCIe Card 2-Port

2-Port ExpressCard SuperSpeed USB 3.0 Card Adapter

USB 3.0 Super High Speed Cables

Throwing Hard Disks With USB 3.0?

It has been difficult to find shipping USB 3.0 adapters, but this research has not been without some humor. I ran across an interesting article titled "Buffalo wants to be the first ones in throwing hard disks with USB 3.0", and could not resist reading it. Given the title, the following sentence from the article seemed to make sense:

It is clear that he has not too much sense to acquire a disk like that if we have not a compatible groove with USB 3.0 in the computer.

I think anyone that throws hard disks with USB 3.0 probably "has not too much sense" to begin with.

Things You Should Know About Windows 7 - God Mode

2010-01-11T08:00:00.000-08:00

Take a tip from Bruce Almighty - use Windows 7 "God Mode" wisely. Windows 7God Mode provide shortcuts to various Windows 7 settings. God Mode is a folder naming trick and it's easy to do.

As the video below will show, you just create a new folder and rename it using one of the God Mode strings.

Special thanks to Marius Oiaga for providing this list of God Mode strings in the article Windows 7 and Windows 8 GodModes, the Complete List:

Action Center Microsoft.ActionCenter (Windows 7 and later only) {BB64F8A7-BEE7-4E1A-AB8D-7D8273F7FDB6}
Backup and Restore Microsoft.BackupAndRestore (Windows 7 and later only) {B98A2BEA-7D42-4558-8BD1-832F41BAC6FD}
Biometric Devices Microsoft.BiometricDevices (Windows 7 and later only) {0142e4d0-fb7a-11dc-ba4a-000ffe7ab428}
Credential Manager Microsoft.CredentialManager (Windows 7 and later only) {1206F5F1-0569-412C-8FEC-3204630DFB70}
Default Location Microsoft.DefaultLocation (Windows 7 and later only) {00C6D95F-329C-409a-81D7-C46C66EA7F33}
Desktop Gadgets Microsoft.DesktopGadgets (Windows 7 and later only) {37efd44d-ef8d-41b1-940d-96973a50e9e0}
Devices and Printers Microsoft.DevicesAndPrinters (Windows 7 and later only) {A8A91A66-3A7D-4424-8D24-04E180695C7A}
Display Microsoft.Display (Windows 7 and later only) {C555438B-3C23-4769-A71F-B6D3D9B6053A}
Getting Started Microsoft.GettingStarted (Windows 7 and later only) {CB1B7F8C-C50A-4176-B604-9E24DEE8D4D1}
HomeGroup Microsoft.HomeGroup (Windows 7 and later only) {67CA7650-96E6-4FDD-BB43-A8E774F73A57}
Infrared Microsoft.Infrared (Windows 7 and later only) {A0275511-0E86-4ECA-97C2-ECD8F1221D08}
Location and Other Sensors Microsoft.LocationAndOtherSensors (Windows 7 and later only) {E9950154-C418-419e-A90A-20C5287AE24B}
Notification Area Icons Microsoft.NotificationAreaIcons (Windows 7 and later only) {05d7b0f4-2121-4eff-bf6b-ed3f69b894d9}
Pen and Touch Microsoft.PenAndTouch (Windows 7 and later only) {F82DF8F7-8B9F-442E-A48C-818EA735FF9B}
Phone and Modem Microsoft.PhoneAndModem (Windows 7 and later only) {40419485-C444-4567-851A-2DD7BFA1684D}
Recovery Microsoft.Recovery (Windows 7 and later only) {9FE63AFD-59CF-4419-9775-ABCC3849F861}
Region and Language Microsoft.RegionAndLanguage (Windows 7 and later only) {62D8ED13-C9D0-4CE8-A914-47DD628FB1B0}
RemoteApp and Desktop Connections Microsoft.RemoteAppAndDesktopConnections (Windows 7 and later only) {241D7C96-F8BF-4F85-B01F-E2B043341A4B}
Sound Microsoft.Sound (Windows 7 and later only) {F2DDFC82-8F12-4CDD-B7DC-D4FE1425AA4D}
Speech Recognition Microsoft.SpeechRecognition (Windows 7 and later only) {58E3C745-D971-4081-9034-86E34B30836A}
Troubleshooting Microsoft.Troubleshooting (Windows 7 and later only) {C58C4893-3BE0-4B45-ABB5-A63E4B8C8651}

Technology Advisory: FireFox 3.5.7 Released

2010-01-08T08:00:00.000-08:00

Firefox 3.5.7 fixes the following issues:

Fixed a common stability issue.
Fixed a problem with how updates were being presented to users.

FireFox 3.6 beta is also available. This soon-to-be-released version is is built on Mozilla's Gecko 1.9.2 web rendering platform, which has been under development for several months and contains many improvements for web developers, Add-on developers and users. This version is also faster and more responsive than previous versions, and has been optimized to run on small device operating systems such as Windows CE and Maemo.

Using Selenium And BrowserMob For Load Testing

2010-01-07T08:00:00.000-08:00

Update: Who doesn't like FREE! The folks at BrowserMob now give you an instant free test. No signup necessary. Uses real browsers. See screenshots, DNS lookup times, time-to-first-byte, and more! It's worth checking out. Click to get your Free Instant Test.

November 30,, 2009 8:00AM PDT
We've written several articles on using Selenium for web testing, including a HOWTO series on "Automated Cross-Browser Testing With Selenium". We plan to expand beyond these functional testing articles by looking at how Selenium can be used for other types of testing. Today's video on BrowserMob shows how you can use Selenium for low-cost load testing service that uses real web browsers.

The video below takes you through recording a Selenium test case using the Selenium IDE and then running it using BrowserMob. I'm particularly interested in how easily you can set up your load tests and the great information provided in their dashboard.

Note: I've not figured out the pricing yet (seems a bit complicated). They claim that you can "Simulate 2,000 VUs for only $150" but if you look at their pricing table, it's difficult to immediately see that sort of pricing.

Latest DeployStudio Tips, Techniques And News

2010-01-06T08:00:00.000-08:00

If you're a DeployStudio fan, or are looking to learn what's behind the growing popularity of this tool, then be sure to attend the upcoming DeployStudio Seminar described in today's article.
LinkedIn Event: If you're a Mac Systems Administrator with more work on your plate than time to complete it you can't miss this seminar. Oliver Block of Skeleton Key will present a demonstration of DeployStudio, which enables you to streamline the deployment of multiple machines. This and similar technology has saved our customers hundreds of hours. DeployStudio Seminar

Here's a nice PDF that covers a lot of the basics of installing and using DeployStudio. DeployStudio Tips & Tricks

This blog has a number of tips and techniques for using DeployStudio. Creating a Master DeployStudio Image

Latest DeployStudio Server with new features that allow you to define an authentication domain when configuring a workflow Active Directory binding task; provides packages management (packages sets and basic ACLs); an option to disable wireless support from DeployStudio NetBoot / external drive systems; and, an option to force UDIF disk images to be converted a second time to the same format before being scanned (multicast configuration panel). DeployStudio Server 1.0rc16

A recent HOWTO article on using one master Deploy Studio server and host the images at different sites. Use one Deploy Studio server with images on many servers

Would You Like To Browsersize That?

2010-01-05T08:00:00.000-08:00

Ever wonder what appears on real users' browsers when they visit your website? If you want a certain percentage of users, let's 90%, to all see the same thing, then you're going to need a tool that tells you in what percentile your web pages fall.

The Google Browser Size is just that tool and is a must have for anyone creating websites. It provides a visualization of browser window sizes, as a percentage of people who visit Google, superimposed on your website. You just enter your website URL, and you get to see contours that match up with different precentages. For example, the "90%" contour means that 90% of people visiting Google have their browser window open to at least this size or larger. Here's how they position this tool at their website:

This is useful for ensuring that important parts of a page's user interface are visible by a wide audience. On the example page that you see when you first visit this site, there is a "donate now" button which falls within the 80% contour, meaning that 20% of users cannot see this button when they first visit the page. 20% is a significant number; knowing this fact would encourage the designer to move the button much higher in the page so it can be seen without scrolling.

And here's what this blog looks like using the Google Browser Size tool:

To view your own Web site with this same visualization overlaid on it, go to http://browsersize.googlelabs.com/, type your URL into the "Enter URL here" textbox at the top of the window and click Go.

Crawljax For Testing AJAX Applications

2010-01-04T08:00:00.000-08:00

Wouldn't it be nice to finally have a tool that solves all the problems we encounter with automated testing? So often we find test teams going down the path of automating individual test cases knowing that the cost to maintain each of those individual test cases will grow. And for those that have looked at advanced automated testing methodologies, such as model-based and keyword driven testing, they know there's a better way, yet they proceed with developing large suites of automated test cases that end up creating a maintenance nightmare later on when things changes.

In 2010, we will continue to work with our clients to use open source frameworks that solve some of the problems inherent with developing automated tests. In particular, we recommend FitNesse for anyone testing web applications and TestOptimal for those interested in using model-based testing. For those of you that have already invested in using Selenium, you should consider FitNesse as THE framework for delivering a more effective automated test development and execution capability.

As for that new tool that will solve all our problems? We haven't found it yet, but we will keep looking. And this year, as we develop more FitNesse automated test solutions for our clients, we'll be keeping an eye on new tools like Crawljax. Crawljax is yet another open source tool that has received great attention through Google's Test Autotmation Conferences. It is a tool for crawling any AJAX/GWT application (GWT = Google Web Toolkit). It uses WebDriver to navigate through the different states of a web application. It creates a state-flow graph of the dynamic DOM states and the transitions between them. This inferred state-flow graph forms a very powerful base for many types of automated web testing. With plugins and invariants Crawljax can be used to perform various automated tests. For example: security testing, regression testing, accessibility testing, performance testing, cross-browser testing.

And as with most new open source tools, it takes time to create polished marketing and support material - the developers wisely spend their time making the tool better and better. The video below has a lot of information and detail on Crawljax. The audio is not great, but if you're motivated to see if Crawljax can solve some of your automated testing problems, then this video is worth watching. And you'll probably want to supplement what you watch with more info at this link.

Happy Holidays!

2009-12-23T08:50:00.000-08:00

We'd like to take a moment to thank all of our readers for your support in 2009. We'll be back with new posts on the world of testing beginning January 4th, 2010.

Wishing you all a safe holiday and happy New Year from everyone here at Testlabs.com.

Testing on Agile Projects

2009-12-22T08:00:00.000-08:00

Do agile projects become the brunt of cubicle Dilbert jokes? Scott Ambler confronts Scott Adams.

Agile projects are both interesting and challenging, especially for testers. Too often these projects and their processes are misunderstood or misapplied. What's most uncomfortable to testers is what appears to be a complete abandonment of any sort of serial testing flow.Moreover, the lack of upfront specifications for all features and functionality seems to be a complete affront to not only testing best practices but also common sense. Suffice it to say that a lot is misunderstood about Agile testing.

Unfortunately this misunderstanding of what can be an effective development and testing process ends up being the brunt of cubicle jokes.

Now I wouldn't say that testers assigned to agile projects see things as comically as Dilbert, but it can be confusing for testers if the development processes don't include them. For this very reason, and based on some past experience, I was happy to find an article, titled Agile Testing Strategies, that spoke to agile testing specifically. Of all the articles I've read on agile development, this one made the most sense to me as a tester.

The author, Scott Ambler, does a great job of relating agile practices to testing and breaks down this subject into 5 parts:

This is a good read and reference document for any tester trying to make sense of their agile development process. I believe that if you spend the time needed to begin to understand the agile process and use Scott's advice and testing process overlays, you'll be able to not only understand your role as an agile tester, but graduate to driving the agile testing strategies for your projects.

University Of Utah Mac Managers Share Their DeployStudio Experience

2009-12-21T08:00:00.000-08:00

DeployStudio University! Mac Managers from the University of Utah share their DeployStudio secrets on iTunes in a presentation from their monthly Mac Managers meeting. Each month they upload their presentations to iTunes U. The video below is a comprehensive presentation on DeployStudio they gave in June 2009:

All of the slides shown in this video presentation are available as a PDF document here. Additionally, all of their presentations are available on iTunes U. Here's the link for all the Mac Manager Meetings presentations.

Technology Advisory: FireFox 3.5.6 Released

2009-12-18T10:18:00.000-08:00

Mozilla has released Firefox 3.5.6, which fixes the following issues:

Several security issues.
Fixed several stability issues.

The update can be downloaded at Mozilla’s site.

You may also be interested in the Firefox 3.5.5 release notes for a list of changes in the previous version.

Using Sauce OnDemand To Run Automated Cross-Browser Tests In The Cloud

2009-12-18T08:00:00.000-08:00

Update: What has the original creator of Selenium (Jason Huggins) been cooking up lately? Hint: it has to do with Sauce.

Jason and his colleagues at Sauce Labs have been very busy since last we wrote about their product offerings. They've renamed their cloud-hosted service to Sauce OnDemand (formerly called "SauceRC") and announced a Sauce Labs enhanced version of Selenium RC (versions for both PC and Mac) which they are calling Sauce RC. To find out more about Sauce RC, here's a link to a screencast video that describes the product and how to get started using it.

August 21, 2009 8:00AM PDT

In May we wrote an article about a service we discovered called Sauce OnDemand (formerly called "SauceRC") from Sauce Labs (Selenium RC In The Cloud - SauceRC). We found this service interesting because it allows you to do browser compatibility testing without the hassle (let alone cost and energy) of configuring local machines and installing browsers. Instead, you push your tests into the Sauce Labs cloud. Simply create test scripts using the Selenium IDE plug-in for FireFox, designate the browser you want to test on (see code below) and let Sauce OnDemand do all the work. Repeat for each browser you want to test on, then go to the Sauce Labs website to retrieve the results. And the cool part is that you get a movie, as shown in the graphic above, of an actual browser session running your tests. Talk about convenience!

It's best if you know how to use Selenium RC before you start using Sauce OnDemand, but, according to the developer, some customers have successfully used the service without previous use of Selenium RC. In our case, we had already written scripts for Selenium RC, so it was simply a matter of changing 1 line in our script using the code below:

DefaultSelenium selenium = new DefaultSelenium(
                "saucelabs.com",
                4444,
                "{\"username\": \"rvizzone\"," +
                "\"access-key\": \"xxxxxxxxxxxxxxxxxxx\"," +
                "\"os\": \"Windows 2003\"," +
                "\"browser\": \"firefox\"," +
                "\"browser-version\": \"3.\"}",
                "http://www.google.com/");

Note the browser and version designated in red.

After we made this change, our scripts ran as they had using Selenium RC on our localhost, with the actually processing taking place at Sauce Labs, and the results, both a movie and log, available to use anytime through links accessible through our Sauce Labs account.

Here's a set of high-level HOWTO steps for using Sauce OnDemand:

1. Record and edit your test script with the Selenium IDE plug-in for Firefox.
2. Export your script and make sure it runs with Selenium RC on your localhost.
3. Make the 1 line change to your script that includes your user name, access key, etc., as shown above.
4. Now run the script again (which will now be directed to the Sauce OnDemand server).
5. Retrieve results from with your Sauce Labs account

If you're looking for your investment in Selenium to pay off even more, I suggest you try Sauce OnDemand. Sauce Labs offers both a subscription plan and pre-paid minutes. And if you sign up during their public beta program, the minutes you buy will be doubled, e.g. buy 250 minutes for $10 and you'll get 250 minutes free.

Automated Testing Institute

2009-12-15T08:00:00.000-08:00

While doing some article research this week I came across a website on automated testing called the Automated Testing Institute. Here's a synopsis of the content:

The ATI Online Reference is the web’s foremost comprehensive online resource for obtaining, exchanging and certifying industry standard test automation knowledge, information, and techniques. With key elements such as automation articles, automation specific news, searchable tools and book, discussion forums, free video tutorials, other automation specific videos, and more, this free, member-based site is the most comprehensive resource for helping you excel at test automation.

From my initial review of the main sections of this website, I was impressed with the variety of articles, video, tips and techniques offered. There's a wide range of articles, from those that appeal to the beginner (How Do I Get Started With Test Automation?), to more advanced topics on ROI, frameworks, etc. Check it out here.

Selenium 2.0 and Beyond!

2009-12-14T08:00:00.000-08:00

For those of you that are fans of Selenium, you'll probably want to hear what Simon Stewart (Google) and Jason Huggins (Sauce Labs) have to say about Selenium 2.0 and beyond.
In particular, the merging of efforts by Huggins (Selenium Core) and Stewart (WebDriver) is very important to understand where Selenium is headed.

Using The National Vulnerability Database During Security Testing

2009-12-11T08:00:00.000-08:00

The National Vulnerability Database (NVD) is a comprehensive cyber security vulnerability database
that integrates all publicly available U.S. Government vulnerability resources and provides references to industry resources. NVD's mission involves warning the public about vulnerabilities in computer systems. NVD provides this information using a search engine while integrating all publicly available U.S. government vulnerability resources. All of this information is given away for free with no licensing restrictions through XML and RSS feeds.

The NVD is easy to use and should be used during security testing. In fact, depending on which security tool you use, the results will most likely reference items in the NVD. I tried searching the NVD with some familiar software we use and the results were impressive. I did a search on the open source defect tracking system Bugzilla, and found numerous security flaws. I did this search by first going to this URL:

http://web.nvd.nist.gov/view/vuln/search?execution=e1s1

And then entering "bugzilla". The result was a list of 88 vulnerabilities that show up in an easy to read list with links that give you more specific details on each item:

Here's a closer look at some of the results from the list above:

CVE-2009-1213
Summary: Cross-site request forgery (CSRF) vulnerability in attachment.cgi in Bugzilla 3.2 before 3.2.3, 3.3 before 3.3.4, and earlier versions allows remote attackers to hijack the authentication of arbitrary users for requests that use attachment editing.
Published: 04/01/2009
CVSS Severity: 6.8 (MEDIUM)

CVE-2008-6098
Summary: Bugzilla 3.2 before 3.2 RC2, 3.0 before 3.0.6, 2.22 before 2.22.6, 2.20 before 2.20.7, and other versions after 2.17.4 allows remote authenticated users to bypass moderation to approve and disapprove quips via a direct request to quips.cgi with the action parameter set to "approve."
Published: 02/09/2009
CVSS Severity: 4.0 (MEDIUM)

Searching for vulnerabilities is just one feature of the NVD. There are number of checklists, statistics and other information available at the main NVD URL:

http://nvd.nist.gov/home.cfm

Add These Website Performance Test Tools To Your Toolkit

2009-12-10T08:00:00.000-08:00

I've listed some performance testing resource links that are worth looking at if you do any sort of website performance testing.
One of the more difficult tasks when conducting load and stress testing is identifying bottlenecks and suggesting areas on which developers should concentrate. Often this type of testing uses tools that are tuned to put a load on a website (e.g. JMeter) and consequently generates a lot of test data, primarily because these tools simulate lots of users. It helps to have another perspective to corroborate the data from your load tests. This is where these website performance tools may help:

Speed Tracer. Speed Tracer is a Google Chrome extension that helps you identify and fix performance problems in your web applications. It visualizes metrics that are taken from low level instrumentation points inside of the browser and analyzes them as your application runs. Using Speed Tracer you are able to get a better picture of where time is being spent in your application.

YSlow. This tool analyzes web pages and why they're slow based on Yahoo!'s rules for high performance web sites. YSlow uses Yahoo!'s Smush.it service.

Page Speed. An open-source Firefox/Firebug Add-on. Webmasters and web developers can use Page Speed to evaluate the performance of their web pages and to get suggestions on how to improve them.

Google Web Toolkit (with Speed Tracer). Google Web Toolkit (GWT) is a development toolkit for building and optimizing complex browser-based applications. GWT is used by many products at Google, including Google Wave and Google AdWords. It's open source, completely free, and used by thousands of developers around the world.

Web Performance Best Practices. When you profile a web page with Page Speed, it evaluates the page's conformance to a number of different rules. These rules are general front-end best practices you can apply at any stage of web development. We provide documentation of each of the rules here, so whether or not you run the Page Speed tool — maybe you're just developing a brand new site and aren't ready to test it — you can refer to these pages at any time. We give you specific tips and suggestions for how you can best implement the rules and incorporate them into your development process.

Tips and Techniques for Using Apache JMeter for Load Testing

2009-12-08T08:00:00.000-08:00

Apache JMeter is a great tool for load testing and we use it for LAN-based products as well as web sites and hosted services. And lately we've used it in conjunction with Amazon's Elastic Cloud Compute (EC2) service. Along the way, we've collected a number of tips and techniques that are sometime difficult to find in the documentation.

Here's a list of changes that we've found to be necessary each time we use jmeter:

Change Memory Settings
The default memory settings are too low need to be increased to the maximum memory available. To do this:

1. Navigate to jmeter/bin.
2. Edit jmeter (Linux command-line: "pico jmeter", Windows: open jmeter.bat with wordpad)
3. Scroll to the line that reads: HEAP="-Xms256m -Xmx256"
4. Change the numbers to meet your system specs (first number is the minimum, second maximum). For example: HEAP="-Xms1024m -Xmx1024"
5. Save changes.

Retries & Timeouts
If you are doing load testing with jmeter, inevitably you'll place enough of a load on your target server that it may have trouble responding. In order to give the server a chance to respond you may want to change the retries and timeout values. To do this, edit the jmeter.properties file for retries = 3 and timeout = 120. Be sure to delete the #s at the start of the lines or they will remain commented out.
--Note: The generic HTTP Request does NOT support these features, and the HTTP Request HTTP Client sampler type must be used.
--Note: The HTTP Request HTTP Client sampler does NOT support automatic redirect so be sure to deselect this (its checked by default) when creating the JMeter tests.

Ports (Windows only)
If you've ever seen this error message when running jmeter "JMeter Exception: java.net.BindException: Address already in use: connect" you've run out of ports. To avoid running out of available ports on Windows (and thus seeing a Address Already In Use error), there's a great explanation and solution posted on the twit88.com blog at this article.

Vint Cerf - The Internet Today

2009-12-07T08:00:00.000-08:00

In today's video, Vint Cerf, often referred to as "the Father of the Internet", gives a lecture at Singularity University (www.singularityu.org) that first provides a history of the internet,
then covers the state of the Internet today, and what issues are arising as it continues to evolve. He talks about technology platforms - IPv6, cloud computing, etc. - that are sure influence how we evolve our test labs.