While the article addresses the importance of end-user experience and the challenges with ensuring performance, it fails to provide a promised solution. The writer advocates applying “new approaches to application performance management (APM), including focusing on end-user transaction performance, consolidating all application delivery chain variables in a ‘single pane of glass’ approach and monitoring all applications 24×7.” However, this is not a truly new approach; this is the traditional approach – monitor and make sure transaction response times adhere to expected service or performance levels. Monitoring remains a critical element in any comprehensive application performance management model; however it does not deliver on the author’s premise that “modern applications require modern performance management approaches.”

The modern approach, increasingly adopted by enterprises worldwide, is to shift performance management left in the software development lifecycle, testing earlier and often, before deployment, when it is more cost-effective to find and resolve issues (and when user experience is not at risk). In addition, modern applications are increasingly distributed and reliant on external services and multiple, dynamic network connections. These dependencies, particularly mobile and cloud network conditions, are beyond an enterprise’s control, yet they are paramount to end-user experience. As a result, modern enterprises are accounting for these variables in their testing by incorporating services and network virtualization into the test environment to ensure that testing is conducted in as accurate a pre-production environment as possible, thus delivering the most reliable and predictive results possible.

As an example, one of the world’s leading providers of digital television entertainment services relies on tens of thousands of employees in broadcast centers around the globe, satellites in space, and hundreds of technology patents to deliver programming to over 30 million people in the western hemisphere. Before rolling out a new CRM application to their call centers, the company needed to certify the system for up to 3,000 users and ensure performance for all call centers. To ensure the validity of their tests, they knew they had to account for load, external services and the network conditions that affect performance for each call center – including bandwidth, latency, jitter and packet loss. As a result of using Shunra NV and services virtualization to capture production network conditions and dependencies from remote call centers and emulate those conditions in their test environment, the entertainment company was able to accurately test and optimize the response times of their CRM application, giving field technicians faster access to the system and allowing call center reps to take in an average of three additional new customers per day.

This modern approach to performance management, with consideration for external services and the real world conditions that affect performance, allows the entertainment company to maintain its position as an organization focused on delivering exceptional end-user experience and to deploy new systems with confidence, increasing revenue and protecting corporate brand.

With modern applications, it is essential to account for their reliance on external services and the variable network conditions that users experience as they move between different network types and different devices. With services and network virtualization, organizations can access real-world network conditions and services and emulate those in the test environment. Unit, functional, regression, integration and performance testing should be conducted in an environment that mirrors the production network, making test results more accurate and reliable. Shifting left and accounting for the most critical conditions that affect end-user experience is a proactive and modern approach.

For the first time, Rational Quality Manager users can capture and virtualize production network conditions in the test environment in order to improve application performance

According to the World Quality Report released in September 2012, two-thirds of enterprises lack the ability to accurately test applications prior to deployment. The Killer Apps Study 2012, a pan-European research study, found that 82 percent of survey respondents had experienced application performance problems, with 74 percent reporting impacts on critical business applications.

As organizations continue to struggle with application quality, the impact of poor performance negatively affects revenue, productivity, and brand. Julie Craig, Research Director at Enterprise Management Associates, addresses the issue of application performance and writes that the network is, “The single most important instrumentation point for tracking and monitoring applications.”

Until now, the production network has been inaccessible to software testers and developers who operate in test labs isolated behind corporate firewalls. That means that traditional testing efforts do not include dynamic, real-world network constraints like bandwidth, latency, jitter, and packet loss, despite the fact that these network conditions are integral to understanding end user experience.

For the first time, with the Shunra NV integration for IBM Rational Quality Manager 8.5, application testers and developers who use Quality Manager can capture and incorporate production network conditions into their test environment. Rational test environments can now account for the variability of the networks over which applications, their dependencies, and their end users communicate. This makes previously inaccessible network constraints available in the test lab, so application testing can be conducted against actual production network conditions.

“Network conditions have a profound impact on application performance,” said Gary Jackson, Shunra CEO. “But too often, they are unaccounted for in testing. Shunra NV’s integration with Rational Quality Manager allows organizations to identify apps that will perform poorly in production due to network-associated issues. They can then isolate and fix performance bottlenecks before the app is released, saving costly post-deployment remediation, and negative user sentiment.”

75 of the Fortune 100 companies currently use Shunra’s network virtualization capabilities in their test environment. A leading automobile manufacturer recently reported saving over $500,000 annually as a result of using Shunra NV to improve the accuracy of their testing which resulted in fewer remediation issues. In addition, that same company reported better testing led to incremental revenue opportunities in excess of $1.5 million.

IBM Rational users can now experience similar results due to Shunra Software’s “Ready for IBM Rational software V7.9.1” technical validation for its Shunra NV integration for Rational Quality Manager 8.5. The plug-in meets the requirements for both the Software Delivery Platform – Software Delivery Lifecycle and Jazz levels of the program. In addition, the Shunra NV integration qualified for the Ready for IBM Rational Best Practices indicator.

The Shunra NV Adapter for Rational Quality Manager 8.5 is an add-on component available from Shunra that empowers Shunra NV users with the ability to leverage Shunra NV capabilities from within the Rational test environment, including:

• Network discovery, capture, and virtualization with Shunra NetworkCatcher;
• A library of millions of mobile and broadband network performance profiles to easily compare app performance across locations and networks;
• Location- and network-aware response time reporting; and
• Automated and customized performance optimization suggestions including mobile specific optimizations.

Shunra NV at IBM Innovate

Shunra will be demonstrating its Shunra NV adapter at IBM Innovate from June 2-6, 2013 in Orlando, Florida. Shunra will be exhibiting at Booth T10 throughout the event and will be presenting testing best practices at “Session QM-1911: Shift Left: Be Inventive at Accelerating Quality and Test Management” on June 5 at 8:30am.

For more information on Shunra NV for Rational Quality Manager 8.5, please visit: http://www.shunra.com/NVRQMAdapter or http://www-304.ibm.com/partnerworld/gsd/solutiondetails.do?solution=49011&lc=en&stateCd=P&tab=2.

On this interview-style, recorded webcast, hear from two APM and network virtualization experts on how you can extend the value of your APM investments:

• find issues before customers are impacted
• reduce unexpected downtime or slowdowns
• identify, isolate, and resolve problems faster
• improve application performance

Watch the webcast: Extending the Value of APM

Dave Berg and Marty Brandwin answer:

• Why isn’t APM delivering on the promise of better performing applications?
• Even though APM helps reduce unexpected downtime and resolve problems, performance is not improving. Why not?
• Why would I want to manage performance differently when I’ve been relying on APM for so long?
• Where does APM fit in the complexity of today’s application architectures and supporting infrastructure?
• What tools and techniques are needed throughout the application life cycle in addition to APM?
• How does testing help APM?

Marty Brandwin (left) and Dave Berg (right)

Featured presenters:

Dave Berg is the Vice President of Product Strategy at Shunra. He is responsible for the management of all Shunra’s products and technical partner relationships, and he leads our product management team. With more than fifteen years of experience in performance engineering, development, automation, vendor management, and professional services, Dave has extensive experience with distributed systems, real-world scenario testing, and complex root cause analysis. He is regarded as an expert in protocol design, mobile performance, and software performance engineering. Dave holds a bachelor’s degree in Computer Science and Discrete Mathematics from the University of Michigan, as well as holds certifications in product management and Agile product management. Dave has worked with dozens of Fortune 100 companies on their performance management strategies. Click his picture to connect with him on LinkedIn.

Marty Brandwin is Shunra’s Vice President of Marketing and a veteran software marketing expert with more than 16 years of experience conceptualizing and implementing integrated marketing programs targeting multiple vertical markets and Fortune 1000 companies. Marty has developed and managed demand generation and social media marketing programs that have garnered national recognition and earned him nomination to the CMO Council. Marty worked in the APM market as it started to take off and understands the promise of monitoring as a component of delivering better performing applications. Click his picture to connect with him on LinkedIn.

One thing each of these projects, or concerns, have in common is the amount of planning that should go into them before any changes are made. On this webinar, we will discuss planning as a critical and invaluable step in the process.

On a recorded webcast, we reveal five risks of data center relocations, and what you can do to avoid them. Watch the webcast: Five Risks of DCRs and How to Avoid Them.

Webcast Summary:

Misaligned expectations

Often, a move or consolidation is driven by a desire to reduce Operating Expenses or upgrade technologies like new, more powerful servers, better power, or better HVAC capabilities. However, better doesn’t always mean faster. The data center represents only one factor contributing to application performance. You have to think about what happens outside of the data center, and how network connections from the end user to the Cloud are impacting user experience. In fact, applications normally exhibit WORSE performance after a DCR, despite upgrades in hardware, and, if you’re not prepared for it, it can be a career-affecting event.

Shunra summary of impact from 150 data center moves

Not showing measurable value

We talk about planning, and that’s because a data center move or consolidation can be high risk. But that risk comes with reward. There is a motivation behind the move; there is some business value that is expected to be gained, and that business value should be communicated. Enterprises need to define the expected benefits and communicate those benefits to all IT constituents so they have a vested interest in the success of the move and understand how business will be impacted. If you don’t know the value, then you can’t understand the true cost of failure.

Not understanding topology and networks

On this webinar, our feature presenters talk about aligning expectations and understanding intended value. There is a complexity to today’s applications that must further complicate any changes made to a data center. You look at how apps have evolved from client-server, to three-tier and to distributed architectures with dozens of dependencies or services outside of the corporate firewall; not to mention the additional communication layers presented by mobile and Cloud. So how does this complexity affect data center relocations and consolidations? Listen to this webinar to hear Todd DeCapua, one of our feature presenters, talk about:

• Application topology/fingerprint in relation to network
• How will apps behave after move, and planning move groups and testing move groups
• Infrastructure supporting apps
• Transaction stitching

Skipping tests in areas of impact, risk, and failure

A test environment needs to accurately reflect the real-world. The test environment is only yielding accurate test results if the environment includes reliable virtualized users, networks and services. Todd and Marty Brandwin, another feature presenter, talk about how you don’t need to virtualize every piece of equipment to test properly. This is a very common misconception that leads many well-meaning people to avoid testing completely. What you really need is to virtualize the BEHAVIOR of the network, from client to server, from server to database, from application to third-party services. So the good news there is that we have tools today that can do this very easily and automatically.

Lack of DCR experience

There’s a lot to consider – what to virtualize, how to virtualize, how to test with those conditions and what to test. The remarkable thing is that many of our customers call us after disaster strikes. If your applications fail, is the DCR still a success? If you move to a “green” facility and reduce your carbon footprint but fail to deliver product to your customers on time, is your executive team still happy? If your failure makes news on social media or, worse, in the press, can you still feel good that your servers now have faster CPUs? Every customer is different. Every DCR is different. 2013 is a major year for DCRs, and our experience at Shunra teaches us that most DCR project teams only do one large DCR in their career, which is why it is sometimes referred to as a “career-limiting project” or a “career-affecting project.” The lack of experience can contribute to a DCR failure. By partnering with an expert, you can reduce that risk. On average, each one of our engineers has conducted a DCR 16.2 times. Through our experience, we have developed a proven four-step DCR methodology that avoids costly failures. Read more about the methodology.

It is no secret that Development and QA managers are looking for solutions that will help mitigate the risks involved with endeavors like mobile app deployments. Unfortunately, few solutions are rising to the challenge. Solutions do exist today that allow automated testing on multiple devices, leveraging virtualized infrastructure and services for testing application performance limits. These solutions are powerful and can be very helpful raising release quality; however, they tend to overlook a single, critical aspect that heavily influences end user experience – the network. This blind spot in traditional testing processes threatens application success and, as a result, the business value of the application.

With mobile, the network is paramount. As developers and testers, we must ask:
• How will the application behave when there is low signal strength?
• How will the application perform when switching between one cell tower and the next?
• How will variability in networks and user locations impact application performance for the end users and backend infrastructure?

While seemingly daunting, this challenge does have a solution. It is possible to test how users will experience an application on different networks. In simple terms, the solution is to use tools that impair the network traffic between application components in a manner that resembles the way the application is used in the real world.

The real problem is awareness of the availability of a solution to the challenges listed above, and ensuring developers and QA are actually taking advantage of the mobile app performance testing capabilities they need. Traditionally, the network and its impact on user experience has not been well understood or even of primary concern for developers and QA. Mobility (and Cloud) is changing that.

Often developers and testers take the approach that it is enough to ensure the mobile application can communicate with the application backend servers – “after all performance and load testing will find the breaking points”. This approach shifts the resolution of network related issues to the right in the development cycle, where the cost of failure is much greater. With this approach, best case is that network related problems will be found at the performance testing stage. In the worst case scenario, problems will be found by end users, thus impacting business.

As development and testing professionals, we must strive to find issues with applications as soon as possible and fix them before creating a negative end user experience – in other words, shift left. The last thing any of us need is a design failure that is only found after the application development is finished, forcing unnecessary remediation efforts and often re-design that could have been avoided.

Instead of relying on developers and QA to remember to run their tests over varying network conditions, consider making network virtualization part of an enterprise testing infrastructure. If even a few testing machines behave like they are deployed in different parts of the world with different carriers, the network impact of those various locations will be visible from the first test.

From Shunra’s perspective, the steps to enable accurate, reliably predictive test environments follow:

1. Measure or look up the network characteristics of the locations of most concern.
2. Configure one machine or VM to work as a router and install a network virtualization for software testing tool on it. Ideally, the network virtualization solutionallows configuring multiple network locations in a single execution. If all testing is done on the same virtual machine host, one VM can be configured to act as a router.
3. Configure all testing machines to route traffic through the above router machine. On a VM host machine, configure the virtualized network using the VM host tools without needing to load and configure each VM individually. In some cases, this will not even require modifying the routing tables. This settings will persist across VM snapshots and is a lot easier to maintain when network topology changes.
4. Configure the network virtualization tool on the router machine to impair network packets coming from all the connected machines – each machine uses its own location information captured in step one – and run it continuously.
5. Ask your QA and Developers to run tests using these test machines.
6. As tests run, network packets are routed through the router machine and are impacted by the network conditions already defined. Note that the person running the tests does not need to have any control on what network condition each machine gets As that can be pre-configured; the tester does not even require access to the emulation.
7. When using real devices, configure them to connect to a wireless network that routes traffic through the same router machine or use a proxy. Network emulation can be modified so each device will operate under different virtualized network conditions (3G, 4G, Edge, etc).

In conclusion, it takes just a few simple steps to enable a very robust testing environment that incorporates virtualized production network conditions for all tests, without changing the way development and QA teams work or the tools they use. Developers and testers do not need to maintain, know, or even care about the virtualized networks, but they will see network related problems emerging early in the development cycle. Performance testing and the ability to accurately predict end user experience will “shift left” in the development lifecycle when it is easier and more cost-effective to find and remediate problems.

Top performance is essential, and accounting for the various network conditions of your users must be part of your development plan. The shelf life of short-term and single-use apps means that traditional monitoring and remediation will not work – the app will expire before it can be fixed.

On this webcast, Mistakes that Cause Single-Use and Short-Term Apps to Fail, Dave Berg, Vice President of Product Strategy at Shunra, answers:

1. What are some considerations that should be taken into account when planning a short-term or single-use app?
2. The amount of effort pre-deployment vs the time the app will actually exist seems disproportionate. Is it worthwhile?
3. What are some of the other mistakes you’ve seen in the industry when it comes to developing these types of apps?

Watch this webcast and learn how to ensure user expectations are met in the small window of time your app is available.

• How can we best face this kind of complexity?
• How can service virtualization lower testing and infrastructure costs
• How can network virtualization speed up release cycles?

On a recorded webcast, we answer these questions, introduce you to the essential capabilities for virtualizing conditions in your test lab, and provide you with a methodology to improve testing accuracy and reliability. Watch Beginner’s Guide to Improving Performance with Service and Network Virtualization.

The virtualized test environment encourages acceleration throughout the development cycle while delivering reliable insight into expected application behavior. The unavailability or high cost of services, systems and networks is no longer an excuse for inaccurate functional, load and performance test results.

However, the success of any testing approach is dependent on a proven three-part strategy that is implemented independent of the application, how it is deployed and how it is managed.

What is that three-part key to software testing? How does the production environment impact your testing strategy? What are the critical factors that must be considered and incorporated into your test efforts? Watch the recorded webcast for this blog post, in which David Berg, Vice President of Product Strategy at Shunra, answers these questions and defines a three-part strategy for software testing.

Load testing by itself is not enough. The application environment is affected by more than just the volume of users. In the real-world, the application has to adjust to services, varying network conditions and user populations. As a result, we have to account for and manage for these factors in the test environment. How? With user virtualization, service virtualization, and network virtualization.

In order to create a reliable testing environment, a “three-legged stool” approach is often employed. The seat of the stool represents any test environment – load, functional, single user, etc. The test platform is held up by three legs:

1. User virtualization: Test environments should emulate the users or load that will be accessing the system. This is accomplished via load generators and load testing tools. Some organizations apply single user test automation or even have manual users to perform tests as real users in the production environment.
2. Services virtualization: In production, applications often rely on third party resources or data feeds. These services can number in the hundreds for larger and more complex applications. It is not always practical, possible or affordable to bring those services into the test environment, yet performance of those services is critical to end user experience. Service virtualization emulates the behavior of specific application dependencies that developers or testers need to exercise in order to complete end-to-end transactions in test environments. Rather than virtualizing entire systems, it virtualizes only specific slices of dependent behavior critical to the execution of development and testing tasks.
3. Network virtualization: As with services, network conditions play a critical role in user experience. With mobile apps, for example, up to 70% of end user experience is dependent on network conditions. Network virtualization is a pre-production process for recreating network conditions from the production environment for use within the test environment. Network conditions such as latency, upstream and downstream bandwidth, packet loss, and jitter, are all critical factors that must be taken into account when testing or validating application performance. Network virtualization enables connections between applications, services, dependencies and end users to be accurately emulated in the test environment.

By looking at production data and virtualizing it, you can engineer your environment to deliver the most reliable and predictable test results.

Myth #1 - There is no real, physical way to test for mobile.

Sticking with the status quo of “we do the best we can, even though we know we can do better” should not be a choice when testing mobile applications. This myth may actually seem plausible in that mobile testing is not an easy task by any measure. The reality is that organizations can accurately test mobile applications today by leveraging production network conditions, using a library of network conditions, and partnering with companies that offer best-of-breed products and services specifically for mobile testing. Check out a few free Shunra tools to help you test mobile.

Myth #2 - You don’t need to test for performance with mobile apps since you already test for functionality.

Functional testing tells you if a function successfully executes. However, with mobile in particular, if it doesn’t execute quickly, then the end user won’t wait around to see it. Mobile users perceive failure after just 3 seconds of delay. Functional testing is just half the story. Performance of function/execution/response time is critical to ensuring the app is used/usable.

Myth #3 - Mobile traffic does not impact my back-end systems.

“There’s not much I can do about mobile networks, so why even bother?” We’ve heard this before, and enough times to need to dispel this myth. It would seem that even a small number of users could do any real harm to the production network and the apps being accessed. Wrong. Even a nominal number of mobile users can impact your back-end systems. Network Virtualization testing must be a critical part of your testing methodology, especially when considering that the traditional performance optimization solutions are not available with mobile right now. Watch the webinar for specific examples of how mobile latency can impact your infrastructure. Liam gives an example of how adding just 9 mobile users to a load test scenario of HP LoadRunner running 500 users against a target production application nearly brought down the system.

Myth #4 - Testing in the wild is the best industry practice.

“In the wild” testing, or testing in production, can be a way to validate performance test results in production, but it is NOT a good way to ensure performance or to avoid the pain/negative impact of poor performance. The pitfalls of “in the wild” include an inability to know the precise network conditions at the time of failure and inability to precisely reproduce the network conditions or root cause at the moment of performance failure. Often, this type of testing may be done during non-peak traffic times so as not to affect real users, but then test results would not be representative of real user experience.

Myth #5 - Testing in pre-production is a waste of time and money because you can’t produce tests at scale.

Mobile application performance is 70% dependent on the network, making scale less imperative. Even if the environment is nearly impossible to accurately recreate within a contained pre-production lab, performance is still a concern because the end-to-end infrastructure (like server dependencies) and the mobile network constitute much more a risk than load.

Myth #6 –  Testing will extend the project deadline, making testing is too expensive and too time-consuming.

What can be the impact of failure? The cost of failure? According to IT Management News, “productivity loss due to application performance issues is estimated at a loss of $42,000 per hour for each user group impacted.” Another study from CA Technologies (2011 survey of 1800 companies) revealed that “the loss associated with IT issues has been quantified as a 63% reduction in productivity, resulting in an average of 552 people hours lost per year per company.” The cost of failure after deployment is greater than the cost of pre-deployment testing. Early testing (shifting left) expedites the development process by finding and fixing issues sooner.

This month, in particular, will challenge IT teams with security and compliance issues as employees to bring their new devices to the office. How a company manages BYOD activity in 2013 will be critical. Even the best-of-breed companies that have already implemented a BYOD policy need to be perpetually aware of application performance issues.

What do employees want to access on their tablets? On their phones? According to InformationWeek, the four most-cited resources are email, Microsoft Office applications, VPN, and company file servers. Performance of these applications can help or hinder employee productivity. Corporate IT policies are challenged now more than ever to evolve and consider BYOD.  “Traditional” approaches to managing and ensuring application performance have not evolved at the speed of mobility, rendering them unreliable. The best way to manage user expectations and to avoid the cost of performance failure is by proactively incorporating planning, testing and performance optimization early in the software development lifecycle, before applications are deployed. As companies plan and test for mobile, it is essential that consideration be given for both the client-side user experience and back-end server capacity and scalability.

Staying on top of network performance with the increase of BYOD into your environment is an important component. Will you support devices? Introduce applications enabling your employees to access email? Files? How are you ensuring your company data is protected and secure? The only way to ensure business continuity is to take a stand – are you going to support or discourage BYOD?