One night, back then, I remember waking up shortly after midnight with a terrible earache. Ouch!

Considering the power of the Internet at the time – with its hypertext markup language being the standard – I fired up my trusty modem and dialed into the Internet to look up ear infection on Yahoo. A list of standard results popped up. I clicked on the one that said ear infection symptoms and causes. I was taken to what appeared to be a medical journal since I didn’t fully understand the words they were using. But I knew I was on the right track. There were two close-up pictures: one of a healthy ear cell that was brightly colored and plump; the other of an unhealthy cell that was black and brown and deteriorated.

I may not have comprehended everything, but I was completely stunned by a sentence that appeared in bold within the text on the page: “If the infection persists, the ear will fall off.”

This sent me into a panic! Alarmed, I screeched to my wife, who was sound asleep in the next room, “Hurry, take me to the ER, my ear is going to fall off!”

Startled from a sound slumber, I heard her quickly pounce out of bed and get dressed. In her tired state, she struggled to find her car keys and slip on her shoes. While she was doing that, I held my left ear with my left hand and clicked on the next page; that’s when I realized my mistake. Before me were two pictures of corn. Once picture was a healthy ear of corn on a stalk brightly colored in yellow, the other a sickly ear of corn about to fall off its stalk.

“Never mind,” I yelled to her. “My ear’s not going to fall off. I was looking at an ear of corn! It’s ok, you can go back to sleep.”

I knew at this point that an earache was the least of my problems. As I heard my wife storm down the hall, I knew I was in trouble. I felt her ominous presence behind me when she asked “which ear is in pain?”

I timidly pointed to my left ear. She took a tight grip of it, twisted and gave it a good yank. “How does it feel now?” she retorted.

Actually, it did feel better.

Since the days of my earache, the power of today’s internet browsers has opened up a whole new business model known as Software as a Service (SaaS). Instead of the static pages of the 1990s with very little user interaction, we have more powerful dynamic browser technology.

The technology back in the ’90s did not lend itself well to robust applications on the Internet. However, that all changed with the evolution of the browser. Technologies such as JavaScript, Dynamic HTML and XML are now available to provide users with robust interfaces that include such elements as menu selections, edit boxes, tabs and tooltips. However, from a user point of view, that would seem to be all that is needed to run applications on the browser, such as Customer Resource Management, Financial systems and Enterprise Resource Management. But just as there was more to the “ear” than what I saw on that Web page 15 years ago, there is more work that goes on behind the scenes of a SaaS application than meets the eye.

To really provide a successful SaaS model, applications need to be engineered in a certain way, with three distinct tenets in mind:

1. Statelessness
2. Load Balancing
3. Multi-tenancy

Statelessness

Back in the old days of software development and server platforms, systems were written to run on a server dedicated to the application; they were known as stateful applications. If a particular task hung the system, all users on the server suffered response time issues. In my opinion, one of the failures of companies porting their legacy systems to the SaaS model was the inability of their architecture to handle such a scenario. Building an application from scratch to run on multiple servers requires the application be stateless. This means the interaction between the user’s browser and the application is disconnected multiple times during the session. Session information is maintained within the user’s computer to allow (recall the cookie concept) the server responding to the request to understand who the user is and what functions were being performed. This all happens with the user unaware of which server is responding to her request. The bottom line is that if a server suffers response time problems, a user can be directed to another server to handle future requests, once again with no impact to the user experience. This group of servers running the same applications is sometimes referred to as a sever farm. The SaaS provider maintains the server farm at their hosting site.

This leads to the next tenet.

Load Balancing

So when a request from a browser enters the SaaS hosting site, how does the system know which server will honor the request? This is accomplished through another server that is known as a load balancer. The function of this server is to check the health of all the servers in the farm. By knowing the state of each server the load balancing can pass the request to the server running with the least amount of work load. This ensures excellent response time for all users, regardless if one server is not feeling well at the moment. There are also additional benefits to this model. One is that the SaaS provider can add new servers to the farm during peak demand. Another is the ability to remove a server that may need maintenance with no impact to the user.

We next move on to the data component.

Multi-tenancy

Once again back to the olden days, if you needed to add another database you typically had to have another instance of the application attached to the new database. So in the simple scenario of a single database on a single server, if you have 500 customers each with their own database you would need 500 servers! Now comes multi-tenancy to the rescue. In this model applications are designed in such a way as to share common code with multiple databases. A single instance of the application can send and receive database requests to the appropriate customer database. In the example above, the 500 databases can reside on one server attached to a farm of application servers, monitored by a load balancing server. Pretty neat stuff!

The point is, the growing sophistication of Internet browsers has indeed facilitated the SaaS model; but also, the underlying technologies that happen at the SaaS hosting site – unbeknownst to the end user – also makes the SaaS model possible. When it comes to Software as a Service, there truly is more than meets the end user’s eyes.

Adding a bit more levity to the story, Alex approached me that night and said, “You know, there really is no souvenir shop at the waste treatment plant.” To which my wife and I had another good chuckle.

So why did the adults find humor at the situation while the teenagers didn’t get it? It has to do with how we view the world. People see things in different ways based on many factors: age, culture, education, etc. These all affect the way people perceive their environment. It particularly impacts how different people interact with technology.

Time and again, I’ve seen scenarios where a CFO purchases a financial system because he loves the sophisticated user interface. But the chosen system falls flat at implementation because his operational staff – those who sit in front of their workstations for 8 hours each day – hate the interface because it is too laborious; they can’t get their work done effectively.

As software developers, we face the difficult task of designing applications for a myriad of users, each with different value systems. Our daily challenge is to build a user interface that allows the casual user and the expert to perform their tasks on the same system. It isn’t always easy!

The casual user wants a simple, intuitive interface that walks them through the process and enables them to complete their task without any confusing or ambiguous icons, links or instructions. The expert, or power user, deals with the application for hours each day and wants quick shortcuts and easy access to functions.

There is no middle ground; the novice expects to be led step by step through each function and feature while too many clicks and too much detail will frustrate the expert. As we start a new decade (another topic for debate: is 1/1/2010 or 1/1/2011 the start of the new decade) the challenge we face as developers is designing an interface that satisfies all.

To illustrate how difficult this can be, I went to Barnes & Noble and looked at books about Microsoft Excel. What I found was staggering. Excel 2010: The Missing Manual is over 800 pages and Excel 2010 Inside and Out is more than 1,000! And Excel is only a software tool to help you get your real job done. Has anyone ever read the entire text? Original versions of Excel, I’m sure, were simpler and required shorter manuals. But as the software evolved and more people started using it for different functions, it became more complex. Today, as the CFO uses it to balance a company’s budget, the project manager uses it to track a multitude of tasks. Different types of professionals need to know how to navigate the same software; therefore the 1,000 + page How To guide is born.

Our society today is filled with technology that has great benefits. We can get more done with our limited time each day. The cost, however, is frustration and, at times, the feeling of stupidity with the surfeit of user interface models we encounter in our daily activities.

My experience with a self checkout machine at a local mega hardware store last month is an example. Accustomed to the self checkout interface at the grocery store closest to my house, imagine my surprise when I accidentally pressed the French button on this unfamiliar system. Since I don’t know a lick of French, I had no idea how to cancel the transaction and start again. Five people were waiting impatiently behind me in line and I had this thing speaking French to me. I eventually got through the transaction, and all the people waiting were likely expecting me to speak in my most alluring French accent. They were a bit surprised when I turned to them and, in my best New York inflection, said “Sorry for taking so long.”

This embarrassment brings to mind the story of a new computer user who had just received a PC-DOS computer (remember them?) with a rather vaguely written installation guide. As he attempted to assemble the components, his frustration at the manual got the better of him. He called customer support for help, which only reinforced his feelings of ignorance. After several fruitless attempts to solve the problem, the support rep – who was obviously on a different experience level than the caller – claimed to have identified the problem.

“Great,” replied the customer.

The support rep said “Stand up in front of the computer parts.”

“Okay, I am,” replied the customer.

“Is your front door to the right or left?”

“To the left,” the customer replied.

“Good, start walking to the left and tell me when you are by your front door.”

“I’m there now.”

“Open the front door,” the support rep commanded.

“Door is now opened.”

“Good,” the computer expert on the other end of the phone paused. “Now step outside and shout: DOES ANYONE OUT THERE KNOW HOW TO USE A COMPUTER?”

What’s the point? During our different ventures in software development, we will encounter both pro users and novices. The trick is to find a way to make both feel like they’re getting a great experience, one that’ll make them want to come back again and again. That’s what makes software development challenging – in a good way. Technology continually evolves, making new techniques and methods available so we can design a user experience that satisfies all participants.

To wrap up this trilogy, I will reflect on a few of the topics I’ve learned during MediClick’s SAS70 travels:

• What makes a SAS70 Type 2 audit successful?
• What are the important components of an audit report for the SaaS consumer?
• What is the fate of the SAS70 audit?

Successful SAS70 Audit

It’s now the end of 2010 and the dust has settled from three successful SAS70 Type 2 audits at MediClick. This begs the question “what exactly is a successful audit?” In the SAS70 report, each control objective is listed and test cases are described. Results are documented for each test case. From a pure report point of view, success is having no relevant exceptions reported from all test cases. That is the simple answer, but reality is more complex; it comes down to two key elements:

    1.       Management Awareness

Is management aware of the activities occurring within its company? Are there meetings to establish corporate direction and confirm results? Is there an approval process for software enhancements and access rights to operational systems? Are all access to networks and physical locations shut down when an employee is terminated? The bottom line is that management must be aware of and approve activities within the organization as revealed through paper or electronic documents.

    2.       Segregation of duties

I remember talking to an IT executive about his SAS70 process; he said his company had to revert to a SAS70 Type 1 audit rather than a Type 2 audit. If you recall from previous blogs, a Type 1 audit states the control objectives without any testing. His company failed the Type 2 audit process because the auditors discovered that developers had been deploying code directly into production from their desktops. As you might imagine fixes were lost when one developer overlaid changes by another. Source code control was nonexistent.

This illustrates the basic point of the segregation of duties, you need a separate team to deploy, test applications in a QA environment and install them into production systems.

Important Components of an Audit Report

I should also note that MediClick also receives SAS70 audits from our partners. The hosting sites for our production and hot fail-over servers provide us with their SAS70 reports. So what do you do if you are a recipient of a SAS70 report? The answer is simple: READ IT.

I recall a fellow Triangle Technology Executive Council member who complained about some loose practices of a SaaS provider he was using. He complained to the provider’s management team and their response was: “You shouldn’t be surprised; these exceptions are noted in our SAS70 report.” I asked him what he did with the report when he received it. His reply: “I filed it in my drawer without turning a page.” Lesson learned!

There is a section of the report to which you should pay particular attention. Titled User Control Considerations, the section essentially lists controls that the SaaS user must implement to mitigate any risks within his operations while using the application. In short: it’s your responsibility! Examples are:

• Ensuring your organization has adequate bandwidth to the Internet
• Providing secure access to each application user
• Creating your own policies and procedures for user access, such as disabling application access upon an employee’s termination

Reading the SAS70 report gives great insight into the SaaS provider’s operations. There were several incidents where I felt certain control objectives of importance to MediClick were missing from our providers’ report. Case in point: one of our hosting sites provides full back-up management services for MediClick. In their report, they tested the control objective regarding the back-up and transfer of the tapes to Iron Mountain. The missing element, which I felt was critical, was the test that validated if the back-up tapes were actually readable! The end result was to add these tests to our own control objectives.

Fate of the SAS70 Audit

Finally, what is the future of the SAS70 audit? It’s changing next year and will be SSAE 16, which is Statement of Standards for Attestation Engagements 16. All service providers with an opinion period ending after July 2011 will be under the new guidelines (please don’t hold me to this as these dates have a tendency to change). You can find further details at http://www.ssae-16.com/. In short, the goal of this process is to align the United States with international standards as designated by the IAASB standards on assurance engagements (http://www.ifac.org/IAASB/); ISAE 3402 (http://www.ssae-16.com/category/isae-3402/).            

It appears new challenges are forming on the horizon. But that’s another trilogy.

To head off any backlash from an audit, we wanted to ensure we had the right team in place. We first set out to find an auditing firm that understood the importance of preserving the successful process we had built during our seven years in business.  Since a certified public accountant must perform each SAS 70 audit, we found a firm that understood SaaS and could effectively work with us to build control objectives that reflected our business model.

After we had the auditing firm in place, we needed to undertake two engagements to achieve a successful SAS70 Type 2 audit: a readiness assessment and the final audit.

The first engagement, a readiness assessment, took place in March 2008. Much like the PSATs that high school students take to gauge their performance prior to taking the full SATs for college entrance, the readiness assessment would allow us to get a first-hand grasp of the entire process and the types of activities the auditors performed. We worked with the auditors to define our control objectives, which we designed so that an auditor could test and report on the effectiveness of the objective. Objectives ranged from the security of the MediClick network environment to the documentation of our development, testing and deployment processes.

Prior to the readiness assessment the auditors requested that we provide them with detailed documents, such as an organization chart, recent employee handbooks, listings of firewall changes, data backup schedules, visitor logs, hot fixes and service packs applied to production systems.

The auditors were on site for a week to perform the assessment; at the end they produced their findings and recommendations for remediation. The auditors assigned each recommendation with a risk level – from low to medium to high – and included a resolution difficulty rating with each.

We felt good about the results of the assessment because it revealed no “gotchas.” The recommended resolutions were doable within our 3-month remediation period. During this time we corrected any shortcomings discovered, such as using generic userids, tightening up our employee access rights and hardening our network and application passwords.

With a better idea of what to expect during the audit, our next step was to talk to the employees about SAS70. Because employee buy-in was essential for success, we needed to ensure all MediClick employees thoroughly understood the purposes of the audit process and what their roles would be. We held internal meetings with all the employees to explain the process and the importance of the audit to MediClick. This was a significant moment in our auditing process because we could see how the employees would react to the audit. It was a success! No one revolted as we had been warned would happen! In fact, just the opposite occurred: acceptance and enthusiasm.

After completing the remediation tasks and employee orientation, we were ready for the second engagement. Our actual audit period was from July 1, 2008, through Dec. 31, 2008. Our first official SAS70 audit would cover a 6-month time period. Then we would have an audit done for each subsequent 12-month cycle.

Two auditors were on site for two weeks during the month of December 2008, pouring through our system logs, documents and development processes. They held interviews with each department manager to review controls and operational procedures. The auditors observed MediClick personnel in their daily activities, looking at such detail as how well they enforced visitor sign-in protocol at the front entrance to our office. The auditors left with tons of data to review.

Even though we successfully navigated the readiness assessment, we held our collective breaths as we waited several weeks for the preliminary findings. Would we experience any of the doomsday scenarios my industry colleagues had painted in my head?

The results were in and we did it! Our first audit was clean and all tests of control objectives showed no relevant exceptions. The final report was delivered in January 2009 and immediately sent out to a backlog of customers who had requested the report.

Despite the tales of woe that we had heard, we had learned for ourselves how to successfully navigate the SAS70 audit process.

Next is The Lord of the Audits Trilogy, Part 3: The Return of Art. In this segment I will reflect on my views on what it takes for a successful audit, what are the critical areas you need to review when receiving a SAS70 Type 2 audit and the future of the SAS70 audit.

That was the advice I received from a fellow IT executive while discussing a SAS70 Type 2 audit at a meeting on IT governances. At the time, the company he worked for had already undergone the audit, which MediClick was considering. The process involved a complete, independent review of his company’s controls and operations, ending in a published report assessing the effectiveness of those controls.

The executive offered his cautionary tale about SAS70 in late 2007 during a Triangle Technology Executive Council (TTEC) meeting in North Carolina’s Research Triangle Park. Now with a membership of more than 100 senior IT executives from over 100 companies, TTEC included only a dozen or so executives at the time MediClick was considering SAS70.

I sought the wisdom of these IT executives because we were experiencing an increased demand for MediClick to become SAS70 compliant, something we purposely put on the back burner. Why? Because we were concerned that our own SAS70 audit would be cost prohibitive for a company of our size and that it would slow our development process to a crawl.

A bit of company history may put our SAS70 dilemma into perspective.

Formed in 2000, executives wanted MediClick to be an Application Service Provider (now referred to as SaaS) for healthcare. We saw a need to advance the industry beyond the traditional software solutions our larger competitors were putting on the market.

Our success depended on our ability to deploy our application quickly and effectively. We had to be lean and mean, which meant employing rapid development methodologies and a hosting site for our software.

Fast forward seven years, we met these obligations and became a successful SaaS company. It was then when our hospital customers began requesting that we become SAS70 compliant. Before we could do that, though, we had to be careful not to change the positive impact we were having on our customers.

“MediClick’s customer support is very helpful. If we have problems, they log on and solve them. They get back to us fairly quickly.”

“MediClick has a sprint process for enhancements instead of doing a rebuild each year. Every eight weeks they do one of these small builds and get things fixed. I have never seen a company be so efficient.”

Published by an independent organization that evaluates healthcare companies, these quotes reaffirm our commitment to efficiency and customer service. We were – and still are – a more nimble software company. We were afraid a SAS70 audit would change that.

For one, we expected that an audit would flag us when our support team signed on to the customer’s application to resolve issues. The SaaS model made this easy because we could access the application via the Internet. Unfortunately, all MediClick personnel used a single sign on to the databases. That was a no-no for SAS70 as it left no audit trail that documented what a particular MediClick support person did.

While those sign-on issues were a top priority, protecting our development methodology became a main concern. Since we had introduced the tenets of agile development and abandoned the standard 12 to 18 month release model, our customers were used to seeing our enhancements every 6 to 8 weeks.

For seven years, we had been fine tuning our process so we could effectively deliver value to our customers. We questioned whether a SAS70 audit would set up road blocks and cause us to lose our competitive edge. Would it frustrate developers resulting in a mass migration to the exit doors? Would we have to revert back to a long release cycle? These questions resulted in many sleepless nights.

So why did we choose the route that my peer advised me against?

First, we wanted to give our customers what they wanted. Plus, as we met more and more industry prospects asking for our SAS70 validation, it became apparent that MediClick would have to take the SAS70 plunge sooner rather than later.

So late in 2007 the decision was made; there was no turning back. We would attack this problem as MediClick has always responded to a challenge: jump into the SAS70 waters with enthusiasm to get the job done completely and correctly. Failure was not an option.

The most difficult part was determining where you start the SAS70 process. How prepared were we? Were there other vulnerabilities we were not aware of? Would there be a potential for a poor report and how would that affect our business?

The storm clouds formed over the horizon and the Dark Lords of the invidious auditors began their approach.

Next is The Lord of the Audits Trilogy, Part 2 – The Two Engagements.

As a consultant, you anticipate that one assignment would eventually turn into a permanent position. I imagined the perks of working for an airline: unlimited travel to exotic locales; a bounty of delectable airline food (which was quite good in those days). I was already charting a course and ordering my first meal when the sound of Dave’s voice yanked me back into reality. He told me there were two negatives to this assignment:

1.       They were still using punch cards

2.       Since the operations were unionized, I had to play by union rules

The second negative was a nonissue. Once a proud member of the New York City teacher’s union, I could deal with a union shop. So the decision came down to punch cards vs. airline food. “Okay, I’m in,” I enthusiastically told Dave, ready to begin my career in this exciting industry.

Unfortunately, working as a contract programmer at a large company like Pan Am turned out to be more of a challenge than I thought. I lasted only two weeks.

My first incident involved a 20 punch card limit. Programmers were only allowed to punch a maximum of 20 cards per day, after that you were required to submit your coding sheets to a key punch operator. I had reached my limit by noon and all key punch operator personnel were out to lunch. I kept punching. End result: first violation.

The second violation happened after work hours when just about everyone had gone home. I sent a listing to the printer and waited for someone to grab it. You see, only operations personnel were allowed to burst listings and put them in cubby holes. But seeing my listing on the printer, just staring me in the face, was too much for me to bear. Knowing it was after hours and operations personnel were on skeleton crew assignments it could be hours before someone would get to it, I went for the printout. Yet again, another rule caught up with me.

That was it; my career at Pan Am was over with two, relatively minor, violations! So much for unlimited travel and tasty cuisine.

In working at Pan Am and a few other large corporations, I started to feel their rules and procedures cuffed my productivity. For me, it is far better to work for a small- to mid-sized company for several reasons.

                 1.       Direct Impact on the Bottom Line

In a small-to mid-sized business, each employee contributes to the success of the company and, most importantly, feels that way. Whether you are a developer writing code, QA analyst testing releases or a customer support representative helping clients, you know your work is critical and everyone in the organization notices it.

                 2.       Everyone is a Star

There really is no place to hide in a smaller shop. If you cannot carry your weight, your underperformance will be noticed. Whether it is a lack of ability or a poor attitude, your career will be a short one. On the positive side, as a viable employee you know the person to your right and to your left is competent, dedicated and will get the job done.

                 3.       Agility

I don’t mean the software development methodology we use at MediClick. When I say agility, I mean the ability to quickly adjust to customer demands and market conditions. Decisions to change product direction can be made quickly without layers upon layers of corporate structure to get through.

A case in point occurred at MediClick several years ago. Marketing approached senior management with a need to develop a contract management application for our SaaS suite that complemented our financial and supply chain products, which had been successfully developed and deployed over the last seven years. Senior management, architects and developers held an offsite meeting. That full day meeting resulted in a decision to pursue the development of this new product line. The next day we laid the foundation and started the project.

These reasons highlight why I feel small- to mid-size companies are often more effective. On the other hand, one might argue that a large corporation offers security and longevity. So what? That security comes with a few corporate rules you have to live with. It’s worth it, right?

My rejoinder to that argument is a single word: Enron.

One of my favorite film quotes is from Ebby Calvin LaLoosh (yes, that’s his name!) the erratic but talented Durham Bull’s baseball pitcher placed under the tutelage of Crash Davis, an experienced but somewhat over the hill catcher, made famous in the movie Bull Durham. “This is a very simple game,” Laloosh says. “You throw the ball; you catch the ball; you hit the ball. Sometimes you win; sometimes you lose; sometimes it rains.”

I find the later part of the quote to be quite insightful and very applicable to marketing and selling the SaaS model to healthcare organizations.  According to a recent InformationWeek survey of 150 organizations not using SaaS, the model raises two red flags in their IT departments:

1. Concerns over security
2. Concerns over data ownership

Any SaaS provider has experienced this bias. Many CIOs and CTOs insist that their data reside in their network and not at the SaaS-hosted site because of a wide spectrum of fears:

• Who can access the data?
• Is the data being shared with other organizations?
• What if the data is financial data or patient records?
• What happens to the data when the contract terminates?

If an organization’s tech execs completely discount SaaS and refuse to let their users evaluate the offering, the rain starts falling and the game is canceled on any effort to get the model adopted there.

So in the SaaS world how does one address these fears? Let’s look at each scenario.

Security

Keeping your own data may give you the illusion that your data is absolutely secure, but there are no guarantees. A case in point is the recent occurrence at a South Shore, Mass., hospital, which lost 800,000 patient and employee records on backup files (http://bit.ly/dt4MyT). There are other such incidents I could easily site as it seems like this type of occurrence happens every day.

The point is, making sure your data is secure – whether it’s kept internally or externally -- is paramount.  With SaaS, this verification is essential.

As I have mentioned in previous posts, the SAS70 Type 2 audit is an essential process that a SaaS provider should undertake. An independent auditing firm tests and reports on control objectives to ensure that the company has defined security procedures and, most importantly, that all employees follow these procedures. Auditors extract sample test cases from visitor logs, maintenance logs, virus scanning logs and system alerts. They then inspect firewall rules and changes. The audit is typically performed yearly, although I have come across some companies that perform them semi-annually. Upon completion, auditors provide a report to the SaaS provider indicating the test results.

If you are a SaaS consumer, you are entitled to a copy of your provider’s report. If you are evaluating a SaaS provider you will not be able to see a full report. In that case, ask for a letter attesting that a Type 2 audit was successfully performed.

I should mention are two types of reports. A Type 1 report only assesses the stated control objectives without any testing. A Type 2 report encompasses test cases and publishes results of the effectiveness of the objectives. In my opinion a Type 1 report provides no value to the SaaS consumer.

Data ownership

There are 2 aspects to the concern over data ownership.

1. Service Termination: A contract with a SaaS provider should indicate that at contract termination, data is turned over to the customer. This ensures that you will have access to the data and be able to populate your replacing application with the SaaS data.
2. In Service Access: This pertains to having your SaaS data available for you to download to your own servers. Any decent SaaS application will provide tools for data extraction to a format that can be used to import into another application such as CSV or pipe delimited files. The caveat here is that once the data is on the consumer’s premises the security onus is entirely on the user of SaaS and not the SaaS provider.

Are the concerns expressed above real? In my opinion they are, but with a thorough review of the SaaS contract, a successful SAS70 Type 2 audit and the ability for the software to provide data, these concerns can be mitigated. It’ll once again be time to play ball, as Ebby Laloosh might say!

As I thumbed through the tattered and fading pages, I chuckled, realizing that it was the first DR manual! The first pig was safe in his straw house until, with one huff, the house collapsed – the first fail. So the smart pig went to his backup plan: his brother’s stick house. But, like the first, the stick house couldn’t withstand the wolf’s hot air.

What saved the pigs from being that night’s house special? It was, of course, the sturdier, well designed, brick fortress that the third brother built in a separate location. The third pig had the most important element of the DR plan – a disaster recovery site.

In healthcare – where patients are the first priority – creating a separate DR site is essential. Unfortunately, many hospitals cannot afford the cost of a complete redundant fail over site.

That’s where the SaaS model proves to be a unique opportunity. With it hospitals can run software that is offsite from their location, which is often essential in a disaster situation. One of MediClick’s clients – a hospital north of New Orleans – experienced something like this during the devastation following Hurricane Katrina.

Communication was down at the hospital and its surrounding community. Resources were limited, but the need for emergency care was greater than ever. MediClick, as this hospital’s materials management software vendor, was able to use the SaaS model to order supplies for them from a separate location. We made sure emergency medical supplies were delivered to the disaster area when time was of the essence. Teamwork, the SaaS model and a solid DR plan benefited thousands of people during the long process of recovery.

MediClick was glad that we were able to help keep them functioning using SaaS. Sadly, choosing a SaaS vendor may not be enough to save you. Not all SaaS vendors have their own disaster recovery plan. If your organization uses a SaaS provider, you should inquire about both their DR plan and site. Review their SAS70 Type 2 audit report to see if the DR site is mentioned in the report.

As a vice president at MediClick, customers have asked me many times for a copy of MediClick’s DR guide. But any decent DR guide has access codes, private phone numbers of employees and system passwords. This information cannot be shared with anyone outside the MediClick offices. That’s why we make it a point to document our DR plan in our SAS70 report. Be sure to check to make sure your SaaS vendor does the same.

There was a time – before caller ID – when I answered these calls. I even fell for one of their pitches. After repeated requests for meetings, lunches and phone conferences – all with promises of project nirvana – I took a chance at outsourcing a project.

We thought the assignment was basic enough: use a standard report writer to produce a purchase order print document from our supply chain system. The technician was given specs on the format and placement of data. We provided a sample purchase order based on a bogus medical device along with the product’s header and detail line. The detail line of this sample purchase order included the comment “Product must be frozen for 24 hours prior to use.”

Off to outsource land went the specs and the sample document. Six weeks later – which seemed forever for such a basic task – the developer sent it back without asking questions. Just “here it is.” Ok, I thought, let’s give it a quick test before releasing it to QA. I created a one line purchase order for 100 boxes of toilet paper. I printed the purchase order and looked at it. Much to my surprise (and outright laughter) the comment “Product must be frozen for 24 hours prior to use” was printed under the toilet paper line. Now that’s some cold toilet paper!

The comment was “hard wired” to the purchase order, creating a purchase order with 20 lines, all with the same comment!

I reviewed my specifications; it was not clear to the developer where the purchase order comments line should have originated. Since specifications are never 100% clear, no matter how detailed they are, the developer needs to make some assumptions related to the business processes. My developers would never make the same mistake because they understand the healthcare business, our application and the intent of the comment line on a purchase order. They would have coded it correctly so only the comments line associated with a particular purchase order printed.

This led me to perform some informal research – and do some internal reflection – about outsourcing. How effective could it be? Would there ever be a case when it is useful for us?

If you do a Google search for “software projects outsourcing success rate”, you’ll get thousands of hits. Read a few documents and see that numbers are all over the map. Some research articles claim that technical outsourcing offers only a 20% success rate while outsourcing companies claim 80% success rate.

Let’s take the middle road and assume a 50% success rate. If you were a baseball player batting 500, that puts you in the hall of fame. As an IT executive, a 50% success rate for projects gets you a job at a fast food restaurant flipping burgers.

So what’s the conclusion? IT is a people business. Software languages change, standards evolve and computers become smaller, faster and more sophisticated. However, the key components for successful project delivery are still people who understand development, business processes and software construction.

There are instances where outsourcing your IT or software projects is acceptable or even advantageous. But the concept doesn’t work for MediClick. Only a dedicated staff of developers, project specialists, QA folks and support personnel could offer the high-quality services that we – and our clients – expect.

In our case, if you take any member of our team out of the equation and replace him or her with someone in an outsourced role, you have a recipe for a failed project.

That’s because our on-site team benefits from things that an outsourced professional doesn’t have:

• Close Communication – I have found that communication is a critical factor in the success of an IT project. Whether you’re holding an impromptu meeting, having a conversation by the coffee maker or making a quick call to a colleague’s desk, you’re facilitating information transfer and building camaraderie. The ability to walk down the hall and talk to each member of my team outweighs any benefits I would derive from an outsource group thousands of miles away.
• Product Knowledge – Communication serves no purpose if the information transfer is obsolete or just plain wrong. Product knowledge is gained over years of experience with a complex software application. Personnel dedicated to a company’s success understand the business and culture of the company.
• Business Acumen – Understanding how a business works – whether you’re working with materials management in hospitals or the unique challenges of another industry – is vital to developing a software product that is actually usable and provides value to an organization.
• Size Counts – Yes, size does count. As a lean and mean, mid-sized SaaS provider, MediClick can turn on a dime. When changing market conditions require us to quickly focus resources on a new business objective, we’re ready. With key personnel working locally, making a change can be rapid and thorough. Try making this adjustment with outsource personnel.

That's why I always answer the phone when one of my team calls me. Better yet, when they call, I just tell them to stop by my desk on the way to the coffee maker.

In the 1970s, as a laid off high school math teacher, I was sitting in my living room watching TV when a commercial caught my eye. Being the days well before the DVR or remote control, I couldn’t fast forward or instantly change the channel. So I watched.

The ad showed a slick, red Maserati sports car pull up in front of a steel and glass building. The car stops directly in front of the entrance and a debonair man of intrigue stepped out and entered the building. Once inside, a beautiful young lady in a long white lab coat greeted him. She helped him into his own lab coat.

The two walk by a set of computers – in the ’70s, a set of computers took up several rooms – with lights blinking and buttons flashing. They pressed buttons on the machines and then huddled over some data they had extracted and printed via their state-of-the-art dot matrix printer.

It seemed the two were part of an exclusive, high-tech club; one that I wanted to join. Their computer wonderland – while rudimentary by today’s microprocessor-based, palm-sized computer standards – intrigued me.

The commercial turned out to be an advertisement for a computer school. I knew what I wanted to do for the rest of my life.

I had no illusions when I enlisted in computer training. I knew the IT lifestyle – this elite society – would be nothing like the glamorous one portrayed in the commercial. In reality, it was much different.

In the early years, I had developed an application in Cobol for a Phase Four minicomputer at CBS, written CICS macro instructions to program the IBM 2260s, and scratched my head in front of a TRS-80 trying to decipher an error message.

It’s no wonder why, several decades later, the professional world gave us the unfortunate moniker of tech geeks! But, like many of you, I wore the label proudly, especially since we tech geeks can make an immense difference in the business world.

The damage to the data center was clearly severe. Water had leaked on servers causing several customers to be offline for three days. What good is diesel backup for seven days if there was nothing to power –all the equipment was flooded!

We already knew how imperative it was for MediClick to be prepared for a similar emergency. Just as they do today, our customers depended on our software to order medical supplies for their departments – often using just-in-time inventory and placing orders days or hours before actual use.

We had a well-documented back up plan to get our clients back up and running after a disaster. We backed up the data and sent it off site to Iron Mountain for protection. In the event of an emergency, we could put together another co-location and restore the data.

With our recovery plan in place, we knew we would be well ahead of a hospital’s ability to get its own data centers operational after a disaster. In fact, our users understand when a short period of down time is necessary after a disaster, but we still weren’t satisfied. Moving the back-ups to a new server location required a long downtime. Unfortunately, there were no good alternatives available yet.

That’s when we decided that we needed to form our own exclusive club within the IT world – one where data downtimes were short and we were the heroes of the entire enterprise.

Enter the hot-backup solution. Technology had finally caught up to the long downtime issue. The first step was server virtualization management, which was just making its headway into IT. Then came the Utility Computing Platform.

Our downtime after a disaster shrunk from a couple of days to a couple of hours.

In future blogs, I will discuss how migrating from co-location to the utility platform satisfied our immediate hot failover needs until MediClick could develop its separate DR site. I’ll also talk more about disaster recovery and utility computing as well as Software-as-a-Service, cloud computing, agile development and more.

It’s a new club, and you’re invited.