The best way to check this out is to use the DIG command. DIG is an acronym for Domain Information Groper. Passing a domain name to the DIG command by default displays the A record of the queried sit (the IP address).

We can use Dig to check the new nameserver are correctly returning the A record and MX records. To do this:

 Dig@<nameserver URL or IP> <DomainName to check>

If this is correct then it means that the name servers have the correct records which means when they are changed at the registrar we can assume they will be correct.

In the case of the company in question the DNS was correctly returning the new NameServer and MX records for the Domain but their local recursor was still returning the old NameServer records as propagation had not taken place.

Other recursors can be checked to identify whether propogation has taken place there i.e.:

dig @4.2.2.1 ns <domain> would check the Verizon recursor

Others of note are:

208.67.222.222, 208.67.220.220 – OpenDNS

8.8.8.8, 8.8.4.4 – Google

Others can be found on the OpenNic Wiki

So in the companies case caching of the prior NameServers and the TTL (time to live) was causing the problem as the new NameServers were not completed propagating. Essentially there were two different “nameservers”, each returning different values, and, being selected randomly (due to cached ns records).

One of the things we were able to do help smooth the transition was to ensure each NameServer returned identical values by making both zones were 100% identical ie. on the original service we changed the NameServer NS records to match the new NameServer NS records. Ideally this would have been done as soon as migration occurred.

Also running vmstat we could see the wait time was quite high, so the question was which process was causing the issue. Interestingly doing a Google web search brings up almost no coherent posts on finding disk bottlenecks. The solution is good old iostat. That provides the information about the disk read and writes per partition but it does not tell you which process is causing the disk i/o. The later versions of linux kernel provide quite good diagnostic information about the disk i/o but this is not documented in the reasonably popular older posts on the subject of disk thrashing.

For the lastest kernel versions you can use iotop to pinpoint the process that is specifically doing the disk i/o. To do this:

1. Start iotop

2. press the left arrow twice so that the sort field is on disk write.

3. You will now be in real time mode of which process i is writing to the disk so you can see specficially

4. If you wish to get a historic view of writes to date then press ‘a’ again (just press ‘a’ one more time to switch back).

In this clients case the issue was their temp directory was on the same physical drive as their site and MySQL DB. Moving the temp diretory to a separate drive resolved the issue.

MySQL ver 14.12 Distrib 5.0.27, for Win32 (ia32)

MongoDB v2.0.4 for Win32 (journaling not enabled)

The test was centred around a table that has 400000 records with numbered names

The table was indexed on two fields, id and an_id

Selection from specific folder by name:

SELECT id FROM table WHERE (an_id=2 AND name=’some name_251504′);

db.files.find({an_id:1, name:’some name_255500′}, {id:1});

* no index for ‘name’

Increased records number to 800 000 (reached limit on 32bit OS for the data file size)

*Added index for ‘name’

Data files size:

Selection of files from specific folder by name pattern:

SELECT count(*) FROM table WHERE (an_id=1 AND name like ‘%ame_2%’);

db.files.find({an_id:0, fi_name:/ame_2/}, {id:1, fi_name:1}).count();

> 202 225 records found

* first run and others (match pattern changes slightly to prevent cache usage)

Count

select count(*) from table where (id > 500000 and id < 550000);

db.files.find({id:{$gt:500000, $lt:550000}}).count()

> 50 000 records found

Delete 10 records:

delete from table where (id > 800000 and id < 800010);

db.files.remove({id:{$gt:800000, $lt:800010}});

Delete 50 000 records: 

delete from table where (id > 600000 and id < 650000);

db.files.remove({id:{$gt:600000, $lt:650000}});

Update 10 records:

UPDATE table SET name=’some new name’ WHERE (an_id=2 AND id > 200000 AND id <= 200010);

db.files.update({an_id:1, id:{$gt:200000, $lte:200010}}, {$set:{name:’some new name’}}, false, true);

Update 50 000 records:

UPDATE table SET name=’sone new name 2′ WHERE (id > 250000 AND id <= 300000);

db.files.update({id:{$gt:250000, $lte:300000}}, {$set:{name:’some new name2′}}, false, true);

Insert 50 records:

Insert 500 records:

Conclusions and other thoughts:

MongoDB has a clear  advantage on speed and this increases as more records are added.

Concerns are:

- MongoDB is not as battle tested or hardened

- The “gotcha’s (lack of our knowledge in part..)

- In MySQL data can be obtained from multiple tables with a single query whereas in mongoDB it seems multiple queries are needed to obtain data from multiple collections. Whereas there are latency advantages when dealing with a single collection these are negligible when dealing with multiple collections. Also, tuning of MySQL buffers and partitioning reduces speed advantages once again.

The conclusion was to stick with MySQL but to keep an eye on MongoDB.

DropBox store the files you give them on Amazon S3. This surprises many people who had assumed that they are stored on DropBox Servers. This means that the entire DropBox business model is beholden to Amazon Web Services. Amazing when you think about it, and highly illustrative that what DropBox really brings to the table is great software with a killer feature, but what is going to happen when every one else has that killer feature, with 10x to 20x more storage for free?

A  recent article had DropBox valued at 4 billion dollars . This is a valuation on a company doing revenues between 100-200 million dollars per year in which investors have poured in 257 million dollars in funding. Perhaps it’s me, but I just don’t see it. Yes, they have a gazillion subscribers but so what? In a commodised industry that struggles to convert more than 2% of the user base, why should that get me excited? But there is DropBox Teams for businesses right? Ever used it? Then try it and you won’t need me to draw a conclusion.

So what for DropBox if there is no mega IPO coming along? They turned down Mr Jobs (a mistake), so who else would be interested? What about Amazon? After all DropBox really is the ultimate sync client for Amazon S3. With Amazon now looking twards  private cloud it would same a match made in heaven. As with all good things, time will tell……

To view the spreadsheet directly on Google Docs click here (it seems Google only supports 50 concurrent connections of a spreadsheet so if you have an issue try again later)

A list stored on the pastebin service shows the extent that the MegaUpload and MegaVideo closure has had on services:

To that end we wrote a small script that can be used to automate the restarting of MySQL in such scenarios.

The script is called mysqlrestart.sh and is listed below. You need root access to be able to use it. If you use it and  ever reboot the server you will need to login as root and run nohup ./mysqlrestart.sh  & to restart it.

Set the script to run every 30 seconds using Cron. It will then check for a number of connections and if it cannot get a connection or the number of connections is greater than the number defined (defined as 90 in the example below), it will restart mysql.

#!/bin/bash

SQLCONNECTION_THRESHOLD=90

echo `date`  sqlrestart started >> run.out

while true; do

        sqlconnections=`mysql –skip-column-names -s   -e  ”SHOW STATUS LIKE ‘Threads_connected’” -u root | awk ‘{print $2}’`

        #exclude myself from the number of thread connections             

        sqlconnections=$((sqlconnections – 1))

        echo `date` sqlrestart connections  $sqlconnections  >> run.out

        if [ $sqlconnections -gt $SQLCONNECTION_THRESHOLD ] || [ $sqlconnections -lt 0 ]

        then

                echo `date` restarting mysql server $sqlconnections  >> restart.out

                service mysql restart >> restart.out 2>&1

                echo `date` restart  complete   >> restart.out

        fi

        sleep 30

done

File Server: StorageMadeEasy

We use SkyDrive (25 GB free) in conjunction with Amazon S3 and an in-house PogoPlug installation to store files. We use the Storage Made Easy Cloud File Server to provide a unified view of all of ur files (we are trialling PogoPlug support with them). This also enables us to assign user / file permissions and governance on the consolidated information stores which is very useful. We can also create collaboration groups across the consolidated information stores for sharing with clients and where the files are stored is abstracted (to the client). We also use the service  to managea nd share files on our iPad (tablet) and Windows Phone 7 clients.

CRM: Zoho CRM

Having used several different CRM systems over time, personally we prefer Zoho. For up to 3 users it is free and adding users and services is reasonably priced. It’s also very easy to change the default templates and to use (using the HTML5 mobile App). SalesForce is synonymous with CRM systems and SaaS, but Zoho is good (cheaper) alternative.

Project Management: Basecamp

Basecamp can be a little expensive but if you manage projects and want to collaborate it is hard  to beat as it’s simplicity and easy to use web interface stand the test of time.

Source Code / Bug Tracker: BitBucket

There are many source code and bug tracking systems, many of them free but we like BitBucket. It’s free for up to 3 users and is a solid source code and bug tracker. For source code editing we use Textastic which enables us to hook into specific source files stored on SkyDrive via WebDav using the SMEStorage CloudDav feature which is enabled when the iPad app is purchased.

Analytics: Google

Google Analytics a essential for tracking website statistics. There are alternatives (such as the fantastic Piwik) but it is hard to beat for ease of use. It is not flawless, the lack of ability to track IP addresses (and therefore do a reverse DNS lookup) is frustrating for example. We use AnalyticsPro on the iPad for mobile access/tracking.

Email: Gmail

Google Apps GMail is a great email system. We’ve used it for years and only have good things to say about it. We backup our Gmail to SkyDrive using the SMEStorage Cloud File Server so that it can be indexed and searchable along with our other files, and of course also for resiliency. For offline access on our iPad’s we use a customised version of Remail that we enhanced for the iPad.

Inbound Lead Tracking: Leadlander

LeadLander enables us to track companies visiting our website, how often they visit, and the new people feature enables us to directly contact leads. It’s a great service and great value at a couple of thousand dollars per year.

Server Monitoring: Server Density & Pingdom & WatchMouse & PagerDuty

Server Density is great for monitoring thresholds on Apache, Server Processes, MySQL connections etc. We plug this into PagerDuty so we can be alerted by phone if major thresholds are breached. Server Density also provide an iOS App for push alerts. Pingdom is used an added check for server outages, and WatchMouse is used to check quality of service for times taken to load pages on a site.

MySQL Admin: PHPMyAdmin

We tend to use the command line but if we need to access MySQL graphically we use PHPMyAdmin.

Call Services: e-Receptionist and Skype and TollFreeforwarding

e-Receptionist is great for virtual teams if you want to route calls to Sales / Support etc and you are not physically in the same location. Skype of course needs no explanation, except that we back up our Skype conversations to SkyDrive using the SMEStorage Cloud File Server so they can be indexed and searchable as we do a lot of communication over Skype. TollFreeForwarding is used to give an international number and then route into the e-receptionist infrastructure.

Online Marketing: Google Adwords and BuySellAds.com and LinkedIn Ads

Google Adwords is synonymous with online marketing and can be a great sales tool if used correctly, and BuySellAds is useful to advertise services to targeted sites using banner ads. LinkedIn Ads is a great way to reach a very targeted audience and in our experience is a great way to compliment any online marketing campaign.

Invoicing Services: BlinkSale

Blinksale is a great, simple SaaS invoicing service that is low cost and very easy to use and administer. There are many others but for simplicity you can’t beat BlinkSale.

Blog: WordPress

There are many others, such as Google Sites, and Tumblr, but there are so many ways to use WordPress and so many plug-in’s that there really is nothing else to compete with it.

Social: Twitter and FriendFeed and Identi.ca and Google Plus and Hootsuite

We use a variety of social sites to push out news. There are of course so many now that it would be impossible to list them all but we covered the major ones. If you use multiple Twitter or social accounts then Hootsuite is a must.

……. and there you have it. The types of SaaS cloud services a distributed virtual company can use to run their business.

This is the third set of certifications Amazon has recently announced coming on top of VPC ISO 27001 certification and SAS 70 Type II certification.

The accreditation covers EC2 (Amazon Elastic Compute Cloud), S3 (Simple Storage Service), VPC (Virtual Private Cloud), and includes Amazon’s underlying infrastructure.

AWS’ accreditation covers FISMA’s low and moderate levels. This level of accreditation requires a set of security configurations and controls that includes documenting the management, operational and technical processes used in securing physical and virtual infrastructure, and a requirement for third-party audits.

Other vendors who recently announced FISMA certification recently where Google with Google Apps for Government and Microsoft with the Microsoft’s Business Productivity Online Suite among cloud services (although there was a spat between Microsoft and Google regarding these claims).

Expect to see further certifications as these are a pre-requisite of expansion into lucrative government and private sector contracts as vendors feels more comfortable choosing Cloud resources as commoditisation marches on.

“A user buys some software over the Internet or Shrink wrapped. They receive it and install it. It either does not work for them or they cannot figure out how to use it so they basically write off the cash and don’t use the App. End of story.”

Now lets looks at what happens on Cloud:

“The user either buys an Application from an App Store be it desktop or mobile, or a holiday from a holiday store, and then decided either the application is rubbish, does not work for them (or they have not RTFM) or has a bad experience on holiday. The user then use Social networks and/or the review forum on the App Store to comment on the bas experience”

In the latter case this “review” and negative experience puts off other people of trying the App / the holiday / the hotel etc. In some cases it can mean the difference between continually selling product or selling nothing as users look at the last bad review and then move somewhere else to continue their search to buy. One person can have the power to seriously undermine your whole product marketing and application strategy.

Even worse, many of the review forums (Apple’s App Store and Google Marketplace come to mind…) don’t even let you post a counter-review to explain that either the user has got it wrong, or misunderstood, or to genuinely offer to correct a bug. Worse still, some vendors can use a strategy of targeting competitive products to “put people off” purchase. In some cases this has led to the vendors involved seeking out legal action.

So what can you do to protect your product and your reputation ?

Well the first step has already been taken, you are at least thinking about it and conscious of it which is half the battle. What you should do is have your marketing or support team have a strategy that includes:

- Monitoring App Stores and review forums where your product features
- Monitoring social media for keywords about your product or company
- Set up Google Alerts keywords to inform you of keywords about your product and company
- Ensure you check your Twitter messages and also posts on LinkedIn and Facebook pages.  These are easily missed.

When you see responses, always make sure to try and follow up with the user and engage and resolve their issue, even if this means refunding them, Even if a refund seems like the last thing you want to do, offer the refund, it is not worth your reputation. As part of the process try and see if they will change their review, even if it is only to neutral.

The Cloud brings power to the masses in more ways than one and a single user can have a dramatic network effect on your business if you are not careful !