on existing slave:

copy data dir from /var/lib/mysql and data from /var/run/mysqld to new slave database:

copy /etc/my.cnf from old slave to new slave
add entry for new server-id

start existing slave:

start new slave:

on masterdb:
e.g.:

test on master:
create database repl;

check on slave:
show databases; /* should show new database */

test on master:
drop database repl;

check on slave:
show databases; /* new database should be dropped */

Now it’s time to turn this into an automated shell script with Expect in there.

So just to be clear. I wasn’t part of the crew responsible for scaling this site. I had already set up a scalable architecture for the site, that would automatically and horizontally scale at Amazon. That idea got shot down for legal reasons that to my surprise haven’t been in play for awhile. Can we say, “Office politics?”

I totally recommend Amazon’s Autoscaling to anybody that’s new to this.

Instead of auto-scaling, the site was architected by a local San Francisco firm who I won’t mention here.

Let’s just hope enough people read this so that they won’t even have to know the name of the company and will just know the smell of an un-scaleable architecture.

Scalability requirement: 100,000 concurrent users

This is how they set it up:

• two web servers
• one database
• four video transcoders that hits the master database
• one more app server that hits the master database
• no slave db

If they had even googled ‘building scalable websites’ they would have come across a book that would have avoided all of this, Cal Henderson’s Building Scalable Websites. It should be mandatory reading for anybody working on a large website, and it just scratches the surface.

So, how did we get to 600 concurrent users?

We tweaked mysql by putting this in /etc/m.cnf:

We ran siege and were able to get to about 300 concurrent users without breaking a sweat, but now apache was dying.

So we tweaked apache. We started out with this:

StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000

And ended up with this:

RAM and CPU were doubled.

Who the hell makes a web app without a slave database and calls themselves website architects? Apparently these guys did.

Please start following if you want to see if this web app can make it to launch.

When you create a provisioning profile for an iPhone app you get an app identifier. It looks like this:
KED2IOCNA.com.codebelay.barceftw

In order to get your iPhone app in development into your iPhone you got to have a bundle identifier. The docs that I’ve found so far tell you that your bundle identifier is the same as your app identifier. After struggling for close to 4 hours, I found out that your bundle identifier really is:
com.codebelay.barceftw

Just strip out the encrypted looking part and the dot (.) before com and you’re set.

It only took twitter about 3 days to recover from this.

Is there a faster way?

First let’s look at the current options:

• TweetBackup (NY Times)
• BackUpMyTweets
• If you are popular enough and folks raise a raucous, Twitter will go into the database back-up and restore your account into its pristine set up.

BackupMyTweets required too much info to get it working. No, you cannot have my gmail password.

I’ve tried Tweetbackup and they get kudos for using OAuth to make it easy to back your tweets up.

The 3rd option, begging Twitter, simply can’t scale and will only work for those few elites close to Twitter or popular enough. There isn’t a consumer solution.

How do we solve the problem of social media backup?

The great thing is the problem is:

• technical
• can have the same business model as insurance
• will gain recognition as more snafus happen

Once again, if you haven’t already, use BackUpMyTweets.

Here’s what I’m comparing:
drizzle r1126 configured with defaults, and
MySQL 5.1.38 configured with

./configure --prefix=/usr/local/mysql --with-extra-charsets=complex \
--enable-thread-safe-client --enable-local-infile --enable-shared \
--with-plugins=partition,innobase

which is really nothing complicated.

SQL query caching is turned off on both database servers. Both are using the InnoDB engine plug-in.

I’m running these benchmarks on a MacBook Pro 2.4 GHz Intel Core 2 Duo with 2GB 1067 MHz DDR3 RAM.

I wrote benchmarking software about 2 years ago to test partitions but I’ve since abstracted the code to be database agnostic.

You can get the benchmarking code at Github.

At the command-line, you type:

where 10000 is the number of rows allocated total, and 4 is the number of partitions for those rows.

You can type the same thing for mysql:

and get interesting results.

Here’s what I got:

MySQL

bash-3.2$ php build_tables.php 10000 4 mysql
Elapsed time between Start and Test_Code_Partition: 13.856538
last table for php partition: users_03
Elapsed time between No_Partition and Code_Partition: 14.740206
-------------------------------------------------------------
marker           time index            ex time         perct
-------------------------------------------------------------
Start            1252376759.26094100   -                0.00%
-------------------------------------------------------------
No_Partition     1252376773.11747900   13.856538       48.45%
-------------------------------------------------------------
Code_Partition   1252376787.85768500   14.740206       51.54%
-------------------------------------------------------------
Stop             1252376787.85815000   0.000465         0.00%
-------------------------------------------------------------
total            -                     28.597209      100.00%
-------------------------------------------------------------
20000 rows inserted...

drizzle

bash-3.2$ php build_tables.php 10000 4 drizzle
Elapsed time between Start and Test_Code_Partition: 7.502141
last table for php partition: users_03
Elapsed time between No_Partition and Code_Partition: 7.072367
-------------------------------------------------------------
marker           time index            ex time         perct
-------------------------------------------------------------
Start            1252376733.68141500   -                0.00%
-------------------------------------------------------------
No_Partition     1252376741.18355600   7.502141        51.47%
-------------------------------------------------------------
Code_Partition   1252376748.25592300   7.072367        48.52%
-------------------------------------------------------------
Stop             1252376748.25627400   0.000351         0.00%
-------------------------------------------------------------
total            -                     14.574859      100.00%
-------------------------------------------------------------
20000 rows inserted...

MySQL: 699 inserts per second
drizzle: 1372 inserts per second
As far as inserts go, drizzle is about 2 times faster out of the box than MySQL.

Long Live j.mp.

Here’s what you need to know to protect yourself and your company from sabotage:

Who’s popular? Yeah, I know. It sounds like highschool, but like then it’s still in important and socially real factor that’s now kept track of on social media sites.

What is your team’s weakness as perceived by those outside? By the team itself? A good manager can appease the two.

Whose skills are the most respected? Yup you have to get along with this douchebag, if she or he is one. Just create enough space between the two of you.

Any others?

1. You’ll need to write your own LDAP plug-ins to interface with any access control lists. E.G. where I work WordPress is used for corporate communications so an LDAP plug-in had to be written to make sure the right people saw the right information.

2. Migration can be expensive if you’re using EBS on the first go. On windows, and I’m not sure why, it can cost about $50 to migrate 2GB of data into EBS. In linux, it happens at a fraction of that cost and as advertised.

3. Windows can be very expensive. Although they say it’s 12 cents per hour per small instance beware of hidden costs like authentication services and SQL server. With both, you are using a server at the cost of $1.35 / hour, which IMHO could be run cheaper with just a small linux instance and do the same thing at 10 cents per hour.

I’m pretty sure that with the right Amazon EC2 set up you could run a cluster of servers for a Fortune 500 company for under $1000.00 (one thousand dollars) per month without the CapEX costs associated with new hardware.

If you have any more questions about Amazon EC2 in the enterprise I’d be happy to answer them. Please ask them in the comments below.

I was kinda amazed about how easy it was.

Prelims: Download the load balancer API software, auto scaling software, and cloud watch software. You can get all three at a download page on Amazon.

Let’s load balancer two servers.

elb-create-lb lb-example –headers \
–listener “lb-port=80,instance-port=80,protocol=http” \
–availability-zones us-east-1a

The above creates a load balancer called “lb-example,” and will load balance traffic on port 80, i.e. the web pages that you serve.

To attach specific servers to the load balancer you just type:

elb-register-instances-with-lb lb-example –headers –instances i-example,i-example2

where i-example and i-example2 are the instance id’s of the servers you want added to the load balancer.

You’ll also want to monitor the health of the load balanced servers, so please add a health check:

elb-configure-healthcheck lb-example –headers \
–target “HTTP:80/index.html” –interval 30 –timeout 3 \
–unhealthy-threshold 2 –healthy-threshold 2

Now let’s set up autoscaling:

as-create-launch-config example3autoscale –image-id ami-mydefaultami \
–instance-type m1.small

as-create-auto-scaling-group example3autoscalegroup \
–launch-configuration example3autoscale \
–availability-zones us-east-1a \
–min-size 2 –max-size 20 \
–load-balancers lb-example

as-create-or-update-trigger example3trigger \
–auto-scaling-group example3autoscalegroup –namespace “AWS/EC2″ \
–measure CPUUtlization –statistic Average \
–dimensions “AutoScalingGroupName=example3autoscalegroup” \
–period 60 –lower-threshold 20 –upper-threshold 40 \
–lower-breach-increment=-1 –upper-breach-increment 1 \
–breach-duration 120

With the 3 commands above I’ve created an auto-scaling scenario where a new server is spawned and added to the load balancer every two minutes if the CPU Utlization is above 20% for more than 1 minute.

Ideally you want to set –lower-threshold to something high like 70 and –upper-threshold to 90, but I set both to 20 and 40 respectively just to be able to test.

I tested using siege.

Caveats: the auto-termination part is buggy, or simply didn’t work. As the load went down, the number of the server on-line remained the same. Anybody have thoughts on this?

What does auto-scaling and load balancing in the cloud mean? Well, the total cost of ownership for scalable, enterprise infrastructure just went down by lots. It also means that IT departments can just hire a cloud expert and deploy solutions from a single laptop instead of having to figure out the cost for hardware load balancers and physical servers.

The age of Just-In-Time IT just got ushered in with auto-scaling and load balancing in the cloud.