At TempoDB, we maintain multiple environments (production, staging, etc), and each environment is in a datacenter (Dallas, Seattle, etc). For the most part, we want strict separation between environments, but we have a growing list of traffic that ought to be allowed to flow between them (see below). We designed a new architectural primitive which allows us to securely permit some traffic, while still blocking everything else.

Business Objectives
Right now, each environment (production/staging) is in one datacenter (Dallas/Seattle). What we developed allows us to connect two datacenters, which means that one environment can span multiple datacenters (intra-environment tunnel) or that two datacenters from different environments can be connected (inter-environment tunnel; eg a connection between production and staging). This allows features such as:

• The ability to set up streaming replication (master/slave) from production to staging for both HBase and postgres
• The ability to easily load a complete backup of production into staging, for developer load-testing (made much easier by streaming replication)
• The ability to expand into other data centers on the SoftLayer internet backbone, in order to expose international Points of Presence (POPs). These POPs wouldn’t be full environments, but lightweight proxies that move the SSL handshake closer to the customer, and then rely on faster (but still encrypted) communication within our system

Such a tunnel must satisfy a few expectations:

• It must provide transparent encryption to all traffic going through it
• It must have a firewall on both ends, and disallow everything except explicitly allowed traffic
• It must be highly available

OpenVPN & Vyatta
There are a number of tunnelling options available for linux, and we went with OpenVPN because it suits our needs (everything is IP, authentication makes sense for us), it’s flexible (supports site-to-site, and remote access, which we’ll want as well), and it’s fairly standard. We also choose to use a Vyatta rather than mix together everything we need on a standard linux box. The reason for this is complexity management; Vyatta is a linux system with a bunch of networking/routing tools that are designed to work well together, including OpenVPN, VRRP (Virtual Router Redundancy Protocol, for high availability), and stateful firewalls.

Solution Overview
Below, you can see the diagram of where we ended up. Each datacenter has a pair of vyatta servers, and each pair exposes a public and a private IP (via a VIP). An encrypted openvpn tunnel is maintained between the two public vips, and it can fail-over automatically if a box goes down. There is a firewall on both ends of the pipe which specifies a short list of allowed traffic types.

Routing
Having the tunnel only allows the Vyatta servers to talk across the pipe. In order for other servers to make use of the connection, they have to have static routes locally which use the vyattas as a gateway for traffic bound for the other datacenter (see the above image for an example). In other words, most servers don’t know anything about the tunnel. They just know that vyatta can send data to the other datacenter. The default gateway for each server is controlled by our datacenter, otherwise we could have just put the routes there (instead of propagating them to every box). Alternatively, we could set up our vyatta cluster as the default gateway, which we may do at some point.

High Availability
We use VRRP for high availability of Vyatta. The way it works is that a pair of Vyatta servers share a public and a private Virtual IP (VIP). The VIPs float (as a pair) between the two servers as appropriate (if one fails, both VIPs move, and the new master re-establishes the connection).

Arity
The setup we’ve discussed and diagrammed is for connecting two datacenters, but it is likely that we’ll want multiple connections to a single datacenter. Conveniently, this doesn’t require much extra work. Using the existing router pair and the existing VIP pair, it’s just a matter of adding an extra OpenVPN config entry onto the routers. Using this strategy, there is little extra effort to go from 1 to N connections.

What’s Next?
For us, an inter-datacenter tunnel is an architectural primitive which we can use in other projects. In later posts, we’ll talk about how TempoDB uses this to expand our production environment into multiple data centers (Points of Presence), as well as set up a channel between environments (streaming replication between production and staging environments).

The platform I’m using is XCP (Xen Cloud Platform; an open-source alternative to Citrix XenServer). It requires the whole drive, so dual-booting isn’t an option (see note below: Dual Booting), which gave me problems. I could install it just fine (hold down ‘c’ while booting to boot from CD), but on reboot I would just get a flashing file folder icon with a question mark in the middle. I finally found the solution on the Ubuntu docs site. After installation is complete, reboot the computer and hold alt/option to get into the recovery partition (it takes a while to download the tool, presumably because they were on the partition that XCP blew away). When you get to the screen showing different options (restore from Time Machine, reinstall OSX, etc), go to Utilities > Terminal (at the top of the screen). Once in a terminal type: bless --device /dev/disk0s1 --setBoot --legacy
. Then reboot (sudo shutdown -r now if you’re still in Terminal), and you should be in XCP.

legacy
According to the online help (in the recovery Terminal, type ‘bless’ without any arguments), “–legacy” isn’t actually a valid argument. I assume that this is correct, and that the Ubuntu docs are wrong/outdated. Still, I included it and Terminal didn’t complain. It probably isn’t necessary, but I didn’t test the command without it. If you try it, let me know and I’ll update the post.

Dual Booting
Instead of overwriting the whole disk (and needing to ‘bless’ it), you could try dual-booting OSX and debian/ubuntu, then installing XCP within that linux environment. According to the docs, this is “*very much* a work-in-progress” (emphasis theirs), so I wanted to avoid it. That said, I bought this computer soley for Xen, so I wasn’t interested in dual-booting. If you don’t want to lose your OSX installation, this may be an option to explore.

What’s a Honeypot?
A honeypot is a fake service that runs on a standard port, waiting for people to try to break in. There are lots of different kinds, covering different protocols and interaction levels. I’m using kippo, which is a medium-interaction (or “research”) honeypot. If someone spends enough time guessing, they’ll eventually get in, and once in they are in a protected area where all their actions are recorded for future analysis. This yields a treasure trove of data, including patterns of who is trying to break in, how are they going about it, and what do they do if they are successful. For a production environment, honeypots may still be a good idea, but a low-interaction environment would be more appropriate (meaning people can attack it all they want, but will never get into anything).

The Data
As of Feb 6, 2013, I have 5 servers running, across two data centers (AWS and Digital Ocean). I started them at various times, all around Feb 3, 2013. The data is shown here on a globe, with markers where attacks have originated. Because the boxes were started at various times, and the results are aggregated, the location of the markers are more relevant than the magnitude. That said, you can also see the data in table form.

Why use a Honeypot?
Honeypots help you gain insight into a wave of attacks that is constantly going on. For instance, there’s a big difference between seeing login attempts for normal system accounts (root, guest, etc) and seeing login attempts for specific employees (john.smith). In the latter case, it may be an indication that you are being targeted specifically, rather than just randomly attacked, and you may want to react differently. You can also get some of this information without a honeypot (for instance ubuntu machines record login failures in /var/log/auth.log).

Should I run a honeypot?
As with any service you expose, running a honeypot increases your surface area on the Internet, so it’s not necessarily a good idea on any particular server. The servers I’m running to collect this data are all single-purpose boxes, with no valuable/sensitive information, and if the boxes actually did get compromised, it wouldn’t be a big deal (they are AWS and Digital Ocean instances). For a box containing data/services you care about, while I wouldn’t use a research (eg medium- or high-interactivity) honeypot, I could see an argument for a low-interactivity one. Given that your servers’ are probably being attacked daily, it might be safer to move your SSH port and put a honeypot on the default port, so it can handle the malicious activity. Only moving your SSH port (and leaving port 22 unused) is another option, but in the case where you’re being specifically targeted, having the default port closed would probably just prompt the attacker to port scan you to find the service.

What Next?
I’d like to bring up other honeypots (on the same servers probably), and compare the relative frequency of attacks against different protocols (I suspect HTTP(S) is the most attacked).

In addition to our REST API and language-specific client libraries, we now offer the ability to bulk import data by uploading CSVs. The intent of this feature is to support the initial load of large amounts of historical data (many millions or billions of data points). By sending us CSVs (instead of just using our API normally), customers save themselves from having to build and monitor a large one-time job, and the problem is simplified to CSV generation.

How It Works
Behind the scenes, we leverage the immutability of HBase’s (our main time series data store) underlying data files, as well as some of the distribution primitives offered by our Hadoop cluster. We use HDFS (a distributed file-system), MapReduce (Hadoop’s distributed batch job framework), and Scala to transform the provided CSVs into HFiles, which is the format HBase uses internally. After generating these HFiles, it’s just a matter of directing HBase to add these files to its collection, which is a fast and efficient process.

Immutability
HBase has an interesting property which permits us to create and load this data out-of-band: the underlying data files of HBase are immutable. While the data in HBase is itself mutable (you can update records as you see fit), the HFiles which hold the data don’t change (with the exception of compactions, which can be ignored here). It achieves this by searching HFiles in reverse chronological order for relevant values, so updates (in more recent HFiles) will be found and returned instead of older (out-dated) values. For our data load, by telling HBase to add the generated files to its cache, we’re effectively doing a single bulk write to the database, with the entire data set. In reality, due to the distributed nature of HBase, we’re not actually talking about a single bulk write, but one per server (where each server holds a portion of the data) but the general idea is the same.

Distribution and Fault Tolerance
At a fundamental level, we work hard to run only horizontally scalable systems, and part of that is the understanding that although all sorts of things can and will go wrong, the system as a whole needs to continue running. For a large data set (let’s say 10 billion data points), the batch job to process the CSVs and generate HFiles can take hours or days, and at a time span like that, getting an error and starting over would be very painful. To address this, we leverage the built-in fault-tolerance and distribution mechanisms provided by Hadoop’s MapReduce framework. This system, coupled with HDFS, means that every step of the process can handle various server/networking/random problems, and the job as a whole will recover.

This feature is new, and (at the moment) we accept a fairly limited CSV format. More info on how to bulk import your data can be found in our Support Center here. We’re eager for feedback on how you’d like to see it fit into your process. Drop us a note to discuss the specifics of your workflow at support@tempo-db.com.

Initial Marketing Efforts
When I opened my store, I used a variety of strategies to get visitors. I was active in the Etsy community forums, I participated in link exchange programs, and I discussed my work in communities outside Etsy (reddit, my blog, etc). I did Etsy’s paid advertising program, and I even advertised on a popular home decor blog. You can see below how effective this was for me, and you can also see the impact of me stopping my efforts, in March. The top (purple) line is visitors to any product in my store, and the bottom (blue) line is visitors to my shop’s homepage.

Why I Quit
I do not have a background in marketing, so a lot of what I was doing was guesswork (mostly based on tips from other sellers). One notable bit of advice was that once you hit some threshold of catalog size, you start to take off. The numbers people gave were different (50/100/200 items in your shop), but the general idea seems to be that having a large catalog has a big impact. I did increase my offerings (I started with 3, and added a couple a week), but I was a long way off from the numbers people were talking about. All of this was a lot of work, and the 3 sales a month I averaged over the first three months was fun and exciting, but didn’t really justify the effort.

Performance After Quitting
I stopped all marketing efforts in March, 3 months into my shop. I still managed the shop, but didn’t add many new products or try to drive in visitors. You can see in the graph above that I did maintain a decent level of visitors, though. My sales (see above) dipped for a few months, then in the summer they started slowly going up (I don’t know why). This resparked my interest, but the only tactic I’ve used since then has been adding products to the store (mostly in August). This month, I’ve hit the first threshold (50 items in my shop), and whether or not there is something special about the number, hopefully the general increase in products continues to improve my sales.

Favorites Don’t Indicate Sales
On Etsy, visitors can “favorite” a product (or an entire shop). Take this metric with a big grain of salt, because people may have all sorts of reasons for favoriting (doesn’t necessary indicate they are more likely to buy), but it does mean you did something to catch their attention for a moment. You can see below that there is not much of a change in favorites in March, when I stopped driving in visitors. Furthermore, the shape of the favorites graph doesn’t really look anything like the orders graph, meaning there isn’t much of a correlation between favorites and sales. I could see an argument for a relationship between the shape of the favorites graph and the visitors graph, though it seems fairly weak.

“Low Quality” Visitors Lead To A Low Conversion Rate
What I get from all this is that for the first three months I was driving in lots of low-quality visitors (meaning people unlikely to buy), and my resulting conversion rate (how often I turn a visitor into a purchaser) was therefore quite low. I read a book on online marketing a while back which talked about (among other things) improving your conversion rate by increasing the quality of your visitors. In my case, any visitor from outside Etsy is less likely to buy simply because there’s a high barrier to entry (creating an account, feeling comfortable buying from a stranger on Etsy, etc). My on-Etsy efforts during the early months tended to target other shop owners, which is better, but still a little off the mark. My latest efforts (adding more products, improving pictures & descriptions) are not going to drive in hordes of visitors, but the people that do find them (mostly through targeted searching) will be much higher quality for me.

Step 1: Extract Graphing Code
To start with the 2-project refactoring, I created a new (third) project: Timewise. This node.js project periodically polls a set of configurable endpoints, each of which serve up a single XML page of RRD’s built-in serialization. The XML gets read in and deserialized to a new RRD file, and this vastly simplifies the postgres replication hacks I had in place for Power Hungry. For now, the project consists only of code I yanked out of Hot or Not, but and right now that’s the only project it serves. This refactoring, though, sets up Power Hungry for the refactoring (to RRD) that is coming its way.

I left the HVAC/thermostat code on the master branch of the code repository, and put my thermometer-only code on a new branch.

Starting Point
To start, I threw the site up on my Linode, using nginx. The server runs a bunch of other sites, in a variety of languages, all with pretty low volumes. The server has keep-alive enabled, but not much else has been tweaked.

Hosting-As-A-Service
I looked at hosting on S3, Cloudfront (amazon’s CDN), and Rackspace’s CDN. Using apachebench (from slicehost), I compared the performance between all 3 of those options, with my nginx as the baseline. S3 was the slowest (slower even then nginx on my little VPS). Cloudfront was comparable to my nginx, and Rackspace would spike to about double the requests per second of my linode, but would also dip to slower than my nginx. Rackspace also has two big drawbacks. It doesn’t let you auto-serve index.html when a directory is requested, so I would still have to run nginx for the html files (as opposed to amazon, which supports this). Rackspace also doesn’t support directories. There are work-arounds, but it’s a pain that I can’t just mirror my directory structure on their cloud. Overall, the cloud hosting solutions were a let-down, and I decided to just host it myself, on linode/nginx.

Testing Methodology
I used Chrome’s debugging tools (with browser caching turned off) to get a sense of pageload times. I would refresh the page several times, and make a note of the spread of times. Not entirely scientific, but good enough for a rough idea of the relative improvements I was making.

External Resources
I had several external resources being referenced. I was sourcing some images from my Etsy store, using a public CDN for jquery, google fonts, etc. I pulled all those into my own repo, rather than rely on whatever bandwidth these providers want to give out for free. I didn’t see any speed change from this, probably because the files in question were pretty small.

Minify CSS & JS
My site goes through a script to convert it from templates and code into static files. As part of this process, I added minifying the CSS and JS files. I combined all the JS files into one file, then ran it through yahoo’s yui compressor, and did likewise for the CSS. Javascript was compressed by 14%, and CSS was compressed by 23%. Page load times dropped from around 1 second to around 700ms.

Resize Images
The same images appear in a number of places on my site, and I was using CSS to change the size. I switched to having the script generate different versions of the image, for every size I needed. This drastically reduced the filesizes, and dropped the pageload time down another 100ms to around 600ms.

Asset Server Pool
The last thing I added was an asset server pool. Browsers limit the number of connections they’ll make to a given server. I made a CNAME DNS alias for the site’s url (so the same nginx serves both domains the same), and sent half the assets (images, js, css, etc) to the alias. This bought me another 100ms, and left me at around 500ms.

Pageload Accuracy
The pageload times I was measuring were definitely anecdotal, and heavily influenced by my internet connection, laptop resources, etc. They are used more to show the relative improvement, rather than as a reliable indication of the speeds users will see.

The Code
All the code and project setup described here is on github.

Goals
I looked at a few options to get my own eCommerce site up and running (Shopify, etc), but everything was really out of my price range (for the volumes I’m doing). There are some open source projects that look interesting, but nothing that seemed a clear winner. For my needs, I just wanted something simple, fast, and secure. Static assets with all the logic client-side gives me that.

Upsides
Static assets are served faster than anything server-side-dynamic, and they require less memory (from the server). Ultimately, the whole site could be thrown on a CDN backed by S3, and I wouldn’t have to manage hosting at all. Because I control the code and the hosting, I have a ton of flexibility for future goals, too (I have several ideas on my TODO list).

Downsides
Having a Javascript shopping cart means that I am not selling to people that don’t use Javascript. About 2% of people in the US have javascript disabled (lower in other countries), and of those most seem to be the super-security conscious or older mobile phones. Not supporting these groups is a price I have to pay with my setup. The other significant downside is that I have to do manual confirmation of the purchases (amount, tax, etc), because the requests could be tampered with. This problem is not specific to my setup, and is not a big deal with my volumes. It doesn’t take a ton of effort, so I imagine this strategy will scale for quite some time before I need to come up with something different.

Tooling
I looked at Jekyll (a similar tool for generating static blogs) for this. I have used Jekyll for non-blog sites before, but in this case I found myself not only wanting to skip the blog-specific features, but also add in eCommerce-specific features. I pretty quickly decided that Jekyll wasn’t the right tool for this, so I built Small Store. It takes a list of products and uses a template to generate product pages. It’s very much a work-in-progress, and needs a lot of cleaning up (and additional features). I’ve tried to pull everything specific to my site into config/ but I’m sure I’ve missed some things. Feel free to check it out if it might match your needs.

Sales Goal
My goal is 1 sale per week. I picked this goal for two reasons. First, it is beyond what I am currently selling, but not that far off (some weeks I make a sale, some weeks I don’t). Second, the 1-week time period is long enough to try out a new marketing strategy or two, but short enough that I have to be really organized and diligent. As soon as a new week rolls around, regardless of the previous week I am ready to go gung-ho towards a sale.

When I Hit My Goal
Hitting a goal is great. It is encouraging, and renews my excitement about this project. I start the week really working hard on making a sale, so when I do make one, I sometimes take the rest of the week to relax and let some product ideas marinate. Other weeks, I spend my extra time working on more long-game marketing strategies, like reaching out to bloggers that might be interested in my product.

When I Miss My Goal
Missing my goal sucks. My week starts on Monday, so as the weekend rolls around I start to get really worried if I haven’t made a sale. I try less targeted marketing strategies, even though they never really seem to pay off.

New Week, New Goal
When a week ends, its success or failure is in the past. If I missed my goal last week, I don’t worry about catching up this week, and if I sold two things last week, it doesn’t make me any less hungry for a sale this week. This is really important for me to stay calm and sane, but on the other hand, if I have a somewhat sustained series of successes or failures, I’ll know I need to reexamine my goals.

Long Game vs Short Game
My goal of 1 sale per week is evaluated every 7 days, but I keep in mind that it is really a long-term goal (“1 sale *every* week.”). For sustained success, I have to look beyond the current 7-day period, so I usually spend some time in the middle of the week working on new product ideas. The new items won’t make it into the store during this week, but hopefully they will help me next week or the week after.