I have forgotten almost everything that I used to know about electronics, and I didn’t know that much to begin with either. I’ve spent the last week or so getting a tentative reintroduction to the fundamentals. Frankly, a current running across a wire is still a novel and exciting thing for me. I used to build my own computers when I was a teenager, but I’ve lost interest in recent years in favour of computers that just work.

With that disclaimer out of the way, the idea of mucking around with Arduino has been stewing in the back of my head for a while. Arduino is an open source, hobbyist computer hardware hacking platform, and can be used for anything from automated plant feeding to powering remote control aircraft. A couple of weeks ago, I was passing by a local electronics store (Maplin), and, on a whim, decided to see if they had any Arduino kits in stock.

I had no real idea what I wanted to do, but I was pretty sure I could figure that out along the way.

Arduino Uno

The first thing I discovered is that there are several types of Arduino for various reasons, with the variable factors seeming to be the physical size of the board (weight/size is important in embedded systems), power of the microcontroller (probably not an issue when you’re starting out), and number of available connector pins (complexity of device you’re building).

Make: Arduino Bots and Gadgets

Available in the store I visited were the Uno and the Mega. There are quite a few other types, including the awesome-looking LilyPad (can be sewn into clothing), but the Uno seems like a good general starting point. I did not know this, however, and decided on balance that the slightly more expensive Mega was a good way to make sure I didn’t have to come visit the shop again too soon. I also found a surprisingly good book on a nearby shelf: Make: Arduino Bots & Gadgets published by O’Reilly. I flipped through the first few pages to discover what was not inside the Arduino box (but was still required to get it working) and discovered that a USB-B (fat, square connector) cable is a necessary component, so picked up one of those (the book, too) and made my first Arduino purchases.

I also picked up two small packets of jumper wires, an adjustable power adapter (3.3V up to 12V with interchangeable heads) and a precision screwdriver set, not really knowing what the heck I was going to use these for, but they all seemed like reasonable purchases. There was a device called an “ethernet shield” on the rack next to the Mega, but the name “shield” did not help me make any immediate logical connections so I ignored it (bit of sensory overload at this point).

I should also mention that the book had a litany of tools listed, which I decided to ignore in favor of getting familiar with the software environment of the Arduino first. I was short on time that day, and didn’t get much done other than installing the development software and running the blinking LED “Hello World”.

I started reading my newly acquired O’Reilly book the following evening, and it quickly became apparent that I had nowhere near enough basic electronics resources to do anything immediately satisfying. Wire cutters, wire strippers, alligator clips, breadboards, soldering irons… I dithered. I let about a week slip by before I decided to press on with the book.

Further reading rekindled my interest, I decided another trip to the electronics store was in order. This time, I picked up an Uno, an Ethernet Shield, a combined wire cutter / stripper and decided I was in business. Satisfied, I spent a good chunk of that evening hunting down and taking apart every old, broken and/or unused electronic device in my apartment to salvage for parts. This yielded me a couple of motors, a 3.7V rechargeable battery, some microphones, speakers, the core of a CD player, and, most promisingly, a pretty simple LCD screen salvaged from a Nokia 3310 (drivers and sample code are available).

Unfortunately, none of these were really morphing into project ideas in my head. Then, while chatting to colleagues the following day, the idea of recreating the “Derezzed on floppy drives” video I’d seen a couple of weeks previously came up, and I had my goal. I was ready for a grueling few weeks of trial and error bit manipulation, but it turns out that floppy drive music is a something of a micro community. One chap, Sammy1Am, seems to have kicked off the recent interest by releasing some software (Moppy) to let people stream MIDI sequences to their Arduinos, and manipulate the floppy drives. This was the software used to create the Derezzed video I’d seen. Not only that, but he had also put together a helpful tutorial video. What a hero!

First problem I encountered: no floppy drives. At least, none near me. Hell, my laptop doesn’t even have a DVD drive. Another week passed, and I visited my family for Easter. I figured I had a few useful bits and pieces in storage, so I rummaged around the attic and pulled out two floppy drives, an old PC power supply and a few other bits and pieces. I also borrowed a digital multimeter from my dad, but I haven’t used it much yet – have been using that 3.7V battery to test whether current is going through something so far.

Power supply turned out to be a dud, but it took me a good while to figure that out; damn thing would work for a little while, then go through periods of shorting out entirely. Eventually I decided that I could power the floppy drives via the Arduino itself, as it has a handy little power passthrough socket. Also, I have no idea how Sam does it, but hooking up wires salvaged from Cat5 to a floppy drive male pin is hard. Well, more precisely, it is really, really fiddly.

I gave up and decided that yet another trip to the store was in order; this time I purchased a couple of small breadboards, a box of variously sized jump wires, some little jumpers (like you get on the main board of your PC – if you’ve ever built one) and some plastic containers to store the small pile of parts I had been accruing. Now I had all the pieces assembled. I cabled everything up as well as I could, then tried to get Moppy working.

Moppy on OSX Lion does not work. Or, at least, does not work without some fiddling – it’s most of the way there. It’s been a long time since I’ve touched Java above the JVM, and I had to install “Netbeans IDE” to get the project built. I’ve forked the project and updated it with the changes that got it working for me, but I’m not comfortable pushing them back since it changes the assumed development OS from Windows to OSX, and that’s not something that should be pushed back upstream.

Eventually, this evening, I got the whole ensemble working together, and recorded my first test (one of the sample MIDIs included):

Now I just need to get a lot more floppy drives, and either Learn How to Music, or find a musician who can create MIDIs and wants to floppydrive-ify some music I’ve a bunch of ideas I want to muck around with on the software side too.

Bootstrapping from zero-electronics was a little expensive, but a lot of fun, and it’s been great to get into the hardware in a way I haven’t done for some time.

The (current) finished product

During initial development of an API client, sometimes you just want to see what the server is receiving in real time, so you can hack things together quickly. This script sets up a HTTP server which spits out the URL requested, along with header information and any POSTed parameters.

Ideally, you could have this running on a parallel screen as you code, so you can see requests as they come in. This could also be used for simple mocking out of responses, but I haven’t gone that far

receive.js on github.

Some of the background for this list can be read here, and for some further context, this is some of what I read to build something probably more powerful than websolr’s top tier offering (those guys are probably worth investigating before building your own cluster, by the way). There were some pretty “out there” requirements for Boards.ie though (potentially thousands of FQ permutations per search phrase, lots of big, ugly, old data, etc).

Some of the issues I ran into scaling Solr are relatively unique, but the general approach should be the same for everyone:

1. Learn how to gather information
2. Gather all the information you can
3. Analyze / Evaluate
4. Make incremental changes
5. Return to Step 2.

Figure out what you’re actually running

If you’re unfamiliar with the world of Java (or a rusty shade of green like me), you might be horrified surprised to discover that there are a few different implementations (let alone versions) of the Java Virtual Machine (JVM) available to you. What’s more, the best documented and supported one, the “Oracle” JVM (still documented almost everywhere as the Sun JVM) is probably not what you’re running if you’re running Ubuntu Server.

There’s also a difference between engineering numbers and product numbers, which may not be immediately apparent from the outset, and often they appear to be used interchangeably.

Understanding the JVM

Understanding Solr

http://www.lucidimagination.com/content/scaling-lucene-and-solr

Somewhat passive aggressive Lucid Imagination advice - http://www.lucidimagination.com/blog/2010/01/21/the-seven-deadly-sins-of-solr/
An example of some of the hilarious bureaucracy in Solr development - https://issues.apache.org/jira/browse/SOLR-1143

There’s probably plenty more, but those are the ones I have saved

At EngineYard, I hope to build upon my experience bringing scale to web applications within Distilled Media, and take it to a wider audience.

I’m looking forward to (and a little daunted by!) working with such an accomplished group of people, and helping to drive our manifesto. Our offices will be down on Barrow Street, near Facebook and Google, in the same building as DogPatchLabs and friends at Intercom, in an area that is being dubbed the “Silicon Dock.” I’m incredibly excited by the sort of buzz that is going to be around this place.

I’m not sure what combination triggers it, but when I was done brainstorming for an app name, I checked Twitter to see if the handle was taken, then registered the domain name. It was quite late and I didn’t want to start fiddling around with email aliases to get the Twitter account, so I decided to leave it since the application is only in early planning and layout stages.

A couple of days later, I decided to grab the account to start playing around with the Twitter API – couldn’t believe it when I discovered the account had been nabbed. No updates, no avatar – just a squatter. It seems obvious in retrospect that someone would be scraping domain registrations and comparing against Twitter handles (there was nothing but the .com taken and I’d never mentioned it anywhere), but it is immensely frustrating. The app is a Twitter-based tool, so having a .com and matching account name is essential.

What’s even more frustrating is that this seems to fall within Twitter’s acceptable usage policy. They have a section on namesquatting, but it states:

“Please note that if an account has had no updates, no profile image, and there is no intent to mislead, it typically means there’s no name-squatting or impersonation.”

You’d be hard-pressed to claim that there is an “intent to mislead” when you’re starting to build something – after all, the fact that you’re only starting to build it means there’s not much to mislead. So I’m back to the drawing board on figuring out a name while I put the app together.

The lesson here, a rather obvious one I suppose, is that Twitter handles are as valuable as the domain for certain classes of applications. If you’re building something that will interface with Twitter, you have to get the Twitter handle at the same time as the domain, or it’ll be swept out from under you.

Edit:
As per Andrew’s comment below, I’ve submitted an impersonation report – hopefully something will come of it.

Update:
Well, it looks like this might be going places, the account in question is now showing as “suspended” – no word from support yet (it’s a weekend, I wasn’t expecting anything until at least Monday) but it’s encouraging!

Update 2011-08-29:
So, received some communication from Twitter today; they will not release the account name to me. This is disappointing, but I will move on.

The reasoning is that there are “privileged” students who begin a computer science degree already knowing how to code, and that this is unfair.

Beyond the impracticality of doing this (you’re going to have to change the language every year so that people don’t “cheat”), the fact that so many educators in the comments agree with this position is further evidence that the current education system is creaking.

People like Salman Khan are already helping to make education more accessible with the likes of Khan Academy. In fact, teachers who have used Khan Academy apparently don’t want their students to be too smart.

When you are effectively looking to punish students for learning how to read before the rest of their peers, you have – at best – lost perspective as an educator.

At some point (actually around the time of the industrial revolution), it seems that education stopped being about the acquisition of knowledge and started being about churning out template humans to fulfil tasks.

If computer science education is broken because people are learning all by themselves, then society should (and already does) route around the problem – the problem being universities holding a monopoly on what it means to be educated. Tech companies in particular have been navigating around this for years, reaping the rewards of hiring self-taught, self-motivated individuals.

If modern enterprises wish to take advantage of this revolution in education, they should recognize this lest they lose some of the best candidates to more modern companies.

If you’re not familiar with Munin, it’s a bit like Nagios, (and if you haven’t heard of Nagios, it’s a network monitoring tool). Munin is particularly nifty because it’s easily extensible in pretty much any language that takes your fancy; Perl seems to be the favorite, but Python is also used.

You can check out the plugins on Distilled’s Github account (parent company of Adverts.ie, Boards.ie, Daft.ie and TheJournal.ie).

There’s a bunch more there that Conor McDermottroe has written for MySQL, Snort, IMPI and FreeBSD too which you might also be interested in!

The Bad

I think one way in which Google really dropped the ball is that Google Apps customers are having serious difficulties using the service. Google is undergoing a massive effort to unify various types of accounts over their network and are mucking it up quite badly for such a smart group of people. They risk alienating some of their most committed users; heck, they’re alienating the minority of users who actually pay.

Something else I’ve heard is that Google should have gone to more effort to allow groups of people to sign up – after all, critical mass is what is going to make this thing successful. Coming back to Google Apps, they had an excellent opportunity to pre-authenticate all Google Apps account holders for the service, thus enabling entire already dedicated groups to bootstrap the service.

Sparks feels like a bit of a miss. I’m not quite sure what mix of keywords is going to give me stuff I should know about. I’m probably trying too hard to recreate the experience of hackernews or Techmeme, but since I don’t know what “topic” to pick and I’m not sure how the results are populated, it feels… off.

The Good

Having the Google+ notification bar as part of my GMail bar is inspired. It’s amazing how in many ways this is not all that different from the time Buzz was lodged below our inboxes – yet it just feels so much more appropriate.

The interface is finally something between the dorky, utilitarian UI of older products and the sweeping strokes of pastel colours, Helveticaesque typefaces and thick lines we’ve become more familiar with in the last few years.

Profiles are very nicely executed. There’s just the right amount of subtle completionist bait to encourage people to add more information and fill out more circles, but without feeling like you’re being pressured into it (a la LinkedIn).

It feels like it has the opportunity to be a richer experience than Twitter (who are struggling to fit the rich world of the web into their 140 character ethos), while retaining the serendipity of interaction with strangers (for want of a more appropriate term) that isn’t really available with Facebook. It doesn’t seem like it would replace the organic, real time stream that is Twitter; but it might supplant a good portion of my (admittedly low) Facebook usage.

Overall, I think if the bootstrapping issues can be addressed then this is going to be a really useful addition to the space that Facebook, Twitter and LinkedIn occupy.

I haven’t owned a copy of Windows for years, and have been using Ubuntu as my solitary home and office OS for some time. Last week though, I decided I should have a copy of Windows 7 since there’s some software I can’t run without it (my hardware is pretty new, and it seems to take Linux about 6-12 months to catch up with changes in hardware, especially graphics).

So, after a bit of hunting, I discovered it was possible to purchase Windows 7 in downloadable ISO format at http://emea.microsoftstore.com

Great, I thought, that’s pretty progressive of Microsoft! Unfortunately their helpfulness ends there, as the only instructions for actually using the ISO revolve around DVD burners and/or assume you already have Windows (of some sort) installed. I didn’t buy a DVD drive with my last computer build because I haven’t actually needed to use one for a long time.

So, if, like me, you have Ubuntu as your only OS and no DVD drive, it seems like you’re kinda up the creek. Fortunately, this is actually pretty easily remedied. This guide does assume some familiarity with both Ubuntu and partitioning and I’m not big on tech support, so consider yourself forewarned

You need:

1. A USB key that you can afford to wipe completely (at least 4GB)
2. Your Windows 7 ISO of choice (32 bit and 64 bit varieties are available from the store) and the serial (it’ll be emailed to you).
3. A connection to the Internet on your Ubuntu machine (or just read the guide and preinstall anything you see mentioned)
4. A willingness to nuke whatever OS you already have installed, because Windows takes no prisoners (blindly overwrites the Master Boot Record).

Now, technically 4 can be alleviated by recovering grub, but it’s a bit of a roundabout process, and my Ubuntu machines are all basically thin clients. Important documents are stored in Dropbox, my code is on Github, my music, etc is on external drives and almost everything else is also stored remotely or run in the browser. If you don’t want to lose your Ubuntu install, I imagine the guide on the Ubuntu Community Wiki is where you’ll want to look after this.

Important: when you’re downloading the ISO, do not bother with “Part 1″ – it is a boot sector editor for Windows and will be useless to you. Download only “Part 2″ as it’s the ISO (you can only do this about three times, so try to download and store the ISO the first time round).

Partitioning the USB key

We’re going to prepare this USB key with gparted. Disk Utility won’t do (since it won’t allow you to set a device as bootable). You might have this installed already, but if you’re running a fairly fresh copy of Ubuntu, you’ll need to install this using Synaptic. Once it’s installed, you should find it under System->Adminstration->GParted Partition Editor.
Insert your USB key, start up Gparted and select the key from the drop down in the top right. It’ll be something like /dev/sdc, with the size of the drive in brackets next to it.
Delete any partitions that are on the drive, and create a new NTFS one, applying the changes.
Right click on the device, and select “manage flags”. Set the “boot” flag and nothing else.

Extracting the ISO

You may need to eject and re-insert the drive for it to appear, at this point you need to open up your freshly downloaded ISO using the Archive Manager. Do not “mount” the ISO as it will not be usable.
Extract the contents of the archive straight onto your newly formatted (and bootable) USB key. Once this is completed, eject the drive and voila, you have a bootable USB key with Windows 7 on it!

Boards.ie recently launched a new search engine – http://www.boards.ie/search/ – which is built upon Amazon Web Services using Solr with PHP as the glue.

Currently, Boards.ie users are searching nearly 30 million posts almost a million times a month.

A little background

Solr has been used in production by Distilled Media (formerly the Daft Media Group) for a couple of years now, first being tested on LetsRent.ie, then for powering the maps functionality on Daft.ie, and even more recently as the core technology for the relaunch of Adverts.ie.

Boards.ie’s usage has been an interesting look into the future for Daft.ie and Adverts.ie; with nearly 26 million posts – “documents” in Solr nomenclature – and growing by more than a million again every month, it presents a challenge in providing a reliable, affordable, relevant and fast search solution to our users.

Boards.ie has relied on MySQL’s “Fulltext” search as long as it has been using MySQL (forever). As time has passed, more and more restrictions had to be placed on search in order to keep it online, and the sheer volume of data was making results less and less useful to people well used to using Google search.

There have been a number of challenges to solve along the way:

Can we make search faster and more relevant with what we have already?

This is sort of an obvious question, but we didn’t want to jump the gun and leap into an entirely new technology stack if it were possible for us to overhaul what was currently there.

The answer was not particularly straight forward, but any solution that involved keeping the MySQL fulltext engine as our primary search system involved a lot of new beefy hardware and a lot of the same slowness, punting relevance issues down the line. As a result, we decided that it was not a realistic option.

Which search technology should we use?

There were only a couple of choices available to us when we started working on this project. First, we wanted to use Open Source – there are plenty of good reasons to choose either open source or proprietary software depending on your requirements, and we have experience with both. We wanted the flexibility to hack away at what we were using and share back to the community where possible. We wanted to be able to scale horizontally without worrying about skyrocketing licensing costs. We wanted access to the huge amount of expertise and goodwill that comes with open source projects, as well as experimental work done by other developers.

It also had to be compatible with the rest of our infrastructure and fit with our areas of experience – we’re pretty evenly split between FreeBSD and Linux on the infrastructure side; we didn’t want to introduce an entirely new stack (Microsoft) if it could be avoided.

There were two major options for us as a result, SphinxSearch and Apache Solr. SphinxSearch appeared to have successful deployments with vBulletin (our forum software), but all the experience in the Distilled Group was with Solr. Daft.ie had already deployed successful Solr installations, so we decided to go with what we had some experience with.

What infrastructure should we use?

We had the option of building using our own hardware, or making the leap to introducing cloud infrastructure. Moving partially into the cloud introduced a data transportation issue – how were we going to get data into the cloud while keeping our core infrastructure in Digiweb? Latency and data transfer costs were likely to be show stoppers. Amazon had recently introduced a new service in beta, Simple Notification Service (SNS). We were pretty sure we could combine this with their Simple Queue Service (SQS) to create a scalable message routing and queuing system for posts. With a few hiccups and some conversation with Amazon along the way, we were successful. As it turns out, getting the data out of our core infrastructure and into Amazon is the cheapest component of the entire operation. For the tens of thousands of messages we send to Amazon on a daily basis, we’re billed a couple of dollars a month.

What are the tradeoffs?

There are always tradeoffs.

The advantages and disadvantages of SQL and NoSQL solutions has been covered at length, I’ll just highlight the areas that were significant for us.

With MySQL, our data was/is potentially consistent. For our usage, it is consistent enough for us to call it the canonical source of our data. Occasionally it chews up a post or thread, but rarely one that can not be rewritten.

The information is also effectively instantly available upon submission. There’s a lag of up to a couple of seconds occasionally as the data waits to propagate across our MySQL cluster due to an extended lock of some sort, but it’s generally not a particularly noticeable to the average user.

With the new search system we have sacrificed some consistency and how immediately data is made available to the searcher. For instance, posts sometimes (more frequently than with MySQL) do not make it into the search system. There is multiple redundancy built into the system to limit this, but occasionally a full resynchronization is required. We judged this to be an acceptable cost.

Immediacy has been sacrificed such that it usually takes about 2-5 minutes for a new post or update to an old one to become available in searches, occasionally an hour, infrequently a day, and very infrequently a couple of months (relating back to consistency). We determined that the normal search profile does not require the absolute newest data, only the most relevant.

At the time we made these decisions, losing real time search was the one I was least happy about, for most users it does not appear to be a concern. In hindsight it’s almost obvious why – Boards.ie is a gigantic repository of historical information. The value is in being able to search this rich back catalogue of conversation, opinion and information, not just the most recent.

Even at that, recency is only sacrificed when measured in seconds.

Architectural choice

As useful as Solr has been for us, it’s a bit of a black box in our architecture. Trying to run Solr through a debugger remotely is a gigantic pain in the arse and not something I had much success with. Fortunately, the Jetty error logs are enough to illuminate most problems:

• Memory management. It’s perhaps unfair to take issue with Solr over this – after all, it’s a far cry from MySQL’s famously labyrinthine memory usage configuration options, and Solr’s problems are really the restrictions of the JVM – but it feels like a single purpose machine with one major application should be able to figure out how much RAM to dedicate to disk cache and how much the application should get for optimal performance.
• XML everywhere. It’s inescapable, and again, this can be ascribed to Java culture, but damn.
• Fixed schema. I suppose this is an ideological argument, but during development this was tedious.
• The book is two inches thick and you need to know it. If you’re building a search service that you expect to take a lot of traffic and contain a lot of documents (why else would you be interested, right?) you will simply have to know all (or at least a good chunk of) the features, quirks, optimizations and architectural choices.

Solr is now a pretty well field-tested application in Distilled, and I’m pretty sure I could rapidly prototype an installation and have it up and running in production inside a week for another site, but I would prefer to investigate ElasticSearch the next time I am revisiting search options.

Further development

During the course of development, I wrote a small node.js server to speed up the relay of post data from our web servers to Amazon SNS. Due to the limitations of deploying node.js on our FreeBSD architecture at the time, this system is, unfortunately, yet to be implemented. Once it is, however, write operations on the site should speed up noticeably for end users. Once this is up and running and has the bugs ironed out, it will be released as an open source project on github.

A side effect of this is that I contributed some code to the AWS Library node.js project, something I would like to continue doing.

I have also amended chunks of the Solr PHP client with a sharded Solr deployment in mind, but it’s quite clunky and I would prefer to have another stab at it, maybe writing my own Solr PHP client and making it available.

In the next posts on this topic, I plan to dive more deeply into the specifics of our implementation, and hopefully release some code in tandem.