Kyle Baley :: The Coding Hillbilly

Outside the shack, or “How to be a gigolo”

Wed, 08 Jul 2015 01:44:51 +0000

Almost four years ago, I waxed hillbilly on how nice it was to stick with what you knew, at least for side projects. At the time, my main project was Java and my side projects were .NET. Now, my main project is .NET and for whatever reason, I thought it would be nice to take on a side project.

The side project is Western Devs, a fairly tight-knit community of developers of similar temperament but only vaguely similar backgrounds. It's a fun group to hang out with online and in person and at one point, someone thought "Wouldn't it be nice to build ourselves a website and have Kyle manage it while we lob increasingly ridiculous feature requests at him from afar?"

Alas, I suffer from an unfortunate condition I inherited from my grandfather on my mother's side called "Good Idea At The Time Syndrome" wherein one sees a community in need and charges in to make things right and damn the consequences on your social life because dammit, these people need help! The disease is common among condo association members and school bus drivers. Regardless, I liked the idea and we're currently trying to pull it off.

The first question: what do we build it in? WordPress was an option we came up with early so we could throw it away as fast as possible. Despite some dabbling, we're all more or less entrenched in .NET so an obvious choice was one of the numerous blog engines in that space. Personally, I'd consider Miniblog only because of its author.

Then someone suggested Jekyll hosted on GitHub pages due to its simplicity. This wasn't a word I usually assocated with hosting a blog, especially one in .NET, so I decided to give it a shot.

Cut to about a month later, and the stack consists of:

Jekyll
GitHub Pages
Markdown
SASS
Slim
Rake
Travis
Octopress
Minimal Mistakes Jekyll theme

Of these, the one and only technology I had any experience with was Rake, which I used to automate UI tests at BookedIN. The rest, including Markdown, were foreign to me.

And Lord Tunderin' Jayzus I can not believe how quickly stuff came together. With GitHub Pages and Jekyll, infrastructure is all but non-existent. Octopress means no database, just file copying. Markdown, Slim and SASS have allowed me to scan and edit content files easier than with plain HTML and CSS. The Minimal Mistakes theme added so much built-in polish that I'm still finding new features in it today.

The most recent addition, and the one the prompted this post, was Travis. I'm a TeamCity guy and have been for years. I managed the TeamCity server for CodeBetter for many moons and on a recent project, had 6 agents running a suite of UI tests in parallel. So when I finally got fed up enough with our deploy process (one can type `git pull origin source && rake site:publish` only so many times), TeamCity was the first hammer* I reached for.

One thing to note: I've been doing all my development so far on a MacBook. My TeamCity server is on Windows. I've done Rake and Ruby stuff on the CI server before without too much trouble but I still cringe inwardly whenever I have to set up builds involving technology where the readme says "Technically, it works on Windows". As it is, I have an older version of Ruby on the server that is still required for another project and on Windows, Jekyll requires Python but not the latest version, and I need to install a later version of DevKit, etc, etc, and so on and so forth.

A couple of hours later, I had a build created and running with no infrastructure errors. Except that it hung somewhere. No indication why in the build logs and at that moment, my 5-year-old said, "Dad, let's play hockey" which sounded less frustrating than having to set up a local Windows environment to debug this problem.

After a rousing game where I schooled the kid 34-0, I left him with his mother to deal with the tears and I sat down to tackle the CI build again. At this point, it occurred to me I could try something non-Windows-based. That's where Travis came in (on a suggestion from Dave Paquette who I also want to say is the one that suggested Jekyll but I might be wrong).

Fifteen minutes. That's how long it took to get my first (admittedly failing) build to run. It was frighteningly easy. I just had to hand over complete access to my GitHub repo, add a config file, and it virtually did the rest for me.

Twenty minutes later, I had my first passing build which only built the website. Less than an hour later and our dream of continuous deployment is done. No mucking with gems, no installing frameworks over RDP. I updated a grand total of four files: .travis.yml, _config.yml, Gemfile, and rakefile. And now, whenever someone checks into the `source` branch, I am officially out of the loop. I had to do virtually nothing on the CI server itself, including setting up the Slack notifications.

This is a long-winded contradiction of my post of four years ago where my uncertainty with Java drove me to the comfort of .NET. And to keep perspective, this isn't exactly a mission critical, LOB application. All the same, for someone with 15-odd years of .NET experience under his obi, I'd be lying if I said I wasn't amazed at how quickly one can put together a functional website for multiple authors with non-Microsoft technology you barely have passing knowledge of.

To be clear, I'm fully aware of what people say about these things. I know Ruby is a fun language and I feel good about myself whenever I do anything substantial with it. And I know Markdown is all the rage with the kids these days. It's not really one technology on its own that made me approach epiphaniness. It's the way all the tools and libraries intermingle so well. Which has this optimistic hillbilly feeling like his personal life and professional life are starting to mirror each other.

Is there a lesson in here for others? I hope so as it would justify me typing all this out and <s>clicking publish</s> committing to the repository. But mostly, like everything else, I'm just happy to be here. As I've always said, if you've learned anything, that's your fault, not mine.

Kyle the Coalescent

* With credit to Brendan Enrick's and Steve Smith's Software Craftsmanship Calendar 2016

On UI Testing

Tue, 30 Jun 2015 14:39:22 +0000

A short while ago, a group of Devs of the Western variety had a chat. It was the latest in a series, depending on how you define “series”, where we gather together to discuss some topic, be it JavaScript frameworks, OO practices, or smoked meat. On this particular day, it was UI testing.

I don’t recall all the participants but it was a good number of the people on this list. Here, I’m going to attempt to summarize the salient points but given my memory, it’ll more likely be a dissertation of my own thoughts. Which is just as well as I recall doing more talking than I should have.

Should you UI test?

This was a common thread throughout. Anyone who has done a significant amount of UI testing has asked a variant of this question. Usually in the form, “Why the &*%$ am I doing this?”

Let it not be said that UI testing is a “set it and forget it” affair. Computers are finicky things, UI’s seemingly more so. Sometimes things can take just that one extra second to render and all of a sudden your test starts acting out a Woody Allen scene: Where’s the button? There’s supposed to be a button. YOU TOLD ME THERE WOULD BE A BUTTON!!!

Eventually, we more or less agreed that they are probably worth the pain. From my own experience, working on a small team with no QA department, they saved us on several occasions. Yes, there are the obvious cases where they catch a potential bug. But there was also a time when we had to re-write a large section of functionality with no change to the UI. I felt really good about having the tests then.

One counter-argument was whether you could just have a comprehensive suite of integration tests. But there’s something to be said for having a test that:

1) Searches for a product 2) Adds it to the shopping cart 3) Browses more products 4) Checks out 5) Goes to PayPal and pays 6) Verifies that you got an email

This kind of integration test is hard to do, especially when you want to verify all the little UI things in between, like whether a success message showed up or whether the number of items in the shopping cart incremented by 1.

We also had the opposite debate: If you have a comprehensive suite of UI tests and are practicing BDD, do you still need TDD and unit tests? That was an interesting side discussion that warrants a separate post.

Maintenance

…is ongoing. There’s no getting around that. No matter how bullet-proof you make your tests, the real world will always get in the way. Especially if you integrate with third-party services (<cough>PayPal<cough>). If you plan to introduce UI tests, know that your tests will be needy at times. They’ll fail for reasons unknown for several consecutive runs, then mysteriously pass again. They’ll fail only at certain times of the day, when Daylight Savings Time kicks in, or only on days when Taylor Swift is playing an outdoor venue in the western hemisphere. There will be no rhyme or reason to the failures and you will never, ever be able to reproduce them locally.

You’ll add sleep calls out of frustration and check in with only a vague hope that it will work. Your pull requests will be riddled with variations of “I swear I wouldn’t normally do this” and “I HAVE NO IDEA WHAT’S GOING ON”. You’ll replace elegant CSS selectors with XPath so grotesque that Alan Turing will rise from his grave only to have his rotting eyeballs burst into flames at the sight of it.

This doesn’t really jibe with the “probably worth it” statement earlier. It depends on how often you have to revisit them and how much effort goes into it. From my experience, early on the answer to that is often and a lot. As you learn the tricks, it dwindles significantly.

One of those tricks is the PageObject pattern. There was universal agreement that it is required when dealing with UI tests. I’ll admit I hadn’t heard of the pattern before the discussion but at the risk of sounding condescending, it sounds more like common sense than an actual pattern. It’s something that, even if you don’t implement it right away, you’ll move toward naturally as you work with your UI tests.

Data setup

…is hard, too. At least in the .NET world. Tools like Tarantino can help by creating scripts to prime and tear down a database. You can also create an endpoint (on a web app) that will clear and reset your database with known data.

The issue with these approaches is that the “known” data has to actually be known when you’re writing your tests. If you change anything in it, Odin knows what ramifications that will have.

You can mitigate this a little depending on your technology. If you use SpecFlow, then you may have direct access to the code necessary to prime your database. Otherwise, maybe you can create a utility or API endpoints that allow you to populate your data in a more transparent manner. This is the sort of thing that a ReST endpoint can probably do pretty well.

Mobile

Consensus for UI testing on mobile devices is that it sucks more than that time after the family dinner when our cousin, Toothless Maggie, cornered—…umm… it’s pretty bad…

We would love to be proven wrong but to our collective knowledge, there are no decent ways to test a mobile UI in an automated fashion. From what I gather, it’s no picnic doing it in a manual fashion. Emulators are laughably bad. And there are more than a few different types and versions of mobile device so you have to use these laughably bad options about a dozen different ways.

Outsourcing

What about companies that will run through all your test scripts on multiple browsers and multiple devices? You could save some development pain that way. But I personally wouldn’t feel comfortable unless the test scripts were extremely prescriptive. And if you’re going to that length, you could argue that it’s not a large effort to take those prescriptive steps and automate them.

That said, you might get some quick bang for your buck going this route. I’ve talked to a couple of them and they are always eager to help you. Some of them will even record their test sessions which I would consider a must-have if you decide to use a company for this.

Tooling

I ain’t gonna lie. I like Cucumber and Capybara. I’ve tried SpecFlow and it’s probably as good as you can get in C#, which is decent enough. But it’s hard to beat fill_in 'Email', :with => 'hill@billy.edu' for conciseness and readability. That said, do not underestimate the effort it takes to introduce Ruby to a .NET shop. There is a certain discipline required to maintain your tests and if everyone is scared to dive into your rakefile, you’re already mixing stripes with plaid.

We also discussed Canopy and there was a general appreciation for how it looks though Amir is the only one who has actually used it. Seems to balance the readability of Capybara with the “it’s still .NET” aspect of companies that fear anything non-Microsoft.

Of course, there’s Selenium both the IDE and the driver. We mentioned it mostly because you’re supposed to.

Some version of Visual Studio also provided support for UI tests, both recorded and coded. The CodedUI tests are supposed to have a pretty nice fluent interface and we generally agreed that coded tests are the way to go instead of recorded ones (as if that were ever in doubt).

Ed. note: Shout out to Protractor as well. We didn’t discuss it directly but as Dave Paquette pointed out later, it helps avoid random Sleep calls in your tests because it knows how to wait until binding is done. Downside is that it’s specific to Angular.

Also: jasmine and PhantomJS got passing mentions, both favorable.

Continuous Integration

This is about as close as we got to disagreement. There was a claim that UI tests shouldn’t be included in CI due to the length of time it takes to run them. Or if they are included, run them on a schedule (i.e. once a night) rather than on every “check in” (by which we mean, every feature).

To me, this is a question of money. If you have a single server and a single build agent, this is probably a valid argument. But if you want to get full value from your UI tests, get a second agent (preferably more) and run only the UI tests on it. If it’s not interfering with your main build, it can run as often as you like. Yes, you may not get the feedback right away but you get it sooner than if you run the UI tests on a schedule.

The main takeaway we drew from the discussion, which you may have gleaned from this summary, is: damn, we should have recorded this. That’s a mistake we hope to rectify for future discussions.

CQRS recap, or “How to resuscitate”

Sun, 26 Apr 2015 22:45:21 +0000

I’m fighting a bit with my ego at the moment which is telling me I need to provide at least four paragraphs of update on what I’ve been doing the last three years when I last posted. The fight is with my more practical side which is saying, “Name three people that have noticed.” I’ll compromise with a bulleted list because some of it does have a little bearing on the rest of this post:

I’m no longer with BookedIN although they are still going strong.
I’ve started recently with Clear Measure who has graciously relaxed their “no hillbilly” hiring policy. Guardedly
For those interested in my extra-curriculars, I’m also blogging at http://kyle.baley.org where you’ll find a recent follow-up to an older post on Life in the Bahamas, which I keep getting emails about for some reason…

This past weekend, Clear Measure hosted a meetup/coding-thingy on CQRS with your host, Gabriel Schenker. Initial intended as an event by and for Clear Measurians, it was opened to the public as a means to garner feedback for future events. It was a 7-hour affair where Gabriel set out the task to perform then left us to our devices to build as much as we could while he provided guidance and answered questions.

The event itself ran as well as I expected, which, me being an optimistic sort, was awesome! And Gabriel, if you’re reading, I did manage to get to the beach today so don’t feel bad about taking that time away from me. I won’t go into logistics but wanted to get my thoughts on CQRS on the table for posterity.

By way of background, my knowledge of CQRS was, up until I started at Clear Measure, pretty vague. Limited mostly to what I knew about the definition of each of the words in the acronym. Gabriel has, in meetings and recently in his blog, increased my awareness of it to some degree to the point where it made sense as an architectural pattern but was still abstract enough that if someone asked me to implement it in a project, I would have first consulted with a local voodoo doctor (i.e. speed dial #4).

The good part

So the major benefit I got from the event is how much CQRS was demystified. It really *is* just segregation of the responsibilities of the commands and the queries. Commands must logically be separated from queries to the point where they don’t even share the same domain model. Even the term “domain model” is misleading since the model for queries is just DTOs, and not even glorified ones at that.

Taking one example, we created ourselves a swanky new TaskService for saving a new task. It takes in a ScheduleTaskDto which contains the basics from the UI: a task name, a due date, some instructions, and a list of assignees. The TaskService uses that info to create a fully-formed Task domain object, setting not only the properties passed in but also maybe the CreateDate, the Status, and the ID. Then maybe it validates the thing, saves it to the repository, and notifies the assignees of the new task. All like a good, well-behaved domain object.

Now we want a list of tasks to show on the dashboard. Here are two things we actively had to *stop* ourselves from doing:

Returning a list of Task objects
Putting the logic to retrieve the tasks in the TaskService

Instead, we created DashboardTask DTO containing the TaskId, Name, DueDate, and Status. All the items needed to display a list of tasks and nothing else. We also created a view in the database that retrieves exactly that information. The code to retrieve that info was in a separate class that goes directly to the database, not through the TaskService.

Given more time, I can see how the command/query separation would play out more easily. For the commands, we may have used NHibernate which gives us all the lazy loading and relationship-handling and property mappings and everything else is does well. For the queries, probably stick with views and Dapper which allow us to query exactly the information we want.

My sense is that we’d have a lot bigger set of classes in the query model than in the command model (which would be a full-fledged domain). Because the query model requires at lease one class almost for each and every screen in the app. Dashboard listing of tasks for supervisors: SupervisorDashboardTask. List of tasks for a dropdown list: TaskListItem. Retrieve a task for printing on a report: OverdueTask. All separate and all very specific.

Wrap up

My partner-in-crime for the day was Alper Sunar who is hosting our day’s efforts, such as they are. The big hurdle I had to jump early on was to stop myself from going infrastructure crazy. Early discussions touched on: Bootstrap, RavenDB, IoC, and Angular, all of which would have kept me from my goal: learning CQRS.

I’ve forked the code with the intent of continuing the journey and perhaps looking into something like RavenDB. I have to admit, all the talk around the virtual water cooler about elastic search has me thinking. And not just about finding new sister-wives…

Kyle the Returned

The Bahamas :: Twelve Years On

Sun, 19 Apr 2015 22:48:32 +0000

The sub-title of this post was originally Ten Years On which still would have made it my first post here in a year. But I’ve been away from Canada long enough that I don’t feel the need to apologize for it (though the fact that I’m passive-aggressively acknowledging all this suggests there’s a still a little Canuck left in me somewhere).

I’ve received a number of requests (and bear in mind, zero is a number) for a follow up on a post that is now seven years old. Mostly around the state of the IT industry in the Bahamas presumably from people looking for greener grasses under bluer skies. So I’ll start with that. Then I’ll talk about the non-career stuff. All this makes for a long post but it’s been three years in the making and nothing pisses me off like a “Tune in next time for the information you *really* want” link at the bottom of a post.

The software industry in the Bahamas

Unchanged. I could almost add “literally” to the beginning of that and point out that I know what the definition of “literally” is. From my perspective, there is still no software industry of any kind in the country. The part that has changed is that I’ve become less frustrated about it. I tried to do my part and met quite a few people who were eager to participate. But I also met more than a few people who were skeptical and even some that were downright hostile.

The Bahamas is xenophobic, plain and simple. Historically, they’ve had plenty of reason to be. The size and location of the country being what it is, they probably still do. But what made this frustrating is that I could never get across the benefit of what I was doing. For the last twelve years, I’ve been living in the Bahamas but collecting a paycheque elsewhere. And willing to share my experience with others. To me, this is exactly the type of person the Bahamas should be pursuing to grow their economy. The Bahamas is fairly unique in the Caribbean for having a good infrastructure (at least with respect to internet connections) as well as being 100 miles off the coast of the US. It’s even the same time zone as Miami. Why someone more enterprising than I am hasn’t set up an off-shore development shop here is beyond me.

So if you’re looking to work in the software industry and live in the Bahamas, you’d best have a good job lined up before you come, probably remotely. Even then, after you move, you have to maintain your skills and network in the event you need to switch jobs.

I’ve been pretty lucky in this regard, working 100% remotely for all but the first 2 or 3 years of my time here. First with remote contracts, then with five years at my own startup, and now with Clear Measure, which was as a direct result of my work in the .NET community, blogging and attending conferences. Do NOT underestimate the power of contacts.

Living in the Bahamas

The nice thing is that once you put your frustration at the wasted opportunity behind you, life becomes several orders of magnitude more enjoyable here. Cost of living is manageable. We live in a gated community on a canal that leads directly to the ocean. It has a 90-foot dock that’s wasted on us. For the price we paid, we’d get a lot less house in Calgary, the city we abandoned to relocate. (At least as of the last 2000s, not sure what’s going on in the housing market up there these days.)

Other things are expensive but not prohibitively so. We pay about $200/month for electricity in the heat of summer, for example. From what I gather, even that is low compared to our neighbours. That might be because we’ve made a conscious effort to stay on top of our consumption, like switching to ductless A/C units rather than central.

As a general rule, stuff you want to buy will be expensive. I priced out a Thunderbolt display for my MacBook Pro at $1387 when it’s retailing for $999 stateside. One company had it at almost $1600. There’s 45% import duty on a good number of items plus the cost of shipping. As of January 2015, there’s also 7.5% VAT (i.e. GST to you Canadians) added to almost everything.

Availability of anything remotely fun for software developers is also a concern. I haven’t found anyone that carries Raspberry Pi and the closest thing to Arduino I’ve seen is a Netduino kit selling for $150 (compared to $70 in the US). You need to find a local shipping company or a friend who will bring something back for you. Which isn’t as hard as you might think since everyone here knows the pain you’re trying to avoid.

This is painting a bleak picture so I’ll re-iterate: it’s not prohibitively expensive. A friend of mine regularly orders guitar pedals and amps (more on that later) and he’s firmly entrenched in the upper-middle class bracket with me.

Beyond technology

The lack of a technology industry has actually been pretty refreshing once you learn to accept it. It certainly frees up time to pursue other interests. Buying a boat is a popular one that Missus Hillbilly and I toss around from time to time. There are islands a’plenty to explore and, quite literally, plenty o’ fish in the sea. Our daughter likes to remind us that we aren’t “boat people” and she’s probably right, given how prevalent motion sickness is in our family. We’ve started experimenting with taking a charter out every few months which is always fun and considerably cheaper than owning and maintaining a boat.

For my part, I unwind by playing in two bands. Both are populated by professionals who have no illusions of superstardom and our practice and gig schedule reflects that. Local tastes are such that we don’t play a whole lot of songs I’d like to play. But I do get to play armchair sociologist trying to figure out why songs like Pour Some Sugar on Me are so &*%$# popular.

Beyond that, one of the things I like most about the country is its small size. It makes things seem very familiar and almost village-like, especially once you decipher some of the local vernacular. It’s common to run into someone you know when you’re out and about which I rarely did in five years living in Calgary. Even if you don’t, Bahamians are a friendly people by nature. It was a little disorienting at first to have everyone say “good morning” to each other when they walk into a doctor’s office waiting room. These days, I’ve become so used to it that I wonder why offices are so stand-offish in the rest of North America.

There are also the other islands besides New Providence/Nassau where I live. The government is taking them for granted because they are all, every single one of them, a national, international, and planetary treasure. If you want to see the Bahamas they show you in the commercials, land in Nassau then get out to one of the out islands as soon as you possibly can. Eleuthera is one of my favoritest places in the world and I’ve only been there maybe 4 times in my life.

Crime and (lack of) punishment

Someone will inevitably bring up crime. It’s high. I don’t think even the current government can deny it. The murder count was 122 in 2014 which is about 34 murders per 100k people. If the Bahamas were a US city, it would have the sixth highest murder rate.

I could explain this away by saying it’s mostly gang-related or it’s the same as any large city or whatever. I’d be explaining it away because I live here and need to justify why I continue to do so. The fact is, statistically speaking, you’re more likely to get killed here than in, say, Toronto or Helsinki. Depending on how you define the statistics. But when you talk to people about it, the concern is rarely about personal safety; it’s about how the crime rate is affecting the world view of the Bahamas and thus, the economy. Three murders last weekend? Hmmm…that’s too bad. The US issues a travel warning for the Bahamas? LORD TUNDERIN’ JAYZUS AND MOSES ON A BOAT, WE GOTS A PROBLEM!

Is the Bahamas for you?

I know you’re expecting me to say “it depends” like a good consultant or adult diaper salesperson. But you know what? Yes, it is. If you’re considering it and are sufficiently employed, stop waffling and get over here.

It wasn’t for me when my wife’s job gave us the opportunity to move here a dozen years ago. But we’ve always moved under the assumption that if it doesn’t work out, we’ll move back. We’re still here.

And as software professionals, that should weigh in on your decision. Day in and day out, we make decisions based on how reversible they are. If you decide to relocate for a year, it’s usually not hard to go back to where you were. Moving to a new country will, at worst, give you some perspective. And a sunburn.

Cron and AppEngine

Sat, 28 Apr 2012 15:45:26 +0000

Quick PSA on using cron jobs with Google App Engine because it almost wreaked havoc for us.

App Engine has a lovely feature of having different versions of your app. You can upload a new version but not make it the default until you’re good and ready. We do this all the time for deployment. Deploy to a new version and try it out, then make it the default when we’re ready to unleash it. Often, we deploy to the new version a day or so in advance.

Cron jobs, it seems, are handled outside this versioning mechanism. If you upload a new cron.xml file, it’s active. Right now. Doesn’t matter if the version it was deployed in is the default or not. As soon as it’s uploaded, it’s the new cron-ness.

Where this almost bit us is that we added a new cron job in our most recent release (deployed yesterday but not active) to use a dynamic backend. As soon as the cron job got uploaded, it started running. I didn’t notice until this morning when our backend usage reflected the new cron job. Some quick research and here we are.

What this means long term is that cron.xml is no longer going to be deployed as part of our application anymore. It now becomes an entirely separate process. I’m a little annoyed that we have to wait until we pull the trigger on the new version before we can upload the new cron.xml but it’s a quick step.

Kyle the Mis-scheduled

The Economics of Ergonomics

Sun, 22 Apr 2012 22:18:48 +0000

Let it not be said there are no downsides to living in the Bahamas (though if you’ll permit a little boasting, a shortage of fantastic venues if you’re lucky enough to be in a band is not one of them).

The desk where I work is too high, plain and simple. So much so that I’ve recently abandoned my Kinesis keyboard because it is not what you might consider “low form factor”. I’ve started feeling some twinges in my lower forearm that my unscientific diagnosis is attributing to the height of my hands while I type. Dumping the Kinesis has helped but it has also led to the return of stress in other areas of my hands. And no amount of raccoon skinning seems to alleviate the pain.

Getting a lower desk is easy enough but I’d actually like to do a little experimenting with two alternatives. Alas, neither are easily done in the Bahamas. The underlying problem is availability. The desks/equipment I want to test are not available here so I would have to order them in. Which means both shipping charges and import duties, the latter of which is a major source of income for the Bahamian government to offset the fact that there is no income tax. So returning said equipment is just not practical if it doesn’t work out. Nor is there much of a reseller market.

So I’m hoping I can get some comments from people who have done something similar.

Adjustable height desk

These are, of courses, desks where you can adjust the height easily. I like the idea of these for two reasons:

They can be set low
They can be set high

I’ve never tried a desk that you stand at but I’ve always wanted to. Working on my own, I tend to get up and wander a lot while I’m thinking. I also pace when I’m on the phone with someone for any length of time so it would be more convenient to walk up to the computer during the conversation should the need arise. (“You want to know the right pattern of plaid for a first date with your second cousin? Let me look that up.”)

I went desk-shopping over the weekend and the closest thing I saw was in an office supply store. And it wasn’t on the showroom floor. Off in the corner of the store were the employee desks. They were all essentially plywood based, laminated desktops all mounted in warehouse style shelving frames. They sat on brackets in the frame which means you could set the height to whatever you want. It wasn’t something you could easily adjust on the fly and my wife wasn’t too thrilled at the industrial look so it was a fleeting idea at best.

Command centre

This is an idea I’ve had ruminating in my head for a while now. I would get rid of the desk altogether in favour of a comfortable command-centre style or gaming chair. In front of it it, I’d mount my monitors on a couple of flexible arms somehow, possibly on the armrests or on stands on either side of the chair. The important thing is that I can slide the monitors out of my way when I want to get out of the chair, and slide them back in when I sit down.

The keyboard would rest either on my lap or on some flat surface on my lap. Or maybe go with a split keyboard (though one without a wire between the two) and have one piece mounted on each armrest. Haven’t quite worked out how the mouse would fit in though. A trackball on some little platform on the side makes sense but I’ve got one now and it doesn’t feel as productive as just a regular mouse.

I feel like this would be more comfortable and would reduce much of the muscle stress that seems to have become more prominent since hitting 40 earlier this year. All of this kind of makes sense in my head but the logistics of getting the stuff here is such that I don’t want to make the investment unless I’ve had a chance to try it out at least for a few days. There’s a chance my tendency to get up and wander might make this impractical. Or maybe cord management would be an ongoing problem.

The device shown at right, which I discovered while researching this article, is essentially what I’ve described. It’s some US$2750. Duty would add about 50% and shipping would likely bring the total price above five large. There’s another potential hurdle in that it may not be available anymore given the company’s domain seems to point to a parking spot. But even building my custom version will cost enough in non-refundable cash dollars for me not to head over eBay.

Instead, I hunt for a standard desk about four to six inches lower than the one I’ve got. Not as exciting, possibly not as ergonomic, but easier to replace.

So my question to you, my honorary hillbillies, for anecdotal evidence. Have you tried either of these devices? What’s good and bad? Good return for the money or does it sit in the garage next to the Bowflex you bought in a fit of New Year’s anxiety?

Kyle the Unreturnable

Audit Fields in Google AppEngine

Sat, 07 Apr 2012 23:53:28 +0000

Executive summary: Here’s how we’re implementing audit fields in AppEngine. IT’S BETTER THAN THE WAY YOU’RE DOING IT!

I considered saying “I hope there’s a better way of doing it” but I believe I’ll get more responses if I frame it in the form of a challenge.

For all entities in our datastore, we want to store:

dateCreated
dateModified
dateDeleted
createdByUser
modifiedByUser
deletedByUser

Here are the options we’ve considered

Datastore callbacks/Lifecycle callbacks

AppEngine supports datastore callbacks natively. If you use Objectify, they have lifecycle callbacks for @PrePersist and @PostLoad. The former works fantastic for dateCreated, dateModified, and dateDeleted. Objectify can handle all three easily as well provided you use soft deletes, which we do. (And they aren’t as bad as people would have you believe, especially in AppEngine. You’d be surprised how many user experience problems you discover strolling through deleted data.)

Both of these led to problems for us when we tried to use them for the createdByUser et al methods. We store the current user in the session and access it through a UserRetrievalService (which, at its core, just retrieves the current HttpSession via a Guice provider).

If we want to use this with the Objectify lifecycle callbacks, we would need to inject either our UserRetrievalService or a Provider<HttpSession> into our domain entities. This isn’t something I’m keen on doing so we didn’t pursue this too rigorously.

The datastore callbacks have an advantage in that they can be stored completely separately from the entities and the repositories. But we ran into two issues.

First, we couldn’t inject anything into them, either via constructor injection or static injection. It looks like there’s something funky about how they hook into the process that I don’t understand and my guess is that they are instantiated explicitly somewhere along the line. Regardless, it meant we couldn’t inject our UserRetrievalService or a Provider<HttpSession> into the class.

The next issue was automating the build. When I try to compile the project with a callback in it, the javac task complained about a missing datastorecallbacks.xml file. This file gets created when you build the project in Eclipse but something about how I was doing it via ant obviously wasn’t right. This also leads me to believe there’s something going on behind the scenes.

Neither of these problems is unsurmountable, I don’t think. There is obviously some way of accessing the current HttpSession somehow because Guice is doing it. And clearly you can compile the application when there’s a callback because Eclipse does it. All the same, both issues remaining unsolved by us, which is a shame because I kind of like this option.

Pass the User to Repository

This is what was suggested in the StackOverflow question I posed on the topic. We have repositories for most of our entities so instead of calling put( appointment ), we’d call put( appointment, userWhoPerformedTheAction ).

I don’t know that I like this solution (as indicated in my comments). To me, passing the current user into the DAO/Repository layer isn’t something the caller should have to worry about. But that’s because in my .NET/NHibernate/SQL Server experience, you can set things up so you don’t have to. Maybe it’s common practice in AppEngine because it’s still relatively new.

(Side note: This question illustrates a number of reasons why I don’t like asking questions on StackOverflow. I usually put a lot of effort into phrasing the question and people often still end up misunderstanding the goal I’m trying to achieve. Which is my fault more than theirs but still means I tend to shy away from SO as a result.)

Add a User property to each Entity

I can’t remember where I saw this suggestion. It’s kind of the opposite of the previous one. Each entity would have a User property (marked as @Transient) and when the object is loaded, this is set to the current user. Then in your repositories, it’s trivial to set the user who modified or deleted. This has the same issue I brought up with the last one in that the caller is responsible for setting the User object.

Also, when new objects are created, we’d need to set the property there as well. If you’re doing this on the client, you may have some issues there since you won’t have access to the HttpSession until you send it off to the server.

Do it yourself

This is our current implementation. In our repositories, we have a prePersist method that is called before the actual “save this to the datastore” method. Each individual repository can override this as necessary. The UserRetrievalService is injected in and we can use it to set the relevant audit fields before saving to the repository.

This works just fine for us and we’ve extended it to perform other domain-specific prePersist actions for certain entities. I’m not entirely happy with it though. Our repositories tend not to favour composition over inheritance and as such, it is easy to forget to make a call to super.prePersist somewhere along the way. Plus there’s the nagging feeling that it should be cleaner and more testable than this.

Related to this is the underlying problem we’re trying to solve: retrieve the user from the session. In AppEngine, the session is really just the datastore (and memcache) with a fancy HttpSession wrapper around it. So when you get the current user from the session, you’re really just getting it from the datastore anyway using a session ID that is passed back and forth from the client. So if we *really* wanted to roll our own here, we’d implement our own session management which would be more easily accessible from our repositories.

So if you’re an AppEngine user, now’s where you speak up and describe if you went with one of these options or something else. Because this is one of the few areas of our app that fall under the category of “It works but…” And I don’t think it should be.

Kyle the Pre-persistent

The other side of AppEngine

Sat, 10 Mar 2012 19:12:54 +0000

I really do love Google AppEngine. That didn’t come out so well in a recent post. I’ve had a bad taste in my mouth ever since then and now that I’ve eliminated the dandelion wine as a potential culprit, I’m forced to conclude that I need to balance my hillbilly feng shui. So in no particular order, here is what I like about AppEngine:

Granular pricing

I complained about the confusing pricing in the last post but since then, a funny thing happened. BookedIN got some traffic (thanks in no small part to the Chrome Web Store.) We were getting feedback and adding features and fixing bugs and hey, howdy, hey, wouldja take a look at what’s happened to our AppEngine budget!

And for all the bitching I did on the different ways your application is charged, this jump in cost has led to some much-delayed but necessary shifts in how we code.

As an example, we are now hitting our datastore read quota regularly. Which leads to the question: “Why so many reads?” Which leads to a number of other questions most of which can be boiled down to “Why am I treating the datastore like a SQL database?”

Similar discussions have led to us taking much greater advantage of AppEngine features we’ve all but ignored, including task queues and the channel API.

This doesn’t mean we’re on a refactoring binge, mind you. Our costs have jumped to the point where we want to waste thousands of dollars in developer hours to save ourselves a few extra bucks every month. But it has affected our coding practices going forward. When discussing new features, there are new questions: Can we denormalize this object? Do we *really* need to send this email synchronously? Can we send a message back to the user via an open channel? That sort of thing. All of these questions we should have been asking but didn’t because we “didn’t have time”. Now that cash dollars are being spent and now that we’ve seen how little time it takes to tweak our way of thinking, we are now working towards a better performing and more scalable application.

Out-of-the-box services

I looked briefly at AWS. It’s very different from AppEngine. From what I can tell, it seems you get a server. What you do with it is up to you. With AppEngine, I have an environment for caching, task queues, logging, mapping, channels, etc.

The price you pay for all these services is that if you ever decide to move away from AppEngine, you’re rewriting. For the most part, this ain’t no generic framework you can swap in your own implementation fer.

Multiple simultaneous versions

You can deploy up to ten versions of an application to the same account. For us, that makes actual deployments very smooth (in theory). We deploy on Saturday nights because statistically, that’s when the app is least used. But any time the week prior, we can deploy to a new version in production and just not make it active. That allows us to do smoke tests (if possible) and make last minute tweaks (or, more likely, add new features). The night of, it’s mostly a matter of switching the default version from the old one to the new one. I say “mostly” because sometimes, we have mappers to run to transform the data in order to handle some new feature. If you structure your model properly, this is an optional feature, but I like to do it anyway. Even with our massive increase in traffic, this still takes less than 15 minutes.

There’s also now a new experimental feature that lets you redirect a percentage of your traffic to a different version. This is designed for A/B testing but something tells me it won’t be used as such. In any case, I haven’t tried it.

Backup and Restore

I’m including this one because my original post said it wasn’t there. Since writing it, an experimental backup/restore feature has been added to the Datastore Admin console. I’ve tried it and it works though the lack of blob support means that it’s usefulness for us is almost non-existent. I suspect it’s not far away from being ready though. In the meantime, there are other solutions. One is DatastoreBackup which looks promising. Another is Google App Engine Backup And Restore, which looks very out of date. I mention it only to serve as an example for a long-standing PSA I have: Acronyms are not always the best name for a software project.

(NOTE: I *still* have to use Firefox to access the Datastore Admin console because it usually doesn’t appear in Chrome.)

Final note: I lied about this list being in no particular order. For all my initial confusion, I can’t understate how much I now love the granular pricing of AppEngine. If we want to pay less for our hosting, there are any number of things we can do to restructure things. And the nice thing is: almost every one of them will result in a better performing application. Before this, I’ve always had a fuzzy feeling of “yes, I know we could optimize but how much is that *really* going to save us?” Now, it’s right there in front of me. “We’re using too many channel API calls.” That’s because there’s an error in our logic when we rejoin a channel. “Our front-end instance hours are going up.” Maybe stuff can be refactored into a task queue or a backend. “Our data reads have spiked.” Perhaps we should cache the list of countries instead of querying the datastore every time the user brings up a certain dialog.

Kyle the Discounted

QA: A Hillbilly Love Story

Mon, 30 Jan 2012 21:31:05 +0000

The hillbilly turned forty last weekend! I’ve been anticipating this giddily because now, I can be crotchety and people will still find me adorable.

I hate QA. Hate, hate, hate, HATE, HATE, HATE it. I despise it so much, I think I might swear. Here goes….I f---, okay, I can’t do it but I still &*%$ hate it.

Side note: I know all QA people weren’t raised by trolls and/or gnomes but I’m going to generalize here. If you are a QA person and were raised by evil leprechauns, please don’t take offense.

As a hillbilly, I’m an innately optimistic and forgiving person. An ugly codebase to me is a sea of opportunity. Changing requirements? Ha! I can put my OCD up against any client. I can see the positive in almost anything. Except the emotional rollercoaster that is QA.

The end of an iteration is a time of celebration and relief for me. We’ve completed a significant piece of functionality on our application. We’ve followed best practices, left the code in a better state than when we started, and all our unit and UI tests are passing (more or less). Despite my past experience, I always feel great pride when I email our QA department:

Hey there, QA folkery! I come bearing tidings of great joy! We have a new version of the application for your revelry and astonishment! You will surely faint in awe at the sheer glory of it! I await your token sign-off. Acclaim and praise is optional (but recommended).

After that, I sit quietly for a few minutes to bask in my accomplishment of the last couple of weeks then dive into the next iteration, the previous one long since gone from the vestiges of my memory by that point.

The inevitable spartan response:

Please fix the following issues:

As per the spec, emails should be sent every hour, not daily

I got a 404 error when I went to the public booking URL

I know this wasn’t part of the iteration but we should display a message when someone confirms an appointment. Can you include this?

You spelled it “cancelled” in the settings screen but “canceled” on the calendar.

Let me know when a new version is up for testing.

By now you will not be surprised when I tell you that the QA process was invented by the Marquis de Sade.

QA goblins: surely you see my point, no? The iteration is over. Done. Finished. There’s no “D.S. al fine” at the end. In my head, I’ve moved on to new functionality. I’ve got tests written and prototypes completed. And you want me to forcibly re-enter the past?!? All the hard problems from that iteration have already been solved. I don’t care about your “cosmetics” or your “functionality”. That new feature you’ve requested? It’s KILLER! It’ll be a great asset to the app. When I get around to it. That 404 error? Just a configuration issue. It’ll be fixed when we deploy. And come on, spelling mistakes? Have you not been on Facebook lately?

In short: I. Don’t. Want. To. Do. It. Right. Now.

And I have to. Virtually everything brought up in a good QA process is valid from bugs to cosmetic issues to features that we need to include this iteration even though they weren’t in the list. Without them, we don’t have a usable product. Users will complain or worse, move on to something else. But my headspace is somewhere else. I’ve already compartmentalized the new features and have shifted my own personal Eye of Sauron on them. Now, there’s leakage that I thought I’d never have to deal with again.

Another psychological issue: I’ve poured everything I’ve got to deliver the best that I’ve got and it wasn’t good enough. It’s never good enough. I’ve just presented to you the end result of four years of university training and over a dozen years of experience as an “expert in the field”. And with a few phrases, you send me slinking back to my computer to do it over again.

It’s a little strange in some regard. I welcome criticism of my coding choices and style. I actively seek it out sometimes. Even code reviews can be made fun again. But something about the subtly adversarial nature of QA raises my dander. The fact that it’s so essential to quality and that I can’t think of an effective way around it doesn’t help…

So to sum up:

QA forces me to change my headspace. And I hate it for that.
QA points out my flaws as a developer. And I hate it for that.
QA is necessary and makes software better. And I hate it most of all for that.

To end on a more positive note, most QA people (including those that do it as part of other duties) I’ve worked with are extremely nice. There’s no accusatory tone in the responses, no blame on messing things up, and no sense of “you’ve *really* screwing things up this time.” In their minds, they’ve done what’s expected and like me, have made the product better for it.

But I still hate you.

Kyle the Unadulterated

AppEngine thoughts

Mon, 16 Jan 2012 20:01:52 +0000

AppEngine has new pricing as of November 7 which has generated much discussion, most of it as exciting as deciding which wine to serve with raccoon (answer: dandelion). So with the segue established, I shall pontificate on what I believe is the single biggest thing keeping it from being a truly great product and a worthy competitor in the cloud hosting space.

First I’ll get out of my system a laundry list of lesser issues:

1. Confusing pricing/configuration

This could be a product of my Microsoft indoctrination but I still have trouble figuring out how we get charged for stuff. It’s based on usage which is a great model on paper. But the usage is broken down into, among other things:

Front-end instance hours
Back-end instance hours
Datastore Write Operations
Datastore Read Operations
Datastore Small Operations
Stanzas Sent
Channels Created

Furthermore, you can tweak things like the minimum and maximum idle instances and the minimum and maximum pending latency.

All this requires a lot of research and testing. And the only way to test various settings is: a) in production, or b) using load testing with a separate paid environment. Both of which will end up costing you dollars. But it must be said, those dollars probably won’t add up to the cost of setting up your own test environment. So far, we spend about the cost of a smoothie every day for all versions of our app (depending on where you find the ingredients).

2. No built-in backup and restore

Backing up the datastore is your responsibility. Now, it’s easy to set up a batch process running nightly. But you’re almost guaranteed to hit your Datastore Read Operations quota on a nightly basis once you reach a certain size. Also, the new-fangled High Replication datastore makes things more interesting by not actually being supported for this scenario.

3. No built-in reporting mechanism

The only way to interact directly with your data is through code you’ve written yourself or with GQL, a SQL-like language for the datastore. But you quickly hit limitations. First is that you can return only 20 results at a time (which you can up to 200 by adding a limit parameter to the URL…manually). Second is that you have to create indexes for fields you want to filter or order by. Doesn’t lend itself to ad hoc querying.

4. Admin console bugs

This one irks me almost as much as the fact that I have to restart my computer whenever I update the glorified Notepad that is Adobe Reader*. Too often for my tastes, when clicking around the console, I’ll get a general “An error has occurred” page. And the error page is not a pretty one:

Even when it does work, until recently, I had to use Firefox or Internet Explorer to access one of the pages (Datastore Admin) because it didn’t load in Chrome.

Which leads me to the point of this post. The one killer issue in AppEngine that keeps its status below world class hosting environment:

Support as a second-class citizen

If you didn’t notice at first glance, take another look at the error page above. In particular, at the URL at the bottom, which is where the “report” link goes. It’s a Google Group page for AppEngine. To their credit, they’ve addressed many of the issues I’ve had with Google Groups recently. Even so, for a world-class hosting environment and especially for apps we’re paying for, I’d much rather see something like this.

Also, my experience with the group site has been pretty dismal. Much of the time, my questions get not a single response except mine. Even reporting issues on the Google Code site leads to sporadic responses. I get slightly better averages on StackOverflow.

This came to a head for us recently when we converted from Master/Slave to High Replication late one Saturday night. Several of the problems I outline here occurred that night, including the 1990s error screen above. And I couldn’t find a single email address or phone number anywhere that I could contact for help and be assured of a response with even an unreasonable time. I made a post to the forum about an issue we saw the next day. The link is included above in the list of posts that have received no response.

Another support-related area that could use work is the roadmap. Something I’ve noticed in all dealings with the AppEngine team is an almost fanatical abhorrence of delivery dates. Even at Google IO, the best we could get from the team was “we’re working on it” with a lot of nervous laughter when someone asked about SSL access on custom domains, which is one feature we’ve had to make decisions around. I gather it’s a hard problem but even if they said “probably 2012”, that would at least indicate to us “okay, it’s not anytime soon, time to decide which is more important in the short term.”

Had I written this post a couple of months ago, after our High Replication migration, it would have been a lot more acidic in tone. For a few weeks after that, I was actively checking out Amazon Web Services. A Microsoft rep reached out to us serendipitously that same week wanting to talk to us about Azure. If he had not dropped the ball and postponed our meeting at least three times, this here blog thingy might be back in .NET land by this time. (Likely not, but this wouldn’t be a blog if I didn’t make grandiose unsupportable claims that back up my argument.)

These days, I’m not so much frustrated as I am disappointed. For all its faults, there is a lot of good being offered by AppEngine. Like many startups, we have discussions about Google and AppEngine and the general consensus is that we’re happy to see Google focusing on what they’re good at. (Do I lose points if I point out my subtle digs?) But when it comes to support for AppEngine, it feels like it’s run by engineers for engineers. Yes, support is boring and customers can be confrontational and much of the time, the answer is a variation of “you’re doing it wrong”, etc, etc. But it’s not just another IoC container; it’s a cloud hosting platform. I believe it needs a higher level of professionalism than what I’ve seen so far.

Kyle the Untenable

*But not nearly as much as the fact that said update invariably adds an icon to my desktop. C’mon Adobe, who actually opens Reader and then opens a PDF file?