Why use MRUnit?

Testing a Hadoop job requires a lot of effort not related to the job. You must configure it to run locally, create a sample input file, run the job on your sample input, and then compare to an expected output file. This not only takes time, but makes your tests run very slow due to all the file I/O.

MRUnit is:

a unit test library designed to facilitate easy integration between your MapReduce development process and standard development and testing tools such as JUnit

With MRUnit, there are no test files to create, no configuration parameters to change, and generally less test code. You can cut the clutter and focus on the meat of your tests.

The Good

Hadoop tests are much simpler to write using MRUnit. Here’s an example of entire test class:

class ExampleTest() {
  private Example.MyMapper mapper
  private Example.MyReducer reducer
  private MapReduceDriver driver

  @Before void setUp() {
    mapper = new Example.MyMapper()
    reducer = new Example.MyReducer()
    driver = new MapReduceDriver(mapper, reducer)
  }

  @Test void testMapReduce() {
    driver.withInput(new Text('key'), new Text('val'))
        .withOutput(new Text('foo'), new Text('bar'))
        .runTest()
  }
}

You can test map and reduce separately, of course. You can also easily verify counters:

driver.withInput(...)
driver.run()

def counters = driver.getCounters()

assertEquals(1, counters.findCounter('foo', 'bar').getValue())

There’s a mess of other cool stuff like MockReporter and MockInputSplit, but I mostly haven’t found a use for them or time to make a simple example.

The Bad

Before I tell you to go grab the latest distribution, I want you to know some of the problems we’ve encountered in the “real-world”.

1. First and foremost, MRUnit is not useful for streaming jobs. If you only write streaming map-reduce jobs, you’ll have to do it the old fashioned way
2. Calling driver.runTest() doesn’t tell you what the failure was (it just throws an AssertionError). Instead, call def output = driver.run() and assert
3. The documentation sucks. There’s only one example and the rest you basically have to figure out from the API
4. setup() is called for the new Hadoop API (mapreduce packages) but not the old API (mapred packages). You have to call it yourself if you need it
5. Finally, tests reuse the same JVM. So if you’re accidentally maintaining state in your job, you will be bitten!

Conclusion

MRUnit makes writing tests for Hadoop easier. It has drawbacks, but they are far outweighed by the benefits.

Grab the latest MRUnit JAR

By the way, here’s how you test a streaming job:

./myMapper.py < test.input | sort | ./myReducer.py > actual.out
diff expected.out actual.out

Related posts:

1. Javascript Stacktrace update
2. Software Engineering and the war of benefits

I hope authors, as well as readers, will gain some insights. Here is my review of MooTools 1.2 Beginner’s Guide.

Book Structure

The book basically takes you, step-by-step, through downloading the MooTools library parts and building simple examples with HTML, CSS, and, of course, MooTools. It seems to touch on many of major parts of MooTools: Core, DOM selection, Events, Ajax, and Fx. Finally, it introduces you to MooTools More and how to write your own MooTools plugins. it approaches everything with very simple, very “hands-on” examples. There are some deviations (like pop quizzes), but they are few.

The examples themselves generally introduce a topic, give an HTML/CSS template and then progressively fill in the MooTools bits.

The Good

You can tell that the authors are bloggers by their very informal writing style. They know how to use simple language so that their writing does not impede the learning ability of the reader. I was able to read the entire book in about 3 hours cover-to-cover, a testament to how easy-to-read it is.

This is a book that a real, never-coded-a-web-page-before dude(tte) could pick up and do something with immediately.

The Bad

It truly is a beginner’s guide, and having some sort of MooTools knowledge myself I was annoyed by the verboseness of the examples. There seemed to be a lot of times I was seeing a small code example stretched out to several pages of text.

I was slightly disappointed by the number of linguistic (spelling and grammar) mistakes throughout the book. These were obviously not cultural differences between the UK-based Packt and the US. I was always able to understand what the writer meant, but I kept getting hung up on the mistakes because that’s the kind of asshole I am ;)

I really missed a good reference or appendix section at the end where I could find links to more information about the parts of MooTools I was learning about. There was some stuff within the chapter texts, but not enough.

You should buy this book if…

… you are truly a beginning web developer. This book is a great introduction to web development with a core focus on MooTools. It seems to generally promote good web development practices in my opinion.

Conversely, this book is not great to use just for reference. If you want to lookup anything advanced, it’s not going to be your best bet.

MooTools 1.2 Beginner’s Guide »

Related posts:

1. Guest Post: Save Text Size Preference Using MooTools and PHP
2. Create a Color Palette Using CSS and MooTools 1.2
3. This month in Bookmarks: April 2008

I worked on some really cool internal (and therefore not shareable) projects in Java, Jython, and Web languages and platforms. They were generally fun, diverse and challenging.

On to Return Path!

I’ll now be joining a small, agile team at Return Path. We will make email safer, and (I hope) win the war over spam. Audacious yes. Impossible… I think not.

I wish all my Sun/Oracle counterparts well. Good luck!

No related posts.

The problem with obtuse triangles and CSS

Obtuse triangles are slightly more complicated, since you can only create acute and right triangles with the CSS shapes method linked above. Therefore I created two triangles: a positive (black) right triangle, and then a negative (white) triangle to emulate an obtuse triangle.

The HTML

<blockquote id="bubble">
    <!-- Black (positive) triangle -->
    <span class="arrow">
        <!-- White (negative) triangle -->
        <span> </span>
    </span>
    Content inside bubble
</blockquote>

Any tag combination will work as long as the CSS shape elements are inline elements.

The CSS

The two key parts of the CSS are the rounded corners (available in major, non-IE browsers) and the arrow that extends from the bubble.

#bubble {
    margin: 8px 0;
    padding: 10px;
    position: relative;
  /* Adjust these to taste */
    width: 175px;
    border: 2px solid #000;
    -moz-border-radius: 15px;
    -webkit-border-radius: 15px;
    border-radius: 15px; /* Opera and Chrome */
}
#bubble .arrow {
    border-bottom: 0px solid #FFF;
    border-right: 20px solid #000;
    border-top: 15px solid #FFF;
    position: absolute;
    left: -20px;
    top: 5px;
    height: 0;
    width: 0;
    line-height: 0;
}
#bubble .arrow span {
    border-bottom: 0px solid transparent;
    border-right: 20px solid #FFF;
    border-top: 8px solid transparent;
    width: 0;
    height: 0;
    line-height: 0;
    position: absolute;
    left: -2px;
    top: -8px;
}

After all this, we have:

I haven’t done so here, but you can can create more interesting elliptical borders with:

    padding: 25px; /* <- increase padding to avoid text overlap of the ellipse */
  /* attribute: width / height */
    -moz-border-radius: 60px / 15px;
    -webkit-border-radius: 60px / 15px;
    border-radius: 60px / 15px;

Note that Opera uses border-radius and that Safari/Chrome does not support percentages. I know this doesn’t look as pretty in IE, but it’s a neat “progressive enhancement”. This might be a great idea for comments, twitter statuses or sub-headings.

There probably is a clever way to inherit a transparent color from a parent element, but I didn’t really take the time to dig into it. If you have improvements, I’d love to hear them.

Related posts:

1. Create a Color Palette Using CSS and MooTools 1.2
2. About the indirect adjacent combinator (~) in CSS
3. Use the table-layout CSS property to speed up table rendering

Andres was kind enough to answer questions that I think are relevant to all of you readers. I hope you learn from them as much as I have. Without further ado…

How do you keep up-to-date with programming advancements and news?

I keep a close eye on RSS feeds on specific blogs, blog aggregators and news sites. DZone is particularly a great source for many things JVM, with some .Net, Ruby, PHP sprinkles here and there. Twitter is another good source for links and interesting readings, but you have to know who you should follow. Local JUG meetings and conferences are a good way to meet people that share your interests, then follow on social networks. Once you’re in you follow the natural process of data selection according to your needs, thus expanding your network as a result. Can’t forget about books too; blogs and online articles are just the tip of the iceberg.

What is the best decision you ever made to improve your programming ability?

I have to say starting an open source project did it for me. It opened the door where people will see your code and give you all kind of feedback. they will take your code and use it in ways you did not foresee. They’ll break it. They’ll mangle it. They’ll enhance it. You can share code and learn from others; it is all about reaching out to developers and users that otherwise you’d never meet. Besides working on open source projects did help me finding employment, which besides a stable income provided me with additional learning opportunities.

What are the most important skills for a programmer to have?

Quoting Heraclitus: “Panta rhei”. Everything flows. A programmer should be able to ride paradigm shifts. Who would have thought back in 95 that Java would be found everywhere years later? (even in space!) A few years ago the ajax revolution changed the web landscape. These days mobile devices seem to be the driving force. Learning a new programming language (at least every 12 months) is a good way to keep you on your toes.

Which programming tools could you not live without?

Vim as editor, Git as source control. These days I don’t use an IDE unless it is 100% Java related, in which case it is a coin flip between Eclipse and NetBeans.

Which programmer has had the biggest influence on you?

I’ve learned a lot from many but I have to save the first spot for Glenn Vanderburg. He is the one responsible for that last nudge that made me jump into contributing back to the open source community. Thank you Glenn!

What first got you interested in Groovy?

I was actually first interested in Ruby after spending a whirlwind Saturday listening to Dave Thomas and Stuart Halloway during an NFJS tour stop in Austin back in 2006. I remember clearly during a panel on the next day that Venkat Subramanian said “Groovy’s future a year ago was very grim, today it’s safe and well”. Because I was not “ready” to embrace Ruby completely (I had strong roots in Java) decided to give Groovy a try, as I heard both languages were close to one another feature-wise. Once I discovered meta-programming via GORM’s dynamic finders I was sold completely, haven’t turned back since and I’m happy with it.

You now lead the Griffon project, what is your motivation for developing it?

I first got into Java because I needed an UI for a term project back in college. Java was a few months old and after seeing applets I convinced the professor to accept Java code rather than C++ code. Java AWT is very quirky when compared to modern toolkits but it did the job, and what’s more it did it better than what I was used to at that moment (low level C/C++ XT bindings), as a result I was hooked into Java. Across the years I keep coming back to desktop app development but longed for a framework that helped speeding things up. It wasn’t until all the pieces came together (Groovy, Grails, SwingBuilder and its siblings) that a framework like Griffon could emerge. Griffon puts a smile back in my face while doing desktop development, I simply cannot describe the joy of doing application development with it. As a result I was able to prototype/finalize more applications in a year than in my whole career (including college projects).

Can you make any predictions about the future of Groovy and Griffon?

Groovy has reached a tipping point IMO, it is a buzzword already. There are a few places where Groovy cannot be found but Java can (android, JME for example), but it’s just a matter of time until that happens. Griffon on the other hand may have a bright future with polyglot programmers, not just Java developers, as it supports (at the moment) 5 languages that can be mixed together on an application: Java, Groovy, Clojure, Scala and JavaFX. On the UI side it is also multi-toolkit. Swing is not the only game in town, Griffon can play nice with JavaFX, Swt, Pivot and Gtk. There may be other toolkits in the future… [dramatic pause] … soon.

Anything you’d like to plug to readers?

Sure, if you’re interested in Groovy/Grails/Griffon then do not hesitate to drop by the mailing lists. You’ll find a thriving community that spans all three projects and then some more (Gradle, Gant, Easyb, etc). Lastly, if you’re really into learning Griffon there’s a book coming up Griffon in Action written by 2 out of 4 of the project’s founders.

I’d like to thank Andres Almiray for his insights. If you found his Groovy stuff interesting, keep up-to-date on the latest on twitter.

Related posts:

1. Why I’m moving my projects to GitHub
2. Get groovy for better shell scripts
3. A programmer’s 2009 resolutions

It’s with a sad heart that we have to announce that the Kenai.com domain will be shutdown as part of the consolidation of project hosting sites now that Sun is a wholly owned subsidiary of Oracle.

This is sad because I thought Kenai had some really killer features like excellent JIRA and NetBeans integration. Nevertheless, it’s not up to me to decide.

Software is only as good as it’s community

A great project cannot thrive without people to improve and maintain it. The reason I am choosing GitHub is the number of people (especially friends) already on it. The is it’s main advantage over something like bitbucket. Git, in my opinion, has great momentum in the OSS community and is roughly equivalent to mercurial in functionality (with a few differences, obviously). Both DVCS are far superior to their non-distributed counterparts. To sum up the biggest benefit in a phrase: “Local commits FTW!”

Moving to GitHub has already paid off because my recent javascript-stacktrace project already has over 50 watchers and a couple forks.

Follow me

You should follow me on GitHub if you are interested in any of my projects, or if you think I might be interested in yours. Only good can come from reaching out. I look forward to seeing your OSS projects.

New links for my other projects (update your bookmarks or whatever):

• Cheqlist – JavaFX desktop application for Remember The Milk
• GroovyRTM – Groovy API for Remember The Milk

Agree about GitHub? Chime in on the “preferred version control system” poll and tell me if I’m full of crap in the comments.

Related posts:

1. Introducing GroovyRTM: A Groovier way to Remember The Milk
2. Follow-up: Why programmers should twitter
3. Interview with Andres Almiray

Lately, I’ve been working on updating my old script. Since it was written, we’ve seen lots of major browser releases and the introduction of the V8 Javascript engine used by Google Chrome.

Updated browser compatibility

Browsers that are fully-supported and well-tested:

1. Firefox (and Iceweasel) 0.9+
2. UPDATE: Chrome 1+ now perfectly supported
3. Safari 3+
4. IE 5.5+
5. Opera 9+
6. Konqueror 3.5+
7. K-Meleon 1.5.3+
8. Epiphany 2.28.0+

Browsers that are supported in almost all cases but not as well-tested:

1. Chrome 1+ – One bug (feature?) that may be in Chrome reporting functions as anonymous when they aren’t. HOWEVER, Chrome’s stack gives us line numbers AND column numbers, so we can see exactly where our problem is – even in minified Javascript! Sweet! Chrome 1+ now fully supported.
2. Opera 7-8 – Opera is dead to me now. Opera 10+ has removed the error.stack info we needed and introduced error.stacktrace, but it seems very unstable. Argh.

More info about compatibility can be shown with the BrowserShots of the test suite.

Now socially coded

I’m not going to post the code here because the source and tests are now on the javascript-stacktrace project on GitHub. You can download it here.

Follow it, file bugs, and make comments there. If you have improvements to make, please fork the project and then contact me or do a “push request”. I’ll make sure you get credit ;)

UPDATE: Øyvind Sean Kinsey has added memoization (caching the implementation) for the mode and XHR bits as well as the ability to pass an existing Javascript Error and get a stacktrace. We’re working on tests and you should see project updates soon. Thanks, Øyvind!

var lastError;
try {
    // error producing code
} catch(e) {
   lastError = e;
   // do something else with error
}

// later...
printStackTrace({e: lastError}); //Returns stacktrace from lastError!

Try it out!

The code is in use on my blog. Click here to give it a spin.

function foo() {
    var blah;
    bar("blah");
}

function bar(blah) {
    var stuff;
    thing();
}

function thing() {
    if (true) { //your error condition here
        var st = printStackTrace();
        alert(st.join("\n\n"));
    }
}

foo();

Random note: one cool suggestion I saw was to assign printStackTrace to window.onerror. Pretty brilliant if you ask me.

I want to thank the guys who contributed to the script: Luke Smith, Loic Dachary and Johan Euphrosine.

I could use a bit of help getting the Chrome and Opera bugs worked out. I’m sure some of you guys who remember how to write software can help. Suggestions and whinings are welcome as long as they don’t get out of hand in the comments.

Javascript Stacktrace on GitHub

Related posts:

1. A Javascript stacktrace in any browser
2. Using Python to update your FeedBurner stats
3. How to highlight search results with JavaScript and CSS

What I’m NOT comparing

In a word: speed. This was a significant hardware upgrade from my last computer, so I’m not going to say anything how everything is so much faster, smoother blah blah because it would’ve been anyway and that’s not useful to you or anyone. Also, virtualization: I know that I can get X or Y if I just use VirtualBox. I’m going to ignore that here for simplicity.

Tools

Before I make stupid lists, I should note that I was working on an Ubuntu Karmic Koala, so I had all of the pre-packaged nice-ities that come with that.

Now, in no order whatsoever:

• Dock – Mac has a built-in dock, Linux has AWN and Gnome-Do Docky. IMO, Linux wins because you have more options for customization.
• Terminal – Both systems have a built-in terminal. I’m a bash user and that came with both. One part where Linux shines is that a lot more tools build themselves to be launched by the Terminal by default. For example, try typing “which firefox” in the Mac terminal. Nope. On Mac, you can use “open -a [application]” to do this. I’ll reluctantly say tie here then ;)
• Browsers – Oh sweet! I can get Safari on a Mac without any hacks. Don’t care. Long as you have Google Chrome you basically have Safari with a faster Javascript engine as far as I’m concerned. No winner
• Code editing – All the dev. tools I had in Linux, I still have on Mac. Same vim. Same NetBeans (well, as far as you care). Same IntelliJ. BUT… I now have access to TextMate and Coda. So far I suck with both but a bunch of cool people say they rock so I’m going to give Mac the win here. Oh, and XCode tools which I’d need if I ever wanted to write an iPhone app. That, too.
• Window organization/effects – I’m a big fan of multiple desktops. Both OSes have this, but I’d argue that Compiz on Linux has way more customization options. Both have cool the exposé. On Mac, though, you can’t move windows between desktops with a keyboard shortcut (Are you kidding me? There has GOT to be a Mac tool that allows this). Linux wins here.
• Multiple monitor support – I’m breaking out the cliché phrase “It just works” here and I bow humbly to Mac OS here. No more messing with xorg.conf files or dealing with the crappy NVidia tools.
• Notifications – Should die anyway because they kill productivity, but if you care more stuff on Mac (that I’ve seen) integrates with Growl than those that integrate with libnotify on Linux. Whatever.
• App launching – Gnome-Do is pretty much a really good rip-off of Mac’s Quicksilver. Love both. To me they’re pretty much equivalent except that I’m seeing more plugins/customization options for Gnome-Do.
• Backups and scheduling – Both systems have cron, so that’s not an issue. Both integrate well with Dropbox (<3 that service). One thing Mac has over Linux here is built-in Time Machine. It integrates really well with my Time Capsule at home, and most of you (except Chris Coyier, sorry dude that sucks) have shared good experiences with it. It does annoy me that I can’t configure when backups run, but I’m not going to whine until it bites me harder.
• Dashboard – Not much of a comparison, really. One thing Linux has that I can’t seem to find a good replacement for is conky. Only sorta-not-really replacement is the Mac dashboard, which does look sweet granted.
• App updates – The Synaptic package manager pretty much kicks the crap out of all other app management systems. That said, I’ve found the Mac AppFresh to be marginally useful for keeping stuff up-to-date. Linux still wins here, though.
• Presentations – I didn’t buy Keynote, so I can’t compare it with Linux offerings. Only reason I put it here is that I do care about it and if anyone has any insights that’d rock.
• UPDATE: VPN – I forgot how cool the built-in VPN is on Macs. It is much more painful in Linux (in my experience) Score +1 for Mac there.

I’m sure I forgot or don’t know about some tools. Leave a comment and I’ll answer.

Keyboard shortcuts

I’ll start by saying that switching to a mac keyboard still f***s me up often. I’ve installed DoubleCommand to help alleviate some of my problems, but this whole “Home not being start of line” stuff really messes with me. If someone could point me to a good guide on with keyboard shortcuts for editing (like selecting a single word) I’ll buy you a beer or something equivalent.

Other than my initial whining, most everything can be hooked to a keyboard action and I hardly have to touch my sweet multi-touch touchpad. I’d say Mac OS generally equivalent to Linux other than the whole can’t move windows to workspaces (seriously, WTF). A beer if you can help me figure out how to do that, too. On second thought, Linux wins because it does have a lot more places you can configure shortcuts (good), but they’re often duplicated and could be conflicting and confuing. Ok, on third thought nobody wins.

Other random carp

This multi-touch thingy is pretty sweet. I know Linux has some multi-touch libs but I haven’t tried them out. More on that later, but I’m betting Mac wins.

I know I omitted a lot of stuff. Probably because I don’t care about it, but maybe I do and I just didn’t think of it. This is my setup and won’t work for you. That said, advice welcome. :)

At this point, there are no winners, just differences. That’ll change once I feel like I’ve given Mac OS a fair shot. FWIW, this is good news for Linux. Even if I switch to Mac, right now I can’t find that much more sweetness here (unless I have to write iPhone apps). Good work Linux community, you’re getting there. Keep it up.

Oh, one more thing… I don’t normally do this, but this video was the best review of Mac OS “Snow Leopard” and probably the funniest thing I’ve seen in 2009. Enjoy!

Related posts:

1. Mindlessly easy ways try out Linux or Solaris
2. Firefox add-ons for productivity
3. 8 steps to my personal Firefox setup for productivity