The problem that those of us with NADD face is the second we are waiting for something to finish, we [alt|cmd]+Tab to a browser and start reading slashdot, rotten tomatoes, daringfireball, or any of a million other sites. Likely, you classify this “quick field trip” as multitasking and because you are so very good at multitasking (despite what all the experts say), you disregard the fact that your build was probably finished 10 minutes ago.

Well, I finally came to grips with these facts:

- I cannot multitask. I can only context switch. And user-to-kernel-to-user is getting slower in my old age.
- Half the time I’m looking at my browser, I’m flipping back to see if the build has finished. Further exacerbating the context switch problem.
- I cannot bring myself to firewall off my favorite, “industry news” sites.

So with that, I decided to fix what I could. That is, decreasing my “response latency” for “asynchronous tasks.” You’ll need to be running on a Mac. But you’re a super leet programmer so I’m sure you already are. OS X has a frickin’ awesome text-to-speech capability. Bundled along with that is a command line app: say.

Try it out, type at the terminal: say “Why didn’t I think of that?”

Now, for those of you who remember Dr. Spaitso, you probably just shit yourself at how great the default voice sounds. (Though, if you are my wife, you are dully unimpressed.)

So I’m sure you see where I’m going with this. Let’s get down to it.

Open up your GRAILS_PROJECT/scripts/Events.groovy file, add the following:

eventStatusFinal = { msg ->

    String.metaClass.say = {
        ["osascript", "-e", "tell application \"iTunes\" to set sound volume to 30"].execute()
        sleep(300)
        "say \"${delegate}\"".execute().waitFor()
        ["osascript", "-e", "tell application \"iTunes\" to set sound volume to 100"].execute()
    }

    if(msg.startsWith("Server running.")) {
        "Application is running".say()
    } else {
        msg.say()
    }
}

I made this a metaClass option originally because it came from a groovy bootstrap groovy script. You could just as easily make it a local closure or method.

The gist is that every time an event completes, it will read aloud the status line. But first, it turns down the music playing in itunes. (It just so happens that I have iTunes turned all the way up and use the system volume to regulate. If you don’t do that, you will probably have to adjust the volume values)

Pretty neat! I actually use this in some fashion in all my projects now… maven, ant, even xcode. It really helps.

Constraints! HasErrors!

Never, EVER forget to add constraints to your composited domain model at the same time you forget to check hasErrors after save. Grails will not tell you when your domain.save() fails. When in doubt, pass failOnError:true to your save() calls.

Where would this hit you? Most of the time scaffolding takes care of your db operations. But, it happened to me in several places. When I was loading a default set of data in Bootstrap.groovy, I never saw my new objects showing up. The other, more devious situation was when I was using the Nimble plugin and extending the User object Nimble provides with some dependent classes. I kept getting the wicked “hibernate transient – must save” exception.  Lastly, most of the time you are wrapping your CRUD in services, you will need to watch out for this.

Watch That Naming!

Many times I’ll create a class manually in the appropriate bucket for what I want to do. For example, instead of going through grails create-integration-test package.Class, I’ll just go create a new class under $GRAILS_APP/tests/integration/package. I spent a good hour trying to figure out why my integration test was not being run. And then I found it. I named my test ClassNameTest.groovy. When it should be ClassNameTest[s].groovy! Doh!

Grails Interactive

The second you have a few maven dependencies and/or plugin dependencies, it can take grails multiple seconds (On average, 3 for me) just to resolve the dependencies of your app. This is a serious flaw in Ivy (the underlying dependency management grails uses). Unfortunately, this is what we are stuck with. Run grails interactive to speed things up. The JVM starts up and keeps running after each command. (The dependency checks go away, but things are still slow.) Further, regardless of how high you specify max memory, it still fails with out of memory errors every few run-app cycles. There must be a few leaks still laying around.

File Uploads

Something no one tells you and is found no where on the grails.org documentation is how to get the true filename of an uploaded file. If you just do what the docs tell you via:

def uploadedFile = request.getFile("file")
uploadedFile.name

the result is that the filename is “file”. Yes, real fucking useful!

Use, instead:

def uploadedFile = request.getFile("file")
uploadedFile.originalFilename

Leveraging established Java codebase

One of the best aspects of Groovy, in my eyes, is being able to leverage Java. Grails goes further and allows you to specify remote maven artifacts your app depends on. This is great. Until you want to reuse Java domain objects.

You see, GORM doesn’t support mapping both Groovy and Java objects if either of those objects refer to each other. So you can’t have a Groovy domain object extend a Java object. You can’t have a Groovy domain object contain a Java domain object. There is an issue open on this:

http://jira.codehaus.org/browse/GRAILS-4996

I’m bringing this up here because this limitation definitely isn’t documented anywhere. And I didn’t discover it until step 98 out of 100 in integrating my codebases.

Further, I found that GORM errors are a bit limited in some circumstances (this one). If you hit a problem where GORM is throwing a cryptic message, and it is about a object in your hasMany map, try removing the hasMany and adding it as a single relationship. The error message gets much more verbose.

(if hasMany, move it to a single to get a better error message from hibernate)

class Tool {
	//static hasMany = [ widgets : Widget, rocks : com.acme.crud.Rock ]
	static hasMany = [ widgets : Widget ]
	Rock rock
	String name

Never *ever* use reserved variables in UrlMappings.

Wait, there are reserved variable names? Yeah, that’s what I said! Say you have a parameter your controller needs called action. $action actually maps to the function in your controller that will be called. So if you use this, suddenly, no request will get to your controller. (Unless it just so happens the type you enter in the URL exists as a function.) This happens even if you use the extended UrlMapping syntax where you can specify the action you want to use. In my case:

"/download/$id/$version/$preview/$action/$filename" {
			controller = "downloadController"
			action = "index"
			constraints {

			}
		}

On no, don’t do that! Change $action to something else instead.

It quickly came time to test out some of the integration. The very first question was how to load resources for testing. E.g., I have a service that handles file uploads. How do I pass it a test file?

The documentation isn’t straightforward on this. In fact, I found the solution attached to a bug report for grails 1.1. It’s pretty simple, once you pull in the spring packages:

import org.springframework.core.io.ClassPathResource
import org.springframework.core.io.Resource

And then in your testXXX() function:

Resource resource = new ClassPathResource("resources/crying-baby.jpg")
def file = resource.getFile()
assert file.exists()

And you are good to go. Note the assert at the end to verify things are working.

1. Download and extract latest binary from solr website.
2. Within solr folder is examples/
3. Remove example-DIH/
4. Remove exampledocs/
5. Create directory /opt/local/share/solr-VERSION (solr-1.4 at time of this writing)
6. Copy contents of examples to /opt/local/share/solr-VERSION
7. Create a symlink /opt/local/share/solr-VERSION -> solr

And here is the SMF:

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM
'/usr/share/lib/xml/dtd/service_bundle.dtd.1'>

<service_bundle type='manifest' name='Solr'>

<service
	name='network/solr'
	type='service'
	version='1'>

	<create_default_instance enabled='false'/>
    <single_instance/>

	<dependency name='fs'
		grouping='require_all'
		restart_on='none'
		type='service'>
		<service_fmri
			value='svc:/system/filesystem/local'/>
    </dependency>

    <dependency name='net'
		grouping='require_all'
		restart_on='none'
		type='service'>
		<service_fmri value='svc:/network/loopback'/>
    </dependency>

    <dependent name='solr_multi-user'
	 	grouping='optional_all'
		restart_on='none'>
		<service_fmri value='svc:/milestone/multi-user'/>
    </dependent>

    <exec_method
		name='start'
		type='method'
		exec='/opt/local/lib/svc/method/svc-solr'
		timeout_seconds='60'>

		<method_context working_directory='/opt/local/share/solr'>
        	<method_credential user='root' group='root' />
      	</method_context>
    </exec_method>

    <exec_method
		name='stop'
		type='method'
		exec=':kill'
		timeout_seconds='60'>
    </exec_method>

	<template>
		<common_name>
			<loctext xml:lang='C'>
				solr
			</loctext>
		</common_name>
	</template>

</service>
</service_bundle>

And the very short script it calls to start Solr:

svc-solr:

#!/bin/sh                                                                                 

nohup java -Dsolr.data.dir=/opt/local/share/solr/data -jar start.jar &

That’s about it.

There is a command called truss which allows you to follow along as something is being executed. See: man truss.

What I found really cool is that you can modify your existing SMF script from the command line. No need to make changes to xml and then reimport.

So back to the tip, if a service isn’t starting and you have eliminated all of the other possibilities with configuration, you need to see what the service script is doing. In my case, I was having a hell of a time with Red5. So here is what you can do:

Use to find what is the start method. In this case, /opt/local/share/red5/red5.sh:

[user@host ~]$ svcprop -p start/exec red5

Install a new start method:

[user@host ~]$ svccfg -s red5 setprop 'start/exec = “truss -f -a -o /tmp/truss.out red5.sh”'

Now try to start the service again:

[user@host ~]$ svcadm clear red5
[user@host ~]$ svcadm enable red5

Notice the clear command! That tripped me up several times when you would enable the service, but no warning was thrown about it being in maintenance mode.

Now go check the contents of /tmp truss.out to see what happened. My problem for red5 is that I was using a relative path to red5.sh. I specified in the environment that the working directory was /opt/local/share/red5. However, it appears that you still must specify an absolute path to the script.

It turns out that the Red5 team actually created an SMF configuration. However, you have to download the source to get to it. I was half way through my own before I saw the change entry on the Red5 commit mailing list. However, it used some funky paths. I took this and fixed it for my Joyent specific installation.

First, download the tar of red5 from red5.org. Extract the contents to /opt/local/share/red5-VERSION. Create a symlink from /opt/local/share/red5 to red5-VERSION.

Here is the SMF config:

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM
'/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<!--
Solaris Service Management Facility for Red5

The idea for this file came from a post by Paul Oswald
URL: http://pauloswald.com/article/29/hudson-solaris-smf-manifest
Modified for use with Red5 by Paul Gregoire (mondain@gmail.com)
Modified for Red5 on Joyent accelerators by The Dude

More info: 

http://www.sun.com/bigadmin/content/selfheal/smf-quickstart.jsp

http://www.sun.com/bigadmin/content/selfheal/sdev_intro.html

-->
<service_bundle type="manifest" name="Red5">

<service
	name="application/red5"
	type="service"
	version="1">

	<create_default_instance enabled="false" />
	<single_instance />

	<dependency name='fs'
		grouping='require_all'
		restart_on='none'
		type='service'>
		<service_fmri
			value='svc:/system/filesystem/local'/>
    </dependency>

    <dependency name='net'
		grouping='require_all'
		restart_on='none'
		type='service'>
		<service_fmri value='svc:/network/loopback'/>
    </dependency>

    <dependent name='solr_multi-user'
 		grouping='optional_all'
		restart_on='none'>
		<service_fmri value='svc:/milestone/multi-user'/>
    </dependent>

	<method_context working_directory='/opt/local/share/red5'>
		<method_credential user='red5' group='red5' />
	</method_context>

	<exec_method
		type="method"
		name="start"
	    exec="/opt/local/share/red5/red5.sh"
	    timeout_seconds="0">
	</exec_method>

	<exec_method
		type="method"
		name="stop"
	    exec=":kill -TERM"
	    timeout_seconds="30"/>

	<!-- Red5 runs as a single child process so we want Wait mode-->
	<property_group name='startd' type='framework'>
		<propval name='duration' type='astring' value='child' />
	</property_group>

	<stability value="Unstable" />

	<template>
		<common_name>
			<loctext xml:lang='C'>
				red5
			</loctext>
		</common_name>
		<documentation>
			<doc_link name='red5.org' uri='http://red5.org/' />
		</documentation>
	</template>
</service>
</service_bundle>

This guy is a bit simpler as it doesn’t require a script to handle starting or stopping. It assumes red5 is installed under /opt/local/share/red5. The same applies as before for installing it:

sudo mv red5.xml /var/svc/manifest/network/

Then run the following (likely with sudo):

chown root:sys /var/svc/manifest/network/red5.xml
chmod 444 /var/svc/manifest/network/red5.xml
svccfg import /var/svc/manifest/network/red5.xml

You’ll need to have a user and group added named red5.

Now, time to go watch Avatar for the 5th time. Who am I kidding? I have to go burp Alec.

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM
'/usr/share/lib/xml/dtd/service_bundle.dtd.1'>

<service_bundle type='manifest' name='Tomcat'>

<service
	name='network/tomcat'
	type='service'
	version='1'>

	<create_default_instance enabled='false' />
	<single_instance />

	<dependency name='fs'
		grouping='require_all'
		restart_on='none'
		type='service'>
		<service_fmri value='svc:/system/filesystem/local' />
	</dependency>

	<dependency name='net'
		grouping='require_all'
		restart_on='none'
		type='service'>
		<service_fmri value='svc:/network/loopback' />
	</dependency>

	<exec_method
		type='method'
		name='start'
		exec='/opt/local/lib/svc/method/svc-tomcat start'
		timeout_seconds='-1'>
		<method_context>
			<method_credential user='root' group='root' />
		</method_context>
	</exec_method>

	<exec_method
		type='method'
		name='stop'
		exec=':kill'
		timeout_seconds='-1'>
	</exec_method>

	<exec_method
		type='method'
		name='restart'
		exec='/opt/local/lib/svc/method/svc-tomcat restart'
		timeout_seconds='-1'>
	</exec_method>

	<stability value='Unstable' />

	<template>
		<common_name>
			<loctext xml:lang='C'>
				tomcat
			</loctext>
		</common_name>
	</template>
</service>
</service_bundle>

The shell script looks like:

#!/usr/bin/sh

. /lib/svc/share/smf_include.sh

case $1 in
'start')
	echo "starting tomcat..."
	/opt/local/share/tomcat/bin/startup.sh & >/dev/null 2> /dev/null
	;;

'stop')
	echo "stopping tomcat..."
	/opt/local/share/tomcat/bin/shutdown.sh &
;;

'restart')
	stop
# give stuff some time to stop before we restart
	sleep 25
	start
	;;
*)
	echo "Usage: $0 { start | stop | restart }"
	exit 1
	;;
esac

Put tomcat.xml in /var/svc/manifest/network/tomcat.xml
Put svc-tomcat in /opt/local/lib/svc/method/svc-tomcat

I had to create the local svc/method directory so before I could put svc-tomcat where it needed to be.

Then run the following (likely with sudo):

chown root:bin /opt/local/lib/svc/method/svc-tomcat
chmod 555 /opt/local/lib/svc/method/svc-tomcat
chown root:sys /var/svc/manifest/network/tomcat.xml
chmod 444 /var/svc/manifest/network/tomcat.xml
svccfg import /var/svc/manifest/network/tomcat.xml

That’s it. You can check on the service with:

[user@host ~]$ svcs -xv tomcat

And start it via:

[user@host ~]$ svcadm enable tomcat

However, it has said that they will no longer be censoring search results for users in China because the attach originated from the Chinese government:

We have decided we are no longer willing to continue censoring our results on Google.cn, and so over the next few weeks we will be discussing with the Chinese government the basis on which we could operate an unfiltered search engine within the law, if at all.

How awesome is that?

What that tells me though is that google.cn will stop operating in the next few months. We all know that there is no way in hell the Chinese government will allow google to provide results, uncensored.

Check it out: http://googleblog.blogspot.com/2010/01/new-approach-to-china.html

Turns out, there are two sshd config settings you should pay attention to:

VerifyReverseMapping and GssAuthentication

The latter only really helps if you are connecting from a Mac (like me).  VerifyReverseMapping will tell the server to look up the host name for client IPs, and if the IP you’re SSH’ing from doesn’t have a reverse DNS entry, this will result in a DNS timeout.  (The source of most of the delay in logging in.)

So open up your sshd config file: /etc/ssh/sshd_config

Remove VerifyReverseMapping, if it exists.  The default value for it is no.  If you don’t see it, this isn’t the source of your problem.

Remove GssAuthentication, if it exists.  It also, defaults to no.

Since my first real world exposure to OpenSolaris is the process of migrating web applications away from Linux, the first thing I really had to get familiar with is the Service Management Facility framework.  SMF for short.  SMF is the Solaris equivalent of UNIX/Linux init.d, Apple’s launchd, or Windows services.

I’m coming from a decade long Linux background so let me compare and contrast init.d and SMF.

• Like init.d, SMF services are first-class objects that you can query for status and start and stop.

• Unlike init.d, if SMF services encounter an error that caused a failure of the service, SMF will AUTOMATICALLY RESTART the service!  However, it isn’t a blind restart.  If the service fails after X number of restarts, SMF will disable the service until manual intervention happens.

• SMF provides a facility to see why a service is not running or stats on running.  The command svcs -x (I remember using SerViCe Status):

[user@host ~]$ svcs -x tomcat
svc:/network/tomcat:default (tomcat)
 State: online since January  6, 2010  7:54:02 PM GMT
   See: /var/svc/log/network-tomcat:default.log
Impact: None.

Or for a disabled service:

[user@host ~]$ svcs -x tomcat
svc:/network/tomcat:default (tomcat)
 State: disabled since January  7, 2010  3:07:47 PM GMT
Reason: Disabled by an administrator.
   See: http://sun.com/msg/SMF-8000-05
   See: /var/svc/log/network-tomcat:default.log
Impact: This service is not running.

Notice the Reason field. Awesome!

• Like init.d, SMF provides start/stop/restart actions on a service.  However, the equivalent actions are named differently: enable/disable/restart.  Use svcadm enable tomcat to start a service.

• Unlike init.d, SMF provides the ability for non root users to start and stop services!

• Like init.d, SMF provides a way for services to fire up in a dependent order.

• Unlike init.d, SMF provides a way to declare those dependencies rather than relying on an integer value to start in a certain order.  On top of being much clearer to admin, this allows systems to boot up faster by starting services in parallel.

• Unlike init.d, SMF provides a logging framework that is transparent to the service.  I’ve found in my use, this is both good and bad.  My SOLR SMF config is moving the SOLR log output from the SOLR install directory to the SMF log directory.  I’ll get to that in another post.

• Unlike init.d, SMF is much more complicated then adding a simple shell script to the init.d folder.  The learning curve is a bit longer/higher than init.d.

This list is certainly not comprehensive.  You can find out more at Sun’s BigAdmin site.  Solaris Service Management Facility – Quickstart Guide

Up next, a SMF config for Tomcat…