blogit ergo sum

My Time in Sweden

2009-10-27T06:26:00.000+11:00

I lived in Sweden from 1988 to 1991. Here is a map which showed where I lived in Stortorget in Gamla Stan in Stockholm.

View Larger Map

I lived in the red building in the two left photos below which are taken from Stortorget. The photo on the right is of the same building taken from Kåkbrinken, the alley to the left the red building.


The photo on the left below is the main street in Gamla Stan and the photo on the centr is of the Grand Hotel as seen from the shore of Gamla Stan and the photo on the right is Karloniska Hospital where I worked.


After I left Stockholm I moved to Umeå which is shown on the left below. When I lived there I used to visit Vaasa in Finland shown on the right.

When I lived in Sweden I took vacations in Norway including Lofoten on the left and Tromsø in the centre of the row of photos below. I also took the Hurtigruten

Photo Credits

Machine Learning While I Work

2009-10-13T13:55:00.008+11:00

I am setting up Postfix so I have spare time as I try things out. This post is about the things I am reading or watching in the background.

Taskforce on Context-Aware Computing
I went to a lecture called Open Mobile Miner (OMM): A System for Real Time Mobile Data Analysis. There is a video here, a description of OMM here and lecture slides here (pdf).

Shonali Krishnaswamy's group are making software that does some analysis of data on a smart phone before uploading it, thereby reducing the phone's power consumption by reducing communications. Their examples include ECG output, traffic congestion metrics and taxi location data. The data in their examples is scalar and sampled at 0.5 Hz or less so it is hard to see why a simple store and forward scheme would not achieve much the same thing. Maybe I need to read their publications more deeply.

Statistical Learning as the Ultimate Agile Development Tool by Peter Norvig is a great overview of practical machine learning. The summary is focus on the data, not the code. I cannot recommend this too highly.

Learning Theory by Mark Reid was an introduction to some theoretical aspects of machine learning presented in a summer school in Canberra in January 2009. Interesting but Peter Norvig is a hard act to follow.

Now some videos of how machine learning can be applied to models of the face.

Changes of facial features on the of dominance, trustworthiness and competence dimensions in a computer model developed by Oosterhof & Todorov (2008). Your browser does not support iframes.

Now it is time to start watching a video on distributed computing

Swarm: Distributed Computation in the Cloud from Ian Clarke on Vimeo.

My First Upside Down Post

2009-09-18T06:40:00.000+10:00

˙ʇǝsdıɥɔ pǝsɐq ɯɹɐ s’ǝןɐɔsǝǝɹɟ 'uoƃɐɹpdɐus ɯɯoɔןɐnb 'ɯoʇɐ ןǝʇuı ˙sɹɐʍ ɹossǝɔoɹd ˙sʞooqʇǝu puɐ sǝuoɥd ʇɹɐɯs ɟo ǝɔuǝƃɹǝʌuoɔ ǝןqıssod ˙ǝƃuɐɥɔ ǝʌıɹp ןןıʍ ǝɔuǝɹǝɟɟıp ǝɔıɹd 000'1$ ˙sʌ 052$ ǝɥʇ ˙pǝʇıns-ןןǝʍ os ʇou sǝop pɹoʍ ʇɟosoɹɔıɯ ǝןıɥʍ pnoןɔ ǝɥʇ ɯoɹɟ ןןǝʍ sʞɹoʍ ʎpɐǝɹןɐ ǝɹɐʍʇɟos ɹǝɥʇo puɐ uozɐɯɐ 'ɯoɔ˙ǝɔɹoɟsǝןɐs 'ǝןƃooƃ ˙ǝɹɐʍʇɟos ʇɟosoɹɔıɯ ǝɥʇ doʇdɐן ǝɔɐןdǝɹ ʎɐɯ pnoןɔ + ʞooqʇǝu os ƃuıʇndɯoɔ pnoןɔ oʇ pǝʇıns-ןןǝʍ ǝɹɐ sʞooqʇǝu ˙ʇɟosoɹɔıɯ puɐ ןǝʇuı oʇ ʇsoɔ ʇɐǝɹƃ ʇɐ sǝןɐs doʇdɐן 0001$sn ǝzıןɐqıuuɐɔ ʎɐɯ sʞooqʇǝu 052$sn ɯɹǝʇ ɹǝƃuoן ǝɥʇ uı ʇnq ǝıd ɔd ǝɥʇ ƃuıʍoɹƃ ǝɹɐ sʞooqʇǝu ʎןʇuǝɹɹnɔ ¿uıɐɥɔ ǝnןɐʌ doʇdɐן puɐ ɔd ǝɥʇ oʇ op sʞooqʇǝu ןןıʍ ʇɐɥʍ ˙sǝןɐs ƃuıʇsıxǝ ɟo uoıʇɐzıןɐqıuuɐɔ pnoןɔ ǝɥʇ oʇ ʇuǝɯǝʌoɯ ɟo sǝɔuǝnbǝsuoɔ ǝɯos sɹoʇıʇǝdɯoɔ ɹıǝɥʇ ɹǝʌo ǝƃɐʇuɐʌpɐ ƃuıɔıɹd ɐ ǝʌǝıɥɔɐ ןןıʍ sıɥʇ ǝpıʌoɹd uɐɔ oɥʍ sɹǝıɹɹɐɔ uoıʇɐɔıunɯɯoɔǝןǝʇ ǝɥʇ ˙suoıʇɔǝuuoɔ ʞɹoʍʇǝu ǝןqɐʇɹod puɐ ǝןqɐıןǝɹ 'ʇsɐɟ sǝɹınbǝɹ ƃuıʇndɯoɔ pnoןɔ ʎɥdɹnɯ ˙ɹɯ sʎɐs „'ɹǝʇʇǝq ʇnq ƃuıɥʇou uǝǝq s,ʇı 'dɯnɥ ƃuıuɹɐǝן ǝɥʇ ɹǝʌo ʇoƃ ǝʍ ǝɔuo ˙sǝƃɐʇuɐʌpɐsıp ןɐǝɹ ʎuɐ ɥʇıʍ dn ƃuıɯoɔ pǝssǝɹd-pɹɐɥ ǝq p,ı ˙ǝɹoɯ ʎuɐ suoıʇɐןןɐʇsuı ǝɹɐʍpɹɐɥ ןɐɔısʎɥd ɹoɟ ƃuıʇıɐʍ sʎɐp puɐ sɹnoɥ puǝds ʇ,uop ǝʍ„ ˙sɹǝʇndɯoɔ uʍo sʇı ɟo ǝuou ɥʇıʍ ʎuɐdɯoɔ ʇuǝɯdoןǝʌǝp qǝʍ ɐ ǝɯoɔǝq sɐɥ ʇı - ǝɹɐʍpɹɐɥ ןɐɔısʎɥd ɹoɟ ʇno ƃuıʞɹoɟ pǝddoʇs sɐɥ ʇı suɐǝɯ ɥɔıɥʍ 'ǝɔıʌɹǝs ןɐnʇɹıʌ ǝɥʇ ɹoɟ ɹnoɥ ɹǝd sʇuǝɔ 08 oʇ 01 sʎɐd ǝƃuɐɹo ʎɔınɾ ˙ǝɯoɔǝq sɐɥ ƃuıʇndɯoɔ pnoןɔ ɯɐǝɹʇsuıɐɯ ʍoɥ sǝʇɐɹʇsnןןı (9002 qǝɟ 02 ǝƃɐ ǝɥʇ) ɯɐǝɹʇsuıɐɯ ǝɥʇ spuǝɔsɐ ƃuıʇndɯoɔ pnoןɔ ”˙ƃuıʍoɹƃ ןןıʇs sı ןɐɹǝuǝƃ uı ƃ3“ ˙sʇdǝɔuoɔ pɹɐʍɹoɟ ʇɐ ןɐdıɔuıɹd 'ssnɐɹʇs ןןıʍ pıɐs ”'sʇods ʇɥƃıɹq ǝɹɐ ǝsǝɥʇ“˙˙˙ ˙pǝʌɹǝsqo ǝʌɐɥ sʇsʎןɐuɐ 'ǝɔuɐɥɔ ƃuıʇɥƃıɟ ɐ ǝʌɐɥ sɯǝpoɯ ƃ3 puɐ sdıɥɔ ɥʇooʇǝnןq puɐ ıɟ-ıʍ 'sdƃ ǝʞɐɯ oɥʍ sɹopuǝʌ puɐ — ɥʇʍoɹƃ ʇɐɥʇ ɟo ɥʇƃuǝɹʇs ǝɥʇ uo ʎɹɐʌ sʇsʎןɐuɐ — ɹɐǝʎ sıɥʇ ʍoɹƃ oʇ pǝʇɔǝɾoɹd ǝɹɐ sǝןɐs ǝuoɥdʇɹɐɯs ˙pɐǝɥɐ ɹɐǝʎ ǝɥʇ uı sʇods ʇɥƃıɹq ƃuıʞǝǝs puɐ sɥʇƃuǝɹʇs ǝɹoɔ oʇ ƃuıʞooן ǝɹɐ sɹopuǝʌ dıɥɔ ʇsoɯ 'ǝןıɥʍuɐǝɯ˙˙˙ ˙sʇǝʞɹɐɯ ʍǝu uǝdo oʇ ʎɐןd ǝuoɥd ʇɹɐɯs ɐ ƃuıɹɐdǝɹd ǝq oʇ pǝɹoɯnɹ sı ˙ɔuı ןןǝp ɹǝʞɐɯ ɔd ˙ǝuoɥdı s’˙ɔuı ǝןddɐ ʎq pǝʌɹǝs ʇǝʞɹɐɯ ǝɥʇ ɟo ǝɯos qɐɹƃ oʇ ƃuıdoɥ 'ʍoןs sǝʇɐɹ sǝןɐs ɔd sɐ spıɯ pǝʇǝƃɹɐʇ sɐɥ — ɹǝʞɐɯ dıɥɔ ʇsǝƃɹɐן s’pןɹoʍ ǝɥʇ — ˙dɹoɔ ןǝʇuı sɐ 'uoıʇıʇǝdɯoɔ ǝʌɐɥ ןןıʍ ɯɯoɔןɐnb˙˙˙spןǝɥpuɐɥ puɐ sdoʇdɐן uǝǝʍʇǝq dɐƃ ǝɔıɹd ǝɥʇ ǝƃpıɹq sʞooqʇǝu sɐ ǝƃɹns oʇ pǝʇɔǝɾoɹd ʎɹoƃǝʇɐɔ ɹǝʇʇɐן ǝɥʇ ɥʇıʍ 'sǝɔıʌǝp ʇǝuɹǝʇuı ǝןıqoɯ puɐ sʞooqʇǝu 'sʞooqǝʇou ɹoɟ ʇǝsdıɥɔ uoƃɐɹpdɐus sʇı uo sısɐɥdɯǝ pǝɔɐןd sɐɥ ɯɯoɔןɐnb 'ǝןıɥʍuɐǝɯ sdıɥɔ :ʇsɐɔǝɹoɟ ssǝןǝɹıʍ 9002 ssǝןǝɹıʍ ɹɔɹ sǝuıɥɔɐɯ dx sʍopuıʍ ɹo nʇunqn ʎןןɐnsn ǝɹɐ ʇɐɥʍ ƃuoɯɐ ǝɥɔıu ɐ puıɟ ןןıʍ ɯɹoɟʇɐןd ǝןıqoɯ ǝɔɹnos-uǝdo s’ǝןƃooƃ ʇɐɥʇ ƃuıʇʇǝq sı - ʎɐpoʇ ʇǝʞɹɐɯ ǝɥʇ uo sʞooqʇǝu ɟo ʎʇıɹoɾɐɯ ʇsɐʌ ǝɥʇ uı punoɟ ɹossǝɔoɹd 072u ɯoʇɐ ןǝʇuı ǝɥʇ ɹoɟ ǝןqısuodsǝɹ - ɹǝʞɐɯdıɥɔ ǝɥʇ ʇɐɥʇ sʇsǝƃƃns oɥʍ '”ǝɔɹnos ǝןqɐıןǝɹ“ s’ʇɐǝqǝɹnʇuǝʌ oʇ ƃuıpɹoɔɔɐ s’ʇɐɥʇ ˙ǝɹɐʍpɹɐɥ ʇǝsdıɥɔ ǝןqɐʇıns ɥʇıʍ sɹǝɹnʇɔɐɟnuɐɯ ʇɹoddns oʇ ƃuıɹɐdǝɹd sı puɐ '0102 ʇnoɥƃnoɹɥʇ puɐ 9002 ǝʇɐן uı sʞooqʇǝu pǝsɐq-pıoɹpuɐ ɟo ʎɹɹnןɟ ɐ ƃuıʇɔǝdxǝ sı ןǝʇuı sʞooqʇǝu pǝsɐq-pıoɹpuɐ ɟo ǝsıɹ ɹoɟ ƃuıʎpɐǝɹ ןǝʇuı ʞooqʇǝu pǝsɐq-pıoɹpuɐ ɟo ǝsıɹ ɹoɟ ƃuıʎpɐǝɹ ןǝʇuı ʎoʎ %6˙12– puɐ bob %7˙12– pǝuıןɔǝp sʇuǝɯdıɥs ʇıun ɹossǝɔoɹd ɔd ǝpıʍpןɹoʍ 'ɯoʇɐ ʇnoɥʇıʍ ˙ǝuıןɔǝp ɔıʇɐɯɐɹp pıoʌɐ ʇǝʞɹɐɯ ǝɥʇ dןǝɥ oʇ ɥƃnouǝ ʇou ʇnq ǝɔuɐɯɹoɟɹǝd ʇǝʞɹɐɯ ןןɐɹǝʌo ǝɥʇ uı ǝɔuǝɹǝɟɟıp ǝןqɐʇou ɐ ǝʞɐɯ oʇ pǝnuıʇuoɔ (,,sʞooqʇǝu,, sןןɐɔ ןǝʇuı ɥɔıɥʍ) sɔd ʞooqǝʇou-ıuıɯ ɹoɟ ɹossǝɔoɹd ɯoʇɐ s,ןǝʇuı ˙˙˙ ؛(ʎoʎ) ɹɐǝʎ ɹǝʌo ɹɐǝʎ %4˙11– puɐ (bob) ɹǝʇɹɐnb ɹǝʌo ɹǝʇɹɐnb %0˙71– pǝuıןɔǝp sʇuǝɯdıɥs ʇıun ɹossǝɔoɹd ɔd ǝpıʍpןɹoʍ '80b4 uı 80b4 uı sʞooqʇǝu oʇ sdoʇdɐן sʍopuıʍ ɯoɹɟ ǝʌoɯ sıɥʇ ʇɹoddns oʇ sǝıɹoʇs ǝɯos ǝɹɐ ǝɹǝɥ ˙ʎʇıןıqoɯ puɐ ʎɹʇsǝɔuɐ ǝuoɥd ɹıǝɥʇ ɯoɹɟ sǝɯoɔ ʇɐɥʇ uoıʇdɯnsuoɔ ɹǝʍod ʍoן ɟo sǝƃɐʇuɐʌpɐ ןɐuoıʇıppɐ ǝɥʇ ǝʌɐɥ ʎǝɥʇ ˙sƃuıɹǝɟɟo pnoןɔ ɹǝɥʇo puɐ uoıʇɐzıןɐnʇɹıʌ 'sɐɐs 'sǝɔıʌɹǝs qǝʍ ɹoɟ sʇuǝıןɔ ǝʇɐnbǝpɐ ǝʞɐɯ ǝuoɥd ʇɹɐɯs ǝןqɐdɐɔ ʎɹǝʌ puɐ sʞooqʇǝu ƃ3 ˙sǝıƃǝʇɐɹʇs ƃuıʇndɯoɔ doʇʞsǝp pǝnsɹnd ʇou ǝʌɐɥ puɐ sǝɔıʌɹǝs ɟo sǝdʎʇ ǝsǝɥʇ uo ʎןǝɹıʇuǝ sǝssǝuısnq ɹıǝɥʇ pǝsɐq ǝʌɐɥ oɥʍ ɯoɔ˙ǝɔɹoɟsǝןɐs puɐ ǝɹɐʍɯʌ 'ǝןƃooƃ sɐ ɥɔns sǝıuɐdɯoɔ ʎq uǝʌıɹp uǝǝq sɐɥ ʇuǝɯdoןǝʌǝp ɹıǝɥʇ ˙sɹɐǝʎ 01 ʇsɐן ǝɥʇ uı ǝʌıʇɔǝɟɟǝ ʎןɥƃıɥ ǝɯoɔǝq ǝʌɐɥ ǝsǝɥʇ ɟo ʇsoɯ ˙ suoıʇɐɔıןddɐ qǝʍ puɐ sɐɐs 'uoıʇɐzıןɐnʇɹıʌ ƃuıpnןɔuı sǝɔıʌɹǝs pnoןɔ ɟo sǝdʎʇ ʎuɐɯ ǝɹɐ ǝɹǝɥʇ ˙uoıʇɐzıuɐƃɹo uɐ ɥƃnoɹɥʇ pǝʇɐɔıןdǝɹ ǝq oʇ pǝǝu ʇou sǝop ǝƃɐɹoʇs ʞsıp ʎʇıןıqɐıןǝɹ-ɥƃıɥ sɐ ɥɔns ǝɹnʇɔnɹʇsɐɹɟuı ǝʌısuǝdxǝ ʇɐɥʇ ʇıɟǝuǝq ןɐuoıʇıppɐ ǝɥʇ sı ǝɹǝɥʇ ˙ʇuǝıɔıɟɟǝ ǝɹoɯ ɥɔnɯ sı ɹǝʌɹǝs ןɐɹʇuǝɔ ɐ uo ǝɹɐʍʇɟos ǝɥʇ ƃuıuunɹ ǝɹoɟǝɹǝɥʇ ˙%01 uɐɥʇ ssǝן ɥɔnɯ 'ʍoן ʎɹǝʌ sı ǝɹɐʍʇɟos sıɥʇ ɟo ǝƃɐsn ǝɔɹnosǝɹ ɹǝʇndɯoɔ ǝƃɐɹǝʌɐ ǝɥʇ ˙ǝʌısuodsǝɹ ǝq oʇ ǝɔɐɟɹǝʇuı ɹǝsn ǝɥʇ ʇuɐʍ noʎ puɐ suoıʇɐʇndɯoɔ ǝsuǝʇuı ǝɥʇ sǝop ʇı uǝɥʍ ʎɐp ɹǝd sǝʇnuıɯ ʍǝɟ ǝɥʇ 'ǝƃɐsn ʞɐǝd ʇɹoddns oʇ ƃuıʎɐd ǝɹɐ noʎ 'ʇı ʇɹoddns oʇ ǝɹɐʍpɹɐɥ ɔd ǝʌısuǝdxǝ ʎnq noʎ uǝɥʍ ˙ǝɯıʇ ǝɥʇ ɟo ʇsoɯ ƃuıɥʇou sǝop ʇı ˙sǝןɔʎɔ ʎʇnp ʍoן ʎɹǝʌ sɐɥ 'sɔıɥdɐɹƃ ʎʇıןɐnb ɥƃıɥ ǝʞıן ǝɹɐʍʇɟos ǝʌısuǝʇuı ʎןןɐuoıʇɐʇndɯoɔ uǝʌǝ 'ǝɹɐʍʇɟos ʇuǝıןɔ ʇsoɯ ˙sɹǝʇndɯoɔ ןɐuosɹǝd s,ǝןdoǝd ʎuɐɯ uo ʇı ƃuıop uɐɥʇ ɹǝısɐǝ sı uoıʇɐɔoן ןɐɔısʎɥd ǝuo uı ƃuıuunɹ ǝɹɐʍʇɟos ƃuıpɐɹƃdn puɐ ƃuıuıɐʇuıɐɯ ǝsnɐɔǝq uoıʇɐzıןɐɹʇuǝɔ ɥʇıʍ ʎןןɐɔıʇɐɯɐɹp sǝsɐǝɹɔǝp (oɔʇ) dıɥsɹǝuʍo ɟo ʇsoɔ ןɐʇoʇ ˙sʇuǝıןɔ ɹǝןןɐɯs ǝʌɐɥ puɐ ƃuıʇndɯoɔ ǝzıןɐɹʇuǝɔ-ǝɹ oʇ sı ʇı ǝsıɹdɹǝʇuǝ uı puǝɹʇ ɹoɾɐɯ ʇuǝɹɹnɔ ɐ

Google Wave

2009-09-15T06:10:00.000+10:00

The has been a lot of news about Google Wave this year.

Wave is Google's new browser-based integrated real-time multi-media collaboration system. As one collaborator types text, or adds a video it appears in the other collaborators' browsers. The sounds like

an online conferencing system like Webex, plus
a collaborative version controlled document sharing system, a real-time Wiki, plus
always-on video conferencing plus
integrated communication like Skype , plus
it is free and browser based (for HTML 5 capable browsers).

PC Mag say a mobile version is in the works.

Not surprisingly, Rasmussen said he sees the mobile experience as one that's absolutely essential to the future of Wave, especially as more and more people generate and consume content while on the go. Gundotra also highlighted an on-stage demo where both an Android phone and an iPhone ran Wave via the Google Web Toolkit, proving that the browser is indeed the OS of the future.

This sounds like a fixed/mobile convergence killer-app.

The mobile phone companies who sell usage based data plans and the companies who provide mobile phone and network infrastructure must be happy about this. (Contact me to inquire about purchasing my report on this subject).

As is often the case, Tim O'Reilly has the most illuminating story I could find in Google Wave: What Might Email Look Like If It Were Invented Today?

We were excited to join Google and help create what would become Google Maps. But we also started thinking about what might come next for us after maps. As always, Jens came up with the answer: communication. He pointed out that two of the most spectacular successes in digital communication, email and instant messaging, were originally designed in the '60s to imitate analog formats — email mimicked snail mail, and IM mimicked phone calls. Since then, so many different forms of communication had been invented — blogs, wikis, collaborative documents, etc. — and computers and networks had dramatically improved. So Jens proposed a new communications model that presumed all these advances as a starting point....

We started with a set of tough questions:

Why do we have to live with divides between different types of communication — email versus chat, or conversations versus documents?
Could a single communications model span all or most of the systems in use on the web today, in one smooth continuum?
How simple could we make it?
What if we tried designing a communications system that took advantage of computers' current abilities, rather than imitating non-electronic forms? ...

Is it a particle or a wave? It's both.

First generation email/IM integration let you see when someone was online, and opt to instant message someone rather than send them an email. Wave simply erases the distinction.

If both people are online at the same time, a wave acts just like an instant message -- except that you see each character as it is typed, just like in subethaedit. "In our experience, a lot of time in IM is spent waiting for the other person to press

It's both. First generation email/IM integration let you see when someone was online, and opt to instant message someone rather than send them an email. Wave simply erases the distinction. If both people are online at the same time, a wave acts just like an instant message -- except that you see each character as it is typed, just like in subethaedit. "In our experience, a lot of time in IM is spent waiting for the other person to press

A key point here is that Google's relentless focus on reducing the latency of online actions is bringing the online experience closer and closer to our real world experience of face-to-face communication. When you're talking with someone, you know what someone is saying before they finish their sentence. You can respond, or even finish their sentence for them. So too with Wave.

What I need from a 3G Netbook

2009-09-14T07:01:00.000+10:00

I could use one know for

working on the train
working at cafes while waiting for the kids
working in the country while visiting friends and family

My PC and laptop seem like overkill for researching on the web, emailing, writing reports and building a few models in a spreadsheet. They also use a lot of electricity and take up space.

To be an effective replacement for the PC and laptop, a netbook would need to have

Be reasonably priced. $200 would be nice.
Have a reasonably priced connection plan.
Have access to cheap or free software

Web browser
Word processor
Drawing tools
Spreadsheet

Reliable connection with good coverage.
Good battery life. 6 hours would be nice

Basic set of applications

Gmail with tasks
Google calender
Google Docs
GIT for source code management
YUML
Ubuntu or Windows XP with cygwin
Gnu tools
VNC or WRD
ssh

That would get me started. It would be nice to have Eclipse and a local word processor but running these over a remote shell would be more than adequate. I worked that way with all my heavy tools on VMware instances for years and it worked well. The VMware instance were hosted in a data center and backed up regularly

Typealyzer Analysis of Blogit Ergo Sum

2009-09-12T06:04:00.000+10:00

About | Blog | Community

The analysis indicates that the author of
http://peterwilliams97.blogspot.com
is of the type:

INTJ - The Scientists

The long-range thinking and individualistic type. They are especially good at looking at almost anything and figuring out a way of improving it - often with a highly creative and imaginative touch. They are intellectually curious and daring, but might be physically hesitant to try new things.

The Scientists enjoy theoretical work that allows them to use their strong minds and bold creativity. Since they tend to be so abstract and theoretical in their communication they often have a problem communicating their visions to other people and need to learn patience and use concrete examples. Since they are extremely good at concentrating they often have no trouble working alone.

about

The Pessimist on Your Team

2009-09-11T22:52:00.001+10:00

Pessimists have a bad reputation.

Today I came across an article called How to Handle the Pessimist on Your Team and it made me think how things can go really bad when you are running a complex project without being regularly subjected to healthy scepticism. A little later in the day I read Paul Krugman's How Did Economists Get It So Wrong? which described how the weight of opinion of economists had drowned out rational scepticism about the "efficient market" theory.

Weight of opinion is a powerful force and when the majority of people in a group believe something is correct it can be a bit rough for the minority who don't share that belief.

I once worked at a software company where some popular project managers told developers who did not believe their project schedules that they were being pessimists and they should cheer up. These project managers followed most of the strategies from How to Handle the Pessimist on Your Team and added a few of their own. They were pretty sweet about it and everybody including the pessimists ended up liking them. They got great results as well. Hard work, personal charm and genuine consideration for others, all of which these guys had, are stronger forces than the things I am discussing in this post.

After a few projects with these guys I started recording people's estimated completion dates at the start of projects and found that everyone was under-estimating the dates (the customer deliverables were in reasonable shape through schedule padding and some pragmatic last minute feature dropping). No surprises there, except that absolutely everybody including the "pessimists" underestimated the schedule. The interesting thing, and the point of this post, was how the estimates of the "pessimists" compared to the estimates of everyone else. The pessimists were the people whose estimated schedules were later than the consensus schedule estimate.

My explanation for this at the time was that everybody was trying to estimate the schedule accurately and fit in with the rest of the team as well. Those who valued giving accurate estimates over fitting in with the team gave estimates closer to the correct date and further from the consensus date. Perceived pessimism correlated with divergence from the consensus schedule and was largely independent of the true schedule.

That reminded me of how NPR thought it was being unbiased when it was in the middle of American mainstream opinion which was roughly halfway between Republicans and Democrats. This automatic reinforcement of the consensus view inhibits clear thinking. If Colombus had been an NPR journalist he would have accepted the consensus view of his time that the Earth was flat and not sailed to the Carribean.

Reinforcement of a core set of beliefs over time is one of the way organisations evolve. That is just how things are. Like most people I find myself drawn to organisations who share my core beliefs. However I find this tendency works against me when I need to be objective, such as when I need to meet schedules for critical projects involving many unknowns. In these cases, when nature cannot be fooled, I know of no substitute for people who speak their minds about what I am doing wrong rather than tell me what I want to hear. Steve McConnell created some procedures for doing this that I have been using for over 10 years: the invaluable Top 10 Risks List, anonymous reporting channels (I prefer people trusting me that I appreciate them telling what I am doing wrong) and objective metrics from automated testing.

I have spent my most of my working life at companies that have encouraged a diversity of opinion and the people I work with are usually comfortable with practices that attempt to discover objective truths, like the Steve McConnell procedures. In many cases they have wanted to know what they were doing wrong so they could improve. However I have also regularly encountered people in nominally technical roles who didn't like the "bad news" these procedures revealed. So, in my experience, there are still some technical managers out there trying to manage perceptions instead of managing reality and leaving the perceptions to the marketing folks. That's why I get suspicious about the motives of technical managers who talk about managing the pessimists in their teams.

The Sociology of Software Development Organisations

2009-09-11T06:38:00.000+10:00

Last week I traded some emails with a person (I will call him Dangerous) who still works at a company I wrote a recent post about. In that post I discussed an event where the company had benefited by acting on experimental evidence rather than following standard processes. I wondered if as a result of this event the company had learned to learn from experimental evidence, or if it had simply incorporotated the solution into its processes.

Dangerous told me he remembered things differently. He remembered me introducing some software development practices and a style of doing software development. There was the continuous integration and testing with high coverage tests and static analysis thatI mentioned in my post and pervasive unit testing, short development cycles, short and focused daily meetings and automatic reporting of predictors of code maturity that I didn't mention. My style was to focus on measurably working software and address customer needs. He liked this approach because he thought it was mostly what effective development teams did anyway and it was easier for him to work with an approach that was mostly right rather than spend time trying to fix what was wrong with the approach.

Dangerous told me that after I left the company most of the practices I had introduced had been downplayed until a new GM arrived. The new GM was an Agile enthusiast and he introduced Agile by re-using many of the tools my group had developed. Dangerous said he liked this approach too because he thought it was mostly what effective developers did anyway and it was easier for him to work with an approach that was mostly right rather than spend time trying to make the approach perfect. He also told me what he didn't like: the buzzwords and the Agile consultants who knew the buzzwords but didn't understand large-scale software development. A small price to pay for sane management in his opinion.

What the new GM did made sense to me but there is not much to learn from people you mostly agree with. This post is about why the organisation moved in a completely different direction between my (and other people's) somewhere-in-middle-management) approach and the new GM's top-down approach.

The biggest resistance to the empirical customer-focused approach I favoured came from other technical middle managers. These middle managers had the key skills of management. They were well-liked got things done, could deliver to a schedule, were well-organised and sensitive to risk, and were technically smart and good communicators.

So how could people who seemed smarter than me take a direction that seemed to be less productive than the one I had left them with?

My guess is that it only looked that way to me and the experienced developers, not to the other middle managers. Software development processes have at most a third-order effect on companies with mature software product lines. All the critical features have already have been added so new software development has only a second order effect on a company's success, and processes have only a second order effect in software development ,way below people. Once the development middle managers deal with the the first and second order issues they may not have time for the third-order issues like effective software development. And the higher they go up the organisational ladder in many development organisations, the less time they have. Thus increasing infuence correlates to decreasing interest. This is a rational reaction to an organisational structure.

The second issue the middle-managers had with the empirical approach was that it was disruptive to the organisational culture. This system I set up was automated and cut through a lot of the human communications that dominated the development work at the time. The middle-managers had spent a lot time making the communications-intensive style work so learning a new style was going to be a lot of effort for them.

Stated in that way, my approach sounds like an irrational reaction to my circumstances in the company. The best I can do to explain it is to trace it back to the time of previous GMs who were focussed on the first and second order issues of customer needs and retaining and motivating good staff. These GMs delegated all the third order stuff and that is how it became my ecological niche when I came across the technical problems of large scale software development.

Further Reading

Electronic Medical Records Bonanza?

2009-09-10T05:58:00.002+10:00

Big Bucks in Health IT!, quoting from http://www.healthcareitnews.com/news/global-market-hospital-it-systems-pegged-35b-2015 , says

SAN JOSE, CA – The global hospital information systems market will climb past $35 billion by 2015, according to a new forecast by Global Industry Analysts. The United States represents the largest market in the world. The U.S. hospital information system market is experiencing an increase in acceptance of customized technology such as laboratory information systems and radiology information systems, the report notes. The market is also a promising ground for electronic medical record systems.

The Asia-Pacific region (excluding Japan) represents the fastest growing hospital information systems market, exhibiting a compounded annual growth rate of 11.5 percent over the next few years, according to analysts. Despite being a smaller market in terms of revenue, the Asia-Pacific promises excellent growth opportunities for hospital information systems, they said.

The global vendors profiled in the report include McKesson , Cerner , Allscripts-Misys Healthcare Solutions, Eclipsys, Computer Programs and Systems, Siemens Medical Solutions USA, QuadraMed, Medical Information Technology, Healthland, GE Healthcare, iSOFT Group, Agfa-Gevaert, Brunie-Software, IBA Health and Integrated Medical Systems.

The full release is here: Global Hospital Information Systems Market to Cross $35 Billion by 2015, According to New Report by Global Industry Analysts, Inc. Increasing awareness among medical service patrons on the benefits of using Information Technology in the healthcare sector, coupled with growing demand for affordable-yet quality healthcare services is forcing hospitals and other medical centers to adopt IT in their daily operations. Subsequently, Healthcare IT systems such as the Hospital Information Systems witnessed a great demand in the healthcare services sector. Adoption of HIS in hospitals is increasingly being encouraged and promoted by the Governments world over. http://www.prweb.com/releases/2009/02/prweb2021984.htm

GAE + GWT Starter Project. 8) Reflections on the First Iteration

2009-09-09T06:23:00.015+10:00

I started posting about this project on 17 August and made my first code commit on 26 August. In my previous post I announced the existence the of the Proof of Concept (PoC) version of the application.

The PoC version will mark the end of the first iteration of the project. This seems like an opportune time to reflect on the work that has been done so far.

The original goals of the project were

Test Datastore access rates.
Test client side responsiveness while fetching a lot of data from the server.
Evaluate the GWT and GAE development environments for doing substantial work quickly and with decent quality.
Evaluate the robustness of the GAE model of doing work in 30 second chunks.
Evaluate datastore scalability.
Create a few re-usable components.
Learn how to create a simple UI. My design skills are limited and I like the sparse Google web page lay-outs so I hope there some widgets and examples for building sparse Googly web pages.

Progress against these goals about one month later has been:

The application tested the Google datastore access rates. The datastore access rates seemed pretty reasonable, typically a few hundredths of a second to fetch a 100 KB data record. In most caches this was helped by the memcached cache sitting in front of the datastore (code is here). As shown in the example in the previous post, the datastore access time was much shorter than the few tenths of a second it takes to fetch the data back to the client.
Please test the client side responsiveness and tell me how well you think the application responds while fetching approximately 1MB data from the server per screen. My experience was that it did a pretty good job on Chrome, was okay on Firefox and Opera, slower on Safari and too slow on IE8. All these browsers were tested on Vista from a home DSL connection in Australia.
I will write a separate post on the GWT and GAE development environments. For now I can say that it was easy to get started with a GWT+GAE+Eclipse+github+cygwin environment from the laptop running Vista that was available to me when I started the project.
GAE could do all the datastore accesses that were required of my application in much less that 30 seconds. I got client response problems on IE8 when fetching 10 records at a time which took GAE about 0.1 second to process. GAE seems well suited to GUI+database applications.
As far as I could tell the datastore has scaled perfectly so far. According to the GAE dashboard, the application is currently using 200 MB of storage, which is a minimum of 2,000 records.
The server cache pipeline is probably re-usable and the client side cache can probably be adapted fairly easily. Apart from that I did not create much re-usable code
I learned how to create a simple UI with GWT. My design skills are still limited but I liked the sparse Googly list and navigation buttons.

The first iteration of the application design was made on 18 August, and the first iteration of the code on 9 September ended up with this design. That was mostly due to keeping the design simple and general until the first iteration was completed. That was helped by this being a toy project where I could define the goals. Fingers crossed for the next iteration.

The code ended up as follows at the end of the first iteration

The datastore access was straightforward. Most of the work was reading the documentation.
The server cache pipeline was simple and straightforward.
The web server was straightforward and simple, just an RPC server, except for the best-effort aspects.
The client cache and UI was where I spent most of my time and learn the most.

It took me a while to get to simple code for the client cache and UI that kept a clean coherent state. The final client cache design was

Block while waiting for visible records that have been requested from server but have not arrived to arrive. This takes care of the case of incomplete but correctly predicted pre-fetches.
Clear all pending server fetches. This is seldom costly because of step 1.
Build a queue of server requests. 0-click away (visible now), 1-click away, 2-clicks, away
Send requests to the server in the above order. Limit the number of pending requests to a specified number (default=2).
When visible records have been fetched, callback to UI. This usually results in an immediate callback (and no UI latency) because the records have been pre-fetched.

Remaining Work

There is a lot left to do for the next iteration. This is being tracked in the TODO list. There are

Optimisations. (Fetching a 100KB record to display about 200 characters is wasteful)
Bugs to be fixed.
Features to be added.

Please give me your feedback on what you think should come next by commenting below.

You can watch this project's progress here.

GAE + GWT Starter Project. 7) Trying out the program

2009-08-29T06:31:00.100+10:00

The program is here. The current version will be saved here and a debug version of the current version will be saved here. he top left of the UI navigate the connections of the main anchor person (the top person on the frame). <<, <, > and >> buttons are used to scroll through the list of connections. Selecting a person with a mouse brings up their details in the bottom frame. Clicking a selected connection person makes that person the anchor person.

Each connection entry has some data that gives some indication of the work the browser, network and server and doing to display each screen. e.g.

Peter Williams - 0, CacheDB, .., 94kb, 0.012 sec, 0.6 sec, level 0

Peter Williams: Person's name
0: Index into connections list
CacheDB: Server cache this person was fetched from. 'CacheDB'=GAE datastore, 'CacheCache'=memcache, anything else is a fetch from an external system.
94 kb: Amount of data fetched for this person in KB
0.012 sec: Time it took to fetch data from cache on server.
0.6 sec: Time it took for data to get back to client.
level 0: The client cache the data was fetched into. Levels 0 and 1 are visible, level 2 is the 1-click-away cache and level 3 is the 2-click-away cache.

Each page of results has about 1 MB of data fetched from the server.

Please try out the program and see if the performance is adequate.

You can watch this project's progress here.

GAE + GWT Starter Project. 6) Building up the Server

2009-08-27T06:22:00.012+10:00

The server side of this design is a cache pipeline wrapped by a GWT RPC service. Today I attempted to implement this.

First I built up the cache pipeline. This was a simple design with each cache stage basically the code below.

public V get(K key) {
V value = cacheActual.get(key);
if (value == null) {
if (nextStage != null) {
value = nextStage.get(key);
if (value != null) {
cacheActual.put(key, value);
}
}
}
return value;
}
public void put(K key, V value) {
V val = cacheActual.get(key);
if (!(val != null && val.equals(value))) {
cacheActual.put(key, value);
if (nextStage != null) {
nextStage.put(key);
}
}
}

Building this up took 5 Java classes

CacheStage.java (the code above)
CachePipeline.java (hooks together CacheStage's in a pipeline)
CacheActual.java (interface for actual caches)
CacheCache.java (memcache implementation)
CacheDB.java (datastore + web-service implementation with datastore intially stubbed)

There were no surprises in this.

Next I wrapped this a GWT style RPC server. This took some classes

PersonService.java (interface for class containing the RPC call)
PersonServiceAsync.java (async version of PersonService.java)
PersonServiceImpl.java (server side implementation of PersonService.java)

Defining these classes required defining the person class that is seen by clients.

PersonClient.java (the client-side person class)
PersonFetch.java (person + tracking data to cross-reference to requests, required for best-effort operation)
PersonClientGroup.java (a class to wrap all the data returned by the RPC call)
PersonTrait.java (since the server and client person classes differ, we need to specify the canonical person traits such as name, address, uniqueID, and have all person classes implement this interface)

That, plus a few helper classes, got the first iteration of the server implemented. There were 21 .java files which seemed like a lot for the amount of functionality created, but most of them were simple. Code is here.

You can watch this project's progress here.

GAE + GWT Starter Project. 5) First check-in

2009-08-26T07:36:00.005+10:00

The initial code is on github.

The Person record and cache pipeline are coming together.. Everything looks okay for now.

Please take a look at the code.

I might try these ideas for the next iteration.

You can watch this project's progress here.

GAE + GWT Starter Project. 4) Just in Time Person Synthesiser

2009-08-24T06:48:00.018+10:00

It is just in time to see the first version of this application here, It loads slowly, especially if you have a slow internet connection or are using Internet Explorer or both, so if you are in this situation then please be patient to start with. Once loaded, the application is responsive.

However this post is about a different "just in time", the just-in-time person record synthesiser.

I have added a person record synthesiser to the end of the cache pipeline so that whenever a non-existent person is referenced, that person is synthesised. This is straight-forward to implement for the dummy persons I am using.

I chose this method instead of pre-populating the datastore with dummy persons because I have not learnt how to use GAE's batch mode yet.

You can watch this project's progress here.

GAE + GWT Starter Project. 3) Getting Started

2009-08-23T11:24:00.014+10:00

After a few days doing other things, I am back working on my GAE + GWT Starter Project. The last thing I did before I took a break was the high-level design. It looked like a layered design with these layers

UI
Person Client Cache
(Client-Server RPC)
Person Server Cache Pipeline
Database

When it comes to implementing this design, the two key observations are

Each layer depends on the layer below it, and
The biggest risks are in the bottom layers.

Therefore there are two reasonable ways to go about the implementation:

Bottom-up
Stubbing of the lower levels

Bottom-up reduces the uncertainties in the project fastest and allows each layer to built on working foundations. Stubbing allows the layers to be built and tested independently.

If there was more than one person working on this project then we would start by building the server cache pipeline (a simple piece of code) and adding stubs for each cache stage. This would allow people to work on different parts of the system independently of each other (and give us the stub framework we would eventually need for testing and tuning).

However I am the only person working on this project and the risk is concentrated in the bottom layers, so I am going to attack uncertainty and start at the bottom.

Build a person canonical meta-data based on using a few social networks.
Get the GAE datastore saving and find person records reliably.
Get the GAE server working reliably to deliver results in series of calls that are timed out in 10 seconds (well below sandbox limit and within human tolerance)

That should keep me busy for a while.

GAE + GWT Starter Project. 2) Product Definition and High Level Design

2009-08-18T06:11:00.024+10:00

In my previous post I outlined a plan to write a GAE+GWT application. (See goals section in the previous post). In this post I will specify the product and give a high level design.

Product Definition
The application will be a Social Network Navigator. It will

navigate a person's list of social network connections
show each connection's information (extracted from or linked to some social networking sites)
allow the connection to become the person so their connections can be navigated

High Level Design - User Interface
UI Components

Main person (one line of summary info: name, address, description).
List of person's connections (one line of summary info per person).
Panel with details of selected person or connection.
Navigation buttons (next, previous, first, last screen of connections)

UI Actions

Selecting a person or connection shows their details in the Details panel.
Clicking on a selected connection makes the connection the main person.
Navigation buttons move between screens of connections

High Level Design - Client Cache
The server will be fetching data from somewhere, either a data store or from some other website or web service. This is going to be slow compared to desired UI response time so we will need to keep a client side cache with the data people will be likely to be requesting. The client cache needs to fetch data from the server before the user requests viewing them. One way to arrange the cache is in terms of the number of mouse clicks a person is from being visible on the main screen.

Visible
1 click from being visible
2 clicks from being visible
...

If there are N people per screen then

N visible persons
<= 4*N 1-click persons (next, previous, first, last)
<= 4*N 2-click persons (next-next, previous-previous, first-next, last-previous

Choosing a new main person will change the few-clicks-away persons a lot. These persons can be predicted but changing the main person is a big change so we will leave this prediction for a future iteration of the design. The cache can be navigated forwards and backwards. Backwards navigation can be cached effectively with an LRU cache. Forwards navigation cached by enumerating the persons that can be accessed by 1 or more mouse clicks. Therefore a sufficient cache hint is a set of lists of persons that can be reached in m mouse clicks for m=0..M where M is a parameter that is to specified and optimised.

High Level Design - UI and Client Cache Interaction
The previous section outlined a cache that depended on the UI state. This leads to a simple set of interactions between the UI and cache.

User sets new UI state
UI sends the m-click hints to client cache and disables user controls (effectively freezing the state of the UI)
Client cache signals UI when visible persons are in cache.
UI enables user controls.

This is simple interaction which is good for a small project like this. Client Cache Effectiveness and Tuning
The effectivenees of the cache will depend on how often it fetched the m-click-away persons from the server before the user makes those clicks. There are some obvious trade-offs.

Fewer navigation controls in the UI makes caching easier but leads to a less flexibiltiy
Bigger caches are faster but use up client side memory and may waste server fetches.

High Level Design - Client Cache and Server Interactions
This is simple in principle. Fetch data from the server. Some things will need to be tuned

number of people fetched per server interaction
timing of feches for different client cache levels (all at once or m-click before m+1-click)

High Level Design - Server
Each person will have some possibly data and some searchable meta-data.

Person = summary (name, address, ...) + detail + list of connections (other persons)

Since we want to access this rapidly for each person we will need to store it on our server in our format. Most likely we will have a multi-stage cache.

Fetch data from datastore (and store it memcache)
Fetch data from memcache.

We will need to run this a little to get some idea of the relative speeds of 1 and 2 to see how we will tune this.

High Level Design - Person Data Structure
The person data structure is mostly straightforward, a big chunk of data with some meta-data. The obvious wrinkle is the relationships between persons. Each person has connections to other persons. If there needs to be any work on connections, such as sorting the connections by the connections' meta-data then we have to decide how much meta-data to store with each connection, or see how the datastore can assist in one to many relationships.

You can watch this project's progress here.

GAE + GWT Starter Project. 1) Introduction

2009-08-17T05:58:00.006+10:00

Google App Engine (GAE) is Google's web application development environment. It makes it easy to create web sites that run on Google's infrastructure. It runs application code in sandboxes, 30 seconds of single-threaded execution with no file access or spawning of other processes. And it provides access to some Google's highly scalable resources, most notably a BigTable based datastore.

Google Web Toolkit (GWT) is Google's JavaScript development environment. It compiles Java into JavaScript, supports Ajax style programming and optimises the JavaScript for the main browsers.

There is a well-thought out RPC communication protocol between GWT and GAE and a GAE hosted mode that supports single-session debugging of GAE and GWT apps in the Eclipse IDE.

The benefits of developing under GAE and GWT appear to be

Massive server and database scalability
Learning the Google server session way: Serving up a reponse in under 30 seconds in a single threaded servlet. If I can do this then Google can make my web applications massively scalable and robust by simply killing and discarding and server response that take more than 30 seconds.
A good AJAX-ey user experience on web clients. GWT (like all AJAX clients) calls the server in the background so that it has server data available locally to update the screen when the user requests it.
Learning and evaluating the GAE and GWT development tools. Hosted mode, Eclipse development environment, server logs, etc
Learning about Google frameworks, tools and applications.

I have been playing with GAE and GWT in my spare time over the last few months, mostly building and tweaking the examples. To find out how GAE+GWT are to use for real development, I am giving myself a simple but non-trivial project to see how long it takes to develop a robust, real world product.

My goals are

Test DataStore access rates.
Test client side responsiveness while fetching a lot of data from the server.
Evaluate the GWT and GAE development environments for doing substantial work quickly and with decent quality.
Evaluate the robustness of the GAE model of doing work in 30 second chunks.
Evaluate data store scalability.
Create a few re-usable components.
Learn how to create a simple UI. My design skills are limited and I like the sparse Google web page lay-outs so I hope there some widgets and examples for building sparse Googly web pages.

That describes my motivations for toying with GAE and GWT. In my next post I will specify my GAE+GWT app. Feel welcome to send me suggestions before then.

You can watch this project progress here.

Open Goverment Made Simple

2009-08-16T10:56:00.005+10:00

There has been a lot of talk about Open Government recently, including this from Peter Williams

"Australian governments should adopt international standards of open publishing as far as possible. Material released for public information by Australian governments should be released under a creative commons licence." or in simple terms "make public data open and free".

That is useful, clear and straightforward.

So how to do it?

My experience in organising company data over the last 10 years is that the teams I have worked in have tried many content management systems (CMSs) and none of them were satisfying to use (though some were interesting to implement.) Inevitably the document taxonomies that made sense to the site administrators did not work for most of the users and the users soon gave up trying to find things through the CMS.

Then one day someone in the company I was working at purchased a Google Search Appliannce (GSA) and indexed most of our intranet with it. After that everybody could find all the documents they knew existed on the intranet and discovered useful ones they did not know existed.

To be fair, things were not quite that simple. Most companies need reliable storage, decent version tracking, access control and many other things that CMSs provide. However people need to be able to find documents much more than they need these other things. Very few people need version tracked, access controlled documents that they cannot find in the first place.

So why don't goverments just make their data visible to internet search engines and store it somewhere secure with some simple versioning system now, and then do the fancy stuff later? Why are they are investing in CMSs like Sharepoint?

The reason we did not do this in the companies I worked in was that many of the features in the CMSs we used were useful and the people who implemented the systems decided they needed all these features. Finding documents was just one of several check-boxes on their requirements documents. They were acting as implementers and experts, not users. The systems they ended up with made perfect sense to everyone except the users.

The interesting thing about this was the implementers were users in most cases. They were aware of the limitations of CMSs but they had to follow either the direction of their users who had had not used CMSs enough and to understand how badly they would work in practice or the direction of their managers and key stakeholders who had heard that CMSs were good. The person who got the GSA was an IT guy who just went out and tried it without surveying users or bringing in CMS vendors to talk to his key stakeholders.

For a different perspective on Open Government, read some Tim O'Reilly.

When Words Fail

2009-07-16T10:37:00.006+10:00

A while back I worked at a company who made software+hardware products in a maturing market. The company found it needed to deliver higher quality products with more features and was struggling to do so from an old codebase. It had become clear to the management team that late-stage serious defects were the major cause of schedule/quality issues but they had been able to fix this problem.

The codebase management team had a lot of ideas about what the causes were and how to fix them. They had discussed "technical debt", "silo-ing" and other causes. However in the end they settled on two key priorities: taking extreme care with code changes and sticking with established QA processes to minimise the number of introduced bugs.

Eventually the project was given to me to manage. One of the (many) things the development team had done well was to document each bug and cross-reference bug fixes against the source code. I analysed about 100 recently fixed serious software bugs, looked up their fixes in the SCM and then looked up the date at which the code changes causing the bug were checked in. This showed that most of the bugs being found had been introduced months before they were discovered. It was clear that the late-stage defects were dominated by latent bugs being unmasked by changes, not by bugs introduced by changes.

Some changes to the development process were needed. The development group was responsible for creating code without introducing bugs and the QA group was responsible for finding the bugs the development team missed. However the QA process was unsuited to discovering latent bugs fast because it had a long cycle based on testing user scenarios. Therefore I got small teams of developers and QAs to work closely together to find, fix and verify bugs and I took some developers away other work to develop a system to find and fix (and eventually prevent the introduction of more) latent bugs. This work is described here. With these changes in place, code stability improved rapidly and late-stage serious bugs essentially ceased to be found.

That was a fairly straightforward technical solution to a fairly straightforward technical problem. So why had the very capable management team who had known the underlying causes (technical debt and silo-ing) not been able to fix the problem for so long?

Change is known to be difficult in organisations and there is an industry built around dealing with this. However our immediate problem was not an inability to persuade people to change. In fact consultation and review had been distracting people from doing the experimentation required to find the underlying causes of the problem was and how to fix them. The more people talked about the problem the further they got from the solution (hence this post's title).

The situation reminded me of Uncle Bob Martin's Agile Smagile

As I said before, going meta is a good thing. However, going meta requires experimental evidence. Unfortunately the industry has latched on to the word "Agile" and has begun to use it as a prefix that means "good". This is very unfortunate, and discerning software professionals should be very wary of any new concept that bears the "agile" prefix. The concept has been taken meta, but there is no experimental evidence that demonstrates that "agile", by itself, is good.

The deeply ingrained practices in the organisation I worked in had grown out of ideas that had worked well in the past. They had been good enough to cover a wide range of development scenarios for a long while and were clearly based on experimental evidence from past development. However somewhere along the way people had stopped experimenting and modifying the rules, and started just following the rules. This is what Uncle Bob called "going meta". The problem for our organisation was that the set of rules it had got to when it stopped experimenting were not universally true, they were only true for the type of the development they were doing when they stopped changing the rules.

The changes I made to detect and fix latent bugs (high-coverage automated system testing, static analysis with Klocwork and refactoring with unit tests) were adopted across the development organisation and became part of the standard development process, at least for the time I was there. That was good but I wondered if those practices would become a fixed part of the new development process because they had worked some time in the past. And I wondered whether they would prevent the company from addressing problems that arose in the future, just as the practises that had worked well in the past had come to do.

Minimal Non-C++Programmer Bamboozling C++ Question

2009-06-22T13:03:00.002+10:00

I recently read a stream of blog posts about why developers don't like C++ for general purpose programming. This post typifies much of the criticism of C++'s complexity. It includes an interview question about creating a C++ class that behaves like a class in a high level language such as Java. The author says that he uses this to weed out job applicants who haven't used C++ for real work.

It strikes me that tripping up developers with C++'s many oddnesses is much easier than that. Here is a simple question that I believe will confuse many non-C++ programmers:

What is the output of this program?

#include <string>
#include <iostream>
using namespace std;
class Parent {
public:
string func() { return "parent"; }
virtual string vfunc() { return "parent+virtual"; }
};
class Child : public Parent {
string func() { return "child"; }
virtual string vfunc() { return "child+virtual"; }
};
string test1(Parent parent) {
return parent.func() + " - " + parent.vfunc();
}
string test2(Parent& parent) {
return parent.func() + " - " + parent.vfunc();
}
int main() {
Child child;
cout << "test1: " << test1(child) << endl;
cout << "test2: " << test2(child) << endl;
}

I have seen C++ interviewers ask questions like this but only show test1 then ask what is on the stack when test1 is invoked.

It's Time for GWTing

2009-06-17T09:47:00.003+10:00

I am sitting up late at night working on my GWT project. The interesting learning is finished and the satisfaction of a working product still to come. It's time to get in the groove of grinding out good code quickly and cleanly.

I need some background music. It's time to get out my record collection and cassette tape recorder and make a programming tape. Was that a flashback to the 70s? Twitter app developers are revisiting the 70s, but I can't even remember the 70s and there are no rumours of the GWT team doing a Commodore 64 port, so I will stick with the modern way of doing a programming tape.

Google Flu Trends for Australia

2009-06-04T08:17:00.001+10:00

This is from the Google Down Under blog this morning.

How does this work? (from the same website)

We have found a close relationship between how many people search for flu-related topics and how many people actually have flu symptoms. Of course, not every person who searches for "flu" is actually sick, but a pattern emerges when all the flu-related search queries are added together. We compared our query counts with traditional flu surveillance systems and found that many search queries tend to be popular exactly when flu season is happening. By counting how often we see these search queries, we can estimate how much flu is circulating in different states and countries around the world. Our results have been published in the journal Nature.

Movement to the Cloud

2009-06-02T10:14:00.005+10:00

A current major trend in enterprise IT is to re-centralize computing and have smaller clients. Total cost of ownership (TCO) decreases dramatically with centralization because maintaining and upgrading software running in one physical location is easier than doing it on many people's personal computers.

Most client software, even computationally intensive software like high quality graphics, has very low duty cycles. It does nothing most of the time. When you buy expensive PC hardware to support it, you are paying to support peak usage, the few minutes per day when it does the intense computations and you want the user interface to be responsive. The average computer resource usage of this software is very low, much less than 10%. Therefore running the software on a central server is much more efficient. There is the additional benefit that expensive infrastructure such as high-reliability disk storage does not need to be replicated through an organization.

There are many types of cloud services including virtualization, SaaS and web applications . Most of these have become highly effective in the last 10 years. Their development has been driven by companies such as Google, VMware and Salesforce.com who have based their businesses entirely on these types of services and have not pursued desktop computing strategies.

3G netbooks and very capable smart phone make adequate clients for Web Services, SaaS, Virtualization and other cloud offerings. They have the additional advantages of low power consumption that comes from their phone ancestry and mobility.

Here are some stories to support this

Move from Windows Laptops to Netbooks in 4Q08 In 4Q08, worldwide PC processor unit shipments declined –17.0% quarter over quarter (QoQ) and –11.4% year over year (YoY); ... Intel's Atom processor for mini-notebook PCs (which Intel calls ''Netbooks'') continued to make a notable difference in the overall market performance but not enough to help the market avoid dramatic decline. Without Atom, worldwide PC processor unit shipments declined –21.7% QoQ and –21.6% YoY
Intel readying for rise of Android-based netbook Intel readying for rise of Android-based netbooks Intel is expecting a flurry of Android-based netbooks in late 2009 and throughout 2010, and is preparing to support manufacturers with suitable chipset hardware. That’s according to VentureBeat’s “reliable source”, who suggests that the chipmaker - responsible for the Intel Atom N270 processor found in the vast majority of netbooks on the market today - is betting that Google’s open-source mobile platform will find a niche among what are usually Ubuntu or Windows XP machines
RCR Wireless 2009 Wireless Forecast: Chips Meanwhile, Qualcomm has placed emphasis on its Snapdragon chipset for notebooks, netbooks and mobile Internet devices, with the latter category projected to surge as netbooks bridge the price gap between laptops and handhelds...Qualcomm will have competition, as Intel Corp. — the world’s largest chip maker — has targeted MIDs as PC sales rates slow, hoping to grab some of the market served by Apple Inc.’s iPhone. PC maker Dell Inc. is rumored to be preparing a smart phone play to open new markets. ...Meanwhile, most chip vendors are looking to core strengths and seeking bright spots in the year ahead. Smartphone sales are projected to grow this year — analysts vary on the strength of that growth — and vendors who make GPS, Wi-Fi and Bluetooth chips and 3G modems have a fighting chance, analysts have observed. ...“These are bright spots,” said Will Strauss, principal at Forward Concepts. “3G in general is still growing.”
Cloud computing ascends the mainstream (The Age 20 Feb 2009) illustrates how mainstream cloud computing has become. Juicy Orange pays 10 to 80 cents per hour for the virtual service, which means it has stopped forking out for physical hardware - it has become a web development company with none of its own computers. "We don't spend hours and days waiting for physical hardware installations any more. I'd be hard-pressed coming up with any real disadvantages. Once we got over the learning hump, it's been nothing but better," says Mr. Murphy

Cloud computing requires fast, reliable and portable network connections. The telecommunication carriers who can provide this will achieve a pricing advantage over their competitors

Some Consequences of Movement to the Cloud

Cannibalization of existing sales. What will netbooks do to the PC and laptop value chain? Currently netbooks are growing the PC pie but in the longer term US$250 netbooks may cannibalize US$1000 laptop sales at great cost to Intel and Microsoft. Netbooks are well-suited to cloud computing so netbook + cloud may replace laptop the Microsoft software. Google, salesforce.com, Amazon and other software already works well from the cloud while Microsoft Word does not so well-suited. The $250 vs. $1,000 price difference will drive change.
Possible convergence of smart phones and netbooks.
Processor wars. Intel Atom, Qualcomm Snapdragon, Freescale’s ARM based chipset.

Trying Out Wolfram Alpha

2009-05-19T10:13:00.003+10:00

Wolfram Alpha, the new computational knowledge engine from Wolfram Research has been getting a good run in the press this week so I decided to try it out.

It gave pretty good answers for most of the following queries. Try them out yourself.

Programming @Home

2009-05-08T19:51:00.026+10:00

	I am writing some software for the kids, mostly to give them a chance to participate in the work by drawing pictures, taking digital photos, writing text, choosing colors and, most importantly, testing. The first thing I need are some tools that will allow me to build a good website fast. I have chosen the following set. Ajax (i.e JavaScript which is available on all good browsers) for clients Eclipse + JDK Java Servlets and JSP for the server. Beanstalk for source code management. Chrome, IE, Firefox, Opera and Safari for testing.
I don't know if these are the best tools for simple web design but they are certainly good enough to get the work done. I have been sitting and knocking out some fairly interesting web pages at a decent rate. But the work is so mechanical that I am getting bored. I need some background music. The videos on this post helped me get through the day. The first application set is here. The kids' application is here .

	Peter Williams Consultant at MAP Research 90 William St Melbourne, VIC, 3000