tag:blogger.com,1999:blog-87205500378673451872024-03-13T23:57:40.570+11:00blogit ergo sumPeter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.comBlogger140125tag:blogger.com,1999:blog-8720550037867345187.post-43052484210127074552013-12-27T20:52:00.000+11:002015-07-12T15:39:12.828+10:00Digital Signal Processing MOOCI just finished <a href="https://www.coursera.org/instructor/~181">Paolo Prandoni</a>'s and <a href="http://en.wikipedia.org/wiki/Martin_Vetterli">Martin Vetterli</a>'s <a href="https://www.coursera.org/course/dsp">Digital </a><a href="https://www.coursera.org/course/dsp">Signal Processing</a> course on Coursera.<br />
<br />
I did not spend much time on this course unlike the MOOCs I had studied before. I have worked on various types of signal processing in my career and I as I progressed through this course I found most of the material was familiar to me.<br />
<br />
Digital Signal Processing is a diverse collection of subjects defined is <a href="http://en.wikipedia.org/wiki/Digital_signal_processing">Wikipedia</a> as <i> being the mathematical manipulation of an information signal to modify or improve it in some way</i>, including e the following laundry list: <i>audio and speech signal processing, sonar and radar signal processing, sensor array processing, spectral estimation, statistical signal processing, digital image processing, signal processing for communications, control of systems, biomedical signal processing, seismic data processing, <b>etc!</b> </i><br />
<br />
The course syllabus was<br />
<a href="https://www.coursera.org/course/dsp" style="clear: right; display: inline !important; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="320" src="http://1.bp.blogspot.com/-pPN-Pz3_uWo/Ur0IplD1dRI/AAAAAAAAMPI/eydWqdyYMr4/s1600/Coursera+dsp+2013.png" width="233" /></a><br />
<ol>
<li>Discrete time signals</li>
<li>Vectors spaces</li>
<li>Fourier Analysis</li>
<li>Linear Filters</li>
<li>Interpolation and Sampling </li>
<li>Stochastic Signal Processing and Quantization </li>
<li>Image Processing</li>
<li>Digital Communication Systems</li>
</ol>
The lectures were based on a very thorough <a href="http://www.sp4comm.org/">online book</a> that was so easy to read, I did not watch many of the lectures. The lectures seemed good for an introduction to the field.<br />
<br />
Assessment was by quiz. I would have preferred programming assignments<br />
<br />
My quiz results suggested that I understood all the material in the course expect design of <a href="http://en.wikipedia.org/wiki/Asymmetric_digital_subscriber_line">ADSL</a>.<br />
<br />Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com38tag:blogger.com,1999:blog-8720550037867345187.post-24866914106217869912013-08-26T20:58:00.001+10:002013-08-30T08:45:36.444+10:00Discrete Optimization Course FinishedI recently finished <a href="https://dl.dropboxusercontent.com/u/62188928/site/Welcome.html">Pascal Van Hentenryck</a>'s <a href="https://www.coursera.org/course/optimization">Discrete Optimization</a> online course.<br />
<br />
The course differed from other tertiary courses and MOOCs I had taken in the past in several ways.<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen='allowfullscreen' webkitallowfullscreen='webkitallowfullscreen' mozallowfullscreen='mozallowfullscreen' width='320' height='266' src='https://www.youtube.com/embed/2IbJf4oXOxU?feature=player_embedded' frameborder='0'></iframe></div>
<ul>
<li>It started with the video on the right instead of the dry introductions I was used to in math and computing courses. </li>
<li>The format of the course was to solve several instances of one <a href="http://en.wikipedia.org/wiki/NP-hard">NP-hard</a> problem per week.</li>
<li>Little guidance were given. Students were left to choose from the methods give in lectures.</li>
<li>There were no quizzes!</li>
</ul>
<div>
Discrete Optimization as the name suggests is about optimizing things that take on discrete (whole number) and usually positive values. That is much of the real world. e.g. items in a knapsack, cities on a route, vehicles in a delivery schedule, so it is widely applicable. e.g. The introductory video told us that Australia spends 10-20% of GDP on logistics, and logistics is made up of discrete things like numbers of packages and vehicles, so discrete optimization is useful.<br />
<h3>
The Course</h3>
The basic ideas of the course were to:<br />
<ul>
<li>Learn the main techniques of discrete optimization. </li>
<ul>
<li>Basics: e.g. randomization, greedy search, dynamic programming, branch and bound.</li>
<li><a href="http://en.wikipedia.org/wiki/Constraint_programming">Constraint programming</a></li>
<li><a href="http://en.wikipedia.org/wiki/Local_search_(optimization)">Local search</a></li>
<li><a href="http://en.wikipedia.org/wiki/Integer_programming">Mixed integer programming</a></li>
</ul>
<li>Solve a series of problems using any of the techniques learned in combination or alternative methods.</li>
<ul>
<li>Knapsack</li>
<li>Graph Coloring</li>
<li>Traveling Salesman</li>
<li>Warehouse Location</li>
<li>Vehicle Routing</li>
<li>Puzzle Challenge (optional)</li>
</ul>
</ul>
<div>
In the words of the<a href="https://www.coursera.org/course/optimizatio"> Course Syllabus</a></div>
<blockquote class="tr_bq">
<span style="font-size: x-small;"><span style="background-color: white; color: #5f5f5f; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; line-height: 21px;"><i>The course has an open format. At the start of the course all of the assignments and lectures are available and each student is free to design their own plan of study and proceed at their own pace. The assessments in the course consist of five programming assignments and one extra credit assignment. In the programming assignments, students experience the challenges of real world optimization problems such as selecting the most profitable locations of retail stores (warehouse location) and the design of package delivery routes (vehicle routing). </i> </span> </span></blockquote>
This sounded practical on one hand, after all I usually want to learn how to solve real world problems in computer courses. On the other hand it does not make it clear how to start. </div>
<br />
Lacking a plan, I watched all the lectures then started solving the problems. This was probably about the worst way to do the course. Later on I noticed there was as study guide that gave a good way of doing the course. Oh well.<br />
<br />
I watched about 40 lectures that contained some deep insights and lots of practical advice about discrete optimization.<br />
<ol>
<li>Introduction, branch and bound, dynamic programming.</li>
<li>Constraint programming</li>
<li>Local search</li>
<li>Linear programming and mixed integer programming.</li>
<li>Advanced topics, column generation, disjunctive global constraints, limited discrepancy search and <a href="http://en.wikipedia.org/wiki/Very_large-scale_neighborhood_search">large neighborhood search</a>.</li>
</ol>
<div>
Then I started on the problem sets.<br />
<br />
The problem sets were in increasing order of difficulty and each set contained instances that were in increasing size and therefore difficulty. The size was important as all the problems were NP-hard which means roughly that they have no known exact solutions that run faster than <span style="font-family: Courier New, Courier, monospace;">a^n</span> where <span style="font-family: Courier New, Courier, monospace;">a > 1.0</span> and <span style="font-family: Courier New, Courier, monospace;">n</span> is the size of the problem (e.g. number of cities in the travelling salesman problem).<br />
<h3>
Solving the Problems</h3>
My problem solving went like this.<br />
<h4>
Knapsack</h4>
Used branch and bound. Spent a lot of time getting tight bounds using a technique that was something like a genetic algorithm and developed a special data structure for it (a fixed size sorted deque). Spent way too much time on this.<br />
<h4>
Graph Coloring</h4>
Used local search. Spent time developing efficient moves and more time developing metaheuristics. Learnt why my original metaheuristic that I took so long to develop is not in wide use. Spent too much time on this.<br />
<h4>
Traveling Salesman</h4>
<div>
Used local search. Spent time developing a visualization and moves. Spent most of my time tuning well-known metaheuristics and experimenting with search strategies. Learnt why my original search strategies are not in wide use.</div>
<h4>
Warehouse Location</h4>
<div>
Used a MIP solution. Spent most of my time getting <a href="http://scip.zib.de/scip.shtml">scip</a> working and formulating the MIP problems.</div>
<h4>
Vehicle Routing</h4>
<div>
Had run out of time when I started this. Sorted vehicle trips by angle with warehouse, jiggled them back to feasibility and ran my TSP solution on each vehicle trip.</div>
<h4>
Puzzle Challenge </h4>
</div>
<div>
Had hours left when I started this so I used whatever method came in to my head first.<br />
<br />
e.g. For the N queens problem, I could remember MIP, local search and constraint programming solutions. Constraint propagation seemed the most natural way to do it so I wrote a simple <a href="http://bit.ly/n_queens">python script</a> for a constraint based solution. This script was too slow to get a high score (valid solution for high N) so I made a few improvements.<br />
<ol>
<a href="http://2.bp.blogspot.com/-vL-jAmKGoRU/Uhsoo5xwpvI/AAAAAAAAKpc/TzK3AvPMvIA/s1600/discrete_optimization_certificate.png" imageanchor="1" style="clear: right; display: inline !important; float: right; margin-bottom: 1em; margin-left: 1em; text-align: center;"><img border="0" height="320" src="http://2.bp.blogspot.com/-vL-jAmKGoRU/Uhsoo5xwpvI/AAAAAAAAKpc/TzK3AvPMvIA/s320/discrete_optimization_certificate.png" width="249" /></a>
<li>Randomized the order of searching rows for new queens as searching every column in increasing row order is very slow. I didn't have time to figure out what a good order was but random seemed unlikely to be bad, and it wasn't/</li>
<li>Coded the inner loop <span style="font-family: Courier New, Courier, monospace;">propagate()</span> in <a href="http://numba.pydata.org/">Numba </a>which is the quickest way to make python code run at C speeds.</li>
</ol>
With those improvements my script could solve for about 900 queens but no more because it was recursive and had reached a python call depth limit. I didn't have time to convert it from a recursive to iterative so I stopped there.<br />
<h3>
Conclusion</h3>
This seemed like a good way to learn. I went about it an inefficient way but I learned first hand why the best methods don't work like the ones I invented myself. I also got to see<br />
<ul>
<li>what was important for each method. e.g. tight bounds for branch and bound, efficient moves for local search</li>
<li>what scales to large N. e.g. not my TSP local search</li>
<li>how much time is involved learning the packages that can be used for discrete optimization, e.g. <a href="http://scip.zib.de/scip.shtml">scip</a></li>
</ul>
You can read other students' comments on the course <a href="http://coursetalk.org/coursera/discrete-optimization">here</a>. <br />
<ul>
</ul>
<br />
<br /></div>
<div>
<br /></div>
<br />Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com5tag:blogger.com,1999:blog-8720550037867345187.post-22953093022160518522012-08-12T14:44:00.002+10:002012-09-22T09:08:00.163+10:00Twitter BotI recently wrote a <a href="https://github.com/peterwilliams97/twitter_bot" target="">Twitter bot</a> to offer sympathy to people who have suffered paper cuts. You can see it <a href="http://twitter.com/OwwwPapercut" target="">here</a>.<br />
<br />
The <i>Twitter bot</i> link in the last paragraph has all the code. There was nothing particularly complex in it. The hardest part was classifying tweets as being from people who suffered paper cuts or not. This was done with a very simple <a href="https://github.com/peterwilliams97/twitter_bot/blob/master/BayesClassifier.py" target="_blank">naive Bayes</a> classifier.<br />
<br />
The code from the previous link was trained on 3,225 tweets to give the following 10-fold cross-validation results. The columns are the numbers of tweets containing "paper cut" that the classifier predicts are from people with paper cuts. The rows are the number of tweets that a human (me) says are from people with paper cuts.<br />
<pre style="background-color: #f8f8f8; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: 1px solid rgb(204, 204, 204); color: #333333; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 13px; line-height: 19px; margin-bottom: 15px; margin-top: 15px; overflow: auto; padding: 6px 10px;"><code style="background-color: transparent; border-bottom-left-radius: 3px; border-bottom-right-radius: 3px; border-top-left-radius: 3px; border-top-right-radius: 3px; border: none; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 12px; margin: 0px; padding: 0px;">===== = ===== = =====
| False | True
----- + ----- + -----
False | 1978 | 59
----- + ----- + -----
True | 308 | 880
===== = ===== = =====
Total = 3225
===== = ===== = =====
| False | True
----- + ----- + -----
False | 61% | 2%
----- + ----- + -----
True | 10% | 27%
===== = ===== = =====
Precision = 0.937, Recall = 0.741, F1 = 0.827
</code></pre>
This precision value of 0.937 means that 0.063 or about 1/16 of the tweets that the Twitter bot classifies as being about paper cuts and replies to are not about paper cuts..<br />
<br />
There is some additional <a href="https://github.com/peterwilliams97/twitter_bot/blob/master/filters.py" target="">filtering </a>and to reduce the number of inappropriate replies. Filtering increases the precision to <a href="https://github.com/peterwilliams97/twitter_bot/blob/master/results/cv.filtered.latest">0.95</a> so the final false positive rate is 1/20. The false positives after filtering are <a href="https://github.com/peterwilliams97/twitter_bot/blob/master/results/false.positives.filtered.latest" target="">here</a>. Examining them gives a feel for the cases where the simple Bayesian n-gram classifier breaks down.<br />
<br />
The most influential n-grams are given <a href="https://github.com/peterwilliams97/twitter_bot#the-classifier">here</a> where you can also find a link to the full list of n-grams.<br />
<br />
You can get some idea of the how well the classification works in the real world from the Twitter <a href="https://twitter.com/#!/search/realtime/%40owwwpapercut" target="">responses</a> to it.<br />
<br />
<b>UPDATE 17 September 2012</b>. Twitter warned the OwwPapercut account for sending automated replies which seemed odd given <a href="https://twitter.com/StealthMountain">@StealthMountain</a>. OwwwPapercut has run in non-replying mode since then.Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com1tag:blogger.com,1999:blog-8720550037867345187.post-86373166977462714462012-06-03T20:42:00.000+10:002012-06-03T21:31:01.365+10:00NLP Class CompletedThe Stanford NLP class finished about a week ago and I got my certificate of accomplishment over the week-end.<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-w6Z5ZKzh3yI/T8s_QkSDKtI/AAAAAAAAKQA/dXbixvrQsz8/s1600/attrition.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="300" src="http://3.bp.blogspot.com/-w6Z5ZKzh3yI/T8s_QkSDKtI/AAAAAAAAKQA/dXbixvrQsz8/s400/attrition.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">NLP Class "Attrition"</td></tr>
</tbody></table>
<br />
Professors <a data-bitly-type="bitly_hover_card" href="http://www.stanford.edu/people/jurafsky">Dan Jurafsky</a> and <a data-bitly-type="bitly_hover_card" href="http://nlp.stanford.edu/~manning/">Chris Manning</a> also sent out some course statistics that I saved <a href="http://bit.ly/N8xxDc">here</a> and graphed on the right using the Python code in the previous link.<br />
<br />
From about the third week of the course onwards I felt like I did not have time to complete watching the lectures and doing the problem sets and programming exercises. Therefore I was wondering what it was like for other people like me with jobs and families.<br />
<br />
The graph on the right shows the number of students completing problem sets and programming exercises each week of the course. I have called this "attrition" which cannot be completely right as the number of students completing programming exercises increased from week 6 to week 7.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-LXxrcDpbxYo/T8tFn3yB-uI/AAAAAAAAKQM/bYaZa63v4tI/s1600/nlp-class-cert.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="320" src="http://2.bp.blogspot.com/-LXxrcDpbxYo/T8tFn3yB-uI/AAAAAAAAKQM/bYaZa63v4tI/s320/nlp-class-cert.png" width="245" /></a></div>
I have not studied university course attrition rates but I suspect they would differ from NLP-class because of incentives like costs and grade penalties that would lead to students leaving at the start of the course. The somewhat smooth and mostly monotonic curves in the above graph seem more natural for a free class like NLP-class where the main benefits are learning. <br />
<br />
So why were people leaving the course all the time? I can only answer for me. In order to learn the skills I wanted, I felt I had to complete the assignments and the assignments were long and tended to get longer through the course. Programming assignment 6 took me the longest and cut into some family time. That seems to be reflected in the graph.Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com2Bruny Island, Tasmania 7150, Australia-43.3937074 147.2679116-43.5783244 146.9520546 -43.209090399999994 147.58376859999998tag:blogger.com,1999:blog-8720550037867345187.post-76831160221851334432012-03-25T20:29:00.001+11:002012-06-03T21:11:32.362+10:00Stanford NLP Class continuedIn one of the early classes one of the lecturers showed some Unix tools for performing low-level tasks such tokenizing and word counting. You can see these in the course notes.<br />
<br />
I prefer not to use such tools because I already use more tools than I want to and it is possible to do all the things taught in lectures with one of the tools I use a lot, Python.<br />
<br />
Here is some Python code that does what I remember the lecturer demonstrating.<br />
<br />
<b>Tokenizing on non-alphabetic boundaries</b><br />
<code>import re<br />
_RE_TOKEN = re.compile('[^a-zA-Z]+')<br />
def tokenize(text):<br />
return [x for x in _RE_TOKEN.split(text) if x]<br />
</code><br />
<b>Finding unique words</b><br />
The lecturers call unique words <i>word types</i>.<br />
<code> tokens = tokenize(text) <br />
types = set(tokens)<br />
</code><br />
<b>Finding unique words, ignoring case variants</b><br />
<code>normalized_types = set(tokenize(text.lower()))<br />
</code><br />
<b>Finding word types that differ only in case</b><br />
<code># variants[x] = set of all case variants of x in types<br />
variants = {} <br />
for w in types:<br />
variants[w.lower()] = variants.get(w.lower(), set([])) | set([w])<br />
<br />
# types that occur in multiple type variants<br />
multi_case = dict([(k,v) for (k,v) in variants.items() if len(v) > 1])<br />
<br />
# number of types that are case variants of other types<br />
num_case_only = sum([len(v) - 1 for v in multi_case.values()])<br />
<br />
# this had better be true<br />
assert(num_case_only == len(types) - len(normalized_types))<br />
</code><br />
<br />
That's it. Basic test manipulations done without having to leave one of the small numbers of programming environments I spend a lot of time in.Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com2Isla Grande de Tierra del Fuego-53.8671117 -69.297214-55.065887200000006 -71.8240695 -52.6683362 -66.7703585tag:blogger.com,1999:blog-8720550037867345187.post-80397062830792703752012-03-10T14:09:00.001+11:002012-06-03T21:11:50.039+10:00Stanford NLP ClassThese are my notes on Stanford's online <a href="http://nlp-class.org/">NLP class</a>. <br />
<br />
The first few lectures said that a lot of the hard work in NLP, notably in tokenizers, is done with regular expressions. This was not entirely surprising as a good fraction of the string processing I have done in my professional career has been done with regular expressions.<br />
<br />
Programming exercises can be done in Python or Java. I chose Python as I have found it well suited to simple string manipulation programs in the past.<br />
<br />
The first programming exercise is to extract phone numbers and email addresses from web pages. A training set of Stanford computer science faculty home pages was supplied along with some starter code to show the required formatting. The starter code helpfully computed lists of true positives, false positives and false negatives.<br />
<br />
My experience with problems like these is to<br />
<ol>
<li>get the test samples to pass, by </li>
<ol>
<li>loosening matches and adding more detection to detect all the addresses and phone numbers</li>
<li>tightening matches to avoid false positives</li>
</ol>
<li>while taking care to make decisions that are likely to generalize well to as yet unseen samples</li>
</ol>
<div>
2 takes some judgement as it is not clear what will generalize well. e.g In the samples " DOT " was used to mask "." in email addresses. It seems wises to match on all cases of " DOT " but then I found that " DOM " was used for the "." alias by one faculty <a href="http://www.stanford.edu/~engler/">member</a> . The question was then whether to generalize from " DOT " and " DOM " to " DO<any character> " or treat " DOM " as a one-off as it had been observed only once.</div>Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com2Tallinn, Estonia59.4426896 24.753197259.3135271 24.437340199999998 59.5718521 25.0690542tag:blogger.com,1999:blog-8720550037867345187.post-48086417445722192542011-01-15T10:09:00.004+11:002011-01-22T22:03:16.257+11:00My Web Browsing Turned into a Newspaper<a href="http://bit.ly/my_life_is_an_open_book">My life is an open book</a>.<br />
What the <a href="http://paper.li/peter_c_william/ml">machine learning people</a> I follow on Twitter are reading.<br />
And a <a href="http://paper.li/peter_c_william/1295693615">custom paper</a>.
<script src="http://widgets.paper.li/javascripts/init.js" type="text/javascript">
</script>
<script>
Paperli.PaperWidget.Show({
pid: 'peter_c_william',
width: 200,
background: '#FB0000'
})
</script>Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com0Bruny Island, Tasmania 7150, Australia-43.3937074 147.2679116-43.6431854 146.8009926 -43.144229399999993 147.73483059999998tag:blogger.com,1999:blog-8720550037867345187.post-58262526648710564572010-10-18T21:42:00.009+11:002012-11-11T21:17:28.142+11:00The Effect of Test Set Selection on Classification AccuracyI was looking at some prediction results for the UCI Michalski and Chilausky <a href="http://archive.ics.uci.edu/ml/machine-learning-databases/soybean/soybean-large.names">soybean data set</a> and wondered how they depended on test set selection. Some had classification accuracy as high as <a href="http://www.cs.umd.edu/~pfontana/CS_Web_Page_Files/Papers/ScholarlyPaperPF.pdf">93.1% accuracy on a 25% training set</a> and <a href="http://www.cs.sfu.ca/~wangk/ucidata/dataset/soybean/soybean-large.names">97.1% on 290 training and 340 test instances</a>.<br />
<br />
A few weeks ago I had been asked to find the best classifier for the soybean data set based on prediction accuracy on a test set of 20% of the data. The remaining 80% could be used for training. That gave <a href="http://www.wolframalpha.com/input/?i=306!+/+(245!+*+61!)">306!/(245!x61!)</a> = 1.3 x 10^65 possible splits of the 306 data points into training and test sets. Could some of these splits lead to better results than others for the classifiers I was about to use?<br />
<br />
The <a href="http://www.cs.waikato.ac.nz/ml/weka/">WEKA </a>data mining package was used for classification. WEKA has many classifiers that can be run on a data set and their performance to be compared.<br />
<br />
WEKA also has a programming interface so I used it to write some <a href="http://bit.ly/weka_tools">Jython tools</a>to explore the performance of a range of classifiers.<br />
<br />
<a href="http://bit.ly/split_data">One of these tools</a> was run on the soybean data to find the training/test splits with best and worst classification accuracy. The results were<br />
<br />
<table border="1"><tbody>
<tr> <th>Classifier</th> <th>Best Accuracy</th> <th>Worst Accuracy</th> </tr>
<tr> <td><a href="http://en.wikipedia.org/wiki/Naive_Bayes_classifier">Naive Bayes</a></td> <td>100%</td> <td>70.5%</td></tr>
<tr> <td><a href="http://en.wikipedia.org/wiki/Bayesian_network">Bayes Net</a></td> <td>100%</td> <td>75.4%</td></tr>
<tr> <td><a href="http://en.wikipedia.org/wiki/C4.5_algorithm">J48</a> (C4.5)</td> <td>95%</td> <td>69%</td></tr>
<tr> <td><a href="http://www.informatik.uni-osnabrueck.de/barbara/lectures/ml/sessions/Ripper.ppt">JRip</a> (RIPPER)</td> <td>98.4%</td> <td>70.5%</td></tr>
<tr> <td><a href="http://weka.sourceforge.net/doc/weka/classifiers/lazy/KStar.html">KStar</a></td> <td>96.7%</td> <td>65.6%</td></tr>
<tr> <td><a href="http://en.wikipedia.org/wiki/Random_forest">Random Forest</a></td> <td>95%</td> <td>62.3%</td></tr>
<tr> <td><a href="http://en.wikipedia.org/wiki/Sequential_Minimal_Optimization">SMO</a> (support vector machine)</td> <td>96.7%</td> <td>82%</td></tr>
<tr> <td><a href="http://en.wikipedia.org/wiki/Multilayer_perceptron">MLP</a> (neural network)</td> <td>100%</td> <td>77%</td></tr>
</tbody></table><span class="Apple-style-span" style="font-size: small;"><b>Fig1. Best and worst accuracies for selected WEKA classifiers run on different training/test splits</b></span><br />
<br />
That was quite a range of test set accuracies for different training/test splits. My simple genetic algorithm may not have found the extremes of the distributions so the actual range may have been higher. <br />
<br />
When I ran the <a href="http://bit.ly/weka_tools">test set selection script</a>a second time (Fig 2) it also found a 100% SMO accuracy. The second test was set up to find a single training/test set split that gave best results for all classifiers at once. It also had a slightly different pre-processing. The 4 duplicate instances were removed and the troublesome single 2-4-5-t sample was left in. Therefore I expected it to give worse results than the pre-processing used for the results in Fig 1.<br />
<br />
<table border="1"><thead>
<tr> <th>Classifier </th> <th>Correct (out of 60) </th> <th>Percent Correct</th> </tr>
</thead> <tbody>
<tr> <td>Naive Bayes </td> <td>57 </td> <td>95 %</td> </tr>
<tr> <td>Bayes Net </td> <td>59 </td> <td>98.3 %</td> </tr>
<tr> <td>J48 </td> <td>58 </td> <td>96.7 %</td> </tr>
<tr> <td>JRip </td> <td>60 </td> <td>100 %</td> </tr>
<tr> <td>KStar </td> <td>60 </td> <td>100 %</td> </tr>
<tr> <td>Random Forest </td> <td>59 </td> <td>98.3 %</td> </tr>
<tr> <td>SMO </td> <td>60 </td> <td>100 %</td> </tr>
<tr> <td>MLP </td> <td>60 </td> <td>100 %</td> </tr>
</tbody> </table><span class="Apple-style-span" style="font-size: small;"><b>Fig2. Best accuracies for selected WEKA classifiers all run on the same training/test split</b></span><br />
<br />
Both the above results were for the default settings of each of the WEKA classifiers. The WEKA classifiers all have parameters that can be tuned and it is possible to select subsets of attributes so they can give better and much worse results than the defaults. However the default parameters are usually close to the best so they may be good indicators of the best possible accuracies.<br />
<br />
It appears that the training/test split of a data set can change classification accuracy by more than 30%. This was observed on a well-known and widely used classification data set.<br />
<br />
Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com0Waikato 7073, New Zealand-40.621377 172.678619-40.686525 172.5618895 -40.556229 172.7953485tag:blogger.com,1999:blog-8720550037867345187.post-91445837980347268532010-09-07T22:21:00.000+10:002011-10-03T22:18:19.371+11:00Watching Percipo<object height="200" width="350"><param name="movie" value="http://www.youtube.com/v/iZZie4tFJaA?fs=1&hl=en_US&color1=0xe1600f&color2=0xfebd01">
</param>
<param name="allowFullScreen" value="true">
</param>
<param name="allowscriptaccess" value="always">
</param>
<embed src="http://www.youtube.com/v/iZZie4tFJaA?fs=1&hl=en_US&color1=0xe1600f&color2=0xfebd01" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="853" height="505"></embed></object>
<object height="200" width="350"><param name="movie" value="http://www.youtube.com/v/zPwTIBlD6d4?fs=1&hl=en_US&color1=0xe1600f&color2=0xfebd01">
</param>
<param name="allowFullScreen" value="true">
</param>
<param name="allowscriptaccess" value="always">
</param>
<embed src="http://www.youtube.com/v/zPwTIBlD6d4?fs=1&hl=en_US&color1=0xe1600f&color2=0xfebd01" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="853" height="505"></embed></object>Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com0San Francisco, CA, USA37.7749295 -122.419415537.707087 -122.536145 37.842772 -122.302686tag:blogger.com,1999:blog-8720550037867345187.post-52446734078641454362010-08-13T21:43:00.003+10:002010-10-18T23:22:12.232+11:00Wrote a Blog PostI have not posted here recently but I wrote <a href="http://www.papercut.com/blog/peter/2010/08/10/25-years-of-digital-printing/">blog post</a> for PaperCut last week.Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com0Mt Waverley VIC, Australia-37.876095 145.129008-37.90997 145.070643 -37.84222 145.187373tag:blogger.com,1999:blog-8720550037867345187.post-81291812848939982912010-03-20T09:50:00.005+11:002010-10-19T22:28:31.619+11:00Blogger supports logical symbols<span class="Apple-style-span" style="font-family: Verdana; font-size: small;"><span class="Apple-style-span" style="font-size: 13px;">e.g. </span></span><span class="Apple-style-span" style="font-family: Verdana; font-size: 13px;">¬(A ∨ B) ⇒ ¬A ∧ ¬B </span><br />
<span class="Apple-style-span" style="font-family: Verdana; font-size: 13px;"><br /></span><br />
<span class="Apple-style-span" style="font-family: Verdana; font-size: 13px;">All symbols: ¬, ∧, ∨, ⇒, ⇔</span>Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com0Nimbin NSW 2480, Australia-28.59677 153.222904-28.672130499999998 153.1061745 -28.5214095 153.3396335tag:blogger.com,1999:blog-8720550037867345187.post-37053980359750609692010-01-04T22:49:00.005+11:002010-01-07T14:35:16.671+11:00C++ Continues to SurpriseSomeone was asking questions about const_cast<>() a few days ago. I was not quite sure how it would work because I try to use as little of the C++ language as possible and it possible to get by in C++ without const_cast<>(). To find out exactly how it worked I tried it out with a test case. The following code gave the same output on g++ on Vista and OS X.<br />
<blockquote class="webkit-indent-blockquote" style="border-color: initial; border-color: initial; border-width: initial; border-width: initial; margin-left: 40px; margin-right: 0px;">
<br />
<div>
<div>
<span style="font-family: 'Courier New';"><span style="color: blue;">int i = 3;</span></span><br />
</div>
<div>
<span style="font-family: 'Courier New';"><span style="color: blue;">const int* ptr = &i;</span></span><br />
</div>
<div>
<span style="font-family: 'Courier New';"><span style="color: blue;">*const_cast<int*>(ptr) = 11;</span></span><br />
</div>
<div>
<span style="font-family: 'Courier New';"><span style="color: blue;">if (&i == ptr && i != *ptr) {</span></span><br />
</div>
<div>
<span style="font-family: 'Courier New';"><span style="color: blue;"> std::cout << "Cannot happen: &i=" << &i << " == ptr=" << ptr << " but i=" << i << " != *ptr=" << *ptr << std::endl;</span></span><br />
</div>
<br />
<div>
<span style="color: blue; font-family: 'Courier New';">}</span><br />
</div>
</div>
</blockquote>
The output in both cases was <span style="color: blue;"><span style="font-family: 'Courier New';">Cannot happen: &i=0x22fe6c == ptr=0x22fe6c but i=3 != *ptr=22</span></span><br />
<br />
How can a single memory address hold two different values?<br />
<br />
<div>
The disassembly was<br />
</div>
<div>
<br />
</div>
<div>
<div>
<blockquote class="webkit-indent-blockquote" style="border: none; margin: 0 0 0 40px;">
<span style="color: blue;"><span style="font-family: 'Courier New';">push %ebp</span></span><br />
<div>
<span style="color: blue;"><span style="font-family: 'Courier New';">mov %esp,%ebp</span></span><br />
</div>
<div>
<span style="color: blue;"><span style="font-family: 'Courier New';">sub $0x18,%esp</span></span><br />
</div>
<div>
<span style="color: blue;"><span style="font-family: 'Courier New';"> int i = 3;</span></span><br />
</div>
<div>
<span style="color: blue;"><span style="font-family: 'Courier New';">movl $0x3,0xfffffffc(%ebp) <b>(i in bp-4)</b></span></span><br />
</div>
<div>
<span style="color: blue;"><span style="font-family: 'Courier New';"> const int* ptr = &i;</span></span><br />
</div>
<div>
<span style="color: blue;"><span style="font-family: 'Courier New';">lea 0xfffffffc(%ebp),%eax <b>(&i in eax)</b></span></span><br />
</div>
<div>
<span style="color: blue;"><span style="font-family: 'Courier New';">mov %eax,0xfffffff8(%ebp) <b>(ptr in bp-8)</b></span></span><br />
</div>
<div>
<span style="color: blue;"><span style="font-family: 'Courier New';"> *const_cast<int*>(ptr) = 11;</span></span><br />
</div>
<div>
<span style="color: blue;"><span style="font-family: 'Courier New';">mov 0xfffffff8(%ebp),%eax <b>(ptr in eax)</b></span></span><br />
</div>
<div>
<span style="color: blue;"><span style="font-family: 'Courier New';">movl $0xb,(%eax) <b>(*ptr set to 11)</b></span></span><br />
</div>
<div>
<span style="color: blue;"><span style="font-family: 'Courier New';"> if (&i == ptr && i != *ptr)</span></span><br />
</div>
<div>
<span style="color: blue;"><span style="font-family: 'Courier New';">lea 0xfffffffc(%ebp),%eax </span></span><br />
</div>
<div>
<span style="color: blue;"><span style="font-family: 'Courier New';">cmp 0xfffffff8(%ebp),%eax</span></span><br />
</div>
<div>
<span style="color: blue;"><span style="font-family: 'Courier New';">jne 0x403214 </span></span><br />
</div>
<div>
<span style="color: blue;"><span style="font-family: 'Courier New';">mov 0xfffffff8(%ebp),%eax </span></span><br />
</div>
<div>
<span style="color: blue;"><span style="font-family: 'Courier New';">mov (%eax),%eax</span></span><br />
</div>
<div>
<span style="color: blue;"><span style="font-family: 'Courier New';">cmp 0xfffffffc(%ebp),%eax</span></span><br />
</div>
<div>
<span style="color: blue;"><span style="font-family: 'Courier New';">je </span></span> <span style="font-family: 'Courier New';"><span style="color: blue;">0x403214</span></span><br />
</div>
</blockquote>
</div>
</div>
<div>
<br />
The disassembly matches the C++ code. i is stored at bp-4 and ptr is stored at bp-8 so the C++ code should work. The observed behaviour does not match the disassembly.<br />
</div>
<br />
This cannot be right. I guess I found a bug in g++.<br />Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com49900 Kirkenes, Norway69.7206509 30.051355169.705776399999991 29.9929901 69.7355254 30.109720099999997tag:blogger.com,1999:blog-8720550037867345187.post-68221315046826880782009-10-27T06:26:00.000+11:002009-10-29T14:07:04.068+11:00My Time in SwedenI lived in Sweden from 1988 to 1991. Here is a map which showed where I lived in <a href="http://en.wikipedia.org/wiki/Stortorget">Stortorget</a> in Gamla Stan in Stockholm.<br />
<br />
<iframe frameborder="0" height="350" marginheight="0" marginwidth="0" scrolling="no" src="http://maps.google.com.au/maps?f=q&source=s_q&hl=en&geocode=&q=stortorget,+gamla+stan&sll=59.328242,18.089161&sspn=0.022943,0.084629&gl=au&ie=UTF8&t=h&ll=59.327191,18.076458&spn=0.015325,0.036478&z=14&output=embed" width="425"></iframe><br />
<small><a href="http://maps.google.com.au/maps?f=q&source=embed&hl=en&geocode=&q=stortorget,+gamla+stan&sll=59.328242,18.089161&sspn=0.022943,0.084629&gl=au&ie=UTF8&t=h&ll=59.327191,18.076458&spn=0.015325,0.036478&z=14" style="color: blue; text-align: left;">View Larger Map</a></small><br />
<br />
<br />
I lived in the red building in the two left photos below which are taken from Stortorget. The photo on the right is of the same building taken from Kåkbrinken, the alley to the left the red building.<br />
<small><a href="http://maps.google.com.au/maps?f=q&source=embed&hl=en&geocode=&q=stortorget,+gamla+stan&sll=59.328242,18.089161&sspn=0.022943,0.084629&gl=au&ie=UTF8&t=h&ll=59.327191,18.076458&spn=0.015325,0.036478&z=14" style="color: blue; text-align: left;"></a></small>
<a href="http://www.flickr.com/photos/miltoncorrea/2051258841/" title="photo sharing"><img alt="" src="http://farm3.static.flickr.com/2011/2051258841_9ad86caf59_m.jpg" style="border: solid 2px #000000;" /></a> <a href="http://www.flickr.com/photos/roberto1956/3110481831/" title="photo sharing"><img alt="" src="http://farm4.static.flickr.com/3021/3110481831_31a07d8f70_m.jpg" style="border-bottom-color: rgb(0, 0, 0); border-bottom-style: solid; border-bottom-width: 2px; border-left-color: rgb(0, 0, 0); border-left-style: solid; border-left-width: 2px; border-right-color: rgb(0, 0, 0); border-right-style: solid; border-right-width: 2px; border-top-color: rgb(0, 0, 0); border-top-style: solid; border-top-width: 2px;" /></a> <a href="http://www.flickr.com/photos/lenora/17081398/" title="photo sharing"><img alt="" src="http://farm1.static.flickr.com/13/17081398_ae4c2f1e54_m.jpg" style="border: solid 2px #000000;" /></a><br />
<br />
The photo on the left below is the main street in Gamla Stan and the photo on the centr is of the Grand Hotel as seen from the shore of Gamla Stan and the photo on the right is Karloniska Hospital where I worked.<br />
<a href="http://www.flickr.com/photos/calvaryzone/118483879/" title="photo sharing"><img alt="" src="http://farm1.static.flickr.com/43/118483879_ce720ed523_m.jpg" style="border: solid 2px #000000;" /></a> <a href="http://www.flickr.com/photos/66538587@N00/82490855/" title="photo sharing"><img alt="" src="http://farm1.static.flickr.com/42/82490855_a3a9fc712a_m.jpg" style="border-bottom-color: rgb(0, 0, 0); border-bottom-style: solid; border-bottom-width: 2px; border-left-color: rgb(0, 0, 0); border-left-style: solid; border-left-width: 2px; border-right-color: rgb(0, 0, 0); border-right-style: solid; border-right-width: 2px; border-top-color: rgb(0, 0, 0); border-top-style: solid; border-top-width: 2px;" /></a> <a href="http://www.flickr.com/photos/toveo/2827805486/" title="photo sharing"><img alt="" src="http://farm4.static.flickr.com/3086/2827805486_cc58851232_m.jpg" style="border-bottom-color: rgb(0, 0, 0); border-bottom-style: solid; border-bottom-width: 2px; border-left-color: rgb(0, 0, 0); border-left-style: solid; border-left-width: 2px; border-right-color: rgb(0, 0, 0); border-right-style: solid; border-right-width: 2px; border-top-color: rgb(0, 0, 0); border-top-style: solid; border-top-width: 2px;" /></a><br />
<br />
After I left Stockholm I moved to Umeå which is shown on the left below. When I lived there I used to visit Vaasa in Finland shown on the right.<br />
<a href="http://www.flickr.com/photos/arendpictures/928126782/" title="photo sharing"><img alt="" src="http://farm2.static.flickr.com/1071/928126782_12281d602e_m.jpg" style="border: solid 2px #000000;" /></a> <a href="http://www.flickr.com/photos/foide/3087146334/" title="photo sharing"><img alt="" src="http://farm4.static.flickr.com/3198/3087146334_a9d4edb4ef_m.jpg" style="border-bottom-color: rgb(0, 0, 0); border-bottom-style: solid; border-bottom-width: 2px; border-left-color: rgb(0, 0, 0); border-left-style: solid; border-left-width: 2px; border-right-color: rgb(0, 0, 0); border-right-style: solid; border-right-width: 2px; border-top-color: rgb(0, 0, 0); border-top-style: solid; border-top-width: 2px;" /></a><br />
<br />
When I lived in Sweden I took vacations in Norway including Lofoten on the left and Tromsø in the centre of the row of photos below. I also took the Hurtigruten<br />
<a href="http://www.flickr.com/photos/16173303@N04/2750315635/" title="photo sharing"><img alt="" src="http://farm4.static.flickr.com/3118/2750315635_a875ba2f50_m.jpg" style="border: solid 2px #000000;" /></a> <a href="http://www.flickr.com/photos/squiz1210/346977907/" title="photo sharing"><img alt="" src="http://farm1.static.flickr.com/162/346977907_b0b4d59ad7_m.jpg" style="border-bottom-color: rgb(0, 0, 0); border-bottom-style: solid; border-bottom-width: 2px; border-left-color: rgb(0, 0, 0); border-left-style: solid; border-left-width: 2px; border-right-color: rgb(0, 0, 0); border-right-style: solid; border-right-width: 2px; border-top-color: rgb(0, 0, 0); border-top-style: solid; border-top-width: 2px;" /></a> <a href="http://www.flickr.com/photos/klutts/128167164/" title="photo sharing"><img alt="" src="http://farm1.static.flickr.com/1/128167164_f9c831ded1_m.jpg" style="border: solid 2px #000000;" /></a><br />
<br />
<hr align="center" width="70%" />
<div style="text-align: center;">
<span class="Apple-style-span" style="font-weight: bold;">Photo Credits</span></div>
<a href="http://www.flickr.com/people/miltoncorrea/">Milton CJ</a><br />
<a href="http://www.flickr.com/photos/roberto1956/3110481831/">StortTorget - Gamla Stan</a> Originally uploaded by <a href="http://www.flickr.com/people/roberto1956/">Roberto1956</a><br />
<a href="http://www.flickr.com/photos/lenora/17081398/">Gamla Stan</a> Originally uploaded by <a href="http://www.flickr.com/people/lenora/">L e n o r a</a><br />
<a href="http://www.flickr.com/photos/calvaryzone/118483879/">Downtown Gamla Stan</a> Originally uploaded by <a href="http://www.flickr.com/people/calvaryzone/">calvaryzone</a><br />
<a href="http://www.flickr.com/photos/arendpictures/928126782/">2007_0331finlandsweden0120</a> Originally uploaded by <a href="http://www.flickr.com/people/arendpictures/">Arend </a><br />
<a href="http://www.flickr.com/photos/toveo/2827805486/">Karolinska Hospital park on a sunny morning</a> Originally uploaded by <a href="http://www.flickr.com/people/toveo/">tove!</a><br />
<a href="http://www.flickr.com/people/arendpictures/">Kuester</a><br />
<a href="http://www.flickr.com/photos/16173303@N04/2750315635/">Lofoten ... fjord panorama (HDR)</a> Originally uploaded by <a href="http://www.flickr.com/people/16173303@N04/">nigel_xf</a><br />
<a href="http://www.flickr.com/photos/squiz1210/346977907/">From our hotel window - taken at midday (that's the moon!)</a> Originally uploaded by <a href="http://www.flickr.com/people/squiz1210/">Squiz1210 (probably back in January 09!)</a><br />
<a href="http://www.flickr.com/photos/klutts/128167164/">Hurtigruten</a> Originally uploaded by <a href="http://www.flickr.com/people/klutts/">klutts</a><br />
<a href="http://www.flickr.com/photos/foide/3087146334/">Vaasa in december 1</a> Originally uploaded by <a href="http://www.flickr.com/people/foide/">Foide</a><br />
<a href="http://www.flickr.com/photos/66538587@N00/82490855/">djurgarden</a> Originally uploaded by <a href="http://www.flickr.com/people/66538587@N00/">daniel dssd</a>Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com0Kåkbrinken, 11127 Stockholm, Sweden59.3243062 18.068825259.3188327 18.0542342 59.3297797 18.0834162tag:blogger.com,1999:blog-8720550037867345187.post-79759514995821539542009-10-13T13:55:00.009+11:002010-03-29T09:30:43.224+11:00Machine Learning While I WorkI am setting up <a href="http://www.postfix.org/TLS_README.html#client_tls">Postfix</a> so I have spare time as I try things out. This post is about the things I am reading or watching in the background.<br />
<br />
<a href="http://hercules.infotech.monash.edu.au/EII-CAC/">Taskforce on Context-Aware Computing</a><br />
I went to a lecture called <a href="https://www.acs.org.au/vic/index.cfm?action=event&area=9003&temID=eventdetails&eveID=10148598860936">Open Mobile Miner (OMM): A System for Real Time Mobile Data Analysis</a>. There is a video <a href="http://www.csse.monash.edu.au/~shonali/OMM/OMM-VideoDemo.asf">here</a>, a description of OMM <a href="http://www-ai.cs.uni-dortmund.de/PROCEEDINGS/SIKDD2009/demos/D02-kdd09demo.pdf">here</a> and lecture slides <a href="http://melbourne-mobility-sig.googlegroups.com/web/1st-Seminar-Oct-09-ACS-Mob-SIG-OMM.pdf?gda=mBKG_F4AAADxBFnGTxFEuFiei4W9wE3pySq_8OVEDIXBhXAec39pA4pxiiyIaEz2AUPUyM07c_VoZ9n0mJ2FR9aAiiZ8r3sxOhI5K8VDwPHDA0n0i2aByeUDNUu6Ev52K7sXoEgGoK8&pli=1">here</a> (pdf).<br />
<br />
Shonali Krishnaswamy's group are making software that does some analysis of data on a smart phone before uploading it, thereby reducing the phone's power consumption by reducing communications. Their examples include ECG output, traffic congestion metrics and taxi location data. The data in their examples is scalar and sampled at 0.5 Hz or less so it is hard to see why a simple store-and-forward scheme would not achieve much the same thing. I guess I need to read their publications more deeply.<br />
<br />
<br />
<a href="http://videolectures.net/cikm08_norvig_slatuad/">
<img border="0/" src="http://videolectures.net/cikm08_norvig_slatuad/thumb.jpg" />
<br />Statistical Learning as the Ultimate Agile Development Tool</a> by
Peter Norvig is an overview of modern practical machine learning. The summary is <b>focus on the data, not the code</b>.<br />
<br />
<br />
<a href="http://videolectures.net/ssll09_reid_leth/">
<img border="0/" src="http://videolectures.net/ssll09_reid_leth/thumb.jpg" />
<br />Learning Theory</a> by
Mark Reid was an introduction to some theoretical aspects of machine learning presented in a summer school in Canberra in January 2009.<br />
<br />
<hr align="left" width="100%" />
Now some videos of how machine learning can be applied to models of the face.<br />
<br />
Changes of facial features on the of dominance, trustworthiness and competence dimensions in a computer model developed by <a href="http://webscript.princeton.edu/~tlab/wp-content/publications/Todorov_PNAS2008.pdf">Oosterhof & Todorov</a> (2008).
<iframe height="600" src="http://webscript.princeton.edu/~tlab/demonstrations/" width="100%"><p><p><p><p><p>Your browser does not support iframes.</p>
</p></p></p></p></iframe>
<br />
<hr align="left" width="100%" />
<br />
Now it is time to start watching a video on distributed computing<br />
<object height="300" width="400"><param name="allowfullscreen" value="true" />
<param name="allowscriptaccess" value="always" />
<param name="movie" value="http://vimeo.com/moogaloop.swf?clip_id=6614042&server=vimeo.com&show_title=1&show_byline=1&show_portrait=0&color=&fullscreen=1" />
<embed src="http://vimeo.com/moogaloop.swf?clip_id=6614042&server=vimeo.com&show_title=1&show_byline=1&show_portrait=0&color=&fullscreen=1" type="application/x-shockwave-flash" allowfullscreen="true" allowscriptaccess="always" width="400" height="300"></embed></object><br />
<a href="http://vimeo.com/6614042">Swarm: Distributed Computation in the Cloud</a> from <a href="http://vimeo.com/user2266455">Ian Clarke</a> on <a href="http://vimeo.com/">Vimeo</a>.Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com1Dandenong Rd, VIC, Australia-37.8697919 145.0339286-37.903669400000005 144.9755636 -37.8359144 145.0922936tag:blogger.com,1999:blog-8720550037867345187.post-39594586027834842242009-09-18T06:40:00.000+10:002009-10-29T14:08:15.225+11:00My First Upside Down Post˙ʇǝsdıɥɔ pǝsɐq ɯɹɐ s’ǝןɐɔsǝǝɹɟ 'uoƃɐɹpdɐus ɯɯoɔןɐnb 'ɯoʇɐ ןǝʇuı ˙sɹɐʍ ɹossǝɔoɹd ˙sʞooqʇǝu puɐ sǝuoɥd ʇɹɐɯs ɟo ǝɔuǝƃɹǝʌuoɔ ǝןqıssod ˙ǝƃuɐɥɔ ǝʌıɹp ןןıʍ ǝɔuǝɹǝɟɟıp ǝɔıɹd 000'1$ ˙sʌ 052$ ǝɥʇ ˙pǝʇıns-ןןǝʍ os ʇou sǝop pɹoʍ ʇɟosoɹɔıɯ ǝןıɥʍ pnoןɔ ǝɥʇ ɯoɹɟ ןןǝʍ sʞɹoʍ ʎpɐǝɹןɐ ǝɹɐʍʇɟos ɹǝɥʇo puɐ uozɐɯɐ 'ɯoɔ˙ǝɔɹoɟsǝןɐs 'ǝןƃooƃ ˙ǝɹɐʍʇɟos ʇɟosoɹɔıɯ ǝɥʇ doʇdɐן ǝɔɐןdǝɹ ʎɐɯ pnoןɔ + ʞooqʇǝu os ƃuıʇndɯoɔ pnoןɔ oʇ pǝʇıns-ןןǝʍ ǝɹɐ sʞooqʇǝu ˙ʇɟosoɹɔıɯ puɐ ןǝʇuı oʇ ʇsoɔ ʇɐǝɹƃ ʇɐ sǝןɐs doʇdɐן 0001$sn ǝzıןɐqıuuɐɔ ʎɐɯ sʞooqʇǝu 052$sn ɯɹǝʇ ɹǝƃuoן ǝɥʇ uı ʇnq ǝıd ɔd ǝɥʇ ƃuıʍoɹƃ ǝɹɐ sʞooqʇǝu ʎןʇuǝɹɹnɔ ¿uıɐɥɔ ǝnןɐʌ doʇdɐן puɐ ɔd ǝɥʇ oʇ op sʞooqʇǝu ןןıʍ ʇɐɥʍ ˙sǝןɐs ƃuıʇsıxǝ ɟo uoıʇɐzıןɐqıuuɐɔ
pnoןɔ ǝɥʇ oʇ ʇuǝɯǝʌoɯ ɟo sǝɔuǝnbǝsuoɔ ǝɯos sɹoʇıʇǝdɯoɔ ɹıǝɥʇ ɹǝʌo ǝƃɐʇuɐʌpɐ ƃuıɔıɹd ɐ ǝʌǝıɥɔɐ ןןıʍ sıɥʇ ǝpıʌoɹd uɐɔ oɥʍ sɹǝıɹɹɐɔ uoıʇɐɔıunɯɯoɔǝןǝʇ ǝɥʇ ˙suoıʇɔǝuuoɔ ʞɹoʍʇǝu ǝןqɐʇɹod puɐ ǝןqɐıןǝɹ 'ʇsɐɟ sǝɹınbǝɹ ƃuıʇndɯoɔ pnoןɔ ʎɥdɹnɯ ˙ɹɯ sʎɐs „'ɹǝʇʇǝq ʇnq ƃuıɥʇou uǝǝq s,ʇı 'dɯnɥ ƃuıuɹɐǝן ǝɥʇ ɹǝʌo ʇoƃ ǝʍ ǝɔuo ˙sǝƃɐʇuɐʌpɐsıp ןɐǝɹ ʎuɐ ɥʇıʍ dn ƃuıɯoɔ pǝssǝɹd-pɹɐɥ ǝq p,ı ˙ǝɹoɯ ʎuɐ suoıʇɐןןɐʇsuı ǝɹɐʍpɹɐɥ ןɐɔısʎɥd ɹoɟ ƃuıʇıɐʍ sʎɐp puɐ sɹnoɥ puǝds ʇ,uop ǝʍ„ ˙sɹǝʇndɯoɔ uʍo sʇı ɟo ǝuou ɥʇıʍ ʎuɐdɯoɔ ʇuǝɯdoןǝʌǝp qǝʍ ɐ ǝɯoɔǝq sɐɥ ʇı - ǝɹɐʍpɹɐɥ ןɐɔısʎɥd ɹoɟ ʇno ƃuıʞɹoɟ pǝddoʇs sɐɥ ʇı suɐǝɯ ɥɔıɥʍ 'ǝɔıʌɹǝs ןɐnʇɹıʌ ǝɥʇ ɹoɟ ɹnoɥ ɹǝd sʇuǝɔ 08 oʇ 01 sʎɐd ǝƃuɐɹo ʎɔınɾ ˙ǝɯoɔǝq sɐɥ ƃuıʇndɯoɔ pnoןɔ ɯɐǝɹʇsuıɐɯ ʍoɥ sǝʇɐɹʇsnןןı (9002 qǝɟ 02 ǝƃɐ ǝɥʇ) ɯɐǝɹʇsuıɐɯ ǝɥʇ spuǝɔsɐ ƃuıʇndɯoɔ pnoןɔ ”˙ƃuıʍoɹƃ ןןıʇs sı ןɐɹǝuǝƃ uı ƃ3“ ˙sʇdǝɔuoɔ pɹɐʍɹoɟ ʇɐ ןɐdıɔuıɹd 'ssnɐɹʇs ןןıʍ pıɐs ”'sʇods ʇɥƃıɹq ǝɹɐ ǝsǝɥʇ“˙˙˙ ˙pǝʌɹǝsqo ǝʌɐɥ sʇsʎןɐuɐ 'ǝɔuɐɥɔ ƃuıʇɥƃıɟ ɐ ǝʌɐɥ sɯǝpoɯ ƃ3 puɐ sdıɥɔ ɥʇooʇǝnןq puɐ ıɟ-ıʍ 'sdƃ ǝʞɐɯ oɥʍ sɹopuǝʌ puɐ — ɥʇʍoɹƃ ʇɐɥʇ ɟo ɥʇƃuǝɹʇs ǝɥʇ uo ʎɹɐʌ sʇsʎןɐuɐ — ɹɐǝʎ sıɥʇ ʍoɹƃ oʇ pǝʇɔǝɾoɹd ǝɹɐ sǝןɐs ǝuoɥdʇɹɐɯs ˙pɐǝɥɐ ɹɐǝʎ ǝɥʇ uı sʇods ʇɥƃıɹq ƃuıʞǝǝs puɐ sɥʇƃuǝɹʇs ǝɹoɔ oʇ ƃuıʞooן ǝɹɐ sɹopuǝʌ dıɥɔ ʇsoɯ 'ǝןıɥʍuɐǝɯ˙˙˙ ˙sʇǝʞɹɐɯ ʍǝu uǝdo oʇ ʎɐןd ǝuoɥd ʇɹɐɯs ɐ ƃuıɹɐdǝɹd ǝq oʇ pǝɹoɯnɹ sı ˙ɔuı ןןǝp ɹǝʞɐɯ ɔd ˙ǝuoɥdı s’˙ɔuı ǝןddɐ ʎq pǝʌɹǝs ʇǝʞɹɐɯ ǝɥʇ ɟo ǝɯos qɐɹƃ oʇ ƃuıdoɥ 'ʍoןs sǝʇɐɹ sǝןɐs ɔd sɐ spıɯ pǝʇǝƃɹɐʇ sɐɥ — ɹǝʞɐɯ dıɥɔ ʇsǝƃɹɐן s’pןɹoʍ ǝɥʇ — ˙dɹoɔ ןǝʇuı sɐ 'uoıʇıʇǝdɯoɔ ǝʌɐɥ ןןıʍ ɯɯoɔןɐnb˙˙˙spןǝɥpuɐɥ puɐ sdoʇdɐן uǝǝʍʇǝq dɐƃ ǝɔıɹd ǝɥʇ ǝƃpıɹq sʞooqʇǝu sɐ ǝƃɹns oʇ pǝʇɔǝɾoɹd ʎɹoƃǝʇɐɔ ɹǝʇʇɐן ǝɥʇ ɥʇıʍ 'sǝɔıʌǝp ʇǝuɹǝʇuı ǝןıqoɯ puɐ sʞooqʇǝu 'sʞooqǝʇou ɹoɟ ʇǝsdıɥɔ uoƃɐɹpdɐus sʇı uo sısɐɥdɯǝ pǝɔɐןd sɐɥ ɯɯoɔןɐnb 'ǝןıɥʍuɐǝɯ sdıɥɔ :ʇsɐɔǝɹoɟ ssǝןǝɹıʍ 9002 ssǝןǝɹıʍ ɹɔɹ sǝuıɥɔɐɯ dx sʍopuıʍ ɹo nʇunqn ʎןןɐnsn ǝɹɐ ʇɐɥʍ ƃuoɯɐ ǝɥɔıu ɐ puıɟ ןןıʍ ɯɹoɟʇɐןd ǝןıqoɯ ǝɔɹnos-uǝdo s’ǝןƃooƃ ʇɐɥʇ ƃuıʇʇǝq sı - ʎɐpoʇ ʇǝʞɹɐɯ ǝɥʇ uo sʞooqʇǝu ɟo ʎʇıɹoɾɐɯ ʇsɐʌ ǝɥʇ uı punoɟ ɹossǝɔoɹd 072u ɯoʇɐ ןǝʇuı ǝɥʇ ɹoɟ ǝןqısuodsǝɹ - ɹǝʞɐɯdıɥɔ ǝɥʇ ʇɐɥʇ sʇsǝƃƃns oɥʍ '”ǝɔɹnos ǝןqɐıןǝɹ“ s’ʇɐǝqǝɹnʇuǝʌ oʇ ƃuıpɹoɔɔɐ s’ʇɐɥʇ ˙ǝɹɐʍpɹɐɥ ʇǝsdıɥɔ ǝןqɐʇıns ɥʇıʍ sɹǝɹnʇɔɐɟnuɐɯ ʇɹoddns oʇ ƃuıɹɐdǝɹd sı puɐ '0102 ʇnoɥƃnoɹɥʇ puɐ 9002 ǝʇɐן uı sʞooqʇǝu pǝsɐq-pıoɹpuɐ ɟo ʎɹɹnןɟ ɐ ƃuıʇɔǝdxǝ sı ןǝʇuı sʞooqʇǝu pǝsɐq-pıoɹpuɐ ɟo ǝsıɹ ɹoɟ ƃuıʎpɐǝɹ ןǝʇuı ʞooqʇǝu pǝsɐq-pıoɹpuɐ ɟo ǝsıɹ ɹoɟ ƃuıʎpɐǝɹ ןǝʇuı ʎoʎ %6˙12– puɐ bob %7˙12– pǝuıןɔǝp sʇuǝɯdıɥs ʇıun ɹossǝɔoɹd ɔd ǝpıʍpןɹoʍ 'ɯoʇɐ ʇnoɥʇıʍ ˙ǝuıןɔǝp ɔıʇɐɯɐɹp pıoʌɐ ʇǝʞɹɐɯ ǝɥʇ dןǝɥ oʇ ɥƃnouǝ ʇou ʇnq ǝɔuɐɯɹoɟɹǝd ʇǝʞɹɐɯ ןןɐɹǝʌo ǝɥʇ uı ǝɔuǝɹǝɟɟıp ǝןqɐʇou ɐ ǝʞɐɯ oʇ pǝnuıʇuoɔ (,,sʞooqʇǝu,, sןןɐɔ ןǝʇuı ɥɔıɥʍ) sɔd ʞooqǝʇou-ıuıɯ ɹoɟ ɹossǝɔoɹd ɯoʇɐ s,ןǝʇuı ˙˙˙ ؛(ʎoʎ) ɹɐǝʎ ɹǝʌo ɹɐǝʎ %4˙11– puɐ (bob) ɹǝʇɹɐnb ɹǝʌo ɹǝʇɹɐnb %0˙71– pǝuıןɔǝp sʇuǝɯdıɥs ʇıun ɹossǝɔoɹd ɔd ǝpıʍpןɹoʍ '80b4 uı 80b4 uı sʞooqʇǝu oʇ sdoʇdɐן sʍopuıʍ ɯoɹɟ ǝʌoɯ sıɥʇ ʇɹoddns oʇ sǝıɹoʇs ǝɯos ǝɹɐ ǝɹǝɥ ˙ʎʇıןıqoɯ puɐ ʎɹʇsǝɔuɐ ǝuoɥd ɹıǝɥʇ ɯoɹɟ sǝɯoɔ ʇɐɥʇ uoıʇdɯnsuoɔ ɹǝʍod ʍoן ɟo sǝƃɐʇuɐʌpɐ ןɐuoıʇıppɐ ǝɥʇ ǝʌɐɥ ʎǝɥʇ ˙sƃuıɹǝɟɟo pnoןɔ ɹǝɥʇo puɐ uoıʇɐzıןɐnʇɹıʌ 'sɐɐs 'sǝɔıʌɹǝs qǝʍ ɹoɟ sʇuǝıןɔ ǝʇɐnbǝpɐ ǝʞɐɯ ǝuoɥd ʇɹɐɯs ǝןqɐdɐɔ ʎɹǝʌ puɐ sʞooqʇǝu ƃ3 ˙sǝıƃǝʇɐɹʇs ƃuıʇndɯoɔ doʇʞsǝp pǝnsɹnd ʇou ǝʌɐɥ puɐ sǝɔıʌɹǝs ɟo sǝdʎʇ ǝsǝɥʇ uo ʎןǝɹıʇuǝ sǝssǝuısnq ɹıǝɥʇ pǝsɐq ǝʌɐɥ oɥʍ ɯoɔ˙ǝɔɹoɟsǝןɐs puɐ ǝɹɐʍɯʌ 'ǝןƃooƃ sɐ ɥɔns sǝıuɐdɯoɔ ʎq uǝʌıɹp uǝǝq sɐɥ ʇuǝɯdoןǝʌǝp ɹıǝɥʇ ˙sɹɐǝʎ 01 ʇsɐן ǝɥʇ uı ǝʌıʇɔǝɟɟǝ ʎןɥƃıɥ ǝɯoɔǝq ǝʌɐɥ ǝsǝɥʇ ɟo ʇsoɯ ˙ suoıʇɐɔıןddɐ qǝʍ puɐ sɐɐs 'uoıʇɐzıןɐnʇɹıʌ ƃuıpnןɔuı sǝɔıʌɹǝs pnoןɔ ɟo sǝdʎʇ ʎuɐɯ ǝɹɐ ǝɹǝɥʇ ˙uoıʇɐzıuɐƃɹo uɐ ɥƃnoɹɥʇ pǝʇɐɔıןdǝɹ ǝq oʇ pǝǝu ʇou sǝop ǝƃɐɹoʇs ʞsıp ʎʇıןıqɐıןǝɹ-ɥƃıɥ sɐ ɥɔns ǝɹnʇɔnɹʇsɐɹɟuı ǝʌısuǝdxǝ ʇɐɥʇ ʇıɟǝuǝq ןɐuoıʇıppɐ ǝɥʇ sı ǝɹǝɥʇ ˙ʇuǝıɔıɟɟǝ ǝɹoɯ ɥɔnɯ sı ɹǝʌɹǝs ןɐɹʇuǝɔ ɐ uo ǝɹɐʍʇɟos ǝɥʇ ƃuıuunɹ ǝɹoɟǝɹǝɥʇ ˙%01 uɐɥʇ ssǝן ɥɔnɯ 'ʍoן ʎɹǝʌ sı ǝɹɐʍʇɟos sıɥʇ ɟo ǝƃɐsn ǝɔɹnosǝɹ ɹǝʇndɯoɔ ǝƃɐɹǝʌɐ ǝɥʇ ˙ǝʌısuodsǝɹ ǝq oʇ ǝɔɐɟɹǝʇuı ɹǝsn ǝɥʇ ʇuɐʍ noʎ puɐ suoıʇɐʇndɯoɔ ǝsuǝʇuı ǝɥʇ sǝop ʇı uǝɥʍ ʎɐp ɹǝd sǝʇnuıɯ ʍǝɟ ǝɥʇ 'ǝƃɐsn ʞɐǝd ʇɹoddns oʇ ƃuıʎɐd ǝɹɐ noʎ 'ʇı ʇɹoddns oʇ ǝɹɐʍpɹɐɥ ɔd ǝʌısuǝdxǝ ʎnq noʎ uǝɥʍ ˙ǝɯıʇ ǝɥʇ ɟo ʇsoɯ ƃuıɥʇou sǝop ʇı ˙sǝןɔʎɔ ʎʇnp ʍoן ʎɹǝʌ sɐɥ 'sɔıɥdɐɹƃ ʎʇıןɐnb ɥƃıɥ ǝʞıן ǝɹɐʍʇɟos ǝʌısuǝʇuı ʎןןɐuoıʇɐʇndɯoɔ uǝʌǝ 'ǝɹɐʍʇɟos ʇuǝıןɔ ʇsoɯ ˙sɹǝʇndɯoɔ ןɐuosɹǝd s,ǝןdoǝd ʎuɐɯ uo ʇı ƃuıop uɐɥʇ ɹǝısɐǝ sı uoıʇɐɔoן ןɐɔısʎɥd ǝuo uı ƃuıuunɹ ǝɹɐʍʇɟos ƃuıpɐɹƃdn puɐ ƃuıuıɐʇuıɐɯ ǝsnɐɔǝq uoıʇɐzıןɐɹʇuǝɔ ɥʇıʍ ʎןןɐɔıʇɐɯɐɹp sǝsɐǝɹɔǝp (oɔʇ) dıɥsɹǝuʍo ɟo ʇsoɔ ןɐʇoʇ ˙sʇuǝıןɔ ɹǝןןɐɯs ǝʌɐɥ puɐ ƃuıʇndɯoɔ ǝzıןɐɹʇuǝɔ-ǝɹ oʇ sı ʇı ǝsıɹdɹǝʇuǝ uı puǝɹʇ ɹoɾɐɯ ʇuǝɹɹnɔ ɐPeter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com1Unknown location-84.7060489350415 -112.5-89.1610814350415 -172.265625 -80.251016435041493 -52.734375tag:blogger.com,1999:blog-8720550037867345187.post-70777355780401152022009-09-14T07:01:00.000+10:002009-09-17T13:31:19.207+10:00What I need from a 3G NetbookI could use one know for<br />
<ul>
<li>working on the train</li>
<li>working at cafes while waiting for the kids</li>
<li>working in the country while visiting friends and family</li>
</ul>
<div>
My PC and laptop seem like overkill for researching on the web, emailing, writing reports and building a few models in a spreadsheet. They also use a lot of electricity and take up space.</div>
<div>
</div>
<div>
To be an effective replacement for the PC and laptop, a netbook would need to have</div>
<div>
<ol>
<li>Be reasonably priced. $200 would be nice.</li>
<li>Have a reasonably priced connection plan.</li>
<li>Have access to cheap or free software</li>
<br />
<ol>
<li>Web browser</li>
<li>Word processor</li>
<li>Drawing tools</li>
<li>Spreadsheet</li>
</ol>
<li>Reliable connection with good coverage.</li>
<li>Good battery life. 6 hours would be nice</li>
</ol>
<div>
Basic set of applications</div>
</div>
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
<ol>
<li>Gmail with tasks</li>
<li>Google calender</li>
<li>Google Docs</li>
<li>GIT for source code management</li>
<li>YUML</li>
<li>Ubuntu or Windows XP with cygwin</li>
<li>Gnu tools</li>
<li>VNC or WRD</li>
<li>ssh</li>
</ol>
</div>
That would get me started. It would be nice to have Eclipse and a local word processor but running these over a remote shell would be more than adequate. I worked that way with all my heavy tools on VMware instances for years and it worked well. The VMware instance were hosted in a data center and backed up regularlyPeter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com01-99 Maza, Buenos Aires, Argentina-34.611781 -58.417309-90 -177.948559 26.511318000000003 61.113941tag:blogger.com,1999:blog-8720550037867345187.post-53963421515590167282009-09-10T05:58:00.002+10:002009-09-13T15:44:54.514+10:00Electronic Medical Records Bonanza?<a href="http://aushealthit.blogspot.com/2009/02/commercial-alert-big-bucks-in-health-it.html">Big Bucks in Health IT!</a>, quoting from <a href="http://www.healthcareitnews.com/news/global-market-hospital-it-systems-pegged-35b-2015">http://www.healthcareitnews.com/news/global-market-hospital-it-systems-pegged-35b-2015</a> , says <br />
<blockquote>
SAN JOSE, CA – <span style="font-weight: bold;">The global hospital information systems market will climb past $35 billion by 2015, </span>according to a new forecast by Global Industry Analysts. The United States represents the largest market in the world.
The U.S. hospital information system market is experiencing an increase in acceptance of customized technology such as laboratory information systems and radiology information systems, the report notes. <span style="font-weight: bold;">The market is also a promising ground for electronic medical record systems.
</span></blockquote>
<blockquote>
The <span style="font-weight: bold;">Asia-Pacific region (excluding Japan) represents the fastest growing hospital information systems market, exhibiting a compounded annual growth rate of 11.5 percent over the next few years</span>, according to analysts. Despite being a smaller market in terms of revenue, the Asia-Pacific promises excellent growth opportunities for hospital information systems, they said.
</blockquote>
<blockquote>
The global vendors profiled in the report include <a href="http://www.mckesson.com/en_us/McKesson.com/Our%2BBusinesses/McKesson%2BProvider%2BTechnologies/Newsroom/2009/McKesson%2BLaunches%2BInformation%2BResources%2Bto%2BHelp%2BPhysicians%2BTake%2BAdvantage%2Bof%2BEconomic%2BStimulus.html">McKesson</a> , <a href="http://www.cerner.com/public/Cerner_3.asp?id=129">Cerner</a> , Allscripts-Misys Healthcare Solutions, Eclipsys, Computer Programs and Systems, Siemens Medical Solutions USA, QuadraMed, Medical Information Technology, Healthland, GE Healthcare, iSOFT Group, Agfa-Gevaert, Brunie-Software, IBA Health and Integrated Medical Systems. </blockquote>
The full release is here:
<span style="font-weight: bold;">Global Hospital Information Systems Market to Cross $35 Billion by 2015</span>, According to New Report by Global Industry Analysts, Inc.
Increasing awareness among medical service patrons on the benefits of using Information Technology in the healthcare sector, coupled with growing demand for affordable-yet quality healthcare services is forcing hospitals and other medical centers to adopt IT in their daily operations. Subsequently, Healthcare IT systems such as the Hospital Information Systems witnessed a great demand in the healthcare services sector. Adoption of HIS in hospitals is increasingly being encouraged and promoted by the Governments world over.
<a href="http://www.prweb.com/releases/2009/02/prweb2021984.htm">http://www.prweb.com/releases/2009/02/prweb2021984.htm</a>Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com0Madeira, Portugal32.7607074 -16.959472332.4719914 -17.426391300000002 33.0494234 -16.4925533tag:blogger.com,1999:blog-8720550037867345187.post-88222412157267511932009-09-09T10:21:00.000+10:002010-10-19T22:27:08.065+11:00bit.ly custom URLsWhere does <a href="http://bit.ly/peterwilliams">http://bit.ly/peterwilliams</a> direct to?<br />
<br />
Is it the same web page as <a href="http://linkd.in/PeterWilliams">http://linkd.in/PeterWilliams</a> ?Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com0Umea, Sweden63.8380201 20.248004262.6268541 16.5126527 65.0491861 23.9833557tag:blogger.com,1999:blog-8720550037867345187.post-46586449132044136802009-08-16T10:56:00.006+10:002010-09-02T20:46:30.043+10:00Open Goverment Made SimpleThere has been a lot of talk about <a href="http://radar.oreilly.com/2009/03/bulk-data-downloads-government-transparency-breakthrough.html">Open Government</a> recently, including <a href="http://deloittedigital.blogspot.com/2009/08/open-government-heating-up.html">this</a> from <a href="http://twitter.com/rexster">Peter Williams</a><br />
<blockquote>
"Australian governments should adopt international standards of open publishing as far as possible. Material released for public information by Australian governments should be released under a creative commons licence."
or in simple terms "make public data open and free".</blockquote>
That is useful, clear and straightforward.<br />
<br />
So how to do it?<br />
<br />
My experience in organising company data over the last 10 years is that the teams I have worked in have tried many <a href="http://en.wikipedia.org/wiki/List_of_content_management_systems">content management systems</a> (CMSs) and none of them were satisfying to use (though some were interesting to implement.) Inevitably the document taxonomies that made sense to the site administrators did not work for most of the users and the users soon gave up trying to find things through the CMS.<br />
<br />
Then one day someone in the company I was working at purchased a <a href="http://en.wikipedia.org/wiki/Google_Search_Appliance">Google Search Appliance</a> (GSA) and indexed most of our intranet with it. After that everybody could find all the documents they knew existed on the intranet and discovered useful ones they did not know existed.<br />
<br />
To be fair, things were not quite that simple. Most companies need reliable storage, decent version tracking, access control and many other things that CMSs provide. However people need to be able to find documents much more than they need these other things. Very few people need version tracked, access controlled documents that they cannot find in the first place.<br />
<br />
So why don't goverments just make their data visible to internet search engines and store it somewhere secure with some simple versioning system now, and then do the fancy stuff later? Why are they are investing in CMSs like <a href="http://egovau.blogspot.com/2008/06/considering-using-microsoft-sharepoint.html">Sharepoint</a>?<br />
<br />
The reason we did not do this in the companies I worked in was that many of the features in the CMSs we used were useful and the people who implemented the systems decided they needed all these features. Finding documents was just one of several check-boxes on their requirements documents. They were acting as implementers and experts, not users. The systems they ended up with made perfect sense to everyone except the users.<br />
<br />
The interesting thing about this was the implementers were users in most cases. They were aware of the limitations of CMSs but they had to follow either the direction of their users who had had not used CMSs enough and to understand how badly they would work in practice or the direction of their managers and key stakeholders who had heard that CMSs were good. The person who got the GSA was an IT guy who just went out and tried it without surveying users or bringing in CMS vendors to talk to his key stakeholders.<br />
<br />
For a different perspective on Open Government, read some <a href="http://radar.oreilly.com/2009/03/bulk-data-downloads-government-transparency-breakthrough.html">Tim O'Reilly</a>.Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com1Canberra ACT, Australia-35.28204 149.12858-35.2995565 149.0993975 -35.2645235 149.1577625tag:blogger.com,1999:blog-8720550037867345187.post-44827596091299676912009-07-16T10:37:00.006+10:002009-09-16T16:55:00.797+10:00When Words FailA while back I worked at a company who made software+hardware products in a maturing market. The company found it needed to deliver higher quality products with more features and was struggling to do so from an old codebase. It had become clear to the management team that late-stage serious defects were the major cause of schedule/quality issues but they had been able to fix this problem.<br />
<br />
The codebase management team had a lot of ideas about what the causes were and how to fix them. They had discussed "technical debt", "silo-ing" and other causes. However in the end they settled on two key priorities: taking extreme care with code changes and sticking with established QA processes to minimise the number of introduced bugs.<br />
<br />
Eventually the project was given to me to manage. One of the (many) things the development team had done well was to document each bug and cross-reference bug fixes against the source code. I analysed about 100 recently fixed serious software bugs, looked up their fixes in the SCM and then looked up the date at which the code changes causing the bug were checked in. This showed that most of the bugs being found had been introduced months before they were discovered. It was clear that the late-stage defects were dominated by latent bugs being unmasked by changes, not by bugs introduced by changes.<br />
<br />
Some changes to the development process were needed. The development group was responsible for creating code without introducing bugs and the QA group was responsible for finding the bugs the development team missed. However the QA process was unsuited to discovering latent bugs fast because it had a long cycle based on testing user scenarios. Therefore I got small teams of developers and QAs to work closely together to find, fix and verify bugs and I took some developers away other work to develop a system to find and fix (and eventually prevent the introduction of more) latent bugs. This work is described <a href="http://peterwilliams97.blogspot.com/2008/10/test-automation-for-complex-systems.html">here</a>. With these changes in place, code stability improved rapidly and late-stage serious bugs essentially ceased to be found.<br />
<br />
That was a fairly straightforward technical solution to a fairly straightforward technical problem. So why had the very capable management team who had known the underlying causes (technical debt and silo-ing) not been able to fix the problem for so long?<br />
<br />
Change is known to be difficult in organisations and there is an <a href="http://en.wikipedia.org/wiki/Change_management_(people)">industry built around dealing with this</a>. However our immediate problem was not an inability to persuade people to change. In fact consultation and review had been distracting people from doing the experimentation required to find the underlying causes of the problem was and how to fix them. The more people talked about the problem the further they got from the solution (hence this post's title). <br />
<br />
The situation reminded me of Uncle Bob Martin's <a href="http://weblogs.java.net/blog/rmartin/archive/2003/09/aristotles_erro.html">Agile Smagile</a><br />
<blockquote>
As I said before, going meta is a good thing. However, going meta requires experimental evidence. Unfortunately the industry has latched on to the word "Agile" and has begun to use it as a prefix that means "good". This is very unfortunate, and discerning software professionals should be very wary of any new concept that bears the "agile" prefix. The concept has been taken meta, but there is no experimental evidence that demonstrates that "agile", by itself, is good.</blockquote>
The deeply ingrained practices in the organisation I worked in had grown out of ideas that had worked well in the past. They had been good enough to cover a wide range of development scenarios for a long while and were clearly based on experimental evidence from past development. However somewhere along the way people had stopped experimenting and modifying the rules, and started just following the rules. This is what Uncle Bob called "going meta". The problem for our organisation was that the set of rules it had got to when it stopped experimenting were not universally true, they were only true for the type of the development they were doing when they stopped changing the rules.<br />
<br />
The changes I made to detect and fix latent bugs (<a href="http://peterwilliams97.blogspot.com/2008/10/test-automation-for-complex-systems.html">high-coverage automated system testing</a>, static analysis with <a href="http://www.klocwork.com/">Klocwork</a> and refactoring with unit tests) were adopted across the development organisation and became part of the standard development process, at least for the time I was there. That was good but I wondered if those practices would become a fixed part of the new development process because they had worked some time in the past. And I wondered whether they would prevent the company from addressing problems that arose in the future, just as the practises that had worked well in the past had come to do.Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com1Creswick Victoria, Australia-37.424838 143.894483-37.4930005 143.77775350000002 -37.3566755 144.0112125tag:blogger.com,1999:blog-8720550037867345187.post-25066418228940607892009-06-22T13:03:00.002+10:002009-06-22T13:09:46.986+10:00Minimal Non-C++Programmer Bamboozling C++ QuestionI recently read a stream of blog posts about why developers don't like C++ for general purpose programming. This <a href="http://www.mistybeach.com/articles/WhyIDontLikeCPlusPlusForLargeProjects.html">post</a> typifies much of the criticism of C++'s complexity. It includes an interview question about creating a C++ class that behaves like a class in a high level language such as Java. The author says that he uses this to weed out job applicants who haven't used C++ for real work.<br />
<br />
It strikes me that tripping up developers with C++'s many oddnesses is much easier than that. Here is a simple question that I believe will confuse many non-C++ programmers:<br />
<br />
What is the output of this program?<br />
<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace;">#include <string></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">#include <iostream></span><br />
<span style="font-family: 'Courier New', Courier, monospace;">using namespace std;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">class Parent {</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">public:</span><br />
<span class="Apple-tab-span" style="white-space: pre;"><span style="font-family: 'Courier New', Courier, monospace;"> </span></span><span style="font-family: 'Courier New', Courier, monospace;">string func() { return "parent"; }</span><br />
<span class="Apple-tab-span" style="white-space: pre;"><span style="font-family: 'Courier New', Courier, monospace;"> </span></span><span style="font-family: 'Courier New', Courier, monospace;">virtual string vfunc() { return "parent+virtual"; }</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">};</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">class Child : public Parent {</span><br />
<span class="Apple-tab-span" style="white-space: pre;"><span style="font-family: 'Courier New', Courier, monospace;"> </span></span><span style="font-family: 'Courier New', Courier, monospace;">string func() { return "child"; }</span><br />
<span class="Apple-tab-span" style="white-space: pre;"><span style="font-family: 'Courier New', Courier, monospace;"> </span></span><span style="font-family: 'Courier New', Courier, monospace;">virtual string vfunc() { return "child+virtual"; }</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">};</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">string test1(Parent parent) {</span><br />
<span class="Apple-tab-span" style="white-space: pre;"><span style="font-family: 'Courier New', Courier, monospace;"> </span></span><span style="font-family: 'Courier New', Courier, monospace;">return parent.func() + " - " + parent.vfunc();</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">}</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">string test2(Parent& parent) {</span><br />
<span class="Apple-tab-span" style="white-space: pre;"><span style="font-family: 'Courier New', Courier, monospace;"> </span></span><span style="font-family: 'Courier New', Courier, monospace;">return parent.func() + " - " + parent.vfunc();</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">}</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">int main() {</span><br />
<span class="Apple-tab-span" style="white-space: pre;"><span style="font-family: 'Courier New', Courier, monospace;"> </span></span><span style="font-family: 'Courier New', Courier, monospace;">Child child;</span><br />
<span class="Apple-tab-span" style="white-space: pre;"><span style="font-family: 'Courier New', Courier, monospace;"> </span></span><span style="font-family: 'Courier New', Courier, monospace;">cout << "test1: " << test1(child) << endl;</span><br />
<span class="Apple-tab-span" style="white-space: pre;"><span style="font-family: 'Courier New', Courier, monospace;"> </span></span><span style="font-family: 'Courier New', Courier, monospace;">cout << "test2: " << test2(child) << endl;</span><br />
<span style="font-family: 'Courier New', Courier, monospace;">}</span><br />
<br />
I have seen C++ interviewers ask questions like this but only show test1 then ask what is on the stack when test1 is invoked.Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com0Creswick VIC, Australia-37.424838 143.894483-37.4930005 143.77775350000002 -37.3566755 144.0112125tag:blogger.com,1999:blog-8720550037867345187.post-38195452824259430492009-06-02T10:14:00.005+10:002009-06-03T12:15:49.882+10:00Movement to the Cloud<div class="western" style="margin-left: 0px; margin-right: 0px;">
A current major trend in enterprise IT is to re-centralize computing and have smaller clients. Total cost of ownership (TCO) decreases dramatically with centralization because maintaining and upgrading software running in one physical location is easier than doing it on many people's personal computers. </div>
<div class="western" style="margin-left: 0px; margin-right: 0px;">
Most client software, even computationally intensive software like high quality graphics, has very low duty cycles. It does nothing most of the time. When you buy expensive PC hardware to support it, you are paying to support peak usage, the few minutes per day when it does the intense computations and you want the user interface to be responsive. The average computer resource usage of this software is very low, much less than 10%. Therefore running the software on a central server is much more efficient. There is the additional benefit that expensive infrastructure such as high-reliability disk storage does not need to be replicated through an organization.</div>
<div class="western" style="margin-left: 0px; margin-right: 0px;">
There are many types of cloud services including <a href="http://en.wikipedia.org/wiki/Virtualization">virtualization</a>, <a href="http://en.wikipedia.org/wiki/Software_as_a_service">SaaS </a> and <a href="http://en.wikipedia.org/wiki/Web_application">web applications</a> . Most of these have become highly effective in the last 10 years. Their development has been driven by companies such as Google, VMware and Salesforce.com who have based their businesses entirely on these types of services and have not pursued desktop computing strategies. </div>
<div class="western" style="margin-left: 0px; margin-right: 0px;">
3G netbooks and very capable smart phone make adequate clients for Web Services, SaaS, Virtualization and other cloud offerings. They have the additional advantages of low power consumption that comes from their phone ancestry and mobility.</div>
<div class="western" style="margin-left: 0px; margin-right: 0px;">
Here are some stories to support this </div>
<ol>
<li><div class="western" style="margin-left: 0px; margin-right: 0px;">
<span style="color: navy;"><u><a class="western" href="http://www.idc.com/getdoc.jsp?containerId=prUS21672009" target="_blank">Move from Windows Laptops to Netbooks in 4Q08 </a></u></span>In 4Q08, worldwide PC processor unit shipments declined –17.0% quarter over quarter (QoQ) and –11.4% year over year (YoY); ... Intel's Atom processor for mini-notebook PCs (which Intel calls ''Netbooks'') continued to make a notable difference in the overall market performance but not enough to help the market avoid dramatic decline. Without Atom, worldwide PC processor unit shipments declined –21.7% QoQ and –21.6% YoY</div>
</li>
<li><div class="western" style="margin-left: 0px; margin-right: 0px;">
<span style="color: navy;"><u><a class="western" href="http://www.slashgear.com/intel-readying-for-rise-of-android-based-netbooks-0232665/" target="_blank">Intel readying for rise of Android-based netbook </a></u></span><span style="color: black;">Intel readying for rise of Android-based netbooks Intel is expecting a flurry of Android-based netbooks in late 2009 and throughout 2010, and is preparing to support manufacturers with suitable chipset hardware. That’s according to VentureBeat’s “reliable source”, who suggests that the chipmaker - responsible for the Intel Atom N270 processor found in the vast majority of netbooks on the market today - is betting that Google’s open-source mobile platform will find a niche among what are usually Ubuntu or Windows XP machines</span></div>
</li>
<li><div class="western" style="margin-left: 0px; margin-right: 0px;">
RCR Wireless <span style="color: navy;"><u><a class="western" href="http://www.rcrwireless.com/article/20090202/WIRELESS/902029984/1081/2009-wireless-forecast-chips" target="_blank">2009 Wireless Forecast: Chips </a></u></span>Meanwhile, Qualcomm has placed emphasis on its Snapdragon chipset for notebooks, netbooks and mobile Internet devices, with the latter category projected to surge as netbooks bridge the price gap between laptops and handhelds...Qualcomm will have competition, as Intel Corp. — the world’s largest chip maker — has targeted MIDs as PC sales rates slow, hoping to grab some of the market served by Apple Inc.’s iPhone. PC maker Dell Inc. is rumored to be preparing a smart phone play to open new markets. ...Meanwhile, most chip vendors are looking to core strengths and seeking bright spots in the year ahead.<span style="color: #333333;"> </span><span style="color: navy;"><u><a class="western" href="http://www.rcrwireless.com/article/20090121/WIRELESS/901199980/Smartphones-a-bright-spot-for-chips">Smartphone sales are projected to grow </a></u></span>this year — analysts vary on the strength of that growth — and vendors who make GPS, Wi-Fi and Bluetooth chips and 3G modems have a fighting chance, analysts have observed. ...“These are bright spots,” said Will Strauss, principal at Forward Concepts. “3G in general is still growing.”</div>
</li>
<li><div class="western" style="margin-left: 0px; margin-right: 0px;">
<span style="color: navy;"><u><a class="western" href="http://www.theage.com.au/news/technology/biztech/cloud-computing-ascends-the-mainstream/2009/02/17/1234632818661.html" target="_blank">Cloud computing ascends the mainstream</a></u></span> (The Age 20 Feb 2009) illustrates how mainstream cloud computing has become. Juicy Orange pays 10 to 80 cents per hour for the virtual service, which means it has stopped forking out for physical hardware - it has become a web development company with none of its own computers. "We don't spend hours and days waiting for physical hardware installations any more. I'd be hard-pressed coming up with any real disadvantages. Once we got over the learning hump, it's been nothing but better," says Mr. Murphy</div>
</li>
</ol>
<div class="western" style="margin-left: 0px; margin-right: 0px;">
Cloud computing requires fast, reliable and portable network connections. The telecommunication carriers who can provide this will achieve a pricing advantage over their competitors</div>
<div class="western" style="margin-left: 0px; margin-right: 0px;">
</div>
<div class="western" style="margin-left: 0px; margin-right: 0px;">
</div>
<h2>
Some Consequences of Movement to the Cloud<br /></h2>
<ol>
<li><div style="margin-left: 0px; margin-right: 0px;">
Cannibalization of existing sales. What will netbooks do to the PC and laptop value chain? Currently netbooks are <span style="color: navy;"><u><a class="western" href="http://news.morningstar.com/articlenet/article.aspx?id=280408">growing the PC pie</a></u></span> but in the longer term US$250 netbooks may cannibalize US$1000 laptop sales at great cost to Intel and Microsoft. Netbooks are well-suited to cloud computing so netbook + cloud may replace laptop the Microsoft software. Google, salesforce.com, Amazon and other software already works well from the cloud while Microsoft Word does not so well-suited. The $250 vs. $1,000 price difference will drive change.</div>
</li>
<li><div style="margin-left: 0px; margin-right: 0px;">
Possible <span style="color: navy;"><u><a class="western" href="http://blogs.zdnet.com/open-source/?p=3643">convergence</a></u></span> of smart phones and netbooks.</div>
</li>
<li><div style="margin-left: 0px; margin-right: 0px;">
Processor wars. Intel Atom, Qualcomm Snapdragon, <span style="color: navy;"><u><a class="western" href="http://uk.reuters.com/article/technologyNewsMolt/idUKTRE51G00620090217?pageNumber=2&virtualBrandChannel=0&sp=true">Freescale</a></u></span>’s ARM based chipset.</div>
</li>
</ol>Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com2Cacique St, Santa Barbara, CA 93103, USA34.421094 -119.674934.416669 -119.68219549999999 34.425518999999994 -119.6676045tag:blogger.com,1999:blog-8720550037867345187.post-85643918003406524022009-05-19T10:13:00.003+10:002009-05-23T07:54:57.065+10:00Trying Out Wolfram Alpha<a href="http://www.wolframalpha.com/">Wolfram Alpha</a>, the new <a href="http://blog.wolfram.com/2009/03/05/wolframalpha-is-coming/">computational knowledge engine</a> from Wolfram Research has been getting a <a href="http://www.google.com/search?q=Wolfram%20Alpha,%20the%20new%20computational%20knowledge%20engine">good run</a> in the press this week so I decided to try it out.<br />
<br />
It gave pretty good answers for most of the following queries. Try them out yourself.<br />
<ul>
<li><a href="http://www44.wolframalpha.com/input/?i=2,+4,+8,+16,+32">2, 4, 8, 16, 32</a><br /></li>
<li><a href="http://www11.wolframalpha.com/input/?i=x+x+%2B+y+y+%3C+1+%26%26+x+%3C+0">x x + y y < 1 && x < 0</a> <a href="http://www11.wolframalpha.com/input/?i=x+x+%2B+y+y+%3C+1+%26%26+x+%3C+0"><br /></a></li>
<li><a href="http://www13.wolframalpha.com/input/?i=sin(x)%3D(x%2B.5)*(.5-x)%2B0.5">Sin[x] == (x + 0.5) (0.5 - x) + 0.5</a><br /></li>
<li><a href="http://www13.wolframalpha.com/input/?i=linear+fit+{{1.3,+2.2},+{2.1,+5.8},+{3.7,+10.2},+{4.2,+11.8},{1.3,+2.5},+{2.1,+2.8},+{3.7,+13.2},+{4.2,+11.8}}">linear fit {{1.3, 2.2}, {2.1, 5.8}, {3.7, 10.2}, {4.2, 11.8},{1.3, 2.5}, {2.1, 2.8}, {3.7, 13.2}, {4.2, 11.8}}</a></li>
<li><a href="http://www.wolframalpha.com/input/?i=eigenvalues+%7B%7B4%2C1%2C2%2C2%7D%2C%7B2%2C-1%2C2%2C2%7D%2C%7B4.1%2C1%2C2%2C2%7D%2C%7B2%2C-1%2C2%2C2%7D%7D">eigenvalues {{4,1,2,2},{2,-1,2,2},{4.1,1,2,2},{2,-1,2,2}}</a><br /></li>
<li><a href="http://www.wolframalpha.com/input/?i=svd+%7B%7B1%2C2%7D%2C%7B2%2C4%7D%2C%7B3%2C6%7D%2C%7B4.0001%2C8.001%7D%7D">svd {{1,2},{2,4},{3,6},{4.0001,8.001}}</a><br /></li>
<li><a href="http://www51.wolframalpha.com/input/?i=google+vs.+yahoo">google vs.yahoo</a><br /></li>
<li><a href="http://www44.wolframalpha.com/input/?i=lung+cancer">lung cancer</a><br /></li>
<li><a href="http://www44.wolframalpha.com/input/?i=bovine+tuberculosis+rate+New+Zealand">bovine tuberculosis rate New Zealand</a><br /></li>
<li><a href="http://www51.wolframalpha.com/input/?i=GDP+per+capita+vs+life+expectancy">GDP per capita vs life expectancy</a><br /></li>
<li><a href="http://www51.wolframalpha.com/input/?i=number+of+moose+in+Canada">number of moose in Canada</a><br /></li>
<li><a href="http://www.wolframalpha.com/input/?i=infinity+%2B+1">infinity + 1</a><br /></li>
<li><a href="http://www.wolframalpha.com/input/?i=infinity+^+infinity">infinity ^ infinity</a><br /></li>
<li><a href="http://www.wolframalpha.com/input/?i=How+long+does+it+take+to+count+to+infinity%3F">How long does it take to count to infinity?</a><br /></li>
</ul>
<object height="385" width="480"><param name="movie" value="http://www.youtube.com/v/5TIOH80Qg7Q&hl=en&fs=1">
</param>
<param name="allowFullScreen" value="true">
</param>
<param name="allowscriptaccess" value="always">
</param>
<embed src="http://www.youtube.com/v/5TIOH80Qg7Q&hl=en&fs=1" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" width="480" height="385"></embed></object>Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com268 Carlton St, Carlton Victoria 3053, Australia-37.800785 144.971747-37.8050235 144.9644515 -37.7965465 144.9790425tag:blogger.com,1999:blog-8720550037867345187.post-75349118748774122062009-05-05T11:46:00.000+10:002009-05-05T11:46:46.261+10:00Software Engineers were on the Right Track It is often sad when software engineers are shown to be correct. One of my first posts was on <a href="http://peterwilliams97.blogspot.com/2008/02/monetizing-social-search.html">monetizing social search</a><br />
<blockquote>
This <a href="http://cbs5.com/local/Internet.gang.recruiting.2.648038.html">innovative use of social networking<img class="snap_preview_icon" id="snap_com_shot_link_icon" src="http://i.ixnp.com/images/v3.79/t.gif" style="background-color: transparent; background-image: url(http://i.ixnp.com/images/v3.79/theme/silver/palette.gif); background-position: -1128px 0px; background-repeat: no-repeat; border-bottom-width: 0px; border-color: initial; border-color: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-style: initial; border-top-width: 0px; display: inline; float: none; height: 12px; left: auto; margin-bottom: 0px !important; margin-left: 0px !important; margin-right: 0px !important; margin-top: 0px !important; max-height: 2000px; max-width: 2000px; min-height: 0px; min-width: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 1px; position: static; top: auto; vertical-align: top; visibility: visible; width: 14px;" /></a> could have a short-term payoff. It turns out to be one of the few successes to date in <a href="http://publishing2.com/2007/07/12/facebook-monetization-lessons-from-google/">making money from social networking<img class="snap_preview_icon" id="snap_com_shot_link_icon" src="http://i.ixnp.com/images/v3.79/t.gif" style="background-color: transparent; background-image: url(http://i.ixnp.com/images/v3.79/theme/silver/palette.gif); background-position: -1128px 0px; background-repeat: no-repeat; border-bottom-width: 0px; border-color: initial; border-color: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-style: initial; border-top-width: 0px; display: inline; float: none; height: 12px; left: auto; margin-bottom: 0px !important; margin-left: 0px !important; margin-right: 0px !important; margin-top: 0px !important; max-height: 2000px; max-width: 2000px; min-height: 0px; min-width: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 1px; position: static; top: auto; vertical-align: top; visibility: visible; width: 14px;" /></a> When I ran a straw poll on making money from social networking with some software engineers, wannabe entrepeneurs and friends, they all came up with essentially the same idea: <em>Mining the users' personal details and finding some way to make them pay to keep this information confidential</em><span style="font-style: italic;">. </span></blockquote>
<br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">
My panel of experts shared the cynicism of the <a href="http://ping.fm/tavDt">sysadmins</a> who say</div>
<blockquote>
NEVER anthropomorphize lusers.</blockquote>
<br />
<a href="http://www.crunchbase.com/company/intelius">Intellius</a> 's recent acquisition of <a href="http://www.spock.com/">Spock</a> appears to have proven my panel of experts were on the right path.<br />
<br />
<a href="http://ping.fm/38Y1P">Michael Arrington</a> warned last week<br />
<blockquote>
But one company may have bitten and are close to buying the company. Sources are saying that the infamous Intelius (founded by the equally infamous <a href="http://www.crunchbase.com/person/naveen-k-jain">Naveen Jain</a> ), a people search engine that charges users to access data, may be buying Spock soon. If these rumors are accurate, God help Spock. Not only is Intelius embroiled in all kinds of legal and ethical disputes, but they also have a shaky history when it comes to acquisitions.</blockquote>
Then <a href="http://ping.fm/A1GIS">Ajit Jaokar</a> summed up<br />
<blockquote>
So, let me see if I get this right</blockquote>
<blockquote>
a) Spock trawls the web looking for our data</blockquote>
<blockquote>
b) It creates a profile about us in their site without approval</blockquote>
<blockquote>
c) It encourages us to enrich that information</blockquote>
<blockquote>
d) It charges us to access our own information</blockquote>
<blockquote>
e) And ultimately .. it sells that same information to a background check company ..</blockquote>
<blockquote>
For instance, my <a href="http://www.spock.com/q/Ajit-Jaokar">'harnessed' profile</a> (i.e. I did not create it) says .. <em>Ajit Jaokar is an Indian-born British author and Web 2.0 specialist. He is the founder and CEO of the publishing company Futuretext. He is also the... </em>and the rest you have to pay for :) (Don't bother .. that information is freely available on my blog .. and lots more .. so you can hopefully make your own judgements about me .. )</blockquote>
<blockquote>
Can I delete my own information in Spock?.. Now it gets MUCH more interesting .. <a href="http://www.spock.com/do/pages/help#claim-search-result"><span style="color: #006600; text-decoration: none;">Spock says on deletion of information</span></a> ..</blockquote>
<blockquote>
I<em><strong><span style="font-style: normal; font-weight: normal;">f you'd like to remove yourself from Spock, please read the following information and click the link below.</span></strong></em></blockquote>
<em></em><br />
<em></em><br />
<em></em><br />
<em><blockquote>
Before requesting removal, please make sure the original source of the information Spock found for you has been removed or made private (MySpace, blog, Friendster, etc). This will prevent you from being re-indexed on the site. Please note that you can only request removal for your Spock search result.</blockquote>
<strong></strong><br />
<strong><blockquote>
When filling out your information please make sure to include your name, e-mail, a link to your Spock Search Result (i.e. http://www.spock.com/Tiger-Woods), and the reason why you'd like to be removed. The Spock Support Team will review your claim and get back to you within 24-48 hours.</blockquote>
</strong></em><br />
<blockquote>
So, I have to ensure that the Original sources of information that they got the profiles from should also be made private(i.e. my blog, my facebook profile etc etc) ..(else they will 'harness' me again!)</blockquote>
<blockquote>
<br /></blockquote>Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com0Cairns QLD, Australia-16.925397 145.775178-16.935661 145.76058700000002 -16.915133 145.789769tag:blogger.com,1999:blog-8720550037867345187.post-23586164016969356522009-04-26T21:28:00.003+10:002010-09-02T20:54:20.448+10:00The Future of ICTIt is worth thinking about the future from time to time. It helps us craft investment strategies and career paths that match the major trends in the world. So what is the future of <a href="http://en.wikipedia.org/wiki/Information_technology">ICT</a>?<br />
<div>
</div>
<div>
My guesses are</div>
<div>
<ol>
<li>Simplification.</li>
<li>Movement to the cloud.</li>
<li>Fixed/mobile convergence.</li>
<li>Integration of simple cloud services. (1+2)</li>
</ol>
<div style="float: left; margin-bottom: 15px; margin-right: 25px; text-align: center;">
<a href="http://www.flickr.com/photos/sybrenstuvel/2468506922/" title="photo sharing"><img alt="threesixtyfive | day 244" src="http://farm3.static.flickr.com/2410/2468506922_c1ed495959_t.jpg" /></a></div>
<div>
Modern ICT systems are insanely complex while the most productive computer users I know all use simple tools. </div>
<div>
</div>
<div>
Most of the things we do we with computers are much simpler than the popular packages are capable of. e.g. Editing some text does not require a full blown desktop publishing program like <a href="http://en.wikipedia.org/wiki/Microsoft_Word">MS Word</a>, yet MS Word is the most popular text editor in the world. Likewise keeping track of some customers and inventory does not require a gigantic package like <a href="http://en.wikipedia.org/wiki/SAP_ERP">SAP</a>, yet SAP is the biggest selling ERP software package in the world.</div>
<div>
</div>
<div style="float: left; margin-bottom: 15px; margin-right: 35px; text-align: center;">
<a href="http://www.flickr.com/photos/jampa/8594608/" title="photo sharing"><img alt="Modern Times" src="http://farm1.static.flickr.com/6/8594608_57ffec8663_t.jpg" /></a></div>
<div>
The costs of learning these immensely complex packages are considerable in terms on time lost. There is probably a much higher cost in working as slaves to these packages which distracts from finding the best solutions to an enterprise's problems. A current trend in corporate ICT is to use "best of breed" packages with the minimum possible customization because the payback from customizing is much less than the cost. (BTW, this <a href="http://www.bpic.co.uk/articles/bobvfis.htm">does not</a> seem to be true for ERP). This means that enterprises that enterprises are paying the cost of not solving their ICT problems as well as they could. This cost has to be a significant fraction of their ICT budgets.</div>
<div>
</div>
<div style="float: left; margin-bottom: 15px; margin-right: 15px; text-align: center;">
<a href="http://www.flickr.com/photos/erpcenter/3285350910/" title="photo sharing"><img alt="What is ERP anyway? (MS&T ERP Center, 01/29/2009)" src="http://farm4.static.flickr.com/3366/3285350910_0e2bc76ba7_t.jpg" /></a></div>
<div>
This <a href="http://en.wikipedia.org/wiki/SAP_ERP#Deployment_and_maintenance_costs">article</a> explains why "best of breed" software packages sell well. It boils down to the promise of lower <a href="http://en.wikipedia.org/wiki/Total_cost_of_ownership">total cost of ownership</a> (TCO) through using a single vendor for all services and a mega-brand that makes buyers feel safe. </div>
<blockquote>
SAP ERP systems effectively implemented can have huge cost benefits. Integration is the key in this process. "<b>Generally, a company's level of data integration is highest when the company uses one vendor to supply all of its modules</b>." An out-of-box software package has some level of integration but it depends on the expertise of the company to install the system and how the package allows the users to integrate the different modules.</blockquote>
<div>
<span style="font-size: large;">Movement to the Cloud</span></div>
<div>
Centralized computing is much more efficient than desktop-centric computing. TCO decreases dramatically with centralization because maintaining and upgrading software running in one physical location is far easier than on many people's personal computers. <br />
Most client software, even computationally intensive software like high quality graphics, has very low duty cycles. It does nothing most of the time. When you buy expensive PC hardware to support it, you are paying to support peak usage, the few minutes per day when it does the tricky computations and you want the user interface to be responsive. The average computer resource usage of this software is very low, much less than 10%. Therefore running the software on a central server is much more the 10x more efficient.<br />
Expensive infrastructure such as high-reliability disk storage does not need to be replicated through an organization. Virtualization, SaaS etc only became effective in the last few years so many software and hardware vendors built their (then efficient) businesses around powerful client PCs running software locally.<br />
<br />
<span style="font-size: large;">Fixed Mobile Convergence </span><br />
<div style="float: left; margin-bottom: 15px; margin-right: 15px; text-align: center;">
<a href="http://www.flickr.com/photos/theleetgeeks/3404790032/" title="photo sharing"><img alt="Skype Crashing on iPhone Fix" src="http://farm4.static.flickr.com/3661/3404790032_a47efd3e14_t.jpg" /></a></div>
When simple applications and cloud computing become dominant, the requirements for terminals become much less. Smart phones and 3G netbooks are already very capable and are becoming more so. They also use little power and are portable.<br />
<br />
The next level of usability is to have one device for fixed and mobile work. That device should be able to work with WiFi and 3G networks and move seamlessly between them. The technology for this is maturing. <br />
<br />
From <a href="http://en.wikipedia.org/wiki/Technological_convergence#Fixed-mobile_convergence">Wikipedia</a><br />
<blockquote>
A clear trend is emerging in the form of fixed and mobile telephony convergence (FMC). The aim is to provide both services with a single phone, which could switch between networks ad hoc. Several industry standardisation activities have been completed in this area such as the Voice call continuity (VCC) specifications defined by the 3GPP.
Typically, these services rely on Dual Mode Handsets, where the customers' mobile terminal can support both the wide-area (cellular) access and the local-area technology (for VoIP). However, an alternative approach achieves FMC over 3G mobile networks - eliminating the requirement for Dual Mode. This approach, broadly termed cellular FMC, is in trials by telecoms operators including BT.</blockquote>
<blockquote>
An alternative approach to achieve similar benefits is that of <a href="http://en.wikipedia.org/wiki/Femtocell"><span style="color: black; text-decoration: none;">femtocells</span></a> .</blockquote>
<span style="font-size: large;">Integration of simple cloud services</span><br />
When cloud computing is widespread and simple cloud services are widely available, <a href="http://en.wikipedia.org/wiki/Integration">integration </a> companies will be able to assemble tools to meet the needs of businesses. This should be a vast business since it <a href="http://www.builderau.com.au/strategy/businessmanagement/soa/Disruptions-in-the-software-fabric/0,339028271,339158080,00.htm?feed=pt_siebel">competes</a> with the mega-apps Microsoft, Oracle, SAP, Siebel etc and the mega-glue <a href="http://www.tibco.com/">Tibco</a> etc.<br />
<br />
If the recent history of software development is a guide, nimble companies will start to build effective suites and grow rapidly to form a foundation for this industry, then they will be followed by specialist companies who will take care to make their software inter-operable. This will evolve into a <a href="http://en.wikipedia.org/wiki/Software_ecosystem">software ecosystem</a> and sales channels will emerge. With fixed mobile convergence in the mix, <a href="http://en.wikipedia.org/wiki/App_Store">application stores</a> may be used for sales, removing the need for sales and marketing teams in the startup companies that start this new business category.<br />
<br />
At this time the setups of hyper-productive software users will be easy to replicate in the cloud. User applications will be available to users as simple serices on simple devices like 3G netbooks. Enterprise applications will run in the cloud with simple interfaces. Business outsourcing will be simple because the software will run in the cloud with well-defined APIs.<br />
<br />
<span style="font-size: large;">The Consequences</span><br />
<ol>
<li>These changes will result in a dramatic increase in productivity that will boost economies world-wide.</li>
<li>Software will be simple so ICT staff will not be slaves to the machines of gigantic software packages.</li>
<li>This will free up ICT staff's time to add business value which will increase productivity even more.</li>
</ol>
<span style="font-size: x-small;"><b>Photo Credits</b></span><br />
<a href="http://www.flickr.com/photos/sybrenstuvel/2468506922/"><span style="font-size: x-small;">threesixtyfive | day 244 </span></a><span style="font-size: x-small;">by </span><a href="http://www.flickr.com/people/sybrenstuvel/"><span style="font-size: x-small;">Sybren A. Stüvel</span></a><span style="font-size: x-small;">.</span><br />
<a href="http://www.flickr.com/photos/erpcenter/3285350910/"><span style="font-size: x-small;">What is ERP anyway? (MS&T ERP Center, 01/29/2009)</span></a><span style="font-size: x-small;"> by </span><a href="http://www.flickr.com/people/erpcenter/"><span style="font-size: x-small;">MS&T Center for ERP</span></a><span style="font-size: x-small;">.</span><br />
<a href="http://www.flickr.com/photos/theleetgeeks/3404790032/"><span style="font-size: x-small;">Skype Crashing on iPhone Fix b</span></a><span style="font-size: x-small;">y </span><a href="http://www.flickr.com/people/theleetgeeks/"><span style="font-size: x-small;">theleetgeeks</span></a><span style="font-size: x-small;">.</span><br />
<a href="http://www.flickr.com/photos/jampa/8594608/"><span style="font-size: x-small;">Modern Times</span></a><span style="font-size: x-small;"> by </span><a href="http://www.flickr.com/people/jampa/"><span style="font-size: x-small;">jampa</span></a><span style="font-size: x-small;">.</span></div>
</div>Peter Williamshttp://www.blogger.com/profile/09350055705252093886noreply@blogger.com1Burke Rd & Toorak Rd, Camberwell VIC 3123, Australia-37.845707 145.053988-37.849942999999996 145.0466925 -37.841471 145.0612835