tag:blogger.com,1999:blog-69275582024-03-08T07:38:20.966+00:00BiofractalLife is the non-random replication of randomly varying replicatorsbiofractalhttp://www.blogger.com/profile/11229842750489025189noreply@blogger.comBlogger17125tag:blogger.com,1999:blog-6927558.post-20321764349368965132013-07-03T21:38:00.000+01:002013-11-25T16:40:35.129+00:00Electro-Flock: Modelling Flocks using Simple Electro-Magnetism<br />
<h2>
Flocks are easy to model if you use physics instead of biology</h2>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-8FOjj89yfxQ/UdAy6MKfq-I/AAAAAAAAIUA/WwI7vV_avWs/s457/electro-flock-1.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="253" src="http://2.bp.blogspot.com/-8FOjj89yfxQ/UdAy6MKfq-I/AAAAAAAAIUA/WwI7vV_avWs/s320/electro-flock-1.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Look - Its all done with magnets!</td></tr>
</tbody></table>
<b>Electro-Flock</b> is a simple flocking algorithm that relies only on <a href="https://en.wikipedia.org/wiki/Coulomb's_law">Coulomb's Law</a>, the single physical equation that describes how electrically charged particles interact. The project uses CoffeeScript, JQuery, HTML5 and Processing.js.<br />
<br />
I started this project with the feeling that flocking algorithms were too specific. I wanted to see if I could write a really simply algorithm that didn't know it was meant to be a flocking algorithm. I had a feeling that electro-magnetism might just be the key to a really fundamental, self-organising flock. Was this feeling justified? - Judge for yourself <a href="http://fdg.staging.goodpractice.net/flock.html" target="_blank">Play with the Electro-Flock Demo</a><br />
<br />
<a name='more'></a><br />
<b><span style="color: #f1c232;">What did you use to build Electro-Flock? </span></b><br />
For a complete description of the technology stack please see the prequel post:<br />
<ul>
<li><a href="http://biofractal.blogspot.co.uk/2013/05/self-organising-fun-force-directed.html" target="_blank">Self Organising Fun : A Force Directed Graph in CoffeeScript</a></li>
</ul>
<br />
<b><span style="color: #f1c232;">Where can I see the demo and get the code?</span></b><br />
<ul>
<li><a href="http://fdg.staging.goodpractice.net/flock.html" target="_blank">Play with the Electro-Flock Demo</a> </li>
<li><a href="https://github.com/biofractal/force-directed-graph" target="_blank">Get all the code from GitHub</a></li>
</ul>
<b><span style="color: #6aa84f;"><br /></span></b>
<b><span style="color: #f1c232;">Not another Flocking Algorithm surely?</span></b><br />
<div class="separator" style="clear: both; text-align: center;">
</div>
It has been done before, many times, but something has always bothered me about the fluffy view of flocking. Being a firm advocate of evolutionary-biology I imagine Mother Nature as a cruel and selfish parent.<br />
<br />
I think that <i>apparent </i>co-operation is most often an illusion created by constrained individualism, in other words individuals acting to serve their own goals in competition with others often end up acting in ways that just happen to <i>look like</i> they are co-operating and coordinating. In fact I had a feeling that the last thing most flock members would want to do is be part of a flock, or worse, part of a bait-ball! (see below for a definition and picture of a bait-ball)<br />
<br />
All this pontificating finally led me to two interesting questions:<br />
<ol>
<li>What would happen if I were to create a selfish bot that actively spurns flocking, a bot that attempts to stay as far as possible from all other bots?</li>
<li>What would happen if a collection of these selfish bots, an <i>anti-flock</i>, started to pursue their own selfish goals in competition with other identical bots?</li>
</ol>
The answer, as you can see from the <a href="http://fdg.staging.goodpractice.net/flock.html" target="_blank">demo</a>, turns out to be amazing. Despite themselves the collection of entirely selfish individual bots magically cohere to form beautiful, fluid, dynamic flocks and tight little bait-balls.<br />
<br />
<b><span style="color: #f1c232;">Does it really need magic though?</span></b><br />
Happily this <i>anti-flock</i> effect is simple to achieve without the aid of magic. Instead it relies on the single, prosaic principle of <i>electro-magnetism</i>. In fact the entire demo consists of nothing more than a combination of various types of magnet - yet the results are complex, very realistic and entirely <i>emergent</i>.<br />
<br />
<b><span style="color: #f1c232;">How do you make an anti-flock out of magnets?</span></b><br />
The rest of this post will try to explain why anti-flocks work but before delving into any more theory you should definitely play with the <a href="http://fdg.staging.goodpractice.net/flock.html" target="_blank">demo</a>. And remember - all the code is available from <a href="https://github.com/biofractal/force-directed-graph" target="_blank">GitHub</a> so you can see how to do it for yourself.<br />
<div>
<div>
<ul>
</ul>
</div>
<div>
<h3>
The Electro-Flock Demo - A Quick Start Guide</h3>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-wuWBbf1mQIc/UdA4DzeMvDI/AAAAAAAAIUQ/Jmxvrv3j-Yo/s441/electro-flock-2.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://4.bp.blogspot.com/-wuWBbf1mQIc/UdA4DzeMvDI/AAAAAAAAIUQ/Jmxvrv3j-Yo/s1600/electro-flock-2.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The top tool-bar</td></tr>
</tbody></table>
<br />
The top tool-bar allows you to set up your flocking environment. Choose the number of bots and, crucially, the number of attractors they will be chasing (see below for an explanation of why the attractors are important). You can also add some obstacles for the bots to avoid. All these things are just different types of magnets.<br />
<br /></div>
<div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-L-Sizeo4MSU/UdBYIMQxLuI/AAAAAAAAIWI/KX29JnVAWTI/s382/electro-flock-3.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="40" src="http://1.bp.blogspot.com/-L-Sizeo4MSU/UdBYIMQxLuI/AAAAAAAAIWI/KX29JnVAWTI/s320/electro-flock-3.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The bottom tool-bar</td></tr>
</tbody></table>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
The bottom tool-bar allows you to switch off both the attractors and the obstacles. Switching off the attractors allows you to see the bots in their natural <i>anti-flock</i> state where they try their best to avoid flocking (see below for a picture of this). You can also reset the obstacles to mix things up. The <i>Help </i>link brings you back to this page.<br />
<h3>
Definitions and Discussion</h3>
<b><span style="color: #f1c232;">A Flock</span></b><br />
Normally a flock is defined as a number of birds that are feeding, resting or travelling together. I would say that a flock is a collection of individuals determined to achieve their own goals whilst avoiding others that are attempting to do the same.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-jLqtwCYvpgA/UdA8VvmiSHI/AAAAAAAAIUw/3Eh85JV7R6o/s800/flock1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="262" src="http://4.bp.blogspot.com/-jLqtwCYvpgA/UdA8VvmiSHI/AAAAAAAAIUw/3Eh85JV7R6o/s320/flock1.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A flock of birds</td></tr>
</tbody></table>
<br />
<b><span style="color: #f1c232;">A Bait-Ball</span></b><br />
A bait-ball occurs when small fish swarm in a tightly packed spherical formation about a common centre. It is commonly described as a <i>last-ditch defensive measure adopted by small schooling fish when they are threatened by predators. </i>I say this is wrong and that a bait-ball occurs when individual fish all selfishly attempt to get as far away as possible from a common set of predators.<br />
<br /></div>
<div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-WJC51hmNPII/UdA-R0aJ0vI/AAAAAAAAIVA/aW0OEce1jmw/s1600/BaitBall.jpeg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="180" src="http://2.bp.blogspot.com/-WJC51hmNPII/UdA-R0aJ0vI/AAAAAAAAIVA/aW0OEce1jmw/s320/BaitBall.jpeg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The more predators, the tighter the ball</td></tr>
</tbody></table>
</div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-31vLB90vIM0/UpN88DjUuXI/AAAAAAAAIbk/meSo-ZE_2Co/s1600/fish-shark-1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="http://2.bp.blogspot.com/-31vLB90vIM0/UpN88DjUuXI/AAAAAAAAIbk/meSo-ZE_2Co/s320/fish-shark-1.jpg" width="287" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A constrained, two dimensional bait-ball</td></tr>
</tbody></table>
<div>
<br />
<b><span style="color: #f1c232;">What is your Point?</span></b></div>
<div>
Simple really. A flock is not a coordinated thing. A bait-ball is not a cooperative thing. These phenomenon emerge coincidentally whenever a collection of genetically similar actors find themselves selfishly pursuing their individual goals in the same location. Birds and fish are individualists, they are not members of a flock. The flock is simply a <i>coincidence </i>of time and space.<br />
<br />
Yes, birds of a feather do flock together but not because they want to. Instead they flock simply because they all want <i>the same things at the same time</i> and so end up trying to occupy the same coordinates in space-time.<br />
<br />
Clearly two things cannot occupy the same space-time coordinates so the individuals just do the best they can whilst selfishly optimising their individual trajectories - each chasing down whatever is attracting them or evading whatever is repulsing them whilst keeping the maximum distance from each other.<br />
<br />
Therefore the effect that we pattern-seeking humans see as a flock or bait-ball does not require any form of co-operation, coordination or communication. Instead it is simply the result of a group of individuals all trying to be in exactly the same place at the same time but <i>failing</i>. The only thing keeping the flock from imploding in a flurry of feathers or fish-scales is the mutual repulsion the individuals have for each other.<br />
<br />
This is the opposite of the usual way we imagine flocks. <br />
<br />
We normally think that the flock has some property of <i>coherence</i>, something that holds the flock together, but that is wrong. The flock is always trying to explode apart, the flock members want to be as far apart from each other as they can be, but since they are effectively identical (genetically) then they are all attracted and repulsed to the same degree by the same things.<br />
<br />
This creates two opposing forces:<br />
<ol>
<li>The repulsion all bots feel for all other bots </li>
<li>The shared attraction bots feel to the things in the environment</li>
</ol>
And the critical, dynamic balance point between these two opposing forces is a flock.<br />
<br />
<b><span style="color: #f1c232;">Um .. OK. So why all this talk of magnets?</span></b><br />
If I am correct then I should be able to model this all really easily. On the one hand I say that each bot should be repulsed by all other bots. On the other hand each bot should be attracted by elements of their environment. For a bird-flock that might be a<i> positive attraction</i> to a number of suitable roosting spots, for fish-bait-ball that might be a <i>negative attraction</i> to a number of predators.<br />
<br />
All this attraction, both positive and negative, can be easily modelled using magnets, or more specifically using <a href="https://en.wikipedia.org/wiki/Coulomb's_law" target="_blank">Coulomb's Law</a> which is a law of physics that describes the electrostatic interaction between electrically charged particles.<br />
<br />
<b><span style="color: #f1c232;">Bots are magnets - right. So what happens when you try it out?</span></b><br />
Unsurprisingly if you model the bots as magnets that simply repel each other you end up with a pure <i>anti-flock</i>, something like this:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-ppxIFu_xL08/UdBGjgbZQQI/AAAAAAAAIVQ/veQsZpI8p98/s430/electro-flock-4.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="186" src="http://3.bp.blogspot.com/-ppxIFu_xL08/UdBGjgbZQQI/AAAAAAAAIVQ/veQsZpI8p98/s320/electro-flock-4.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The bots fly away from each other just as fast as they can</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-iTdWqppmFfg/UdBG9OnaFgI/AAAAAAAAIVY/lqhbVzBCHmU/s429/electro-flock-5.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="187" src="http://1.bp.blogspot.com/-iTdWqppmFfg/UdBG9OnaFgI/AAAAAAAAIVY/lqhbVzBCHmU/s320/electro-flock-5.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Until they are equally spaced as far as possible</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-08thyBW8H24/UdBHW-Qj6lI/AAAAAAAAIVg/QdE-vWF3dvc/s241/electro-flock-6.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://1.bp.blogspot.com/-08thyBW8H24/UdBHW-Qj6lI/AAAAAAAAIVg/QdE-vWF3dvc/s241/electro-flock-6.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">You can see this for yourself if you switch the attractors off in the <a href="http://fdg.staging.goodpractice.net/flock.html" target="_blank">demo</a></td></tr>
</tbody></table>
<br />
The next step is to add some more magnets called <i>attractors</i>. The attractors positively attract bots to themselves causing the bots to chase them. And wouldn't you know it, as soon as you add attractors into the mix something amazing happens - the anti-flock turns into a beautiful, flowing dynamic <i>flock </i>and the more attractors you add the tighter and more coherent the flock becomes.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-B-f_g1HeIkw/UdBjOyZ_XQI/AAAAAAAAIWY/YlKXttsiCqM/s431/electro-flock-7.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="240" src="http://2.bp.blogspot.com/-B-f_g1HeIkw/UdBjOyZ_XQI/AAAAAAAAIWY/YlKXttsiCqM/s320/electro-flock-7.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A loose flock created by two attractors</td></tr>
</tbody></table>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<i><br /></i>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-82AtpksonBU/UdBKyIHlMGI/AAAAAAAAIV4/IJGBV9cc1yw/s432/electro-flock-8.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="235" src="http://4.bp.blogspot.com/-82AtpksonBU/UdBKyIHlMGI/AAAAAAAAIV4/IJGBV9cc1yw/s320/electro-flock-8.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A tight bait-ball style flock created by twenty attractors</td></tr>
</tbody></table>
<i><br /></i>
<b><span style="color: #f1c232;">It works! That's great. But remind me - what is going on?</span></b><br />
Its so simple - bots are repelled by bots and attracted to attractors - that's it.<br />
<br />
This creates a system that seeks a balance point where the bots are as close as they can be to an attractor whilst being as far away as possible from each other. If the attractors were stationary then this balance point would be a static cloud of bots. However (and I didn't mention this before) the attractors are also repelled by the bots. So as the bots try to chase them down the attractors try to escape. Thus the balance point of the system is forever changing. Adding more than one attractor makes the balance point both dynamic and complex and the result is a beautiful flowing cloud of bots, in other words, a flock.<br />
<br />
<b><span style="color: #f1c232;">It looks like a flock sure but come on, birds and fish don't work like this do they?</span></b><br />
Don't they? Maybe we have been making our explanations of flocking more complex and specific than they need to be because we have been thinking too much like biologists. Thinking like a physicist, that is thinking in terms of<i> electrically charged particles</i> means we get the same (or better) results using a single incredibly simple formula - Coulomb's law. <a href="http://en.wikipedia.org/wiki/Occam's_razor" target="_blank">Occam's razor</a> surely applies.<br />
<br />
<b><span style="color: #f1c232;">Go on then, tell me what you really think.</span></b><br />
Thinking electrically then - birds flock not because they want to flock or try to flock but because each individual bird finds itself flying through a type of five dimensional <a href="http://en.wikipedia.org/wiki/Gauge_theory" target="_blank">Gauge Field</a>. That's right - five dimensions. There are the four dimensions of space-time plus a bird-perceived fifth dimension of <i>environmental attractiveness</i>. Its just like flying through an electro-magnetic field with a compass.<br />
<br />
At each point in space-time the bird experiences a greater or lesser degree of attraction to any given direction of flight and simply responds accordingly, like iron filings lining up in a magnetic field. The varying field strengths throughout four dimensional space-time are a product of everything in the bird's environment, for example the desirability of roosting spots, the presence of predators, the availability or otherwise of food sources, the wind direction - you name it, its an infinity of causes.<br />
<br />
Birds of the<i> same species</i> will perceive and react to their attractor-field in essentially the same way and so they will end up trying to be in the same place at the same time. Since this is impossible it implies that birds must avoid each other, in other words that birds are repulsed by each other. Thus we have the tension between the two forces of attraction and repulsion - and a flock is the dynamically balanced result.<br />
<br />
The same goes for a bait ball. The attractor field is in this case is <i>negative</i>. Instead of being positively attracted to elements in their environment a bait-ball is characterised by fish being repulsed by predators - but the direction of attraction does not matter as the effect is just the same. Lots of functionally identical fish all experiencing the same attractor-field all want to go to the same place at the same time but they can't because of all the other fish that are doing the exact same thing.<br />
<br />
The bait ball is a pure and therefore very clear demonstration of the attractor field effect. Individually each fish acutely senses the varying degrees of repulsion as it travels through points in attractor-space-time. This has the effect of aligning the fish to the underlying attractor field lines and, in so doing, maximising the distance between the fish and the predators. With enough predators arranged around them in three dimensional space the least repulsive location becomes a perfect singularity, a point in five dimensional attractor-space-time around which all the fish align themselves with a precision that is distorted only by the presence of the other fish.<br />
<br />
This bait-ball effect is exactly what we see happening with the <a href="http://fdg.staging.goodpractice.net/flock.html" target="_blank">demo</a> when we add lots of attractors.<br />
<br />
<b><span style="color: #f1c232;">I sense a summary...</span></b><br />
Don't think of a fluffy, co-operating flock. Instead think of electrically charged particles moving through an electro-magnetic field. Swap the word 'magnetism' for 'environmental attractiveness' (both positive and negative) and just use the standard formula as described by Coulomb. What you get is flocking!<br />
<br />
This strikes me as a beautiful and potentially useful simplification of the current descriptions of natural flocking behaviours and that, for me at least, serves to make this apparently complex but actually fundamentally simple phenomenon even more awe inspiring.</div>
</div>
biofractalhttp://www.blogger.com/profile/11229842750489025189noreply@blogger.com1tag:blogger.com,1999:blog-6927558.post-24790691101246063572013-05-04T14:52:00.001+01:002013-11-06T13:10:23.467+00:00Self Organising Fun : A Force Directed Graph in CoffeeScript<br />
<h2>
Emergent Self Organising Behaviour using CoffeeScript, JQuery and Processing.js</h2>
<b><span style="color: #6aa84f;">What is a Force Directed Graph and why bother spending time coding one in CoffeeScript? </span></b><br />
A Force Directed Graph is a collection of nodes and links that self-organises until its nodes are as far apart as possible and the links do not cross. They appeal because <i>self-organisation</i> is just so intrinsically fascinating and, more professionally, because the project allowed me to bring together a whole set of exciting web technologies including CoffeeScript, JQuery, HTML5 and Processing.js.<br />
<br />
Before reading any further why not <a href="http://fdg.staging.goodpractice.net/forcedirectedgraph.html" target="_blank">Play with the Force Directed Graph Demo</a><br />
<br />
Or if you really want to blow your mind why not take a look at the innovative and beautiful flocking system based solely on the principles of electromagnetism <a href="http://biofractal.blogspot.co.uk/2013/07/electro-flock-modelling-flocks-using.html" target="_blank">Electro-Flock: Modelling Flocks using Simple Electro-Magnetism</a>.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-J-J8-Tm1tHU/UYjGxVuiotI/AAAAAAAAH4U/gf_xGvoyRtQ/s1600/fdg_swarm-ordered.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://3.bp.blogspot.com/-J-J8-Tm1tHU/UYjGxVuiotI/AAAAAAAAH4U/gf_xGvoyRtQ/s1600/fdg_swarm-ordered.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">They did it all by themselves</td></tr>
</tbody></table>
<b><span style="color: #6aa84f;">How did it turn out? </span></b><br />
I started this project wondering if CoffeeScript would be worth learning and finished it vowing never again to write another line of naked JavaScript. I also found the combination of CoffeeScript and JQuery to be a powerful and elegant solution to the problem of coding against the browser. Finally, using the Processing.js library coupled to an Html5 canvas tag meant I could tap the GPU of the client's graphics card and so achieve the smooth visualisation of forces and vectors I wanted.<br />
<a name='more'></a><br />
<ul>
<li><a href="http://fdg.staging.goodpractice.net/forcedirectedgraph.html" target="_blank">Play with the Force Directed Graph Demo</a> </li>
<li><a href="https://github.com/biofractal/force-directed-graph" target="_blank">Get all the code from GitHub</a></li>
</ul>
<br />
For more information on Force Directed Graphs please see the Definitions and Discussion section below.<br />
<div>
<div>
<h3>
Technology Stack</h3>
<ul>
<li><a href="http://www.microsoft.com/visualstudio/eng/downloads#d-2012-express" target="_blank">Visual Studio 2012 Express</a><br />Free editor. No extensions supported</li>
<li><a href="http://www.microsoft.com/visualstudio/eng/downloads#d-2012-express" target="_blank">Jurassic Coffee</a><br />A coffescript compiler and linker. I run it as a build event.</li>
<li><a href="http://coffeescript.org/" target="_blank">CoffeeScript</a><br />The shorter, better version of javascript</li>
<li><a href="http://jquery.com/" target="_blank">JQuery</a><br />A javascript library for dealing with the browser DOM</li>
<li><a href="http://processingjs.org/" target="_blank">Processing.js</a><br />A javascript library for high performance 2D drawing onto an html5 Canvas</li>
<li><a href="http://twitter.github.io/bootstrap/" target="_blank">Bootstrap</a><br />A CSS library so I don't have to think about it.</li>
<li><a href="http://en.wikipedia.org/wiki/HTML5" target="_blank">html5</a><br />Its got the new Canvas element that allows for modern animation in the browser.</li>
</ul>
</div>
<div>
<h3>
The Demo</h3>
The <a href="http://fdg.staging.goodpractice.net/forcedirectedgraph.html" target="_blank">demo</a> allows you play with Force Directed Graphs of various sorts. Here is a quick run-down.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-jeg7k7CVHA8/UYP1a-o90gI/AAAAAAAAH4E/P2DAqLEPyHI/s1600/fdg_click-on-screen.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://1.bp.blogspot.com/-jeg7k7CVHA8/UYP1a-o90gI/AAAAAAAAH4E/P2DAqLEPyHI/s1600/fdg_click-on-screen.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">This is very important</td></tr>
</tbody></table>
The first thing - don't overlook clicking on the screen. This toggles a Mouse attractor on and off. The effect is dramatic as you convert a Force Directed Graph into a poor man's flocking algorithm. Its also handy for generally mixing things up.<br />
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-yOo-BFu43vY/UYPSbh5BSHI/AAAAAAAAH3E/DHzPIdtQhfM/s1600/fdg_top-menu.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="28" src="http://4.bp.blogspot.com/-yOo-BFu43vY/UYPSbh5BSHI/AAAAAAAAH3E/DHzPIdtQhfM/s320/fdg_top-menu.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The top tool-bar</td></tr>
</tbody></table>
<div>
The first drop-down determines the number of copies of each graph to generate. Each copy will repel the others so multiple copies can introduce some interesting, higher-order patterns.<br />
<br />
<a href="http://www.blogger.com/blogger.g?blogID=6927558"></a><a href="http://www.blogger.com/blogger.g?blogID=6927558"></a>The second drop-down determines the type of graph to be generated. Each graph type differs in the number of nodes, the interconnections between the nodes and the node mass. The physical constants that determine magnetic repulsion and spring strength remain the same for all.<br />
<br />
The third drop-down allows you to change the behaviour of a node as it encounters the edge of the drawing area.<br />
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-eRgAYaBGA0U/UYPSTHPCDsI/AAAAAAAAH28/eHw6JCP2wHw/s1600/fdg_bottom-menu.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="42" src="http://2.bp.blogspot.com/-eRgAYaBGA0U/UYPSTHPCDsI/AAAAAAAAH28/eHw6JCP2wHw/s320/fdg_bottom-menu.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The bottom tool-bar</td></tr>
</tbody></table>
<div>
The <i>Noise </i>drop-down allows you to introduce random noise into the mix. It is known that Force Directed Graphs will often beach themselves on the desolate shores of sub-optimality . These locally optimal solutions trap the pattern close to the globally optimal solution. Randomness can help fix that by jiggling the system to overcome the local optima and allow the system to keep falling towards perfection.<br />
<br />
The <i>Springs </i>check-box toggles the springs on and off. How do the nodes act when they are unfettered?<br />
<br />
The <i>Magnets </i>check-box toggles the magnetic repulsor fields on and off. <br />
<br />
The <i>Help </i>link brings you back to this page.<br />
<h3>
Resources</h3>
<br />
<ul>
<li><a href="http://natureofcode.com/book/chapter-1-vectors/" target="_blank">The Nature of Code by Daniel Shiffman</a><br />This was my number 1 resource. It is well written and engaging but most of all it tells you everything you need to know about vectors, forces, springs and magnets and how to code them up.</li>
<li><a href="http://www.brad-smith.info/blog/archives/129" target="_blank">A Force-Directed Diagram Layout Algorithm</a><br />Another force-directed example this time written in C#. I reckon I probably stole some code from here.</li>
<li><a href="http://processing.org/learning/topics/springs.html" target="_blank">Springs</a><br />An example of how to code various springs in Processing.js, the 2D drawing package I used.</li>
</ul>
<br />
<h3>
Definitions and Discussion</h3>
<b><span style="color: #6aa84f;">Graph</span></b><br />
A picture of an interconnected set of items where each item is represented by a node and each connection by a link. <br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://3.bp.blogspot.com/-UPCu0ckpyAk/UYPVocWVTgI/AAAAAAAAH34/HAZPEL1SCV8/s1600/fdg_graph.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://3.bp.blogspot.com/-UPCu0ckpyAk/UYPVocWVTgI/AAAAAAAAH34/HAZPEL1SCV8/s1600/fdg_graph.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A simple, static Graph</td></tr>
</tbody></table>
<br />
<b><span style="color: #6aa84f;">Force Directed Graph</span></b><br />
A graph that self-organises until its nodes are as far apart as possible and the links do not cross. The nodes repulse one another whereas the links act like springs.</div>
<div>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://4.bp.blogspot.com/-MnNALYeFr_U/UYPVoYkT_II/AAAAAAAAH30/cco8TUayDIE/s1600/fdg_blossom-ordered.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://4.bp.blogspot.com/-MnNALYeFr_U/UYPVoYkT_II/AAAAAAAAH30/cco8TUayDIE/s1600/fdg_blossom-ordered.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A dynamic Force Directed Graph achieves a state of cybernetic balance</td></tr>
</tbody></table>
<br />
<span style="color: #6aa84f;"><b>No Really, What is a Force Directed Graph?</b></span></div>
<div>
Imagine a single tennis ball connected to ten otherwise identical balls by equal lengths of string. How to disentangle this?<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-1uakWoSVKUk/UYPVpKKVrhI/AAAAAAAAH3o/O-YwhEZLQq8/s1600/fdg_spokes-tangled.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://2.bp.blogspot.com/-1uakWoSVKUk/UYPVpKKVrhI/AAAAAAAAH3o/O-YwhEZLQq8/s1600/fdg_spokes-tangled.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Triangular tennis balls?</td></tr>
</tbody></table>
<br />
You could take the brute force approach and sit cross-legged unpicking the knots and arranging the balls. Or you could just let the whole thing self-organise until no balls touch and no strings cross.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://1.bp.blogspot.com/-MMZVL8CXiC4/UYPVo6ttCkI/AAAAAAAAH3s/GVDkpzWYI1E/s1600/fdg_spokes-ordered.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://1.bp.blogspot.com/-MMZVL8CXiC4/UYPVo6ttCkI/AAAAAAAAH3s/GVDkpzWYI1E/s1600/fdg_spokes-ordered.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A globally optimal solution</td></tr>
</tbody></table>
<br />
You may think that a system of tennis balls and string cannot self-organise but that is not true. This system, like all physical systems, will organise itself until it reaches a low-energy state. It will stop self-organising the moment no energy-lowering moves remain, in other words, when every remaining move implies a rise in the energy state of the system.<br />
<br />
The problem: For tennis balls and string this final state is vastly more likely to be a tangled mess rather than a pleasing, symmetrical pattern. We <i>could </i>re-engineer the system using exotic materials but doing this in the real world would be impractical.<br />
<br />
The trick is to simulate a node-and-link system (a graph) so that its lowest energy state happens to resemble the pattern you want. Such a system should then untangle itself and, as a ball rolls downhill, self-organise toward the desired pattern.<br />
<br />
But we don’t want to just define a low energy state and have the system go there. That’s no fun. Instead we want to simulate a real system, albeit one with the impractical physical properties. It is the nature of these properties, their interactions, and the final dynamic balance point that determines the final pattern. Once modelled we should be able to just sit back and watch as the complex behaviour spontaneously emerges, powered only by the laws of pseudo-thermodynamics.<br />
<br />
To achieve this we must replace the tennis balls with imaginary <i>repulsor </i>magnets. The closer these magnets get to each other the greater the repulsive force they feel. The links are replaced with <i>springs</i>. As a link is stretched so it pulls with ever increasing force on the magnets at either end. All that remains is to tweak the strength of the springs and magnets until a nice, critical balance is reached.<br />
<br />
A system like this will seek a low energy state just as any real physical system would. The combination of the magnets repelling plus the springs attracting make the system <i>spontaneously </i>untangle and re-order. To stretch the analogy - the system is powered by virtual kinetic energy harvested from virtual potential energy as the system spontaneously tumbles down an imaginary energy gradient. The process stops when the system finally achieves the lowest energy state, which we have cunningly engineered to be both ordered and aesthetically pleasing.<br />
<br />
It’s not magic it but it sometimes looks that way. The fascination of watching Force Directed Graphs gracefully disentangle themselves may well stem from the fact that real-life systems overwhelmingly tend not be ordered at low energy states. Disordered states so vastly outnumber ordered states that our evolutionary expectations assume that any non-living system’s low energy state will be disordered, and this is usually correct. However it is not true for Force Directed Graphs and so, as they spontaneously transition from disorder to order they create a strong illusion of purposeful behaviour, sometimes they seem almost eerily life-like.<br />
<br />
A Force Directed Graph is therefore a computer representation of an edge-case physical system. It models a system of repulsor-nodes and springy-links with the final shape of the optimally ordered pattern being determined by the interconnections and the precise balance of the opposing forces of attraction and repulsion.<br />
<br />
And finally, don't forget to take a look at the exciting electromagnetic flocking system <a href="http://biofractal.blogspot.co.uk/2013/07/electro-flock-modelling-flocks-using.html" target="_blank">Electro-Flock: Modelling Flocks using Simple Electro-Magnetism</a>.</div>
</div>
biofractalhttp://www.blogger.com/profile/11229842750489025189noreply@blogger.com1tag:blogger.com,1999:blog-6927558.post-41383064217814454332012-09-10T12:34:00.004+01:002013-05-04T17:13:13.137+01:00Goodbye OpenRasta - Hello Nancy<br />
This is a painful post to write as I still feel a strong residual loyalty to <a href="http://openrasta.org/" target="_blank">OpenRasta</a> (OR) however I have now moved over the <a href="http://nancyfx.org/" target="_blank">Nancy </a>and so I would like to give you my reasons why. These are not technical reasons, far from it, instead I give you nothing more than sorrowful account of feelings hurt, a comparison between the brutal, distant and dominant love provided by OR versus the beautiful, seductive and submissive framework that is sweet Nancy.<br />
<br />
<a name='more'></a><br /><br />
To establish my bona fides: The pain I have experienced using OR cannot be easily dismissed, the problems were not simply due to a lack of familiarity or a sanguine welcome to the razor's edge of open-source development. I have now delivered two projects with OR as well as spending time (and money) with <a href="http://serialseb.blogspot.co.uk/" target="_blank">Sebastien Lambla</a> (Seb) taking his REST course down in London - and I think that's enough experience to get the gist of the framework, even if I am not exactly a glassy-eyed contributor.<br />
<br />
The final result, in my opinion, is this: OR was almost great, and may yet be great still, but it has been hamstrung by the side of the tracks that spawned it. It was not born of the grotesque, flabby, Microsoft world (that pays my bills), but sprang forth from the lean, hyper-cool loins of the command-line world where Linux is something other than a Charlie Brown character and eidetic memories for endless strings of command-line linear-B are <i>de-rigueur</i>.<br />
<br />
And I almost made it. I almost stopped being a child-coder with Microsoft chocolate smeared around my mouth, I almost became a ethically-focused, command-line vegan coder because OR, for a while, bridged those two worlds. But then Seb took his eye off the ball and, like Macbeth, the vaunting ambition that made him great now led to his downfall. And fall down he did, back into the world of black-windows with strange commands, the no-mouse, no GUI world, the <a href="http://en.wikipedia.org/wiki/Essenes" target="_blank">Essene </a>asceticism of keyboard short-cuts, and instead of continuing with OR, pushing hard to fix its early-years verbosity and birthing difficulties he wandered away to become an absent father.<br />
<br />
Perhaps this is how open-source is meant to work, perhaps it was now down to the acolytes to push the code forward, and for a long while I hoped so but I am not a source-code Github-jockey, I want something I can install and use. And yes, I know, I should be cool with incanting my own assemblies from magical hieroglyphic streams but my flesh is weak. I don't really care what a <b>make.bat</b> is - something to do with windows 3.1 if memory serves - which it doesn't usually.<br />
<br />
So when it came to being a player, a contributor, an open-source-head, I failed, and I wept as I failed, I raged against the purity of others as I failed, but I failed all the same. OR worked, yes it did, and compared to MVC (oh, I am not that bad come on, MVC, really ... anybody? ...) it was a revelation of RESTful cleanliness, but it did not work well enough, not for MS fan-boi coder like me, with my fat behind and nervous giggle, sweating in a wannabee black turtle-neck, stretched and lumpy, splattered with dribbles of full-fat Starbucks coffee. The marathon running dream receded with each month that went by without an official OpenRasta release, the vegan dream with each cheeseburger I crammed into my eggy-bearded mouth, the Linux dream when I realised I couldn't even master the miniature piano far less remember <span style="font-family: courier, lucidatypewriter, monospace; font-size: 15px;">du -s * | sort -nr</span><br />
<br />
Personally I think Seb got bored and needed a new challenge. OpenWrap (his package manager) was a brilliant idea but MS got there second with the inferior <a href="http://nuget.org/" target="_blank">Nuget</a> and, if Seb had not been so pure, he would have realised that his true calling was to follow through with his beautiful, but flawed, OR and thus raise himself as the RESTful saviour for soft-palmed coders like me who just don't get why icons are bad, coders who guiltily stuff their mouths with creamy Nuget packages because its so easy, coders who right-click on their projects and select 'Manage Nuget' even though they know that for some vague, undefined reason this is meant to be bad, bad, bad.<br />
<br />
Then Nancy sashayed into the room and one look was all it took. She was just so <i>easy</i>. Everything just worked (as it would have <i>just worked</i> in OR if Seb had not wandered back into the desert seeking further visions). The Nancy site looked great (as the OR site would have looked if such things had any interest to hard-muscled, flinty-eyed Linux coders). And now I love Nancy, I just can't help it, she is beautiful and she is getting better all the time - with regular updates to help smooth my brow and keep my underpants clean. She smiles with concerned understanding at my code-smells and quietly helps me to REST at night, she doesn't even seem to mind my guilty habits, my perverse but harmless need to update my project with her latest code, even as I cast a side-ways glance before furtively click-click-clicking my dirty little mouse.<br />
<br />
I know I am wrong, I know that OR and OpenWrap are probably better, but I also think that none of that matters for the majority of coders like me, who want to impress the boss with the sites they make, coders who try to care about the command line and silently weep tears of acid guilt because it <i>makes no sense</i>, coders who cannot resist unstopping their ears because they <i>want </i>to hear the sirens calling, calling us back to the colours and the pictures and the pretty-little-clicky-things, coders who will work with impure products just as long as it makes their grubby lives that little bit easier.<br />
<br />
I am sorry Seb, I really am. I think you are great, a visionary, but your eye wandered from the little people and I felt unloved. I am just a humble C# coder, I need my visual studio crutch because I love all those code colours and I am addicted to the soft whisperings of my metham-resharper. The command-line is a cold place to be and I need warm cuddles - I think you lost me when you forgot that.biofractalhttp://www.blogger.com/profile/11229842750489025189noreply@blogger.com7tag:blogger.com,1999:blog-6927558.post-44372743644692328862012-05-09T16:19:00.004+01:002013-05-04T17:13:36.560+01:00RavenDB or SQL Server? Which one should I use?SQL Server is not a database for storing anything except ad-hoc reporting data. It is splendid for that, ad-hoc reports, data-mining, real time relationship discovery - and its optimal for these uses because way back when Codd designed the rules, that is what it was designed for (see <a href="http://biofractal.blogspot.co.uk/2009/02/data-driven-conspiracy.html" target="_blank">The Data Driven Conspiracy</a> for more details)<br />
<div>
<br />
Of course you can store other types of data in a relational database. For example you can serialise your domain state to a SQL Server database and retrieve that state at a later date but this is a terrible use of a relational system. So bad, in fact, that whole layers of code (so called 'data-layers', or DALs), many thousands of lines, are required to make this kind of task remotely do-able, testable, maintainable.</div>
<div>
<br />
The great, decades long confidence trick has been for the SQL based retailers to convince us that there was no better way to store hierarchical / document data other than real-time conversion between radically different data-structures. DALs and the latest ORM have all been trumpeted but, in the end, these are nothing more than codecs to thunk your bits back and forth between different data organisations - different patterns on the disk.</div>
<div>
<br />
Madness!</div>
<div>
<br />
<a name='more'></a><br /><br />
So forget all the itsy-bitsy, angels-on-a-pinhead arguments about atomicity, consistency, isolation, durability and all that jazz. The elephant in the room is that if you are using SQL Server and you are not a pharmaceutical company that is dynamically mining data-cubes for hidden inferences within large populations (are you?) then you are using the wrong technology to store your data.<br />
<br />
If, on the other hand, you are a humble application developer - a coder who, like most of us, simply wishes to serialize domain-state in a way that does not mean learning a whole new set of ideologies and syntax then, for goodness sake, do not use a relational database. Just don't.<br />
<br />
RavenDB? I have been using this for a while (having evolved through DB4O and CouchBase) and I can say that it does what it says on the tin. It stores my stuff and gives me it back when I ask. I did not have to write any data-layers nor use a third party ORM. I did not have to learn a structured query language and I did not have to bolt on extras to make Full Text search work (RavenDB is based on Lucene so it all 'just works'). So far so good.<br />
<br />
But really, does it matter what you use as long as it is, for the task at hand, easy to code, fast and efficient? For normal application development RavenDB is all of these things whereas SQL Server is not. That is not to say that relational DBs are bad - let's not blame the victim - I fully acknowledge that for certain specialised developments you may indeed need something a little more ... exotic, such a relational datastore.<br />
<br />
But just in case you are still clinging to the wreckage, you need to ask yourself this: If we had all been using simple, fast, elegant datastores that closely fitted our needs as application developers, and then I wrote an opinion piece urging you to adopt SQL Server then you would either laugh at me or pity me, since that is the rational response. One thing is for sure - you would not swap to SQL Server. Thus we demonstrate that inertia alone is the guiding force that keeps SQL Server tacked into the mainstream. Inertia and ignorance. Inertia, ignorance and corporate greed ....</div>
biofractalhttp://www.blogger.com/profile/11229842750489025189noreply@blogger.com1tag:blogger.com,1999:blog-6927558.post-3694825355532072522012-05-09T14:03:00.000+01:002013-05-04T17:21:13.913+01:00Agile and Cost Breakdowns<br />
At the beginning of an Agile project it is often very tricky to provide a cost breakdown for individual items of work. This is because central to the Agile methodology is the demand that at the start of a project everybody admits they do not really know what details to expect! This is a brave thing to do. It is brave of a developer to admit that the implementation details of a proposed project are not clear. It is brave of a client to accept that they do not really know, in real-life detail, what it is they want. The pay back for all this honesty is a much better chance that the project will be useful, delivered on time and delivered within budget.<br />
<br />
<a name='more'></a><br /><br />
That said, many clients require the sort of detailed costing that Agile does not supply. In response it has become common to adopt the following approach: Each unit of work is assigned a number of points where the more points the more difficult the task is considered. This number encapsulates all the experience of the developers, their previous projects and their expectations for the current project, however it does not directly relate to cost. Instead, as the project unfolds so the developers keep careful track of the effort required to deliver each unit of work. This creates a relationship between the points and the actual, real-life cost of delivering those points for the current project under the current conditions. As the project continues the mapping between the abstract points and the concrete costs becomes ever more accurate and useful. This is in direct contrast to other project management techniques where, as the project unfolds, the predictions made at the start become ever more inaccurate and useless.<br />
<br />
This still does not answer the client’s needs however. They need a hard and fast prediction, an amount of cash per unit of work, a crystal ball that allows them to peer 6 months into the future and follow the convoluted, unpredictable, surprise strewn trajectory of a hand-made, complex software artefact that, as yet, nobody can truthfully define in detail. As unfair as it may seem, the business demands that we need to answer the following question: “How much will it cost to build something you don’t really understand, given that you don’t know how long it is going to take and given that what is possible and what is desired will certainly change along the way”. The answer has to be “I don’t know”, any other answer is a guess.<br />
<br />
Can we do better than guessing? Yes, a little. We know the total (estimated) number of points. From this we can generate a cost estimate, informed by experience, for a total cost expressed as a min->max range. A detailed breakdown of costs can now be derived (for what it is worth) by simply dividing the range values by the total number of points to give a min->max range per point. From this a cost value per unit of work can be estimated. For example:<br />
<br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">Total Points for Project = 100 </span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">Estimated Total Cost = 30K -> 45K </span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;">Estimated Cost per Point = [30K / 100] -> [45K / 100]</span><br />
<span style="font-family: 'Courier New', Courier, monospace; font-size: x-small;"> = <b>£300</b> -> <b>£500</b> per point</span><br />
<br />
This estimate will be wrong but, because this is an Agile project, as time goes by the estimated costs will become steadily more accurate. This is good, because most of the cost overruns on a traditional product come near the end. In contrast, the final stages of an Agile project are the least risky and the most productive. This is when everybody finally understands what the project is about, the client knows what is possible, the developer knows what is required and given this shared knowledge, both realise there is more that should or can be done – budget allowing.biofractalhttp://www.blogger.com/profile/11229842750489025189noreply@blogger.com2tag:blogger.com,1999:blog-6927558.post-84108865607078874382010-04-02T14:43:00.036+01:002013-05-04T17:21:42.586+01:00Predicting complex, dynamic systems or why Stalin, Hitler and stock market analysts have failed usThe ability to predict a future event lies at the heart of what it is to be human. A hunter predicting the presence of animals at a waterhole can drastically optimise the chances of a successful kill. Empathetically predicting a rival's or ally's actions can determine the outcome in hierarchical social competition. Predicting in advance the consequences of leaving a small fire unquenched can save the lives of your family. The need to predict the consequences of complex, dynamic systems has been a primary driver of human evolution, both biological and technical.<br />
<br />
What does it mean to predict something? It is a simple question with a relatively straightforward answer but its misapplication can have profound and disturbing consequences. Mistaking the systems that can or cannot be usefully predicted is a core political and financial flaw of the last 100 years that has lead to spectacularly negative consequences, from the rise of totalitarianism, both right and left, to financial bubbles whose froth still fizzes through our financial institutions today.<br />
<br />
<a name='more'></a><br /><br />
Predicting an event is easy - all you need is a model. Simply ensure the model's initial state resembles reality and run it for as long as you need. When the model stops its final state is your prediction. <br />
<br />
If making a "model whose initial state resembles reality" sounds a bit complicated then, fortunately for us, all humans come with some pretty good models right out of the box. As babies we all quickly boot up an hard-wired physics engine in our heads. As your hand collides with a glass of water and you see it tip over the edge of the table it is a matter of a few trillion, near instantaneous yet autonomous calculations for you to know, with crashing certainty, the fate of the glass. There may be the odd slightly surprising variation, the glass may not break, it may teeter then just recover from the edge of catastrophe, but as sure as apples it will not fall half way to the floor, stop and then float back to its original position. Philosophers may argue that you can't tell what the glass will do until you try but evolution makes the assumption that gravity is universal and so we have evolved an extremely powerful, ultra-fast physical modelling system that allows us to determine the likely environmental effect of a given cause. Without this internal modelling ability we would be sea anemones, which is perhaps a little cruel to anemones who, with their reflex withdrawal of their tentacles when touched are also making cause and effect assumptions with associated energy expenditure cost / benefit ratios all encoded in their simple neural nets. It is almost safe to say that if something is alive then it is in the business of modelling reality because, to remain alive, the future must, to some extent, be predicted.<br />
<br />
Prediction is a fundamental property of life and so it is fortunate that making predictions for simple linear systems is really quite easy - the model must be able to run faster than reality. Imagine firing a cannon and then sitting down with pencil and paper to work out the distance the cannon ball will travel. If you know the equations (the model) and have practised the arithmetic beforehand then you could probably calculate the answer before the ball lands and so predict the future. If you have to first remember the equations from your school physics class, then labour over the arithmetic then the ball may land before you have finished your calculation and you may as well use a measuring tape to find the actual answer. What makes a dynamic prediction worthwhile is that it happens before the event itself and for that to occur the model must be able to run faster than reality it is modelling.<br />
<br />
For a model to run faster than reality it must be different from reality. If you were to model a portion of reality with absolute fidelity then it would be running at real-time speed. Thus a predictive model must take short cuts, it must round numbers up or down, it must miss out extraneous details and ignore apparently unconnected variables. Thus a model is <span style="font-style: italic;">compressed</span>, its is squeezed of its useless details to leave the essential, highly optimised set of relationships that describe a precise set of cause and effect relationships. The equations for the trajectory of a cannon ball work very well and give answers that are more than good enough yet they do not include any information on the type, weight or nutritional content of cannon manufacturer's favourite breakfast cereal. Not including this information saves valuable time and enables dynamic predictions about firing distances to be made before the actual event occurs.<br />
<br />
This is a very seductive idea. It would seem that in order to predict something accurately then all that is required is a sufficiently accurate model. Certainly Stalin and Hitler thought so. Interestingly both Stalin and Hitler lived in Vienna at exactly the same time, just a few years before the outbreak of the first world war and whilst there they both wrote about walking in the same park and how they enjoyed watching the spectacle of Emperor Franz Josef I clattering past in his gilded carriage surrounded by his glittering horse guards. From this imperial demonstration, and many other life experiences, they both drew the same lessons concerning the absolute centralization of state power. They wanted to create a bureaucratic model of both a society and the economy upon which it depends then, using this model, a tiny elite could efficiently predict and so command the real society and economy thus satisfying the needs of the people. <br />
<br />
What was behind this revolutionary notion and was it coincidence that the idea seemed to ripen at just that time? In those hopeful, pre-war days it seemed as though science had amply demonstrated that careful observation allowed for the creation of genuinely accurate models and these models then allowed for useful predictions and finally that those predictions allowed complicated systems, such as manufacturing processes, to be accurately and profitably controlled. Empires were forged on the back of this assumption and both Stalin and Hitler simply extended it to come to an identical conclusion, that with enough observations, card indexing and information, an entire economy and an entire society could be usefully modelled and so predicted and so controlled - scientifically - from the top down. <br />
<br />
After leaving Vienna they both set about creating a compressed model of their societies. Vast armies of bureaucrats and technicians set about gathering and collating data to create a huge, hive-like interactive model of the economy and the society. The human cogs in the state calculating machine churned their way through mountains of detail. They gathered data from the secret police, the shop floor and factory gate to feed the model and so generate predictions for future consumption. These would then become production targets. The targets would be met or not, the differences observed and fed back into the model, future targets created - and so the whole system would hum along in a dynamic, cybernetic balance freeing up the workers to enjoy a völkisch idyll or a glorious dictatorship-of-the-proletariat depending on your taste in utopian fantasy. <br />
<br />
Were the great political and economic modellers of the 20th Century a success? No, the goddess of the eternal court of history did not acquit them. Their models failed, their societies required repression to keep the elites in power and dystopia reigned - but why? The historically accurate answer is, as you would expect, complex and multi-stranded but at its heart lies the misapplication of predicative modelling, specifically a missing piece of the explanatory jigsaw, the notion of chaotic systems. It is not that the science of the early 20th Century got it fundamentally wrong. Modelling does allow predictions and predictions do enable the exertion of control however this process cannot be usefully applied to every complex system. There exists a certain type of complex, dynamic system that is not amenable to prediction and it is these systems that today we label as <span style="font-style: italic;">chaotic</span>.<br />
<br />
Consider a system that is not chaotic. Such as system can guarantee that for every observable cause there will be a equivalent observable effect and that cause and effect scale together in a linear fashion. These systems can certainly be modelled and therefore predicted. Firing a cannon ball at a certain velocity will cause it to travel a certain distance. The distance the ball travels scales well with the initial muzzle velocity, the faster it is travelling the further it will go. <br />
<br />
Now imagine a cannon where this cause and effect linkage did not scale in a simple linear fashion. Firing the cannon with a little gunpowder causes the ball to dribble out of the muzzle (as expected), a grain or two more powder and the ball sails into the heavens (a shock), add a bucket load more and the ball lands just a few feet away (profoundly disturbing). In a non-linear system the results are not random but the primary cause, in this case the amount of gunpowder, cannot be used to usefully predict the primary effect, the distance travelled by the cannon ball. This is because non-linear systems are sensitive to much more than their primary cause. The results they produce are combinations of many subtle interactions that feedback into each other, amplifying some minuscule causes and attenuating other gross causes to create predictive confusion. In the case of the non-linear cannon it may be that the temperature of the barrel affects the trajectory and therefore, without realising it, the amount of gunpowder used in the previous firings is feeding back into the performance of the current firing causing complex results. Eliminate this variable and you discover that another subtle set of interactions takes its place, and another, and another. The system is perfectly rational and explicable its just that the amount of detail required to usefully model it is intractable. <br />
<br />
Not all non-linear systems are chaotic, some may just be complex, and the recent rise of staggering computational power has allowed more of these complex systems to be usefully predicted. What defines a truly chaotic system is its extreme, perhaps infinite, sensitivity to any change in its state. Modelling such a system now becomes officially impossible because, as we have seen, all models are short cuts. A model must be compressed in order to be predictive and since it is compressed it is different from the reality it is modelling. Yet for a chaotic system <span style="font-style: italic;">any</span> deviation from reality will cause an unpredictable outcome. Surely though, it is just a matter of the degree of precision? If the model is not working because it is imprecise then simply add more precision. This is the basic flaw ignored by all experts who purport to predict and control complex, dynamic systems such as economies and societies.<br />
<br />
To give a fundamental example, a model cannot have the ability to represent a number to infinite precision, at some stage a model's numerical values must be rounded either up or down. Applied to a model of a chaotic system, this deviation from reality is deadly. The rounding error, no matter how infinitesimal, will be churned through internal feedback loops within the system and amplified and attenuated. It is like taking a nice round lump of pizza dough and drawing two spots on it side by side. Now stretch the dough (amplification) and then fold it back on itself (attenuation), stretch and fold. The dough never breaks, the system is never disjointed, but after a few cycles where are the two dots? Just about the only thing you can predict is that they are not still side by side.<br />
<br />
Now imagine a number with two values after the decimal point, say 1.23 and multiply that number by ten and we get 12.3X. What is the new number X? Is it zero or some other random number? There is nothing special about zero, it is a just a guess at the true value and it has a 1 in ten chance of being correct. Now divide the number by 1000 to give 0.01 and them multiply is by 1000 to give 1.00 and now information is lost. This is the same as stretching and folding the dough.Positive and negative feedback loops in a model constrained by finite precision must lead to information generation and information destruction and since all models must enforce a finite precision then the stretching and folding of finite precision numbers must perturb a model's trajectory through the space of all possible results and cause it to veer away from reality.<br />
<br />
It must be appreciated that upping the precision of a model cannot solve this problem. It simply does not matter how small the error is, the feedback loops will eventually inflate it and so cause the model to spiral away into an alternative, non-useful, reality. The more accurate you make the model the more useful time you can buy but always and eventually the model must deviate from reality simply because it is a finite model and not the infinitely detailed reality it pretends to represent - the map is not the terrain. <br />
<br />
This then is the reason top-down, command economies always fail and why Stalin and Hitler are not revered today as our economic and social saviours. The reality of an economy or society is a web of interlinked interactions within which feedback loops, both positive and negative, nested and isolated, exists in rampant abundance. This means that models of such systems, with their inevitable short cuts, must ultimately fail in their predictions. A chaotic system cannot be controlled by commands from the top because the models required to generate the commands cannot be compressed without radically disconnecting their predictions from reality. This is the lesson that the totalitarians of the 20th Century have bequeathed to us. Have we taken it to heart?<br />
<br />
The recent financial crash has shown that, at least in the realm of predicting the stock market, we have not. It is actually trivially simple to make money from the stock market, a sure-fire thing, and here is how you do it. Pick a global company with an instantly recognisable name whose shares are traded daily. Now send off 1024 spam emails, half predicting that the stock price will rise and half predicting it will fall. Wait a week and send out 512 mails to the people who received your previous correct prediction, as before half predict up, half predict down. Repeat the process for the 256 recipients who have now received two correct predictions, then again for the 128 who have now received three correct predictions, 64 for four correct predictions, 32 for five, 16 for six, 8 for seven, 4 for eight correct predictions. These last four will think either that you are a magician or that you have insider knowledge. They may even have already made themselves very rich, entirely legitimately, from your astonishing ability to predict the market. You now ask the last four to send you £1000 for your final prediction with the promise that, if you get it wrong, you will pay them back - the recipients simply cannot lose. You receive £4000 and send back £2000. You make £2000 and walk away. Rinse and repeat.<br />
<br />
If that sounds far fetched then you have never worked in an investment bank. This is pretty much how they make their money. They simply employ enough traders and analysts so that some of them are right for long periods of time and they look like they can predict the system. There will always be a more than enough traders who are currently on a lucky run so that investors can be deluded into thinking they are magicians or, in our scientific times, have some esoteric, possibly algorithmic system that gives them the ability to predict the movements of the market. Of course they can't predict those movements no matter how often their predictions turn out to be correct. They can't predict the chaotic market for all the same reasons of finite precision models attempting to simulate an uncompressible, chaotic system we have just seen, yet they look like they are for the same reason the spam trick works. <br />
<br />
The global economic system is the very definition of a chaotic, complex, dynamic system and to think that it can be modelled and so predicted is our 21st Century hubris that simply invites Nemesis to do her worst - and so she does - conjuring for our arrogance stock market analysts with their liquid promises just as she conjured, in the last century, Stalin and Hitler with their utopian visions. <br />
<br />
In short - <span style="font-style: italic;">Beware the prophet bearing finite precision models</span>.biofractalhttp://www.blogger.com/profile/11229842750489025189noreply@blogger.com0tag:blogger.com,1999:blog-6927558.post-77100863575599694842009-09-07T15:19:00.053+01:002013-05-04T17:21:59.432+01:00An Introduction to Applied Evolutionary Metaheuristics<div>
<b>Jonathan Anderson</b></div>
<div>
<br /></div>
First delivered by me at "Selected Topics on Complex Systems Engineering" an international symposium held at Morelia, Mexico in October 2008. It was subsequently published in the <i>European Journal of Operational Research : Applications of metaheuristics</i><br />
<br />
<a href="http://docs.google.com/present/view?id=dhck8ddg_132fs45f2ff"><b>View slide show</b></a><br />
<h4>
Abstract</h4>
This paper introduces some of the main themes in modern evolutionary algorithm research while emphasising their application to problems that exhibit real-world complexity. Evolutionary metaheuristics represent the latest breed of biologically inspired computer algorithms that promise to usefully optimise models that display fuzzy, complex and often conflicting objectives. Until recently, evolutionary algorithms have circumvented much of this complexity by defining a single objective to be optimised. Unfortunately nearly all real-world problems do not compress neatly to a single optimisation objective especially when the problem being modelled is non-linear. Recent research into multi-objective evolutionary metaheuristic algorithms has demonstrated that this single-objective constraint is no longer necessary and so new opportunities have opened up in many fields including environmental health and sustainability.<br />
<div>
<br /></div>
<div>
With their proven ability to simultaneously optimise multiple, conflicting objectives, evolutionary metaheuristics appear well suited to tackle ecological problems. Such algorithms deliver a range of optimal trade-off solutions that allow an appropriate profit / cost balance to be selected according to the decision maker's imperatives. This paper concludes with an examination of a powerful multi-objective evolutionary algorithm called IC-SPEA2 (Martínez-García & Anderson, 2007) and its application to a real world problem namely the maximisation of net revenue for a beef cattle farm running on temperate pastures and fodder crops in Chalco, Mexico State. Some counter-intuitive results and their impact on the farm's overall sustainability are discussed.<br />
<a name='more'></a><br />
<h4>
What is a Metaheuristic?</h4>
A heuristic is a 'rule of thumb', that is a algorithm that provides a solution to a problem without considering whether the solution is formally optimal but which will, nonetheless, tend to be good enough for real-world application.<br />
<br />
<a href="http://1.bp.blogspot.com/_kbCCKz8F0ok/SqgXcH62dRI/AAAAAAAAAHA/g9WdDoVx2lQ/s1600-h/good_enough_3.jpg" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5379575526823720210" src="http://1.bp.blogspot.com/_kbCCKz8F0ok/SqgXcH62dRI/AAAAAAAAAHA/g9WdDoVx2lQ/s320/good_enough_3.jpg" style="cursor: hand; cursor: pointer; height: 315px; width: 313px;" /></a><br />
<span class="Apple-style-span" style="font-size: small;"><b>Figure 1</b>: The law of diminishing returns indicates that perfection is not a realistic goal.</span><br />
<br />
A metaheuristic is an algorithm that synthesises two or more heuristics into a single compound. A metaheuristic algorithm is therefore a heuristic that relies on other heuristics either as sub-components or outsourced to black-box functions.<br />
<br />
A metaheuristic example is found in neural-networks. Researchers found that manually training neural networks was inefficient (Alba, Enrique, Marti, Rafael. 2006.). This issue has been solved by employing search heuristics, including evolutionary algorithms, to discover appropriate training regimes for the neural networks (Alba et al). The compound algorithms that results from the synthesis of a neural-networks and evolutionary search heuristics are metaheuristic algorithms.<br />
<h4>
No Free Lunch</h4>
It may seem as though metaheuristics have the potential to create a general or universal search / optimisation mechanism to solve complex problems for which no problem-specific heuristic currently exists. However a theoretical stumbling block stands in the way of this goal.<br />
<br />
The No Free Lunch Theorem states that : "any two algorithms are equivalent when their performance is averaged across all possible problems" (Wolpert & Macready 1995)<br />
<br />
This implies that for any optimisation algorithm, gaining additional performance over one class of problems is exactly paid for in performance over another class. A universal problem solver is therefore not possible.<br />
<br />
Despite this lack of universality metaheuristics remains a vibrant topic of research with many exciting approaches under investigation:<br />
<ul>
<li>Simulated annealing</li>
<li>Ant colony optimization</li>
<li>Harmony search</li>
<li>Evolutionary algorithms</li>
</ul>
<h4>
What is an Evolutionary Algorithm?</h4>
Being alive is a very complex objective, perhaps the most complex there is, and yet solutions to this problem exist in astonishing variety and subtly.<br />
If we cast biology in the role of an optimisation strategy then we can say that biology seeks optimal solutions to the problem of being alive. If, as computer scientists, we could unlock the process by which this most complex of combinatorial-optimisation problems is successfully solved then we would surely have found a powerful heuristic tool.<br />
The process by which biological complexity emerges is Evolution by Natural Selection. In his book “The Blind Watchmaker”, professor Richard Dawkins provides a precise definition of this algorithmic process, precise enough for eventual translation into an computer code:<br />
"<i>Natural Selection is the result of the non-random replication of randomly varying replicators</i>" (Dawkins. 1986)<br />
Natural selection discovers optimal replicators via iterative replication, random variation and environmentally guided selection. If being alive is the problem then, according to Dawkins, the non-random replication of randomly varying replicators is the solution. An evolutionary algorithm is simply this definition converted directly into a computer code.<br />
<h4>
A General Evolutionary Algorithm</h4>
<ol>
<li><i>Initialization</i>First an initial population must be created. This population will contain replicators whose characteristics have been randomly generated.<br /><br /><a href="http://4.bp.blogspot.com/_kbCCKz8F0ok/SqgXA4TilII/AAAAAAAAAG4/Vklrl-XlqXk/s1600-h/general_ea_2.jpg" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5379575058775839874" src="http://4.bp.blogspot.com/_kbCCKz8F0ok/SqgXA4TilII/AAAAAAAAAG4/Vklrl-XlqXk/s320/general_ea_2.jpg" style="cursor: hand; cursor: pointer; height: 320px; width: 233px;" /></a><br /><span class="Apple-style-span" style="font-size: small;"><b>Figure 2</b>: The main loop at the heart of every evolutionary algorithm.</span><br /></li>
<li><i>Fitness assignment</i><br />Each replicator must be evaluated and assigned a fitness value according to a problem-specific definition of fitness.<br /></li>
<li><i>Environmental selection</i><br />A subset of the fittest replicators is selected to be used as the breeding stock when breeding a new population of child replicators. The selection process is deterministic and based on the problem specific fitness values assigned previously.<br /></li>
<li><i>Termination</i><br />If stopping condition is met then Stop.<br /></li>
<li><i>Breeding and Variation</i><br />Breed fittest to create a new population of child replicators. Breeding will mainly involve the mixing of parent characteristics, multiple parents to produce a single child, with the additional chance of a random mutation to slightly alter randomly selected child characteristics.<br /></li>
<li><i>Go to Fitness Assignment</i></li>
</ol>
Over many generations the population will efficiently search the replicator state-space, the n-dimensional set of possible replicator states where n = the number of mutable replicator characteristics, producing replicators with ever increasing fitness (Goldberg 1989).<br />
<br />
Another way of conceiving the evolutionary process is to imagine a complex process deconstructed into a set of rules that together make up a model of that process. The purpose of the evolutionary algorithm is therefore to optimise this model. A model will have many variables that go to define its current state any given point in time. These are usually called the model's decision values. A collection of decision values represents the variability between different instances of a model.<br />
<br />
When optimising a model certain characteristics are selected to be the targets of optimisation. These are usually called the objective values. The purpose of the model is to convert decision values into objective values according to the model's internal rules. The objective values are then used by the evolutionary algorithm to calculate a fitness value. Thus a replicator is a set of decision values with an associated set of objective values. The model is the set of environment rules within which the replicator finds its physical expression. The evolutionary algorithm is the set of rules that evaluates, selects and breeds future generations of replicators ready for their brief flicker of life within the model.<br />
<br />
Replicators consist of a series of values, analogous to genes, and so can be conveniently encoded as a set of characters, or string, which is analogous to the genome. Often binary is used to encode the replicator values and so the genome will consist of a long string of binary '1' and '0' characters. This makes it relatively easy to manipulate the genome, breaking it into pieces and splicing those pieces back together using standard string handling techniques common to most computer languages. For example during breeding when parental values are mixed together to create a novel child using the mechanism called crossover.<br />
<h4>
The Crossover Mechanism</h4>
During breeding the characteristics of two selected parents are mixed together using a mechanism called crossover to create a novel child. The selection process depends on the fitness values of the parents where the fitter the parent the more likely it is to be selected. This means that the average fitness of the population tends to increase over the generations.<br />
<ul>
<li>Select two parents</li>
<li>Pick a locus somewhere on the parental genome</li>
<li>Split both the parent's genomes at the selected locus</li>
<li>Take the first portion of the first parent's genome and join it to the second portion of the second parent's genome to create a new child genome</li>
</ul>
<a href="http://2.bp.blogspot.com/_kbCCKz8F0ok/SqUmnZbVfaI/AAAAAAAAAGA/TM21BW9do78/s1600-h/dhck8ddg_124czz4w4fr_b.png" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5378747788246220194" src="http://2.bp.blogspot.com/_kbCCKz8F0ok/SqUmnZbVfaI/AAAAAAAAAGA/TM21BW9do78/s400/dhck8ddg_124czz4w4fr_b.png" style="cursor: hand; cursor: pointer; height: 158px; width: 300px;" /></a><br />
<span class="Apple-style-span" style="font-size: small;"><b>Figure 3</b>: Illustration of the crossover mechanism that recombines parental characteristics to create novel child combinations.</span><br />
<div>
<br />
The mixing of parental values does not affect the values themselves but it does create novel combinations of pre-existing values. This means that short combinations of values which happen to result in higher fitness are preserved between generations. During crossover multiple, high fitness values are moved as a single unit or schema (Goldberg. 1989) between parents and their offspring.</div>
<div>
<h4>
Implicit Parallelism</h4>
As a result of selection and crossover fit building blocks or schemata are propagated from generation to generation. Each population member thus provides multiple points at which the state-space is being sampled and tested. Since a population has many members and each member is sampling the state-space with many schemata the resulting search is implicitly parallel.(Goldberg. 1989)<br />
<br />
This multiple parallel sampling of the state-space is a unique feature of evolutionary algorithms and it dramatically leverages the efficiency of the search without requiring any special book keeping, processor or memory overheads.<br />
<br />
When attempting to solve NP-hard problems, increasing the number of values that define a replicator will exponentially increases the size of the state-space to be searched (NP-hard is a class of problem that cannot be solved in less time than the exponential of the problem size). Evolutionary algorithms display a remarkable insensitivity to the inflation of their target state-space. This insensitivity is due to implicit parallelism. As the number of replicator variables grows then so must the length of the string used to encode them. This simply creates more schemata and so increases the degree of implicit parallelism. In this way an evolutionary algorithm leverages the inflation of the state-space to amplify the effects of implicit parallelism and so mitigates the effects of state-space inflation.<br />
<h4>
Single Objective Evolutionary Algorithms</h4>
Single objective problems are problems whose objectives can be collapsed or aggregated into a single overall objective to be maximised or minimised.A standard NP-hard benchmark single objective problem is the travelling salesperson problem or TSP which states: Given a number of cities and the distance from any city to any other city, what is the shortest round-trip route that visits each city exactly once and then returns to the starting city?Exact, non-heuristic algorithms will give precisely the shortest possible route. For example various branch-and-bound algorithms1 can be used to process TSPs containing 0-60 cities whereas progressive improvement algorithms2 work well for up to 200 cities (Dekker 2008)<br />
<br />
Evolutionary algorithms do not necessarily return the shortest route but they do promise to return a route that is sufficiently short to be useful. The payback for accepting this heuristic uncertainty is the insensitivity to state-space inflation. Evolutionary algorithms can usefully process TSPs up to 100,000 cities and beyond (Dekker 2008).<br />
<br />
Single-objective evolutionary algorithms allow single objective problems to be usefully solved. By contrast multi-objective problems do not resolve to a single global optimum but are instead optimise to a potentially large set of trade-off solutions. Finding this set of optimal trade-offs requires a more powerful class of evolutionary algorithm.<br />
<h4>
What is a Multi-Objective Problem?</h4>
A multi-objective problem is characterised by having two or more conflicting objectives. If objectives do not conflict then they can always be aggregated back into a single compound objective. It is the conflict between different objectives that defines a multi-objective problem and creates the additional complexity that defeats the single-objective evolutionary algorithm approach.</div>
<div>
<br />
Multi-objective problems can be found in various fields:<br />
<ul>
<li>Product and process design</li>
<li>Finance</li>
<li>Aircraft design</li>
<li>Oil and gas industry</li>
<li>Vehicle design</li>
</ul>
</div>
<div>
Or wherever decisions are taken where trade-offs exist between two or more conflicting objectives:<br />
<ul>
<li>Maximising performance and minimising fuel consumption</li>
<li>Maximising strength and minimising weight</li>
</ul>
<br />
<a href="http://2.bp.blogspot.com/_kbCCKz8F0ok/SqgZuh-9lNI/AAAAAAAAAHQ/VxN0TVy2ZIA/s1600-h/strength_weight_1.jpg" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5379578042081187026" src="http://2.bp.blogspot.com/_kbCCKz8F0ok/SqgZuh-9lNI/AAAAAAAAAHQ/VxN0TVy2ZIA/s320/strength_weight_1.jpg" style="cursor: hand; cursor: pointer; height: 294px; width: 320px;" /></a><br />
<span class="Apple-style-span" style="font-size: small;"><b>Figure 4</b>: A real world evolutionary solution to the strength versus weight contradiction.</span><br />
<br />
Multi-objective problems cannot be resolved to a single solution that simultaneously optimises all of the objectives. As soon as two or more conflicting objectives are present then the single global optimum solution disappears to be replaced by a set of non-dominated or Pareto-optimal solutions.<br />
<br />
A solution is considered Pareto-optimal if every objective has been optimised to such an extent that attempting to further optimise one of the objectives will result in the degradation of the other objectives. Therefore every Pareto-optimal solution represents an optimal trade-off between the conflicting objectives. Since there are many combinations of objective trade-off there are also many different Pareto-optimal solutions.<br />
<br />
The members of a Pareto-optimal set cannot be distinguished in terms of optimality but they can be distinguished in terms of there perceived utility. In order to make a choice between Pareto-optimal solutions external information must be employed. This information can come directly from a human decision maker via a software interface or by encoding the known preferences of a human decision maker as set of post-optimisation rules.<br />
<br />
Solving problems with multiple conflicting objectives is challenging. The simplest approach is to construct a single aggregate objective function by assigning each objective a scalar weight which is combined into a single function that can be solved by any single-objective optimisation algorithm (Messac, A., Ismail-Yahaya, A., & Mattson. 2003). The problem with this approach is the globally optimal solution obtained will strongly depend on the values of the arbitrary objective weights. If a higher weight is specified for one objective relative to the others then the optimal solution will be one that favours that objective over the others. Solutions obtained using the weighted sum are always Pareto-optimal, but establishing a meaningful combinations of objective weightings can be very challenging (Messac et al).<br />
<h4>
Multi-Objective Evolutionary Algorithms</h4>
Evolutionary algorithms are well-suited for optimisation problems involving several conflicting objectives. A multi-objective evolutionary algorithm simultaneously optimises multiple conflicting objectives for a given problem resulting a Pareto-optimal set of solutions where each solution represents an optimal trade-off between the conflicting objectives. Although various evolutionary approaches to multi-objective optimisation are capable of searching for multiple solutions concurrently in a single run, in general the discovery of the Pareto-optimal set of solutions involves two implicit goals (Zitzler, Eckart and Laumanns, Marco and Thiele, Lothar. 2001):<br />
<ol>
<li>Minimise the distance from the evolving Pareto front to the undiscovered Pareto-optimal set.</li>
<li>Maximise the diversity of the generated solutions.</li>
</ol>
<h4>
Minimise the Distance</h4>
The set of intermediate solutions generated as the algorithm progresses is called the Pareto front. It is an n-dimensional surface (where n is the number of objectives) whose shape evolves through state-space as the algorithm discovers ever more optimal solutions (Zitzler et al)<br />
<br />
Over time the shape of the Pareto front approaches the shape of the Pareto-optimal set or the set of solutions that fully dominates all other possible solutions (Zitzler et al). One solution is said to fully dominate another if each of its objective values is superior (smaller if the objective is being minimised or greater if the objective is being maximised).<br />
<br />
To guide the evolving Pareto front, multi-objective algorithms can employ a technique called elitism whereby the best solutions found to date are retained in a secondary population called the archive. An elite solution is replaced in the archive when another solution is discovered that fully dominates it. This avoids the population being dominated by solutions that improve one objective at the expense of another.<br />
<h4>
Maximise the Diversity</h4>
As the algorithm progresses the elite members of the archive tend to converge on a series of points along the non-dominated front creating clusters of solutions. Such clusters indicate a lack of diversity within the emerging Pareto front. Solution diversity is important as rarefied areas of the front will remain poorly investigated with the potential to miss interstitial solutions that may be good enough for application in the real world.<br />
<br />
To prevent this gradual loss of diversity modern algorithms tend to incorporate a pruning mechanism whereby solutions that lie too close to their neighbours are removed or pruned to leave room for additional novel solutions. The distance between solutions is calculated in terms of the position in the multi-dimensional state-space with solutions that appear in dense clusters identified and removed (Zitzler et al).<br />
<br />
<a href="http://3.bp.blogspot.com/_kbCCKz8F0ok/SqUnC-8Yd2I/AAAAAAAAAGI/CjHmkNztGqM/s1600-h/dhck8ddg_125wcbqb6hj_b.png" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5378748262173407074" src="http://3.bp.blogspot.com/_kbCCKz8F0ok/SqUnC-8Yd2I/AAAAAAAAAGI/CjHmkNztGqM/s400/dhck8ddg_125wcbqb6hj_b.png" style="cursor: hand; cursor: pointer;" /></a><br />
<span class="Apple-style-span" style="font-size: small;"><b>Figure 5</b>: The pruning mechanism. The non-dominated set is shown on the left. The centre indicates those solutions selected for removal. The final, pruned set is shown on the right. This figure has been adapted from Zitler et al (pp8).</span><br />
<br />
The pruning mechanism needs to be carefully designed if good solutions, especially good outrider solutions lying at the edges of the Pareto front, are not to be discarded. The goal is to maintain solution diversity by maximising the spread of solutions across the Pareto front with each solution evenly spaced with respect to its neighbours (Zitzler et al).<br />
<h4>
Stopping the Algorithm</h4>
Due to the intractability of NP-hard, real-world problems the globally optimal Pareto set is typically unknown. This implies that it is impossible to know when the optimisation process should end since there is no metric to determine whether the current Pareto set is the unknown global optimum set. However this is a common feature of heuristic algorithms and is usually resolved by accepting that solutions only need to be good enough and not necessarily optimal in the formal sense.<br />
<h4>
What is a Constraint?</h4>
All real world problems are constrained by physical limits. Solutions that lie outside of these limits are infeasible. Constraints model the physical limits of a problem and therefore a constraint defines a component of solution feasibility. If a solution violates one or more of its constraints then the solution is considered infeasible (Wright J. A., Loosemore H. 2001).<br />
<br />
<a href="http://2.bp.blogspot.com/_kbCCKz8F0ok/Sqgd6KavBwI/AAAAAAAAAHo/U87A8jFs1PI/s1600-h/constraint.jpg" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5379582639960164098" src="http://2.bp.blogspot.com/_kbCCKz8F0ok/Sqgd6KavBwI/AAAAAAAAAHo/U87A8jFs1PI/s320/constraint.jpg" style="cursor: hand; cursor: pointer; height: 320px; width: 313px;" /></a><br />
<span class="Apple-style-span" style="font-size: small;"><b>Figure 6</b>: Constraints are the rules that bound the feasible set of solutions. All solutions that do not satisfy the constraints are considered infeasible.</span><br />
<br />
Where advanced knowledge of the problem domain is available it is often possible to define constraints and so confer upon a solution a measure of feasibility. For example the Travelling Salesperson Problem typically does not consider the needs of the salesperson however in real life a salesperson will have a physical limit to the distance they can travel before they must return home. In other words there is an upper limit to the route length and any route that exceeds this length must be considered infeasible. The longer a route is over the upper limit the greater the degree to which the constraint is violated. This allows infeasible routes to be compared and ranked according their degree of infeasibility.<br />
<h4>
The Problem with Constraints</h4>
Since the infeasible solutions will vastly outnumber feasible solutions then throwing away infeasible solutions as they are discovered forces the algorithm to spend most of its time randomly walking through the state-space trying to find a population of feasible solutions to get the ball rolling. The algorithm is being presented with a cliff-face to climb at the outset of any run.<br />
<br />
A more effective approach is to provide a mechanism whereby the algorithm can evolve its way from infeasibility to feasibility so converting the cliff-face into a gradual incline. A good way to create a gradual slope is to add an infeasibility objective to the problem(Wright et al). An infeasibility objective is a single measure of a solution's infeasibility. It is treated as an independent objective by a multi-objective evolutionary algorithm. The measure of infeasibility should represent both the number of active constraints and the extent to which each constraint is violated. A measure of infeasibility that has these properties is the sum of the normalised constraint violation values for all violated constraints (Wright et al). The new infeasibility objective is set to be minimised, the smaller the objective value the more feasible the solution is. For feasible solutions the value for the infeasibility objective should equal zero.<br />
<br />
With the infeasibility objective in place it is not necessary for a given solution to be feasible in order for it to take part in the ongoing evolutionary process. This allows a population of infeasible solutions to be generated at the start of a run and used to slowly evolve towards a set of feasible solutions.<br />
<h4>
Applying an Evolutionary Metaheuristic</h4>
The hybrid metaheuristic algorithm IC-SPEA2 was first proposed by Martínez-García & Anderson, 2007. It consists of an an infeasibility constrained (IC) version of the Strength Pareto Evolutionary Algorithm 2 (SPEA2) (Zitzler et al).<br />
<br />
The core algorithm, SPEA2, is a powerful multi-objective evolutionary metaheuristic that evolves a set of initially infeasible solutions towards the feasible Pareto-optimal set using elitism, a fine-grained fitness assignment strategy and a sophisticated pruning mechanism for maintaining solution diversity without losing outriders.<br />
<br />
The state-space is constrained by the addition of an infeasibility objective that forces the algorithm to evolve and maintain a feasible set of solutions. At the end of each optimisation run the non-dominated Pareto-optimal set of solutions is presented to the human decision maker for evaluation and selection.<br />
<br />
<b>SPEA2 Main Loop</b> (Zitzler et al)</div>
<div>
<b><br /></b><span class="Apple-style-span" style="font-size: small;">Input: N population size<br />N² archive size<br />T maximum number of generations<br />Output: ND (non-dominated set)</span></div>
<div>
<br />
<i><b>Step 1</b>. Initialization</i><br />
Generate an initial population P0 and create the empty archive P²0 → Ø. Set t = 0.<br />
<br />
<i><b>Step 2</b>. Fitness assignment</i><br />
Calculate fitness values of individuals in Pt and P²(t)<br />
<br />
<i><b>Step 3</b>. Environmental selection</i><br />
Copy all non-dominated individuals in Pt and P²(t) to P²(t+1). If size of P²(t+1) exceeds N² then reduce P²(t+1) by means of the truncation operator, otherwise if size of P²(t+1) is less than N² then fill P²(t+1) with dominated individuals in Pt and Pt<br />
<br />
<i><b>Step 4</b>. Termination</i><br />
If t »= T or another stopping condition is satisfied then set ND to the set of decision vectors represented by the non-dominated individuals in P²(t+1). Stop.<br />
<br />
<i><b>Step 5</b>. Mating selection</i><br />
Perform binary tournament selection with replacement3 on P²(t+1) in order to fill the mating pool.<br />
<br />
<i><b>Step 6</b>. Variation</i><br />
Apply crossover and mutation operators to the mating pool and set P(t+1) to the resulting population. Increment generation counter (t = t + 1) and go to Step 2.<br />
<h4>
Testing the Algorithm: The [0/1] Knapsack Problem</h4>
The [0-1] knapsack problem is a classic test problem in combinatorial optimisation (Goddard). It seeks the best choice of essential equipment that can fit into one knapsack to be carried on a trip. Given a set of items, each with a weight and a profit value, determine the combination of items to pack into the knapsack so that the total weight is less than or equal to the knapsack's total capacity and the total profit is as large as possible. The problem is called a “[0-1]” problem because each item must be entirely accepted or rejected, that is you cannot sub-divide an item.<br />
<h4>
Problem Description</h4>
Given a knapsack with maximum capacity and a set of items where each item has some weight and profit value, what items should be packed into the knapsack to achieve the maximum profit for the minimum weight?<br />
<br />
<a href="http://3.bp.blogspot.com/_kbCCKz8F0ok/Sqgck7-9xfI/AAAAAAAAAHg/oYqqH3tYMxM/s1600-h/knapsack_2.jpg" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5379581175796712946" src="http://3.bp.blogspot.com/_kbCCKz8F0ok/Sqgck7-9xfI/AAAAAAAAAHg/oYqqH3tYMxM/s400/knapsack_2.jpg" style="cursor: hand; cursor: pointer; height: 299px; width: 400px;" /></a><br />
<span class="Apple-style-span" style="font-size: small;"><b>Figure 7</b>: The knapsack test rig showing a perfect Pareto optimal curving through the 2D state space. Note the lack of clumping. The red dots are infeasible solutions due to a constrained sack size.</span><br />
<br />
Problem Objectives:<br />
<ul>
<li>Maximise Profit</li>
<li>Minimise Weight</li>
</ul>
Since each item has both a positive profit and weight then adding more items increases the profit, which is desired, but also increases the weight, which is not desired. Conversely removing items decreases the weight, which is desired, but also decreases the profit which is not desired. In other words the objectives are conflict.<br />
<br />
Conflicting objectives means that no single knapsack will represents the best possible combination of both weight and profit. Instead a range of non-dominated knapsacks, the Pareto-optimal set, is required to represent the full range of trade-offs between weight and profit leaving the external human decision maker to choose according to their own preferences.<br />
<h4>
Applying the IC-SPEA2 Algorithm: A Mexican Beef Cattle Farm</h4>
A farm is a dynamic system emerging from the imperatives of the environment coupled to the experience of the farmer and the way the available resources are manipulated when trying to achieve multiple and most probably conflicting objectives. A farmer's capacity to act is always bounded by multiple trade-offs, environmental constraints and shifting measures of success (Martínez-García A, Anderson J. 2005).<br />
<br />
To explore the problem of farm profitability a model of a Mexican beef cattle farm running on temperate pastures and fodder crops was created. The model served as a black-box objective function taking a set of decision variables and convert these values, via the complex, iterative model rules, into multiple objective values for evaluation by IC-SPEA2, the multi-objective evolutionary algorithm.<br />
<a href="http://3.bp.blogspot.com/_kbCCKz8F0ok/SqUndIETd0I/AAAAAAAAAGQ/WkeaTEcUo1k/s1600-h/dhck8ddg_126ch4zvjc8_b.png" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5378748711299151682" src="http://3.bp.blogspot.com/_kbCCKz8F0ok/SqUndIETd0I/AAAAAAAAAGQ/WkeaTEcUo1k/s400/dhck8ddg_126ch4zvjc8_b.png" style="cursor: hand; cursor: pointer; height: 340px; width: 400px;" /></a><br />
<span class="Apple-style-span" style="font-size: small;"><b>Figure 3</b>: An illustration of the relationships between the farm manager, the farm, the farm model and the IC-SPEA2 algorithm used to generate Pareto optimal farming strategies</span><br />
<br />
The purpose of the modelled beef farm is make money by converting food into live animal weight. The farmer can control the rate of weight gain by varying the amount and type of food to be included in the diet formulation for each of the twelve months of the season. The different food types have different nutritional benefit and purchase / production costs which vary throughout the season.<br />
<br />
The optimisation objectives for the farm were:<br />
<ol>
<li><i>Maximise net revenue</i></li>
<li><i>Minimise the cost of the diet</i><br />This conflicts with maximising net revenue since cheap food tends to have low nutritional value.</li>
<li><i>Maximise average daily weight gain</i><br />This conflicts with minimising diet costs since increasing the weight gain involves using food with the highest nutritional value of the diet resulting in increased diet costs. It may also conflict with maximising net revenue since heavy animals lead to higher variable costs.</li>
</ol>
The farm manager was asked to suggest an optimal strategy based on his experience and perceived objectives. He suggested the use of as much pasture as possible to feed the herd, supplementing this with alfalfa only when seasonal shortages of pasture occurred, and further only using silage and stubble when both pasture and alfalfa were in short supply.<br />
<br />
In other words the farmer suggests feeding his animals in such a way as to maximise the weight gain while minimising the diet costs. If these two objectives are achieved then it seems perfectly reasonable to assume that third objective maximise the net revenue will be achieved since fatter animals should sell for more (Martínez-García et al).<br />
<h4>
Interpreting the Results – A Counter Intuitive Strategy is Discovered</h4>
<a href="http://1.bp.blogspot.com/_kbCCKz8F0ok/SqUno6YK3PI/AAAAAAAAAGY/NLgcbpIMW-Q/s1600-h/dhck8ddg_127g6trhndp_b.png" onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}"><img alt="" border="0" id="BLOGGER_PHOTO_ID_5378748913782807794" src="http://1.bp.blogspot.com/_kbCCKz8F0ok/SqUno6YK3PI/AAAAAAAAAGY/NLgcbpIMW-Q/s400/dhck8ddg_127g6trhndp_b.png" style="cursor: hand; cursor: pointer; height: 254px; width: 343px;" /></a><br />
<span class="Apple-style-span" style="font-size: small;"><b>Table 9</b>: Martínez-García et al (pp. 647)</span><br />
<br />
IC-SPEA2 returned multiple, distinct Pareto-optimal strategies, of which two were of particular interest:<br />
<ul>
<li><i>Solution 2</i><br />The strategy that gave the highest net revenue.</li>
<li><i>Solution 1</i><br />The strategy that gave the highest live weight gain.</li>
</ul>
These results were counter-intuitive as it was assumed that the strategy that gave the highest live weight gain would be the same strategy that gave the highest net revenue.<br />
Even the farmer assumed that the goal was to make the animals as heavy as possible since heavy animals sell for more money.<br />
<br />
Yet IC-SPEA2 found a strategy that gave the highest net revenue while delivering lighter animals. This has potentially profound implications for the sustainable management of the farm.<br />
<h4>
The Implications for Farm Sustainability</h4>
Among the most surprising discoveries resulting from using IC-SPEA2 were the counter-intuitive nature of the solutions, their diversity and the fact that such a set allows for a compromise between the manager's goals and the precautionary principle4.<br />
<br />
This point is demonstrated by the fact that for the solution with the highest net revenue the animals eat less allowing for a compromise between the farmer's objectives and the sustainability of the farm. Maintaining net revenue while reducing live weight gain can correspondingly reduces the stress on the available pasture and allows the farmer to meet his profit obligations while operating the farm at below its maximum carrying capacity.<br />
<br />
Furthermore, IC-SPEA2 showed the potential usefulness of on demand optimisation displaying an ability to simulate and optimise a large, non-linear model with an intractable state-space using a cheap laptop computer. Martínez-García et al (2005) states that:<br />
<br />
<i>... as self-generated, cognitive systems with humans at their core, the main technical problem for achieving sustainable [farming systems] is how to enhance the decision-makers skills for choosing appropriate courses of action, in real time, in response to their own internal, dynamic purposes, while increasing their number of choices to face complex environmental conditions.</i><br />
<br />
This suggests farmers making regular use of real-time evolutionary metaheuristic software may be able to respond to changes in their dynamic farm systems and so perform frequent, incremental, strategic re-evaluations in the field. These small adjustments measured against the backdrop of a shifting reality may well buffer against the over-shoot and performance fluctuations that can prove damaging to the sustainability of a complex, dynamic farm system.</div>
<div>
<br />
Finally it suggests that the ability of multi-objective evolutionary heuristics to generate a Pareto set of multiple, optimal choices gives the farmer the control and scope to dynamically re-balance the farm's objectives to maintain its sustainability in a measured and controlled manner even as the environmental conditions fluctuate and the farmer's measures of success evolve.<br />
<h4>
References</h4>
Alba, Enrique, Marti, Rafael (Eds). 2006. Metaheuristic Procedures for Training Neural Networks. Operations Research/Computer Science Interfaces Series, Vol.35<br />
<br />
Dawkins, R. 1996. The Blind Watchmaker. UK: Penguin Books<br />
<br />
Dekker, M.D. 2008. Travelling salesman problem. http://en.wikipedia.org/wiki/Travelling_salesman_problem<br />
<br />
Goddard, S. Dynamic programming 0-1 Knapsack problem. CSCE 310J. Data Structures & Algorithms. http://www.cse.unl.edu/goddard/Courses/CSCE310J/Lectures/Lecture8-DynamicProgramming.pdf<br />
<br />
Goldberg, D. E. (1989). Genetic Algorithms in Search, Optimization, and Machine Learning. Reading, MA: Addison-Wesley.<br />
<br />
Martínez-García A, Anderson J. (2005). Cárnico-ICSPEA2 – A metaheurístic co-evolutionary navigator for a complex co-evolutionary farming system. European Journal of Operational Research 179 (2007) 634–655<br />
<br />
Messac, A., Ismail-Yahaya, A., and Mattson, C.A. (2003) The Normalized Normal Constraint Method for Generating the Pareto Frontier, Structural and Multidisciplinary Optimization, Vol. 25, No. 2, 2003, pp. 86-98.<br />
<br />
Wolpert, D.H., Macready, W.G. (1995), No Free Lunch Theorems for Search, Technical Report SFI-TR-95-02-010 (Santa Fe Institute).<br />
Wright J. A., Loosemore H. (2001). An Infeasibility Objective for Use in Constrained Pareto Optimization. EMO 2001: 256-268<br />
<br />
Zitzler, Eckart and Laumanns, Marco and Thiele, Lothar (2001) SPEA2: Improving the Strength Pareto Evolutionary Algorithm. Evolutionary Methods for Design, Optimisation, and Control.<br />
<h4>
Notes</h4>
1 A general optimisation algorithm where all possible solutions are processed and discarded in blocks if they lie outside the estimated minimum and maximum bounds of the quantity being optimized<br />
<br />
2 Progressive improvement algorithms follow a direction of improvement until no further optimisation can be made. With luck the global optimum is now located.<br />
<br />
3 Two random solutions are selected from the population and the more fit of the two wins the tournament. A clone of the winning solution is made leaving the original in the population so it can continue to take part in future tournaments.<br />
<br />
4 A moral and political principle that invokes caution to avoid damaging the environment in ways that cannot be reversed where the scientific evidence is not conclusive.</div>
</div>
<br />
<br />
<iframe frameborder="0" height="342" src="http://docs.google.com/present/embed?id=dhck8ddg_132fs45f2ff" width="410"></iframe>biofractalhttp://www.blogger.com/profile/11229842750489025189noreply@blogger.com4tag:blogger.com,1999:blog-6927558.post-12109503030358898112009-08-09T15:58:00.033+01:002009-08-11T13:05:15.199+01:00The Broken Waterfall<blockquote><p><i>The traditional </i>predictive<i> approach to project management is being rejected in favour an </i>adaptive<i> or </i><span><i>Agile </i></span><i>approach.</i></p><p><i><span class="Apple-style-span" style="font-style: normal; "><i>This is not a matter of buzz-words or faddish management technologies, instead it is a genuine commitment to help clients get the software they </i><span><i>actually want</i></span><i> - on time and within budget.</i></span></i></p></blockquote><h2><span class="Apple-style-span" style="font-size:medium;">The Problem</span></h2><p>There is a problem with the delivery of software. The more complex a project the greater the chance the project will be delivered over budget and behind schedule. As a project grows in complexity there comes a point where this potential for failure becomes almost a guarantee. Most experienced project managers understand this and strain their sinews to prevent it from happening and most experienced programmers have lived through the intense disappointment of seeing their work fail to achieve its initial promise. Yet time and again, despite the best efforts of genuinely talented and motivated people, software projects are delivered late, cost too much and do not function as the client expected - Why is this?</p><p>For each failed software project the problem typically turns out to be <em>the plan</em>. Now that may seem trivially obvious. Looking back over a failed project it is easy to suggest that if only the plan had been more precise then the project could have been more controlled and so more successful.</p><p>This is not correct.</p><p>The problem does not lie in the quality of the planning, the problem lies in the type of plan, specifically the attempt to create an <em>up-front</em> plan that covers the entire project life-cycle. This is not so obvious - how can you run a project without deciding what you need <em>up-front</em>?</p><p>To understand why up-front planning impedes the successful delivery of quality software it is first necessary to understand what is meant by a plan in this traditional sense, and then see how this concept can be dispensed with and replaced with a new type of planning mechanism.</p><h2><span class="Apple-style-span" style="font-size:medium;">What's in a Plan</span></h2><p>At the start of a traditional project there is the familiar <em>requirements-capture</em> phase. This typically involves the writing of various specifications, a user specification that outlines the requirements in the language of the client, a functional specification that outlines the requirements in the language of the programmer and then perhaps a fully detailed technical specification that describes the requirements in a pseudo programming language.</p><p>Once complete, these detailed specifications provide the basis for all future work. They allow predictions to be made about the project's costs as well as its anticipated schedule. Specification documents also serve a secondary function. They give both the client and the engineers a form of 'contract' that, upon project delivery, allows everybody to compare what was promised with what was actually delivered.</p><p>This up-front planning process is often called the 'waterfall' model, it is a highly structured methodology that steps through requirements-capture, analysis, design, coding, and testing in a strict, pre-planned sequence. Progress is generally measured in terms of deliverable artifacts: requirement specifications, design documents, test plans and code reviews.</p><h2><span class="Apple-style-span" style="font-size:medium;">The Waterfall is Broken </span></h2><p>There are good reasons why traditional, up-front planning fails. Unfortunately these reasons tend to make both clients and engineers feel uncomfortable so they are rarely spoken out loud.</p><p>Firstly, up-front planning means that the specification documents are written before any software is built. Experts, using all their intellectual powers and experience, attempt to <em>imagine</em> the software and in doing so mentally traverse all of its myriad details. Since no software has yet been built, the hypothetical assertions contained within these documents cannot be tested experimentally. In science an hypothesis that cannot be tested is called <em>pseudo-science</em> and by the same token a specification whose assumptions cannot be tested should be considered <em>pseudo-planning</em>. </p><p>Secondly, at the start of any reasonably complex project there is always an inescapable knowledge gap. This gap exists between:</p><ul><br /><li>The business knowledge brought by the client</li><li>The technical knowledge brought by the engineers</li></ul><p>To begin with these two bodies of knowledge do not mix well as the clients do not really understand the language of software engineering and the engineers do not really understand the language of the client's specific business. This will change as time goes on and eventually the distinct bodies of information will mix and become one shared information landscape. However, at the start of a project when traditional up-front planning occurs, this inevitable knowledge gap leads to two critical and incorrect assumptions:</p><p style="padding-left: 30px;"><strong>1. The client knows what they want their new software to do<br /><span style="font-weight: normal;">Many clients come to a project with good idea of what they want, perhaps they have spent time and effort working this out, perhaps they have a legacy system that shows them much of they want and what they do not want. However at the start of a project the client cannot know what they want in sufficient detail to create a <em>complete</em> and <em>precise</em> plan. They can provide a business vision and they can provide business constraints but they cannot state in detail the processes required to deliver their vision because they have not yet absorbed the necessary details of the engineering environment. A superficial understanding can be gleaned during the initial planning meetings but this will not produce a sufficient understanding of the software they are commissioning. </span></strong></p><p style="padding-left: 30px;"><strong>2. The engineers know how to implement the client's business vision</strong><br />Many engineers come to a project with a good idea of how to build business systems. They will have spent considerable time and effort building other, perhaps similar systems. However at the start of a project engineers cannot know how to implement the precise details of a specific business application because they have not yet absorbed the detailed business knowledge brought by the client. A superficial understanding can be gleaned during the initial planning meetings but this will not produce a sufficient understanding of the software they are being asked to deliver.</p><p>Predictive planning fails because an accurate plan requires a genuine, non-superficial understanding of both the client's business knowledge and the engineer's technical knowledge. Traditional specifications are created at the start of a project when both parties have not had enough time to come to such an understanding. It takes much effort to synthesize the two bodies of knowledge into a coherent whole, far more than can reasonably be assigned during the requirements-capture phase. </p><p>This means that plans created at the start of the project cannot be more than partially informed guesswork. Given that the nature of complex systems make them particularly sensitive to changes in small details, a plan for a complex system created with incomplete knowledge must perforce be a recipe for failure by degrees.</p><p>Does this really make up-front planning redundant? Is there a way to make the synthesis of the client and technical knowledge more efficient, perhaps by using advanced planning software? If this could be achieved then perhaps the planners could write effective up-front specifications that lead to accurate long-term costings and schedules.</p><p>Unfortunately there is another, more fundamental reason why detailed specifications must fail - regardless of their precision.</p><p>A specification is a description that attempts to outline features and functions in a natural language such as English. Yet software is actually written in the very precise syntax of a machine language. Engineers know that only computer code can truly express the details of a software vision, a natural language specification cannot be logically accurate enough. This means that natural language specifications must leave many implementation details open to interpretation forcing the engineer to skilfully choose from a set of implied options. Yet complex systems are sensitive to precisely these sorts of technical details, different choices will lead to different systems and, as often as not, unfulfilled client expectations.</p><p>Therefore, even where a specification guesses correctly, the natural language descriptions will contain subtle choices and hidden contradictions. It is only when the fuzzy language of the specification is transformed into the precise reality of the code that these choices and contradictions become apparent.</p><p>This leads to a profound truth about the nature of specifications: G<em>reater precision does not lead to greater control. </em>Instead the greater the precision the more varied and subtle the choices and contradictions become.</p><h2><span class="Apple-style-span" style="font-size:medium;">Planning For Success</span></h2><p>Understanding these fundamental flaws at the heart of traditional software delivery, many forward looking managers and engineers are now moving towards a new project control methodology. In contrast to up-front or predictive planning this new methodology uses repeated bursts of short-term <em>adaptive </em>planning.</p><p><em>Agile Software Development</em> throws out long-term planning and with it the traditional concept of a specification. Instead agile projects start with everybody discussing and sharing a simple vision of the end product. The vision is really no more than a mission statement that, at this early stage, explicitly removes the need for engineers to fully understand the business and for the client to fully understand the technology.</p><p>This means that an agile project can get started almost straight away, with the absolute minimum of requirements-capture. Instead of a long, costly and ultimately self-defeating planning phase, the engineers get to work building the first version (iteration) of what will become a <em>rolling beta</em>. Armed with a very short term plan covering just one or two weeks of work, the engineers build the first iteration and deliver it to the client for discussion and criticism. The rolling-beta is still only a sketch, an outline of the most important functions and how they might fit together. Mistakes and incorrect assumptions will have been made, indeed given the knowledge gap they cannot be avoided, but the mistakes are identified and quickly eliminated as the rolling-beta is regularly assessed by the client and engineers in close collaboration.</p><p>Once the first iteration is signed-off then the process begins again, a new short term plan is created and work begins on the second iteration. This <em>iterative</em> development continues and as the knowledge gap closes so the requirements and hence the software become ever more detailed and coherent. </p><h2><span class="Apple-style-span" style="font-size:medium;">Embracing Function Creep</span></h2><p>As this hands-on process continues the client comes to properly understand the technical environment, what is expensive and what is possible, and as their knowledge grows so they begin to see new possibilities. </p><p>Clients changing their minds or adding new features during development is traditionally called <em>function creep</em> and remains the enemy of traditional planners. Yet to suppress this is to deny that clients can learn and modify their expectations as they see their software progressing. Rather than trying to ignore the client's input, the agile iterative process welcomes it as new and valuable knowledge. </p><p>Thus the client is encouraged to re-specify their product as it is being written. This is the ultimate guarantee that, in the end, the client will be satisfied. It is hard for a client to be surprised or disappointed with their software if they have played an active part in designing and deciding the goals at each iteration.</p><p>Equally, as the iterative process progresses the engineers will also come to a genuine understanding of the business. This allows the engineers to discuss the business processes with the client in a manner that allows a useful exchange of knowledge to take place. Questions to the client can be appropriately framed using the business terminology both the client and the engineers now share. Since the frequent iterations and short-term planning means that any incorrect business assumptions are quickly discovered, such mistakes can be corrected with the minimum of effort. </p><p>Engineers too, once they come to a genuine understanding of the business, can start to usefully contribute to the re-specification of the rolling-beta. New ideas and inspirations, whatever their source, can be welcomed, discussed and possibly incorporated as the software adapts over time.</p><h2><span class="Apple-style-span" style="font-size:medium;">Job Satisfaction</span></h2><p>In summary, an agile software system evolves under the twin constraints of the client's business vision and the engineering environment's technical limitations. As the client and engineers come to a mutual understanding so new ideas bubble up and are incorporated as bad old ideas are identified and discarded. Before starting each iteration everybody discusses, negotiates and quickly reaches an understanding of what is <em>actually required</em> to fulfil the next set of short-term goals. </p><p></p><blockquote><i>Thus an agile system organically grows its natural complexity out of a fundamental simplicity. As a result there are fewer surprises, the project risks are minimised and the client is more likely to get software that works.</i></blockquote><p></p><p><br /></p>biofractalhttp://www.blogger.com/profile/11229842750489025189noreply@blogger.com7tag:blogger.com,1999:blog-6927558.post-47219640082621866502009-04-28T20:38:00.166+01:002010-08-27T16:33:51.923+01:00Domain Driven RIA: Managing Deep Object Graphs in Silverlight-3<em><span class="Apple-style-span" style="font-size:large;">Using RIA Services, can a simple n-tier application manage a deep object graph with eager fetching, lazy loading and silverlight databinding?</span></em><div><br /></div><div><br /><object width="400" height="300"><param name="allowfullscreen" value="true"><param name="allowscriptaccess" value="always"><param name="movie" value="http://vimeo.com/moogaloop.swf?clip_id=4589802&server=vimeo.com&show_title=1&show_byline=1&show_portrait=0&color=&fullscreen=1"><embed src="http://vimeo.com/moogaloop.swf?clip_id=4589802&server=vimeo.com&show_title=1&show_byline=1&show_portrait=0&color=&fullscreen=1" type="application/x-shockwave-flash" allowfullscreen="true" allowscriptaccess="always" width="400" height="300"></embed></object><br /><br /><div><span class="Apple-style-span" style="font-weight: bold;">Downloads</span><br /><ul><li><a href="https://www.assembla.com/code/biofractal/subversion/nodes/trunk/Blog/DomainDrivenRIA/DeepLazyBinding">The Source Code from Assembla</a><br /></li><li><a href="http://cid-07794e6d29b7e05c.skydrive.live.com/self.aspx/.Public/DeepLazyBinding.mp4?sa=84332759">The screencast: A detailed walk-through</a> [duration 28 minutes]</li></ul><div><br /><span class="Apple-style-span" style="color: rgb(204, 0, 0);"><span class="Apple-style-span" style="font-weight: bold;">Note</span></span>: If you have no experience with RIA Services then you may prefer to start with my previous demo, <a href="http://biofractal.blogspot.com/2009/03/domain-driven-db4o-silverlight-3-ria.html">A Domain-Driven DB4O Silverlight-3 RIA</a>, which has links to RIA Services documentation and Microsoft presentations to get you started.<br /><br /><strong>Introduction</strong><br />RIA Services is a Rich Internet Application (RIA) framework that promises to streamline n-tier Line of Business application development. Reading through the <a href="http://download.microsoft.com/download/F/B/8/FB8CA635-296B-487F-965C-8148F08B5319/riaservicesoverviewpreview.pdf">RIA documentation</a> and listening to the <a href="http://videos.visitmix.com/MIX09/T40F">RIA team's presentations</a> I was struck by two things:<div><ul><li>How potentially useful this framework was. <br /></li><li>How skewed the material was in favour of a <a href="http://en.wikipedia.org/wiki/Data-driven_design">data-driven design</a> approach<br /></li></ul><div>In this post I want to investigate how RIA Services can be used in a Domain-Driven context with a special focus on how it can help with the eager and lazy loading of domain entities.<br /><br /><span class="Apple-style-span" style="font-weight: bold;">Where is the Database?</span></div><div>I have chosen not to use a relational database in this example. This is because I want to ensure that my domain instance data can be easily stored and retrieved in the most efficient and maintainable <span class="Apple-style-span" style="font-style: italic;">domain-centric</span> manner. I have therefore elected to use an object datastore, in this case <a href="http://www.db4o.com/">DB4O</a>, which provides all the ease, speed and functionality I need. For more information see:</div><div><div><ul><li><a href="http://biofractal.blogspot.com/2009/02/irrational-love-of-relational.html">An Irrational Love of the Relational</a><br /></li><li><a href="http://biofractal.blogspot.com/2009/02/data-driven-conspiracy.html">The Data Driven Conspiracy</a><br /></li></ul></div><div><strong>The Technology Stack</strong><ul><li><a href="http://silverlight.net/getstarted/silverlight3/default.aspx">Silverlight 3</a><br />Handles the client-side application logic and user interface</li><li><a href="http://www.nikhilk.net/RIA-Services-MIX09.aspx">RIA Services</a><br />Provides the client<->server interaction and client-side domain</li><li><a href="http://www.db4o.com/">DB4O</a><br />A server-side datastore for domain entity de/serialization</li></ul><strong>The Software</strong><br />Here is a sneak preview of the software in action.<br /><br /><a href="http://1.bp.blogspot.com/_kbCCKz8F0ok/Sfi_kdkln7I/AAAAAAAAACc/lNKf-0xnMhY/s1600-h/DeepLazyBinding_UI.png"><img style="TEXT-ALIGN: center; MARGIN: 0px auto 10px; WIDTH: 400px; DISPLAY: block; HEIGHT: 314px; CURSOR: hand" id="BLOGGER_PHOTO_ID_5330220792127266738" border="0" alt="" src="http://1.bp.blogspot.com/_kbCCKz8F0ok/Sfi_kdkln7I/AAAAAAAAACc/lNKf-0xnMhY/s400/DeepLazyBinding_UI.png" /></a><br /><br /><strong>The Objectives</strong><br />Using a combination of RIA Services and DB4O I want to test the following:<ul><li>Server - When I fetch an instance of the <a href="http://www.lostechies.com/blogs/jimmy_bogard/archive/2008/05/20/entities-value-objects-aggregates-and-roots.aspx">aggregrate root</a> class I expect its inner hierarchy be <span class="Apple-style-span" style="font-style: italic;">eagerly fetched.</span><br /><br /></li><li>Client - I want certain collections to be <span class="Apple-style-span" style="font-style: italic;">lazy-loaded</span> and so remain unloaded until they are requested.<br /><br /></li><li>I do not expect to write my own <a href="http://en.wikipedia.org/wiki/Windows_Communication_Foundation">WCF Service</a> nor do I want to write and <a href="http://en.wikipedia.org/wiki/Data_transfer_object">Data Transfer Objects</a> (DTOs).<br /><br /></li><li>I want to databind my domain entities to silverlight controls. I expect the controls to correctly display my eagerly fetched data as well as handling lazy-loaded data.<br /><br /></li><li>Finally, I want to prove that new domain entities can be created on the client and efficiently serialized to the server-side data-store as a batched <a href="http://framework.zend.com/wiki/display/ZFDEV/Unit+of+Work+pattern">unit-of-work</a></li></ul><br /><strong>The Domain</strong><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/_kbCCKz8F0ok/SgiSZi7tgXI/AAAAAAAAADM/7D3cCE1q2s8/s1600-h/DeepLazyBindingDomain.png"><img style="cursor:pointer; cursor:hand;width: 219px; height: 383px;" src="http://2.bp.blogspot.com/_kbCCKz8F0ok/SgiSZi7tgXI/AAAAAAAAADM/7D3cCE1q2s8/s400/DeepLazyBindingDomain.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5334674726191989106" /></a><br /><span><span><br />I have a small hierarchical domain consisting of a single <span class="Apple-style-span" style="font-style: italic;">User</span> <a href="http://domaindrivendesign.org/node/88">aggregate</a> root that bounds a one-to-many inner collection of <span class="Apple-style-span" style="font-style: italic;">Holding</span> <a href="http://domaindrivendesign.org/node/109">Entities</a> that each contain a further collection of <span class="Apple-style-span" style="font-style: italic;">Transaction</span> entities. </span></span></div><div><br /></div><div>The test domain therefore consists of the following hierarchy:</div><div><ul><li><span class="Apple-style-span" style=""><span class="Apple-style-span" style="font-weight: bold;">User.Holdings[n].Transactions[n]</span></span><br /></li></ul></div><div>Here is the code for the domain hierarchy.<br /></div><div><br /><div class="csharpcode"><pre class="alt"><span class="lnum"> 3: </span><span class="kwrd">public</span> <span class="kwrd">abstract</span> <span class="kwrd">class</span> Entity</pre><pre><span class="lnum"> 4: </span>{</pre><pre class="alt"><span class="lnum"> 5: </span> [Key]</pre><pre><span class="lnum"> 6: </span> <span class="kwrd">public</span> Guid Id { get; set; }</pre><pre class="alt"><span class="lnum"> 7: </span>}</pre><pre><span class="lnum"> 8: </span> </pre><pre class="alt"><span class="lnum"> 9: </span><span class="kwrd">public</span> <span class="kwrd">partial</span> <span class="kwrd">class</span> User : Entity, IAggregateRoot</pre><pre><span class="lnum"> 10: </span>{</pre><pre class="alt"><span class="lnum"> 11: </span> <span class="kwrd">public</span> <span class="kwrd">string</span> Name { get; set; }</pre><pre><span class="lnum"> 12: </span> <span class="kwrd">public</span> <span class="kwrd">string</span> Password { get; set; }</pre><pre class="alt"><span class="lnum"> 13: </span> <span class="kwrd">private</span> List<Holding> _holdings = <span class="kwrd">new</span> List<Holding>();</pre><pre><span class="lnum"> 14: </span> <span class="kwrd">public</span> List<Holding> Holdings</pre><pre class="alt"><span class="lnum"> 15: </span> {</pre><pre><span class="lnum"> 16: </span> get { <span class="kwrd">return</span> <span class="kwrd">this</span>._holdings; }</pre><pre class="alt"><span class="lnum"> 17: </span> set { <span class="kwrd">this</span>._holdings = <span class="kwrd">value</span>; }</pre><pre><span class="lnum"> 18: </span> }</pre><pre class="alt"><span class="lnum"> 19: </span>}</pre><pre><span class="lnum"> 20: </span> </pre><pre class="alt"><span class="lnum"> 21: </span><span class="kwrd">public</span> <span class="kwrd">partial</span> <span class="kwrd">class</span> Holding : Entity</pre><pre><span class="lnum"> 22: </span>{</pre><pre class="alt"><span class="lnum"> 23: </span> <span class="kwrd">public</span> Guid UserId { get; set; }</pre><pre><span class="lnum"> 24: </span> <span class="kwrd">public</span> <span class="kwrd">string</span> Symbol { get; set; }</pre><pre class="alt"><span class="lnum"> 25: </span> <span class="kwrd">private</span> List<Transaction> _transactions = <span class="kwrd">new</span> List<Transaction>();</pre><pre><span class="lnum"> 26: </span> <span class="kwrd">public</span> List<Transaction> Transactions</pre><pre class="alt"><span class="lnum"> 27: </span> {</pre><pre><span class="lnum"> 28: </span> get { <span class="kwrd">return</span> <span class="kwrd">this</span>._transactions; }</pre><pre class="alt"><span class="lnum"> 29: </span> set { <span class="kwrd">this</span>._transactions = <span class="kwrd">value</span>; }</pre><pre><span class="lnum"> 30: </span> }</pre><pre class="alt"><span class="lnum"> 31: </span>}</pre><pre><span class="lnum"> 32: </span></pre><pre class="alt"><span class="lnum"> 33: </span><span class="kwrd">public</span> <span class="kwrd">class</span> Transaction : Entity</pre><pre><span class="lnum"> 34: </span>{</pre><pre class="alt"><span class="lnum"> 35: </span> <span class="kwrd">public</span> Guid HoldingId { get; set; }</pre><pre><span class="lnum"> 36: </span> <span class="kwrd">public</span> TransactionType Type { get; set; }</pre><pre class="alt"><span class="lnum"> 37: </span> <span class="kwrd">public</span> <span class="kwrd">int</span> Quantity { get; set; }</pre><pre><span class="lnum"> 38: </span> <span class="kwrd">public</span> <span class="kwrd">decimal</span> Price { get; set; }</pre><pre class="alt"><span class="lnum"> 39: </span>}</pre></div><div><br /><span class="Apple-style-span" style="font-weight: bold; ">Domain Loading Strategy</span></div><div><ul><li><span class="Apple-style-span" style="font-style: italic; ">Server</span><br />Fetching a User should eagerly fetch all of its dependent Holdings. Each Holding should eagerly fetch all its dependent Transactions.<br /><br /></li><li><span class="Apple-style-span" style="font-style: italic; ">Client</span><br />Fetching a User should eagerly fetch all of its dependent Holdings. However due to the potential for large numbers of Transactions, each Holding should <span class="Apple-style-span" style="font-style: italic; ">not </span>fetch any Transactions instead the Transactions collection must be <span class="Apple-style-span" style="font-style: italic; ">lazy-loaded</span>.</li></ul></div><br /><strong>The Datastore Setup</strong></div><div>Before plunging in to the RIA Services code I want to show you just how easy it is to use the <a href="http://www.db4o.com/">DB4O </a>object database. </div><div><br /></div><div>In the web.config there are two application settings (shown below). </div><div><ol><li><span class="Apple-style-span" style="font-style: italic;">DataFile.Name</span><br />Specifies the name of the DB4O datastore file held in the App_Data folder<br /><br /></li><li><span class="Apple-style-span" style="font-style: italic;">DataFile.GenerateSampleData</span><br />Determines whether the datastore is reset with newly generated sample data whenever the Cassini web application is re-started (useful for testing). <br /></li></ol></div><div><span class="Apple-style-span" style="color: rgb(204, 0, 0);"><span class="Apple-style-span" style="font-weight: bold;">Important</span></span>: Ensure the <span class="Apple-style-span" style="font-style: italic;">DataFile.GenerateSampleData</span> setting is <span class="Apple-style-span" style="font-weight: bold;">false </span>if you want to retain any changes between application runs.<br /><br /><div class="csharpcode"><pre class="alt"><span class="lnum"> 1: </span><appSettings></pre><pre><span class="lnum"> 2: </span> <add key=<span class="str">"DataFile.Name"</span> <span class="kwrd">value</span>=<span class="str">"DataStore.db4o"</span>/></pre><pre class="alt"><span class="lnum"> 3: </span> <add key=<span class="str">"DataFile.GenerateSampleData"</span> <span class="kwrd">value</span>=<span class="str">"true"</span>/></pre><pre><span class="lnum"> 4: </span></appSettings></pre><pre class="alt"><span class="lnum"> 5: </span></pre><pre><span class="lnum"> 6: </span><span class="kwrd">public</span> <span class="kwrd">static</span> <span class="kwrd">void</span> ServerOpen()</pre><pre class="alt"><span class="lnum"> 7: </span>{</pre><pre><span class="lnum"> 8: </span> <span class="kwrd">if</span> (db4oServer != <span class="kwrd">null</span>)</pre><pre class="alt"><span class="lnum"> 9: </span> {</pre><pre><span class="lnum"> 10: </span> <span class="kwrd">return</span>;</pre><pre class="alt"><span class="lnum"> 11: </span> }</pre><pre><span class="lnum"> 12: </span> </pre><pre class="alt"><span class="lnum"> 13: </span> var filename = Path.Combine(HttpContext.Current.Server.MapPath(<span class="str">"~/App_Data"</span>), ConfigFileName);</pre><pre><span class="lnum"> 14: </span> </pre><pre class="alt"><span class="lnum"> 15: </span> var generateSampleData = <span class="kwrd">bool</span>.Parse(GenerateSampleData);</pre><pre><span class="lnum"> 16: </span> <span class="kwrd">if</span> (generateSampleData && File.Exists(filename))</pre><pre class="alt"><span class="lnum"> 17: </span> {</pre><pre><span class="lnum"> 18: </span> File.Delete(filename);</pre><pre class="alt"><span class="lnum"> 19: </span> }</pre><pre><span class="lnum"> 20: </span> db4oServer = Db4oFactory.OpenServer(GetConfig(), filename, 0);</pre><pre class="alt"><span class="lnum"> 21: </span> <span class="kwrd">if</span> (generateSampleData)</pre><pre><span class="lnum"> 22: </span> {</pre><pre class="alt"><span class="lnum"> 23: </span> SampleData.Generate();</pre><pre><span class="lnum"> 24: </span> }</pre><pre class="alt"><span class="lnum"> 25: </span>}</pre></div><br />In order to create the server-side eager fetch strategy outlined above, the DB4O datastore requires some configuration. The following <span class="Apple-style-span" style="font-style: italic;">GetConfig()</span> method shows the Domain being scanned for types that implement <span class="Apple-style-span" style="font-style: italic;">IAggregateRoot </span>with DB4O instructed to automatically fetch, save and delete the inner dependecies for those types.<br /><br /><div class="csharpcode"><pre class="alt"><span class="lnum"> 1: </span><span class="kwrd">private</span> <span class="kwrd">static</span> IConfiguration GetConfig()</pre><pre><span class="lnum"> 2: </span>{</pre><pre class="alt"><span class="lnum"> 3: </span> var config = Db4oFactory.NewConfiguration();</pre><pre><span class="lnum"> 4: </span> config.UpdateDepth(2);</pre><pre class="alt"><span class="lnum"> 5: </span> var types = Assembly.GetExecutingAssembly().GetTypes();</pre><pre><span class="lnum"> 6: </span> <span class="kwrd">for</span> (var i = 0; i < types.Length; i++)</pre><pre class="alt"><span class="lnum"> 7: </span> {</pre><pre><span class="lnum"> 8: </span> var type = types[i];</pre><pre class="alt"><span class="lnum"> 9: </span> <span class="kwrd">if</span> (type.GetInterface(<span class="kwrd">typeof</span> (IAggregateRoot).Name) == <span class="kwrd">null</span>)</pre><pre><span class="lnum"> 10: </span> {</pre><pre class="alt"><span class="lnum"> 11: </span> <span class="kwrd">continue</span>;</pre><pre><span class="lnum"> 12: </span> }</pre><pre class="alt"><span class="lnum"> 13: </span> var objectClass = config.ObjectClass(type);</pre><pre><span class="lnum"> 14: </span> objectClass.CascadeOnUpdate(<span class="kwrd">true</span>);</pre><pre class="alt"><span class="lnum"> 15: </span> objectClass.CascadeOnActivate(<span class="kwrd">true</span>);</pre><pre><span class="lnum"> 16: </span> objectClass.CascadeOnDelete(<span class="kwrd">true</span>);</pre><pre class="alt"><span class="lnum"> 17: </span> objectClass.Indexed(<span class="kwrd">true</span>);</pre><pre><span class="lnum"> 18: </span> }</pre><pre class="alt"><span class="lnum"> 19: </span> <span class="kwrd">return</span> config;</pre><pre><span class="lnum"> 20: </span>}</pre></div><br /><strong>RIA Services</strong><br />N-Tier applications are defined by the machine boundary that exists between the client and the server. Getting to grips with RIA Services begins by understanding how it tries to help you write applications that span that machine boundary. </div><div><br /></div><div>As you write your server-side domain code RIA Services tries to discover the way you <span class="Apple-style-span" style="font-style: italic;">intend to use</span> this domain on the client. As it does so it generates a client-side version of your domain that fulfils those <span class="Apple-style-span" style="font-style: italic;">intentions</span>. This means that you do not need to write a client-side version of your domain in order to use its features on the client, nor do you need to write any explicit mechanism for transferring domain instance data across the machine boundary (no WCF, no DTOs).</div><div><br />RIA Services discovers your intentions via a combination of <span class="Apple-style-span" style="font-style: italic;">Convention </span>and <span class="Apple-style-span" style="font-style: italic;">Metadata</span>. For example, I intend to utilize my User class on the client and so I need to be able to fetch User instances from the data store. This implies that somewhere I must write a server-side service method to perform the User fetch.<br /><br />RIA Services simply asks that I put that User fetch service method in a class that derives from the RIA <span class="Apple-style-span" style="font-style: italic;">DataService</span> class and that I follow some simple naming rules for the method signature. For more information on these conventions see <a href="http://us.brd.yahoo.com/_ylt=A0wNBqCMpAFKou0A4Y1hRh54;_ylv=0/SIG=13li2378l/**http%3A//download.microsoft.com/download/F/B/8/FB8CA635-296B-487F-965C-8148F08B5319/riaservicesoverviewpreview.pdf">.NET RIA Services Overview for Mix 2009 Preview</a></div><div><br /></div><div>If I follow the prescribed <span class="Apple-style-span" style="font-style: italic;">conventions</span> then RIA will be able to determine that I intend utilizing the User class on the client and so generate a client-side version of my User class. This generated version is not the same class as my 'real' server-side User class, it only has as much or as little functionality as I decide to share (see later) but it does allow the client-code to operate as if I had access to the User class so I can use it in my silverlight code.<br /><br />This is what the conventional User fetch method looks like.<br /><br /><div class="csharpcode"><pre class="alt"><span class="lnum"> 1: </span>[EnableClientAccess]</pre><pre><span class="lnum"> 2: </span><span class="kwrd">public</span> <span class="kwrd">class</span> DataStore : DomainService</pre><pre class="alt"><span class="lnum"> 3: </span>{</pre><pre><span class="lnum"> 4: </span> </pre><pre class="alt"><span class="lnum"> 5: </span> <span class="kwrd">public</span> IQueryable<User> GetUser(<span class="kwrd">string</span> name, <span class="kwrd">string</span> password)</pre><pre><span class="lnum"> 6: </span> {</pre><pre class="alt"><span class="lnum"> 7: </span> <span class="kwrd">using</span> (var db = DataService.GetSession<User>())</pre><pre><span class="lnum"> 8: </span> {</pre><pre class="alt"><span class="lnum"> 9: </span> <span class="kwrd">return</span> db.GetList(x => x.Name.Equals(name) && x.Password.Equals(password)).AsQueryable();</pre><pre><span class="lnum"> 10: </span> }</pre><pre class="alt"><span class="lnum"> 11: </span> }</pre><pre><span class="lnum"> 12: </span> ... other code</pre><pre class="alt"><span class="lnum"> 13: </span>}</pre></div><br />The presence of this method stimulates RIA into generating a client-side version of my User class however it will only carry over simple properties such as the <span class="Apple-style-span" style="font-style: italic;">User.Name </span><span class="Apple-style-span" style="">and </span><span class="Apple-style-span" style="font-style: italic;">User.Password</span>. So what happens if I want to make client-side use of a more complex property such as the <span class="Apple-style-span" style="font-style: italic;">User.Holdings</span> collection? </div><div><br /></div><div>This is a new intention so I must tell RIA about it. Only then can RIA generate the appropriate client-side code to fulfil the new intention.<br /><br />This is achieved in two steps.<ol><li>The Holding class must define a UserId property. When a new Holding is instantiated this property must be set to the Id of its parent User</li><br /><li>The User.Holdings Collection must be decorated with the appropriate attributes.</li></ol>To decorate a server-side domain entity with attributes targeted to client-side behaviour seems impure but fortunately RIA provides a pattern that brushes it all under the carpet and allows you to retain your domain-driven dignity. </div><div><br /></div><div>First you must ensure the main User class is <span class="Apple-style-span" style="font-style: italic;">partial</span>. This allows you to create a new partial User segment in a separate code file called <span class="Apple-style-span" style="font-style: italic;">User.meta.cs</span>. You can then add the following code to that file. In this way you can keep all the RIA meta-data tucked away in their own partial file segments.<br /><br /><div class="csharpcode"><pre class="alt"><span class="lnum"> 1: </span>[MetadataType(<span class="kwrd">typeof</span> (UserMetadata))]</pre><pre><span class="lnum"> 2: </span><span class="kwrd">public</span> <span class="kwrd">partial</span> <span class="kwrd">class</span> User</pre><pre class="alt"><span class="lnum"> 3: </span>{</pre><pre><span class="lnum"> 4: </span> <span class="kwrd">internal</span> <span class="kwrd">sealed</span> <span class="kwrd">class</span> UserMetadata</pre><pre class="alt"><span class="lnum"> 5: </span> {</pre><pre><span class="lnum"> 6: </span> [Include]</pre><pre class="alt"><span class="lnum"> 7: </span> [Association(<span class="str">"User_Holdings"</span>, <span class="str">"Id"</span>, <span class="str">"UserId"</span>)]</pre><pre><span class="lnum"> 8: </span> <span class="kwrd">public</span> List<Holding> Holdings { get; set; }</pre><pre class="alt"><span class="lnum"> 9: </span> }</pre><pre><span class="lnum"> 10: </span>}</pre></div><br />You will note there are two attributes being used here. What are they doing? </div><div><ul><li><span class="Apple-style-span" style="font-style: italic;">[Association]</span><br />This attribute is informing RIA that the Holdings collection can be reconstructed on the client by comparing the User.Id to the Holding.UserId. When these match the Holding belongs to the collection. <br /><br /></li><li><span class="Apple-style-span" style="font-style: italic;">[Include]</span><br />This attribute is more mysterious. Perhaps, like me, you might assume it means "Include this property in the generated code". This is not correct. In fact it means "Automatically recreate this collection on the client", in other words the client-side collection will be eagerly fetched and made available without any further intervention on your part. This is the behaviour we want for the User.Holdings collection and gives us our first clue about how we might set up the lazy loading for the Holding.Transactions collection.</li></ul></div><div><br /></div><div>RIA allows us to define the shape of our hierarchy on the client using a combination of <span class="Apple-style-span" style="font-style: italic;">convention </span>for the fetch method signatures and <span class="Apple-style-span" style="font-style: italic;">metadata </span>using the [Include] + [Association] attributes. But a class must also define functionality or it is just a DTO. </div><div><br /></div><div>Can I pick and choose the functions I want to appear in the client-side versions of my domain entities?<br /><br /><strong>Sharing Domain Functions</strong><br />On the client I want to add a new Holding to my User.Holdings collection. Being a conscientious domain-driven coder I want to ensure that my code follows the <a href="http://en.wikipedia.org/wiki/Law_Of_Demeter">Law of Demeter</a>, which means I cannot reach into the Holdings collection directly like this:<br /><br /><span class="Apple-style-span" style="font-style: italic;">User.Holdings.Add(...)</span><br /><br />Instead I need to write a method to do this for me:<br /><br /><span class="Apple-style-span" style="font-style: italic;">User.AddHolding(...)</span><br /><br />This is easy to write for my server-side domain but if I intend the same features to be available on the client I must tell RIA services about those intentions and so allow it to generate the appropriate client-side code.<ol><li>Ensure the class with shared features is partial<br /><br /></li><li>Put the shared code in a partial segment stored a code file called <span class="Apple-style-span" style="font-style: italic;">MyClass.shared.cs<br /><br /></span></li><li>Decorate the shared methods with the [Shared] attribute<br /><br /></li></ol>Here is the code for the shared AddHolding method held in the <span class="Apple-style-span" style="font-style: italic;">User.shared.cs</span> file.<br /><br /><div class="csharpcode"><pre class="alt"><span class="lnum"> 1: </span><span class="kwrd">public</span> <span class="kwrd">partial</span> <span class="kwrd">class</span> User</pre><pre><span class="lnum"> 2: </span>{</pre><pre class="alt"><span class="lnum"> 3: </span> [Shared]</pre><pre><span class="lnum"> 4: </span> <span class="kwrd">public</span> Holding AddHolding(Holding holding)</pre><pre class="alt"><span class="lnum"> 5: </span> {</pre><pre><span class="lnum"> 6: </span> <span class="kwrd">this</span>.Holdings.Add(holding);</pre><pre class="alt"><span class="lnum"> 7: </span> <span class="kwrd">return</span> holding;</pre><pre><span class="lnum"> 8: </span> }</pre><pre class="alt"><span class="lnum"> 9: </span>}</pre></div><strong><br />More Shared Code</strong><br />When I create a new Holding I would prefer to use a <a href="http://en.wikipedia.org/wiki/Factory_method">factory method</a> found in my DomainFactory class. This is a useful method so I want it to be available on the client as well as the server. As it happens the Factory class also contains a number of methods I would like to share, so instead of creating a partial class and sharing out individual methods as before I can just share the entire Factory class.<br /><br />The following code is held in a file <span class="Apple-style-span" style="font-style: italic;">DomainFactory.shared.cs<br /><br /></span><div class="csharpcode"><pre class="alt"><span class="lnum"> 1: </span>[Shared]</pre><pre><span class="lnum"> 2: </span><span class="kwrd">public</span> <span class="kwrd">class</span> DomainFactory</pre><pre class="alt"><span class="lnum"> 3: </span>{</pre><pre><span class="lnum"> 4: </span> </pre><pre class="alt"><span class="lnum"> 5: </span> [Shared]</pre><pre><span class="lnum"> 6: </span> <span class="kwrd">public</span> <span class="kwrd">static</span> User User(<span class="kwrd">string</span> name, <span class="kwrd">string</span> password)</pre><pre class="alt"><span class="lnum"> 7: </span> {</pre><pre><span class="lnum"> 8: </span> <span class="kwrd">return</span> <span class="kwrd">new</span> User</pre><pre class="alt"><span class="lnum"> 9: </span> {</pre><pre><span class="lnum"> 10: </span> Id = Guid.NewGuid(),</pre><pre class="alt"><span class="lnum"> 11: </span> Name = name,</pre><pre><span class="lnum"> 12: </span> Password = password,</pre><pre class="alt"><span class="lnum"> 13: </span> };</pre><pre><span class="lnum"> 14: </span> }</pre><pre class="alt"><span class="lnum"> 15: </span> </pre><pre><span class="lnum"> 16: </span> [Shared]</pre><pre class="alt"><span class="lnum"> 17: </span> <span class="kwrd">public</span> <span class="kwrd">static</span> Holding Holding(User user, <span class="kwrd">string</span> symbol)</pre><pre><span class="lnum"> 18: </span> {</pre><pre class="alt"><span class="lnum"> 19: </span> <span class="kwrd">return</span> <span class="kwrd">new</span> Holding</pre><pre><span class="lnum"> 20: </span> {</pre><pre class="alt"><span class="lnum"> 21: </span> Id = Guid.NewGuid(),</pre><pre><span class="lnum"> 22: </span> UserId = user.Id,</pre><pre class="alt"><span class="lnum"> 23: </span> Symbol = symbol</pre><pre><span class="lnum"> 24: </span> };</pre><pre class="alt"><span class="lnum"> 25: </span> }</pre><pre><span class="lnum"> 26: </span> </pre><pre class="alt"><span class="lnum"> 27: </span> [Shared]</pre><pre><span class="lnum"> 28: </span> <span class="kwrd">public</span> <span class="kwrd">static</span> Transaction Transaction(Holding holding, TransactionType type, <span class="kwrd">int</span> quantity, <span class="kwrd">decimal</span> price)</pre><pre class="alt"><span class="lnum"> 29: </span> {</pre><pre><span class="lnum"> 30: </span> <span class="kwrd">return</span> <span class="kwrd">new</span> Transaction</pre><pre class="alt"><span class="lnum"> 31: </span> {</pre><pre><span class="lnum"> 32: </span> Id = Guid.NewGuid(),</pre><pre class="alt"><span class="lnum"> 33: </span> HoldingId = holding.Id,</pre><pre><span class="lnum"> 34: </span> Type = type,</pre><pre class="alt"><span class="lnum"> 35: </span> Quantity = quantity,</pre><pre><span class="lnum"> 36: </span> Price = price</pre><pre class="alt"><span class="lnum"> 37: </span> };</pre><pre><span class="lnum"> 38: </span> }</pre><pre class="alt"><span class="lnum"> 39: </span>}</pre></div><strong><br />Some Client-Side Code</strong><br />Now we have informed RIA about our intentions it is time to see some client-side code that shows the resulting RIA generated client domain in use. This code is taken from the silverlight application that accompanies the web application.<br /><br />First of all, here is the code that does some setup and then the initial fetch for the User.<br /><br /><div class="csharpcode"><pre class="alt"><span class="lnum"> 1: </span><span class="kwrd">public</span> HomePage()</pre><pre><span class="lnum"> 2: </span>{</pre><pre class="alt"><span class="lnum"> 3: </span> <span class="kwrd">this</span>.InitializeComponent();</pre><pre><span class="lnum"> 4: </span> <span class="kwrd">this</span>._dataStore.Submitted += <span class="kwrd">this</span>.DataStoreSubmitted;</pre><pre class="alt"><span class="lnum"> 5: </span> <span class="kwrd">this</span>._dataStore.Loaded += <span class="kwrd">this</span>.DataStoreLoaded;</pre><pre><span class="lnum"> 6: </span> <span class="kwrd">this</span>._dataStore.LoadUser(<span class="str">"biofractal"</span>, <span class="str">"x"</span>, <span class="kwrd">null</span>, <span class="str">"LoadUser"</span>);</pre><pre class="alt"><span class="lnum"> 7: </span> <span class="kwrd">this</span>.Holdings.SelectionChanged += <span class="kwrd">this</span>.Holdings_SelectionChanged;</pre><pre><span class="lnum"> 8: </span>}<span class="lnum"> </span> </pre><pre><span class="lnum"> 12: </span><span class="kwrd">private</span> <span class="kwrd">void</span> DataStoreLoaded(<span class="kwrd">object</span> sender, LoadedDataEventArgs e)</pre><pre class="alt"><span class="lnum"> 13: </span>{</pre><pre><span class="lnum"> 14: </span> var userState = e.UserState;</pre><pre class="alt"><span class="lnum"> 15: </span> <span class="kwrd">if</span>(userState==<span class="kwrd">null</span>)</pre><pre><span class="lnum"> 16: </span> {</pre><pre class="alt"><span class="lnum"> 17: </span> <span class="kwrd">return</span>;</pre><pre><span class="lnum"> 18: </span> }</pre><pre class="alt"><span class="lnum"> 19: </span> <span class="kwrd">switch</span> (userState.ToString())</pre><pre><span class="lnum"> 20: </span> {</pre><pre class="alt"><span class="lnum"> 21: </span> <span class="kwrd">case</span> <span class="str">"LoadUser"</span>:</pre><pre><span class="lnum"> 22: </span> var user = e.LoadedEntities.First() <span class="kwrd">as</span> User;</pre><pre class="alt"><span class="lnum"> 23: </span> <span class="kwrd">this</span>.User.DataContext = user;</pre><pre><span class="lnum"> 24: </span> <span class="kwrd">this</span>.Holdings.ItemsSource = user.Holdings;</pre><pre class="alt"><span class="lnum"> 25: </span> <span class="kwrd">break</span>;</pre><pre><span class="lnum"> 26: </span> }</pre><pre class="alt"><span class="lnum"> 27: </span>}</pre></div><br />The _dataStore variable references an instance of the DataStore class which is derived from the RIA client-side <span class="Apple-style-span" style="font-style: italic;">DomainContext</span> class. This class is auto-generated by RIA Services. It is the primary RIA generated artefact.<br /><br />The <span class="Apple-style-span" style="font-style: italic;">DataStore.LoadUser()</span> calls the <span class="Apple-style-span" style="font-style: italic;">GetUser()</span> service method on the server. This is an asynchronous service call so the return must be caught in the <span class="Apple-style-span" style="font-style: italic;">DataStore.Loaded()</span> event handler. Here the silverlight controls can be data-bound to their data sources and, because the User.Holdings collection was decorated with the [Include] attribute, RIA will ensure that it is automatically fetched. Using the Holdings collection as a binding data source will therefore display the correct list of Holdings for the current User without requiring an explicit fetch.<br /><br /><strong>Lazy Loading the Transactions</strong><br />In contrast to the User.Holdings collection, the Holding.Transactions collection is not automatically loaded when the User is initially fetched. Instead the client-side domain behaviour requires that the Transactions collection is lazy loaded on-demand. How is this achieved using RIA Services?<br /><br />As before, the metadata is used to inform RIA of our intentions. The [Association] attribute is again used to decorate the collection definition in a partial class segment held in distinct code file (<span class="Apple-style-span" style="font-style: italic;">Holding.meta.cs</span>). However this time there is no [Include] attribute.<br /><br /></div><div><div class="csharpcode"><pre class="alt"><span class="lnum"> 1: </span>[MetadataType(<span class="kwrd">typeof</span> (HoldingMetadata))]</pre><pre><span class="lnum"> 2: </span><span class="kwrd">public</span> <span class="kwrd">partial</span> <span class="kwrd">class</span> Holding</pre><pre class="alt"><span class="lnum"> 3: </span>{</pre><pre><span class="lnum"> 4: </span> <span class="kwrd">internal</span> <span class="kwrd">sealed</span> <span class="kwrd">class</span> HoldingMetadata</pre><pre class="alt"><span class="lnum"> 5: </span> {</pre><pre><span class="lnum"> 6: </span> <span class="preproc">#region</span> Properties</pre><pre class="alt"><span class="lnum"> 7: </span></pre><pre><span class="lnum"> 8: </span> [Association(<span class="str">"Holding_Transactions"</span>, <span class="str">"Id"</span>, <span class="str">"HoldingId"</span>)]</pre><pre class="alt"><span class="lnum"> 9: </span> <span class="kwrd">public</span> List<Transaction> Transactions { get; set; }</pre><pre><span class="lnum"> 10: </span> </pre><pre class="alt"><span class="lnum"> 11: </span> <span class="preproc">#endregion</span></pre><pre><span class="lnum"> 12: </span> }</pre><pre class="alt"><span class="lnum"> 13: </span>}</pre></div><br />As a result RIA Services will generate the appropriate client-side code for the manipulation of Transactions however as there is no [Include] attribute RIA will not automatically fetch the members of a Transactions collection when its parent Holding is instantiated.<br /><br />To manually load a list of Transactions it is necessary to write a parameterized server-side service method to perform the datastore lookup.<br /><br /><div class="csharpcode"><pre class="alt"><span class="lnum"> 1: </span>[EnableClientAccess]</pre><pre><span class="lnum"> 2: </span><span class="kwrd">public</span> <span class="kwrd">class</span> DataStore : DomainService</pre><pre class="alt"><span class="lnum"> 3: </span>{</pre><pre><span class="lnum"> 4: </span> ... other code</pre><pre class="alt"><span class="lnum"> 5: </span> </pre><pre><span class="lnum"> 6: </span> <span class="kwrd">public</span> IQueryable<Transaction> GetTransactionsForHolding(Guid holdingId)</pre><pre class="alt"><span class="lnum"> 7: </span> {</pre><pre><span class="lnum"> 8: </span> <span class="kwrd">using</span> (var db = DataService.GetSession<Transaction>())</pre><pre class="alt"><span class="lnum"> 9: </span> {</pre><pre><span class="lnum"> 10: </span> <span class="kwrd">return</span> db.GetList(x => x.HoldingId.Equals(holdingId)).AsQueryable();</pre><pre class="alt"><span class="lnum"> 11: </span> }</pre><pre><span class="lnum"> 12: </span> }</pre><pre class="alt"><span class="lnum"> 13: </span>}</pre></div><br />The<span class="Apple-style-span" style="font-style: italic;"> GetTransactionsForHolding(...)</span> method is scanned by RIA Services causing it to generate a client-side equivalent method on the DataStore class. This can then be used in client-side code to fetch a set of Transactions belonging to a specified Holding. The code below shows this happening. The call is being made within the SelectionChanged event of the Accordion control.<br /><br /><div class="csharpcode"><pre class="alt"><span class="lnum"> 1: </span><span class="kwrd">private</span> <span class="kwrd">void</span> Holdings_SelectionChanged(<span class="kwrd">object</span> sender, SelectionChangedEventArgs e)</pre><pre><span class="lnum"> 2: </span>{</pre><pre class="alt"><span class="lnum"> 3: </span> <span class="kwrd">if</span> (e.AddedItems.Count == 0)</pre><pre><span class="lnum"> 4: </span> {</pre><pre class="alt"><span class="lnum"> 5: </span> <span class="kwrd">return</span>;</pre><pre><span class="lnum"> 6: </span> }</pre><pre class="alt"><span class="lnum"> 7: </span> var holding = e.AddedItems[0] <span class="kwrd">as</span> Holding;</pre><pre><span class="lnum"> 8: </span> <span class="kwrd">if</span> (holding == <span class="kwrd">null</span> || holding.Transactions.Count > 0)</pre><pre class="alt"><span class="lnum"> 9: </span> {</pre><pre><span class="lnum"> 10: </span> <span class="kwrd">return</span>;</pre><pre class="alt"><span class="lnum"> 11: </span> }</pre><pre><span class="lnum"> 12: </span> <span class="kwrd">this</span>._dataStore.LoadTransactionsForHolding(holding.Id);</pre><pre class="alt"><span class="lnum"> 13: </span>}</pre></div><br />When an Accordion item is opened by a user click it fires the SelectionChanged event above. The newly selected Holding is extracted from the Accordion and its Holding.Id is passed into the RIA generated <span class="Apple-style-span" style="font-style: italic;">LoadTransactionsForHolding(...)</span> method. This automatically calls the <span class="Apple-style-span" style="font-style: italic;">GetTransactionsForHolding(...)</span> service method which returns the appropriate list of Transactions for the specified Holding.Id.<br /><br />Where do these Transactions go? How is it that simply calling this method automatically fills the correct Holding.Transactions collection and displays that collection in the data-bound Accordion?<br /><br />The list of Transactions is loaded into a flat list of Transactions generated and maintained by RIA Services. When a Holding.Transactions collection is requested RIA will dynamically create and return the correct list of Transactions as a conseqence of the information specified in the [Association] attribute. This is why each Transaction needs a HoldingId and each Holding a UserId. Finally, because RIA generated collections are <a href="http://weblogs.asp.net/joelvarty/archive/2008/11/17/silverlight-databinding-the-observable-collection.aspx">ObservableCollections</a> then changes automatically stimulate any data-bound containers to refresh themselves.<br /><br />This means that a call to the <span class="Apple-style-span" style="font-style: italic;">LoadTransactionsForHolding()</span> method will set off a chain of events that results in the lazy-loading of the selected list of Holding.Transactions and its subsequent display in the newly expanded Accordion item.<br /></div></div></div><br /><strong>Creating and Saving Domain Instances</strong><br />RIA Services makes the creating and saving new domain instances particularly easy. Once again the process begins with a statement of intention. This time RIA must be informed of our intention to add new Holdings to the User.Holdings collection and new Transactions to the Holding.Transactions collection. This is achieved via convention, by adding service methods whose signatures follow the convention shown below. <br /><br /><div class="csharpcode"><br /><pre class="alt"><span class="lnum"> 1: </span>[EnableClientAccess]</pre><pre><span class="lnum"> 2: </span><span class="kwrd">public</span> <span class="kwrd">class</span> DataStore : DomainService</pre><pre class="alt"><span class="lnum"> 3: </span>{</pre><pre><span class="lnum"> 4: </span> ...other code</pre><pre class="alt"><span class="lnum"> 5: </span> </pre><pre><span class="lnum"> 6: </span> <span class="kwrd">public</span> <span class="kwrd">void</span> CreateHolding(Holding holding)</pre><pre class="alt"><span class="lnum"> 7: </span> {</pre><pre><span class="lnum"> 8: </span> <span class="kwrd">using</span> (var db = DataService.GetSession<User>())</pre><pre class="alt"><span class="lnum"> 9: </span> {</pre><pre><span class="lnum"> 10: </span> var user = db.GetFirst(x => x.Id.Equals(holding.UserId));</pre><pre class="alt"><span class="lnum"> 11: </span> user.AddHolding(holding);</pre><pre><span class="lnum"> 12: </span> db.Save(user);</pre><pre class="alt"><span class="lnum"> 13: </span> }</pre><pre><span class="lnum"> 14: </span> }</pre><pre class="alt"><span class="lnum"> 15: </span> </pre><pre><span class="lnum"> 16: </span> <span class="kwrd">public</span> <span class="kwrd">void</span> CreateTransaction(Transaction transaction)</pre><pre class="alt"><span class="lnum"> 17: </span> {</pre><pre><span class="lnum"> 18: </span> <span class="kwrd">using</span> (var db = DataService.GetSession<Holding>())</pre><pre class="alt"><span class="lnum"> 19: </span> {</pre><pre><span class="lnum"> 20: </span> var holding = db.GetFirst(x => x.Id.Equals(transaction.HoldingId));</pre><pre class="alt"><span class="lnum"> 21: </span> holding.AddTransaction(transaction);</pre><pre><span class="lnum"> 22: </span> db.Save(holding);</pre><pre class="alt"><span class="lnum"> 23: </span> }</pre><pre><span class="lnum"> 24: </span> }</pre><pre class="alt"><span class="lnum"> 25: </span>}</pre></div><br />Adding these service methods tells RIA that we intend to add new Holdings and Transactions via client-side code. Without these methods any attempt to add an item will result in a runtime error. For example, if the <span class="Apple-style-span" style="font-style: italic;">CreateHolding()</span> method above is commented out and a new Holding is added to the User.Holdings collection via client-side code, the following error is displayed.<br /><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/_kbCCKz8F0ok/SgGowVmm1-I/AAAAAAAAAC8/as4mwtKRMSY/s1600-h/edit_error.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 235px;" src="http://3.bp.blogspot.com/_kbCCKz8F0ok/SgGowVmm1-I/AAAAAAAAAC8/as4mwtKRMSY/s400/edit_error.png" border="0" alt="" id="BLOGGER_PHOTO_ID_5332728982169638882" /></a><br /><br /><span class="Apple-style-span" style="font-weight: bold;">Serializing New Entities</span><br /><div>Domain entities added on the client are not automatically serialized to the server-side data-store. Instead RIA services keeps track of the changes you have made so that when a save is requested only the changes are submitted for server-side serialization. </div><div><br /></div><div>This is a good example of the <a href="http://martinfowler.com/eaaCatalog/unitOfWork.html">Unit of Work</a> pattern and in this way RIA helps to minimise the traffic over the wire as well as giving you much more flexibility with respect to rolling back or cancelling changes, providing save on demand or automatic timed-interval saves.<br /><br />The following code shows how to add and save new domain items.<br /><br /><div class="csharpcode"><br /><pre class="alt"><span class="lnum"> 1: </span><span class="kwrd">public</span> <span class="kwrd">partial</span> <span class="kwrd">class</span> HomePage : Page</pre><pre><span class="lnum"> 2: </span>{</pre><pre class="alt"><span class="lnum"> 3: </span> <span class="kwrd">private</span> <span class="kwrd">readonly</span> DataStore _dataStore = <span class="kwrd">new</span> DataStore();</pre><pre><span class="lnum"> 4: </span> <span class="kwrd">private</span> ProgressDialog _progressDialog;</pre><pre class="alt"><span class="lnum"> 5: </span> </pre><pre><span class="lnum"> 6: </span> <span class="kwrd">public</span> HomePage()</pre><pre class="alt"><span class="lnum"> 7: </span> {</pre><pre><span class="lnum"> 8: </span> <span class="kwrd">this</span>.InitializeComponent();</pre><pre class="alt"><span class="lnum"> 9: </span> <span class="kwrd">this</span>._dataStore.Submitted += <span class="kwrd">this</span>.DataStoreSubmitted;</pre><pre><span class="lnum"> 10: </span> </pre><pre class="alt"><span class="lnum"> 11: </span> ...other code</pre><pre><span class="lnum"> 12: </span> }</pre><pre class="alt"><span class="lnum"> 13: </span> </pre><pre><span class="lnum"> 14: </span> <span class="kwrd">private</span> <span class="kwrd">void</span> ShowProgressDialog(<span class="kwrd">string</span> message)</pre><pre class="alt"><span class="lnum"> 15: </span> {</pre><pre><span class="lnum"> 16: </span> <span class="kwrd">this</span>._progressDialog = <span class="kwrd">new</span> ProgressDialog(message);</pre><pre class="alt"><span class="lnum"> 17: </span> <span class="kwrd">this</span>._progressDialog.Show();</pre><pre><span class="lnum"> 18: </span> }</pre><pre class="alt"><span class="lnum"> 19: </span> </pre><pre><span class="lnum"> 20: </span> <span class="kwrd">private</span> <span class="kwrd">void</span> DataStoreSubmitted(<span class="kwrd">object</span> sender, SubmittedChangesEventArgs e)</pre><pre class="alt"><span class="lnum"> 21: </span> { </pre><pre><span class="lnum"> 22: </span> <span class="kwrd">if</span> (e.EntitiesInError.Count() != 0)</pre><pre class="alt"><span class="lnum"> 23: </span> {</pre><pre><span class="lnum"> 24: </span> <span class="kwrd">this</span>._progressDialog.ShowError();</pre><pre class="alt"><span class="lnum"> 25: </span> }</pre><pre><span class="lnum"> 26: </span> <span class="kwrd">else</span></pre><pre class="alt"><span class="lnum"> 27: </span> {</pre><pre><span class="lnum"> 28: </span> <span class="kwrd">this</span>._progressDialog.Close();</pre><pre class="alt"><span class="lnum"> 29: </span> }</pre><pre><span class="lnum"> 30: </span> }</pre><pre class="alt"><span class="lnum"> 31: </span> </pre><pre><span class="lnum"> 32: </span> <span class="kwrd">private</span> <span class="kwrd">void</span> SubmitChanges_Click(<span class="kwrd">object</span> sender, RoutedEventArgs e)</pre><pre class="alt"><span class="lnum"> 33: </span> {</pre><pre><span class="lnum"> 34: </span> <span class="kwrd">this</span>.ShowProgressDialog(<span class="str">"Saving Changes..."</span>);</pre><pre class="alt"><span class="lnum"> 35: </span> <span class="kwrd">this</span>._dataStore.SubmitChanges();</pre><pre><span class="lnum"> 36: </span> }</pre><pre class="alt"><span class="lnum"> 37: </span> </pre><pre><span class="lnum"> 38: </span> <span class="kwrd">private</span> <span class="kwrd">void</span> NewHolding_Click(<span class="kwrd">object</span> sender, RoutedEventArgs e)</pre><pre class="alt"><span class="lnum"> 39: </span> {</pre><pre><span class="lnum"> 40: </span> var user = ((User) <span class="kwrd">this</span>.User.DataContext);</pre><pre class="alt"><span class="lnum"> 41: </span> <span class="kwrd">if</span> (user == <span class="kwrd">null</span>)</pre><pre><span class="lnum"> 42: </span> {</pre><pre class="alt"><span class="lnum"> 43: </span> <span class="kwrd">return</span>;</pre><pre><span class="lnum"> 44: </span> }</pre><pre class="alt"><span class="lnum"> 45: </span> user.Holdings.Add(DomainFactory.Holding(user, NewHoldingSymbol.Text));</pre><pre><span class="lnum"> 46: </span> <span class="kwrd">this</span>.Holdings.SelectedItem = <span class="kwrd">this</span>.Holdings.Items[<span class="kwrd">this</span>.Holdings.Items.Count-1];</pre><pre class="alt"><span class="lnum"> 47: </span> }</pre><pre class="alt"><span class="lnum"> 49: </span> <span class="kwrd">private</span> <span class="kwrd">void</span> Buy_Click(<span class="kwrd">object</span> sender, RoutedEventArgs e)</pre><pre><span class="lnum"> 50: </span> {</pre><pre class="alt"><span class="lnum"> 51: </span> var holding = ((Button) e.OriginalSource).DataContext <span class="kwrd">as</span> Holding;</pre><pre><span class="lnum"> 52: </span> <span class="kwrd">if</span> (holding == <span class="kwrd">null</span>)</pre><pre class="alt"><span class="lnum"> 53: </span> {</pre><pre><span class="lnum"> 54: </span> <span class="kwrd">return</span>;</pre><pre class="alt"><span class="lnum"> 55: </span> }</pre><pre><span class="lnum"> 56: </span> holding.AddTransaction(DomainFactory.Transaction(holding, TransactionType.Buy, 42, 0.42m));</pre><pre class="alt"><span class="lnum"> 57: </span> }</pre><pre><span class="lnum"> 58: </span> </pre><pre class="alt"><span class="lnum"> 59: </span> <span class="kwrd">private</span> <span class="kwrd">void</span> Sell_Click(<span class="kwrd">object</span> sender, RoutedEventArgs e)</pre><pre><span class="lnum"> 60: </span> {</pre><pre class="alt"><span class="lnum"> 61: </span> var holding = ((Button) e.OriginalSource).DataContext <span class="kwrd">as</span> Holding;</pre><pre><span class="lnum"> 62: </span> <span class="kwrd">if</span> (holding == <span class="kwrd">null</span>)</pre><pre class="alt"><span class="lnum"> 63: </span> {</pre><pre><span class="lnum"> 64: </span> <span class="kwrd">return</span>;</pre><pre class="alt"><span class="lnum"> 65: </span> }</pre><pre><span class="lnum"> 66: </span> holding.AddTransaction(DomainFactory.Transaction(holding, TransactionType.Sell, 42, 0.42m));</pre><pre class="alt"><span class="lnum"> 67: </span> }</pre><pre><span class="lnum"> 68: </span> </pre><pre class="alt"><span class="lnum"> 69: </span> ...other code</pre><pre><span class="lnum"> 70: </span> </pre><pre class="alt"><span class="lnum"> 71: </span>}</pre></div><br />This code shows how to add new domain items to their correct location in the domain hierarchy using the <span class="Apple-style-span" style="font-style: italic;">shared </span>DomainFactory class discussed earlier. These changes are then asynchronously submitted as a batched <span class="Apple-style-span" style="">unit of work</span> to the server, displaying a progress dialog to keep the user informed. The return is trapped so that the progress dialog can be dismissed and any errors displayed.<br /><br /><strong>The Verdict</strong><br />How did RIA Services and DB4O manage?</div><div><div><ul><li><span><span><span class="Apple-style-span" style="font-style: italic;">Server - When I fetch an instance of the aggregrate root class I expect its inner hierarchy be eagerly fetched.</span><br /><span class="Apple-style-span" style="font-style: italic;"><span class="Apple-style-span" style="font-style: normal; ">The server-side domain de/serialisation behaviour was handled by DB4O. Being an object database it is simple to create this behaviour using a few lines of initialisation code.<br /><span class="Apple-style-span" style="font-style: italic;"><br /></span></span></span></span></span></li><li><span class="Apple-style-span" style="font-style: italic;">Client - I want certain collections to be lazy-loaded and so remain unloaded until they are requested.</span><br />RIA Services provides a set of attributes that allow both eager and lazy loading to be specified as client-side behaviour and wired up with minimal code.<br /><br /></li><li><span class="Apple-style-span" style=""></span><span><span><span class="Apple-style-span" style="font-style: italic;">I do not expect to write my own WCF Service nor do I want to write and Data Transfer Objects (DTOs).</span></span></span><span class="Apple-style-span" style="font-style: italic;"></span><br />RIA Services replaces the explicit WCF layer with an implicit data transafer layer via its DomainService class and the data manipulation methods you write to extend it. <br /><br /></li><li><span class="Apple-style-span" style="font-style: italic;">I want to databind my domain entities to silverlight controls. I expect the controls to correctly display my eagerly fetched data as well as handling lazy-loaded data.</span><br />Because RIA Services generates its own observable collections the silverlight databinding flows smoothly with little intervention. The lazy loading of new data stimulates the silverlight bound controls to refresh and so display changes as they occur.<br /><br /></li><li><span><span><span class="Apple-style-span" style="font-style: italic;">Finally, I want to prove that new domain entities can be created on the client and efficiently serialized to the server-side data-store as a batched unit-of-work.</span></span></span><a href="http://framework.zend.com/wiki/display/ZFDEV/Unit+of+Work+pattern"></a><br />RIA Services implements a Unit of Work pattern that allows only those items that have been changed to be batched and serialized to the server-side data-store when required.<br /></li></ul></div><br />I think that RIA Services plus DB4O performed well in handling the demands of my simple Line of Business Rich Internet Application. I would certainly recommend you try it out for yourself to see what you think. Good Luck.<div><br /></div></div></div></div></div>biofractalhttp://www.blogger.com/profile/11229842750489025189noreply@blogger.com5tag:blogger.com,1999:blog-6927558.post-51102332546351823132009-03-31T20:54:00.024+01:002011-05-18T17:19:49.633+01:00A Domain-Driven, DB4O Silverlight RIA<div><span class="Apple-style-span" style="font-style: italic; "><span class="Apple-style-span" style="font-size:large;">Build a </span><span class="Apple-style-span" style="font-weight: bold;"><span class="Apple-style-span" style="font-size:large;">Domain Driven</span></span><span class="Apple-style-span" style="font-size:large;"> Rich Internet Application using Silverlight, RIA Services and DB4O</span></span></div><div style="text-align: center;"><br /><ul><li style="text-align: left;"><a href="http://www.assembla.com/code/biofractal/subversion/nodes/trunk/Blog/DomainDrivenRIA/GettingStarted">Download the Example Solution for Silverlight 3</a><br /></li><br /><li style="text-align: left;"><a href="http://cid-9803e97b07345d0c.office.live.com/self.aspx/.Public/DomainDrivenRIA-GettingStarted.zip">Download the Example Solution for Silverlight 4</a><br /></li><br /></ul></div>I was <a href="http://biofractal.blogspot.com/2009/03/silverlight-and-object-databases.html">recently ranting</a> about Silverlight-2 and my annoyance with the WCF layer needed to serialize object instance data to the server. Well Microsoft must have had their mind-reading machines turned up high because the newly announced Mix09 Silverlight 3 & RIA services preview has solved nearly all the problems I was having.<div><br /><div>Its the RIA (<a href="http://en.wikipedia.org/wiki/Rich_Internet_application">Rich Internet Application</a>) Services that has the real wow-factor. This is Silverlight finally growing up. Instead of creating Frankenstein client-server apps crudely stitched together with WCF, RIA services allows you to treat your client and server as one, almost seamless application. You share domain design <span class="Apple-style-span" style="font-style: italic;">intentions</span> between the client and the server so that your domain acts as you intended regardless of which side of the machine boundary you are on. This is how RIA development was always meant to be. </div><div><br /></div><div>You can download all the software kit you need from here the <a href="http://silverlight.net/getstarted/silverlight3/default.aspx">main Silverlight-3 site</a>. I found the best way to get started was to watch the following Mix09 presentations then read the <a href="http://download.microsoft.com/download/F/B/8/FB8CA635-296B-487F-965C-8148F08B5319/riaservicesoverviewpreview.pdf">RIA Services Overview</a>.</div><div><ul><li><a href="http://videos.visitmix.com/MIX09/T40F">Building Amazing Business Centric Applications with Microsoft Silverlight 3</a><br /><span class="Apple-style-span" style="font-style: italic;">"Come hear how simple it is to build end-to-end data-intensive Silverlight applications with the new set of features in Silverlight 3 and .NET RIA Services. Explore Silverlight improvements that help to enable rapid development for business applications and to make your development process more productive"</span><br /><br /></li><li><a href="http://videos.visitmix.com/MIX09/T41F">Building Data-Driven Applications in ASP.NET and Silverlight</a><br /><span class="Apple-style-span" style="font-style: italic;">"Learn how Microsoft is simplifying the traditional n-tier application pattern by bringing together ASP.NET and Silverlight. Learn about patterns for working with data, implementing reusable and independently testable application logic, and application services that readily scale with growing requirements"</span></li></ul></div><div><div>Of course Microsoft just could not help plugging their dystopian Data-Driven-Design vision. Look at the title of that second presentation for goodness sake. </div><div><br /></div><div>To counter this and to provide a solid Domain-Driven template for future Silverlight RIA apps I have created an example that does away with the monolithic SqlServer and Igor, the Entity Framework, in favour of an light, fast object database repository that promotes good Code Cohesion, Separation of Concerns and Inversion of Control.</div><div><br /></div><div>Here is the tech I use in the example:</div><div><ul><li><a href="http://silverlight.net/getstarted/silverlight3/default.aspx">Silverlight 3</a><br /></li><li><a href="http://www.nikhilk.net/RIA-Services-MIX09.aspx">RIA Services</a><br /></li><li><a href="http://www.db4o.com/">DB4O</a><br /></li><li><a href="http://structuremap.sourceforge.net/QuickStart.htm">StructureMap</a></li></ul></div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;"><span class="Apple-style-span" style="font-size:large;">An Overview of the Domain Driven RIA Example</span></span></div><div>This example does just enough to highlight the primary functions of RIA services as I see them.</div><div><ol><li>You can define and use your domain objects on the server and then effectively re-use domain logic on the client. <br /><br /></li><li>You can transmit domain objects between the client and server without polluting your domain or requiring an additional DTO transformation layer.<br /></li></ol></div><img src="http://4.bp.blogspot.com/_kbCCKz8F0ok/SdKBCzuYM8I/AAAAAAAAAB0/QzZO6l2SnRI/s400/SolutionExplorer.gif" style="float:left; margin:0 10px 10px 0;cursor:pointer; cursor:hand;width: 177px; height: 400px;" border="0" alt="" id="BLOGGER_PHOTO_ID_5319455995122430914" /><div>The example is a stripped down version of an app I am currently writing so I have left in all the domain structure even if it is not actually used in the example. This makes the application ready to go if you want to start fleshing it out with your own code.<br /></div><div><br /></div><div style="text-align: left;">The functionality is quite simple. You are presented with a Silverlight 3 navigation application. There is a sign-in link that takes you a view containing a username and password field, sign-in and register buttons. </div><div style="text-align: left;"><br /></div><div>The code works as you would expect with the additional extra that typing into the <span class="Apple-style-span" style="font-style: italic; "><span class="Apple-style-span" style="font-weight: bold; ">Name</span></span><span class="Apple-style-span" style="font-weight: bold; "> </span>field triggers a 1 second timer that will check (on the server) that the name is unique. Clicking the <span class="Apple-style-span" style="font-style: italic; "><span class="Apple-style-span" style="font-weight: bold; ">Register</span> </span>button creates a new user in the server-side DB40 database. Clicking the <span class="Apple-style-span" style="font-style: italic; "><span class="Apple-style-span" style="font-weight: bold; ">Sign In</span></span> button checks the db for the supplied credentials and, if they exist will return the appropriate User instance and cleverly navigate you back to the home view. <br /></div><div style="text-align: center;"><br /></div><div>If you have watched the videos you might at this point think I have cheated and used the built in Authentication Domain Service that come with RIA. Not so. That service relies on having a fat SQLServer file squatting in your solution and that is no good for a Domain-Driven purist. Instead I have implemented a very simple custom identity authentication that could easily be linked up to either Forms or Windows authentication in the usual way.<br /></div><div style="text-align: center;"><br /></div><img src="http://2.bp.blogspot.com/_kbCCKz8F0ok/SdKDJtP_-JI/AAAAAAAAAB8/gmhzySwZQjE/s400/Sign-In.gif" style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;width: 400px; height: 266px;" border="0" alt="" id="BLOGGER_PHOTO_ID_5319458312666740882" /><div style="text-align: center;"><br /></div><div style="text-align: center;"><a href="http://www.assembla.com/code/biofractal/subversion/nodes/trunk/Blog/DomainDrivenRIA/GettingStarted">Download the Example Solution From Assembla</a><br /></div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;">NB:</span> If you get an error about the URI Prefix then you need to reset the startup application to <span class="Apple-style-span" style="font-style: italic;">DomainDrivenRIA.Web</span> and the startup page to <span class="Apple-style-span" style="font-style: italic;">DomainDrivenRIA.SilverlightTestPage.aspx<br /></span></div><div><span class="Apple-style-span" style="font-style: italic;"><br /></span></div></div></div>biofractalhttp://www.blogger.com/profile/11229842750489025189noreply@blogger.com18tag:blogger.com,1999:blog-6927558.post-48888227704540177782009-03-17T20:58:00.046+00:002010-08-27T16:41:18.406+01:00Convert dotnet assemblies to Silverlight<div><span style="font-size:130%;">The SilverLighter</span></div><div><ul><li><span class="Apple-style-span" style=" "><span class="Apple-style-span" style="font-size:small;"><a href="https://www.assembla.com/code/biofractal/subversion/nodes/trunk/Blog/Silverlighter/Silverlighter.Setup/Debug">Download Installer</a></span></span><span class="Apple-style-span" style="font-size:small;"><br /></span></li><li><span class="Apple-style-span" style=" "><span class="Apple-style-span" style="font-size:small;"><a href="https://www.assembla.com/code/biofractal/subversion/nodes/trunk/Blog/Silverlighter">Download Source</a></span><span class="Apple-style-span" style="font-size:small;"> <a href="http://tortoisesvn.net/"><span class="Apple-style-span" style="font-size:small;"></span></a></span></span></li><li><span class="Apple-style-span" style=" "><span class="Apple-style-span" style="font-size:small;"><a href="http://tortoisesvn.net/"><span class="Apple-style-span" style="font-size:small;">SVN</span></a><span class="Apple-style-span" style="font-size:small;"> http://subversion.assembla.com/svn/biofractal/trunk/Blog/Silverlighter</span></span></span></li></ul></div><div>I have recently been trying to convert an object database (<a href="http://www.db4o.com/">db40</a> and <a href="http://www.neodatis.org/">NeoDatis</a>) for use in a Silverlight application. Along the way I discovered a few interesting things. For example I found that although Visual Studio will not allow you to reference dotnet assemblies from a Silverlight application or class library, this restriction is actually a bit heavy handed. Sometimes it is very useful to use a known Silverlight-safe dotnet assembly, as long as you take responsibility for your own actions.</div><div><br /><img src="http://4.bp.blogspot.com/_kbCCKz8F0ok/ScAPp0AuBKI/AAAAAAAAABc/CYHBsYfexnE/s320/Silverlighter_1.png" id="BLOGGER_PHOTO_ID_5314264771307898018" style="FLOAT: left; MARGIN: 0px 10px 10px 0px; WIDTH: 320px; CURSOR: hand; HEIGHT: 184px" alt="" border="0" />To enable the re-use of dotnet assemblies in Silverlight I wrote a handy WPF application called "The Silverlighter". This tool allows you to convert any dotnet assembly into a Silverlight assembly ready for use in your Silverlight applications or class libraries. </div><div><br />The knowledge I needed to write The Silverlighter was gleaned primarily from the excellent <span>article <span>by</span><span class="Apple-style-span" style="font-style: italic; "> David Betz <span class="Apple-style-span" style="font-style: normal;">called </span><span class="Apple-style-span" style="font-style: normal; "><span><a href="http://www.netfxharmonics.com/2008/12/Reusing-NET-Assemblies-in-Silverlight">Reusing .NET Assemblies in Silverlight</a></span>. This article clearly explains how the dotnet->Silverlight conversion process works and why it is not a hack. I thought this article was a fascinating insight into the similarities between the dotnet and Silverlight assemblies and is well worth a read.</span></span></span></div><div><br />But before you get carried away (I know I did) and imagine that you are just a click away from re-using your favourite 3rd party assemblies, a word or two of caution. </div><div><br /></div><div>Unfortunately just because you can reference an assembly from Silverlight does not mean it will work with Silverlight. If you have read David Betz' article you will know that Silverlight uses a distinct set of System.* assemblies (v 2.0.5.0). These assemblies do not contain all the features of their equivalent dotnet assemblies. For example the following collection types are missing in Silverlight:</div><ul><li>ArrayList</li><li>Hashtable</li><li>SortedList</li><li>NameValueCollection</li></ul><div>Instead Silverlight only allows generic collections to be used, which is a good thing unless your referenced ex-dotnet assembly happens to use one or more of the missing types, in which case your code will blow up with an error similar to "<span class="Apple-style-span" style="font-style: italic;">cannot load type ArrayList from assembly System 2.0.5.0</span>"<br /></div><br /><div>Replacing these missing collections with the generic equivalents (you can decompile most assembly code using <a href="http://www.red-gate.com/products/reflector/">Reflector</a>) is actually quite trivial however more serious problems are lurking. Silverlight is also missing core features such as:</div><ul><li><a href="http://www.lhotka.net/weblog/SilverlightSerializer.aspx">BinarySerializer</a> and the [Serializable} attribute</li><li>Threading.SetData</li><li><a href="http://dotnetslackers.com/Silverlight/re-90376_Silverlight_2_and_Sockets.aspx">TCP Socket related features</a></li></ul><div>You can see that Silverlight has removed or redesigned features that are security risks. Again that is a good thing, until you try to re-use assemblies that use these features - then boom. </div><p>Anyway, The Silverlighter app does some cool IL trickery so if you have dotnet dlls you know are fine and just want to use them in Silverlight without any nonsense from Visual Studio then it might be just the thing you need.</p><p><span style="font-size:130%;">Additional Notes</span></p><p><span style="font-size:100%;">There are a few options available to help tweak the functionality.</span></p><div><a href="http://4.bp.blogspot.com/_kbCCKz8F0ok/ScAXYFYRHiI/AAAAAAAAABk/rY1v3UeHQxU/s1600-h/Silverlighter_2.png"><img id="BLOGGER_PHOTO_ID_5314273262825446946" style="FLOAT: left; MARGIN: 0px 10px 10px 0px; WIDTH: 320px; CURSOR: hand; HEIGHT: 184px" alt="" src="http://4.bp.blogspot.com/_kbCCKz8F0ok/ScAXYFYRHiI/AAAAAAAAABk/rY1v3UeHQxU/s320/Silverlighter_2.png" border="0" /></a></div><br /><div>You can choose to convert only selected System assemblies, although it is recommended that you just leave them all selected unless you have a good reason not to.</div><div> </div><br /><div>The "Recursively process dependent assemblies" feature will, when checked, pick out references from the IL to non System assemblies (your own or 3rd party references) and recursively convert these to Silverlight compliant assemblies as well. The entire dependency tree will be processed in this way.<br /><br />Finally the path to the ILdasm.exe is exposed just in case you have it at a different location on your system. If you don't have ILdasm.exe anywhere you can get by installing <a href="http://www.microsoft.com/downloads/details.aspx?FamilyID=fe6f2099-b7b4-4f47-a244-c96d69c35dec&displaylang=en">.NET Framework 2.0 Software Development Kit (SDK)</a></div><div><br /></div><div>If you notice any bugs or want to add new features then please feel free to checkout the source code and make the updates. Just paste the SVN url into your <a href="http://tortoisesvn.net/">TortoiseSVN</a> repo-browser, check out the source code under subversion and you are off and running. Good luck.</div><div><br /></div><div><span class="Apple-style-span" style="font-size:small;">SVN URL = http://subversion.assembla.com/svn/biofractal/trunk/Blog/Silverlighter</span></div><div><br /></div><div><br /></div>biofractalhttp://www.blogger.com/profile/11229842750489025189noreply@blogger.com14tag:blogger.com,1999:blog-6927558.post-23249701258255479522009-03-16T21:46:00.008+00:002009-04-01T08:47:40.515+01:00Silverlight and Object Databases<div><span class="Apple-style-span" style="font-weight: bold;"><span class="Apple-style-span" style="color: rgb(204, 0, 0);">Stop Press</span></span>: Most of the issues below have now been resolved with the release of Silverlight-3 & RIA Services. See the following post for more details - <a href="http://biofractal.blogspot.com/2009/03/domain-driven-db4o-silverlight-3-ria.html">A Domain-Driven, DB4O Silverlight-3 RIA</a></div><div><br /></div><div>I recently decided to write myself a quick Silverlight application for a bit of fun. I wanted an app that could grab stock quotes, do a calculation of my current losses and maybe have some nice blue-gel buttons and what-not. A few evenings of light coding pleasure and good break from all that architectural stuff. Naturally it did not quite work out that way.<br /><div><br /></div><div>I got a wireframe working pretty quickly but that was not good enough. I wanted to serialize my data so I could start to get some fancy Silverlight graphs with bouncy bars. So I found myself grabbing multiple quotes and using the data to instantiate domain objects in the Silverlight client. Now what? I needed to serialize these instances on the web server. So how do I do that?</div><br /><div>I had two options</div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;">1</span>. Use isolated storage on the client then use Mesh to synch the data. The only problem with this approach is that the last sentence is the total extent of my knowledge. I want a slope not a cliff face.</div><br /><div><span class="Apple-style-span" style="font-weight: bold;">2</span>. Use a good old database on the server, why not, and since I am about it lets make that an object database so we don't have to concern ourselves with all that old-fashioned NHibernate, ORM tosh (ah - how quickly I become intolerant).<br /></div><br /><div>I chose option #2. And right there it started to get annoying. What I really wanted to do was just use my ODB in the Silverlight client code as if I was coding normally. But of course the Silverlight sandbox had lots to say about that.</div><br /><div>To get around this you must write a WCF service layer. Call this an <span class="Apple-style-span" style="font-style: italic;">API</span> and you will feel much better because that makes it sound quite cool and computery. You need this comfort blanket because you are about to climb into a time machine a go back 10 years to when you were a script kiddie banging out ASP code. </div><div><br /></div><div>This WCF 'API' service is going to be the only means you have to serialize data and transmit it between the Silverlight client and the web server. The [DataContract] will require [DataMember] methods to cover every possible action, just like writing all those CRUD sprocs for a relational DB. Of course you could break this jumbled, untestable mess into smaller WCF contracts but that is just more fantasy to cover the poor design, like putting prefixes on your sproc names so they group together to simulate a business layer.</div><div><br /></div><div>No, the real solution is to do away with the WCF data-layer before you even start. Now don't get me wrong, WCF is fine and very useful but I think it is an abuse to use it a domain serialization interface. We just got rid of RDB DALs so why create another DAL for SaaS?</div><br /><div>What is needed is grown up client / server relationship. That would be much better. Then, instead of writing a sprawling WCF thunking layer replete with DTO auto-mappers and all that junk paraphanalia, you get a nice generic client-centric solution with code like this:</div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;"> myODB.Store(myObject) ;</span></div><div><span class="Apple-style-span" style="font-weight: bold;"> myODB.Commit();</span></div><div><br /></div><div>This focuses coding attention away from the boilerplate WCF data transformation layer and back on to the business useful world of the Domain - where it belongs.</div><br /><div>By the way, you <span class="Apple-style-span" style="font-weight: bold;">can </span>use dotnet assemblies in a Silverlight project. It needs a few quick (but legitimate) 'fixes' to the IL to get them past the Visual Studio fascist guards. I am finishing off a tool that automates the IL work for you makes it nice and easy to convert dotnet assemblies into Silverlight reference-ready assemblies. I will post it here soon.</div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;"><span class="Apple-style-span" style="color: rgb(204, 0, 0);">Stop Press</span></span>: You can now download the IL Tool mentioned above. See the following blog post for more details - <a href="http://biofractal.blogspot.com/2009/03/convert-dotnet-assemblies-to.html">Convert dotnet assemblies to Silverlight</a></div><div><br /></div><div></div></div>biofractalhttp://www.blogger.com/profile/11229842750489025189noreply@blogger.com0tag:blogger.com,1999:blog-6927558.post-16205231111539089702009-02-27T12:01:00.015+00:002009-03-04T12:12:36.935+00:00Generic Lists of Anonymous Type<p>[cross-posted to <a href="http://blog.stormid.com/archive/2009/02/27/generic-lists-of-anonymous-type.aspx">StormId blog</a>]</p> <p>Anonymous types can be very useful when you need a few transient classes for use in the middle of a process. </p> <p>Of course you could just write a class in the usual way but this can quickly clutter up your domain with class definitions that have little meaning beyond the scope of their transient use as part of another process. </p> <p>For example, I often use anonymous types when I am generating reports from my domain. The snippet below shows me using an anonymous type to store data values that I have collected from my domain.</p> <div class="csharpcode"><pre class="alt"><span class="kwrd">for</span> (var k = 0; k < optionCount; k++)</pre><pre class="alteven">{</pre><pre class="alt"> var option = options[k];</pre><pre class="alteven"> var optionTotal = results[option.Id];</pre><pre class="alt"> var percent = (questionTotal > 0) ? ((optionTotal/(<span class="kwrd">float</span>)questionTotal) * 100): 0;</pre><pre class="alteven"> reportList.Add(<span class="kwrd">new</span></pre><pre class="alt"> {</pre><pre class="alteven"> Diagnostic = diagnostic.Name, </pre><pre class="alt"> Question = question.Text, </pre><pre class="alteven"> Option = option.Text, </pre><pre class="alt"> Count = optionTotal, </pre><pre class="alteven"> Percent = percent</pre><pre class="alt"> });</pre><pre class="alteven">}</pre></div><p>Here I am generating a report on the use of diagnostics (a type of survey). It shows how often each option of each question in each diagnostic has been selected by a user, both count and percent.</p><p>You can see that the new anonymous type instance is being added to a list called <strong>reportList</strong>. This list is strongly typed as can been seen by this next bit of code where I order the list using LINQ.</p><div class="csharpcode"><pre class="alt">reportList = reportList</pre><pre class="alteven"> .OrderBy(x => x.Diagnostic)</pre><pre class="alt"> .ThenBy (x => x.Question)</pre><pre class="alteven"> .ThenBy (x => x.Percent)</pre><pre class="alt"> .ToList();</pre></div><p>This is where the problem comes in, how is it possible to create a strongly typed (generic) list for an anonymous type? The answer is to use a generics trick, as the following code snippet shows.</p><div class="csharpcode"><pre class="alt"><span class="kwrd">public</span> <span class="kwrd">static</span> List<T> MakeList<T>(T example)</pre><pre class="alteven">{</pre><pre class="alt"> <span class="kwrd">return</span> <span class="kwrd">new</span> List<T>();</pre><pre class="alteven">}</pre></div><p>The <strong>MakeList</strong> method takes in a parameter of type <strong><T></strong> and returns a generic list of the same type. Since this method will accept any type then we can pass an anonymous type instance with no problems. The next snippet shows this happening.</p><div class="csharpcode"><pre class="alt">var exampleReportItem = <span class="kwrd">new</span></pre><pre class="alteven"> {</pre><pre class="alt"> Diagnostic = <span class="kwrd">string</span>.Empty, </pre><pre class="alteven"> Question = <span class="kwrd">string</span>.Empty, </pre><pre class="alt"> Option = <span class="kwrd">string</span>.Empty, </pre><pre class="alteven"> Count = 0, </pre><pre class="alt"> Percent = 0f</pre><pre class="alteven"> };</pre><pre class="alt">var reportList = MakeList(exampleReportItem);</pre></div><p>So here is the context for all these snippets. The following code gathers my report data and stores it in a strongly typed list containing a transient anonymous type.</p><div class="csharpcode"><pre class="alt">var exampleReportItem = <span class="kwrd">new</span></pre><pre class="alteven"> {</pre><pre class="alt"> Diagnostic = <span class="kwrd">string</span>.Empty, </pre><pre class="alteven"> Question = <span class="kwrd">string</span>.Empty, </pre><pre class="alt"> Option = <span class="kwrd">string</span>.Empty, </pre><pre class="alteven"> Count = 0, </pre><pre class="alt"> Percent = 0f</pre><pre class="alteven"> };</pre><pre class="alt">var reportList = MakeList(exampleReportItem);</pre><pre class="alteven"><span class="kwrd">for</span> (var i = 0; i < count; i++)</pre><pre class="alt">{</pre><pre class="alteven"> var diagnostic = diagnostics[i];</pre><pre class="alt"> var questionCount = diagnostic.Questions.Count;</pre><pre class="alteven"> <span class="kwrd">for</span> (var j = 0; j < questionCount; j++)</pre><pre class="alt"> {</pre><pre class="alteven"> var question = diagnostic.Questions[j];</pre><pre class="alt"> var questionTotal = results[question.Id];</pre><pre class="alteven"> var options = question.Options;</pre><pre class="alt"> var optionCount = options.Count;</pre><pre class="alteven"> <span class="kwrd">for</span> (var k = 0; k < optionCount; k++)</pre><pre class="alt"> {</pre><pre class="alteven"> var option = options[k];</pre><pre class="alt"> var optionTotal = results[option.Id];</pre><pre class="alteven"> var percent = (questionTotal > 0) ? ((optionTotal/(<span class="kwrd">float</span>)questionTotal) * 100): 0;</pre><pre class="alt"> reportList.Add(<span class="kwrd">new</span></pre><pre class="alteven"> {</pre><pre class="alt"> Diagnostic = diagnostic.Name, </pre><pre class="alteven"> Question = question.Text, </pre><pre class="alt"> Option = option.Text, </pre><pre class="alteven"> Count = optionTotal, </pre><pre class="alt"> Percent = percent</pre><pre class="alteven"> });</pre><pre class="alt"> }</pre><pre class="alteven"> }</pre><pre class="alt">}</pre></div><p>Perhaps you are wondering how the type of the anonymous <strong>exampleReportItem</strong> is the same as the type of the anonymous object I add to the <strong>reportList</strong>?</p><p>This works because of the way the type identities are assigned for anonymous types. If two anonymous types share the same public signature, that is if their property names and types are the same (you can't have methods on anonymous types) then the compiler treats them <em>as the same type</em>.</p><p>This is how the <strong>MakeList</strong> method can do its job. The <strong>exampleReportItem</strong> instance sent to the <strong>MakeList</strong> function has exactly the same properties as the anonymous type added to the generic <strong>reportList</strong><em>.</em> Because they have the same signatures then they are recognised as the same anonymous type and all is well.</p><p><br /></p>biofractalhttp://www.blogger.com/profile/11229842750489025189noreply@blogger.com0tag:blogger.com,1999:blog-6927558.post-30731045691511979852009-02-18T20:50:00.031+00:002009-03-05T11:24:39.679+00:00The Data Driven ConspiracyIn a previous post called <a href="http://biofractal.blogspot.com/2009/02/irrational-love-of-relational.html">An Irrational Love of the Relational</a> I asked whether the relational database (RDB) is the best data store for an object-oriented application. Along the way I mentioned that RDBs tend to make coders adopt a <span class="Apple-style-span" style="font-style: italic;"><span class="Apple-style-span" style=""><a href="http://en.wikipedia.org/wiki/Data-driven_design">data driven design</a></span><span class="Apple-style-span" style="font-style: normal;"> approach instead of the more object-oriented </span><span class="Apple-style-span" style=""><a href="http://en.wikipedia.org/wiki/Domain_driven_design">domain driven design</a></span><span class="Apple-style-span" style="font-style: normal;">. </span></span><div><br /></div><div><span class="Apple-style-span" style="font-style: italic;"><span class="Apple-style-span" style="font-style: normal;">That got me thinking: "</span><span class="Apple-style-span" style="">Why has the RDB remained at the heart of our technology stack despite a steadily growing demand for a domain-centric design methodology?"</span></span></div><div><br /></div><div>I think the answer to that comes in two parts.</div><div><ol><li>Historical</li><li>Conspiratorial<br /></li></ol></div><div><span class="Apple-style-span" style="font-size:large;"><span class="Apple-style-span" style="font-weight: bold;">1. Historical</span></span></div><div><a href="http://en.wikipedia.org/wiki/Edgar_F._Codd">Edgar Codd</a>, an employee of IBM in the 1970's, was getting frustrated at the lack of a <span class="Apple-style-span" style="font-style: italic;">search feature</span> that would allow him to quickly retrieve information from his IBM mainframe hard-drive array. So he worked out a cool way of structuring the data on the drive that would allow <span class="Apple-style-span" style="font-style: italic;">queries</span> to be constructed and so allow for ad hoc data retrieval. He released a paper describing this work called <span class="Apple-style-span" style="font-style: italic;">A Relational Model of Data for Large Shared Data Banks </span><span class="Apple-style-span" style="">[1]. </span></div><div><br /></div><div><span class="Apple-style-span" style="">In this paper he described, for the first time, all the features of an RDB that we find so familiar today. Data normalisation, columns , rows, tables, foreign keys <span class="Apple-style-span" style="font-style: italic;">etc</span>. <a href="http://en.wikipedia.org/wiki/Codd's_12_rules">Codd's rules</a> laid out clearly what constituted a true relational database and allowed manufacturers to create their own RDB management systems. </span></div><div><br /></div><div>This tech was very cool and so much better than anything that had gone before. It allowed for data-integrity, which helped the coders and it facilitated a whole new class of program that relied on queries that sliced and diced huge data-sets revealing subtle and surprising inter-connections. </div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;"><span class="Apple-style-span" style="font-size:large;">2. Conspiratorial</span></span></div><div>By the 1980's most young coders were convinced that relational data and the RDB was <span class="Apple-style-span" style="font-style: italic;">the </span><span class="Apple-style-span" style="font-style: italic;">answer </span><span class="Apple-style-span" style=""><span class="Apple-style-span" style="font-style: italic;">to life, the universe and everything</span></span><span class="Apple-style-span" style="font-style: italic;">. <span class="Apple-style-span" style="font-style: normal;">N</span><span class="Apple-style-span" style="font-style: normal; ">ew companies grew up around the RDB, <span class="Apple-style-span" style="font-weight: bold;">Microsoft </span>got in on the act with SqlServer and began, slowly, to dominate the market. Time passed and eventually those young turks grew older, their beards grew greyer and, like the RDB software they depended on, they became bloated. Then, just when people were forgetting that data could be stored in any other way, the first cracks began to appear in the RDB monolith. </span></span></div><div><br /></div><div>It started when <a href="http://en.wikipedia.org/wiki/Object_orientation">object-orientation</a> finally broke free of the university labs and escaped into the wider coding world. Programs written as a hierarchical collection objects (an <span class="Apple-style-span" style="font-style: italic;">object domain</span>) had been around since the 1960's when a pair of Norwegian academics invented a language called <a href="http://en.wikipedia.org/wiki/Simula">Simula-67</a>. But this technology did not really take off until the 1990's with the widespread adoption of <a href="http://en.wikipedia.org/wiki/C%3D%3D">C++</a>.</div><div><br /></div><div>As soon as business coders started to regularly cast complex business systems into objects they began to notice a fundamental problem. Hierarchical object domains of de-normalised state data do not look anything like the relational, normalised data used by RDBs.</div><div><br /></div><div>But the companies who provided the RDB monolith software and the generation of coders who had invested their youth evangelising RDB tech could not accept the <a href="http://en.wikipedia.org/wiki/Cognitive_dissonance">cognitive dissonance</a> this observation created. It became imperative to the profits of the RDB suppliers and the reputations of the RDB evangelists that some way to <span class="Apple-style-span" style="font-style: italic;">ignore the problem</span> was found. Thus the<span class="Apple-style-span" style="font-style: italic; "> data-driven design </span><span class="Apple-style-span" style="">methodology</span><span class="Apple-style-span" style="font-style: italic; "> </span>was born.</div><div><br /></div><div>Data-driven design demands that the<span class="Apple-style-span" style="font-style: italic;"> relational database </span>must be designed first. Only then can the database be <span class="Apple-style-span" style="font-style: italic;">translated </span>into the radically different structure of the object domain and to perform this translation we must write not one but <span class="Apple-style-span" style="font-style: italic;">two </span>additional layers of logic.</div><div><ol><li><a href="http://en.wikipedia.org/wiki/Stored_procedures">Stored Procedures</a> (Sprocs)<br /></li><li>The <span class="Apple-style-span" style="font-style: italic; "><a href="http://en.wikipedia.org/wiki/Data_access_layer">Data Access Layer</a> (<span class="Apple-style-span" style="font-style: normal; ">DAL)</span></span><br /></li></ol></div><div><span class="Apple-style-span" style="font-weight: bold;"><span class="Apple-style-span" style="font-size:large;">Stored Procedures</span></span></div><div>Codd's original stored procedure language was called SEQUEL, which IBM reworked and cheekily named the 'new' language SQL.</div><div><br /></div><div>SQL is an excellent language for constructing data queries and hence mining data sets. SQL also has a range of management features - you can Add, Edit and Delete the structured data on the disk and the RDB data itegrity rules will help prevent the data getting messed up. These management features are nice but they are not the core purpose of SQL, which remains the ability to <span class="Apple-style-span" style="font-style: italic;">perform complex data retrieval by exploiting the relationship</span><span class="Apple-style-span" style="font-style: italic;">s</span> created in the structured data. </div><div><br /></div><div>Yet it is precisely these data management features (add, delete etc), along with some typically trivial queries, that are used 99% by object-oriented coders in order to save and retrieve (serialize) their object state data. </div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;"><span class="Apple-style-span" style="font-size:large;">Data Access Layer</span></span></div><div>Communicating with sprocs from code is a complicated business and requires its own set of coding techniques, objects libraries and tools. These all come together in the data access layer (DAL). The DAL is the code that <span class="Apple-style-span" style="font-style: italic;">marshals </span>object state to and from the Sprocs. </div><div><br /></div><div>This code is typically complex and fragile. As the application requirements evolve, or bugs are discovered and fixed, so the RDB tables, columns etc must be updated to reflect those design changes. This is the heart of <span class="Apple-style-span" style="font-style: italic;">data driven design. </span>It forces a cascade of changes to the sprocs and then changes to the DAL code before finally allowing the class design to be updated.</div><div><br /></div><div style="text-align: center;">Data-Driven Design = <span class="Apple-style-span" style="font-style: italic;"><span class="Apple-style-span" style="font-style: normal; "><span class="Apple-style-span" style="font-style: italic; ">Change</span> <span class="Apple-style-span" style="font-style: italic;">DB </span><span class="Apple-style-span" style="font-style: italic; ">-> <span class="Apple-style-span" style="font-style: normal; "><span class="Apple-style-span" style="font-style: italic; ">Sprocs -> <span class="Apple-style-span" style="font-style: normal; "><span class="Apple-style-span" style="font-style: italic; ">DAL -> <span class="Apple-style-span" style="">Class</span></span></span></span></span></span></span></span></div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;"><span class="Apple-style-span" style="font-size:large;">The Enlightenment</span></span></div><div>At last things are changing. Leading the way is <span class="Apple-style-span" style="font-style: italic;">Domain Driven Design</span>. This is, at its heart, simply a statement of the obvious - that the best way to design an object hierarchy is to design the constituent objects. </div><div><br /></div><div style="text-align: center;">Domain Driven Design = <span class="Apple-style-span" style="font-style: italic; "><span class="Apple-style-span" style="font-style: normal; "><span class="Apple-style-span" style="font-style: italic; ">Change</span> </span><span class="Apple-style-span" style="">Class</span></span></div><div style="text-align: left;"><br /></div><div style="text-align: left;">The domain driven enlightenment has been born out of the fundamental realisation that the old ways of writing software just <span class="Apple-style-span" style="font-style: italic;">do not work</span>. Replacing them is <a href="http://en.wikipedia.org/wiki/Agile_software_development">Agile</a>, a coding methodology that demands that we <span class="Apple-style-span" style="font-style: italic;">refactor</span>, <span class="Apple-style-span" style="font-style: italic;">evolve </span>and <span class="Apple-style-span" style="font-style: italic;">simplify. <span class="Apple-style-span" style="font-style: normal; ">These Agile<span class="Apple-style-span" style="font-style: italic; "> </span>concepts are the very antithesis of data driven design, with its multi-layered, stultifying, <a href="http://en.wikipedia.org/wiki/Baroque">baroque</a> complexity. </span></span></div><div style="text-align: left;"><br /></div><div style="text-align: left;"><span class="Apple-style-span" style="font-style: italic;"><span class="Apple-style-span" style="font-style: normal; ">Thus Agile <span class="Apple-style-span" style="font-style: italic;">demands </span>that we throw out the DAL, the Sprocs and the RDB because they are not, and never were, an appropriate <span class="Apple-style-span" style="font-style: italic;">minimal solution</span> to the problem of object state serialisation.</span></span></div><div style="text-align: left;"><br /></div><div style="text-align: center;"><span class="Apple-style-span" style="font-style: italic;"><span class="Apple-style-span" style="font-size:large;">Object state serialisation </span></span><span class="Apple-style-span" style="font-weight: bold;"><span class="Apple-style-span" style="font-style: italic;"><span class="Apple-style-span" style="font-size:large;">does not</span></span></span><span class="Apple-style-span" style="font-style: italic;"><span class="Apple-style-span" style="font-size:large;"> imply a relational database</span></span><br /></div><div style="text-align: left;"><br /></div><div><span class="Apple-style-span" style="font-weight: bold;">References</span></div><div><ol><li>Codd, E.F. (1970). "A Relational Model of Data for Large Shared Data Banks" In: <i>Communications of the ACM</i> 13 (6): 377–387.<br /></li></ol></div>biofractalhttp://www.blogger.com/profile/11229842750489025189noreply@blogger.com3tag:blogger.com,1999:blog-6927558.post-62023331667854314602009-02-13T16:20:00.066+00:002009-02-19T16:02:18.613+00:00An Irrational Love of the Relational<div style="text-align: center;"><span class="Apple-style-span" style="font-style: italic; font-weight: bold; "><br />Is a relational database the best data-store for serializing object-graphs?</span><br /></div><div><div><span class="Apple-style-span" style="font-style: italic; font-weight: bold;"><br /></span></div><div><span class="Apple-style-span" style="font-weight: bold;">Caveats</span></div><div><ol><li>This applies to coders who employ <a href="http://en.wikipedia.org/wiki/Agile_software_development">Agile</a>, <a href="http://en.wikipedia.org/wiki/Domain_driven_design">Domain Driven Design</a><br /></li><li>The application you are building is not a <a href="http://en.wikipedia.org/wiki/Data_mining">data mining</a> app<br /></li></ol><div>Imagine building an object-oriented application. You find that you need to save and subsequently retrieve an object's state (serialize and de-serialize your object graph). Is a relational database (RDB) the best data store for you?</div><div><br /></div><div>The answer seems pretty clear to me - <span class="Apple-style-span" style="font-style: italic;">Relational databases are just about the worst kind of data store for storing an hierarchical object-graph</span>. </div><div><br /></div><div>Of course an RDB can store object state. Mapping-tables, foreign keys and normalisation can all be used to <span class="Apple-style-span" style="font-style: italic;">project</span> your natural object hierarchy into tables, rows and columns in the way that a globe can be projected onto the flat page of an atlas. </div><div><br /></div><div>But the map is not the terrain, the RDB projection is nothing less than a <span class="Apple-style-span" style="font-style: italic;">distortion</span> of the object graph. Translating between the object-graph and its distorted, relational representation requires work - and not just computer cycles, although it needs plenty of those, but also the hand-rolled code / bugs required to manage the transformation of data to and from the RDB object-store.</div><div><br /></div><div>Traditionally this code was encapsulated into a bespoke Data Access Layer (DAL). It makes me shudder to remember just how much of my life I have wasted writing and debugging DAL code. But all that wasted time is not the critical problem, the real kicker is that a DAL severely limits how <a href="http://en.wikipedia.org/wiki/Agile_software_development">Agile</a> you can be. </div><div><br /></div><div>Take a typical agile process, the quick refactoring of a class definition, say the addition of a new public property.</div><div><ul><li>Add new property to class<br /></li><li>Add equivalent mapping to DAL class<br /></li><li>Add equivalent parameter to SPROC<br /></li><li>Add equivalent column to RDB table<br /></li></ul></div><div>Notice how that list feels back to front? Surely life would easier if I did things the other way around?</div><div><ul><li>Add new column to RDB table</li><li>Add equivalent parameter to SPROC<br /></li><li>Add equivalent mapping to DAL class<br /></li><li>Add equivalent property to class<br /></li></ul></div><div>This shows up another problem with using RDBs - It promotes <span class="Apple-style-span" style="font-style: italic;">data-driven design</span>. </div><div><ul>[<a href="http://en.wikipedia.org/wiki/Data-driven_design">Data Driven Design</a>] = Design your objects by designing <span class="Apple-style-span" style="font-style: italic;">tables</span><br /></ul><ul><span><span>[<a href="http://en.wikipedia.org/wiki/Domain_driven_design">Domain Driven Design</a>] = Design your objects by designing <span class="Apple-style-span" style="font-style: italic;">objects</span></span></span></ul></div>To me the <span class="Apple-style-span" style="">natural</span> way to design a domain is to play with the objects but an RDB + DAL approach flips this natural flow on its head and makes you design backwards. It makes you design the data-store <span class="Apple-style-span" style="font-style: italic;">before </span>the domain, from tables -> objects.</div><div><br /><div>Data driven design means that you do all the work up-front (table, sproc, DAL then finally object) which greatly increases the cost of experimentation. This <span class="Apple-style-span" style="font-style: italic;">ossifies the design process</span>. Data-driven design strongly inhibits a successful design evolving out a series of cheap experiments. This is why agile coders tend to use domain driven design.</div><div><br /></div><div>So why suffer all this RDB pain? Why not use a data store whose intrinsic architecture fits the structure of an object graph and does not require piles of buggy DAL code just to satisfy the basics of object serialization? DBAs often cite two main reasons:</div><div><ol><li>You can run reports that cut across the object graph<br /></li><li>You can keep you data <span class="Apple-style-span" style="font-style: italic;">application agnostic</span> allowing future applications to use the data</li></ol></div><div><span class="Apple-style-span" style="font-weight: bold;">Reason 1</span> - Because I am not writing a data-mining app I know my report designs in advance. Therefore I don't need ad-hoc, dynamic reports. Since my reports are pre-defined they can be represented in my object-graph as a collection of serializable report objects. Reports are just filtered collections of report objects. </div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;">Reason 2</span> - I adhere to strict <a href="http://en.wikipedia.org/wiki/You_Ain't_Gonna_Need_It">YAGNI</a> (you aren't going to need it) principles therefore <span class="Apple-style-span" style="font-style: italic;">I only write code to do the job</span>. No matter how tempting it is, I do not write code <span class="Apple-style-span" style="font-style: italic;">just in case</span> <span class="Apple-style-span" style="font-style: italic;">it might be needed in the future</span>. </div><div><br /></div><div>So what can the modern object-oriented coder do to make life a bit easier?</div><div><br /></div><div><span class="Apple-style-span" style="font-weight: bold;">Use an Object Database</span></div><div>You can use a database that has been specifically designed to store object state data. They are called <a href="http://en.wikipedia.org/wiki/ODBMS">object databases</a>. I have played around with the open source object database called <a href="http://www.db4o.com/">DB4Objects</a> which I found to be very fast and easy to use (as easy as NHibernate) but when I was playing its development was in a state of rapid flux. If you have used an object database then please leave a comment.</div><div><br /></div><div><span class="Apple-style-span" style=""><span class="Apple-style-span" style="font-weight: bold;">Scrap the DAL</span></span></div><div>Object Relational Mappers (ORM) e.g <a href="http://en.wikipedia.org/wiki/NHibernate">NHibernate</a> allow you to<span class="Apple-style-span" style="font-style: italic;"> scrap the DAL</span>. ORMs automate, as far as possible, the cruddy DAL code and get you closer to the agile ideal of fast and cheap refactoring. You can create new classes, add and remove and interface elements and the ORM will take care of adding new tables and columns. You need never write another object persistence save / update / delete SPROC again (almost). </div><div><br /></div><div>Whilst much better than writing DAL code, ORMs are not perfect and are not pain free. Every once in a while you have to go to the DB and mess around. This might seem trivial but you will be amazed how quickly all that DB experience evaporates. </div><div><br /></div><div>Also, to ORM-enable your objects you need to somehow provide the mapping information that links the objects to their equivalent tables. These days this can be handled by marking up your classes with fairly simple [attribute] metadata but it is still a surprising amount of work to keep the attributes up to date.</div><div><br /></div><div>So ORMs are not the final answer because they are really just papering over the the underlying problem: A<span class="Apple-style-span" style="font-style: italic;"> relational database is <span class="Apple-style-span" style="font-weight: bold;">not </span>the best data store for serializing object-graphs</span>.</div><div><br /></div></div></div>biofractalhttp://www.blogger.com/profile/11229842750489025189noreply@blogger.com1tag:blogger.com,1999:blog-6927558.post-61147658153263421712009-02-11T20:28:00.023+00:002009-02-19T16:02:36.075+00:00Code Coverage: How much is enough?How much code should you cover with unit tests. How much code coverage is enough?<br /><br />This was a question that came up in our regular technical team meeting down the pub the other day. I have been pretty evangelical about unit tests ever since the penny dropped about a year ago but it has to be said that I am struggling to stick to the high moral standards set by the 100% code-coverage purists.<br /><br />You all know my grubby little excuses. Unless you live in some coding Utopia then time pressure, last minute changes that need to be deployed now, flashes of code-god inspiration that require input <span style="font-style:italic;">right now</span> because it is just too exciting to wait etc. I admit it - my flesh is weak.<br /><br />But should I be so hard on myself? Is 100% really worth aiming for? At what point does the law of diminishing returns kick in? Nobody I have spoken to seems to have any evidence (as opposed to anecdotal opinion) one way or the other. It boils down to unsupported assertions like "<span class="Apple-style-span" style="font-style: italic;">you should at least aim for > 95% code coverage</span>".<br /><br />I wonder if there is more efficient way to balance time spent on unit tests and the value those test return. Fighting entropy requires a lot of work so perhaps we reduce the work by breaking the problem into smaller pieces and create a <span class="Apple-style-span" style="font-style: italic;">hierarchy of code coverage</span>.<br /><ul><li><span class="Apple-style-span" style="font-weight: bold;">Public Interfaces</span> <br />These should have unit tests created for each possible interaction with the interface members. There should be an iron rule that public interfaces have 100% code coverage. <br /><br /></li><li><span class="Apple-style-span" style="font-weight: bold; ">Black Box Code<br /><span class="Apple-style-span" style="font-weight: normal; ">The plumbing code that supports the public interface functionality will be covered by tests as required by TDD (test driven development) but new tests should only be written for new bug fixes and any obvious TDD development. </span></span></li></ul><br />In a perfect world I accept that 100% is the ideal, but in a world where I need to make money for my company then <span class="Apple-style-span" style="font-style: italic;">100% code coverage 100% of the time</span> is surely too much.<br /><br />100% coverage for selected, critical code and significantly less elsewhere might just be the way to go.biofractalhttp://www.blogger.com/profile/11229842750489025189noreply@blogger.com0tag:blogger.com,1999:blog-6927558.post-31190644892117075352009-02-10T21:42:00.002+00:002011-09-06T10:42:48.094+01:00Agile Technical SpecificationsA technical specification document attempts to bridge the gap between a business driven functional specification and the raw code pouring from a developers mind.<br />
<br />
Yet technical specifications now belong to a bygone, pre-Agile age. Back then business people and technical managers collaborated to produce a functional specification intended to describe all the features of the final system. The technical specification was then derived from this functional specification. It was a structured document that described the system's architecture and intended features in unambiguous terms.<br />
<br />
It is now understood that this top-down process is unrealistic. Agile is, in part, a response to the divergence between pragmatic coding and 'command-economy' management. Agile accepts that nobody can really specify a software system before the first versions of the code are written. Nowadays we realise that code must evolve alongside the specification with each informing the other and the whole informing the client in a system of mutual feedback loops. The rigid, top-down information flows of yesterday have given way to a much more flexible and dynamic approach to system specification, planning and construction.<br />
<br />
But you don't know what you have until you lose it. Now technical specifications have gone the way of the Dodo I can't help thinking that they were not entirely evil after all. If we imagine a utopian technical specification that really did describe a complete system with no mistakes then this would surely be a very useful document.<br />
<br />
Is there any way we can get to an Agile version of the technical specification? Can we achieve a technical specification that delivers the usefully specific technical information whilst still being agile, flexible and evolutionary?<br />
<br />
I think so. I think that if I was given a set of automatically generated, pre-defined class interfaces and a corresponding set of unit tests then together these would constitute an Agile Technical Specification.<br />
<br />
<span style="font-size: small; font-weight: bold;">[Public Interfaces] + [Unit Tests] = [Agile Technical Specification]</span><br />
<br />
Given a set of interfaces, my job as a coder would be to create an equivalent set of domain classes that implemented those interface contracts.<br />
<br />
Given a set of unit tests my job would be to use all my creative powers to flesh out my domain classes until those unit tests passed.<br />
<br />
If all the unit tests passed then I would have satisfied the functional requirements and this functionality would, perforce, be presented via a public API specified by the interfaces. In other words I would have translated my Agile Technical Specification into working code that both defined and self-certified its own features.<br />
<br />
So how do we 'generate' the interfaces and unit tests that comprise an Agile Technical Specification? Well we certainly don't want to be writing these by hand. That would just put us back into the old boat where time constraints cause the code to relentlessly drift away from the specification. Instead these artefacts need to be automatically generated if they are to be of any use.<br />
<br />
If they are to be auto-generated then what specifies and drives the auto-generation? The answer is an Agile Functional Specification. Technical specifications are always derived from functional specifications, the difference is that now this process will be automatic.<br />
<br />
Fortunately Agile Functional Specifications already exist. They are the sets of user stories that evolve in conjunction with the client or their business representatives. This implies that as the user stories evolve so the Technical Specification will evolve as an automatic derivative.<br />
<br />
To achieve this the user stories must be written in a context-aware structured syntax, in other words a domain specific language (DSL). Then it will be possible to consume the user stories and from them auto-generate both the interfaces and the unit tests required to create an Agile Technical Specification. The interfaces are derived from the user story setups and the unit tests from the user story 'Where' clauses (constraints).<br />
<br />
<span style="font-weight: bold;">References</span><br />
<ul>
<li><a href="http://www.agilemodeling.com/essays/agileDocumentation.htm">Agile Documentation</a></li>
<br />
<li><a href="http://www.agilemodeling.com/essays/tagri.htm">TAGRI </a>(they aren't going to read it)</li>
</ul>
biofractalhttp://www.blogger.com/profile/11229842750489025189noreply@blogger.com3