khigia2009-10-19T21:53:33+08:00/Ludovic Coquellelcoquelle@gmail.comerlocaml wc map-reduce example2008-05-20T00:00:00+08:00//wp-import/erlang/2008/05/20/erlocaml-wc-map-reduce-exampleNew example in ocamerl lib of <a href="http://code.google.com/p/erlocaml/">erlocaml</a> project: a tiny map-reduce-like (1) word-count program (word-count is the simple example <a href="http://www.michael-noll.com/wiki/Writing_An_Hadoop_MapReduce_Program_In_Python">often</a> <a href="http://code.google.com/edu/parallel/mapreduce-tutorial.html">used</a> to illustrate map-reduce principle).<br/>
<a href="http://code.google.com/p/erlocaml/source/browse/trunk/lib/ocamerl/ex/ex_node_wc.ml?r=51">First part</a> of this example is an ocaml distributed erlang node [<a href="http://code.google.com/p/erlocaml/source/browse/trunk/lib/ocamerl/ex/ex_node_wc.ml?r=51#52">l52</a>]. It is composed of one permanent registered ('wc') mbox [<a href="http://code.google.com/p/erlocaml/source/browse/trunk/lib/ocamerl/ex/ex_node_wc.ml?r=51#40">l40</a>] which upon request create other mboxes implementing the mappers [<a href="http://code.google.com/p/erlocaml/source/browse/trunk/lib/ocamerl/ex/ex_node_wc.ml?r=51#19">l19</a>] (on request, a mapper read a text file [<a href="http://code.google.com/p/erlocaml/source/browse/trunk/lib/ocamerl/ex/ex_node_wc.ml?r=51#8">l8</a>] and send it word by word to reducer [<a href="http://code.google.com/p/erlocaml/source/browse/trunk/lib/ocamerl/ex/ex_node_wc.ml?r=51#29">l29</a>]).<br/>
<a href="http://code.google.com/p/erlocaml/source/browse/trunk/lib/ocamerl/ex/ex_wc.erl?r=51">Second part</a> of this map-reduce example is an erlang module implementing two functionalities. One of them is the reduce part implementation [<a href="http://code.google.com/p/erlocaml/source/browse/trunk/lib/ocamerl/ex/ex_wc.erl?r=51#34">l34</a>] done by a process which accept messages from mappers and update a dictionary accordingly.
Second responsibility of this module is orchestration of the map-reduce flow [<a href="http://code.google.com/p/erlocaml/source/browse/trunk/lib/ocamerl/ex/ex_wc.erl?r=51#90">l90</a>], consisting of running all mappers [<a href="http://code.google.com/p/erlocaml/source/browse/trunk/lib/ocamerl/ex/ex_wc.erl?r=51#66">l66</a>], waiting end of all those executions [<a href="http://code.google.com/p/erlocaml/source/browse/trunk/lib/ocamerl/ex/ex_wc.erl?r=51#79">l79</a>], and collecting result of all reducers [<a href="http://code.google.com/p/erlocaml/source/browse/trunk/lib/ocamerl/ex/ex_wc.erl?r=51#95">l95</a>].<br/>
Assuming EPMD is running, <a href="http://code.google.com/p/erlocaml/source/browse/trunk/lib/ocamerl/test/wc.sh?r=51">this script</a> run an ocaml node and an erlang node, and run a simple map-reduce to count words in two test files.<br/>
As usual, not sure it's a useful example, but it was sure fun to write :)<br/>
<em>(1) this is not a map-reduce lib ... there is no failure case taken care of, processes pipe is mostly done through messages over network (while map-reduce is mainly meant to use an efficient filesystem based communication), etc!</em>Ocamerl - Erlang ... echo-ing in shells2008-05-13T00:00:00+08:00//wp-import/erlang/2008/05/13/ocamerl-erlang-echo-ing-in-shellsLittle intro step-by-step to the ocaml-erlang message exchange mechanism offered by ocamerl (a lib of erlocaml).<br/>
Subject of the following code is a wonderfull application ... named "echo server"!
<ol>
<li>In the interpretor shell of ocaml, we create a hidden erlang node and an activity to echo all received message (both on console as well as in respond to the caller).</li>
<li>In the erlang interpretor shell, we send few message and compare them to the received ones.</li>
</ol>
The following assumes code is running on machine "comp1".<br/>
Ocaml shell:<br/>
<pre><code>
(* getting Ocamerl available in toplevel *)
ocaml> #use "topfind";; (* ocamerl used findlib for install *)
ocaml> #thread;; (* ocamerl is multithreaded app *)
ocaml> #require "ocamerl";;
(* creating/declaring a node, assuming epmd is running *)
ocaml> let o1 = Ocamerl.Enode.run "o1" ~cookie:"cookie";;
val o1 : Ocamerl.Enode.t = <abstr>
(* creating/declaring a mbox, equivalent of erlang process *)
ocaml> let m = Ocamerl.Enode.create_mbox o1;;
val m : Ocamerl.Enode.Mbox.t = <abstr>
ocaml> Ocamerl.Enode.register_mbox o1 m "echoer";; (* give it a name *)
- : unit = ()
ocaml> Ocamerl.Enode.Mbox.create_activity m (fun msg -> match msg with
| Ocamerl.Eterm.ET_tuple [|pid; any;|] ->
Printf.eprintf "MSG:%s\n%!" (Ocamerl.Eterm.to_string msg);
Ocamerl.Enode.send o1 pid any
| _ ->
() (* drop unexpected msg *)
);;
- : unit = ()
</code></pre>
Erlang shell:<br/>
<pre><code>
# starting erlang node with same cookie
erl -sname e1 -setcookie cookie
% check connection
erl> pong = net_adm:ping(o1@comp1).
pong
% utility to print whatever is in message queue
erl> F = fun() -> receive W -> io:format("got back: ~w~n", [W]) after 1 -> error end end.
#Fun<erl_eval.20.67289768>
% some tests ... send data, received it back
erl> {echo1, o1@comp1} ! {self(), {1,2,"test"}}.
{<0.37.0>,{1,2,"test"}}
erl> F().
got back: {1,2,[116,101,115,116]}
ok
% in the mean time, ocaml shell also display the data
</code></pre>
That's it! A wonderfull echo server :)<br/>
Amongst things on the to-do list:
<ul>
<li>Should not have to create a mbox and set its activity separately (need some wrapper)</li>
<li>Could have an onode ocaml toplevel which run one node by default and offer direct interface (e.g. "send").</li>
</ul>
Erlang to call an Ocaml seam carving lib2008-05-10T00:00:00+08:00//wp-import/erlang/2008/05/10/erlang-to-call-an-ocaml-seam-carving-libYet another useless example for <a href="http://code.google.com/p/erlocaml">erlocaml</a>, but this time it is not completely silly ... it does something :)<br/>
In fact the project <a href="http://github.com/khigia/eocarve/tree/master">eocarve</a> uses:
<ul>
<li>the <a href="http://code.google.com/p/erlocaml/wiki/OcamerlIntro">ocamerl</a> library (which got a <a href="http://code.google.com/p/erlocaml/source/browse/trunk/lib/ocamerl/Makefile">makefile install rule</a> for the occasion);</li>
<li>the <a href="http://eigenclass.org/hiki/seam-carving-in-ocaml">seamcarving</a> library provided by mce (thanks!) (it's LGPL so the full code can be found in eocarve).</li>
</ul>
Idea is simple: an ocaml node run and provide a API for Erlang node to call the seamcarving library. See <a href="http://github.com/khigia/eocarve/wikis">eocarve wiki</a> for details/example.<br/>
Aim of this project was mainly to demonstrate use of ocamerl lib ... however it may be usefull for an Erlang web app which would need seamcarving (heavy weight for CPU!). Had fun to integrate those lib in same app.Tiny progress on erlocaml2008-05-06T00:00:00+08:00//wp-import/erlang/2008/05/06/tiny-progress-on-erlocamlMinor updates for <a href="http://code.google.com/p/erlocaml/">erlocaml</a> (more exactly on <a href="http://code.google.com/p/erlocaml/wiki/OcamerlIntro">ocamerl</a>) ... mostly code cleaning and minor refactoring!<br/>
Only one new feature: ability for Erlang processes to send message to unregistered ocaml processes. An example of that is "ex_node_mult" which generalize "ex_node_double" by dynamically creating ocaml process to perform a multiplication ... yep I know, not very useful<br/>
<pre><code>
% erlang code using the ocaml node
{byn, OcamlNode} ! {self(), 2},
By2 = receive M -> M after 1 -> error,
{byn, OcamlNode} ! {self(), 3},
By3 = receive M -> M after 1 -> error,
P1 = make_ref(),
P2 = make_ref(),
By2 ! {self(), P1, 21},
By3 ! {self(), P1, 21},
ok = receive {P1, 42} -> ok after 1 -> error end,
ok = receive {P2, 63} -> ok after 1 -> error end
</code></pre>
With this feature, ocamerl begin to be usable ... and that's exactly what I will be doing next: experiment with more useful (at least less silly) examples! Some ideas are:
<ul>
<li>tiny mapreduce example with ocaml workers (erlang doing the network part);</li>
<li>using the ocaml image processing lib from erlang (not best example as lot of data need to be exchange ... this is task to be solved for future erlocaml development);</li>
<li>others???</li>
</ul>
ocamerl ... no update :(2008-03-13T00:00:00+08:00//wp-import/erlang/2008/03/13/ocamerl-no-updateQuick news about ocamerl:
<ul>
<li>I changed the build system to ocamlbuild (need ocaml 3.10).</li>
<li>... and that's all! can't find time to clean all the mess.</li>
</ul>
So anyway, here is the current state of ocamerl:
<ul>
<li>ocaml can register a hidden erlang node (epmd).</li>
<li>ocaml node can respond to ping and keep connection up with erlang node (net_kernel process)</li>
<li>erlang can send (some) terms to named ocaml process</li>
<li>ocaml can send (some) terms to erlang process if it received the pid in a previous message</li>
</ul>
That's all for now! Argh!<br/>
Amongst things I'd like to change:
<ul>
<li>simplify terms manipulation in ocaml</li>
<li>add features (most important initial message from ocaml to erlang)</li>
<li>complete redesign of concurrency (using events or JoCaml maybe!)</li>
<li>add a minimum of documentation</li>
</ul>
I need motivation! In fact I have no project which would need ocaml + erlang ... for now.Coming soon: an interesting post about ocamerl2008-01-08T00:00:00+08:00//wp-import/erlang/2008/01/08/coming-soon-an-interesting-post-about-ocamerlManaged to have some time to play with ocamerl: ocaml node is now able to receive data from erlang node, and also send data to pid on connected nodes.<br/>
I begin to understand what I'm doing, so that I will be able to clean the code and refactor some parts very soon.<br/>
As example, <a href="http://erlocaml.googlecode.com/svn/trunk/lib/ocamerl/ocaml/ex/node_double.ml">an ocaml process</a> which multiply integer by 2! wow! What' an interesting example!<br/>
<pre><code>
bash> cd trunk/lib/ocamerl
bash> ./ex/node_double >/dev/null 2>&1 &
bash> erl -setcookie cookie -sname erl
erl@devhost 1> net_adm:ping(ocaml@devhost).
pong
erl@devhost 2> {bytwo, ocaml@devhost} ! {self(), 8}.
{<0.37.0>,8}
erl@devhost 3> receive I -> io:format("Got: ~w~n", [I]) after 0 -> ko end.
Got: 16
ok</code></pre>
Note: the ping was optional and the answer seems correct: 8 * 2 = 16!
Simple game2008-01-02T00:00:00+08:00//wp-import/erlang/2008/01/02/simple-gameIt's kind of surprising how people can like such a simple game as pong. Isn't it even more odd to find programmer which enjoy their program to play?<br/>
All that to say: ocaml node (using the ocamerl lib of <a href="http://code.google.com/p/erlocaml/">erlocaml</a>) can now reply pong to a ping request from erlang node ;) ... and that's all it can do!<br/>
Next step: receive data :)ocamerl update: handshake2007-12-28T00:00:00+08:00//wp-import/erlang/2007/12/28/ocamerl-update-handshakeCode in ocamerl (part of erlocaml) pass the node handshake. Hopefully nobody will read this code before I clean it up a bit ... because right now it is a furious mess!!!<br/>
I spent some time (arg, that's a lot in fact) on a stupid (as the bug's author) bug. The (simplified) handshake is a follow:
<ul>
<li>the 2 nodes exchange a challenge which is 4 random bytes interpreted as a Int32;</li>
<li>each node computes the MD5 of (their own cookie + challenge);</li>
<li>each node compares its result with the digest of the other node.</li>
</ul>
My bug was to concatenate the cookie (string) with the raw 4 bytes of the challenge ... where I had to concatenate the cookie with the string representation of the 32 bits long unsigned integer!<br/>
(in fact the handshake is done with 2 challenges: each node send a challenge to the other)<br/>
Anyway, the handshake pass correctly now, and ocaml node can receive the ping control message ... but do not reply yet! after it reply, sure I clean the code!!!Concurrent behaviours2007-12-22T00:00:00+08:00//wp-import/erlang/2007/12/22/concurrent-behavioursAs spotted in this <a href="http://gilesbowkett.blogspot.com/2007/12/erlang-robotics.html">blog post</a> using Erlang in robotics seems a pretty good idea to implement robust systems and concurrent behaviours.<br/>
However I do not consider the subsumption architecture (Brooks, 1986) as the best example of action selection architecture for concurrent behaviours. As far as I remember, the subsumption architecture assumes predefined priorities on behaviours (hierarchy), and has a fix set of rules on how some behaviours can subsume others.<br/>
As far as reactive architectures are concerned, IMHO Erlang style of concurrency could be more effectively used on architecture using activation networks (Maes, 1989), or a voting mechanism <i>a la</i> DAMN (Rosenblatt, 1995), or a pertinence-based selection like Creature (Blumberg, 1997), or whatever architecture which enable a distribution of control.<br/>
But yes, Erlang seems quite a good match to implement autonomous entities (robots or software agents (MAS)).ocamerl2007-12-16T00:00:00+08:00//wp-import/erlang/2007/12/16/ocamerl Christmas is coming, and I found a new toy to play with: ocamerl. This is a very simple solution to exchange data between erlang and ocaml. In fact, this is mainly an ocaml implementation of the erlang external binary format (based on <a href="http://groups.google.com/group/erlocaml-discuss/browse_thread/thread/920bc75a60a3e2e5">this discussion/code</a>). I just added few tools to make a TCP connection to exchange those data.<br/>
On erlang side, I use 2 processes: the receiver process listen on a TCP port,read data and send it back to any erlang process meant to handle incoming data; the second process is busy transfering all data in its message queue to some TCP server (the ocaml part). On ocaml side, there are a couple a function to transform erlang forms in ocaml entities and vice versa.<br/>
I only wrote a very very simple exemple: an ocaml echo server, which multiply by 2 all integers sent in message (deep search in tuple).<br/>
Erlang side init:<br/>
<pre><code>
1>Receiver = spawn(oe, print_mailbox, []).
2>Sender = oe:messenger(12345, 54321, Receiver).</code></pre>
Ocaml side init:
<pre><code>
./ex/double -recv 12345 -send 54321</code></pre>
Then, any term sent to from erlang is sent back by ocaml.
<pre><code>
3> Sender ! {true, 12, {1,2}}.
{true,12,{1,2}}
Sending data: <<131,104,3,100,0,4,116,114,117,101,97,12,104,2,97,1,97,2>>
Received binary: <<131,104,3,100,0,4,116,114,117,101,97,24,104,2,97,2,97,4>>
Converted to term: {true,24,{2,4}}
Received: {true,24,{2,4}}</code></pre>
Funny, isn't it ? :) This is an early stage of course. Code is part of <a href="http://code.google.com/p/erlocaml">erlocaml project</a> (in lib/ocamerl). My goal is to write a hidden erlang node in ocaml. Not sure it is useful, but quite certain it is interesting ;)
Morphing ruby to erlang just for fun2007-11-11T00:00:00+08:00//wp-import/erlang/2007/11/11/morphing-ruby-to-erlang-just-for-fun<p>Read some introductions to theory of categories (a <a href="http://math.ucr.edu/home/baez/week73.html">very good introduction </a>to the theory, or this one <a href="http://reperiendi.wordpress.com/2007/11/03/category-theory-for-the-java-programmer">more programmer oriented</a>). As I also stumble upon <a href="http://groups.google.com/group/comp.lang.ruby/browse_thread/thread/1052c289b22c60a5">this thread</a>, I discover a new concept: "hylomorphism is a composite of an anamorphism (unfold) and an catamorphism (fold/inject) <a href="http://tunes.org/wiki/Morphism">[1]</a>, <a href="http://www.cs.nott.ac.uk/~cvh/hylos/hylos.pdf">[2]</a>", along with a ruby code to implement it in some way (this the googlegroup thread above).</p>
<p>Just for fun (and be sure to understand it), I translated the code in Erlang. I've used the property list module to implement some kind of default value, but it's not really elegant. Anyway, code is here.</p>
<pre><code>
-module(hylo).
-export([
% hylomorphism API
new/1,
% examples/toys
evens/1,
fact/1,
to_bin/1,
expand/1
]).
% internal declarations
-record(hylo, {
do,
till,
collecting,
injecting
}).
% API
new(PL) when is_list(PL) ->
H = #hylo{
do = proplists:get_value(
do,
PL,
fun(X) -> X + 1 end
),
till = proplists:get_value(
till,
PL,
fun(X) -> X =:= undefined end
),
collecting = proplists:get_value(
collecting,
PL,
fun(X) -> X end
),
injecting = proplists:get_value(
injecting,
PL,
{ [],
fun(A,E) -> [E|A] end,
fun lists:reverse/1
}
)
},
fun(S) -> eval(H, S, element(1,H#hylo.injecting)) end.
% internal implementation
eval(H, S1, R) ->
{_, InjF, InjR} = H#hylo.injecting,
case (H#hylo.till)(S1) of
false ->
V = (H#hylo.collecting)(S1),
S2 = (H#hylo.do)(S1),
eval(H, S2, InjF(R, V));
_ ->
InjR(R)
end.
% examples
evens(N) ->
H = new([
{do, fun(X) -> X + 2 end},
{till, fun(X) -> X >= N end}
]),
H(0).
fact(N) when N > 0 ->
H = new([
{do, fun(X) -> X - 1 end},
{till, fun(X) -> X =< 1 end},
{injecting, {1, fun(A,E) -> A * E end, fun(A) -> A end}}
]),
H(N).
to_bin(N) when N > 0 ->
H = new([
{do, fun(X) -> X div 2 end},
{till, fun(X) -> X =< 0 end},
{collecting, fun(X) -> (X rem 2) end},
{injecting, {[], fun(A,E) -> [E|A] end, fun(A) -> A end}}
]),
H(N).
expand(L) ->
H = new([
{do, fun ([_|T]) -> T; ([]) -> [] end},
{till, fun ([]) -> true; (_) -> false end},
{collecting, fun([{C,N}|_]) -> lists:duplicate(N, C) end}
]),
H(L).
-ifdef(EUNIT).
-include_lib("eunit/include/eunit.hrl").
evens_test() ->
?assert(evens(10) =:= [0,2,4,6,8]).
fact_test() ->
?assert(fact(5) =:= 120).
to_bin_test() ->
?assert(to_bin(10) =:= [1,0,1,0]).
expand_test() ->
?assert(expand([{$a,2},{$b,3},{$c,4}]) =:= ["aa", "bbb", "cccc"]).
-endif.</code></pre>
Erlang walk in AVI file2007-10-09T00:00:00+08:00//wp-import/erlang/2007/10/09/erlang-walk-in-avi-fileWarning: long post ahead ... but mostly code :)<br/>
This post contains absolutely no idea nor thought: it is just a recap of my attempt to read an AVI file format (or RIFF file format, as I do not parse AVI data but only document structure). Let's go directly in code with this simple module header!
<pre><font size="-1">
<font color="#2e8b57"><strong>-module</strong></font>(avir)<font color="#6a5acd">.</font><font color="#2e8b57"><strong>
-compile</strong></font>([export_all])<font color="#6a5acd">.</font>
<font color="#2e8b57"><strong>-include</strong></font><font color="#6a5acd">_</font>lib(<font color="#ff00ff">"kernel/include/file.hrl"</font>)<font color="#6a5acd">.</font><br/>
dbg(Level, Template, Args) <font color="#a52a2a"><strong>-></strong></font>
Indent <font color="#a52a2a"><strong>=</strong></font> <font color="#008b8b">lists</font><font color="#6a5acd">:</font><font color="#008b8b">flatten</font>(<font color="#008b8b">lists</font><font color="#6a5acd">:</font><font color="#008b8b">duplicate</font>(Level, <font color="#ff00ff">" "</font>)),
<font color="#008b8b">io</font><font color="#6a5acd">:</font><font color="#008b8b">format</font>(Indent <font color="#a52a2a"><strong>++</strong></font> Template, Args)<font color="#6a5acd">.</font><br/>
go() <font color="#a52a2a"><strong>-></strong></font>
go(<font color="#ff00ff">"test.avi"</font>)<font color="#6a5acd">.</font><br/>
go(Filename) <font color="#a52a2a"><strong>-></strong></font>
{ok, #file_info{<font color="#008b8b">size</font><font color="#a52a2a"><strong>=</strong></font>Size}} <font color="#a52a2a"><strong>=</strong></font> <font color="#008b8b">file</font><font color="#6a5acd">:</font><font color="#008b8b">read</font><font color="#6a5acd">_</font><font color="#008b8b">file</font><font color="#6a5acd">_</font><font color="#008b8b">info</font>(Filename),
{ok, IODev} <font color="#a52a2a"><strong>=</strong></font> <font color="#008b8b">file</font><font color="#6a5acd">:</font><font color="#008b8b">open</font>(Filename, [read, <font color="#008b8b">binary</font>]),
{ok, Parts} <font color="#a52a2a"><strong>=</strong></font> walk_data(<font color="#ff00ff">0</font>, [], IODev, <font color="#ff00ff">0</font>, Size)<font color="#6a5acd">.</font></font></pre>
So, <code>dbg</code> is a crap function to print debug message ... yeah, the old fashion way, it's so simple for just a post! <code>go</code> is the main entry point and call the 'real' code: the approach is to call the <code>walk_data</code> function which will build and return a list of AVI structures (first parameter will be level of nesting, used for printing comment with a meaningful indentation, and second one is an accumulator for recursion to come).<br/>
I mainly use this <a href="http://www.alexander-noe.com/video/documentation/avi.pdf">short document</a>: AVI is a (nested) sequence of two kind of structure, either LIST or CHUNK. More precisely, first come a mandatory RIFF-AVI LIST then multiple (and optional) RIFF-AVIX kind of LIST. Let's walk those structures:
<pre><font size="-1">
walk_data(Level, Parts, File, From, To) <font color="#a52a2a"><strong>when</strong></font> From <font color="#a52a2a"><strong><</strong></font> To <font color="#a52a2a"><strong>-></strong></font>
<font color="#a52a2a"><strong>case</strong></font> chunk_or_list(File, From) <font color="#a52a2a"><strong>of</strong></font>
avichunk <font color="#a52a2a"><strong>-></strong></font>
{ok, Part, NextPos} <font color="#a52a2a"><strong>=</strong></font> walk_chunk(Level, File, From, To),
walk_data(Level, [Part|Parts], File, NextPos, To);
avilist <font color="#a52a2a"><strong>-></strong></font>
{ok, Part, NextPos} <font color="#a52a2a"><strong>=</strong></font> walk_list(Level, File, From, To),
walk_data(Level, [Part|Parts], File, NextPos, To);
Error <font color="#a52a2a"><strong>-></strong></font>
{error, <font color="#ff00ff">"maybe unexpected EOF"</font>, Error}
<font color="#a52a2a"><strong>end</strong></font>;
walk_data(<font color="#6a5acd">_</font>Level, Parts, <font color="#6a5acd">_</font>File, <font color="#6a5acd">_</font>From, <font color="#6a5acd">_</font>To) <font color="#a52a2a"><strong>-></strong></font>
{ok, <font color="#008b8b">lists</font><font color="#6a5acd">:</font><font color="#008b8b">reverse</font>(Parts)}<font color="#6a5acd">.</font><br/>
chunk_or_list(File, Pos) <font color="#a52a2a"><strong>-></strong></font>
<font color="#a52a2a"><strong>case</strong></font> <font color="#008b8b">file</font><font color="#6a5acd">:</font><font color="#008b8b">pread</font>(File, Pos, <font color="#ff00ff">4</font>) <font color="#a52a2a"><strong>of</strong></font>
{ok, <font color="#a52a2a"><strong><<</strong></font><font color="#ff00ff">"RIFF"</font><font color="#a52a2a"><strong>>></strong></font>} <font color="#a52a2a"><strong>-></strong></font>
avilist;
{ok, <font color="#a52a2a"><strong><<</strong></font><font color="#ff00ff">"LIST"</font><font color="#a52a2a"><strong>>></strong></font>} <font color="#a52a2a"><strong>-></strong></font>
avilist;
{ok, <font color="#6a5acd">_</font>FourCC} <font color="#a52a2a"><strong>-></strong></font>
avichunk;
eof <font color="#a52a2a"><strong>-></strong></font>
eof
<font color="#a52a2a"><strong>end</strong></font><font color="#6a5acd">.</font>
</font></pre>
Walk is straightforward, from position <code>From</code> to <code>To</code>, accumulating result in reverse order (I love this <code>[head|tail]</code> list notation ... was Prolog the first to use it?). <code>chunk_or_list</code> read few bytes (the <code>FourCC</code> header) to guess the kind of the next structure (CHUNK or LIST) in file; this structure is loaded, and walk continue.
<pre><font size="-1">
walk_list(Level, File, From, To) <font color="#a52a2a"><strong>-></strong></font>
<font color="#a52a2a"><strong>case</strong></font> read_list_header(File, From) <font color="#a52a2a"><strong>of</strong></font>
{ok, AviList<font color="#a52a2a"><strong>=</strong></font>{avilist, List, FourCC, DataPos, DataSize}, NextPos} <font color="#a52a2a"><strong>-></strong></font>
dbg(Level, <font color="#ff00ff">"read list header (pos=</font><font color="#6a5acd">~p</font><font color="#ff00ff">, next=</font><font color="#6a5acd">~p</font><font color="#ff00ff">): List=</font><font color="#6a5acd">~p</font><font color="#ff00ff"> FourCC=</font><font color="#6a5acd">~p~n</font><font color="#ff00ff">"</font>, [From, NextPos, List, FourCC]),
{ok, SubPart} <font color="#a52a2a"><strong>=</strong></font> <font color="#a52a2a"><strong>case</strong></font> FourCC <font color="#a52a2a"><strong>of</strong></font>
<font color="#a52a2a"><strong><<</strong></font><font color="#ff00ff">"movi"</font><font color="#a52a2a"><strong>>></strong></font> <font color="#a52a2a"><strong>-></strong></font>
dbg(Level, <font color="#ff00ff">"... skipping list FourCC=</font><font color="#6a5acd">~p</font><font color="#ff00ff">...</font><font color="#6a5acd">~n</font><font color="#ff00ff">"</font>, [FourCC]),
{ok, []};
<font color="#6a5acd">_</font> <font color="#a52a2a"><strong>-></strong></font>
walk_data(Level <font color="#a52a2a"><strong>+</strong></font> <font color="#ff00ff">1</font>, [], File, DataPos, DataPos <font color="#a52a2a"><strong>+</strong></font> DataSize)
<font color="#a52a2a"><strong>end</strong></font>,
{ok, {AviList, SubPart}, NextPos};
eof <font color="#a52a2a"><strong>-></strong></font>
dbg(Level, <font color="#ff00ff">"end of file</font><font color="#6a5acd">~n</font><font color="#ff00ff">"</font>, []),
eof
<font color="#a52a2a"><strong>end</strong></font><font color="#6a5acd">.</font><br/>
read_list_header(File, Pos) <font color="#a52a2a"><b>-></b></font>
<font color="#a52a2a"><b>case</b></font> <font color="#008b8b">file</font><font color="#6a5acd">:</font><font color="#008b8b">pread</font>(File, [{Pos, <font color="#ff00ff">4</font>}, {Pos <font color="#a52a2a"><b>+</b></font> <font color="#ff00ff">4</font>, <font color="#ff00ff">4</font>}, {Pos <font color="#a52a2a"><b>+</b></font> <font color="#ff00ff">8</font>, <font color="#ff00ff">4</font>}]) <font color="#a52a2a"><b>of</b></font>
{ok, [List, <font color="#a52a2a"><b><<</b></font><font color="#008b8b">Size</font><font color="#6a5acd">:</font><font color="#ff00ff">4</font><font color="#a52a2a"><b>/</b></font>little-unsigned<font color="#a52a2a"><b>-</b></font><font color="#008b8b">integer</font><font color="#a52a2a"><b>-</b></font><font color="#008b8b">unit</font><font color="#6a5acd">:</font><font color="#ff00ff">8</font><font color="#a52a2a"><b>>></b></font>, FourCC]} <font color="#a52a2a"><b>-></b></font>
{ok, {avilist, List, FourCC, Pos <font color="#a52a2a"><b>+</b></font> 12, Size <font color="#a52a2a"><b>-</b></font> <font color="#ff00ff">4</font>}, Pos <font color="#a52a2a"><b>+</b></font> <font color="#ff00ff">8</font> <font color="#a52a2a"><b>+</b></font> Size};
{ok, [eof, eof, eof]} <font color="#a52a2a"><b>-></b></font>
eof;
<font color="#6a5acd">_</font> <font color="#a52a2a"><b>-></b></font>
{error, <font color="#ff00ff">"no list header to read, but not empty data</font><font color="#6a5acd">~n</font><font color="#ff00ff">"</font>}
<font color="#a52a2a"><b>end</b></font><font color="#6a5acd">.</font>
</font></pre>
To walk a LIST, read the header (remember that the FourCC field length is part of the data size ...), read the nested data (this re-use the walk_data), and return the LIST representation: a 2-tuple with first the header (could be a record) and then a list of sub parts. There is a useless test to not walk the real data because my test file is kind of big. Walking the CHUNK is quite the same.
<pre><font size="-1">
walk_chunk(Level, File, From, To) <font color="#a52a2a"><strong>-></strong></font>
<font color="#a52a2a"><strong>case</strong></font> read_chunk_header(File, From) <font color="#a52a2a"><strong>of</strong></font>
{ok, AviChunk<font color="#a52a2a"><strong>=</strong></font>{avichunk, FourCC, DataPos, DataSize}, NextPos} <font color="#a52a2a"><strong>-></strong></font>
<font color="#0000ff">%FourCC = <<_StreamNumber:2/binary, _DataType:2/binary>>},</font>
dbg(Level, <font color="#ff00ff">"read chunk header (pos=</font><font color="#6a5acd">~p</font><font color="#ff00ff">, next=</font><font color="#6a5acd">~p</font><font color="#ff00ff">): FourCC=</font><font color="#6a5acd">~p</font><font color="#ff00ff"> DataSize=</font><font color="#6a5acd">~p~n</font><font color="#ff00ff">"</font>, [From, NextPos, FourCC, DataSize]),
chunk_spy(FourCC, File, DataPos, DataSize),
{ok, AviChunk, NextPos};
eof <font color="#a52a2a"><strong>-></strong></font>
dbg(Level, <font color="#ff00ff">"end of file</font><font color="#6a5acd">~n</font><font color="#ff00ff">"</font>, []),
eof
<font color="#a52a2a"><strong>end</strong></font><font color="#6a5acd">.</font><br/>
read_chunk_header(File, Pos) <font color="#a52a2a"><strong>-></strong></font>
<font color="#a52a2a"><strong>case</strong></font> <font color="#008b8b">file</font><font color="#6a5acd">:</font><font color="#008b8b">pread</font>(File, [{Pos, <font color="#ff00ff">4</font>}, {Pos <font color="#a52a2a"><strong>+</strong></font> <font color="#ff00ff">4</font>, <font color="#ff00ff">4</font>}]) <font color="#a52a2a"><strong>of</strong></font>
{ok, [FourCC, <font color="#a52a2a"><strong><<</strong></font><font color="#008b8b">Size</font><font color="#6a5acd">:</font><font color="#ff00ff">4</font><font color="#a52a2a"><strong>/</strong></font>little-unsigned<font color="#a52a2a"><strong>-</strong></font><font color="#008b8b">integer</font><font color="#a52a2a"><strong>-</strong></font><font color="#008b8b">unit</font><font color="#6a5acd">:</font><font color="#ff00ff">8</font><font color="#a52a2a"><strong>>></strong></font>]} <font color="#a52a2a"><strong>-></strong></font>
NextPos <font color="#a52a2a"><strong>=</strong></font> Pos <font color="#a52a2a"><strong>+</strong></font> <font color="#ff00ff">8</font> <font color="#a52a2a"><strong>+</strong></font> Size,
PaddedNextPos <font color="#a52a2a"><strong>=</strong></font> NextPos <font color="#a52a2a"><strong>+</strong></font> (NextPos <font color="#a52a2a"><strong>rem</strong></font> <font color="#ff00ff">2</font>),
{ok, {avichunk, FourCC, Pos <font color="#a52a2a"><strong>+</strong></font> <font color="#ff00ff">8</font>, Size}, PaddedNextPos};
{ok, [eof, eof]} <font color="#a52a2a"><strong>-></strong></font>
eof;
<font color="#6a5acd">_</font> <font color="#a52a2a"><strong>-></strong></font>
{error, <font color="#ff00ff">"no chunk header to read, but not empty data</font><font color="#6a5acd">~n</font><font color="#ff00ff">"</font>}
<font color="#a52a2a"><strong>end</strong></font><font color="#6a5acd">.</font>
</font></pre>
Similar to LIST, without nested data. Also, this went wrong at the first attempt: I found in <a href="http://msdn2.microsoft.com/en-us/library/ms779636.aspx">this page</a> that CHUNK data is padded to word boundary (grr).<br/>
But that's all it take to read a well formated RIFF file. And for those wondering about the <code>chunk_spy</code> function, continue to read this blog :).Erlang/OTP installation from sources2007-09-26T00:00:00+08:00//wp-import/erlang/2007/09/26/erlangotp-installation-from-sourcesI'm probably missing something in Erlang/OTP installation from sources ... my goal is to install multiple version of OTP in user space.<br/>
Here is one way to do it:
<ol>
<li> remove any symlink if needed
<code>rm /home/khigia/local/lib/erlang</code></li>
<li>apply the "normal" quick install steps for the new OTP version, let say otp-R11B5
<code>cd /home/khigia/apps/otp-src-R11B5; ./configure --prefix=/home/khigia/local ; make; make install</code></li>
<li>modify the install to be version specific
<code>mv /home/khigia/local/lib/erlang /home/khigia/local/lib/otp-R11B5
ln -s /home/khigia/local/lib/otp-R11B5 /home/khigia/local/lib/erlang</code></li>
</ol>
Changing the symlink enable to switch Erlang/OTP version. There must be an easiest way ... how?Different DBMS2007-09-07T00:00:00+08:00//wp-import/erlang/2007/09/07/different-dbmsIt's common knowledge that DBMS is a very important part of a fast scalable distributed systems as far as performance is concerned. But I often tend to forget it and blindly rely on well known relational DBMS. Recently I stumble upon the following other DBMS/discussions and the last one really opened my eyes:
<ul>
<li>The Erlang/OTP come with <a href="http://www.erlang.org/doc/apps/mnesia/index.html">mnesia</a> allowing transaction on distributed DB. However it has been argue (note: this need some reference...) that distributed mnesia cannot scale (it needs some agreement between all nodes for transaction). For huge distributed system (or nodes on low network) the application has to maintain data consistency without relying on transaction. <a href="http://en.wikipedia.org/wiki/Berkeley_DB">Berkeley DB</a> seems to be the preferred way to go for this, but as I understand it, it is not a DBMS which manage distribution but a tool to help you to do so.</li>
</ul>
<ul>
<li><a href="http://couchdb.org/CouchDB/CouchDBWeb.nsf/Home?OpenForm">CouchDB</a> is one DBMS trying to handle the distribution of DB.</li>
</ul>
<ul>
<li>Also I just discover the terminology column-oriented DBMS through this (too?) simple <a href="http://en.wikipedia.org/wiki/Column-oriented_DBMS">wikipedia article</a>. Being used to R statistics system helped me to (try to) understand what it really is. At least I better understand why Google developed BigTable and I have to <a href="http://labs.google.com/papers/bigtable.html">read</a> or <a href="http://video.google.com/videoplay?docid=7278544055668715642">listen</a> more about it ...</li>
</ul>Sieve again2007-07-27T00:00:00+08:00//wp-import/erlang/2007/07/27/sieve-againI didn't known this was a so popular problem when I coded <a href="http://khigia.wordpress.com/2007/05/07/44/">a sieve</a> of Eratosthene using stream design from SICP.<br/>
In fact, I saw <a href="http://video.google.com/videoplay?docid=810232012617965344">this video</a> about NewSqueak which present an example of a concurrent algorithm of the sieve, and in <a href="http://groups.google.de/group/comp.os.plan9/browse_thread/thread/951130abcfcf506e/8c4267a634a0524b?lnk=raot">this related discussion</a> on plan9 group I also found <a href="http://video.google.com/videoplay?docid=810232012617965344">an Haskell implementation</a> (more related to the stream design than to the concurrent design IMHO). I also had a look at <a href="http://swtch.com/libtask/">libtask</a> which offer a C version of the concurrent sieve.<br/>
That's already a lot, so one more shouldn't be useful ... but I like this example, and thus here is <a href="http://paste.lisp.org/display/45190">my Erlang implementation</a> using a spawn for each prime in the sieve.Simple stream implementation in Erlang2007-05-07T00:00:00+08:00//wp-import/erlang/2007/05/07/44My projects are all suspended ... I have been busy playing with <a href="http://mitpress.mit.edu/sicp/full-text/book/book-Z-H-24.html#%_sec_3.5">stream examples of chapter 3 of SICP</a> :).<br/>
I wrote in Erlang a simple implementation of stream as cons-list. It is not meant to be efficient but easy to understand (and debug):
<ul>
<li>a stream is here a tuple <span style="font-family:courier new,monospace;">{stream, Head, DelayedTail}</span>, where <span style="font-family:courier new,monospace;">DelayedTail</span> is a 0-arity's function which return the next stream element tuple;</li>
<li>a macro implements the delay function used to construct a stream tail:
<span style="font-family:courier new,monospace;">-define(DELAY(Any), fun() -> Any end)</span>.</li>
</ul>
Note: the head is not delayed, and if we need a delayed head this probably means that we need a lazy-evaluation model, to which I'll come later. Using abstract functions, <span style="font-family:courier new,monospace;">head/1</span> to access current value and <span style="font-family:courier new,monospace;">tail/1</span> to force evaluation of next stream state, will enables to hook a memoization process if needed.
<p style="text-align:center;"> </p>
<p style="text-align:left;">Anyway, the stream concatenation and the delay's macro are enough to build all stream manipulations. Here are some examples:</p><br/>
<pre>
% map: application of a procedure to all elements of a stream
map(Proc, Stream={stream, undefined, undefined}) ->
% empty stream: do nothing
Stream;
map(Proc, Stream) ->
% apply Proc to current, and delay apply to all tail elements
{stream, Proc(head(Stream)), ?DELAY(map(Proc, tail(Stream)))}<br/>
% filter elements of stream with respect to some predicate
filter(Pred, Stream={stream, undefined, undefined}) ->
% empty stream: do nothing
Stream;
filter(Pred, Stream={stream, _Head, _Tail}) ->
Current = head(Stream),
case Pred(Current) of
true ->
{stream, Current, ?DELAY(filter(Pred, tail(Stream)))};
_False ->
filter(Pred, tail(Stream))
end.<br/>
% generalized map: apply proc to multiple streams (traverse streams "in parallel")
gmap(Proc, Streams=[First={stream, undefined,undefined}|_Others]) ->
% assume all stream have the same length
% found one stream empty: end of process
First;
gmap(Proc, Streams=[First|_Others]) ->
% create a list of heads
Heads = lists:map(fun(S) -> stream:head(S) end, Streams),
% create a generator of the list of tails
FTails = fun() -> lists:map(fun(S) -> stream:tail(S) end, Streams) end,
% result is the application to heads and delayed mapping to tails
{stream, Proc(Heads), ?DELAY(gmap(Proc, FTails()))}.<br/>
</pre>
And now, we have enough to use the streams:
<pre>
% explicit definition of infinite sequence of integers
explicit_integers_from(N) ->
{stream, N, ?DELAY(explicit_integers_from(N+1))}.<br/>
% add values of 2 streams
add(S1, S2) ->
gmap(fun([X,Y]) -> X + Y end, [S1, S2]).<br/>
% infinite sequence of Fibonacci numbers
fibs() ->
{stream, 0, ?DELAY({stream, 1, ?DELAY(add(fibs(), tail(fibs())))}) }.<br/>
</pre>
A bit more usefull: 2 infinite sequences of prime numbers. First one uses the sieve of Eratosthene; second one uses basic definition of prime: number not divisible by any previous primes.
<pre>
% Sieve of Eratosthene:<br/>
sieve(Stream) ->
Current = head(Stream),
{stream, Current, ?DELAY(sieve(filter(
fun(N) -> N rem Current /= 0 end,
tail(Stream)
)))}.<br/>
sieve_primes() ->
sieve(explicit_integers_from(2)).<br/>
% prime numbers using the factors decomposition property<br/>
primes() ->
{stream, 2, ?DELAY(filter(fun is_prime/1, explicit_integers_from(3)))}.<br/>
is_prime(N) ->
iter_is_prime(N, primes()).<br/>
iter_is_prime(N, Primes) ->
FirstPrime = head(Primes),
if
FirstPrime * FirstPrime > N ->
true;
N rem FirstPrime == 0 ->
false;
true ->
iter_is_prime(N, tail(Primes))
end.<br/>
</pre>
And of course there are lot of stream examples in SICP chapter, like the impressive Euler transformation to accelerate convergence of sequence.<br/>
In this thread <a href="http://erlang.mirror.su.se/ml-archive/erlang-questions/200010/msg00165.html" target="_blank">http://erlang.mirror.su.se/ml-archive/erlang-questions/200010/msg00165.html </a> you can find a real stream implementation by Richard Carlsson, as well as interesting discussions about stream and lazy evaluation. Also check this other thread <a href="http://www.erlang.org/ml-archive/erlang-questions/200602/msg00003.html" target="_blank"> http://www.erlang.org/ml-archive/erlang-questions/200602/msg00003.html</a> if you're interested by memoization in Erlang. I may test it later to learn the magic of Erlang parse_transform ;), but I think an easy possible way to implement memoization for my stream is to model a stream as a tuple of 4 elements, adding a dictionary of memoized values as last element (my current implementation uses the process dictionary to store memoized values).<br/>
Actually I'm not sure if this post is too short to be a tutorial or too long to be a interesting post! Anyway, it exists now ;) ... mostly because a friend gave me motivation to do so, thanks!8-queens ... me too!2007-04-10T00:00:00+08:00//wp-import/erlang/2007/04/10/8-queens-me-tooEven it the <a href="http://en.wikipedia.org/wiki/Eight_queens_puzzle">8-queens puzzle</a> is often used to illustrate the recursivity principle, I never tried to solve it myself. I only wrote my N-queens solution recently (using Erlang), and was pretty happy to see that <a href="http://en.wikipedia.org/wiki/List_comprehension">list-comprehension</a> is quite good for the principle of generate-and-filter. Oh! and I tried the <a href="http://esdl.sourceforge.net/">ESDL</a> library to display the result :) ... I was just as excited as the day I did my first openGL tutorial in C.Positive integer and recursion2007-04-05T00:00:00+08:00//wp-import/erlang/2007/04/05/positive-integer-and-recursionI remember my first steps of Prolog, I was following an exercise to define positive integer as a recursion of the same function ... I did not really known why I did that, but it was fun to define the integers (the funniest Prolog exercise is still to write a Prolog interpretor in Prolog). Today I'm reading <a href="http://mitpress.mit.edu/sicp/full-text/book/book.html">SICP</a> and I just found in Ex2.6 that the recursive integer definition is the basic of lambda calculus: Church' numerals! Definitively, I have to learn this lambda calculus. At least I have now a better understanding of it (<a href="http://www.rubrication.net/2007/01/13/addition-in-the-lambda-calculus/">thanks also to this post</a>) and I wrote <a href="http://paste.lisp.org/display/39267">the addition and multiplication operators in Erlang</a>.grinderl is sinan-ized2007-04-05T00:00:00+08:00//wp-import/erlang/2007/04/05/grinderl-is-sinan-izedForgot to mention: <a href="http://code.google.com/p/grinderl">grinderl</a> now use <a href="http://code.google.com/p/sinan/">sinan</a> as build system (makefiles are still around but should disappear as soon as the tests will be handle by sinan). Hopefully sinan will help to deals with dependencies and release handling.<br/>
Btw I continue to follow the development of the <a href="http://cean.process-one.net/" target="_blank">CEAN</a> repository ...
<a href="http://cean.process-one.net/" target="_blank"> </a>grinderl: less tsung, more capistrano2007-03-31T00:00:00+08:00//wp-import/erlang/2007/03/31/grinderl-less-tsung-more-capistranoI decided to change the roadmap of the <a href="http://code.google.com/p/grinderl/">grinderl</a> project.<br/>
It first application was to be a tiny load-test framework (at least a testing framework as I'm not sure it is efficient enough to create heavy load on a well design server). And I still have to finish the part of creating useful statistics from all the results.<br/>
But now I think that I could implement a better node management API, and then grinderl can become one way to call code on remote host (not exactly as RPC since command are asynchronous and supervised by remote host). One application could be to install an external application on all remote host, or manage deployment like does <a href="http://manuals.rubyonrails.com/read/book/17">capistrano</a> (but with the full Erlang environment available!).<br/>
So the new (of course, it's the first!) roadmap:
<ul>
<li> Stabilize the current version:
<ul>
<li> move test to become command</li>
<li> better handling of results sent by remote hosts</li>
</ul>
</li>
<li> Develop concurrently the remote node API and a UI:
<ul>
<li> API to manage remote hosts
<ul>
<li>use slave module</li>
<li>have a look at <a href="https://support.process-one.net/doc/display/ERL/Starting+a+set+of+Erlang+cluster+nodes">this how-to</a></li>
<li>can we install a minimal Erlang node on a remote machine through ssh ... using CEAN or erlware maybe?</li>
</ul>
</li>
<li>UI: network tools seems to require a network-able UI (probably using yaws) to be able to:
<ul>
<li> see/manage nodes/hosts</li>
<li> send command to all/any hosts, monitor results (looks like each command will need a specialize UI)</li>
</ul>
</li>
</ul>
</li>
</ul>
Conclusion:
<ul>
<li>Sad note: nothing new in grinderl (this project don't seems to appeal my creativity)</li>
<li>Good note: simple project to learn Erlang distributed programming</li>
</ul>List of Erlang data structures2007-02-24T00:00:00+08:00//wp-import/erlang/2007/02/24/list-of-erlang-data-structures-stdlib-contribHow would you implement a huge-but-sparse 2D matrix in Erlang?
<ul>
<li>First solution who came to my mind was to google for some linear algebra library ... that failed!</li>
<li>The second one, implement a dict-of-dict structure.</li>
</ul>
That was quite easy, but as I was wondering on efficiency I needed to understand differences between dict, orddict etc. On my way I came up with this (ongoing) <a href="http://khigia.wordpress.com/erlang-data-structures/">list of Erlang data structures</a>.grinderl: early stage2007-02-10T00:00:00+08:00//wp-import/erlang/2007/02/10/grinderl-early-stageI did it! I have try to code my first OTP applications following most of the recommendation of the documentation (and the French book <a href="http://www.eyrolles.com/Informatique/Livre/index.php?ouv_ean13=9782212110791&societe=remond" target="_blank" title="//www.eyrolles.com/Informatique/Livre/index.php?ouv_ean13=9782212110791&societe=remond" class="externalLink">Erlang programmation</a> was a great help).<br/>
One of those applications became an opensource project: <strong>grinderl</strong>. I did blog a bit about it <a href="http://khigia.wordpress.com/2006/11/25/baby-steps-in-the-erlang-world/" target="_blank" title="//khigia.wordpress.com/2006/11/25/baby-steps-in-the-erlang-world/" class="externalLink">in this blog post</a>, and now it has evolved from the all-in-one-module code to a small ongoing project (<a href="http://code.google.com/p/grinderl/" target="_blank" title="//code.google.com/p/grinderl/" class="externalLink">hosted by</a> googlecode). To do that I used:
<ul>
<li>application/supervisor/server/fsm/event ... all the generic modules: not complicated</li>
<li><a href="http://support.process-one.net/doc/display/CONTRIBS/EUnit" title="eunit">eunit</a>: simple enough, and tests should be compatible with the test_server application</li>
<li>dialyzer: wow! it can save a lot of time, especially because Erlang has no type system: a MUST.</li>
<li> appmon: great and easy to see your supervision tree</li>
<li> debugger: I still prefer to write message in the console ... please don't tell that to anybody!</li>
<li> edoc: I learned how to use it, but sadly it didn't teach me to write good documentation...</li>
</ul>
I may like to have a look at cover and profiling tools, I may want to learn about Erlang port, but I guess the two main points I missed are:
<ul>
<li> mnesia: looks so great!</li>
<li> release handling: this is a tricky part of OTP application distribution I think!</li>
</ul>
And for the last one, I hope the <a href="http://code.google.com/p/sinan/" target="_blank" title="//code.google.com/p/sinan/" class="externalLink">sinan built system</a> can help me (<a href="http://erlangish.blogspot.com/2007/02/code-complete.html" target="_blank" title="//erlangish.blogspot.com/2007/02/code-complete.html" class="externalLink">Eric Merritt promised us a high level user documentation</a> soon).<br/>
My only regret is that I haven't found a complete tutorial to go through all the steps of transformation of an Erlang program in an OTP application release. And that's too sad that I don't feel able to write one!Erlang: the good, the bad and the chubby2007-01-23T00:00:00+08:00//wp-import/erlang/2007/01/23/erlang-the-good-the-bad-and-the-chubby(hoping my <a href="http://www.popachubby.com/" target="_blank" title="//www.popachubby.com" class="externalLink">favorite guitarist</a> don't mind that I borrow one of <a href="http://www.bluesweb.com/p_disque.php3?id_article=286" target="_blank" title="//www.bluesweb.com/p_disque.php3?id_article=286" class="externalLink">his title</a>)<br/>
Some (mostly subjective) comments after few days playing around Erlang:
<ul>
<li><strong> The good:</strong> I have never before implemented a server (relatively efficient TCP proxy) so quickly. Asynchronous call are as easy as synchronous ones and network is invisible! that's good! physician can wrote as many differential equation as they need, synchronous model are a <strong>strong</strong> hypothesis of the world!</li>
<li><strong> The bad:</strong> syntax! I was used to <a href="http://en.wikipedia.org/wiki/Prolog" target="_blank" title="//en.wikipedia.org/wiki/Prolog" class="externalLink">Prolog</a> notation, but Erlang introduce too many meaning for punctuation while newline or indentation has no meaning at all! Anybody interested in Pyerlang (a syntax translator python-inspired)?</li>
<li><strong> The chubby:</strong> my next blog will be about transformation of a simple Erlang server to a (nearly) complete OTP application. Yes, OTP is a good framework, but learning it is far to be as easy as learning Erlang!</li>
</ul>Baby steps in the Erlang world!2006-11-25T00:00:00+08:00//wp-import/erlang/2006/11/25/baby-steps-in-the-erlang-worldIt took a long time to decide <a href="http://khigia.wordpress.com/2006/11/02/what-is-my-next-programming-language/" title="to learn Erlang instead of Haskell or OCaml/F#">to learn Erlang instead of Haskell or OCaml/F#</a>. What I really want to learn for now are the OTP principles of distributed application. So I wrote my first Erlang script (not yet an application) and it confirmed my point of view: Erlang make easy the development of distributed program! really!!!<br/>
At my work we needed to test performance of a TCP server (and we'll used Grinder because most of our API is Python). I've decided to use Erlang to do the same thing as an exercise: grind.erl (of course it will never become neither as huge as <a href="http://grinder.sourceforge.net/" title="Grinder">Grinder</a> neither as efficient as <a href="http://tsung.erlang-projects.org/" title="Tsung">Tsung</a>; I even do not hope it can be usefull). But with few lines of code, I was able to reproduce the same behavior as Grinder: launch on multiple machine multiple tester agents executing some test function and gather the results of all the tests.<br/>
Ok, I did not told all the truth: that's not a very small number of lines: arround 260 (with tests and all). But all the code is for running multiple time a test function and gathering some statistics on its result (run_task). Distributed this behaviour among multiple Erlang node is only a map call (distribute_task)!<br/>
Here is a code extract:
<pre>
distribute_task(NodeLst, Task) ->
NodeLstLen = length(NodeLst),
StatListener = spawn(grind, statistic_gathering, [now(), NodeLstLen]),
io:format(
"~w create statistic gathering process ~w on node ~w~n",
[self(), StatListener, node()]
),
TaskRunnerCreator = fun(Node) -> spawn(
Node,
grind,
run_task,
[StatListener, Task]
) end,
Runners = lists:map(TaskRunnerCreator, NodeLst),
io:format(
"~w create a list of task runners: ~w~n",
[self(), Runners]
),
Runners.</pre>
And here is a usage example:
<pre>
grind:init([node(), grinderl@node.net]).
grind:distribute_task(
[node(), grinderl@node.net], % use 2 nodes
{
% on each node run the test function in 50 concurrent process
{concurrent, 50},
% two statistics to gather
[{mean, writetime}, % a real value (mean, std. dev., min, max, med will be retrieved
{count, writer_val} % a occurence counter
],
% foo function to test: must return a tuple {ok|error, Pid, ValLst}
fun(Writer, WritenValue) ->
FWrite = fun() -> io:format("~s got ~w~n", [Writer, WritenValue]) end,
{WriteTime, _Res} = timeit(FWrite),
{ok, self(), [WriteTime, {Writer, WritenValue}]}
end,
% arguments to used for each call of the test function
[{rr, ["bea", "pouf"]}, % first is taken in the list with a round-robin style
{choice, [0, 1, 12, 42, 77]} % second argument is randomly choosen from the list
]
}
)</pre>
This first steps was to become more familiar with Erlang language. My next steps will be:
<ul>
<li> use edoc ...</li>
<li> use logger/trace services instead of printing every function call</li>
<li> use generic services (gen_event, gen_server)</li>
<li> create an OTP application (application, supervisor)</li>
<li> use Mnesia to gather statistics</li>
<li> what about OTP release</li>
<li> embed everything in distribution package (automake/autoconf ?)</li>
</ul>