Yet Another Dev BlogCodeception2018-06-09T00:00:00+5:30Sreejith Kesavansreejithemk@gmail.comDemystifying Python's del, __del__ and garbage collection2015-07-09T00:00:00+5:30http://foobarnbaz.com/2015/07/09/understanding-python-del-and-garbage-collection<p>People are often confused about Python's <code>del</code> keyword and <code>__del__()</code> method.
Most people think that applying <code>del</code> on an object causes the object's
<code>__del__()</code> method to be called. But this is not true. <code>del</code> keyword merely
decrements the objects reference count and de-scopes the variable on its
application. But it doesn't necessarily invoke the object's <code>__del__()</code> method.
<code>__del__</code> method will be called only when the garbage collection kicks in.
And this would happens automatically when the reference count of an object
becomes zero.</p>
<p><code>del</code>: Decrements reference count<br />
<code>__del__()</code>: Object destructor. Called when an object is garbage collected</p>
<div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">Foo</span><span class="p">():</span>
<span class="k">def</span> <span class="nf">__del__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">print</span> <span class="s1">'__del__() called'</span>
</pre></div>
<p>Now let's create some instances of this <code>Foo</code> class and call <code>del</code> upon these
instances to see what happens.</p>
<div class="codehilite"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">1</span><span class="p">]:</span> <span class="o">%</span><span class="k">paste</span>
<span class="k">class</span> <span class="nc">Foo</span><span class="p">():</span>
<span class="k">def</span> <span class="nf">__del__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">print</span> <span class="s1">'__del__() called'</span>
<span class="c1">## -- End pasted text --</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">2</span><span class="p">]:</span> <span class="kn">from</span> <span class="nn">sys</span> <span class="kn">import</span> <span class="n">getrefcount</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">3</span><span class="p">]:</span> <span class="n">getrefcount</span><span class="p">(</span><span class="n">Foo</span><span class="p">)</span>
<span class="n">Out</span><span class="p">[</span><span class="mi">3</span><span class="p">]:</span> <span class="mi">2</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">4</span><span class="p">]:</span> <span class="n">a</span> <span class="o">=</span> <span class="n">Foo</span><span class="p">()</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">5</span><span class="p">]:</span> <span class="n">getrefcount</span><span class="p">(</span><span class="n">a</span><span class="p">)</span>
<span class="n">Out</span><span class="p">[</span><span class="mi">5</span><span class="p">]:</span> <span class="mi">2</span>
</pre></div>
<p>At this point you might be wondering why <code>getrefcount(Foo)</code> and <code>getrefcount(a)</code>
shows 2 instead of 1. Well, the reason is when you call the method <code>getrefcount</code>
the argument is passed by value, effectively bumping the reference count by 1.
This extra reference will be freed up once the <code>getrefcount</code> method exits. So
effectively we have a reference count of 1 for the instance <code>a</code> here. Now let's
make more references to the same instance.</p>
<div class="codehilite"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">6</span><span class="p">]:</span> <span class="n">b</span> <span class="o">=</span> <span class="n">a</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">7</span><span class="p">]:</span> <span class="n">getrefcount</span><span class="p">(</span><span class="n">a</span><span class="p">)</span>
<span class="n">Out</span><span class="p">[</span><span class="mi">7</span><span class="p">]:</span> <span class="mi">3</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span class="n">getrefcount</span><span class="p">(</span><span class="n">b</span><span class="p">)</span>
<span class="n">Out</span><span class="p">[</span><span class="mi">8</span><span class="p">]:</span> <span class="mi">3</span>
</pre></div>
<p>As expected, the reference count is incremented. Now let's call <code>del</code> on a to
see if the object's <code>__del__()</code> is invoked.</p>
<div class="codehilite"><pre><span></span><span class="n">In</span> <span class="p">[</span><span class="mi">9</span><span class="p">]:</span> <span class="k">del</span> <span class="n">a</span>
<span class="n">In</span> <span class="p">[</span><span class="mi">10</span><span class="p">]:</span> <span class="k">del</span> <span class="n">b</span>
<span class="n">__del__</span><span class="p">()</span> <span class="n">called</span>
</pre></div>
<p>As you can see, <code>__del__()</code> is not invoked on applying <code>del</code> the first reference.
This is because the reference count is still 1. On calling <code>del</code> on <code>b</code> causes
the reference count become zero, garbage collection kicks in and finally
our long awaited <code>__del__()</code> is called.</p>
<p>But there are situation where the destructor goes astray. These situations and
the alternative approaches are explained in detail in the following brilliant
blog post.</p>
<p><a href="http://eli.thegreenplace.net/2009/06/12/safely-using-destructors-in-python">Safely using destructors in Python</a></p>UNIX Series: System Calls - Part I2014-01-27T00:00:00+5:30http://foobarnbaz.com/2014/01/27/unix-series-system-calls-part-1<p>System programmers must know about <em>system calls</em> and for for those who are not system programmers,
it is still good getting to know about how things work under the hood. My idea here is to do a series
of blog posts on UNIX/Linux system internals for the upcoming weeks. I'm doing this as a part of my
learnings from <a href="https://www.codeschool.com">CodeSchool</a> and reading
<em><a href="http://shop.oreilly.com/product/9780596009588.do">Linux System Programming</a></em>. Let's get started
with UNIX <em>System Calls</em> in the first place.</p>
<h5>What are System Calls?</h5>
<p>From <em>Linux System Programming</em> book,</p>
<blockquote>
<p>System programming starts and ends with system calls. System calls (often shortened to syscalls)
are function invocations made from user space—your text editor, favorite game, and so on—into the
kernel (the core internals of the system) in order to request some service or resource from the
operating system. System calls range from the familiar, such as read() and write(), to the exotic,
such as get_thread_area() and set_tid_address().</p>
</blockquote>
<p>So system calls acts as an intermediary between userspace applications and the kernel space. It
allows programs to request the operating system to do something on its behalf. The system calls
are functions used in the kernel itself. If there was no clear distinction between the kernel
space and user space, then all the user applications will be able to do absolutely anything
they want to do in the operating system. Now that would be insecure and chaotic. System calls
exposes only those functionalities of the kernel through an API which will get only that task
done for the user application. In short, its a secure way of getting things done in the operating
system.</p>
<h5>Some of the popular System Calls</h5>
<p>All of us are atleast familiar with the <code>read()</code> and <code>write()</code> system calls. But applications need
even more of these. the following table shows the most commonly used system calls.</p>
<table>
<colgroup>
<col style="text-align:left;"/>
<col style="text-align:center;"/>
<col style="text-align:right;"/>
</colgroup>
<thead>
<tr>
<th style="text-align:left;">General Class</th>
<th style="text-align:center;">Specific Class</th>
<th style="text-align:right;">System Call</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:left;">File related</td>
<td style="text-align:center;">Creating a Channel</td>
<td style="text-align:right;">creat()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">open()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">open()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;">Input/Output</td>
<td style="text-align:right;">read()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">write()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;">Random Access</td>
<td style="text-align:right;">lseek()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;">Channel duplication</td>
<td style="text-align:right;">dup()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;">Aliassing Files</td>
<td style="text-align:right;">link()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;">Removing Files</td>
<td style="text-align:right;">unlink()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;">File Status</td>
<td style="text-align:right;">stat()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">fstat()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;">Access Control</td>
<td style="text-align:right;">access()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">chmod()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">chown()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">umask()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;">Device Control</td>
<td style="text-align:right;">ioctl()</td>
</tr>
<tr>
<td style="text-align:left;">Process related</td>
<td style="text-align:center;">Process creation</td>
<td style="text-align:right;">exec()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;">Process creation</td>
<td style="text-align:right;">fork()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;">Process wait</td>
<td style="text-align:right;">wait()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;">Process termination</td>
<td style="text-align:right;">exit()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;">Process Owner & Group</td>
<td style="text-align:right;">getuid()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">geteuid()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">getgid()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">getegid()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;">Process identity</td>
<td style="text-align:right;">getpid()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">getppid()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;">Process control</td>
<td style="text-align:right;">signal()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">kill()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">alarm()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;">Change working directory</td>
<td style="text-align:right;">chdir()</td>
</tr>
<tr>
<td style="text-align:left;">IPC</td>
<td style="text-align:center;">Pipelines</td>
<td style="text-align:right;">pipe()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;">Messages</td>
<td style="text-align:right;">msgget()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">msgsnd()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">msgrcv()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">msgctl()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;">Semaphores</td>
<td style="text-align:right;">semget()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">semop()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;">Shared memory</td>
<td style="text-align:right;">shmget()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">shmat()</td>
</tr>
<tr>
<td style="text-align:left;"></td>
<td style="text-align:center;"></td>
<td style="text-align:right;">shmdt()</td>
</tr>
</tbody>
</table>
<h5>How System Calls are being called?</h5>
<h5>UNIX Process Address Space</h5>
<h5>References</h5>
<p><a href="http://www.di.uevora.pt/~lmr/syscalls.html">syscalls</a></p>IDEs vs. Text Editors: What should developers use?2014-01-24T00:00:00+5:30http://foobarnbaz.com/2014/01/24/text-editors-vs-ides-which-one-to-use<p>The answer to this question may vary from developers to developers. I truly believe that it also depends on the kind of programming language they use for developing applications. For example, I have heard many times from my fellow Java programmers that they couldn't live without the <a href="http://www.eclipse.org/">Eclipse</a> IDE, that they will not be able to write a working Java program with the help of a simple text editor. Predominantly being a Python developer, I use <a href="http://www.gnu.org/software/emacs/">Emacs</a> for most of my coding. I do not feel code completion or any other fancy IDE features are required for coding in Python, atleast for what I'm doing. The only difficult thing I encountered using just Emacs for coding is <em>debugging</em> the code I've written. For most of the time <code>print</code> statements will do the job fine. But at times I have to manually initialize <code>pdb</code> to debug my code. This could be painful sometimes. Although there are some tricks in Emacs to do the debugging, it didn't seem neat enough to me.</p>
<blockquote>
<p>Then when I started writing C and using GDB inside of Emacs I kind of tried to
keep up the same way of doing things. That was the model we built Energize
around. And that just never seemed like it worked well. And as time went by I
gradually stopped even trying to use those tools like that and just stick in
print statements and run the thing again. Repeat until done. Especially when
you get to more and more primitive environments like JavaScript and Perl and
stuff like that, it's the only choice you have -- There aren't any debuggers.
<cite>Jamie Zawinski, "Coders At Work: Reflections on the Craft of Programming"
</cite></p>
</blockquote>
<p>That's when I found <a href="http://www.jetbrains.com/pycharm/">PyCharm</a> IDE for Python. They had this program where they provide free licenses for the commercial version of PyCharm for doing opensource projects. I applied for one mentioning my <a href="https://github.com/semk/voldemort">voldemort</a> project and I got one free license.</p>
<p>PyCharm comes with a brilliant visual Python debugger and I found it really easier to debug the code using breakpoints and variable watchers. I spent a month or two coding on PyCharm and it really helped me code faster. One of the main reasons I love Emacs is that I could do all the tasks just by using the keyboard. Be it cutting & pasting text, switching between buffers it was all easy. There was also this good feeling of using just the keyboard. After using PyCharm for quite some time, I felt like using the mouse a lot for most of these taks. Another advantage of using a text editor is that you will get to know the programming language a lot better since there are no autocompletion or any other IDE features. You have to write the code on your own. Then there are things like version control integration for IDEs which I believe eases the effort for most of the people. But I belive they conceal a lot of activities from the user. I prefer doing the version control activities by myself on a terminal rather than handing it to an IDE. In this pypix <a href="http://pypix.com/roundups/tools-python-super-stars/">article</a> some of the world's top Python developers talk about the tools preferred by them.</p>
<p>So what developer's should ultimately use? Hmm, I believe, there is no concrete answer. Nowadays, I use Emacs for relatively smaller projects and PyCharm for the behemoths. What do you think? What are you using for development?. Give your opinions/suggestions in the comment section.</p>RegexMate - A visual Regular Expression testing toolkit2013-08-03T00:00:00+5:30http://foobarnbaz.com/2013/08/03/regexmate-the-visual-regular-expression-tester<p>I have been doing a lot of regular expression parsing at work. Writing regular expressions are hard, and testing it is even harder. <a href="http://regexpal.com/">Regexpal</a> and <a href="http://www.debuggex.com/">Debuggex</a> are some of the popular web based Regular expression testing tools available. Inspired by the idea behind these tools, I wrote a simple cross-platform regular expression tester, <a href="https://github.com/semk/RegexMate">RegexMate</a> in PyQt. Initial version of RegexMate can highlight matches, groups and supports some regex flags as well.</p>
<p><img alt="RegexMate" src="https://raw.github.com/semk/RegexMate/master/screenshots/RegexMate.png" /></p>
<p>I'm planning to add syntax highlighting for the regex pattern and additional options like replace and split in the upcoming releases. Stay tuned.</p>Org2OPML - Make awesome mindmaps using Emacs Org mode and Mindnode2013-08-02T00:00:00+5:30http://foobarnbaz.com/2013/08/02/awesome-mindmaps-using-org<p>Whoever using <a href="http://www.gnu.org/software/emacs/">Emacs</a> as their primary text-editor would have liked the <a href="http://orgmode.org/">Org mode</a>. As the Org website puts it:-</p>
<blockquote>
<p>Org mode is for keeping notes, maintaining TODO lists, planning projects, and authoring documents with a fast and effective plain-text system.</p>
</blockquote>
<p>Org mode allows you to create documents with a simple syntax. Different headers/sections in the document can be represented by a <code>*</code> symbol. A single <code>*</code> corresponds to an <code><h1></code> block in HTML and <code>**</code> represents a <code><h2></code>.</p>
<p>Org mode has the capability to export documents to various formats like ASCII text, PDF, LaTeX, XOXO, HTML, Freemind Mindmap etc. I was not quite happy with the output produced by <a href="http://freemind.sourceforge.net/wiki/index.php/Main_Page">Freemind</a> from the <code>.mm</code> files exported by Org mode. So I switched to <a href="http://mindnode.com/">Mindnode</a>, another mindmapping software which is available for Mac OSX and produces visually pleasing mindmaps. But the <code>.mm</code> exports from Org mode doesn't work pretty well with Mindnode. So I wrote a simple <a href="https://github.com/semk/Org2OPML">Org2OPML</a> converter that converts <code>.org</code> files to <code>.opml</code> files. OPML is another formats for mindmps/outlining and well supported by Mindnode.</p>
<p>The <code>.org</code> representation of the mindmap</p>
<div class="codehilite"><pre><span></span>#+TITLE: Windows Versions
#+AUTHOR: Sreejith Kesavan
#+ROOT: Windows Versions
* Windows 8 Home
** Windows 8
** Windows 8
** Windows 8 Enterprise
** Windows RT
* Windows 7
** Windows 7 Starter
** Windows 7 Home Basic
** Windows 7 Home Premium
** Windows 7 Professional
** Windows 7 Enterprise
** Windows 7 Ultimate
** Windows 7 Service Pack 1
* Windows Vista
** Windows Vista Starter
** Windows Vista Home Basic
** Windows Vista Home Premium
** Windows Vista Business
** Windows Vista Enterprise
** Windows Vista Ultimate
* Windows XP
** Windows XP Starter Edition
** Windows XP Home Edition
** Windows XP Edition N
** Windows XP Professional,
** Windows XP Edition N
** Windows XP Professional x64 Edition
** Windows XP for specialized hardware
** Windows XP Media Center Edition
*** Windows XP Media Center Edition
*** Windows XP Media Center Edition 2003
*** Windows XP Media Center Edition 2004
*** Windows XP Media Center Edition 2005
** Windows XP 64-bit Edition
** Windows XP Tablet PC Edition
** Windows Fundamentals for Legacy PCs
</pre></div>
<p>Have a look at the mindmap generated by Mindnode once the <code>.org</code> file is converted to OPML using Org2OPML script.</p>
<p><img alt="Windows Versions" src="/images/posts/awesome-mindmaps-using-org/mindmap.png" /></p>Simple file upload progressbar in PyQt2012-12-31T00:00:00+5:30http://foobarnbaz.com/2012/12/31/file-upload-progressbar-in-pyqt<p>I've been learning the <a href="http://qt-project.org/">Qt</a> Framework for the last few months and righfully it took
the honour of being my favourite GUI toolkit ;-). But still I don't like KDE. I have used
<a href="http://wiki.python.org/moin/TkInter">Tkinter</a> for quite some time when I was studying engineering. But Qt is
much more powerful than Tkinter and comes with an enormous bundle of GUI widgets and elements. Qt documentation
is one of the best documentations available for a GUI framework. I must admit that I started liking Qt even more
as soon as I started reading the documentation and trying out sample programs.</p>
<p>I encountered a challenge of displaying a progress dialog for files being uploaded to a central server. In this
case the files were uploaded using sending a POST request to the server using the <code>urllib2</code> library. The data was
multipart encoded. The problem here was, when you send the data to the server, how would you show the incremental
progress of how much data got uploaded in the progress dialog view? The <code>urllib2</code> library doesn't provide any special
methods to help in this situation. Neither does <code>Qt</code>. But we know the upper and lower limits of the progress bar: 0 to
size of the file. But how would you tell Qt that how much data is uploaded at a given time?</p>
<p>Well, I found a tricky way to let Qt know about the file upload status using the <code>StringIO</code> module. The solution is to give
<code>urllib2</code> a custom <code>StringIO</code> instance that will periodically alert <code>QProgressDialog</code> about the status of the upload.
Checkout my approach to this problem.</p>
<div class="codehilite"><pre><span></span><span class="k">class</span> <span class="nc">CancelledError</span><span class="p">(</span><span class="ne">Exception</span><span class="p">):</span>
<span class="sd">"""Error denoting user interruption.</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">msg</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">msg</span> <span class="o">=</span> <span class="n">msg</span>
<span class="ne">Exception</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">msg</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">msg</span>
<span class="n">__repr__</span> <span class="o">=</span> <span class="n">__str__</span>
<span class="k">class</span> <span class="nc">BufferReader</span><span class="p">(</span><span class="n">StringIO</span><span class="p">):</span>
<span class="sd">"""StringIO with a callback.</span>
<span class="sd"> """</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">buf</span><span class="o">=</span><span class="s1">''</span><span class="p">,</span>
<span class="n">callback</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span>
<span class="n">cb_args</span><span class="o">=</span><span class="p">(),</span>
<span class="n">cb_kwargs</span><span class="o">=</span><span class="p">{}):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_callback</span> <span class="o">=</span> <span class="n">callback</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_cb_args</span> <span class="o">=</span> <span class="n">cb_args</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_cb_kwargs</span> <span class="o">=</span> <span class="n">cb_kwargs</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_progress</span> <span class="o">=</span> <span class="mi">0</span>
<span class="n">StringIO</span><span class="o">.</span><span class="n">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">buf</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">read</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">n</span><span class="o">=-</span><span class="mi">1</span><span class="p">):</span>
<span class="sd">"""Read the chunk. Alert the callback.</span>
<span class="sd"> """</span>
<span class="n">chunk</span> <span class="o">=</span> <span class="n">StringIO</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">n</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_progress</span> <span class="o">+=</span> <span class="nb">int</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">chunk</span><span class="p">))</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_cb_kwargs</span><span class="o">.</span><span class="n">update</span><span class="p">({</span><span class="s1">'progress'</span><span class="p">:</span> <span class="bp">self</span><span class="o">.</span><span class="n">_progress</span><span class="p">})</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">_callback</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_callback</span><span class="p">(</span><span class="o">*</span><span class="bp">self</span><span class="o">.</span><span class="n">_cb_args</span><span class="p">,</span> <span class="o">**</span><span class="bp">self</span><span class="o">.</span><span class="n">_cb_kwargs</span><span class="p">)</span>
<span class="k">except</span><span class="p">:</span> <span class="c1"># catches exception from the callback</span>
<span class="k">raise</span> <span class="n">CancelledError</span><span class="p">(</span><span class="s1">'The upload was cancelled.'</span><span class="p">)</span>
<span class="k">return</span> <span class="n">chunk</span>
</pre></div>
<p>We can use the <code>BufferReader</code> instance as the input to the <code>urlllib2.Request</code>. It takes the actual data to be
sent as the input and registers a callback function which will be periodically called whenever a read occurs from
the <code>urllib2</code> module. Let's create a progress dialog box in Qt which shows the progress.</p>
<div class="codehilite"><pre><span></span><span class="c1"># create the progress dialog</span>
<span class="n">progressDialog</span> <span class="o">=</span> <span class="n">QProgressDialog</span><span class="p">(</span><span class="s1">'Uploading </span><span class="si">%s</span><span class="s1"> ...'</span> <span class="o">%</span> <span class="n">file_path</span><span class="p">,</span>
<span class="n">QString</span><span class="p">(</span><span class="s2">"Cancel"</span><span class="p">),</span> <span class="mi">0</span><span class="p">,</span> <span class="n">file_size</span><span class="p">)</span>
<span class="n">progressDialog</span><span class="o">.</span><span class="n">setWindowTitle</span><span class="p">(</span><span class="s1">'Upload status'</span><span class="p">)</span>
<span class="c1"># create a file stream that supports callback</span>
<span class="n">databuf</span> <span class="o">=</span> <span class="n">BufferReader</span><span class="p">(</span><span class="n">buf</span><span class="o">=</span><span class="n">body</span><span class="p">,</span> <span class="n">callback</span><span class="o">=</span><span class="n">update_progress</span><span class="p">,</span>
<span class="n">cb_args</span><span class="o">=</span><span class="p">(</span><span class="n">progressDialog</span><span class="p">,</span> <span class="n">file_size</span><span class="p">))</span>
<span class="c1"># upload the file using the databuf</span>
<span class="n">req</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">Request</span><span class="p">(</span><span class="n">URL</span><span class="p">,</span> <span class="n">databuf</span><span class="p">,</span> <span class="n">headers</span><span class="p">)</span>
<span class="n">urlobj</span> <span class="o">=</span> <span class="n">urllib2</span><span class="o">.</span><span class="n">urlopen</span><span class="p">(</span><span class="n">req</span><span class="p">)</span>
<span class="n">result</span> <span class="o">=</span> <span class="n">urlobj</span><span class="o">.</span><span class="n">read</span><span class="p">()</span>
</pre></div>
<p>And we can use the callback function <code>update_progress</code> to update the progressbar with additional <code>progress</code> keyword argument
which reports the bytes uploaded/read by the <code>urllib2</code> library.</p>
<div class="codehilite"><pre><span></span><span class="k">def</span> <span class="nf">update_progress</span><span class="p">(</span><span class="n">progressbar</span><span class="p">,</span> <span class="n">size</span><span class="p">,</span> <span class="n">progress</span><span class="o">=</span><span class="bp">None</span><span class="p">):</span>
<span class="sd">"""Callback method for reporting upload status.</span>
<span class="sd"> """</span>
<span class="k">if</span> <span class="n">progressbar</span><span class="o">.</span><span class="n">wasCanceled</span><span class="p">():</span>
<span class="k">raise</span> <span class="c1"># notify uploader and stop file upload</span>
<span class="n">progressbar</span><span class="o">.</span><span class="n">setLabelText</span><span class="p">(</span><span class="s1">'Uploading </span><span class="si">%d</span><span class="s1"> KB of </span><span class="si">%d</span><span class="s1"> KB'</span>
<span class="o">%</span> <span class="p">(</span><span class="n">progress</span><span class="o">/</span><span class="mi">1024</span><span class="p">,</span> <span class="n">size</span><span class="o">/</span><span class="mi">1024</span><span class="p">))</span>
<span class="n">progressbar</span><span class="o">.</span><span class="n">setValue</span><span class="p">(</span><span class="n">progress</span><span class="p">)</span>
<span class="n">QApplication</span><span class="o">.</span><span class="n">processEvents</span><span class="p">()</span>
</pre></div>
<p>And this happens to be my last blog post of 2012. Happy to have survived the apocalypse, no zombies, earthquakes or Tsunamis.
Wishing you all a prosperous and happy new year. Welcome 2K13.</p>Understanding Python variables and Memory Management2012-07-08T00:00:00+5:30http://foobarnbaz.com/2012/07/08/understanding-python-variables<p>Have you ever noticed any difference between variables in Python and C? For example, when you do an assignment like the following in C, it actually creates a block of memory space so that it can hold the value for that variable.</p>
<div class="codehilite"><pre><span></span><span class="kt">int</span> <span class="n">a</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
</pre></div>
<p>You can think of it as putting the value assigned in a box with the variable name as shown below.</p>
<p><img alt="int a =1;" src="/images/posts/2012-07-08-understanding-python-variables/a1box.png" /></p>
<p>And for all the variables you create a new box is created with the variable name to hold the value. If you change the value of the variable the box will be updated with the new value. That means doing</p>
<div class="codehilite"><pre><span></span><span class="n">a</span> <span class="o">=</span> <span class="mi">2</span><span class="p">;</span>
</pre></div>
<p>will result in</p>
<p><img alt="a = 2;" src="/images/posts/2012-07-08-understanding-python-variables/a2box.png" /></p>
<p>Assigning one variable to another makes a copy of the value and put that value in the new box.</p>
<div class="codehilite"><pre><span></span><span class="kt">int</span> <span class="n">b</span> <span class="o">=</span> <span class="n">a</span><span class="p">;</span>
</pre></div>
<p><img alt="b=2" src="/images/posts/2012-07-08-understanding-python-variables/b2box_a2box.png" /></p>
<p>But in Python variables work more like tags unlike the boxes you have seen before. When you do an assignment in Python, it tags the value with the variable name.</p>
<div class="codehilite"><pre><span></span><span class="n">a</span> <span class="o">=</span> <span class="mi">1</span>
</pre></div>
<p><img alt="a = 1" src="/images/posts/2012-07-08-understanding-python-variables/a1tag.png" /></p>
<p>and if you change the value of the varaible, it just changes the tag to the new value in memory. You dont need to do the housekeeping job of freeing the memory here. Python's Automatic Garbage Collection does it for you. When a value is without names/tags it is automatically removed from memory.</p>
<div class="codehilite"><pre><span></span><span class="n">a</span> <span class="o">=</span> <span class="mi">2</span>
</pre></div>
<p><img alt="a = 2" src="/images/posts/2012-07-08-understanding-python-variables/a2tag_1.png" /></p>
<p>Assigning one variable to another makes a new tag bound to the same value as show below.</p>
<div class="codehilite"><pre><span></span><span class="n">b</span> <span class="o">=</span> <span class="n">a</span>
</pre></div>
<p><img alt="b = a" src="/images/posts/2012-07-08-understanding-python-variables/ab2tag.png" /></p>
<p>Other languages have 'variables'. Python has 'names'.</p>
<h3>A bit about Python's memory management</h3>
<p>As you have seen before, a value will have only one copy in memory and all the variables having this value will refer to this memory location. For example when you have variables <code>a</code>, <code>b</code>, <code>c</code> having a value 10, it doesn't mean that there will be 3 copy of <code>10</code>s in memory. There will be only one <code>10</code> and all the variables <code>a</code>, <code>b</code>, <code>c</code> will point to this value. Once a variable is updated, say you are doing <code>a += 1</code> a new value <code>11</code> will be allocated in memory and <code>a</code> will be pointing to this.</p>
<p>Let's check this behaviour with Python Interpreter. Start the Python Shell and try the following for yourselves.</p>
<div class="codehilite"><pre><span></span><span class="o">>>></span> <span class="n">a</span> <span class="o">=</span> <span class="mi">10</span>
<span class="o">>>></span> <span class="n">b</span> <span class="o">=</span> <span class="mi">10</span>
<span class="o">>>></span> <span class="n">c</span> <span class="o">=</span> <span class="mi">10</span>
<span class="o">>>></span> <span class="nb">id</span><span class="p">(</span><span class="n">a</span><span class="p">),</span> <span class="nb">id</span><span class="p">(</span><span class="n">b</span><span class="p">),</span> <span class="nb">id</span><span class="p">(</span><span class="n">c</span><span class="p">)</span>
<span class="p">(</span><span class="mi">140621897573616</span><span class="p">,</span> <span class="mi">140621897573616</span><span class="p">,</span> <span class="mi">140621897573616</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">a</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="o">>>></span> <span class="nb">id</span><span class="p">(</span><span class="n">a</span><span class="p">)</span>
<span class="mi">140621897573592</span>
</pre></div>
<p><code>id()</code> will return an objects memory address (object's identity). As you have noticed, when you assign the same integer value to the variables, we see the same ids. But this assumption does not hold true all the time. See the following for example</p>
<div class="codehilite"><pre><span></span><span class="o">>>></span> <span class="n">x</span> <span class="o">=</span> <span class="mi">500</span>
<span class="o">>>></span> <span class="n">y</span> <span class="o">=</span> <span class="mi">500</span>
<span class="o">>>></span> <span class="nb">id</span><span class="p">(</span><span class="n">x</span><span class="p">)</span>
<span class="mi">4338740848</span>
<span class="o">>>></span> <span class="nb">id</span><span class="p">(</span><span class="n">y</span><span class="p">)</span>
<span class="mi">4338741040</span>
</pre></div>
<p>What happened here? Even after assigning the same integer values to different variable names, we are getting two different ids here. These are actually the effects of CPython optimization we are observing here. CPython implementation keeps an array of integer objects for all integers between -5 and 256. So when we create an integer in that range, they simply back reference to the existing object. You may refer the following links for more information.</p>
<p><a href="http://stackoverflow.com/questions/306313/is-operator-behaves-unexpectedly-with-integers">Stack Overflow: "is" operator behaves unexpectedly with integers</a></p>
<p>Let's take a look at strings now.</p>
<div class="codehilite"><pre><span></span><span class="o">>>></span> <span class="n">s1</span> <span class="o">=</span> <span class="s1">'hello'</span>
<span class="o">>>></span> <span class="n">s2</span> <span class="o">=</span> <span class="s1">'hello'</span>
<span class="o">>>></span> <span class="nb">id</span><span class="p">(</span><span class="n">s1</span><span class="p">),</span> <span class="nb">id</span><span class="p">(</span><span class="n">s2</span><span class="p">)</span>
<span class="p">(</span><span class="mi">4454725888</span><span class="p">,</span> <span class="mi">4454725888</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">s1</span> <span class="o">==</span> <span class="n">s2</span>
<span class="bp">True</span>
<span class="o">>>></span> <span class="n">s1</span> <span class="ow">is</span> <span class="n">s2</span>
<span class="bp">True</span>
<span class="o">>>></span> <span class="n">s3</span> <span class="o">=</span> <span class="s1">'hello, world!'</span>
<span class="o">>>></span> <span class="n">s4</span> <span class="o">=</span> <span class="s1">'hello, world!'</span>
<span class="o">>>></span> <span class="nb">id</span><span class="p">(</span><span class="n">s3</span><span class="p">),</span> <span class="nb">id</span><span class="p">(</span><span class="n">s4</span><span class="p">)</span>
<span class="p">(</span><span class="mi">4454721608</span><span class="p">,</span> <span class="mi">4454721664</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">s3</span> <span class="o">==</span> <span class="n">s4</span>
<span class="bp">True</span>
<span class="o">>>></span> <span class="n">s3</span> <span class="ow">is</span> <span class="n">s4</span>
<span class="bp">False</span>
</pre></div>
<p>Looks interesting, isn't it? When the string was a simple and shorter one the variable names where referring to the same object in memory. But when they became bigger, this was not the case. This is called interning, and Python does interning (to some extent) of shorter string literals (as in <code>s1</code> and <code>s2</code>) which are created at compile time. But in general, Python string literals creates a new string object each time (as in <code>s3</code> and <code>s4</code>). Interning is runtime dependant and is always a trade-off between memory use and the cost of checking if you are creating the same string. There's a built-in <code>intern()</code> function to forcefully apply interning. Read more about interning from the following links.</p>
<p><a href="http://stackoverflow.com/questions/17679861/does-python-intern-strings">Stack Overflow: Does Python intern Strings?</a><br/>
<a href="http://stackoverflow.com/questions/15541404/python-string-interning">Stack Overflow: Python String Interning</a><br/>
<a href="http://guilload.com/python-string-interning/">Internals of Python String Interning</a><br/></p>
<p>Now we will try to create custom objects and try to find their identities.</p>
<div class="codehilite"><pre><span></span><span class="o">>>></span> <span class="k">class</span> <span class="nc">Foo</span><span class="p">:</span>
<span class="o">...</span> <span class="k">pass</span>
<span class="o">...</span>
<span class="o">>>></span> <span class="n">bar</span> <span class="o">=</span> <span class="n">Foo</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">baz</span> <span class="o">=</span> <span class="n">Foo</span><span class="p">()</span>
<span class="o">>>></span> <span class="nb">id</span><span class="p">(</span><span class="n">bar</span><span class="p">)</span>
<span class="mi">140730612513248</span>
<span class="o">>>></span> <span class="nb">id</span><span class="p">(</span><span class="n">baz</span><span class="p">)</span>
<span class="mi">140730612513320</span>
</pre></div>
<p>As you can see, the two instances have different identities. That means, there are two different copies of the same object in memory. When you are creating custom objects, they will have unique identities unless you are using <a href="http://foobarnbaz.com/2010/10/06/borg-pattern/">Singleton Pattern</a> which overrides this behaviour (in <code>__new__()</code>) by giving out the same instance upon instance creation.</p>
<p>Thanks to Jay Pablo (see comments) for correcting the mistakes and making this post a better one.</p>The era of online payments2012-05-31T00:00:00+5:30http://foobarnbaz.com/2012/05/31/the-era-of-online-payments<p>We live in a century defined and driven by technology. Internet has changed the way approach the needs in our life. Very rarely people are going to an Electronic/Computer store to buy stuffs when there is an option to order it online. Now, all of the consumer centric business have an online version too.</p>
<p>I was using <a href="http://rechargeitnow.com">RechargeItNow</a> for recharging my cell phone and DTH services. But their website was too cluttered and filled with advertisements all over the place. Then I found <a href="https://justrechargeit.com">JustRechargeIt</a>. Their web application was way too better than the RechargeItNow website. They even provided options to save lists of phone numbers and DTH accounts to ease the payment. I was happily using this service for a long time until I found <a href="http://paytm.com">Paytm</a>. Paytm web application is so simle and intuitive. Just have a look.</p>
<p><img alt="Paytm" src="/images/posts/the-era-of-online-payments/paytm.png" /></p>
<p>The simplicity and way of presentation is an important aspect when it comes to the web business. And Paytm has succeeded in doing so. I really want to see more and more web applications doing the same like what Paytm did for user experience, hopefully <a href="http://ebay.in">eBay</a> :-)</p>Pagination, Tags and more in Voldemort2012-04-06T00:00:00+5:30http://foobarnbaz.com/2012/04/06/pagination-tags-and-more-in-voldemort<p><img alt="Voldemort" src="/images/posts/2012-04-06-pagination-tags-and-more-in-voldemort/term.png" /></p>
<p>I have been adding a few useul features to <a href="https://github.com/semk/voldemort">Voldemort</a> for the last few months and it did come out pretty good. It now sports automatic Atom Feed and sitemap generation, thanks to the contibutions from my friend <a href="http://endlessinsomnia.com">Stanislav Yudin</a>. Here are the highlights.</p>
<ul>
<li>Generates Atom Feed automatically for your blog</li>
<li>Generates sitemap.xml</li>
<li>Pagination supported. Just add <code>paginate: True</code> in the page metadata section.</li>
<li>You can tag posts using metadata. Eg: <code>tags: ['Python', 'Tips']</code>. A special template file named <code>tag.html</code> is used to generate tags on the fly.</li>
</ul>
<p>You can see these changes live in this blog. Try clicking any of the tags below to see it yourself.</p>New set of Developer Tools in Firefox2012-04-06T00:00:00+5:30http://foobarnbaz.com/2012/04/06/firefox-developer-tools<p>My friends at work were using <a href="http://chrome.google.com">Google Chrome</a> for Debugging web pages saying that Chrome's Developer Tools are superior to Firefox. But starting with the new <a href="http://hacks.mozilla.org/2011/11/developer-tools-in-firefox-aurora-10/">Aurora update</a> Firefox Developer Tools looks way cooler with the new Page Inspector, Style Editor and Scratchpad.</p>
<p>The all new <code>Right Click -> Inspect Element</code> feature lets you select any HTML element and see the HTML code or Styles associated with it in clean Developer-Friendly manner.</p>
<p><img alt="Inspect Element" src="/images/posts/2012-04-06-firefox-developer-tools/inspect.png" /></p>
<p>Breadcrumbs of the HTML tags are also provided by the Page Inspector.</p>
<p><img alt="Breadcrumbs" src="/images/posts/2012-04-06-firefox-developer-tools/breadcrumbs.png" /></p>
<p>You can also switch to a 3D view of the HTML Document to see how elements are organized.</p>
<p><img alt="Document View" src="/images/posts/2012-04-06-firefox-developer-tools/view.png" /></p>
<p>Using the Style Editor, you can edit the CSS dynamically and see the effect instantaneously without reloading your web service.</p>
<p><img alt="Style Editor" src="/images/posts/2012-04-06-firefox-developer-tools/style_editor.png" /></p>
<p>And finally for testing Javascript code snippets, there is Scratchpad which also sports code highlighting.</p>
<p><img alt="Scratchpad" src="/images/posts/2012-04-06-firefox-developer-tools/scratchpad.png" /></p>
<p>I find it quite useful and intuitive. How about you?</p>Multiple Index queries in Riak using Python2012-02-25T00:00:00+5:30http://foobarnbaz.com/2012/02/25/multi-index-queries-in-riak<p><img alt="Riak Logo" src="http://upload.wikimedia.org/wikipedia/en/5/53/Riak_product_logo.png" /></p>
<p><a href="http://wiki.basho.com/Riak.html">Riak</a> is a <a href="http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html">Amazon Dynamo</a> inspired masterless Key-Value store written in Erlang. It is one of those <a href="http://en.wikipedia.org/wiki/NoSQL">NoSQL</a> databases that is rock stable, production ready and promises zero downtime. I have been using Riak at work and was literally blown away by its simplicity (Setting up a three node cluster wouldn't even take ten minutes) and the kind of support the Riak Community provides. And it is amazingly fast too. The Data retrieval operations in Riak are basically using methods like Get (by key), MapReduce and Key Filters.</p>
<p>Riak also supports multiple backends for Key-value store. And Google's own <a href="code.google.com/p/leveldb">LevelDB</a> is one of them. One of the advantage of using LevelDB with Riak is that they support Secondary Indexes. This is a way to retrieve data faster when you want to use an SQL like Query interface. But the problem is that Riak only supports single index queries. That means, you will be able to query only one field at a time.</p>
<p>I wrote a Python wrapper that allows multiple index queries using Secondary indexes and MapReduce. The basic steps are</p>
<ol>
<li>Query Multiple Indexes and get the associated keys</li>
<li>Pass the keys to a MapReduce job where Multiple filters are again evaluated. The map phase applies all the conditions to individual keys.</li>
</ol>
<blockquote>
<p><em>As suggested by Elias Levy, it is ideal to compute the intersection of all the index queries rather than evaluating filters on Map Phase as this involves parsing the data and validating filters. This could become very slow when the number of keys returned by a single index query is larger compared to other indexes. The sources are updated to reflect this change.</em></p>
</blockquote>
<p>So the new steps are</p>
<ol>
<li>Query Indexes and get the key sets</li>
<li>Find the intersection of these key sets</li>
<li>Pass the resulting keys to MapReduce where we fetch and sort the data.</li>
</ol>
<p>Using this, you can write queries like</p>
<div class="codehilite"><pre><span></span><span class="n">client</span> <span class="o">=</span> <span class="n">riak</span><span class="o">.</span><span class="n">RiakClient</span><span class="p">(</span><span class="s1">'localhost'</span><span class="p">,</span> <span class="mi">8091</span><span class="p">)</span>
<span class="n">bucket</span> <span class="o">=</span> <span class="n">client</span><span class="o">.</span><span class="n">bucket</span><span class="p">(</span><span class="s1">'test_multi_index'</span><span class="p">)</span>
<span class="n">bucket</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="s1">'sree'</span><span class="p">,</span> <span class="p">{</span><span class="s1">'name'</span><span class="p">:</span> <span class="s1">'Sreejith'</span><span class="p">,</span> <span class="s1">'age'</span><span class="p">:</span> <span class="mi">25</span><span class="p">})</span><span class="o">.</span>\
<span class="n">add_index</span><span class="p">(</span><span class="s1">'name_bin'</span><span class="p">,</span> <span class="s1">'Sreejith'</span><span class="p">)</span><span class="o">.</span>\
<span class="n">add_index</span><span class="p">(</span><span class="s1">'age_int'</span><span class="p">,</span> <span class="mi">25</span><span class="p">)</span><span class="o">.</span><span class="n">store</span><span class="p">()</span>
<span class="n">bucket</span><span class="o">.</span><span class="n">new</span><span class="p">(</span><span class="s1">'vishnu'</span><span class="p">,</span> <span class="p">{</span><span class="s1">'name'</span><span class="p">:</span> <span class="s1">'Vishnu'</span><span class="p">,</span> <span class="s1">'age'</span><span class="p">:</span> <span class="mi">31</span><span class="p">})</span><span class="o">.</span>\
<span class="n">add_index</span><span class="p">(</span><span class="s1">'name_bin'</span><span class="p">,</span> <span class="s1">'Vishnu'</span><span class="p">)</span><span class="o">.</span>\
<span class="n">add_index</span><span class="p">(</span><span class="s1">'age_int'</span><span class="p">,</span> <span class="mi">31</span><span class="p">)</span><span class="o">.</span><span class="n">store</span><span class="p">()</span>
<span class="n">query</span> <span class="o">=</span> <span class="n">RiakMultiIndexQuery</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="s1">'test_multi_index'</span><span class="p">)</span>
<span class="k">for</span> <span class="n">res</span> <span class="ow">in</span> <span class="n">query</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="s1">'name'</span><span class="p">,</span> <span class="s1">'=='</span><span class="p">,</span> <span class="s1">'Sreejith'</span><span class="p">)</span><span class="o">.</span><span class="n">run</span><span class="p">():</span>
<span class="k">print</span> <span class="n">res</span>
<span class="n">query</span><span class="o">.</span><span class="n">reset</span><span class="p">()</span>
<span class="k">for</span> <span class="n">res</span> <span class="ow">in</span> <span class="n">query</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="s1">'age'</span><span class="p">,</span> <span class="s1">'<'</span><span class="p">,</span> <span class="mi">50</span><span class="p">)</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="s1">'name'</span><span class="p">,</span> <span class="s1">'=='</span><span class="p">,</span> <span class="s1">'Vishnu'</span><span class="p">)</span><span class="o">.</span><span class="n">run</span><span class="p">():</span>
<span class="k">print</span> <span class="n">res</span>
<span class="n">query</span><span class="o">.</span><span class="n">reset</span><span class="p">()</span>
<span class="k">for</span> <span class="n">res</span> <span class="ow">in</span> <span class="n">query</span><span class="o">.</span><span class="n">filter</span><span class="p">(</span><span class="s1">'age'</span><span class="p">,</span> <span class="s1">'<'</span><span class="p">,</span> <span class="mi">50</span><span class="p">)</span><span class="o">.</span><span class="n">order</span><span class="p">(</span><span class="s1">'age'</span><span class="p">,</span> <span class="s1">'ASC'</span><span class="p">)</span><span class="o">.</span><span class="n">run</span><span class="p">():</span>
<span class="k">print</span> <span class="n">res</span>
<span class="n">query</span><span class="o">.</span><span class="n">reset</span><span class="p">()</span>
<span class="k">for</span> <span class="n">res</span> <span class="ow">in</span> <span class="n">query</span><span class="o">.</span><span class="n">limit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">run</span><span class="p">():</span>
<span class="k">print</span> <span class="n">res</span>
<span class="n">query</span><span class="o">.</span><span class="n">reset</span><span class="p">()</span>
<span class="k">for</span> <span class="n">res</span> <span class="ow">in</span> <span class="n">query</span><span class="o">.</span><span class="n">order</span><span class="p">(</span><span class="s1">'age'</span><span class="p">,</span> <span class="s1">'ASC'</span><span class="p">)</span><span class="o">.</span><span class="n">offset</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">limit</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span><span class="o">.</span><span class="n">run</span><span class="p">():</span>
<span class="k">print</span> <span class="n">res</span>
</pre></div>
<p>You can find the full source code at <a href="https://github.com/semk/utils/blob/master/riak_multi_query.py">GitHub</a>.</p>Voldemort: A Jinja2 powered static site generator2011-10-05T00:00:00+5:30http://foobarnbaz.com/2011/10/05/voldemort-a-jinja-powered-static-site-generator<p><a href="https://github.com/semk/voldemort">Voldemort</a> is a blog-aware static site generator inspired by <a href="http://jinja.pocoo.org/2/">Jinja2</a>. All these times this blog was generated using <a href="https://github.com/mojombo/jekyll">Jekyll</a> and I always wanted to use something Pythonic. <a href="http://ringce.com/hyde">Hyde</a> was there, but the awesomeness of Jinja2 forced me to write Voldemort on my own.</p>
<p><img alt="Voldemort" src="/images/posts/2011-10-05-voldemort-a-jinja-powered-static-site-generator/voldemort.jpg" /></p>
<p>Voldemort has its own advantages. You can templatize your HTML pages using Jinja and write posts using <a href="http://daringfireball.net/projects/markdown">Markdown</a>. And if you like this website, then you've got one reason to tryout Voldemort. Go for an <a href="http://pypi.python.org/pypi/voldemort/0.6.0">install</a> or grab the <a href="https://github.com/semk/voldemort">sources</a> if you want to tweak it your own way.</p>Packaging Python Applications2011-10-02T00:00:00+5:30http://foobarnbaz.com/2011/10/02/packaging-python-applications<p>A few days ago, I came across a situation where I needed to create Debian packages for some Python libraries on which our software was dependant on. All these time we were creating and distributing the application as Eggs built using <a href="http://pypi.python.org/pypi/setuptools">setuptools</a> <code>setup.py</code> script. Later on this became a problem since some other applications which we were using were not Python applications and were packaged as <code>.deb</code> packages. This situation made us to build <code>.deb</code> packages for our Python software as well.</p>
<p>The real challenge here was packaging all the dependencies our Python application has brought in, as most of the dependencies were having only <code>egg</code> distributions. Luckily there was an extension to setuptools called <a href="https://github.com/astraw/stdeb">stdeb</a> which will allow you to generate debian packages using your setup.py script. It'll automatically add the dependencies in the <code>debian/control</code> file after searching for the dependencies using <code>apt-cache</code>. stdeb will do something like <code>apt-cache dump | grep <package></code> to find whether there is a proper debian package for the dependencies. If they found the correct one in apt-cache, <code>debian/control</code> file will be updated with that information. But as I said, most of the dependencies were not packaged as <code>.deb</code>. So we downloaded all the dependencies using the <code>easy_install</code> command and created <code>.deb</code> packages for them using <code>stdeb</code>.</p>
<p>I will demonstrate the procedure with the <a href="https://github.com/semk/voldemort">Voldemort</a> project from my GitHub Repo. The <code>setup.py</code> looks like this,</p>
<div class="codehilite"><pre><span></span><span class="k">try</span><span class="p">:</span>
<span class="kn">from</span> <span class="nn">setuptools</span> <span class="kn">import</span> <span class="n">setup</span><span class="p">,</span> <span class="n">find_packages</span>
<span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span>
<span class="kn">from</span> <span class="nn">ez_setup</span> <span class="kn">import</span> <span class="n">use_setuptools</span>
<span class="n">use_setuptools</span><span class="p">()</span>
<span class="kn">from</span> <span class="nn">setuptools</span> <span class="kn">import</span> <span class="n">setup</span><span class="p">,</span> <span class="n">find_packages</span>
<span class="n">setup</span><span class="p">(</span>
<span class="n">name</span><span class="o">=</span><span class="s1">'voldemort'</span><span class="p">,</span>
<span class="n">version</span><span class="o">=</span><span class="s1">'0.5.0'</span><span class="p">,</span>
<span class="n">description</span><span class="o">=</span><span class="s1">'Voldemort is a simple static site generator</span><span class="se">\</span>
<span class="s1"> using Jinja2 and markdown templates.'</span><span class="p">,</span>
<span class="n">author</span><span class="o">=</span><span class="s1">'Sreejith K / K7Computing Pvt Ltd'</span><span class="p">,</span>
<span class="n">author_email</span><span class="o">=</span><span class="s1">'sreejithemk@gmail.com'</span><span class="p">,</span>
<span class="n">url</span><span class="o">=</span><span class="s1">'http://www.foobarnbaz.com'</span><span class="p">,</span>
<span class="n">install_requires</span><span class="o">=</span><span class="p">[</span>
<span class="s1">'Pygments >= 1.4'</span><span class="p">,</span>
<span class="s1">'PyYAML >= 3.10'</span><span class="p">,</span>
<span class="s1">'Markdown >= 2.0'</span><span class="p">,</span>
<span class="s1">'Jinja2 >= 2.5'</span>
<span class="p">],</span>
<span class="n">setup_requires</span><span class="o">=</span><span class="p">[],</span>
<span class="n">packages</span><span class="o">=</span><span class="n">find_packages</span><span class="p">(</span><span class="n">exclude</span><span class="o">=</span><span class="p">[</span><span class="s1">'ez_setup'</span><span class="p">]),</span>
<span class="n">test_suite</span><span class="o">=</span><span class="s1">'tests'</span><span class="p">,</span>
<span class="n">scripts</span> <span class="o">=</span> <span class="p">[</span><span class="s1">'scripts/voldemort'</span><span class="p">],</span>
<span class="n">include_package_data</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
<span class="n">zip_safe</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span>
<span class="p">)</span>
</pre></div>
<p>In this case all the dependencies were lacking <code>.deb</code> packages. So, we need to download them using <code>easy_install</code>.</p>
<div class="codehilite"><pre><span></span>mkdir deps
easy_install -e -b deps/ <span class="s2">"Pygments>=1.4"</span>
easy_install -e -b deps/ <span class="s2">"PyYAML>=3.10"</span>
easy_install -e -b deps/ <span class="s2">"Markdown>=2.0"</span>
easy_install -e -b deps/ <span class="s2">"Jinja2>=2.5"</span>
ls deps/
jinja2 markdown pygments pyyaml
</pre></div>
<p>Now you need to go to each dependency directory and issue the following command</p>
<div class="codehilite"><pre><span></span>python setup.py --command-packages<span class="o">=</span>stdeb.command debianize
</pre></div>
<p>That will create a directory named <code>debian</code> which contain all the stuffs needed to create a <code>.deb</code> package. Now it is upto us to edit the control file and add any additional dependency if needed. To generate the debian package, do</p>
<div class="codehilite"><pre><span></span>dpkg-buildpackage -us -uc
</pre></div>
<p>Now you can build the main application debs by editing the control file manually since stdeb won't add the dependencies. But you know you have them in your hand :-).</p>Developing scalable services with Python2011-08-30T00:00:00+5:30http://foobarnbaz.com/2011/08/30/developing-scalable-services-with-python<p>Developing multi-threaded applications in python is a "Pain In The Ass". And the <a href="http://wiki.python.org/moin/GlobalInterpreterLock">GIL</a> (Global Interpreter Lock) takes away the advantage of utilizing multiple cores in a machine. It doesn't matter how many cores a CPU have, GIL prevents threads from running in multiple cores. So python programs would't get the maximum performance out of the CPU when they use threads in their services.</p>
<p>In many cases you might need to write services where you need to listen on a port and wait for the client connection to do some task. If multiple clients are connecting to this service simultaneously then you might need to spawn threads to handle the requests. Considering the fact that GIL introduces a performance bottleneck, the best way to solve this situation is to use python's <a href="http://docs.python.org/library/multiprocessing.html">multiprocessing</a> capabilities. This library provides almost <code>threading</code> like class implementations. </p>
<p>Then a question might arise. How do you share a socket created by the server process to the newly spawned processes? It is possible since all the <code>fork</code>ed processes will have the parent's file descriptors. <code>multiprocessing</code> library already has this package named <code>multiprocessing.reduction</code> that provides a method <code>reduce_handle</code> which can serialize a socket and you can send this socket to another process using pipes. The child processes can read from the pipe and re-create the socket using <code>rebuild_handle</code>. The following example will make this idea clear to you.</p>
<div class="codehilite"><pre><span></span><span class="c1"># Main Process</span>
<span class="kn">from</span> <span class="nn">multiprocessing.reduction</span> <span class="kn">import</span> <span class="n">reduce_handle</span>
<span class="c1"># serialize the socket</span>
<span class="n">serialized_socket</span> <span class="o">=</span> <span class="n">reduce_handle</span><span class="p">(</span><span class="n">client_socket</span><span class="o">.</span><span class="n">fileno</span><span class="p">())</span>
<span class="c1"># send it to the child/worker process</span>
<span class="n">pipe_to_worker</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="n">serialized_socket</span><span class="p">)</span>
<span class="c1"># Worker Process</span>
<span class="kn">from</span> <span class="nn">multiprocessing.reduction</span> <span class="kn">import</span> <span class="n">rebuild_handle</span>
<span class="c1"># get the socket from parent</span>
<span class="n">serialized_socket</span> <span class="o">=</span> <span class="n">pipe_from_parent</span><span class="o">.</span><span class="n">recv</span><span class="p">()</span>
<span class="c1"># rebuild the file descriptor</span>
<span class="n">fd</span> <span class="o">=</span> <span class="n">rebuild_handle</span><span class="p">(</span><span class="n">serialized_socket</span><span class="p">)</span>
<span class="c1"># create socket from fd</span>
<span class="n">client_socket</span> <span class="o">=</span> <span class="n">socket</span><span class="o">.</span><span class="n">fromfd</span><span class="p">(</span><span class="n">fd</span><span class="p">,</span> <span class="n">socket</span><span class="o">.</span><span class="n">AF_INET</span><span class="p">,</span> <span class="n">socket</span><span class="o">.</span><span class="n">SOCK_STREAM</span><span class="p">)</span>
<span class="c1"># use the socket as usual. eg: send a message to the client</span>
<span class="n">client_socket</span><span class="o">.</span><span class="n">send</span><span class="p">(</span><span class="s2">"Baby, I</span><span class="se">\'</span><span class="s2">m so fast</span><span class="se">\r\n</span><span class="s2">"</span><span class="p">)</span>
</pre></div>
<h3>Preforking</h3>
<p>Another way to solve this issue is by spawning multiple process from the main server which is listening on a socket/port and letting all the child processes to <code>accept()</code> connections from the client. Apache uses this style of process scaling known as "Preforking". A simple example using <code>multiprocessing</code> module which runs an instance of a <code>BaseHTTPServer.HTTPServer</code> on a pool of worker processes can be written very easily as follows.</p>
<div class="codehilite"><pre><span></span><span class="c1">#</span>
<span class="c1"># Example where a pool of http servers share a single listening socket</span>
<span class="c1">#</span>
<span class="c1"># On Windows this module depends on the ability to pickle a socket</span>
<span class="c1"># object so that the worker processes can inherit a copy of the server</span>
<span class="c1"># object. (We import `multiprocessing.reduction` to enable this pickling.)</span>
<span class="c1">#</span>
<span class="c1"># Not sure if we should synchronize access to `socket.accept()` method by</span>
<span class="c1"># using a process-shared lock -- does not seem to be necessary.</span>
<span class="c1">#</span>
<span class="c1"># Copyright (c) 2006-2008, R Oudkerk</span>
<span class="c1"># All rights reserved.</span>
<span class="c1">#</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">from</span> <span class="nn">multiprocessing</span> <span class="kn">import</span> <span class="n">Process</span><span class="p">,</span> <span class="n">current_process</span><span class="p">,</span> <span class="n">freeze_support</span>
<span class="kn">from</span> <span class="nn">BaseHTTPServer</span> <span class="kn">import</span> <span class="n">HTTPServer</span>
<span class="kn">from</span> <span class="nn">SimpleHTTPServer</span> <span class="kn">import</span> <span class="n">SimpleHTTPRequestHandler</span>
<span class="k">if</span> <span class="n">sys</span><span class="o">.</span><span class="n">platform</span> <span class="o">==</span> <span class="s1">'win32'</span><span class="p">:</span>
<span class="kn">import</span> <span class="nn">multiprocessing.reduction</span> <span class="c1"># make sockets pickable/inheritable</span>
<span class="k">def</span> <span class="nf">note</span><span class="p">(</span><span class="n">format</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">):</span>
<span class="n">sys</span><span class="o">.</span><span class="n">stderr</span><span class="o">.</span><span class="n">write</span><span class="p">(</span><span class="s1">'[</span><span class="si">%s</span><span class="s1">]</span><span class="se">\t</span><span class="si">%s</span><span class="se">\n</span><span class="s1">'</span> <span class="o">%</span> <span class="p">(</span><span class="n">current_process</span><span class="p">()</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="n">format</span><span class="o">%</span><span class="n">args</span><span class="p">))</span>
<span class="k">class</span> <span class="nc">RequestHandler</span><span class="p">(</span><span class="n">SimpleHTTPRequestHandler</span><span class="p">):</span>
<span class="c1"># we override log_message() to show which process is handling the request</span>
<span class="k">def</span> <span class="nf">log_message</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">format</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">):</span>
<span class="n">note</span><span class="p">(</span><span class="n">format</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">serve_forever</span><span class="p">(</span><span class="n">server</span><span class="p">):</span>
<span class="n">note</span><span class="p">(</span><span class="s1">'starting server'</span><span class="p">)</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">server</span><span class="o">.</span><span class="n">serve_forever</span><span class="p">()</span>
<span class="k">except</span> <span class="ne">KeyboardInterrupt</span><span class="p">:</span>
<span class="k">pass</span>
<span class="k">def</span> <span class="nf">runpool</span><span class="p">(</span><span class="n">address</span><span class="p">,</span> <span class="n">number_of_processes</span><span class="p">):</span>
<span class="c1"># create a single server object -- children will each inherit a copy</span>
<span class="n">server</span> <span class="o">=</span> <span class="n">HTTPServer</span><span class="p">(</span><span class="n">address</span><span class="p">,</span> <span class="n">RequestHandler</span><span class="p">)</span>
<span class="c1"># create child processes to act as workers</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">number_of_processes</span><span class="o">-</span><span class="mi">1</span><span class="p">):</span>
<span class="n">Process</span><span class="p">(</span><span class="n">target</span><span class="o">=</span><span class="n">serve_forever</span><span class="p">,</span> <span class="n">args</span><span class="o">=</span><span class="p">(</span><span class="n">server</span><span class="p">,))</span><span class="o">.</span><span class="n">start</span><span class="p">()</span>
<span class="c1"># main process also acts as a worker</span>
<span class="n">serve_forever</span><span class="p">(</span><span class="n">server</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">test</span><span class="p">():</span>
<span class="n">DIR</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">dirname</span><span class="p">(</span><span class="n">__file__</span><span class="p">),</span> <span class="s1">'..'</span><span class="p">)</span>
<span class="n">ADDRESS</span> <span class="o">=</span> <span class="p">(</span><span class="s1">'localhost'</span><span class="p">,</span> <span class="mi">8000</span><span class="p">)</span>
<span class="n">NUMBER_OF_PROCESSES</span> <span class="o">=</span> <span class="mi">4</span>
<span class="k">print</span> <span class="s1">'Serving at http://</span><span class="si">%s</span><span class="s1">:</span><span class="si">%d</span><span class="s1"> using </span><span class="si">%d</span><span class="s1"> worker processes'</span> <span class="o">%</span> \
<span class="p">(</span><span class="n">ADDRESS</span><span class="p">[</span><span class="mi">0</span><span class="p">],</span> <span class="n">ADDRESS</span><span class="p">[</span><span class="mi">1</span><span class="p">],</span> <span class="n">NUMBER_OF_PROCESSES</span><span class="p">)</span>
<span class="k">print</span> <span class="s1">'To exit press Ctrl-'</span> <span class="o">+</span> <span class="p">[</span><span class="s1">'C'</span><span class="p">,</span> <span class="s1">'Break'</span><span class="p">][</span><span class="n">sys</span><span class="o">.</span><span class="n">platform</span><span class="o">==</span><span class="s1">'win32'</span><span class="p">]</span>
<span class="n">os</span><span class="o">.</span><span class="n">chdir</span><span class="p">(</span><span class="n">DIR</span><span class="p">)</span>
<span class="n">runpool</span><span class="p">(</span><span class="n">ADDRESS</span><span class="p">,</span> <span class="n">NUMBER_OF_PROCESSES</span><span class="p">)</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
<span class="n">freeze_support</span><span class="p">()</span>
<span class="n">test</span><span class="p">()</span>
</pre></div>
<p>I wrote a simple wrapper for this kind of services which can be scaled. You can find it <a href="https://github.com/semk/utils/blob/master/prefork_server.py">here</a>.</p>Custom Authentication for Google App Engine apps2011-07-31T00:00:00+5:30http://foobarnbaz.com/2011/07/31/custom-authentication-on-gae<p><img alt="Google App Engine" src="http://upload.wikimedia.org/wikipedia/en/3/38/Google_App_Engine_Logo.png" /></p>
<p><a href="http://code.google.com/appengine">Google App Engine</a> is a widely used and most popular <a href="en.wikipedia.org/wiki/Platform_as_a_service">PaaS</a> solution provided by Google. App Engine provides the developer with a wide range of apis which can be used to develop web applications using any <a href="http://en.wikipedia.org/wiki/Web_Server_Gateway_Interface">WSGI</a> compliant Frameworks (<a href="http://code.google.com/appengine/docs/python/tools/webapp/">Webapp</a>, <a href="http://www.tipfy.org">Tipfy</a>, <a href="http://www.django.org">Django</a>, <a href="http://bottlepy.org">Bottle</a>, <a href="http://www.tornadoweb.org">Tornado</a> etc.). One of the apis App Engine provides is the <a href="http://code.google.com/appengine/docs/python/users/overview.html">users api</a>, which most of the developers confuses for an api which provides user creation, authentication for the application. But this api only authenticates Google Accounts (can be the application developer or any third-party Google Account) using <a href="http://oauth.net/">OAuth</a>. You can't really user this api to create or manage users for your application.</p>
<p>Remember when I told you that every application you write for GAE is a WSGI Application? WSGI is just a standard for the web application to talk to the backend HTTP server. That means a WSGI application can't run by itself. It needs an HTTP server to listen on and execute the code you have written. Its the HTTP Server which handles all the server stuffs used for authentication such as setting cookies. Now, GAE has a <a href="code.google.com/appengine/docs/python/runtime.html">sandbox</a>, which is a restrictive environment for your application code to run. For example, your application is restricted for file operations and certain modules are restricted from importing. So you can't really set up a cookie from your application code since its not an HTTP Server code. Here is our problem now. How do you do a custom authentication for a Google App Engine application?</p>
<p>You can achieve this by writing a middleware to your WSGI application. There are many authentication libraries available for this purpose. Popular ones are <a href="http://beaker.groovie.org/">Beaker</a>, <a href="https://github.com/dound/gae-sessions">GAE-Sessions</a>, <a href="http://gaeutilities.appspot.com/session">gaeutilities</a>. But I liked the GAE-Sessions library better than the other ones since its the <a href="https://github.com/dound/gae-sessions/wiki/comparison-with-alternative-libraries">fastest</a> of them all. GAE-Sessions use <a href="code.google.com/appengine/docs/memcache/">memcache</a>/<a href="code.google.com/appengine/docs/datastore/">datastore</a> to store the session information. To use this library, just copy the gaesessions directory to your application directory. The middleware for your application is as simple as shown in the code below</p>
<div class="codehilite"><pre><span></span><span class="kn">from</span> <span class="nn">gaesessions</span> <span class="kn">import</span> <span class="n">SessionMiddleware</span>
<span class="k">def</span> <span class="nf">webapp_add_wsgi_middleware</span><span class="p">(</span><span class="n">app</span><span class="p">):</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">SessionMiddleware</span><span class="p">(</span><span class="n">app</span><span class="p">,</span> <span class="n">cookie_key</span><span class="o">=</span><span class="s2">"a random and long string"</span><span class="p">)</span>
<span class="k">return</span> <span class="n">app</span>
</pre></div>
<p>Save the above code to a file named <code>appengine_config.py</code>. You can use <code>gaesessions.get_current_session()</code> to get a dictionary to store all the session information. You can either use <code>session.set_quick(<session-var>, <value>)</code> method to use application memcache for storing session info. Otherwise it'll be stored to the datastore. Getting session info is as easy as <code>session.get(<session-var>)</code> and <code>session.pop_quick(<session-var>)</code> will remove the information from the session. All the dictionary like indexed operations will be persisted to the database.</p>
<div class="codehilite"><pre><span></span><span class="kn">from</span> <span class="nn">gaesessions</span> <span class="kn">import</span> <span class="n">get_current_session</span>
<span class="n">session</span> <span class="o">=</span> <span class="n">get_current_session</span><span class="p">()</span>
<span class="c1"># setting user session information</span>
<span class="n">session</span><span class="o">.</span><span class="n">set_quick</span><span class="p">(</span><span class="s1">'user'</span><span class="p">,</span> <span class="s1">'authenticated_user_info'</span><span class="p">)</span>
<span class="c1"># getting user session</span>
<span class="n">user</span> <span class="o">=</span> <span class="n">session</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">'user'</span><span class="p">)</span>
<span class="c1"># removing session info</span>
<span class="n">session</span><span class="o">.</span><span class="n">pop_quick</span><span class="p">(</span><span class="s1">'user'</span><span class="p">)</span>
</pre></div>
<p>The default session lifetime is 7 days. You may configure how long a session lasts by calling <code>SessionMiddleware</code> with a <code>lifetime</code> parameter, e.g., <code>lifetime=datetime.timedelta(hours=2)</code>). You can schedule a cron job for cleaning up all the expired session info for your application by creating a handler file like this.</p>
<div class="codehilite"><pre><span></span><span class="c1"># cleanup_sessions.py</span>
<span class="kn">from</span> <span class="nn">gaesessions</span> <span class="kn">import</span> <span class="n">delete_expired_sessions</span>
<span class="k">while</span> <span class="ow">not</span> <span class="n">delete_expired_sessions</span><span class="p">():</span>
<span class="k">pass</span>
</pre></div>
<p>Make sure you have a <code>cron.yaml</code> with the correct info.</p>
<div class="codehilite"><pre><span></span><span class="l l-Scalar l-Scalar-Plain">cron</span><span class="p p-Indicator">:</span>
<span class="p p-Indicator">-</span> <span class="l l-Scalar l-Scalar-Plain">description</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">daily session cleanup</span>
<span class="l l-Scalar l-Scalar-Plain">url</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">/cleanup_sessions</span>
<span class="l l-Scalar l-Scalar-Plain">schedule</span><span class="p p-Indicator">:</span> <span class="l l-Scalar l-Scalar-Plain">every 24 hours</span>
</pre></div>
<p>You can find a complete sample application <a href="https://github.com/dound/gae-sessions/blob/master/demo">here</a>.</p>Fix Parallels Desktop on Mac OSX Lion2011-07-27T00:00:00+5:30http://foobarnbaz.com/2011/07/27/fix-parallels-desktop-on-lion<p><img alt="Parallels Desktop" src="http://cdn2.digitaltrends.com/wp-content/uploads/2010/09/pdfm-box-278x266.jpg" /></p>
<p>Most of the <a href="www.parallels.com/products/desktop/">Parallels Desktop 6</a> users are experiencing an <a href="http://www.youtube.com/watch?v=g4yhICBqpfk">issue</a> not finding the virtual machine windows after upgrading to <a href="www.apple.com/macosx/">Mac OSX Lion</a>. This might be caused by the Desktop (inwhich Parallels to run on) being set to <code>None</code> during upgrade. <a href="http://www.apple.com/macosx/whats-new/mission-control.html">Mission Control</a> to blame? But this can be fixed very easily. Follow these steps,</p>
<ol>
<li>Create a Dock Icon for Parallels Desktop if not</li>
<li>Launch Parallels Desktop</li>
<li>Right click the Parallels Dock Icon and go to <code>Options</code>. Then set <code>Assign To</code> to <code>This Desktop</code>.</li>
<li>Done.</li>
</ol>
<p>According to Parallels FAQ, there is one more workaround to fix this issue,</p>
<ol>
<li>Delete all the spaces except the default one</li>
<li>Reboot and restart Parallels</li>
</ol>
<p>I hope this will be helpful for those struggling to get around this issue.</p>Meet Raspberry Pi, the $25 computer2011-05-07T00:00:00+5:30http://foobarnbaz.com/2011/05/07/cheapest-personal-computer-on-a-usb<p>No one would have thought that they could ever buy a computer for $25. Thanks to the smart game developer from <a href="http://www.frontier.co.uk/">Frontier Developments</a>, <a href="http://en.wikipedia.org/wiki/David_Braben">David Braben</a> who designed <a href="http://www.raspberrypi.org/">Raspberry Pi</a>, a $25 dollar computer with the size of a USB Stick, infact it is actually a USB Stick Computer. Have a look,</p>
<p><img alt="Raspberry Pi" src="http://www.geek.com/wp-content/uploads/2011/05/rasperry_pi_pcb-580x435.jpg" /></p>
<p>This tiny device comes with an HDMI port in one end and a USB port on the other. You can plug it into a HDMI socket and then connect a keyboard via the USB port turns it into a fully functioning machine running a flavour of Linux. And the Hardware Specs are not that bad at all. It boasts a 700MHz ARM11 processor coupled with 128MB of RAM and runs OpenGL ES 2.0 allowing for decent graphics performance with 1080p. Yes 1080p. You can also add an SD card to the SD card slot for your storage needs. Did I already mention that you can also mount a 12MP camera into this?</p>
<p>It looks like the stock OS it ships with is Ubuntu. That means you can browse, run office apps or do whatever you want to do with a computer. Braben argues that the new ICT syllabus in schools have changed a lot and no one is teaching programming and the internals of computer systems. They are now teaching other <em>useful</em> skills like writing documents in a word processor, how to create presentations etc. This could be his effort to bring back the good stuff back to schools by making an affordable computer for students.</p>Scratch - A Magic Tool for Tux Paint2011-04-11T00:00:00+5:30http://foobarnbaz.com/2011/04/11/a-magic-tool-for-tuxpaint<p><a href="http://tuxpaint.org/">Tux Paint</a> is a drawing software for children that provides a kids-like user interface. Tuxpaint does have some nice tools integrated in it which helps the kids to create their drawing <a href="http://tuxpaint.org/gallery/">masterpieces</a> using a computer. Simply speaking its the GIMP for kids of 3-12 years.</p>
<p>But what I was excited about Tux Paint is the availability of <a href="http://www.tuxpaint.org/presentations/tuxpaint-magic-api.pdf">Magic Tool Plug-in API</a> using which you can create Magic Tools that creates some sort of graphical effect on the canvas. Tux Paint already provides some Magic Tools by default like blur, bricks etc. Tux paint is fully written in C and uses <a href="http://libsdl.org">SDL</a> for graphics programming. There are basically two kinds of tools we can create using the API. One that affects the whole canvas and one that updates a specific region of the canvas (eg. place where you drag the cursor).</p>
<p><img alt="Scratch effect on Bricks" src="/images/posts/2011-04-11-a-magic-tool-for-tux-paint/scratch_on_bricks.png" /></p>
<p>To test how this API works, I actually developed a <a href="https://github.com/semk/tp-scratch">Scratch Magic Tool</a> which when you apply, creates a <em>scratched</em> effect on the image (as shown above). It also produces a nice sound effect while applying the effect. Still this tool has a lot of logical bugs so that sometimes it doesn't even look like a scratch when applied over certain colors :-). Other than that, you can use this code for a reference to create your own Magic Tools. If you need documentation this presentation <a href="http://www.tuxpaint.org/presentations/tuxpaint-magic-api.pdf">slide</a> is more than enough. Once you have downloaded the Tux Paint source code go through <code>src/tp_magic_api.h</code> to see the all API definitions and usage.</p>
<p>Back in the college days, we had this subject on <em>Computer Graphics</em> where you need to study some popular <a href="http://en.wikipedia.org/wiki/Bresenham%27s_line_algorithm">line-drawing</a> and <a href="http://en.wikipedia.org/wiki/Midpoint_circle_algorithm">circle-drawing</a> algorithms. But there were no lab sessions for those subjects. Everyone where literally by-hearting those algorithms at that time. I would suggest all the CS faculty to conduct a lab session on Computer Graphics and ask students to implement those algorithms as a Magic Tool in Tux Paint using the Magic Tool API. It will be useful for them to see those algorithms in action. You may use my plugin or other plugins available in Tux Paint's <code>magic/src/</code> directory as a reference for your development. Please do give a feedback here in the comment section when you implement some of these algorithms as a Magic Tool.</p>
<p><em>NOTE:</em> For compilation and installation of Tux Paint from sources, follow the instructions from <a href="https://github.com/semk/tp-scratch">tp-scratch repository</a>.</p>I have an'droid2011-02-21T00:00:00+5:30http://foobarnbaz.com/2011/02/21/i-have-an-droid<p>Finally my wish came true. Now I'm a proud owner of an Android phone. Its LG Optimus One, better known as LG P500. Good that it comes with Froyo and LG promises an update to Gingerbread. Here you can find the phone <a href="http://www.gsmarena.com/lg_optimus_one_p500-3516.php">specs</a>.</p>
<p><img alt="LG Optimus One" src="http://www.gadgetvenue.com/wp-content/uploads/2010/09/LG-optimus-one-300x300.jpg" /></p>
<p>This one is a good choice for anyone looking for a budget Android phone (10.8K when I bought) whose battery lasts a complete day with internet on. Thanks to <a href="http://binarykarma.org">Shuveb</a> for providing a review of the phone and atlast making me buy it :-P</p>
<p>Eventhough a Pythonista like me hates Java, the Android platform was tempting me to look at Java with some interest. So I bought <em>Android Application Development by O'Reilly</em> and setup a Development Environment on my laptop using Eclipse. This was so easy that I could run the <em>Hello World</em> program without even writing a piece of Java code. Once you create a Project, it auto-generates files needed for the application to show on the foreground and other resource files and build scripts. By default, they'll create an activity source file with an <em>Activity</em> name you've given. This is pretty much the <em>Hello World</em> program we wanted.</p>
<div class="codehilite"><pre><span></span><span class="kn">package</span> <span class="nn">com.semk.helloworld</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">android.app.Activity</span><span class="o">;</span>
<span class="kn">import</span> <span class="nn">android.os.Bundle</span><span class="o">;</span>
<span class="kd">public</span> <span class="kd">class</span> <span class="nc">HelloWorldActivity</span> <span class="kd">extends</span> <span class="n">Activity</span> <span class="o">{</span>
<span class="cm">/** Called when the activity is first created. */</span>
<span class="nd">@Override</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">onCreate</span><span class="o">(</span><span class="n">Bundle</span> <span class="n">savedInstanceState</span><span class="o">)</span> <span class="o">{</span>
<span class="kd">super</span><span class="o">.</span><span class="na">onCreate</span><span class="o">(</span><span class="n">savedInstanceState</span><span class="o">);</span>
<span class="n">setContentView</span><span class="o">(</span><span class="n">R</span><span class="o">.</span><span class="na">layout</span><span class="o">.</span><span class="na">main</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</pre></div>
<p>Google has done a great job in developing an easy-to-use and feature-abundant SDK for Android. The Android emulator which comes with the SDK is an engineering masterpiece. It helps the developer to run and test their applications locally before running them on the actual devices.</p>
<p><img alt="Android Emulator" src="/images/posts/2011-02-21-i-have-an-droid/emulator.png" /></p>
<p>I'm hoping to develop some useful applications for Android while I finish reading the book. So keep an eye on my <a href="http://github.com/semk">Git Repo</a>.</p>HyperGAE - Use Hypertable for App Engine Datastore2011-01-08T00:00:00+5:30http://foobarnbaz.com/2011/01/08/hypergae-use-hypertable-for-appengine-datastore<p>After few days of hacking on the <a href="http://code.google.com/appengine/docs/python/overview.html">Google App Engine SDK</a> and <a href="http://code.google.com/apis/protocolbuffers/docs/pythontutorial.html">ProtocolBuffers</a>, finally I succeeded in creating a datastore driver for GAE that talks to <a href="http://hypertable.org">Hypertable</a> and stores the data there <em>fully protocol buffer encoded</em>. If you want to checkout this implementation head to <a href="https://github.com/semk/hypergae">HyperGAE</a> repository and see <a href="https://github.com/semk/hypergae/tree/master/google/appengine/datastore">files</a> <code>datastore_hypertable_ht.py</code> and <code>datastore_hypertable_thrift.py</code>. HyperGAE basically uses two methods to connect to hypertable. Using the thrift api and using the boost-python library <a href="http://code.google.com/p/python-hypertable/">ht</a>. The mentioned drivers provides these api connections to hypertable.</p>
<p>To run App Engine sdk with hypertable support, do</p>
<div class="codehilite"><pre><span></span>python dev_appserver.py demos/guestbook/ --use_hypertable
</pre></div>
<p>The above command assumes hypertable <a href="http://www.hypertable.com/download/0.9.3.4.html">0.9.3.4</a> installed and running on your machine. It uses the thrift connection by default. I need to add support for configuring hypertable options though <code>dev_appserver.py</code> script. Suggestions and patches are always welcome.</p>
<p><em>UPDATE:</em> The drivers api has been modified to accommodate the new Hypertable and Thrift api changes. You must install Hypertable version <a href="http://www.hypertable.com/download/">0.9.4.3</a> for HyperGAE to work. More updates to follow.</p>Building Custom Firmware for PS32011-01-05T00:00:00+5:30http://foobarnbaz.com/2011/01/05/building-custom-firmware-for-ps3<p>The 27th <a href="http://events.ccc.de/congress/2010/wiki/Welcome">Chaos Communication Congress</a> revealed the <a href="http://www.engadget.com/2010/12/29/hackers-obtain-ps3-private-cryptography-key-due-to-epic-programm/"><em>Epic Programming Fail</em></a> on the PlayStation 3 security. Finally it turned out that Sony's ECDSA algorithm for signing apps use a random number generator which always returns the same number (No, I'm not kidding). The <a href="http://fail0verflow.com">fail0verflow</a> team, who found out this fail, used the following famous <a href="http://xkcd.com">xkcd</a> comic to describe Sony's ECDSA algorithm.</p>
<p><img alt="Random Number" src="http://imgs.xkcd.com/comics/random_number.png" /></p>
<p><em>fail0verflow</em> demonstrated their work on <a href="http://marcansoft.com/blog/2010/10/asbestos-running-linux-as-gameos/">AsbestOS</a>, which is a bootloader for ps3 using which you can boot <em>Linux</em> on it. Currently this needs some wiring and soldering works but they have promised to release an easy to use tool sometime next month.</p>
<p>But <a href="https://github.com/kakaroto">kakaroto</a>(the hacker behind aMsn and PL3 payload for <em>psgroove</em>) wasn't patient enough. He went on to develop some tools to create <em>custom firmware</em> for ps3. You can find the custom firmware generator code from <a href="https://github.com/kakaroto/ps3utils">here</a>. Currently it works on Linux and Mac. For those who doesn't know how to use these tools, the following steps would help.</p>
<p>First of all checkout the tools developed by <em>fail0verflow</em> team and build it. You might need to install <code>build-essential</code> and <code>zlib1g-dev</code> packages if you are using Ubuntu (use <code>sudo apt-get install</code>).</p>
<div class="codehilite"><pre><span></span>mkdir ~/src
<span class="nb">cd</span> ~/src
git clone git://git.fail0verflow.com/ps3tools.git
<span class="nb">cd</span> ps3tools
make
</pre></div>
<p>Now checkout the firmware generator code from kakaroto's GitHub repository and use the official firmware update (PS3UPDAT.PUP) to create the CFW.</p>
<div class="codehilite"><pre><span></span>mkdir ~/.ps3
<span class="nb">cd</span> ~/.ps3
git clone https://github.com/kakaroto/ps3keys.git .
<span class="nb">cd</span> ~/src
git clone https://github.com/kakaroto/ps3utils.git
<span class="nb">cd</span> ps3utils
make
./create_cfw.sh PS3UPDAT.PUP CFW.PUP
</pre></div>
<p>You can rename this CFW.PUP to PS3UPDAT.PUP and copy it to PS3->UPDATE directory of your pendrive and update the ps3 from the XMB. Still, this CFW does nothing but shows the <em>Install Application</em> option on XMB. You still have to wait until the homebrews are signed and repackaged with the retail <code>.pkg</code> format to install on this firmware.</p>
<h4>Links</h4>
<ul>
<li>
<p><a href="http://events.ccc.de/congress/2010/Fahrplan/attachments/1780_27c3_console_hacking_2010.pdf">fail0verflow's presentation on Console Hacking</a> - 27th Chaos Communication Congress</p>
</li>
<li>
<p><a href="http://www.bbc.co.uk/news/technology-12116051">fail0verflow and Geohot Interview with BBC</a></p>
</li>
</ul>Reads for the New Year2010-12-31T00:00:00+5:30http://foobarnbaz.com/2010/12/31/reads-for-the-new-year<p>Now I have two new good books to read for this New Year. The classic <a href="http://mitpress.mit.edu/sicp/full-text/book/book.html">Structure and Interpretation of Computer Programs/SICP</a> and <a href="http://www.amazon.com/Mock-Mockingbird-Other-Logic-Puzzles/dp/0394534913">To Mock a Mockingbird</a> (<a href="http://en.wikipedia.org/wiki/Raymond_Smullyan">Raymond Smullyan</a>), a humorous introduction to <a href="http://en.wikipedia.org/wiki/Combinatory_logic">combinatory logic</a> with the help of <a href="http://www.codinghorror.com/blog/2007/09/classic-computer-science-puzzles.html">interesting puzzles</a>.</p>
<p>SICP was used as the textbook of MIT's old introductory programming class <a href="http://sicp.ai.mit.edu/Spring-2007/">6.001</a>. Later this class has been replaced by <a href="http://mit.edu/6.01/mercurial/spring10/www/index.html">6.01</a>, which uses Python. It is also known as the wizard book beacause it features the image of a wizard on its cover page.</p>
<p><img alt="Books" src="/images/posts/2010-12-31-reads-for-the-new-year/books.png" /></p>
<p>To Mock a Mockingbird contains numerous logical puzzles and by solving them, you will eventually learn some theories and concepts in combinatorics and <a href="http://en.wikipedia.org/wiki/Lambda_calculus">lambda calulus</a>, the base of <a href="http://en.wikipedia.org/wiki/Lisp_programming_language">Lisp</a> programming.</p>
<p>Well. Afterall, its time to say good bye to 2010 and welcome 2011.</p>Learn web development with Flask2010-12-25T00:00:00+5:30http://foobarnbaz.com/2010/12/25/learn-web-development-with-flask<p>I've seen many people jumping into web development with feature-complete frameworks like <a href="http://www.djangoproject.com/">Django</a> and <a href="http://rubyonrails.org/">Rails</a>. But most of them will find it difficult to assimilate the web development concepts because of cluttered documentation (incase of Rails) or complexity (incase of Django). One might need to look at a simpler microframework to learn from scratch. And yes, <a href="http://flask.pocoo.org/">Flask</a> is the one you'd want to have a look at.</p>
<p>Flask is a microframework for Python based on <a href="http://www.pocoo.org/projects/werkzeug/">Werkzeug</a>, and <a href="http://www.pocoo.org/projects/jinja2/">Jinja 2</a>. It is developed by the same guys at <a href="http://www.pocoo.org/">Pocoo</a> who gave us <a href="http://www.pocoo.org/projects/sphinx/">Sphinx</a> and <a href="http://www.pocoo.org/projects/pygments/">Pygments</a> (a Python based syntax highlighter which beautifies the codes shown in this website). If you are new to the world of web development, with Flask you'd find it easier to understand the basics concepts like <em>sessions</em>, <em>cookie</em>, <em>templates</em> etc.</p>
<p>Flask provides a more developer friendly routing mechanism. You can define the routes to handlers with the help of a simple python decorator. That means no pain-in-the-ass for defining routes and writing handlers separately. Let's have a look at a sample <em>Hello World</em> application in Flask. I don't think you might need an explanation to this code :-)</p>
<div class="codehilite"><pre><span></span><span class="c1"># Hello World in Flask</span>
<span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span>
<span class="nd">@app.route</span><span class="p">(</span><span class="s2">"/"</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">hello</span><span class="p">():</span>
<span class="k">return</span> <span class="s2">"Hello World!"</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">app</span><span class="o">.</span><span class="n">run</span><span class="p">()</span>
</pre></div>
<p>If you want to look at some really cool applications using Flask, never hesitate to checkout the code from <a href="https://github.com/mitsuhiko/flask/tree/master/examples">here</a>.</p>Factory/Service mode on PlayStation 32010-12-12T00:00:00+5:30http://foobarnbaz.com/2010/12/12/factory-service-mode-om-playstation-3<p>Here is a happy news for those who updated to firmware <code>>3.41</code> and didn't get a chance to enjoy the homwbrews for PS3. Yes, atlast we'll be able to downgrade the PS3 firmware to any version <code><3.50</code>. The actual PSJailbreak team were the first team to release the downgrade. But many thanks to <a href="https://github.com/zAxis">zAxis</a> who created <a href="https://github.com/zAxis/PSGrade">PSGrade</a>, an opensouce version of PSDowngrade. Using this we can put the PS3 into Factory/Service mode (Downgrade) using popular development boards like Teensy and AT90USBKEY.</p>
<p>We should also be thankful to <a href="https://github.com/kakaroto">KaKaRaTo</a> who reversed the PS3Yes Key and made it available for PSGrade. Here is PS3Yes key which he reversed from its hex.</p>
<blockquote>
<p>0×04, 0x4E, 0×61, 0x1B, 0xA6, 0xA6, 0xE3, 0x9A, 0×98, 0xCF, 0×35, 0×81, 0x2C, 0×80, 0×68, 0xC7, 0xFC, 0x5F, 0x7A, 0xE8</p>
</blockquote>
<p>You'll need to specify this key in PSGrade sources (key.c as shown below) for the service mode hack to work. But now zAxis has updated the code with this key. So all you have to do is download the souces and compile the hex for your board.</p>
<div class="codehilite"><pre><span></span><span class="k">const</span> <span class="kt">uint8_t</span> <span class="n">jig_key</span><span class="p">[</span><span class="mi">20</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
<span class="mh">0x04</span><span class="p">,</span> <span class="mh">0x4E</span><span class="p">,</span> <span class="mh">0x61</span><span class="p">,</span> <span class="mh">0x1B</span><span class="p">,</span> <span class="mh">0xA6</span><span class="p">,</span> <span class="mh">0xA6</span><span class="p">,</span> <span class="mh">0xE3</span><span class="p">,</span> <span class="mh">0x9A</span><span class="p">,</span> <span class="mh">0x98</span><span class="p">,</span> <span class="mh">0xCF</span><span class="p">,</span>
<span class="mh">0x35</span><span class="p">,</span> <span class="mh">0x81</span><span class="p">,</span> <span class="mh">0x2C</span><span class="p">,</span> <span class="mh">0x80</span><span class="p">,</span> <span class="mh">0x68</span><span class="p">,</span> <span class="mh">0xC7</span><span class="p">,</span> <span class="mh">0xFC</span><span class="p">,</span> <span class="mh">0x5F</span><span class="p">,</span> <span class="mh">0x7A</span><span class="p">,</span> <span class="mh">0xE8</span>
<span class="p">};</span>
</pre></div>
<p>As I was already there in firmware 3.50, I had a chance to test PSGrade. I compiled the sources for my Teensy 2.0 board and wrote the hex files to it. Using the same PSJailbreak methods I booted the PS3 with the Teensy board in the usb port and rebooted after the hack got executed (Teensy LED will glow). It booted into the service mode quickly and I got a screen as shown below.</p>
<p><img alt="Service Mode" src="http://www.ps3hax.net/wp-content/uploads/2010/12/SERVICE.jpg" /></p>
<p>For exiting from the service mode you need to copy a special file (Lv2diag.self) to a usb storage drive and boot the PS3. Otherwise your PS3 will still be on service mode.</p>
<p>If you wish to downgrade the firmware (which I haven't performed yet) follow these downgrade steps from <a href="http://psgroove.com/content.php?501-PS3Yes-Release-Free-PSGRADE-Downgrade-Hex-Works-on-All-AT90usb162-Boards">here</a>.</p>Cricinfo api (unofficial) for Python2010-11-30T00:00:00+5:30http://foobarnbaz.com/2010/11/30/cricinfo-api-for-python<p>Recently I started developing a Python <a href="https://github.com/semk/cricinfo">library</a> for accessing live information from <a href="http://espncricinfo.com">ESPN Cricinfo</a> like live scores, innings details and player profiles. Currently, it only fetches live match information using a simple Python iterator.</p>
<div class="codehilite"><pre><span></span><span class="c1"># instantiate</span>
<span class="n">matches</span> <span class="o">=</span> <span class="n">CricInfo</span><span class="p">()</span>
<span class="c1"># iterate though matches</span>
<span class="k">for</span> <span class="n">match</span> <span class="ow">in</span> <span class="n">matches</span><span class="p">:</span>
<span class="c1"># match title</span>
<span class="k">print</span> <span class="n">match</span><span class="o">.</span><span class="n">title</span>
<span class="c1"># a short desciption for the match</span>
<span class="k">print</span> <span class="n">match</span><span class="o">.</span><span class="n">description</span>
<span class="c1"># url to live scorecard</span>
<span class="k">print</span> <span class="n">match</span><span class="o">.</span><span class="n">link</span>
<span class="k">print</span> <span class="n">match</span><span class="o">.</span><span class="n">guid</span>
</pre></div>
<p>I wish to add features for every match like full scorecard, innings details etc in a similar manner. A plan for getting player profile is also in my mind. I'd like to make this as human-readable as possible. For example, innings information can be fetched by an iteration over the Match object. </p>
<div class="codehilite"><pre><span></span><span class="c1"># getting innings info</span>
<span class="k">for</span> <span class="n">innings</span> <span class="ow">in</span> <span class="n">match</span><span class="p">:</span>
<span class="k">print</span> <span class="n">innings</span><span class="o">.</span><span class="n">score</span>
<span class="k">print</span> <span class="n">innings</span><span class="o">.</span><span class="n">current_batsmen</span>
<span class="k">print</span> <span class="n">innings</span><span class="o">.</span><span class="n">overs</span>
<span class="k">print</span> <span class="n">innings</span><span class="o">.</span><span class="n">wickets</span>
</pre></div>
<p>I'll update about the changes being done for the api. So keep reading my blog. I need your suggestions to make it a useful one.</p>Ubuntu Maverick Meerkat2010-11-07T00:00:00+5:30http://foobarnbaz.com/2010/11/07/ubuntu-maverick-meerkat<p>I installed the new <a href="http://www.ubuntu.com/desktop">Ubuntu 10.10 Maverick Meerkat</a> on my Laptop and on my Desktop at home. Now both systems doesn't have Windows in it. My family is now very comfortable with Ubuntu and they use the Ubuntu machine for working with <a href="http://www.openoffice.org">OpenOffice</a>, watching movies, playing <a href="http://2dboy.com/games.php">games</a> and for using Internet.</p>
<p>BTW, I really like the new Ubuntu font which gives Ubuntu a brandish look. I mean like Mac :-)</p>Borg Pattern2010-10-06T00:00:00+5:30http://foobarnbaz.com/2010/10/06/borg-pattern<p>Singleton Design Patterns create all sorts of problems as you have exactly one instance for the singleton class throughout the program.</p>
<div class="codehilite"><pre><span></span><span class="c1"># Singleton implementation using new-style classes</span>
<span class="k">class</span> <span class="nc">Singleton</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">__new__</span><span class="p">(</span><span class="nb">type</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="s1">'_the_instance'</span> <span class="ow">in</span> <span class="nb">type</span><span class="o">.</span><span class="n">__dict__</span><span class="p">:</span>
<span class="nb">type</span><span class="o">.</span><span class="n">_the_instance</span> <span class="o">=</span> <span class="nb">object</span><span class="o">.</span><span class="n">__new__</span><span class="p">(</span><span class="nb">type</span><span class="p">)</span>
<span class="k">return</span> <span class="nb">type</span><span class="o">.</span><span class="n">_the_instance</span>
<span class="k">class</span> <span class="nc">Foo</span><span class="p">(</span><span class="n">Singleton</span><span class="p">):</span>
<span class="k">pass</span>
<span class="o">>>></span> <span class="n">foo</span> <span class="o">=</span> <span class="n">Foo</span><span class="p">()</span>
<span class="o">>>></span> <span class="n">bar</span> <span class="o">=</span> <span class="n">Foo</span><span class="p">()</span>
<span class="o">>>></span> <span class="nb">id</span><span class="p">(</span><span class="n">foo</span><span class="p">),</span> <span class="nb">id</span><span class="p">(</span><span class="n">bar</span><span class="p">)</span>
<span class="p">(</span><span class="mi">10049912</span><span class="p">,</span> <span class="mi">10049912</span><span class="p">)</span>
</pre></div>
<p>Usually programmers use Singleton Patterns as a global entry point to database connections. This is a bad programming habit. This will also break tests as most unittests use dummy implementations of real objects that emulate real objects. Suppose you have protected your class to instantiate only once using this pattern, then it would become impossible for him to stub/mock this class for tests. In short, Singleton patterns are not recommended for test driven development.</p>
<p>What we really care about class objects are its identity/state and behaviour, not the number of instances; unless you really need that kind of implementation. This is where Borg pattern comes as useful. Borg pattern share the same state across all its class instances. Its fairly easy to implement a Borg pattern in python. Just initialize the class <code>__dict__</code> with a class attribute in its <code>__init__</code>.</p>
<div class="codehilite"><pre><span></span><span class="c1"># Borg Pattern</span>
<span class="k">class</span> <span class="nc">Borg</span><span class="p">:</span>
<span class="n">__shared_state</span> <span class="o">=</span> <span class="p">{}</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">__dict__</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">__shared_state</span>
</pre></div>
<p>Subclass this <em>Borg</em> class and you can have a shared-state class implementation and all the instances will have the same state. For more information about different patterns in python, checkout this <a href="http://www.suttoncourtenay.org.uk/duncan/accu/pythonpatterns.html">link</a>.</p>A commandline mapper2010-10-02T00:00:00+5:30http://foobarnbaz.com/2010/10/02/commandline-mapper<p>Python provides a builtin <code>map</code> function which applies a method over a list of entities. This function comes handy in a lot of situations as in</p>
<div class="codehilite"><pre><span></span><span class="c1"># find the square of all integers in a list</span>
<span class="c1"># Eg: </span>
<span class="c1"># input: [1, 2, 3, 4]</span>
<span class="c1"># return: [1, 4, 9, 16]</span>
<span class="nb">map</span><span class="p">(</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="o">*</span><span class="n">x</span><span class="p">,</span> <span class="n">list_of_integers</span><span class="p">)</span>
</pre></div>
<p>Similar functionality can be achieved in linux commandline using a combination of unix pipe <code>|</code> and <code>xargs</code> command. For my previous blog post I needed to resize the image sizes so that it fits correcly in the post. This is how I mapped the <code>convert</code> utility over a list of image files in a directory.</p>
<div class="codehilite"><pre><span></span>$ find . -name <span class="s2">"*.jpg"</span> -print0 <span class="p">|</span> xargs -0 -I img convert -resize 600x450 img img
</pre></div>
<p>The <code>-print0</code> option for find list files without the EOF marker so that it can be used efficiently in <code>xargs</code>. The -0 option indicates this. You can use a replace string similar to <code>img</code> used here, to replace initial arguments of the command from standard input. See <code>man xargs</code> for more information about this utility.</p>Hacking PlayStation 3 with Teensy 2.02010-10-02T00:00:00+5:30http://foobarnbaz.com/2010/10/02/hacking-playstation3-with-teensy<p>I've always been a proud and loyal owner of a <a href="http://en.wikipedia.org/wiki/PlayStation_3">Playstation 3</a> and still I consider it as the most powerful game console of this generation. But things were super-cool back on the days when <a href="http://en.wikipedia.org/wiki/PlayStation_3">Playstation 3</a> supported <a href="http://en.wikipedia.org/wiki/OtherOS">OtherOS</a> by which you could boot and use Linux on your console. This was the very feature (other than GameOS which is why someone should buy a PS3 :-)) every PS3 owner was proud of when comparing his console to its counterparts. This feature was removed by Sony after <a href="http://en.wikipedia.org/wiki/George_Hotz">Geohot's</a> attempts to hack <a href="http://en.wikipedia.org/wiki/PlayStation_3">Playstation 3</a> using Linux. Although he was partially succeeded in doing so, soon he disappeared from the PlayStation hacking scene and asked other hackers to continue his work. The PS3 users gained nothing from this untill the mighty <a href="http://psjailbreak.com">PSJailbreak</a> was announced.</p>
<p>I must say, Sony guys have made <a href="http://en.wikipedia.org/wiki/PlayStation_3">Playstation 3</a> almost unhackable as it took more than 4 years for the console hackers to find an exploit to allow homebrews on it. This exploit which was found in firmware 3.41, allows users to backup <em>Legitimate</em> Copies of PlayStation Games to internal/external harddisk. This is achieved by installing an application called Backup Manager. The team to announce this hack was <a href="http://psjailbreak.com">PSJailbreak</a>, who used a USB dongle to boot from which took advantage of the exploit found in 3.41. Later open source implementations of <a href="http://psjailbreak.com">PSJailbreak</a> like <a href="http://github.com/psgroove/psgroove">psgroove</a>, psfreedom etc. came which used popular development boards like <a href="http://www.pjrc.com/store/teensypp.html">Teensy ++</a> and <a href="http://www.atmel.com/dyn/products/tools_card.asp?tool_id=3879">Atmel AT90USBKEY</a>.</p>
<p>I was really excited to see this progress and I liked the idea of backing up games. If you buy Original Games for $50, atleast there should be one way to make backups so that you can play games even if yor media is damaged. So I logged on to <a href="http://pjrc.com">pjrc</a> and ordered one <a href="http://www.pjrc.com/teensy/">Teensy 2.0 USB Development Board</a> which used <a href="http://www.atmel.com/dyn/products/product_card.asp?part_id=4317">Atmel ATMEGA32U4 AVR</a>. It took almost 3 weeks to ship to India from US. I checked out <a href="http://github.com/psgroove/psgroove">psgroove</a> code from GitHub and used the Teensy Loader utility to flash the compiled hex code to the board.</p>
<p>For this jailbreak to work, you need to hard power your PS3 after connecting the usb board. Turn on the console and immediately press eject button for the exploit to work. If everything goes proper, the orange led in Teensy board will light up. Tada! The PlayStation is Jailbroken. Take a look at the snaps I took during Jailbreaking my console.</p>
<p><img alt="Flashed Teensy" src="/images/posts/2010-10-02-hacking-playstation3-with-teensy/teensy_loaded.jpg" /></p>
<p><img alt="Exploited" src="/images/posts/2010-10-02-hacking-playstation3-with-teensy/exploited.jpg" /></p>
<p><img alt="Installing" src="/images/posts/2010-10-02-hacking-playstation3-with-teensy/installing.jpg" /></p>
<p><img alt="Backup Manager" src="/images/posts/2010-10-02-hacking-playstation3-with-teensy/backup_manager.jpg" /></p>
<p><img alt="Backup Manager Page" src="/images/posts/2010-10-02-hacking-playstation3-with-teensy/bm_screen.jpg" /></p>
<p><img alt="Backing up Uncharted" src="/images/posts/2010-10-02-hacking-playstation3-with-teensy/backing_up.jpg" /></p>
<p><img alt="FTP Server" src="/images/posts/2010-10-02-hacking-playstation3-with-teensy/ftp_server.jpg" /></p>
<p>The latest firmware update from Sony patches this exploit and even they are banning consoles using Backup Manager when they connect to <a href="http://en.wikipedia.org/wiki/PlayStation_Network">PSN</a>. As I love <a href="http://en.wikipedia.org/wiki/PlayStation_Network">PSN</a> and playing <a href="http://en.wikipedia.org/wiki/Uncharted_2:_Among_Thieves">Uncharted 2</a> online, soon I have to update my firmware to 3.50. Even legitimate users are irritated by this move by Sony. If iPhone Jailbreak is legal, this is also legal. Right ?</p>
<p>Had Sony not removed <a href="http://en.wikipedia.org/wiki/OtherOS">OtherOS</a>, none of this would have happened OR atleast it would have taken more time for the hackers to hack the console. "Sony, never do stupid things like this anymore. Please bring back <a href="http://en.wikipedia.org/wiki/OtherOS">OtherOS</a>."</p>GitHub powered blogging2010-09-23T00:00:00+5:30http://foobarnbaz.com/2010/09/23/github-powered-blogging<p>I have learned a big lesson from losing all the blog archives from my previous blog <a href="http://semk.in">semk.in</a>, that never ever host a blog without any backups. I didn't pay any attention to domain/hosting expiration date since I was busy working at that time. But finally everything was deleted in a matter of seconds. I never blamed the hosting provider for this. They kept mailing me all the day. Recently I came to know about using <a href="http://github.om">Github</a> + <a href="http://github.com/mojombo/jekyll">Jekyll</a> for blogging from <a href="http://pages.github.com/">Github Pages</a> which I think is the best way to maintain the website code. Thats why this new site got spawned up. </p>
<p>This website layout is a fork of <a href="http://tatey.com">Tate Johnson's site</a> with some minor modifications. Stay tuned for interesting stuffs ...</p>
<p>Thanks,</p>
<p>Sreejith K</p>