foo, bar n baz

Voldemort: A Jinja2 powered static site generator

2011-10-05T00:00:00+5:30

Voldemort is a blog-aware static site generator inspired by Jinja2. All these times this blog was generated using Jekyll and I always wanted to use something Pythonic. Hyde was there, but the awesomeness of Jinja2 forced me to write Voldemort on my own.

Voldemort has its own advantages. You can templatize your HTML pages using Jinja and write posts using Markdown. And if you like this website, then you've got one reason to tryout Voldemort. Go for an install or grab the sources if you want to tweak it your own way.

Packaging Python Applications

2011-10-02T00:00:00+5:30

A few days ago, I came across a situation where I needed to create Debian packages for some Python libraries on which our software was dependant on. All these time we were creating and distributing the application as Eggs built using setuptools setup.py script. Later on this became a problem since some other applications which we were using were not Python applications and were packaged as .deb packages. This situation made us to build .deb packages for our Python software as well.

The real challenge here was packaging all the dependencies our Python application has brought in, as most of the dependencies were having only egg distributions. Luckily there was an extension to setuptools called stdeb which will allow you to generate debian packages using your setup.py script. It'll automatically add the dependencies in the debian/control file after searching for the dependencies using apt-cache. stdeb will do something like apt-cache dump | grep <package> to find whether there is a proper debian package for the dependencies. If they found the correct one in apt-cache, debian/control file will be updated with that information. But as I said, most of the dependencies were not packaged as .deb. So we downloaded all the dependencies using the easy_install command and created .deb packages for them using stdeb.

I will demonstrate the procedure with the Voldemort project from my GitHub Repo. The setup.py looks like this,

try:
    from setuptools import setup, find_packages
except ImportError:
    from ez_setup import use_setuptools
    use_setuptools()
    from setuptools import setup, find_packages

setup(
    name='voldemort',
    version='0.5.0',
    description='Voldemort is a simple static site generator\
                    using Jinja2 and markdown templates.',
    author='Sreejith K / K7Computing Pvt Ltd',
    author_email='sreejithemk@gmail.com',
    url='http://www.foobarnbaz.com',
    install_requires=[
        'Pygments >= 1.4',
        'PyYAML >= 3.10',
        'Markdown >= 2.0',
        'Jinja2 >= 2.5'
    ],
    setup_requires=[],
    packages=find_packages(exclude=['ez_setup']),
    test_suite='tests',
    scripts = ['scripts/voldemort'],
    include_package_data=True,
    zip_safe=True,
)

In this case all the dependencies were lacking .deb packages. So, we need to download them using easy_install.

mkdir deps
easy_install -e -b deps/ "Pygments>=1.4"
easy_install -e -b deps/ "PyYAML>=3.10"
easy_install -e -b deps/ "Markdown>=2.0"
easy_install -e -b deps/ "Jinja2>=2.5"
ls deps/
jinja2   markdown pygments pyyaml

Now you need to go to each dependency directory and issue the following command

python setup.py --command-packages=stdeb.command debianize

That will create a directory named debian which contain all the stuffs needed to create a .deb package. Now it is upto us to edit the control file and add any additional dependency if needed. To generate the debian package, do

dpkg-buildpackage -us -uc

Now you can build the main application debs by editing the control file manually since stdeb won't add the dependencies. But you know you have them in your hand :-).

Developing scalable services with Python

2011-08-30T00:00:00+5:30

Developing multi-threaded applications in python is a "Pain In The Ass". And the GIL (Global Interpreter Lock) takes away the advantage of utilizing multiple cores in a machine. It doesn't matter how many cores a CPU have, GIL prevents threads from running in multiple cores. So python programs would't get the maximum performance out of the CPU when they use threads in their services.

In many cases you might need to write services where you need to listen on a port and wait for the client connection to do some task. If multiple clients are connecting to this service simultaneously then you might need to spawn threads to handle the requests. Considering the fact that GIL introduces a performance bottleneck, the best way to solve this situation is to use python's multiprocessing capabilities. This library provides almost threading like class implementations.

Then a question might arise. How do you share a socket created by the server process to the newly spawned processes? It is possible since all the forked processes will have the parent's file descriptors. multiprocessing library already has this package named multiprocessing.reduction that provides a method reduce_handle which can serialize a socket and you can send this socket to another process using pipes. The child processes can read from the pipe and re-create the socket using rebuild_handle. The following example will make this idea clear to you.

# Main Process
from multiprocessing.reduction import reduce_handle
# serialize the socket
serialized_socket = reduce_handle(client_socket.fileno())
# send it to the child/worker process
pipe_to_worker.send(serialized_socket)

# Worker Process
from multiprocessing.reduction import rebuild_handle
# get the socket from parent
serialized_socket = pipe_from_parent.recv()
# rebuild the file descriptor
fd = rebuild_handle(serialized_socket)
# create socket from fd
client_socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)
# use the socket as usual. eg: send a message to the client
client_socket.send("Baby, I\'m so fast\r\n")

Preforking

Another way to solve this issue is by spawning multiple process from the main server which is listening on a socket/port and letting all the child processes to accept() connections from the client. Apache uses this style of process scaling known as "Preforking". A simple example using multiprocessing module which runs an instance of a BaseHTTPServer.HTTPServer on a pool of worker processes can be written very easily as follows.

#
# Example where a pool of http servers share a single listening socket
#
# On Windows this module depends on the ability to pickle a socket
# object so that the worker processes can inherit a copy of the server
# object.  (We import `multiprocessing.reduction` to enable this pickling.)
#
# Not sure if we should synchronize access to `socket.accept()` method by
# using a process-shared lock -- does not seem to be necessary.
#
# Copyright (c) 2006-2008, R Oudkerk
# All rights reserved.
#

import os
import sys

from multiprocessing import Process, current_process, freeze_support
from BaseHTTPServer import HTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler

if sys.platform == 'win32':
    import multiprocessing.reduction    # make sockets pickable/inheritable

def note(format, *args):
    sys.stderr.write('[%s]\t%s\n' % (current_process().name, format%args))

class RequestHandler(SimpleHTTPRequestHandler):
    # we override log_message() to show which process is handling the request
    def log_message(self, format, *args):
        note(format, *args)

def serve_forever(server):
    note('starting server')
    try:
        server.serve_forever()
    except KeyboardInterrupt:
        pass

def runpool(address, number_of_processes):
    # create a single server object -- children will each inherit a copy
    server = HTTPServer(address, RequestHandler)

    # create child processes to act as workers
    for i in range(number_of_processes-1):
        Process(target=serve_forever, args=(server,)).start()

    # main process also acts as a worker
    serve_forever(server)

def test():
    DIR = os.path.join(os.path.dirname(__file__), '..')
    ADDRESS = ('localhost', 8000)
    NUMBER_OF_PROCESSES = 4

    print 'Serving at http://%s:%d using %d worker processes' % \
          (ADDRESS[0], ADDRESS[1], NUMBER_OF_PROCESSES)
    print 'To exit press Ctrl-' + ['C', 'Break'][sys.platform=='win32']

    os.chdir(DIR)
    runpool(ADDRESS, NUMBER_OF_PROCESSES)

if __name__ == '__main__':
    freeze_support()
    test()

I wrote a simple wrapper for this kind of services which can be scaled. You can find it here.

Custom Authentication for Google App Engine apps

2011-07-31T00:00:00+5:30

Google App Engine is a widely used and most popular PaaS solution provided by Google. App Engine provides the developer with a wide range of apis which can be used to develop web applications using any WSGI compliant Frameworks (Webapp, Tipfy, Django, Bottle, Tornado etc.). One of the apis App Engine provides is the users api, which most of the developers confuses for an api which provides user creation, authentication for the application. But this api only authenticates Google Accounts (can be the application developer or any third-party Google Account) using OAuth. You can't really user this api to create or manage users for your application.

Remember when I told you that every application you write for GAE is a WSGI Application? WSGI is just a standard for the web application to talk to the backend HTTP server. That means a WSGI application can't run by itself. It needs an HTTP server to listen on and execute the code you have written. Its the HTTP Server which handles all the server stuffs used for authentication such as setting cookies. Now, GAE has a sandbox, which is a restrictive environment for your application code to run. For example, your application is restricted for file operations and certain modules are restricted from importing. So you can't really set up a cookie from your application code since its not an HTTP Server code. Here is our problem now. How do you do a custom authentication for a Google App Engine application?

You can achieve this by writing a middleware to your WSGI application. There are many authentication libraries available for this purpose. Popular ones are Beaker, GAE-Sessions, gaeutilities. But I liked the GAE-Sessions library better than the other ones since its the fastest of them all. GAE-Sessions use memcache/datastore to store the session information. To use this library, just copy the gaesessions directory to your application directory. The middleware for your application is as simple as shown in the code below

from gaesessions import SessionMiddleware
def webapp_add_wsgi_middleware(app):
    app = SessionMiddleware(app, cookie_key="a random and long string")
    return app

Save the above code to a file named appengine_config.py. You can use gaesessions.get_current_session() to get a dictionary to store all the session information. You can either use session.set_quick(<session-var>, <value>) method to use application memcache for storing session info. Otherwise it'll be stored to the datastore. Getting session info is as easy as session.get(<session-var>) and session.pop_quick(<session-var>) will remove the information from the session. All the dictionary like indexed operations will be persisted to the database.

from gaesessions import get_current_session
session = get_current_session()

# setting user session information
session.set_quick('user', 'authenticated_user_info')
# getting user session
user = session.get('user')
# removing session info
session.pop_quick('user')

The default session lifetime is 7 days. You may configure how long a session lasts by calling SessionMiddleware with a lifetime parameter, e.g., lifetime=datetime.timedelta(hours=2)). You can schedule a cron job for cleaning up all the expired session info for your application by creating a handler file like this.

# cleanup_sessions.py
from gaesessions import delete_expired_sessions
while not delete_expired_sessions():
    pass

Make sure you have a cron.yaml with the correct info.

cron:
- description: daily session cleanup
  url: /cleanup_sessions
  schedule: every 24 hours

You can find a complete sample application here.

Fix Parallels Desktop on Mac OSX Lion

2011-07-27T00:00:00+5:30

Most of the Parallels Desktop 6 users are experiencing an issue not finding the virtual machine windows after upgrading to Mac OSX Lion. This might be caused by the Desktop (inwhich Parallels to run on) being set to None during upgrade. Mission Control to blame? But this can be fixed very easily. Follow these steps,

Create a Dock Icon for Parallels Desktop if not
Launch Parallels Desktop
Right click the Parallels Dock Icon and go to Options. Then set Assign To to This Desktop.
Done.

According to Parallels FAQ, there is one more workaround to fix this issue,

Delete all the spaces except the default one
Reboot and restart Parallels

I hope this will be helpful for those struggling to get around this issue.

Meet Raspberry Pi, the $25 computer

2011-05-07T00:00:00+5:30

No one would have thought that they could ever buy a computer for $25. Thanks to the smart game developer from Frontier Developments, David Braben who designed Raspberry Pi, a $25 dollar computer with the size of a USB Stick, infact it is actually a USB Stick Computer. Have a look,

This tiny device comes with an HDMI port in one end and a USB port on the other. You can plug it into a HDMI socket and then connect a keyboard via the USB port turns it into a fully functioning machine running a flavour of Linux. And the Hardware Specs are not that bad at all. It boasts a 700MHz ARM11 processor coupled with 128MB of RAM and runs OpenGL ES 2.0 allowing for decent graphics performance with 1080p. Yes 1080p. You can also add an SD card to the SD card slot for your storage needs. Did I already mention that you can also mount a 12MP camera into this?

It looks like the stock OS it ships with is Ubuntu. That means you can browse, run office apps or do whatever you want to do with a computer. Braben argues that the new ICT syllabus in schools have changed a lot and no one is teaching programming and the internals of computer systems. They are now teaching other useful skills like writing documents in a word processor, how to create presentations etc. This could be his effort to bring back the good stuff back to schools by making an affordable computer for students.

Scratch - A Magic Tool for Tux Paint

2011-04-11T00:00:00+5:30

Tux Paint is a drawing software for children that provides a kids-like user interface. Tuxpaint does have some nice tools integrated in it which helps the kids to create their drawing masterpieces using a computer. Simply speaking its the GIMP for kids of 3-12 years.

But what I was excited about Tux Paint is the availability of Magic Tool Plug-in API using which you can create Magic Tools that creates some sort of graphical effect on the canvas. Tux Paint already provides some Magic Tools by default like blur, bricks etc. Tux paint is fully written in C and uses SDL for graphics programming. There are basically two kinds of tools we can create using the API. One that affects the whole canvas and one that updates a specific region of the canvas (eg. place where you drag the cursor).

To test how this API works, I actually developed a Scratch Magic Tool which when you apply, creates a scratched effect on the image (as shown above). It also produces a nice sound effect while applying the effect. Still this tool has a lot of logical bugs so that sometimes it doesn't even look like a scratch when applied over certain colors :-). Other than that, you can use this code for a reference to create your own Magic Tools. If you need documentation this presentation slide is more than enough. Once you have downloaded the Tux Paint source code go through src/tp_magic_api.h to see the all API definitions and usage.

Back in the college days, we had this subject on Computer Graphics where you need to study some popular line-drawing and circle-drawing algorithms. But there were no lab sessions for those subjects. Everyone where literally by-hearting those algorithms at that time. I would suggest all the CS faculty to conduct a lab session on Computer Graphics and ask students to implement those algorithms as a Magic Tool in Tux Paint using the Magic Tool API. It will be useful for them to see those algorithms in action. You may use my plugin or other plugins available in Tux Paint's magic/src/ directory as a reference for your development. Please do give a feedback here in the comment section when you implement some of these algorithms as a Magic Tool.

NOTE: For compilation and installation of Tux Paint from sources, follow the instructions from tp-scratch repository.

I have an'droid

2011-02-21T00:00:00+5:30

Finally my wish came true. Now I'm a proud owner of an Android phone. Its LG Optimus One, better known as LG P500. Good that it comes with Froyo and LG promises an update to Gingerbread. Here you can find the phone specs.

This one is a good choice for anyone looking for a budget Android phone (10.8K when I bought) whose battery lasts a complete day with internet on. Thanks to Shuveb for providing a review of the phone and atlast making me buy it :-P

Eventhough a Pythonista like me hates Java, the Android platform was tempting me to look at Java with some interest. So I bought Android Application Development by O'Reilly and setup a Development Environment on my laptop using Eclipse. This was so easy that I could run the Hello World program without even writing a piece of Java code. Once you create a Project, it auto-generates files needed for the application to show on the foreground and other resource files and build scripts. By default, they'll create an activity source file with an Activity name you've given. This is pretty much the Hello World program we wanted.

package com.semk.helloworld;

import android.app.Activity;
import android.os.Bundle;

public class HelloWorldActivity extends Activity {
    /** Called when the activity is first created. */
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);
    }
}

Google has done a great job in developing an easy-to-use and feature-abundant SDK for Android. The Android emulator which comes with the SDK is an engineering masterpiece. It helps the developer to run and test their applications locally before running them on the actual devices.

I'm hoping to develop some useful applications for Android while I finish reading the book. So keep an eye on my Git Repo.

HyperGAE - Use Hypertable for App Engine Datastore

2011-01-08T00:00:00+5:30

After few days of hacking on the Google App Engine SDK and ProtocolBuffers, finally I succeeded in creating a datastore driver for GAE that talks to Hypertable and stores the data there fully protocol buffer encoded. If you want to checkout this implementation head to HyperGAE repository and see files datastore_hypertable_ht.py and datastore_hypertable_thrift.py. HyperGAE basically uses two methods to connect to hypertable. Using the thrift api and using the boost-python library ht. The mentioned drivers provides these api connections to hypertable.

To run App Engine sdk with hypertable support, do

python dev_appserver.py demos/guestbook/ --use_hypertable

The above command assumes hypertable 0.9.3.4 installed and running on your machine. It uses the thrift connection by default. I need to add support for configuring hypertable options though dev_appserver.py script. Suggestions and patches are always welcome.

UPDATE: The drivers api has been modified to accommodate the new Hypertable and Thrift api changes. You must install Hypertable version 0.9.4.3 for HyperGAE to work. More updates to follow.

Building Custom Firmware for PS3

2011-01-05T00:00:00+5:30

The 27th Chaos Communication Congress revealed the Epic Programming Fail on the PlayStation 3 security. Finally it turned out that Sony's ECDSA algorithm for signing apps use a random number generator which always returns the same number (No, I'm not kidding). The fail0verflow team, who found out this fail, used the following famous xkcd comic to describe Sony's ECDSA algorithm.

fail0verflow demonstrated their work on AsbestOS, which is a bootloader for ps3 using which you can boot Linux on it. Currently this needs some wiring and soldering works but they have promised to release an easy to use tool sometime next month.

But kakaroto(the hacker behind aMsn and PL3 payload for psgroove) wasn't patient enough. He went on to develop some tools to create custom firmware for ps3. You can find the custom firmware generator code from here. Currently it works on Linux and Mac. For those who doesn't know how to use these tools, the following steps would help.

First of all checkout the tools developed by fail0verflow team and build it. You might need to install build-essential and zlib1g-dev packages if you are using Ubuntu (use sudo apt-get install).

mkdir ~/src
cd ~/src
git clone git://git.fail0verflow.com/ps3tools.git
cd ps3tools
make

Now checkout the firmware generator code from kakaroto's GitHub repository and use the official firmware update (PS3UPDAT.PUP) to create the CFW.

mkdir ~/.ps3
cd ~/.ps3
git clone https://github.com/kakaroto/ps3keys.git .
cd ~/src
git clone https://github.com/kakaroto/ps3utils.git
cd ps3utils
make
./create_cfw.sh PS3UPDAT.PUP CFW.PUP

You can rename this CFW.PUP to PS3UPDAT.PUP and copy it to PS3->UPDATE directory of your pendrive and update the ps3 from the XMB. Still, this CFW does nothing but shows the Install Application option on XMB. You still have to wait until the homebrews are signed and repackaged with the retail .pkg format to install on this firmware.

Reads for the New Year

2010-12-31T00:00:00+5:30

Now I have two new good books to read for this New Year. The classic Structure and Interpretation of Computer Programs/SICP and To Mock a Mockingbird (Raymond Smullyan), a humorous introduction to combinatory logic with the help of interesting puzzles.

SICP was used as the textbook of MIT's old introductory programming class 6.001. Later this class has been replaced by 6.01, which uses Python. It is also known as the wizard book beacause it features the image of a wizard on its cover page.

To Mock a Mockingbird contains numerous logical puzzles and by solving them, you will eventually learn some theories and concepts in combinatorics and lambda calulus, the base of Lisp programming.

Well. Afterall, its time to say good bye to 2010 and welcome 2011.

Learn web development with Flask

2010-12-25T00:00:00+5:30

I've seen many people jumping into web development with feature-complete frameworks like Django and Rails. But most of them will find it difficult to assimilate the web development concepts because of cluttered documentation (incase of Rails) or complexity (incase of Django). One might need to look at a simpler microframework to learn from scratch. And yes, Flask is the one you'd want to have a look at.

Flask is a microframework for Python based on Werkzeug, and Jinja 2. It is developed by the same guys at Pocoo who gave us Sphinx and Pygments (a Python based syntax highlighter which beautifies the codes shown in this website). If you are new to the world of web development, with Flask you'd find it easier to understand the basics concepts like sessions, cookie, templates etc.

Flask provides a more developer friendly routing mechanism. You can define the routes to handlers with the help of a simple python decorator. That means no pain-in-the-ass for defining routes and writing handlers separately. Let's have a look at a sample Hello World application in Flask. I don't think you might need an explanation to this code :-)

# Hello World in Flask
from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run()

If you want to look at some really cool applications using Flask, never hesitate to checkout the code from here.

Factory/Service mode on PlayStation 3

2010-12-12T00:00:00+5:30

Here is a happy news for those who updated to firmware >3.41 and didn't get a chance to enjoy the homwbrews for PS3. Yes, atlast we'll be able to downgrade the PS3 firmware to any version <3.50. The actual PSJailbreak team were the first team to release the downgrade. But many thanks to zAxis who created PSGrade, an opensouce version of PSDowngrade. Using this we can put the PS3 into Factory/Service mode (Downgrade) using popular development boards like Teensy and AT90USBKEY.

We should also be thankful to KaKaRaTo who reversed the PS3Yes Key and made it available for PSGrade. Here is PS3Yes key which he reversed from its hex.

0×04, 0x4E, 0×61, 0x1B, 0xA6, 0xA6, 0xE3, 0x9A, 0×98, 0xCF, 0×35, 0×81, 0x2C, 0×80, 0×68, 0xC7, 0xFC, 0x5F, 0x7A, 0xE8

You'll need to specify this key in PSGrade sources (key.c as shown below) for the service mode hack to work. But now zAxis has updated the code with this key. So all you have to do is download the souces and compile the hex for your board.

const uint8_t jig_key[20] = {
0x04, 0x4E, 0x61, 0x1B, 0xA6, 0xA6, 0xE3, 0x9A, 0x98, 0xCF, 
0x35, 0x81, 0x2C, 0x80, 0x68, 0xC7, 0xFC, 0x5F, 0x7A, 0xE8 
};

As I was already there in firmware 3.50, I had a chance to test PSGrade. I compiled the sources for my Teensy 2.0 board and wrote the hex files to it. Using the same PSJailbreak methods I booted the PS3 with the Teensy board in the usb port and rebooted after the hack got executed (Teensy LED will glow). It booted into the service mode quickly and I got a screen as shown below.

For exiting from the service mode you need to copy a special file (Lv2diag.self) to a usb storage drive and boot the PS3. Otherwise your PS3 will still be on service mode.

If you wish to downgrade the firmware (which I haven't performed yet) follow these downgrade steps from here.

Cricinfo api (unofficial) for Python

2010-11-30T00:00:00+5:30

Recently I started developing a Python library for accessing live information from ESPN Cricinfo like live scores, innings details and player profiles. Currently, it only fetches live match information using a simple Python iterator.

# instantiate
matches = CricInfo()
# iterate though matches
for match in matches:
    # match title
    print match.title
    # a short desciption for the match
    print match.description
    # url to live scorecard
    print match.link
    print match.guid

I wish to add features for every match like full scorecard, innings details etc in a similar manner. A plan for getting player profile is also in my mind. I'd like to make this as human-readable as possible. For example, innings information can be fetched by an iteration over the Match object.

# getting innings info
for innings in match:
    print innings.score
    print innings.current_batsmen
    print innings.overs
    print innings.wickets

I'll update about the changes being done for the api. So keep reading my blog. I need your suggestions to make it a useful one.

Ubuntu Maverick Meerkat

2010-11-07T00:00:00+5:30

I installed the new Ubuntu 10.10 Maverick Meerkat on my Laptop and on my Desktop at home. Now both systems doesn't have Windows in it. My family is now very comfortable with Ubuntu and they use the Ubuntu machine for working with OpenOffice, watching movies, playing games and for using Internet.

BTW, I really like the new Ubuntu font which gives Ubuntu a brandish look. I mean like Mac :-)

Borg Pattern

2010-10-06T00:00:00+5:30

Singleton Design Patterns create all sorts of problems as you have exactly one instance for the singleton class throughout the program.

# Singleton implementation using new-style classes
class Singleton(object):
    def __new__(type):
        if not '_the_instance' in type.__dict__:
            type._the_instance = object.__new__(type)
        return type._the_instance

class Foo(Singleton):
    pass

>>> foo = Foo()
>>> bar = Foo()
>>> id(foo), id(bar)
(10049912, 10049912)

Usually programmers use Singleton Patterns as a global entry point to database connections. This is a bad programming habit. This will also break tests as most unittests use dummy implementations of real objects that emulate real objects. Suppose you have protected your class to instantiate only once using this pattern, then it would become impossible for him to stub/mock this class for tests. In short, Singleton patterns are not recommended for test driven development.

What we really care about class objects are its identity/state and behaviour, not the number of instances; unless you really need that kind of implementation. This is where Borg pattern comes as useful. Borg pattern share the same state across all its class instances. Its fairly easy to implement a Borg pattern in python. Just initialize the class __dict__ with a class attribute in its __init__.

# Borg Pattern
class Borg:
    __shared_state = {}
    def __init__(self):
        self.__dict__ = self.__shared_state

Subclass this Borg class and you can have a shared-state class implementation and all the instances will have the same state. For more information about different patterns in python, checkout this link.

A commandline mapper

2010-10-02T00:00:00+5:30

Python provides a builtin map function which applies a method over a list of entities. This function comes handy in a lot of situations as in

# find the square of all integers in a list
# Eg: 
#   input: [1, 2, 3, 4]
#   return: [1, 4, 9, 16]
map(lambda x: x*x, list_of_integers)

Similar functionality can be achieved in linux commandline using a combination of unix pipe | and xargs command. For my previous blog post I needed to resize the image sizes so that it fits correcly in the post. This is how I mapped the convert utility over a list of image files in a directory.

$ find . -name "*.jpg" -print0 | xargs -0 -I img convert -resize 600x450 img img

The -print0 option for find list files without the EOF marker so that it can be used efficiently in xargs. The -0 option indicates this. You can use a replace string similar to img used here, to replace initial arguments of the command from standard input. See man xargs for more information about this utility.

Hacking PlayStation 3 with Teensy 2.0

2010-10-02T00:00:00+5:30

I've always been a proud and loyal owner of a Playstation 3 and still I consider it as the most powerful game console of this generation. But things were super-cool back on the days when Playstation 3 supported OtherOS by which you could boot and use Linux on your console. This was the very feature (other than GameOS which is why someone should buy a PS3 :-)) every PS3 owner was proud of when comparing his console to its counterparts. This feature was removed by Sony after Geohot's attempts to hack Playstation 3 using Linux. Although he was partially succeeded in doing so, soon he disappeared from the PlayStation hacking scene and asked other hackers to continue his work. The PS3 users gained nothing from this untill the mighty PSJailbreak was announced.

I must say, Sony guys have made Playstation 3 almost unhackable as it took more than 4 years for the console hackers to find an exploit to allow homebrews on it. This exploit which was found in firmware 3.41, allows users to backup Legitimate Copies of PlayStation Games to internal/external harddisk. This is achieved by installing an application called Backup Manager. The team to announce this hack was PSJailbreak, who used a USB dongle to boot from which took advantage of the exploit found in 3.41. Later open source implementations of PSJailbreak like psgroove, psfreedom etc. came which used popular development boards like Teensy ++ and Atmel AT90USBKEY.

I was really excited to see this progress and I liked the idea of backing up games. If you buy Original Games for $50, atleast there should be one way to make backups so that you can play games even if yor media is damaged. So I logged on to pjrc and ordered one Teensy 2.0 USB Development Board which used Atmel ATMEGA32U4 AVR. It took almost 3 weeks to ship to India from US. I checked out psgroove code from GitHub and used the Teensy Loader utility to flash the compiled hex code to the board.

For this jailbreak to work, you need to hard power your PS3 after connecting the usb board. Turn on the console and immediately press eject button for the exploit to work. If everything goes proper, the orange led in Teensy board will light up. Tada! The PlayStation is Jailbroken. Take a look at the snaps I took during Jailbreaking my console.

The latest firmware update from Sony patches this exploit and even they are banning consoles using Backup Manager when they connect to PSN. As I love PSN and playing Uncharted 2 online, soon I have to update my firmware to 3.50. Even legitimate users are irritated by this move by Sony. If iPhone Jailbreak is legal, this is also legal. Right ?

Had Sony not removed OtherOS, none of this would have happened OR atleast it would have taken more time for the hackers to hack the console. "Sony, never do stupid things like this anymore. Please bring back OtherOS."

GitHub powered blogging

2010-09-23T00:00:00+5:30

I have learned a big lesson from losing all the blog archives from my previous blog semk.in, that never ever host a blog without any backups. I didn't pay any attention to domain/hosting expiration date since I was busy working at that time. But finally everything was deleted in a matter of seconds. I never blamed the hosting provider for this. They kept mailing me all the day. Recently I came to know about using Github + Jekyll for blogging from Github Pages which I think is the best way to maintain the website code. Thats why this new site got spawned up.

This website layout is a fork of Tate Johnson's site with some minor modifications. Stay tuned for interesting stuffs ...

Thanks,

Sreejith K