Unified Python Planet

complete genetic algorithm framework written in pure python · significant

    $ git checkout some_repo
    $ cd some_repo/
    $ mkvirtualenv project_name

    $ workon project_name

$ virtualenv --distribute ENV
$ python bootstrap.py --distribute

    @tag(register, [Constant("for"), Variable(), Optional([Constant("as"), Name()])]):
    def example_tag(context, val, asvar=None):

    {% example_tag for variable %}
    {% example_tag for variable as new_var %}

scypi.io.netcdf
  from zope.interface import implements

  from twisted.cred.portal import IRealm
  from twisted.web.static import File

  class PublicHTMLRealm(object):
      implements(IRealm)

      def requestAvatar(self, avatarId, mind, *interfaces):
          if IResource in interfaces:
              return (IResource, File("/home/%s/public_html" % (avatarId,)), lambda: None)
          raise NotImplementedError()

  from twisted.cred.portal import Portal
  from twisted.cred.checkers import FilePasswordDB

  portal = Portal(PublicHTMLRealm(), [FilePasswordDB('httpd.password')])

  from twisted.web.guard import DigestCredentialFactory

  credentialFactory = DigestCredentialFactory("md5", "example.org")

  from twisted.web.guard import HTTPAuthSessionWrapper

  resource = HTTPAuthSessionWrapper(portal, [credentialFactory])

  cache()

cache()

from zope.interface import implements

from twisted.cred.portal import IRealm, Portal
from twisted.cred.checkers import FilePasswordDB
from twisted.web.static import File
from twisted.web.resource import IResource
from twisted.web.guard import HTTPAuthSessionWrapper, DigestCredentialFactory

class PublicHTMLRealm(object):
    implements(IRealm)

    def requestAvatar(self, avatarId, mind, *interfaces):
        if IResource in interfaces:
            return (IResource, File("/home/%s/public_html" % (avatarId,)), lambda: None)
        raise NotImplementedError()

portal = Portal(PublicHTMLRealm(), [FilePasswordDB('httpd.password')])

credentialFactory = DigestCredentialFactory("md5", "localhost:8080")
resource = HTTPAuthSessionWrapper(portal, [credentialFactory])

def clean_directives( page ):
"""
Stupid Microsoft "Directive"-like comments!
Must remove all  sequences.  Which can be nested.
Must remove all ... sequences.  Which appear to be the nested version.
"""
if_endif_pat= re.compile(  r"(\)|()" )
context= []
start= 0
for m in if_endif_pat.finditer( page ):
   if "[if" in m.group(0):
       if start is not None:
           yield page[start:m.start()]
       context.append(m)
       start= None
   elif "[endif" in m.group(0):
       context.pop(-1)
       if len(context) == 0:
           start= m.end()+1
if start is not None:
   yield page[start:]

python pip.py install --test wsgiref

 tag. The closest Disqus supported was the
 tag which doesn't preserve white-space (useless for Python code
samples).
So I paid my US$12 (it's the only service that doesn't have a free
option) and started looking at how to import my existing comments
using their API ... and quickly found that it sucks. Comments can be
submitted but you can't specify a timestamp so they are dated with the
import date. Far from ideal. Then there's the API for retrieving
comments: it returns the data as JavaScript code (no not JSON)! It's
pretty clear that the API is what they use with the JavaScript for
Echo itself and geared for that use only. They've just thrown it out
there and documented it, warts and all.
Back to the drawing board.
The only showstopper for Disqus was the lack of 
. Everything else
about it was great: it met all my requirements and the API was clean
and comprehensive. If only there was a way to have properly formatted
source code in the comments.
Light bulb moment: use a CSS hack to make  in comments behave
like 
. The trick is to turn code into a block element and change
how white-space is handled. The CSS snippet looks like:
.dsq-comment-message code {
  display:block;
  white-space:pre;
}

Works great.
With the only blocker gone, I wrote a Python script with the help of
Ian Lewis' excellent disqus-python-client package to pull in the
existing comments from the old system. Within an hour or so it was
ready to go.
Hopefully this article saves someone else some time if they decide to
use one of these systems. Getting things running chewed up a lot more
time then I had expected.

def test_fib_of_0_is_0():
        assert fib(0) == 0
def fib(n):
        return 0
def test_fib_of_1_is_1():
        assert fib(1) == 1
def fib(n):
        return n
def test_fib_of_2_is_1():
        assert fib(2) == 1
def fib(n):
        if n == 0:
            return 0
        else:
            return 1
def test_fib_of_3_is_2():
        assert fib(3) == 2
def fib(n):
        if n == 0:
            return 0
        elif n  3:
            return 1
        else:
            return 2
def fib(n):
        if n  2:
            return n
        else:
            return n - 1
def test_fib_of_4_is_2():
        assert fib(4) == 2
def test_fib_of_5_is_5():
        assert fib(5) == 5
def fib(n):
        if n  2:
            return n
        elif n == 5:
            return 5
        else:
            return n - 1
def fib(n):
        if n = 1:
            return n
        else:
            return fib(n - 1) + fib(n - 2)
1
2

sudo rm  /tmp/mysql.sock
sudo ln  /Applications/MAMP/tmp/mysql/mysql.sock /tmp/mysql.sock
svn+ssh
class tornado {
 $tornado = "tornado-0.2"
 $url = "http://mydomain.com/download"

 $tornado_root_dir = "/opt/tornado"
 $tornado_log_dir = "/opt/tornado/logs"
 $tornado_src_dir = "/opt/tornado/$tornado"

 Exec {
  logoutput => on_failure,
  path => ["/bin", "/sbin", "/usr/bin", "/usr/sbin", "/usr/local/bin",  "/usr/local/sbin"]
 }

 file { 
  "$tornado_root_dir":
  ensure => directory,
  recurse => true,
  source =>  "puppet:///tornado/bin";
 }

 file { 
  "$tornado_log_dir":
  ensure => directory,
 }

 package {
  ["curl", "libcurl3", "libcurl3-gnutls", "python-setuptools", "python-pycurl", "python-simplejson", "python-memcache", "python-mysqldb", "python-imaging"]:
  ensure => installed;
 }

 define install_pkg ($pkgname, $extra_easy_install_args = "", $module_to_test_import) {
  exec {
   "InstallPkg_$pkgname":
   command => "easy_install-2.6 $extra_easy_install_args $pkgname",
   unless => "python2.6 -c 'import $module_to_test_import'",
   require => Package["python-setuptools"];
  }
 }

 install_pkg {
  "virtualenv":
  pkgname => "virtualenv",
  module_to_test_import => "virtualenv";

  "boto":
  pkgname => "boto",
  module_to_test_import => "boto";

  "grizzled":
  pkgname => "grizzled",
  module_to_test_import => "grizzled.os";
 }

 $oracle_root_dir = "/opt/oracle"

 case $architecture {
  i386, i686: { 
   $oracle_instant_client_pkg = "instantclient_11_2-linux-i386"
   $oracle_instant_client_dir = "instantclient_11_2"
  }
  x86_64: { 
   $oracle_instant_client_pkg = "instantclient_11_1-linux-x86_64"
   $oracle_instant_client_dir = "instantclient_11_1"
  }
 }

 package {
  ["libaio-dev", "gcc"]:
  ensure => installed;
 }

 file { 
  "$oracle_root_dir":
  ensure => directory;
 }

 exec {
  "InstallOracleInstantclient":
  command => "(cd $oracle_root_dir; wget $url/$oracle_instant_client_pkg.tar.gz; tar xvfz $oracle_instant_client_pkg.tar.gz; rm $oracle_instant_client_pkg.tar.gz; 
cd $oracle_instant_client_dir; ln -s libclntsh.so.11.1 libclntsh.so); echo $oracle_root_dir/$oracle_instant_client_dir > /etc/ld.so.conf.d/oracleinstantclient.conf; ldconfig",
  creates => "$oracle_root_dir/$oracle_instant_client_dir",
  require => File[$oracle_root_dir];
 }

 $cx_oracle = "cx_Oracle-5.0.2"
 exec {
  "InstallCxOracle":
  command => "(cd $oracle_root_dir; wget $url/$cx_oracle.tar.gz; tar xvfz $cx_oracle.tar.gz; rm $cx_oracle.tar.gz; cd $oracle_root_dir/$cx_oracle; export ORACLE_HO
ME=$oracle_root_dir/$oracle_instant_client_dir; python2.6 setup.py install)",
  unless => "python2.6 -c 'import cx_Oracle'",
  require => [Package["libaio-dev"], Package["gcc"], Exec["InstallOracleInstantclient"]];
 }

 exec {
  "InstallTornado":
  command => "(cd $tornado_root_dir; wget $url/$tornado.tar.gz; tar xvfz $tornado.tar.gz; rm $tornado.tar.gz; cd $tornado; python2.6 setup.py install)",
  creates => $tornado_src_dir,
  unless => "python2.6 -c 'import tornado.web'",
  require => [File[$tornado_root_dir], Package["python-pycurl"], Package["python-simplejson"], Package["python-memcache"], Package["python-mysqldb"]];
 }
}

import "common" 
import "tornado"

node basenode {
    include common
}

node default inherits basenode {
}

node webserver inherits basenode {
    include scribe
    include apache2
    $required_apache2_modules = ["rewrite", "proxy", "proxy_http", "proxy_balancer", "deflate", "headers", "expires"]
    apache2::module {
        $required_apache2_modules:
        ensure => 'present',
    }
    include tomcat
    include tornado
}

node "web1.mydomain.com" inherits webserver {}

import "modules"
import "nodetemplates"
import "nodes" 

from __future__ import with_statement
import os
from fabric.api import *
from fabric.contrib.files import comment, sed

# Globals

env.user = 'myuser'
env.password = 'mypass'
env.nginx_conf_dir = '/usr/local/nginx/conf'
env.nginx_conf_file = '%(nginx_conf_dir)s/nginx.conf' % env

# Environments

def prod():
    """Nginx production environment."""
    env.hosts = ['nginx1', 'nginx2']

def test():
    """Nginx test environment."""
    env.hosts = ['nginx3']

# Tasks

def disable_server_in_lb(hostname):
    require('hosts', provided_by=[nginx,nginxtest])
    comment(env.nginx_conf_file, "server %s" % hostname, use_sudo=True)
    restart_nginx()

def enable_server_in_lb(hostname):
    require('hosts', provided_by=[nginx,nginxtest])
    sed(env.nginx_conf_file, "#server %s" % hostname, "server %s" % hostname, use_sudo=True)
    restart_nginx()

def restart_nginx():
    require('hosts', provided_by=[nginx,nginxtest])
    sudo('/etc/init.d/nginx restart')
    is_nginx_running()

def is_nginx_running(warn_only=False):
    with settings(warn_only=warn_only):
        output = run('ps -def|grep nginx|grep -v grep')
        if warn_only:
            print 'output:', output
            print 'failed:', output.failed
            print 'return_code:', output.return_code

cmd = 'mv -f %(crt_egg)s %(backup_dir)s' % env

fab -f fab_nginx.py test disable_server_in_lb:web1.mydomain.com

fab -f fab_nginx.py prod disable_server_in_lb:web1.mydomain.com

fab -f fab_nginx.py prod disable_server_in_lb:web1.mydomain.com
fab -f fab_tornado.py prod deploy
fab -f fab_nginx.py prod enable_server_in_lb:web1.mydomain.com

import collections
import random

class _ChainLink(object):
    """ Internal use only.
        This is used for managing the possibilities for a next item (usually a
        word or letter) randomly through keeping track of quantity.
    """
    def __init__(self):
        self._list = []

    def add(self, item):
        """ Add an item to the chain link.
            If the item is already in the chain link, simply increment the 
            quantity counter.
        """
        for i, (qty, value) in enumerate(self._list):
            if value == item:
                self._list[i] = (qty+1, item)
                return
        self._list.append((1, item))

    def get_random(self):
        """ Retreive a random item.
            This works by using the quantity as a weight for selecting a random
            value.
        """
        total = sum(qty for qty, val in self._list)
        random_num = random.randint(0, total)
        total_pos = 0
        for qty, value in self._list:
            total_pos += qty
            if random_num <= total_pos:
                return value

class MarkovChain(collections.defaultdict):
    """ Yet another markov chain implementation.
        This one differs in the sense that it is able to better support
        huge amounts of data since the weighted randomization doesn't rely
        on duplicates.
    """

    # Sentinals 
    # Discussion here: http://stackoverflow.com/questions/1677726
    class START(object):pass
    class END(object):pass

    def __init__(self):
        collections.defaultdict.__init__(self, _ChainLink)

    def add(self, iterable):
        """ Insert an iterable (pattern) item into the markov chain.
            The order of the pattern will define more of the chain.
        """
        item1 = item2 = MarkovChain.START
        for item3 in iterable:
            self[(item1, item2)].add(item3)
            item1 = item2
            item2 = item3
        self[(item1, item2)].add(MarkovChain.END)

    def random_output(self, max=100):
        """ Generate a list of elements from the markov chain.
            The `max` value is in place in order to prevent excessive iteration.
        """
        output = []
        item1 = item2 = MarkovChain.START
        for i in range(max-2):
            item3 = self[(item1, item2)].get_random()
            if item3 is MarkovChain.END:
                break
            output.append(item3)
            item1 = item2
            item2 = item3
        return output
import markov
chain = markov.MarkovChain()

# Load the dictionary into the markov chain
dictionary = "/usr/share/dict/words"
for word in open(dictionary):
    word = word.strip()
    if word != "" and not word.endswith("'s"):
        chain.add(word)

# Let's generate 10 random pseudowords
for i in range(10):
    print "".join(chain.random_output())

Unified Python Planet

Mike C. Fletcher: Getting the old OpenGL.Tk module working again...

Paul Bissex: Branching and merging in real life

Alex Gaynor: My Workflow

Jarrod Millman: SciPy 2009 proceedings coming soon ...

Gael Varoquaux: Acceleration estimation in atom-interferometric tests of the Einstein equivalence principle

Atoms, light, gravity fields and free-fall planes

Measuring free fall, while in free fall?

Tim Golden: London Python Dojo: that was interesting…

IronPython-URLs: QCon Presentation: Real World IronPython

IronPython-URLs: A Good Mix 28: NTornado, WPF, Testing in Italian, More Benchmarking, and adodbapi

Mike C. Fletcher: Any Win64 PyOpenGL devs?

Tarek Ziade: virtualenv and zc.buildout now with Distribute included

Alex Gaynor: Towards a Better Template Tag Definition Syntax

Shannon -jj Behrens: Suicide

Python Secret Weblog: The Success of the Zope Component Architecture

What is the ZCA used for?

The ZCA is successful

Frustration with the ZCA

Conclusions

Greg Wilson: Misa Sakamoto on DB2 Technology Explorer

Logilab: First contact with pupynere

Jean-Paul Calderone: Twisted Web in 60 seconds: HTTP authentication

Patrick Stinson: An interesting thread problem

Simon Willison: Python in the Scientific World

S. Lott: BBEdit Configuration

Thomas Vander Stichele: Dedicated separate Firefox windows

Ludvig Ericson: Multitouch on Unibody MacBooks

S. Lott: Parsing HTML from Microsoft Products (Like Front Page, etc.)

Thomas Vander Stichele: Best post-sale screen ever

Ned Batchelder's blog: Continuous integration for Python with Hudson

PyPy Development: Düsseldorf Sprint Started

Graham Dumpleton: A roadmap for the Python WSGI specification.

Eric Holscher: Adding testing to pip

Python News: SciPy India Call for Papers

Malthe Borch: Emperor's new clothes

Alex Gaynor: The Pycon Program Committee and my PyCon Talk

Rene Dudfield: rakarrack is a decent effects rack for linux

Patrick Stinson: Pure Art

Menno's Musings: Why I chose Disqus Comments

Greg Wilson: The Greatest Failing of High School Science

Jason Baker: How programming language webpages should be designed

Show me the code!

Feature lists are useless, documentation is priceless

But we just met!

Gary Bernhardt: How I Started TDD

Jean-Paul Calderone: Free Agent

Kenneth Reitz: OSX + MAMP + Python + PHP + MySQL

Patrick Stinson: Sell Out

Greg Wilson: Toronto Innovations Showcase

William's Journal: Non blocking connections

Aaron Brady: Drying, with free heat

V.S. Babu: SVN client over SSH to remote Unix server from Windows

Grig Gheorghiu: Automated deployments with Puppet and Fabric

Guido van Rossum: Python in the Scientific World

Python Software Foundation: First pyArkansas an Unqualified Success

Catherine Devlin: pernicious python.org proxy problem

Jarrod Millman: SciPy India 2009 Call for Presentations

Evan Fosmark: Python Markov Chains and how to use them

Mikeal Rogers: CouchDB View Performance (Python vs JavaScript)

Improving Performance