Nasuni

Yup, still running Python and Django! We’re actually pretty proud to be a sponsor for DjangoCon 2010 coming up in September – I’ll be attending, so I hope to see all the familiar Django faces I know, and meet some new ones.

I’ve been blogging semi-regularly for the Nasuni blog itself – my posts are focused on product-things more than anything else. Here’s a small list of posts which I’ve done:

• The Road to Release – Feature Previews – this is actually my latest one, and the first in a series where I’ll be showing off some of the new features we’re adding in the latest release.
• Looking at OpenStack, a Rackspace and NASA initiative – For those of you who don’t know, Rackspace and NASA announced OpenStack – the awesome part? It’s all python – I had the swift component (which powers Rackspace’s cloudfiles system) of OpenStack running pretty quickly. I’d recommend snagging the code from launchpad and taking a look. Swift (the storage component) uses eventlet – and Nova (the compute part) uses Tornado and Twisted.
• Storage Switzerland Test Drives the Filer – This is a response to an article written about the product – I actually used it to preview some of the work going into the next release of the Filer.
• Thanks to Django – This piece goes into some detail about our use of Django, it’s one of our ways of saying thanks. I still need to rework it so we can send it over for the Django Success Stories page.
• Thanks to the Supporting Cast – This is an earlier thank you post – but to the other people who have helped out a ton, including Greg Newman, Lincoln Loop, and Revsys.
• The Donut Solution – This was a fun one, mainly to show that yes – we’re listening hard to customer feedback, and we’re improving/iterating quickly. Also, I get to show off UI improvements.
• Finally – The Nasuni Blog team – this is the rosetta stone for the authors of the blog, describing who we are. I didn’t write this piece, but it’s good reading to figure out who is who.

If you’re interested in Nasuni – or cloud storage in general – I’d encourage you to sign up for the RSS feed. We’re trying to keep the information useful outside of “just us” (despite my urge and predilection to churning out completely product-related posts) – and if you ever have feedback, drop us a line.

PSF Sponsored Sprints

The project continues on – we’ve funded two sprints so far, and have several more coming down the pike. We’re always in need of volunteers to help us do things like the manuals and site maintenance/content authoring. Here’s some highlights:

• The call for applications is open – The call for applications is open – and now I suspect we won’t be closing it. Originally, I thought we’d have to do things in waves of apply-approve. As time has progressed, I no longer think this is the case.
• Montreal Python Packaging sprint wrap up – the wrap up report for our first sprint!
• Europython core sprint report - another wrap up report for the core sprint we provided funds to.
• Just added the locations page – we now have people/companies offering up space for sprinters! Check it out!
• Finally - Sprints at PyOhio – PyOhio is going on this weekend, if you’re in the area you should really go check it out! Catherine has gone above and beyond with the entire “become a contributor” effort going on.

Please! If you’re thinking about holding a sprint - send us an application! Heck, even if we’re not sponsoring it, we’ll help promote you via the blog, and the sprint calendar we have up. A little fact? The sprints we’ve funded so far, and that are on deck for funding are all outside of the US, which is both awesome, and surprising!

PSF Board

Some of you probably know that I’m currently on the board of directors for the PSF – things progress well here, but I mainly wanted to call out the excellent blog Doug Hellmann has been authoring for PSF news. You should really be watching that because yes – we do do things, and hopefully over the next year, we’ll be doing more awesome things.

I’ve actually got a bigger post in the works for what I think the ultimate mission of the PSF is/should be as well as “how do you get money from us” as well. Must find the time!

Even though I’ve spent more time then not on “the other side” of the table (Developer, noun – “focus on making software (ergo, bugs)”) I find that James’ words ring pretty loudly for me still, especially his part on risk analysis:

I am thankful that the vast majority of bugs that affect entire user populations are generally nuisance-class issues. These are typically bugs concerning awkward UI elements or the occasional misfiring of some feature or another where workarounds and alternatives will suffice until a minor update can be made. Serious bugs tend to have a more localized effect. True recall class bugs, serious failures that affect large populations of users, are far less common. Testers can take advantage of the fact that not all bugs are equally damaging and prioritize their effort to find bugs in the order of their seriousness. The futility of finding every bug can be replaced by an investigation based on risk.

I’d recommend James’ post amongst the others there on that blog – it reminded me of an old rant of mine “The cost of (not) testing software“. Anyone in the business of making something is also in the business of making bugs. It’s important for us to keep that in mind when we deal with our day to day job – and when we think about our customers. It’s also important for us to keep that in mind when criticizing or dragging any person or company or code through the muck.

Tonight, I “officially accepted” the PEP Brian’s put a lot of work into. You can check out the PEP itself, or play with the code over here. The credit goes to Brian, and the small army of people who gave him feedback including Jeffery Yasskin who spent a fair amount of time iterating and expounding on things.

Below, is a snippet from one of my original emails to python-dev on my reasoning for supporting the library, and also adding in the “concurrent” namespace.

Baloney. A young library providing some syntactic sugar which uses
primitives in the standard library to implement a common pattern is
fine for a PEP. We’ve hashed this out pretty heavily on the stdlib-sig
list prior to bringing it here. By the same argument, we should shunt
all of the recent unittest changes and improvements into space, since
golly, other people did it, why should we.

This is something relatively simple, which I would gladly add in an
instant to the multiprocessing package – but Brian’s one-upped me in
that regard and is providing something which works with both threads
and processes handily. Take a look at multiprocessing.Pool for example
- all that is some sugar on top of the primitives, but it’s good
sugar, and is used by a fair number of people.

Let me also state – “my” vision of where futures would live would be
in a concurrent package – for example:

from concurrent import futures

The reason *why* is that I would like to also move the abstractions I
have in multiprocessing *out* of that module, make them work with both
threads and processes (if it makes sense) and reduce the
multiprocessing module to the base primitive Process object. A
concurrent package which implements common patterns built on top of
the primitives we support is an objectively Good Thing.

For example, how many of us have sat down and implemented a thread
pool on top of threading, I would hazard to say that most of us who
use threading have done this, and probably more than once. It stands
to reason that this is a common enough pattern to include in the
standard library.

In any case; consider me a strong +1 to adding it.

With that done, I am back to being only the benevolent dictator of my lawn, which is mostly dead.

Also, for added fun – check out the PyCon AU talk Brian did!

This is great news; most of all, I’m looking forward to picking through the collective knowledge of their porting effort to see if there information which can be given/provided to other porting efforts. You can see their Python3 porting notes here.

And a quick plug – the PSF Sprints project will help fund sprints focused on porting libraries to Python 3 – go here.

Awhile ago, I started wondering if there was a way that the Python Software Foundation (the PSF) could reach out, and help encourage python development a little bit more than it currently does (it actually does a lot of stuff). I then spent a fair amount of time thinking about core contribution, and ways of getting people not just to become contributors to say, python-core, but also to the “ecosystem” of big-P-Python. It’s part of my job now that I volunteered to be on the PSF Board.

Then, a few weeks ago, I asked the question – “Why aren’t you contributing (to python)?” – that post alone got my little blog here about 30,000 hits in around 48 hours. Pretty impressive for navel gazing (not even controversial navel gazing, at that). In the days following that post, I’ve had a lot of public and private discussions with people about various ways to help improve things, what could work, what wouldn’t work, etc.

One thing has been stuck in my head – sprints – sprints are places where a bunch of programmers and hackers of all skill levels, sit in a room and work on a particular code domain, it could be a website, it could be python bugs, it could be a module, library, or anything, really. The point is getting a bunch of people together in a room, feed them, water them and enable the hacking.

And finally – there’s Python 3.

Python 3 is a bit of an interesting animal – and while it is the future of the language, and a much brighter one at that, the rate at which people are porting to Python 3 is a little slower than many of us would like. Porting libraries to Python 3 can be relatively simple, but it can also be boring, tedious and require mentorship and guidance.

Given these itches of mine – I’ve decided to try to find (or at least, start the formulation) of a cure. A few weeks ago, I approached the rest of the Python Software Foundation board, and presented them with a “sponsored sprint” proposal. The short version being that the PSF would spin up a committee which would accept and evaluate sprint proposals from the world as a whole, and give each sprint a relatively set amount of money.

The sprints themselves are relatively focused – Python-core, the Python website, docs, bugs – and finally, porting Python 2 libraries to Python 3.

So – all of that said, I’m officially publicly announcing the new program. I’m still working on initial committee membership, and putting together things like the mailing list, but I will soon be announcing a call for applications. Below, you will find the full proposal – please feel free to post comments, or send me private email (jnoller at python dot org) about questions or concerns. If you want to help – please please contact me – I can not possibly make this a success on my own.

We all know that the fact is that there are plenty of areas for anyone (both programmer and not) to contribute to the Python project/ecosystem, this is actually true for most, if not all, open source, and open communities out there. While I am looking at this through the myopic lens of Python-core, the same discussion could be applied anywhere.

In the discussions I’ve had on mailing lists, and in private conversations with people about why they don’t contribute, I’ve found a set of recurring issues cited by people who could be contributors, roughly speaking these are:

• Don’t know how (documentation issue)
• Don’t know where (site layout, organization issue)
• Don’t want to fight/argue with the “but this is the way we’ve always done it” people (perception issue)
• For many issues, see checkout (svn), build (learn a “new” build system), run tests/write tests, generate patch, file bug, defend bug as a “high bar” (process/tool issue)
• Don’t feel that they’ll be valued (perception issue)
• Can’t invest the time to learn the tool chain(s) (make, ReST, regrtest, svnmerge/svndiff, sphinx, etc) (I got nothin here)

And so on.

What interests me more than the things I’ve listed about (which is a distillation of months of discussions) is what, if anything, prevents you, my reader(s) from contributing to the following areas:

• Filing bug reports (including the stdlib and documentation)
• Filing patches for bugs/improvements (including the stdlib and documentation)
• Fixing issues/sending patches on website content
• Proposing new website or documentation content
• Contributing to website maintenance
• Contributing to the Documentation

And anything else you can think of. What I’m asking for, is for you to give me thoughts, ideas and feedback on how we can lower the bar and reduce the friction of any of this so that you too can help out. Everything is fair game, even “I find the method of arguing for PEPs to cause lower intestinal irritation” (although, I hope to get more constructive comments than that).

Here’s the reason I’m asking/reaching out – I firmly, and totally believe that every single person who uses Python today can become a contributor to the project and ecosystem beyond that of a user. While it is true that philosophically, everyone who uses Python is contributing, I would like to see if we can make it so that even the most “basic” user feels that they can file a bug, submit a patch, fix the documentation, fix/add website content, etc.

Much like I think anyone can, and should be empowered to contribute back to the “Python Project” – I believe the culture of “everyone contributes” is an important one. Citing the wonderful keynote from PyCon 2010 by Antonio Rodriguez (video here), I maintain this is should be a goal for everyone, not just companies – but open source projects as well.

There are lots of things we can change/fix/improve – but before we can do that, we have to be able to identify the “bugs” in what we have now. Do we need a giant flowchart with flashing text (I joke!) or do we need simplified “how do I…” text to the website? Everything’s fair game (except “just put it on github”).

So, do you have any thoughts? If you don’t feel comfortable airing them publicly, you can email comments to jnoller at (gmail.com|python.org) as you see fit. I promise not to share any identifiable information, but I reserve the right to share the feedback you provide, albeit anonymously. I also promise to ask meaningful questions so I can squirrel out more detail if needed.

So – my goal of the lightning talk was to say thank you to the python community, not only from me, but from the point of view of the company I work for (Nasuni) which recently launched. Thanks can be given back to the community in a variety of ways – money, code, sponsorship, programs, etc.

Sometimes, a simple Thank You can be a good start, and that was my goal. Unfortunately, I also completely deleted my slides (intentionally, what was I thinking?) after my talk was done.

What is Nasuni? We’re pretty small – obviously much smaller when starting (5 core engineers). The goal was to build a simple to manage, secure, reliable cloud-storage backed virtual storage appliance. Everyone helped code and design the product. We had to build it fast – when you’re a startup on VC money, you’ve got to be fast – you have finite time. Limited time, money and people.

From a technology standpoint – it simply made sense to put the core data path in C and C++ – you need speed, security, and the compactness C offers you when you’re talking speed. Not to mention, this is a device which has a finite amount of space, so keeping it relatively compact is a Good Thing. So the brain and central nervous system are in C – but we’re lacking something:

The rest of the person. D’oh. You know, arms and legs and stuff.

Let’s look at all of the other stuff we needed:

• tests
• tools/glue scripts
• prototypes
• user interface(s)
• other system daemons
• infrastructure
• web site
• deployment tools

And that’s just the stuff I could think of off the top of my head right now. So, we needed to compliment our super strong core with something pretty flexible, something that could also easily mesh with the core system as well.

When picking the technology; we needed speed of development, a broad ecosystem to build from, and it needed to ramp up and down. It had to be flexible enough that anyone involved in the project could become immediately effective with it.

Enter Python.

“Duct tape is like the force. It has a light side, a dark side, and it holds the universe together.”

Python Ramps “up”, and it ramps “down” – it’s good for “hard” programmers, and people just trying to get things done. It’s simple enough that anyone who has scripted in the past can get something done, and quickly. My motto has slowly become “there is no such thing as bad code – there’s just code that works, and code that doesn’t”. Code that comes out of someone who is just learning, and gets the job done is just as valuable as code that is well thought out and designed.

Python, and it’s broad ecosystem. It’s the glue, the face, the hands, the legs. It, plus insanely smart engineers let us move quickly, adapt to anything that popped up and put together a kick ass product.

8 months.

With Python’s help, we put together a ready-to-ship simple to manage, secure, reliable cloud-storage backed virtual storage appliance. 8 months! This isn’t a web app, or a another web service. This is something just shy of putting actual hardware into a datacenter! We’ve got python deep in our DNA. (And heck – the president of the company wrote a good chunk of it! How awesome is that?)

The moral of the story, however brief it is – when you need to sell Python to your boss, or that client – or if you’re just starting out. Don’t sell it with words. Sell it by showing it’s effectiveness, it’s speed. Just do it – that’s what Python is here for. helping you get your job done and succeed.

Python is not meant to be the prettiest, or the fastest-to-run, or the one with the latest feature du-jour. It’s there to let you get your job done. That’s why I love it, and much of the community around it – it’s the pragmatism, how will this let people be effective.

I wanted to take a moment not just to cheerlead – but to also specifically thank the projects and tools out there in the community that helped us get the job done.

• Python (duh)
• The Python standard library: This can not be stressed enough! Despite anything else we use from the ecosystem, the standard library is rich enough that it’s a massive time, and life saver. This is why I don’t believe that “getting rid of it” makes any sense at all. Python’s success is not just a by product of the language itself – the standard library is such a big selling point it can’t be stated enough.
• Django
• Virtualenv / Virtualenvwrapper
• Nose
• pip
• fabric
• paramiko
• lxml
• pycurl

And many, many others (including obviously non-python projects). Thank you, thank you, thank you!

In closing – Thank You!. Without Python, as a language – Python as a community, Django, and so many others – we would not have been able to achieve our goals. There’s so much to be proud of as a community, and sometimes we loose sight of it while debating frameworks, this, that or the next thing. Python is the perfect secret weapon for people and companies just trying to get things done

Thank you all – hopefully we can give back as much as we’ve gotten.

The Nasuni team is an excellent one – and one I am very, very proud to be a part of. Our product is called the Nasuni Filer – a simple-to-use, versioned, encrypted and cloud-storage backed virtual NAS (network attached storage) server (click here for more information).

Without going into all of the features, our goal in making this was to make cloud storage simple, accessible and secure – and I know we’ve accomplished all three. All you do is download it, boot it and start using it – once you do so you have access to truly unlimited storage. It’s an unlimited filesystem for the cloud. Here’s the elevator pitch:

Nasuni has developed a virtual file server, called the Nasuni Filer, that delivers unlimited file storage and complete file protection for businesses. Working in partnership with leading cloud storage vendors, the Nasuni Filer leverages the vast capacity of the cloud to store and protect company files offsite, while retaining the local functionality and performance of a traditional NAS.

This technology allows businesses to use the cloud provider of their choice as a replacement for traditional primary storage. Snapshots, file versioning, and offsite storage are integrated into the file server itself – ensuring business file are safe and secure at all times. No need to manage complex backup and DR schemes – if the file server is running, files are protected.

We’ve launched the Beta of the product today – anyone can sign up, download and use it. Anyone can give us feedback and suggestions – I encourage all of you who might need something like this to download and give it a try. If you want – go check out the videos we’ve put together showcasing the Filer (and better yet – check out the awesome animated cartoon we have on the front page).

Most of you know that my blog is mainly Python oriented. Suffice it to say, Nasuni – and the Nasuni Filer make use of Python for a wide range of tasks. We use Python, Django and as much of the Python ecosystem as we can to drive everything from the website, to the GUI on the appliance itself – Python is part of the DNA of the company, and it has served us well. Without Open Source and Python – I don’t think it would have been possible to build what we have built in as little time as we have.

We have a strong dedication to not just Python, but open source in general (and a fair number of us will be at PyCon this month). As time progresses, now that we’re exiting stealth mode we plan on possibly open sourcing stuff we feel would benefit the community. Some of us already push patches back where and when we can, but as I said – as time progresses this involvement will only increase.

So not only am I proud to announce the product, be part of this team and to see what we’ve made, I’m also happy to thank so many people in the Python and OSS community which have helped us reach this point.

So go – check it out, let us know what you think.

You probably also saw the 2009Q3 testing/performance results (here), showing a hefty speed increase for various operation. The 2009Q4 will be coming soon.

What you probably don’t know – yet, is that a PEP is coming, proposing Unladen Swallow be merged back to Python-core – specifically, the Python 3k branch. Yeah, that’s right – a PEP to merge Unladen Swallow back into the mothership, but to Py3k only. Talk about a shot in the arm!

Quoting Collin Winter:

Python 3 is the natural place for something like Unladen Swallow to land: Python 3 is clearly the future, and I think improved performance will be a strong selling point in the language’s favor. What we and others have done for Python 2 applies directly to Python 3.

…

One of the things we’ve really focused on with Unladen Swallow is creating a solid foundation that the wider CPython developer community can build on top of. We’ve fixed some nasty problems in the x86-64 JIT, for example, so that the CPython community can focus on the fun stuff — more optimizations! Having the basic JIT infrastructure in place and well-tested opens up a world of opportunities that didn’t exist before.

…

I think PyPy’s focus on research is incredibly valuable. Unladen Swallow has always been about, “what can we achieve right now?” One of our guiding principles has been, Do nothing original, don’t be innovative. PyPy is looking 10 years down the road and thinking, What can we do if we question everything? I think having those different perspectives is important to the community. So, no, I don’t see us in competition (with PyPy) at all.

There’s been a lot of talk about how slow adoption of 3k has been – a backwards incompatible implementation of a widely used language will always have slow(er) adoption then releases of the same language, backwards compatible – but I’d argue that to an extent, part of the reason is that there “haven’t been enough carrots” in the Py3k cart to provide enough of an incentive for projects, libraries, etc to move to it at an accelerated rate.

This, well – this changes things. A significantly faster interpreter, a more rational GIL thanks to Antoine Pitrou – these are the improvements to Python 3 which make it the perfect target to aim for for new projects and libraries. It should drastically help provide the impetus for larger libraries to move over. Things like these are exactly why something like the moratorium make sense, efforts can be focused on the ecosystem of python outside of the syntax, on the interpreter, the standard library, etc.

This is huge; and with a little luck, a lot of discussion and debate – and a ton of work, by the end of 2010, we’ll have not only the best implementation of Python, as a language to date in Python 3 – but the best interpreter as well.

The PEP should go out for review soon, probably within the next two weeks – and more details/information will be coming out at PyCon in February (you’re coming, right?!).

Also see Michael Foord’s post about this news too.