intranation.com - Latest Entries

Using Dropbox as a Git repository

2010-02-14T10:52:53+00:00

So last month I wrote a bit about setting up your own personal Git repositories on a Linux box, and how to use that for sharing code.

I’ve had a slight epiphany since then: what if I just used the awesome Dropbox (my referral link, if you’re likely to sign up) to share Git repositories between computers? Dropbox seems able to get through most corporate firewalls (my previous empoyer blocked SSH, for example), and is very unobtrusive in its synchronisation behaviour.

Enough introductions, make with the commands

Okay, here we go. Basically, we’re just going add a new remote which points at Dropbox (in the same way the origin remote typically points at your primary external repository). Please note these instructions should be mostly *Nix agnostic—but they’re only tested on OS X.

First, create the Git repository in Dropbox (assuming your repository is named myrepo):

cd ~/Dropbox
mkdir -p repos/myrepo.git
cd !$
git --bare init

And that’s the repository created. Basically we made a bare repository in the Dropbox directory.

Now we can add the new remote to our existing repository (again, assuming it lives at ~/Projects/myrepo).

cd ~/Projects/myrepo
git remote add dropbox file://$HOME/Dropbox/repos/myrepo.git
git push dropbox master

And we’re done. We’ve created the repository, linked a Git remote to it, and pushed the master branch to the repository. This Git repository will now be available on all computers that your Dropbox account is.

Pulling from the repository

When you get to a computer that shares this Dropbox account, but hasn’t checked out Git yet, do as follows:

cd ~/Projects
git clone -o dropbox file://$HOME/Dropbox/repos/myrepo.git

Which will add your repository locally, and automatically set up a remote called dropbox which auto–merges with master.

I think this approach could be valuable for things like keeping personal documents or text files in version control (or indeed personal coding projects) without bothering to set up your own Linux box or server. Git really does make these things incredibly easy.

How to set up your own private Git server on Linux

2010-01-09T18:06:55+00:00

Update: the ever–vigilant Mike West has pointed out that my instructions for permissions and git checkout were slightly askew. These errors have been rectified.

One of the things I’m attempting to achieve this year is simplifying my life somewhat. Given how much of my life revolves around technology, a large part of this will be consolidating the various services I consume (and often pay for). The mention of payment is important, as up until now I’ve been paying the awesome GitHub for their basic plan.

I don’t have many private repositories with them, and all of them are strictly private code (this blog; Amanda’s blog templates and styles; and some other bits) which don't require collaborators. For this reason, paying money to GitHub (awesome though they may be) seemed wasteful.

So I decided to move all my private repositories to my own server. This is how I did it.

Set up the server

These instructions were performed on a Debian 5 "Lenny" box, so assume them to be the same on Ubuntu. Substitute the package installation commands as required if you’re on an alternative distribution.

First, if you haven’t done so already, add your public key to the server:

ssh myuser@server.com mkdir .ssh
scp ~/.ssh/id_rsa.pub myuser@server.com:.ssh/authorized_keys

Now we can SSH into our server and install Git:

ssh myserver.com
sudo apt-get update
sudo apt-get install git-core

...and that’s it.

Adding a user

If you intend to share these repositories with any collaborators, at this point you’ll either:

Want to install something like Gitosis (outside the scope of this article, but this is a good, if old, tutorial); or
Add a “shared” Git user.

We’ll be following the latter option. So, add a Git user:

sudo adduser git

Now you’ll need to add your public key to the Git user’s authorized_keys:

sudo mkdir /home/git/.ssh
sudo cp ~/.ssh/authorized_keys /home/git/.ssh/
sudo chown -R git:git /home/git/.ssh
sudo chmod 700 !$
sudo chmod 600 /home/git/.ssh/*

Now you’ll be able to authenticate as the Git user via SSH. Test it out:

ssh git@myserver.com

Add your repositories

If you were to not share the repositories, and just wanted to access them for yourself (like I did, since I have no collaborators), you’d do the following as yourself. Otherwise, do it as the Git user we added above.

If using the Git user, log in as them:

login git

Now we can create our repositories:

mkdir myrepo.git
cd !$
git --bare init

The last steps creates an empty repository. We’re assuming you already have a local repository that you just want to push to a remote server.

Repeat that last step for each remote Git repository you want.

Log out of the server as the remaining operations will be completed on your local machine.

Configure your development machine

First, we add the remotes to your local machine. If you’ve already defined a remote named origin (for example, if you followed GitHub’s instructions), you’ll want to delete the remote first:

git remote rm origin

Now we can add our new remote:

git remote add origin git@server.com/myrepo.git
git push origin master

And that’s it. You'll probably also want to make sure you add a default merge and remote:

git config branch.master.remote origin && git config branch.master.merge refs/heads/master

And that’s all. Now you can push/pull from origin as much as you like, and it’ll be stored remotely on your own myserver.com remote repository.

Bonus points: Make SSH more secure

This has been extensively covered by the excellent Slicehost tutorial, but just to recap:

Edit the SSH config:

sudo vi /etc/ssh/sshd_config

And change the following values:

Port 2207
...
PermitRootLogin no
...
AllowUsers myuser git
...
PasswordAuthentication no

Where 2207 is a port of your choosing. Make sure to add this so your Git remote:

git remote add origin ssh://git@myserver.com:2207/~/myrepo.git

Python virtualenv quickstart with Django

2009-12-20T21:59:00+00:00

One of the solutions to the ghetto of Python package management is to use a sand-boxed Python install like virtualenv. This is the very same technique we use at Smarkets to simultaneously run Pylons and Django alongside each other on a single server.

This post is briefly about how to get up and running in a few minutes using virtualenv and Django.

Install virtualenv

Installing virtualenv is easy on a Linux or Mac system, but the instructions that follow are Linux (Ubuntu, actually) specific. First you'll need setuptools:

sudo apt-get install python-setuptools

Then we can easy_install virtualenv:

sudo easy_install virtualenv

We need to use sudo here because it has to install to a global location. Don’t worry, this is the last time we’ll need to do something as root.

Create your virtualenv

cd to wherever it is you keep your projects (for me, in ~/src), and run:

virtualenv --no-site-packages venv

In this instance I’ve chosen venv as the name for my virtual environment. The --no-site-packages command tells virtualenv not to symlink the global site packages into my local environment, just take the Python standard library. This is important, because it helps us avoid the dependency difficulties mentioned above.

At this stage you might want to add venv to your list of ignored files, as you don’t want it to be committed to source control:

echo "venv" >> .gitignore

Installing Django

Now, the trick with virtualenv is that it creates its own Python and easy_install binaries, which means you can install/run things specifically in your environment. Let’s install Django:

./venv/bin/easy_install django

And it’s done. easy. You might also want to install the MySQL bindings and IPython for ease of use:

./venv/bin/easy_install ipython python-mysql

To start a new Django project, you’ll note that a django-admin.py file will have been installed for you in the environment:

./venv/bin/django-admin.py startproject myapp

Obviously you can skip this step if you have an existing Django project.

Running Django

Now the last step, which is probably obvious by now, is to run Django’s runserver with the virtual Python binary:

cd myapp
../venv/bin/python manage.py runserver 0.0.0.0:8000

And you’re away!

Closing

Just make sure whenever you need to add another package that you install it in the virtualenv, and not in the global packages directory. If you want to deploy using virtualenv, here are some instructions for using mod_wsgi with virtualenvs.

A week using Emacs

2009-06-29T13:04:56+00:00

From a consensus vote of two (Hunter and Jason) out of three, Emacs is the preferred editor of Smarkets.

Historically, my editor experience runs roughly thus:

Notepad (early days of learning HTML from liberally using “View source”);
Dreamweaver (college);
Homesite (post-college until 2007); and
TextMate (2007 onwards, when thankfully I’ve had a Mac).

with a little Vi(M) interspersed here and there as my server—side editor.

In an effort to fit in a bit better with the team (and because they continually bang on about it like it’s the greatest thing since the Internet), I’ve committed to trying Emacs for a week of full time development. I’ll write up a review of my thoughts at week end.

Misgivings

In the interests of disclosure, I have some misgivings about Emacs:

The lack of respect it will have for OS X–wide keyboard shortcuts. Having to re-learn keyboard commands just to cut and paste (for example), will drive me up the wall;
Using Elisp to configure it;
The auto—indent mode of Emacs doesn’t work very well with my HTML coding style. Emacs thinks it knows when/where you’d like to indent code, and pressing “tab” will guess for you. I prefer HTML to just indent 4 spaces when I tab.

See you back here in a week for the review (and hopefully solutions to common web developer problems that Emacs causes).

Development virtual machines on OS X using VMWare and Ubuntu

2009-03-22T00:04:00+00:00

Update 21st Febuary, 2010: If you get this error when installing VMWare Tools: mount: unknown filesystem type 'iso9660', it’s because you’re using Ubuntu 9.10 in JeOS mode. The solution is to upgrade your kernel to the generic, rather than virtual, kernel.

Update 26th November, 2009: Mike West has pointed out that the Linux headers required differ from version to version, so I’ve updated the instructions to reflect the header install required. Also, the JeOS version of Ubuntu 8.04 LTS is available at a different URL than the normal distribution.

Update 29th June, 2009: VMWare Fusion 2.0.5 has been released, which contains an updated VMWare Tools package for Ubuntu 9.04. This removes the requirement to edit the source files as outlined here.

Update 21st June, 2009: it’s been pointed out that Ubuntu 8.10, which this article was originally written for, has been replaced by the 8.04 LTS release. If you want a stable version of Ubuntu, rather than the bleeding edge, you’re best off using this release.

I’ve been using Linux as my primary development environment for a number of months now, despite being an abject Mac fanboy. Why?

I often want to try new software (Python 2.6, Nginx, mod_wsgi, or CouchDB, for example), and neither MacPorts nor Fink are really appropriate to my needs, as it’s too easy to break your system using it (and to give you an idea of how invasive MacPorts is, just take a look at the uninstallation instructions!);
OS X, being both a consumer desktop OS and a BSD–derived UNIX, is nothing like any of the commodity hosting available (usually Debian–derived systems like Ubuntu for me), so releasing stuff requires more testing (both locally and then in a “production–like” environment) before I can be sure it’ll work; but mostly
The penalty for screwing up is too high. I live in fear of trashing the default Apache or Python on my Mac and not being able to recover from it.

I will continue to use my Mac as my main machine, because I love OmniFocus, and Safari, and VMWare Fusion, and all the other applications on my Mac. I love the toolset available to me, and value the user experience.

So what’s the solution? Virtualisation, of course. Thanks to VMWare’s shared folders feature, we can mirror our local folders into the VM, making it super–easy to run our code in Linux (and it also means we don’t have to use Samba).

Following are some (hopefully) bulletproof instructions for setting up JeOS (a virtual–machine friendly version of Ubuntu Server, as pointed out to me by Danny Amey) in VMWare in such a way as to make it an ideal development environment.

Firstly, get the latest Ubuntu Server from the download page (via BitTorrent or any other convenient method). Note that JeOS is listed as a feature of the distribution: Ubuntu Server Edition JeOS (pronounced "Juice") is an efficient variant of our server operating system, configured specifically for virtual appliances;
Then start VMWare Fusion up, and create a new Virtual Machine. I recommend 256MB RAM and HDD of about 10GiB. Name it whatever you like (let's say “JeOS” for this example);
When asked for a CD–ROM, point VMWare at the location of the ISO you just downloaded;
You’ll want to turn printers off, as JeOS doesn't seem to deal very well with printers—probably because it’s quite minimal and doesn't ship with printer capability;
Follow the other steps as normal.

You now have a fresh virtual machine. Polish the settings (change HDD size, for example), and start it up.

Initially you’ll see a language screen. Choose the appropriate language (“English” for me, obviously);
After choosing the language, you’re presented with an install menu. At this point, we need to differ from the usual Ubuntu install process and change our install mode. Hit “F4” (or “fn + F4”) to open the modes menu, and select “Install a minimal virtual machine”;
Follow the rest of the install process as usual (name your user, name your server etc.).

It’s worth nothing that at this point I don’t normally install any of the pre–configured server options—I just leave it as bare bones as possible.

Now the machine will start up, so we can install VMWare tools. This gives us the benefit of Shared Folders, which let us treat the machine as a local folder, so we can use TextMate or other local tools to edit files within the system. This is the real key to making the development VM usable.

After the machine starts up in a VMWare window, log in and type ifconfig. Take note of the IP address in the eth0 section alongside inet addr:. This is the IP address of your virtual machine (if you’re using NAT—if you’re using bridged it’ll be a non-local IP).

Since the terminal provided by basic Linux is a bit sparse on features, install SSH with sudo apt-get install ssh. When this completes, log out and open Terminal.app.

SSH into the system with ssh [username]@[ip address]. You’ll need to accept the dialogue about the MAC address fingerprint.

Now we need to update the system a bit. Run:

sudo apt-get update (this updates the package sources);
sudo apt-get upgrade (this updates the installed packages); and
sudo apt-get dist-upgrade (only if there’s a newer version of Ubuntu floating around—probably not at time of writing)—if you do this, see below.

Note: if you ran dist-upgrade above, you should restart your virtual machine now with: sudo shutdown -r now. Otherwise you may get clashes with kernel versions in the next step (thanks Mike Davies).

Now we can get on with installing VMWare tools.

sudo apt-get install linux-headers-$(uname -r) build-essential (installs some software required to build the components of VMWare tools);
In Fusion, install VMWare tools with the “Virtual Machine > Install VMWare Tools” menu dialogue. This causes a fake CD–ROM to be inserted into the machine;
Back in SSH, mount the CD with sudo mount /cdrom;
Create a new directory to compile source code: mkdir -p ~/src;
Enter the CD: cd /cdrom/
Copy the source code of VMWare across: cp VMwareTools-[HIT TAB TO COMPLETE] ~/src;
cd ~/src;
tar zxfv VMwareTools-[HIT TAB TO COMPLETE];
cd vmware-tools-distrib;
Now we can install VMWare Tools: sudo ./vmware-install.pl [--default].

At this point you can basically accept all the default dialogue options until it completes. Edit: as Mike West has rightly pointed out, passing the --default flag to the install command automatically accepts all defaults for you.

Now you’ve installed VMWare Tools, so it’s time to mount the remote folder:

In the VMWare Library, edit the Settings of your currently running virtual machine;
Select the “Sharing” settings option;
Add a new folder (in mine my code is stored in ~/Projects) and share it with Read/Write permissions;
Close the menu dialogue to save your choice.

This folder is now available to the VM at /mnt/hgfs/[NAME OF FOLDER], except there’s one issue: because of the way the mount works, the VM can’t write back to the shared folder (which is a problem if you’re compiling anything, for example). Here’s how you enable read–write for real:

Work out your user and main group ID by typing: id (this will usually be 1000 and 1000);
Edit your sharing settings with: sudo vi /etc/fstab;
Edit the line starting with: .host:/ /mnt/hgf to look like: .host:/ /mnt/hgfs vmhgfs defaults,ttl=5,uid=1000,gid=1000 0, where uid=1000 and gid=1000 are the actual user and group IDs we discovered back in the first step.

Now your VM can write back to the shared folder, and all is well.

Bonus topics

Link your shared folder into your home directory

If you want the shared folder reflected in your user home directory, just use a symlink:

ln -s /mnt/hgfs/Projects ~/projects

Run the VM in headless mode

So you don’t need to start VMWare each time, you can run the virtual machines as a separate process outside of VMWare Fusion. This is called “headless mode”. From a local terminal on your Mac, type:

defaults write com.vmware.fusion fluxCapacitor -bool YES

and restart VMWare. When you start your machine, choose “View > Enter Headless”. The screen will disappear for that machine, and it’ll keep running. Now you can close VMWare with a running VM and not have the machine stop. Use this script I wrote to start machines via the command line (so you can even bypass VMWare Fusion altogether).

Final notes

At this point it’s wise to use the snapshot feature of VMWare Fusion to save the state of your machine, in case you completely trash it and want to restore from scratch. This is one of the biggest wins of virtualisation in my opinion—making it easy and non–dangerous to experiment.

Also, thanks to Danny for helping me out a bit and Tim Huegdon for being my guinea pig.

Update: A note on Symfony

A few people have written and asked me about how to get Symfony running under this set up. Symfony has slightly peculiar requirements in that it needs to have a PHP/Apache writable directory in which it can dump the compiled YML files it uses for configuration. The solution to this is to patch /etc/apache2/apache2.conf as follows:

User [your username here]
Group [your groupname here (usually the same as the username)]

and then reboot Apache.

This configuration means that Symfony runs with the same permissions as your primary user, which gives it full control over your project directories, and also inherits the permissions granted by HGFS. Needless to say: never do this on a public web server.