aleatory

Deploying Updates from Windows Dev Box to Linux Server

2012-01-17T18:19:43-08:00

I develop on my local Windows 7 (or XP netbook when out of town) and to test/deploy projects I need to send them to my Gentoo Linux server. While manual copying of files quickly becomes a pain in the arse a fully fledged continuous integration environment is needlessly complex for a single programmer project so I created the following batch script that once run checks my windows folders for new & updated files since the last time the script was run then sends them across the ether using scp ¹.

The script has been tested on XP & 7 successfully.

For the linux side of things I run a python paster instance with the —reload flag that means paster will pickup changes and restart automatically, making deployment from my windows box a single click action.

Initially I started out a little rusty in my command line knowledge and had set about getting timestamps from first principles – lots of find, substring comparisons and messy embedded if control statements. Then I found command extensions which gave convenient access to file properties. Forfile and robocopy were potentially very useful shortcuts but neither are found in the base WinXP system so I eventually arrived at the following solution. The key part where datetime comparisons are made is in the main loop:

dir /b /tw /od !folder!%timestampfile% !file! | more +1 > %tmpfile%
set /p latest= < %tmpfile%
del %tmpfile%

Here the [tmp] timestamp file is compared with the current file in the recursive loop and ranked in ascending order of modification datetime (/b cuts out unwanted output from dir, /tw tells it to use file modification time for sorting and /od tells it to order by oldest first). This is piped to more which takes the bottom line (the newest file) and throws it into %tmpfile%. The variable !latest! then gets set to the value (/p switch tells set to expect a single line of input) and the temporary file is deleted. Afterwards we test to ensure !latest! is actually a file we want to transfer and not the timestamp file (or the batch file for that matter) and if so we copy it across to our transfer folder. Anyways:

@echo OFF
 
REM IMPORTANT: 
REM 1. after you checkout your project into the working directory, create this batch file AND a blank txt file named %timestampfile% in your top level project directory. 
REM 1. make sure timestampfile IS unique (no other FILES with this NAME) within all subfolders of your working directory
REM 2. make sure tmpfile IS unique within pwd
REM 3. make sure tmp IS unique directory
REM 4. set your pscp, conn & pw strings TO appropriate values
REM
REM original post: http://aleatory.clientsideweb.net/2012/01/18/deploying-windows-dev-to-linux-server
 
set tmp=C:\tmp\
set tmpfile=tmp.txt
set timestampfile=timestamp.txt
set pscp="C:\path\to\pscp.exe"
set conn=user@server:/path/TO/directory
set pw=yourpasswdhere
 
REM ensure w2k command extensions are enabled AND delayed expansion FOR the control LOOP
setlocal enableextensions enabledelayedexpansion
 
REM tmp directory FOR copying modified FILES across
MKDIR %tmp%
CALL :strlen prelen %~dp0
set /a LEN=2+prelen
 
REM need TO copy timestampfile temporarily TO each subfolder FOR datetime comparison in main LOOP.
FOR /d /r %%j in (*) DO (
	copy %timestampfile% %%j
)
 
REM substring %var:~start,length% - 0-based
REM LOOP through all FILES in directory & SUB directories
REM IF last modified time greater than %timestampfile% - copy TO set TO ssh across
FOR /r %%i in (*) DO (
	set file=%%~dpnxi
	set file=!file:~%len%!
	set folder=%%~dpi
	set folder=!folder:~%len%!
 
	REM below will find last modified time FOR each AND RETURN latest file
	REM always returns 2nd file IF both were modified in the same minute, so we should PUT source file 2nd.
	REM t the moment /tw doesn't work as expected when comparing timestampfile with files inside different folders; 
	REM so the only immediate solution IS TO have a temp copy of timestampfile in the same directory ON each LOOP...
	dir /b /tw /od !folder!%timestampfile% !file! | more +1 > %tmpfile%
	set /p latest= < %tmpfile%
	del %tmpfile%
 
	REM IF latest isn't this batch file or the timestamp file then copy across, maintaining relative folder structure
	IF NOT !latest!==%~nx0 IF NOT !latest!==%timestampfile% ( 
		echo %tmp%!folder!!file!
		MKDIR %tmp%!folder! & copy !file! %tmp%!folder!
	)
)
 
REM timestampfile cleanup
FOR /d /r %%j in (*) DO (
	del %%j\%timestampfile%
)
 
%pscp% -batch -pw %pw% -r %tmp%* %conn%
RMDIR /s /q %tmp%
 
REM update timestamp file
echo %date%-%time%>%timestampfile%
 
GOTO :EOF
 
REM FUNCTION TO count 'string' length
:strlen <resultVar> <stringVar>
(   
	setlocal EnableDelayedExpansion
    set "s=!%~2!#"
    set "len=0"
    FOR %%P in (4096 2048 1024 512 256 128 64 32 16 8 4 2 1) DO (
        IF "!s:~%%P,1!" NEQ "" ( 
            set /a "len+=%%P"
            set "s=!s:~%%P!"
        )
    )
)
( 
	endlocal
    set "%~1=%len%"
    EXIT /b
)

(apologies for the non-batch file syntax highlighting – there’s a bug in GeSHi syntax highlighting engine that inserts HTML cruft into the code.)

You’ll note the intermediate step it takes of creating a temporary folder to place the files to be copied. This means I can copy the whole thing across as a oner rather than a long-winded new ssh connection for each one. The ideal windows solution would be to create a network folder using SAMBA on the Linux server where I could simply copy the files across in one go. If the file sizes were massive I might do this but I already use SSH for sysadmin so to cut down on one less potential security hole I’ll keep it simple.

Notably I had to use pukka REM statements and not the faster double colon to avoid the comments within the FOR loop being interpreted as commands and returning the following error:

The filename, directory name, or volume label syntax is incorrect

Also the one caveat with the above batch code is it won’t copy anything that has the same name as the script or the timestampfile value – so name them accordingly. If there is a better way of doing any of the steps I took then you are welcome to comment below.

Overall, yeah batch file processing is hard.

¹. pscp – a command line scp utility.

First Impressions on Dropping ICT/Gaining Programming

2012-01-11T06:21:10-08:00

Dropping the old ICT curriculum with it’s much maligned emphasis on M$ Office is a brave move by education minister Michael Gove. It could be argued that from purely a learning point of view this one action is more important than any of the recent decisions on increasing education cost. Sending children back to where it all began – the formulation of grammars and statements thereof into machine-readable instructions that produce new computing tasks rather than just learn old ones by rote – is in many ways fundamental to getting a British engineering discipline back on track.

But it’s easy to get carried away here. The gaming industry has long had luminaries making the case for radical change to computing education. But it isn’t as simple as replacing building a spreadsheet with building a bouncing red ball. Spreadsheets may be boring but real world spreadsheet skills are one of the most essential skills in office jobs of all hues in the world today. Replacing that with an app that does something ‘cool’ is not what ICT is about either.

Reducing things to the bare minimum, programming has two innate parts – the design process and the act of realising this design in code. To design effectively, a student must be aware of what has gone before them, whether to use as building blocks or to pick holes, improve and iterate. How many people would be familiar with Excel prior to their job if they didn’t use it at school? Similarly how do we expect programmers to create something better if they have not experienced what went before?

Purely dropping perceivably boring essentials is not the answer – it’s akin to removing addition from maths or spelling from English – but we do need a better balance between application and creation. Far from being opposite ends of the spectrum, both must go hand in hand to rebuild our design culture.

2011: Marking 10 Years of Evolution in Evil Lair Design

2011-12-31T03:27:34-08:00

circa. 2001

2011

Fiscal Union Probably Means an End to Irish Republic’s USP

2011-12-03T18:23:20-08:00

This Friday on the anniversary of the Maastricht Treaty draft and amid a likely worsening economic outlook (Eurozone GDP out Tuesday) the EZ economies should announce moves to rein in the spending of the southern European states & complement this with a plan to harmonise fiscal policy. This will mean among other things bringing tax rates in peripheral nation states into line with eurozone leaders France & Germany.

For Dublin, this is undoubtedly the calm before the storm. Cheap corporation tax has been the primary reason behind many non-European multinationals setting up European HQ shop in the Republic. European resentment of Dublin’s intransigence on the maintenance of their undercut tax rate when asking (and indeed, still asking) for bailout money is well known. Sarkozy made thinly veiled comments during the week past. Merkel’s rhetoric leaves little doubt tax rate parity is number 1 on German plans as a price for sorting out the mess. The trouble with politicians is that they all need to get elected again.

Looking further out, there appears little chance of ratification of such a move passing any mandatory RoI referendum. The political risk inherent in an angry southern Europe has been ignored by markets giving the EZ political establishment the benefit of the doubt that any eventual agreement will also be passed through the EU’s tortuous legislative routes.

In Defence of Ideas

2011-11-17T06:57:50-08:00

An insightful blog post¹ the other day railed against having lots of ideas and emphasised the need to stay on track, to concentrate on execution of The Big Idea. Here’s the other side of that argument.

There are many economic activities in which the rapid generation of ideas is a must have – take trading as one example. This is purely the game of projecting your ideas for where the market is going onto what is happening in reality. And just as you must see both sides of the trade as they say, so too do you need to foresee multiple scenarios ahead in order to understand all potential market movement. Because how it gets to the projected end result is just as important as when/if it gets there. Multiple & even contradictory ideas are the lifeblood then on which trading activity is based.

Sure, unlike startups it may be argued watching a single number make various shapes on computer screens is an inherently transitive activity. Execution is simply clicking a button. Perhaps, but the two are not binary opposites – far from it. What links the two, and thus supports the premise that lots of ideas can have a greater value than just one, is leverage. Traders piggy back on the economic activity of others just as startups can utilise the web services & APIs of others in order to create immense value with relatively little effort.

Take the likes of Zynga. Yes Zynga makes something. They churn out what has become mass media entertainment. But there’s plenty of developers doing that in Flash and still only getting a commission per title (if at all). Zynga on the other hand leveraged. They didn’t even create their first social game, YoVille, they just bought the thing inheriting it’s facebook userbase & social hooks and combined it with an offers service for monetization. This simple use of leverage provided the holy grail of a monetization channel for the social graph. It didn’t require enormously original or hi-tech games it just required leveraging services that were already available on the web. The idea – the route to market – was the important bit while the actual end-product was not i.e. the games are crap!

So just like the trader playing the trend, so too it’s worthwhile for startups to consider the prevalent mood and dare I say it go with the path of least resistance. Instead of seeking to single-mindedly disrupt it armed with The One Big Idea and seeking to create a platform from the outset, maybe being swift, nimble and able to jump on at the right time can also be the key to success. And for that you’re going to need a lot of ideas.

Refs:
1. “The Problem With Ideas”

Leaving Vilnius

2011-09-04T07:15:54-07:00

Last time round. The blur matches the state of my inebriation.

The urban sprawl is rapidly outpacing the idyllic rural lifestyle it engulfs at the city limits – a man & his wife tend to their vegetable field against a backdrop of high rise apartment blocks. The buildings are new, but retain a drabness that is subtly Soviet. Elsewhere the other chief external influence on modern Baltic architecture is evident – Scandinavian ‘box’ living spaces with their dark wooded veneers & pastel green panelling.

Modern Warfare…

2011-08-29T03:42:26-07:00

Less

The Waiting Game

2011-03-10T05:00:24-08:00

Modern Warfare 3 – ONN from kenny yuan on Vimeo.

It was the largest roll of notes I’d ever seen. 3 grand in fifty pound notes. The guy who hauled it out did so as if it were a rolled up copy of the Sun. One night on sentry I had a conversation with another guy on how his law exams were going. Another evening those of us new to the unit went through a joining process at the end of which we were judged by our peers as suitable to serve alongside them.

The side effect of leaving us in a state of complete physical incapability as we crawled into our pits in the early morning was only a minor talking point, getting up the next day for annual fitness tests just another non event none of us stopped to think about for a second. No one failed.

No one ever fails.

The daily grind of camp life is a very different experience from the outside world. But in here our unit attracts all walks of life – academics, tradesmen & school leavers alike. It’s a varied environment. Which is just as well, as the amount of skills our squad needs to pick up en route to Afghan is also pretty diverse.

Forensics, electronic counter measures, IED search, team medics, language – the modern nature of warfighting has morphed the battlefield into a highly dynamic multidimensional theatre where quick thinking across any number of domains previously the reserve of the trained specialist is required.

To aid the man on the ground in this our kit has seen a near constant flux as new equipment is both trialled & issued to the field in rapid evolutions. On an individual level, we’ve never been better supplied. Which is just as well, as the Helmand insurgency proper has been going on for upwards of 5 years now.

To our North the US has created blitzkrieg-like conditions to take the war to the Sangin Talebs who had proven so stubborn when we were there. It’s a bloody tactic but appears to be yielding results. The Taleban soon adapt to new tactics, the question is whether coalition forces can manoeuvre more effectively.

I haven’t tasted war yet but I’m glad I’m going in with those I’ve trained alongside these past months. The ever-changing situation in a tribal matrix such as Helmand demands fighters who adapt breathlessly.

There is undoubtedly a feeling of relief as much as anything that we can get a chance to make the training count for real. And while reconstruction and war fighting at times seem to occupy the same ground – just completely different dimensions – it will also feel good to ascertain first hand just what is happening on the dusty desert wastelands of southern Afghanistan.

And so too, while I believe in the benefits of troops developing their own ideas of the way ahead, the inevitable clash with reality on the ground is another interesting context for the experiences of deployment.

We’ve been hanging around a while, things are now getting decidedly faster.

Netbook Upgrading & Housekeeping

2011-02-26T13:50:21-08:00

I’m a longtime fan of netbooks over other mobile tech gadgets, finding their versatility an important stepchange in the prior arms race for increasingly powerful ‘portables’ that weigh a metric ton and need their own special carrying case, or mobiles with a preposterous array of unusable functions.

May have missed the hackathon at QUB but I’ve still managed to do a bit of hardware related fiddling with this guide on boosting my 4211′s performance & removing a niggle.

As refreshingly contrarian as a netbook can be, depending on one is a double edged sword – the more tasks you want to run on it, the greater the load on it’s relatively tiny engine. One of the few areas you can upgrade a netbook is it’s RAM. When messing around with portable versions of memory-intensive graphics apps, adding in an extra stick of DDR can boost performance significantly.

Similarly the more you depend on a gadget no matter where you go it will soon become encased in a fine layer of dust that destroys fan performance while general wear and tear of budget bits of plastic like the Acer 4211 will start to loosen moving parts and in the case of the fan it’ll be noticed by a pretty irritating clicking noise.

Tools I Used:
Small-head star screwdriver
mini knife
Leatherman Squirt P4
WD40

Removing dust & that fucking clicking noise from the fan

First take out the battery supply via the two slides on the underside of the laptop. Then unfasten the 7 screws dotted around the side edges of the bottom which allows you to unclip the black and silver halves, exposing the guts of the Advent. There were no seals broken in doing this so it’s doubtful as to whether this invalidates any warranty as is normally the case with fiddling with computer innards.

You can now see how big a mess has built up round the netbook fan and clean out accordingly. A toothbrush is recommended around the nooks and crannies while a cloth is fine for the rest of the surface area.

Now for that annoying noise the fan makes when it whirrs up after a few months of near continuous use. Simply spray some WD40 in around the fan mechanism itself, not to much is needed. Hopefully when the fan starts up again the oil will be spread around the fan parts and help prevent whatever friction may have been taking place.

Secondly the 4 fan screws binding the fan to the motherboard should be tightened to ensure this isn’t responsible for the noise as well. It turns out a couple of mine were pretty loose.

Adding more RAM

Upgrading RAM is likely the single biggest performance win on machines that have a free slot. Make sure when you purchase it’s of a similar tech as the existing bus and RAM – for example my Advent 4211 uses DDR2 which is incompatible with DDR and DDR3. Don’t rely on shop assistants to proactively check for you either, because in my experience they don’t. I got a PNY 1GB 533MHz module for about £20 but I seen a better deal on Amazon for a Crucial RAM stick at £15.

On the Advent it’s a simple task to fit it – locate the long black bus in the centre of the motherboard and the original OEM memory will be on one side with an empty space on the other. There’s even a cover sheet to protect the circuitry beneath your memory once you install it. Angle the stick so it lines up with the pins of the bus and fit it in ‘teeth’ first then snap the rest of the stick flat against the board, secured in place with the harness on either side.

Re-clip the base, replace the screws and battery then power back up to confirm the OS is running with the newly available capacity.

Sorted.

The World’s Oldest Direct Marketing Trick?

2011-02-11T03:13:06-08:00

Confusing some of the people some of the time is enough.