All commands

Apache CLF access log format to CSV converter

hute37 — Fri, 10 Feb 2012 16:58:50 +0000

$ #(see sample) $ cat x | perl -pe 'BEGIN{ print "TIME;...\n"; } s!(\S+) - (\S+) - \[(\d\d)/(\S\S\S)/(\S+):(\d\d):(\d\d:\d\d) \S+\] "(\S+) (.*/)(\S+)(?:\.([^?]*)(\?\S*)?) HTTP/\S+" (\d+) (\S+)!$3-$4-$5 $6:$7;$6;$2;$1;$8;$13;1;$14;$11;$10;$9;$12;!' > x.csv

- excel date compatible with a separate hour field

- added a fixed 1 for easier request counter aggregation

- split URL in directory, filename, fileext, query

- used with tomcat valve with response bytes replaced by elapsed time

by David Winterbottom (codeinthehole.com)

Cleanup a (source) text file, removing trailing spaces/tabs and multiple consecutive blank lines

bibe (http://www.1729.it/blog) — Fri, 10 Feb 2012 11:09:54 +0000

$ sed -i -e 's/[ \t]*$//;/^$/N;/\n$/D' sourcefiletocleanup

I used this command (in addition to a code formatting tool) to "cleanup" a bunch of PHP files

by David Winterbottom (codeinthehole.com)

Default value or argument

f4m8 — Fri, 10 Feb 2012 06:24:20 +0000

$ num_lines=${1:-42}

Very similar but no use of `tr` for

function liner() {

local num_lines=${1:-42}

local line=$(printf %${num_lines}s)

echo ${line// /#}

}

by David Winterbottom (codeinthehole.com)

Default value or argument

robinsonaarond — Thu, 09 Feb 2012 17:31:33 +0000

$ num_lines=${1:-42}

I used it for this function, which prints hashes:

function liner {

num_lines=${1:-42}

echo

printf %${num_lines}s|tr " " "#"

echo

}

Note the colon-dash, instead of the usual colon-equals.

by David Winterbottom (codeinthehole.com)

Print current running shell, PID

robinsonaarond — Thu, 09 Feb 2012 17:27:45 +0000

$ ps -p $$

works as well as echo $0, but also prints process id, which pts you're using. echo $SHELL doesn't always get updated when changing shells, so this is a better solution than that. Just one more variation on a theme.

by David Winterbottom (codeinthehole.com)

Revert selection with find.

ohe (http://www.twitter.com/_olivier_) — Thu, 09 Feb 2012 13:10:36 +0000

$ find . ! -name "*.tar.gz"

by David Winterbottom (codeinthehole.com)

Function that show the full path of every argument.

flxndn — Thu, 09 Feb 2012 11:40:05 +0000

$ full() {local i;for i in $*;do readlink -f $i;done}

You can use the command "full" not only with one item, with items with diferent paths, and with relatives and absolute paths.

by David Winterbottom (codeinthehole.com)

Check if a machine is online

UnixNeko — Thu, 09 Feb 2012 06:30:55 +0000

$ ping -c 1 -q MACHINE_IP_OR_NAME >/dev/null 2>&1 && echo ONLINE || echo OFFLINE

PING

parameters

c 1 limits to 1 pinging attempt

q makes the command quiet (or silent mode)

/dev/null 2>&1 is to remove the display

&& echo ONLINE is executed if previous command is successful (return value 0)

|| echo OFFLINE is executed otherwise (return value of 1 if unreachable or 2 if you're offline yourself).

I personally use this command as an alias with a predefined machine name but there are at least 2 improvements that may be done.

Asking for the machine name or IP

Escaping the output so that it displays ONLINE in green and OFFLINE in red (for instance).

by David Winterbottom (codeinthehole.com)

ttyS0 - terminal on serial connection

0disse0 (http://gplus.to/0disse0) — Wed, 08 Feb 2012 20:44:09 +0000

$ setserial -q /dev/ttyS0

I actually planned to do this for quite a long time, but since I haven't had any suitable client hardware, I procrastinated this.

Now, the old laptop I've got from my dad, features an RS-232 port. So, now that I had technically a client, that I could test my RS-232 connection with, I ordered a null modem cable.

There is no RS-232 outlet on my desktop computer directly on the mainboard, but theres a connector on the mainbord, where a RS-232 outlet can be attached to. The outlet will then cover up a PCI slot.

# Activating RS-232

Ok, once all cables were in place, I tried to access both RS-232 ports by reading from them directly. They're usually ttyS0 or ttyS1, depending what COM-Port it is.

From the file /proc/tty/driver/serial, information about the serial setup can be obtained.

A setserial -q /dev/ttyS0 might be usefull as well. Usually, the UART Type is 16550A, on a standard PC. Mine wasn't working though.

At leas not right from the start, when I tried to read the interface with cat /dev/ttyS0 I got the following error:

# cat /dev/ttyS0

cat: /dev/ttyS0: Input/output error

Obviously, the driver couldn't activate the hardware. Reason was, it was deactivated in BIOS. After activating RS-232 there, it worked well.

As a last action, I added myself to the uucp group, so I have user permission to the serial lines. It is not necessary for the terminal setup, but a good idea to do so, just for future projects, maybe...

# Setting up a terminal

Once the Serial line is configured and working properly, it's time to let a terminal run on that port. This is what I added to my /etc/inittab :

s0:2345:respawn:/sbin/agetty -L 38400 ttyS0

I added it quite on the top of that file, right below the 'si' statement, mingetty cannot be used for serial connections, it cannot be run in a console, too. I tried it for testing purposes, but the cosole - along with your login program - will log you out, as soon as you log in over your serial line.

'-L' means this is a local line, with no carrier signal. 38400 is the standard speed of a Linux console, it might be a bit high, I was told, but it works well. I tested that with some higher values as well (115200) and it worked too, I guess it tepends on things like cable length, etc. Last parameter, is the serial tty to listen on. The terminal type can be specified as an additional parameter at the end of the parameter list, vt102, for instance. This is sometimes required, depending on the client.

After finishing editing /etc/inittab, an init q will make the system re-read /etc/inittab and apply changes. The agetty should now be listening on ttyS0.

#Setting up a client

It's time to establish a connection and test the serial line. I use a laptop, that has an RS-232 port, so some preliminary setup is required. I tried minicom as terminal initially, but it turned out, not to be the best client. It initializes the modem, this lasts quite long, and it doesn't convey ANSI colors. So the better option is cu, it's part of the UUCP-Package.

Oh, and the serial port of that computer, has to be accessible as well, of course.

Once everything was set up, I established the connection:

cu -l ttyS0 -38400 --nostop

Pretty self explanatory, I think. The --nostop option disables XON/XOFF handling.

# root access over ttyS0

In order to become root over the serial terminal, the tty needs to be added to /etc/securetty

I appended ttyS0 to the end of the file. It is now possible, to gain root access over the serial terminal. The agetty process needs to be restarted to apply changes.

# Accessing GRUB over ttyS0

To make bootloader access possible over ttyS0, some changes to /boot/grub/menu.lst need to be done. (GRUB is the bootloader I use, I suppose LiLo has similar capabilities.)

Those are the lines, I appended to the top of my menu.lst :

serial --unit=0 --speed=38400 --word=8 --parity=no --stop=1

terminal --timeout=3 serial console

The serial command initiates the serial terminal option, --unit=0 defines our first serial connector, I my case, it's the only one I have on my machine. I used the standard Linux-Console speed, as well as the "8N1" connection strategy.

terminal defines the terminal priorities, first terminal (serial) is the standard one, the last one is the secondary terminal (console).

--timeout=3 enables a delay on both consoles, with a prompt for a keystroke. Depending on which terminal, the key is pressed, this terminal, will be used. If no key is pressed after the timeout, the standard console (in my case serial) will be used.

# Relaying Kernel output on boot

The Kernel accepts multiple console options, of which the last one, is the standard console, and the one that will be used in Single User mode. These are my Kernel options:

title Fedora Core (2.6.20-1.2316.fc5)

root (hd0,0)

kernel /vmlinuz-2.6.20-1.2316.fc5 ro root=/dev/VolGroup00/LogVol00 rhgb quiet vga=795 console=tty0 console=ttyS0,38400

initrd /initrd-2.6.20-1.2316.fc5.img

console=tty0 is the standard console, located on the machine, i.e. monitor and keyboard.

by David Winterbottom (codeinthehole.com)

Reverse SSH

0disse0 (http://gplus.to/0disse0) — Wed, 08 Feb 2012 20:24:38 +0000

$ ssh -f -N -R 8888:localhost:22 user@somedomain.org

this command from the source server and this follow in the destination server:

ssh user@localhost -p 8888

by David Winterbottom (codeinthehole.com)

shell function which allows you to tag files by creating symbolic links directories in a 'tags' folder.

bartonski — Wed, 08 Feb 2012 12:40:45 +0000

$ tag() { local t="$HOME/tags/$1"; [ -d $t ] || mkdir -p $t; shift; ln $* $t;}

The tag function takes a tag name as its first argument, then a list of files which take that tag. The directory $HOME/tags/tagname will then hold symbolic links to each of the tagged files. This function was inspired by tmsu (found at https://bitbucket.org/oniony/tmsu/wiki/Home).

Example:

tag dog airedale.txt .shizturc weimeraner.pl

This will create $HOME/tags/dog which contains symbolic links to airedale.txt .shizturc and weimeraner.pl

by David Winterbottom (codeinthehole.com)

keylogger

totti — Wed, 08 Feb 2012 12:11:23 +0000

$ python -c  "DEV = '/dev/input/event4' fo = open(DEV) def inn(k,s):     if s == 0:        print '%i up'%k     if s == 1:        print '%i down'%k  while 1:     l = fo.read(16)     if ord(l[10]) != 0:        k,s = l[10],l[12]        inn(ord(k),ord(s)) "

python -c "DEV = '/dev/input/event4'

#if event0 doesn't work, try event1 event2 etc

fo = open(DEV)

def interpret(keycode,state):

if state == 0:

print '%i up'%keycode

if state == 1:

print '%i down'%keycode

if state == 2:

print '%i repeat'%keycode

while 1:

line = fo.read(16)

if ord(line[10]) != 0:

keycode,state = line[10],line[12]

interpret(ord(keycode),ord(state))

by David Winterbottom (codeinthehole.com)

grep directory and sub-directories

totti — Wed, 08 Feb 2012 11:48:08 +0000

$ grep -r <searchterm> *

by David Winterbottom (codeinthehole.com)

gawk gets fixed width field

unixmonkey21861 — Wed, 08 Feb 2012 08:18:25 +0000

$ ls -l | gawk -v FIELDWIDTHS='1 3 3 3' '{print $2}'

by David Winterbottom (codeinthehole.com)

what model of computer I'm using?

bbbco (http://www.briandgoad.com) — Tue, 07 Feb 2012 16:26:23 +0000

$ sudo dmidecode | grep Product

by David Winterbottom (codeinthehole.com)

Create new user with home dir and given password

0disse0 (http://gplus.to/0disse0) — Mon, 06 Feb 2012 19:53:01 +0000

$ useradd -m -p $(perl -e'print crypt("passwordscelta", "stigghiola")') user

The crypt function takes a password, key, as a string, and a salt character array which is described below, and returns a printable ASCII string which starts with another salt. It is believed that, given the output of the function, the best way to find a key that will produce that output is to guess values of key until the original value of key is found.

from http://en.wikipedia.org/wiki/Crypt_(Unix)

by David Winterbottom (codeinthehole.com)

I am using a desktop?

0disse0 (http://gplus.to/0disse0) — Mon, 06 Feb 2012 19:44:44 +0000

$ sudo hal-get-property --udi /org/freedesktop/Hal/devices/computer --key 'system.formfactor'

by David Winterbottom (codeinthehole.com)

what model of computer I'm using?

0disse0 (http://gplus.to/0disse0) — Mon, 06 Feb 2012 19:43:57 +0000

$ sudo hal-get-property --udi /org/freedesktop/Hal/devices/computer --key 'system.hardware.product'

by David Winterbottom (codeinthehole.com)

how to allow a program to listen through the firewall

0disse0 (http://gplus.to/0disse0) — Mon, 06 Feb 2012 19:33:57 +0000

$ netsh firewall add programmaautorizzato C:\nltest.exe mltest enable

To allow a program ("programmaautorizzato" in example) to listen through the firewall

by David Winterbottom (codeinthehole.com)

Send e-mail if host is 'dead' or not reachable

bashrc — Mon, 06 Feb 2012 14:54:15 +0000

$ 10,30,50 * * * * ping -c1 -w3 192.168.0.14 >/dev/null

Cleaner with a mailto assignment in crontab (if the command fails you get an email):

MAILTO=admin@example.com

10,30,50 * * * * ping -q -c1 -w3 192.168.0.14 >/dev/null

by David Winterbottom (codeinthehole.com)

Send e-mail if host is 'dead' or not reachable

knoppix5 — Mon, 06 Feb 2012 10:42:46 +0000

$ 10,30,50 * * * * ping -q -c1 -w3 192.168.0.14 | grep '1 received' - || mail -ne -s'Host 192.168.0.14 not reachable' admin@example.com

Every 20 minutes test if host with IP 192.168.0.14 is 'dead' or not reachable.

The line should be put in your crontab file.

by David Winterbottom (codeinthehole.com)

Make alert if host is 'dead' or not reachable

knoppix5 — Mon, 06 Feb 2012 09:40:11 +0000

$ 10,30,50 * * * * ping -q -c1 -w3 192.168.0.14 | grep '1 received' - || env DISPLAY=:0 xeyes

Every 20 minutes ping host with IP address 192.168.0.14. If it's not 'alive' or not reachable, then display something eye-catching (here xeyes) on the desktop.

Hint for newbies: edit crontab with

crontab -e

by David Winterbottom (codeinthehole.com)

before writing a new script

sharfah (http://fahdshariff.blogspot.com) — Mon, 06 Feb 2012 08:25:27 +0000

$ echo '#!'$(which bash) > script.sh

Writes out the shebang line (#!/bin/bash) to the script.

by David Winterbottom (codeinthehole.com)

Progress bar for MySQL import

wr8cr8 — Mon, 06 Feb 2012 08:16:55 +0000

$ pv -i 1 -p -t -e /path/to/sql/dump | mysql -u USERNAME -p DATABASE_NAME

Print out the progress of MySQL import using the pv command. Updates every second.

by David Winterbottom (codeinthehole.com)

before writing a new script

unixmonkey21861 — Mon, 06 Feb 2012 01:03:22 +0000

$ which awk > script.awk

write first line `#!`

by David Winterbottom (codeinthehole.com)