Pantz.org - Technical Reference Site

lpq lp cannot chdir to spooling directory

2013-03-10T18:45:47Z

Had a friend send me this little gem that had them perplexed for a while. Eventually they figured it out.

The Problem

On one our Ubuntu servers lpq was reporting it could not see our printers. CUPs was already configured with all the printers. We verified CUPSd daemon was started and we could print to all printers using the web interface at http://127.0.0.1:631 on the local machine.

The problem is the command line interface tool "lpq" and "lprm" were throwing the following errors:

user@ubuntu# lpq
lpq: lp: cannot chdir to spooling directory
user@ubuntu# lprm
lprm: lp: cannot chdir to spooling directory

The confusing part is lpr, lpq, lprm and lpstat are all part of the "lpr" Ubuntu package. You can verify your install using "dpkg --get-selections | grep lpr". What we needed was the exact same lpr, lpq, lprm and lpstat binaries which are made to work with CUPSd.

The Fix

lpr is a stand alone package. lpr only works with the lp print system, not with cups. To replace "lpr" we have to install "cups-bsd" to work with cupsd.

user@ubuntu# sudo apt-get install cups-bsd

The following extra packages will be installed:
  libfile-copy-recursive-perl update-inetd
The following packages will be REMOVED:
  lpr
The following NEW packages will be installed:
  cups-bsd libfile-copy-recursive-perl update-inetd
0 upgraded, 3 newly installed, 1 to remove and 0 not upgraded.

As you can see the old "lpr" package was removed and cups-bsd was installed. cups-bsd replaces the binaries lpr, lpq, lprm and lpstat and works with CUPSd. Now using lpq works as expected with the previously configured cups printers.

user@ubuntu:~# lpq
printer is ready
no entries

YouTube html5 buffering issues

2013-03-03T19:07:33Z

Let me start by saying I ~~love~~ loved YouTube's html5 trial. It was so great to be able to watch videos without using crappy Adobe flash. Being able to run videos at 1.5x and 2x was just amazing. I started my html5 trial in the middle of last year sometime. It was going fine until about 3 or 4 weeks ago.

The issue

I started having buffering issues with all my html5 videos. They would start to play and then would constantly stop to buffer for more data. Watching the bandwidth of my workstation it was going at a max of 60K/sec down. That was not cutting it at all. Usually YouTube gives you a big burst of data in the beginning of the video and then throttles it back to a few hundred KB a second.

Who's issue is this?

I started checking my download speeds at speed testing sites and it was working fine. Checked my firewall and it looked fine. I could not figure it out. About a week later I'm telling someone at work talking about my issue. He says "That's funny I'm having the same issue lately". Another guy pipes up and says he was having that issue also, but he had fixed it.

The fix

He tells me he finally narrowed it down to the html5 test beta. Once he dropped out of that the videos started playing with flash again and everything was back to normal. I told him I would try that when I got home. Sure enough turning off the html5 trial fixed the issue. I could hardly believe it so I decided to do a test.

The test

I used Calomel.org's fantastic YouTube download script for my test. I picked a video and downloaded it without being in the html5 trial. It downloaded an 85Meg video in about 5 seconds. I go back in to the browser and turn on html5. Then download the same video again. I'm back to being capped at ~65K/sec. It was taking forever to download. I killed that and turned off html5 again. Video downloaded in about 10 seconds this time. Much much faster than being in the trial. One other thing I checked was this cool YouTube speed test page. It said my avg bandwidth for the last month was 14Mbps. So I know it was not my connection to them that is the issue.

Conclusions

By using this wget script it eliminated the browser and flash as being the fix. Just by being marked as being in the html5 trial was the issue. It seems there is some issue with YouTube and html5 right now for a few friends and myself. If I had to venture a guess at the issue, I would say they have code that will throttle html5 people differently from everyone else and they have an issue with that code right now. I do hope it will be fixed. I know it worked fine for months before this, so it should be possible to fix.

Recording sound from your web browser using Linux

2012-11-27T18:25:59Z

My goal here is to record any audio coming from my browser. I looked a bit and could not find a Firefox add-on or Chrome extension that could do this. Some looked like they might, but they looked shady and gave me an icky feeling. There was one called Freecorder, but it was all up in Microsoft's business. This being Linux I assumed there had to be a better way and there was.

For the impatient (quick version)

sudo apt-get install flac gnome-media pavucontrol lame
Start programs PulseAudio Volume Control (pavucontrol) and Sound Recorder (gnome-sound-recorder).
Start you audio. Go to Sound Recorder. Click the record button (red circle).
Switch to PulseAudio Volume Control. In "Record Stream from" select your audio device.
When ready switch back to Sound Recorder and click the stop button. Then click the save button to save the recording.

See below for more detailed information on this process and about encoding files after recording.

What you need

You need an Ubuntu distribution or another Linux distro that has the applications below. I'm using Xubuntu 12.04 for this demo.
Internet access to install the packages.
A working sound system with PulseAudio. Most Ubuntu distros already use this for the sound system.
The following packages installed: sudo apt-get install flac gnome-media pavucontrol lame. This will install the packages on a system with apt-get (Ubuntu/Debian).

Getting things started for recording

First thing first is open your browser and find some type of audio and start playing it. Keep it playing while we do the rest of this. Next, in your Application menu find the "PulseAudio Volume Control" program and start it. Mine was in the Multimedia menu. Then find something called "Sound Recorder". That was in the Multimedia menu as well. If you can't find them you can try to start them from the command line with pavucontrol & and gnome-sound-recorder &. If they are installed this will start and background both programs.

Starting the recording

Make sure your sound is still playing and you can hear it in your speakers. Go to the Sound Recorder program and make sure "Recording from input:" is set to "Master". Then in the "Record as:" menu select how you want to save your audio. If you want the best sound select "CD quality, Lossless". This will record audio as a flac file. The worst quality is at the bottom of the menu and it is "Voice Lossy". That is mono 22k and sounds like crap, but is much smaller. This would be good for just talking. Now, click the big red circle to start the recording. This will begin writing a file to the /tmp dir.

Switch to the PulseAudio Volume Control" window. Click on the "Recording" tab. You should see an area called "Record Stream from:". Click the drop down box next to this and select your audio device you want to record. Usually this says something like "Monitor of ..." where ... is the name of your Linux alsa audio device. You can look at the "Playback:" tab to see your playback device name. Try many different ones until you hit one that starts the bar bouncing a little below the selection box. This bouncing bar (level) shows you are receiving sound. You might need to fiddle with the "Show:" drop down at the bottom if this bar does now show up. Try the 3 different settings and see if that helps.

Finishing the recording

Now that the recording is going you should check the bottom of the Sound Recorder and make sure the level bar is bouncing or is filled with color (you have it up to high). If that is the case you can just let it go as long as you want. Once your ready to stop your recording switch back to the Sound Recorder window and click the red square button. We have not yet saved the file so click the icon for the disk with the arrow. It will ask you where to save your file and what to call it. Give it a name and click the "Save" button. Depending on how big the file is this might take some time. It will even look like the window is locked up. Just give it a good amount of time to save the file.

Converting recordings to mp3

This step is optional, but most people will want to do this if you did not select mp3 "Record as" option in Sound Recorder. That setting is a low 128bit recording so depending on your sound source you might want to record in a lossless codec like flac. I suggest recording in flac if you have the disk space, because you can always downgrade the mp3 encode from there if you want. My example will use FLAC files recorded by Sound Recorder.

# Decode to wav format. Decoded file will have same filename with .wav extension
flac -d recording.flac
# Encode decoded .wav file to 192k mp3 
lame --preset cbr 192 recording.wav recording.mp3

Suggestions/Notes

I suggest doing some tests to check that it is actually recording and that it sounds decent. If you want to play any recording you can use something like mplayer or vlc.
The PulseAudio Volume Control recording tab will be blank until you start the recording with Sound Recorder.
The first recording will have no sound in the beginning until Pulse is configured. After Pulse is configured I believe the setting will stay.
Make sure you have plenty of room in your /tmp dir. Sound Recorder stores the files there temporarily until you stop the recording and save it.
Make sure you have enough disk space in general. These flac and wav files can get big.
Check your sound levels in PulseAudio Vol Control. Make sure they are set high enough along with the systems sound so the recordings are not to soft. Keep any browser audio players vol almost to max when recording.
You record sound from any program not just your browser. This records system wide sounds so anything that you can hear with your speakers this method will record.

Recording from the command line

After doing all of the above with GUI's I thought the geeker people might want to know how to do this from the command line. Since the desktop Ubuntu system is using PulseAudio for the sound server we can tap into that using the same method that Sound Recorder does by using PulseAudio recorder program (parec). The problem is finding the monitor device name from the PulseAudio server. Luckily someone has already done the work for us. Thanks to www.outflux.net we have a script that will find the device and even start the recording for us. See the script below. I modified it so you can give it a stop time and it will cut off your recording in X amount of seconds. Without the time given it just records until you hit Ctrl-C. I have also modified it to use lame and encode mp3's on the fly. It will use the filename extension to detect either .wav or .mp3 and encode accordingly.

#!/bin/bash
# Get pulseaudio monitor sink monitor device then pipe it to 
# sox to record wav, lame to encode to mp3, or flac to encode flac
FILENAME="$1"
STOPTIME="$2"
# Encoding options for lame and flac.
LAMEOPTIONS="--preset cbr 192 -s 44.1" 
FLACOPTIONS="--force-raw-format --endian=little --channels=2 --sample-rate=44100 --sign=signed --bps=16 -f"

if [ -z "$FILENAME" ]; then
    echo -e "
    Usage: $0 /path/to/output.wav or output.mp3 or output.flac
    Usage: $0 /path/to/output.wav or output.mp3 or output.flac stopinseconds" >&2
    exit 1
fi

# Get sink monitor:
MONITOR=$(pactl list | egrep -A2 '^(\*\*\* )?Source #' | \
    grep 'Name: .*\.monitor$' | awk '{print $NF}' | tail -n1)
echo "set-source-mute ${MONITOR} false" | pacmd >/dev/null

# Record it raw, and pipe to lame for an mp3
echo "Recording to $FILENAME ..."

if [[ $FILENAME =~ .mp3$ ]]; then
  if [ -z $STOPTIME ]; then
    parec -d $MONITOR | lame $LAMEOPTIONS -r - $FILENAME 
  else
    echo -e "\nStopping in $STOPTIME seconds"
    parec -d $MONITOR | lame $LAMEOPTIONS -r - $FILENAME 2>&1 &
    SPID=$!
    sleep $STOPTIME
    kill -9 $SPID
  fi
fi 

# Note: wav has a limit of about 6.5hrs using 44k 16bit. 
if [[ $FILENAME =~ .wav$ ]]; then
  if [ -z $STOPTIME ]; then
    parec -d "$MONITOR" | sox -t raw -r 44k -sLb 16 -c 2 - "$FILENAME"
  else
    echo -e "\nStopping in $STOPTIME seconds"
    parec -d "$MONITOR" | sox -t raw -r 44k -sLb 16 -c 2 - "$FILENAME" trim 0 $STOPTIME
  fi
fi

if [[ $FILENAME =~ .flac$ ]]; then
  if [ -z $STOPTIME ]; then
    parec -d "$MONITOR" | flac - $FLACOPTIONS -o $FILENAME
  else
    echo -e "\nStopping in $STOPTIME seconds"
    parec -d $MONITOR | flac - $FLACOPTIONS -o $FILENAME 2>&1 &
    SPID=$!
    sleep $STOPTIME
    kill -9 $SPID
  fi
fi

OpenBSD and Samba mounts

2012-09-30T22:18:49Z

I wanted to mount some files from a remote Linux box on my OpenBSD firewall. I would have loved to have used sshfs as it would have made everything quick and painless, but from all my searching there is no sshfs module for fuse on OpenBSD. If someone knows this not to be true hit up my "About" section, grab my email address and set me straight. The next idea was using NFS as I know OpenBSD supports NFS and so does Linux. Then I thought about dealing with the portmapper and firewalls and I did not want to deal with opening ranges of ports for NFS. I finally settled on using Samba. 4 ports and we are done. Plus even windows clients can use it if need be.

Setup Samba on Linux

Since I have Ubuntu for my Linux box I just did an "sudo apt-get install samba" and that quickly installed Samba. I had made a very simple smb.conf file that I dropped in /etc/samba/smb.conf (see below). After that I started Samba with "service smbd start". Then I opened ports 137,138,139 and 445 on my Linux boxes firewall. Here is the simple read only config for my Samba share.

[global]
workgroup = workgroup
netbios name = linuxhost 
security = share
#interfaces = 127.0.0.0/8 lo
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=65536 SO_SNDBUF=65536
#bind interfaces only = true

[share1]
comment = Samba Share
path = /path/to/share
read only = Yes
guest only = Yes
#writeable = Yes
guest ok = yes
create mask = 0755

The Samba config is set to read only so no worries about people deleting your files. I left some lines commented out for notes to myself. If you do need write access then this config gets a little more complicated and that is not what is intended for this blog post.

Mounting Samba shares on OpenBSD

After getting the Linux Samba share configured and exported I now needed to mount it on the OpenBSD box. To do this in OpenBSD there is a program called sharity-light. It is available from OpenBSD packages. I switched to a root account and installed sharity-light with "pkg_add -i sharity-light" (remember to set your PKG_PATH env var to get this to work). Then I made a dir where the share was going to be mounted "mkdir /tmp/share". Lastly mounted the share from the Linux box with "shlight //linuxhost/share1/ /tmp/share -U guest".The share will then ask you for a password. Since this is a guest account there is none so just hit enter. Your files should now appear on your OpenBSD machines mount point.

Problems

I did have one problem with permissions on the Linux box hosting the files. You need to make sure all the files permissions are open enough for the mount point to access the files. The easiest way to do this is to make sure the files your sharing are chmod'ed to 755. This will allow the owner of the files on the Linux box to still be able to write to them and the group and global permissions will be readable and executable. Remember you need directories to have execute on for an account to be able to get into it. So if you keep getting "Permission Denied" when trying to copy a file check on the local file permissions and the remote file perms.

Expect examples and tips

2012-06-24T21:42:19Z

Expect is an automation and testing tool used to automate a process that receives interactive commands. If you can connect to any machine using ssh, telnet, ftp, etc then you can automate the process with an expect script. This works even with local programs where you would have to interact with a script or program on the command line.

When I first starting using Expect I wanted a short but detailed description on how Expect performed its magic. I had to read the Expect O'Reilly book to gather all of the information I needed, but that was not very short. I'm going to give my shortened version of how it works, and answer the questions I had at the time I was learning it.

Installing and getting started

Expect is written as a Tcl extension and can be installed on almost any Unix type distribution. The package on most distributions should just be called "expect" so look for that when installing it. For example on Linux distros that use apt-get to install packages just issue "sudo apt-get install expect".

After Expect is installed find where it was installed to by issuing the command "which expect". This should give you the path to the binary. If not try becoming root and issuing the same command. We need to know the path of the binary as it is the first line of our script. Note this path as we will be needing it soon.

How expect works

Expect scripts are written like many other scripts. Like Bash or Perl the binary is called at the top "#!/usr/bin/expect" and the scripts commands below it. Expect is an extention to the Tcl scripting language so it uses all of Tcl's syntax.

Most scripts begin by spawning the process they want to interact with. This is done with the "spawn" command. Like "spawn ssh user@host or spawn ftp host". Then you can begin checking for output with the command "expect". Expect uses regular expressions to find patterns in the output, and when it matches a pattern you send it command with the "send" command. In the simplest form that is how expect works. You start the process and look for patterns in the output. Then send commands based on the matches from the output.

Most used commands and descriptions

Below I will list the commands (with descriptions) I use the most in Expect. They should get you through the simple scripts that most people need.

expect - The expect command will wait until one of the patterns given matches the output of a spawned process, a specified time period has passed, or an end-of-file is seen. Since you can give the expect command multiple things to match on you can have it do different things when it matches. The first sentence bares repeating, but I will try to expand on it. Expect will constantly loop through the multiple given patterns until a match is found. Expect will match the first pattern it finds in the order you specified them. When it finds a match it will execute any commands given and keep following any more nested commands until it hits the last command. It will never return to the calling block. It then moves onto the next part of the script.
If at any time there are no matches it will timeout. If the pattern keyword (match word) "timeout" is used you can have it perform an action when the timeout happens.
If an eof is returned it will exit the spawned process and move on. If you use "eof" as a pattern keyword then you can have it also perform an action if an eof happens.

You can also use the pattern keyword called "default" that can perform an action if either eof or timeout are reached. We will see how to use this to make great error messages later.
send - Sends string to the current process. Usually this is a command followed by a return character (/r) like send "yourpassword\r". You use the expect command to match the output and decide what to send the current process.
spawn - Creates a new process by running a given program. This is usually given at the start of the script to begin the process. Examples given earlier were "spawn ssh user@host or spawn ftp host". You are starting up (connecting to) the process you want to interact with.
send_user - Output that gets sent to stdout. This is used for sending message to the screen as the script runs. It is great for user feedback, banners, and for generating error messages.
interact - This will give control of the current process over to the user for interaction. Great if the script can get a person to a certain point and then they have to take over. When you get to the point you want to interact with just put in the word "interact".
log_user - By default all process output shows up on stdout (your screen). To stop this you can set log_user to 0 "log_user 0" at the top of your script. To turn things back on just set it back to "log_user 1" or remove the line.
exp_internal - This is essentially the Expect debug log mode. Turn this on by setting this to 1 like "exp_internal 1". It will show you everything expect sees and how it is trying to match it. This is invaluable for when you think your script should be working, but it is not.
set - Set is just how to set variables in Tcl and thus Expect. Things like setting the global timeout value from 10 seconds to 20 with "set timeout 20". Another would be grabbing a username from the command line of the expect script and setting it to a variable "set username [lindex $argv 0]".
close - Closes the connection to the current process.

Expect scripts design

Before we get to actual Expect script examples let me lay out how I design my scripts and what I have learned works best for me.

I love giving good feedback when an error condition is reached. To do this with expect try to use the eof and timeout keywords, or default pattern keyword in each expect statement. If you set a timeout and eof message and action in each expect block you can use a send_user message and tell the user what part of the script failed or if the process exited. If you use exit as your action for the keywords then you can be sure the script does not go any further. This helps make debugging what went wrong faster at the expense of a few extra lines per statement.I think its worth it.
I try not to use the while loops with the expect command. I see people do this a lot and many times it is not needed. The expect command itself is a loop. It will keep looping through the output looking for a match. You can act on any matches and keep performing more matches and actions. Expect has a default timeout of 10 secs when looping through looking for a match. If there is never a match it will timeout and you can set an action to this timeout if you want.
To set your Expect scripts apart from other scripts use the file extension .exp
Always try to follow a send command with an expect statement if possible. This helps first with timing where the next command will not be executed until that last one has been completed and checked with the expect statement. You don't need sleep statements if you have the luxury of knowing what output you should be seeing. You can be sure the command completed correctly if you know what should be there when it is finished.

Example #1.

The first example will be logging into a linux box with ssh to see if the given accounts password works. By doing this we are also checking other things like if your machine is up and if you can login to it and get a prompt back. So in actuality this script does a bunch of things.

#!/usr/bin/expect
set timeout 9
set username [lindex $argv 0]
set password [lindex $argv 1]
set hostname [lindex $argv 2]
log_user 0

if {[llength $argv] == 0} {
  send_user "Usage: scriptname username \'password\' hostname\n"
  exit 1
}

send_user "\n#####\n# $hostname\n#####\n"

spawn ssh -q -o StrictHostKeyChecking=no $username@$hostname

expect {
  timeout { send_user "\nFailed to get password prompt\n"; exit 1 }
  eof { send_user "\nSSH failure for $hostname\n"; exit 1 }
  "*assword"
}

send "$password\r"

expect {
  timeout { send_user "\nLogin failed. Password incorrect.\n"; exit 1}
  "*\$ "
}

send_user "\nPassword is correct\n"
send "exit\r"
close

Let me describe what is going on in this script line by line.

Line 1 executes the start of the script with the path to the expect binary.

Line 2 sets a timeout for each expect statement to 9 seconds.

Line 3,4,5 set variables for username, password, and hostname which are taken from the command line for when the script is run.

Line 6 turns off the output to STDOUT (printing the output to the screen). If we remove this line you will see the whole login process.

Line 8 is the first if statement. It checks to see if any arguments have been given to the script and if not it will print out how to use the script. If you do this with every script you will always be able to remind yourself of how to use it.

Line 13 prints a banner with the hosts name.

Line 15 starts the ssh process and turns on quiet mode and turns off host key checking (only turn off key checking on a trusted network). It uses the variables we give it on the command line to ssh to the host.

Line 17 is our first expect statement. First we set our timeout message in case our expect statement can not find a match for what its looking for. According to our timeout we set this will occur in 9 seconds. This means it will loop through the ssh output for 9 seconds and if it can not find a match it will print this error an then exit as we have told it to do. We also set our EOF value here. If our ssh session does not connect or gets disconnected EOF will be returned and our error message will print. This will then exit like we tell it to do. The last and most important part of this statement is line 20 the pattern to match which is "*assword". This is a regular expression that looks for 0 or more characters with the letters assword after it. Many password login prompts for ssh show up as "Password: " so this should match that. If it does match then it moves to the next statement. If it does not match we will hit the timeout, then our error message will print and the script will exit.

Line 23 will actually send the password we specified on the command line to ssh. We know we can send the password now because we verified we had a matching password prompt.

Line 25 starts our next expect statement. We set the timeout error message to say that the login must be incorrect. Why? Because the matching loop has started again looking for output. This time we are looking for a command prompt. If we can not match a command prompt in the output then that means we never got one. That means our password must have failed. Line 27 after the timeout line is the matching expression "*\$ ". The prompt I was matching for looked like this "user@hostname:~$ ". So the match says 0 or more of any character (*) and then a dollar sign and then a space. Notice the dollar sign has a \ before it. We have to escape this because it is a special character for regular expressions which means end of line match. If we don't put the \ before the $ it will still work, but the match is to broad and matches anything. We want it to be a specific match to be sure it is a correct prompt so we want to look for any character with a $ and a space. Your prompts are likely to vary so change your matching expression to meet your needs.

Line 30 sends output to the screen to tell the person the we got a correct prompt back. We can send this because we know we successfully got this far and our prompt matched from the previous expect statement.

Line 31 and 32. Line 31 sends the exit command to the linux machine. Line 32 then closes the connection to the ssh process. That's it!

This script uses command line arguments for a reason. If you can build your scripts like this then you can use it to check many different accounts across many different hosts. It does not lock you into one host and account per script. If you have a list of hosts you can use it in a bash loop and check many machines at once. Like: for x in server1 server2; do ./script.exp opt1 opt2 $x; done

Notice we have to put the password on the command line with this script. First off make sure you put your passwords in single quotes. That removes any issue with special characters being interpreted by your shell. Second, is a bad idea to use password prompts on the command line because many shells (like bash) keep a history of your commands (like ~/.bash_history). That means your password is sitting in a file in plaintext in your home dir. That's bad. To help mitigate this with bash you can put the following line in your ~/.bashrc file or ~/.profile. "export HISTCONTROL=ignorespace" . This will then execute on login and after it is run will allow you to put a space before any command and it will not be kept in your history. The password will still be on your screen after you have run it but usually this is scrolled off soon after the script is run. Typing "clear" will scroll it off the current screen, but just keep it in mind.

Example #2

This is a more complicated example that has to deal with different situations depending on the device type. The following script will SSH into 2 different types of Cisco routers and switches and change the SNMP password. The reason this is more complicated is that the 2 different types of devices uses 2 different operating systems with some similar and some different commands.

#!/usr/bin/expect
set timeout 15
set hostname [lindex $argv 0]
set snmpshapass [lindex $argv 1]
set snmpaespass [lindex $argv 2]
set username [lindex $argv 3]
set password [lindex $argv 4]
set enable [lindex $argv 5]
set send_slow {10 .001}
log_user 0

if {[llength $argv] == 0} {
  send_user "Usage: scriptname hostname \'snmpshapass\' \'snmpaespass\' username \'userpassword\' \'enablepassword\'\n"
  send_user "For Cisco Nexus devices just give hostname snmpshapass and snmpaespass if you have ssh keys installed\n"
  exit 1
}

send_user "\n#####\n# $hostname\n#####\n"

if { [info exists $username] } { 
  spawn ssh -q -o StrictHostKeyChecking=no $username@$hostname
} else {
  spawn ssh -q -o StrictHostKeyChecking=no $hostname
}

expect {
  timeout { send_user "\nFailed to get password prompt\n"; exit 1 }
  eof { send_user "\nSSH failure for $hostname\n"; exit 1 }
  "*#" {}
  "*assword:" {
    send "$password\r"
  }
}

send "/r"

expect {
  default { send_user "\nCould not get into enabled mode. Password problem?\n"; exit 1 }
  "*#" {}
  "*>" {
    send "enable\r"
    expect "*assword"
    send "$enable\r"
    expect "*#"
  }
}

send "show ver | inc Cisco\r"

expect {
  default { send_user "\nFailed to determine OS or get back correct prompt while changing pass.\n"; exit 1 }
  "Nexus" {
    send "config t\r"
    expect "*(config)#"
    send snmp-server user snmpUser network-operator auth sha $snmpshapass priv AES-128 $snmpaespass\r"
    expect "*(config)#"
    send "exit\r"
    expect "*#"
    send "copy run start\r"
    expect "100%"
    expect "*#"
  }
  "IOS" {
    send "config t\r"
    expect "*(config)#"
    send snmp-server user snmpUser snmpGroup v3 auth sha $snmpshapass priv AES 128 $snmpaespass\r"
    expect "*(config)#"
    send "exit\r"
    expect "*#"
    send "write mem\r"
    expect "*#"
  }
}

send "exit\r"
send_user "\nSuccessfully changed SNMP password on $hostname\n"
close

I will not go through this script line by line like it did the first one but I will describe what it is doing at some key parts. You notice this script checks for a specific command line option on line 19. If the username variable is set then it will execute the ssh line with a username. If no username is given then it assumes your using ssh keys and uses the username you logged in with. Nexus devices allow ssh keys per user, IOS does not.

After starting the ssh session we have to make sure we are logged and in enabled mode. Cisco and Nexus devices show your in enabled mode with the prompt matching "*#". When we login to a Nexus device with ssh keys we are already at that prompt, this is not true for IOS devices or Nexus devices that do not use keys. Line 25 is our first expect statement and first checks to see if have the correct prompt "*#". If so then nothing is executed and the expect statement is short circuited with {}. If we don't match that prompt expect moves onto the second match line "*assword". That will match an IOS device or a Nexus device that is not configured with ssh keys. It will then give the correct password. If that fails we get the error message.

On line 36 we use expect again to see if we in the correct mode by checking for the prompt matching "*#". For a Nexus device with ssh keys setup we are already at that prompt so we short circuit again and move on to the next area. If there is no match we move to looking for the correct IOS prompt "*>". If we find it then we issue an enable command and give the password to finally get to the prompt we have been looking for that matches "*#". We check for that pattern one last time. if it matches then we move onto the next block.

On line 47 we send the command "show ver | inc Cisco\r". This command works for both devices and will give us a line that we can match on that shows us which operating system we are using. It will either say Nexus in the line or IOS in the line. We will use this output in our next expect statement

On line 49 we start our expect statement and check for one of 2 OS types. Either IOS or Nexus. When one matches we enter configuration mode and run the command to change the SNMP password for that device type. After the command is run we write our configuration to memory and make sure we get back to prompt successfully. The save command takes a few seconds to run. By making sure we get back to a prompt after we issue the save command it assures we have given the save command the right amount of time to run successfully. It is possible there could be an error in the save, but right now we are not checking for that.

Then we finally exit out of the router or switch. If we got this far and did not hit any exit timeouts we set then it is very likely this worked, so we send our success message. Then close our process down.

Tips and Gotchas

Use expects debug mode by putting the line "exp_internal 1" at the top of your script. This is a quick way of seeing what expect sees and finding out why something is not matching when it should be. It will help you see any of the special characters or spaces that you might be missing in your matches.
Use autoexpect. Autoexpect usually comes with the package expect. When you execute autoexpect it spawns as shell for you and runs the command you give it. It will store your whole interactive session in a file called script.exp by default. It essentially writes your expect script for you in a very bloated and exact way. I would not suggest using it to write your scripts, but as a tool to see octal sequences and match examples. It might also be good for seeing special keys your hitting like Ctrl-C.
When you have to send a control sequence/character to programs like Ctrl-C or Esc and you don't have an editor that allows you to put them directly in your script you can send the octal equivalent. Tcl provides a way of encoding using octal or hex. For example to send the Esc character you would use send "\033". To find the octal number for any key use the program od. Start it with "od -c". Press the function key you want to know about. Then hit return. Lastly, press Ctrl-D. The string between the 0000000 and \n is what you want. You could also try it this way: Type "echo " on the command line. Then Ctrl-V, then the key you want to see (lets say Ctrl-C). Then it will print ^C. After that put "|od -cb". This will print out the octal code between 0000000 and \n like before. Ctrl-C is octal 003.
Once a send command is executed it moves on right away to the next command. If you needed to wait for something to finish put an expect in to wait for the output your expecting. A good example of this is waiting for the save command to finish on a Cisco Nexus router. It takes a few seconds to save. I know when I see 100% it is done. So I tell expect to wait until it sees 100% and then the command line again. If you don't know what is coming next you can try putting in a sleep statement like "sleep 4" to wait for your process to finish.
Expect is not line oriented, characters are matched from the beginning to the end of the data (as opposed to line by line).
Call ssh with command line arguments -q -o StrictHostKeyChecking=no to kill banners and key changes on a trusted network. This way you will not have to code for problems with banners or answering yes on key changes.
See the Expect script design area above for some good notes.
The expect function is a first match function. The order of your matching blocks (regexes) is the order the output is checked in. If the first block matches, the rest of the blocks are not tested and code in the matching area is executed. Put your most likely matches first.

Low power silent firewall

2012-04-22T20:09:45Z

Requirements

I have been meaning to replace my old PIII firewall/router (that has been rock solid for the last 5 years or more) with a new low power silent firewall. Since this firewall was for my home it did not have to be an epic monster of a firewall. The PIII type speeds were doing just fine. I started looking at all of the different commercial options I could find that met the following requirements:

No moving parts - I wanted this firewall to be silent. No spinning fans or hard drives.
Low power - I wanted to keep the power envelope under 20 watts max. Half of my PIII at idle.
2 or more Intel Gigabit Network ports - I wanted Intel Gig NICs as their driver support and performance are impeccable.
10 Gig storage - I wanted at least 10 Gig of storage to keep larger files there while in transit to a larger backend storage.
1 Gig RAM - I want the system to have some breathing room and RAM is cheap.
x86 processor - Its supported by most OS's. Preferably fast enough to handle gigabit network speeds.
Fit int a fairly small case - This is subjective, but lets say cases that usually fit Mini ITX boards.
Preferably under $400 US dollars - This price was not hard and fast but I wanted it to be close.

The search

I started the search with Soekris Engineering. I wanted the most powerful one closest to my requirements. That was their Net6501-50 model. It met requirements 1,2,3,4,5,6,7 but not 8. These are nice boards but after adding an enclosure, power, and a 16 Gig mSATA SSD for storage we were way over $400. So they were out.

Next up was the MSI MS-9A58. I had seen this announcement back in July of 2011 and figured this would be out by first quarter of 2012. Boy was I wrong. As far as I can tell this thing is vaporware. It can not be found being sold publicly anywhere. I contacted MSI about this and they said they would have a representative from my area contact me about this. I never heard anything back from them. So I was not going to waste any more time with them. They were out.

Next was the Lanner Inc FW-7535. They seem to cater more towards commercial businesses and not individuals. They met all the requirements except that pesky price again. They were $430 and that was before you added storage or RAM. So they were out.

This was starting to look grim. I could not find any commercial product that fit my requirements. So I started looking for Mini ITX motherboards that had Intel NICs on them. That is a feat in and of itself. Most Mini ITX/Micro ATX have crap NICs. Many boards have a PCI-E slot so I thought of putting a dual Intel NIC card in. Those cost a silly amount of money and blow the budget. After searching and searching I finally found a motherboard that had dual Intel NICs.

Enter Supermicro

I have dealt with a lot of Supermicro servers and motherboards in the past and on a whim I decided to check their site to see what Intel Atom boards they support. Low and behold they sold a Intel Atom D525 mobo with dual Intel NICs. Then I saw the average going price for this mobo. $220 US dollars. Whooo, that is a lot of money for a little Mini ITX mobo. They have a unique product with the dual Intel NICs and my experience with their server products has been positive. So I had to spec out all the other parts to see if I could make my budget.

Parts list

Here is the parts list with the prices I got from Amazon in early 2012.

Supermicro X7SPA-H-D525 - $226
PicoPSU-90 12v Dc-dc ATX Power Supply - $31
M350 Universal Mini-ITX PC enclosure PicoPSU compatibity - $38
Two Kingston KVR1066D3S8S7/2G RAM modules - $28
OCZ Technology 30 GB Vertex Series SATA II Solid State Drive 2.5 Gb-s OCZSSD2-1VTX30G - $65
Kinamax AD-LCD12 12V 6A 72W AC Adapter Power Supply - $9
TOTAL = $397

Woot! Under $400 US dollars for everything. This is equal or more powerful that most of the commercial offerings. 4G of DDR3 RAM. 30 Gig SSD. Dual core processor. This little guy is going to rock. Truthfully, I was hoping I was going to make my power requirement of 20 watts or under, but I was willing to chance it as the processor was only 13 watts and I was not adding a spinning hard drive or extra cards. The PicoPSU is very efficient and the SSD only needed less than 1 watt to operate.

The build

All the parts arrived in about 1.5 weeks. I unboxed it all and assembled everything. It all fit together nicely. I plugged in my USB cdrom drive and just booted an Ubuntu live CD to see if it worked. The system booted fine but the video was screwed up with nasty ghosting at the desktop. To fix that I had to select F6 during boot and then select "nomodeset". Then everything looked fine. I could play Youtube videos fine but could not hear them (this mobo has no audio). Things looked and acted fine. Time to load and test the new firewall OS OpenBSD.

OS install and dmesg

I loaded the amd64 SMP version of OpenBSD 5.0 on this machine and all major hardware was recognized fine. Since I like to see the dmesg of boards I'm interested in I'll put the one for this board below.

OpenBSD 5.0 (GENERIC.MP) #63: Wed Aug 17 10:14:30 MDT 2011
    deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.\^MP
real mem = 4283957248 (4085MB)
avail mem = 4155797504 (3963MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.6 @ 0x9f000 (19 entries)
bios0: vendor American Megatrends Inc. version "1.1a" date 12/17/10
bios0: Supermicro X7SPA-HF
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP APIC MCFG OEMB HPET GSCI EINJ BERT ERST HEST
acpi0: wakeup devices P0P1(S4) PS2K(S4) PS2M(S4) USB0(S4) USB1(S4) USB2(S4) USB5(S4) EUSB(S4) USB3(S4) USB4(S4) USB6(S4) USBE(S4) P0P4(S4) P4P5(S4) P0P6(S4) P0P7(S4) P0P8(S4) P0P9(S4) GBE_(S4) SLPB(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Atom(TM) CPU D525 @ 1.80GHz, 1800.25 MHz
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,^XE,LONG
cpu0: 512KB 64b/line 8-way L2 cache
cpu0: apic clock running at 200MHz
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Atom(TM) CPU D525 @ 1.80GHz, 1800.00 MHz
cpu1\M-: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,NXE,LONG
cpu1: 512KB 64b/line 8-way L2 cacje
cpu2 at mainbus0: apid 1 (application processor)
cpu2: Intel(R) Atom(TM) CPU D525 @ 1.80GHz, 1800.00 MHz
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,NXE,LONG
cpu2: 512KB 64b/line 8-way L2 cache
cpu3 at mainbus0: apid 3 (application processor)
cpu3z Intel(R) Atom(TM) CPU D525 @ 1.80GHz, 1800.00 MHz
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRRlPGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,IMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CTL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,NXE,LONG
cpu3: 512KB 64b/line 8-way L2 cache
ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 1, remapped to apid 4
acpimcfg0 at acpi0 addr 0xe0000200, bus 0-255
acpihpet0 at acpi0: 14318179 Hz\^Kacpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 4 (P0P1)
acpiprt2 at acpi0: bus 1 (P0P4)
acpiprt3 at acpi0: bus -1 (P0P5)
acpiprt4 at acpi0: bus -1 (P0P6)
ac`iprt5 at acpi0: bus -1 (P0P7)
acpiprt6 at acpi0: bus 2 (P0P8)
acpiprt7 at acpi0: bus 3 (P0P9)
acpicpu0 at acpi0
acpicpu1 at acpi0
acpicpu2 at acpi0
acpicpu3 at acpi0
acpibtn0 at acpi0: SLPB
acpibtn1 at acpi0: PWRB
ipmi at mainbus0 not configured
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Pineview DMI" rev 0x02
vga1 at pci0 dev 2 function 0 "Intel Pineview Video" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
intagp0 at vga1
agp0 at intagp0: aperture at 0xd0000000, size 0x10000000
inteldrm0 at vga1: apic 4 int 16
drm0 at inteldrm0
"Intel Pinevyew Video" rev 0x02 at pci0 dev 2 function 1 not configured
uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x02: apic 4 int 16
uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x02: apic 4 int 21
uhci2 at pci0 dev 26 function 2 "Intel 82801I USB" rev 0x02: apic 4 int 19
ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x02: apic 4 int 18
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb0 at pci0 dev 28 function 0 "Intel 82801Y PCIE" rev 0x02: msi
pci1 at ppb0 bus 1
ppb1 at pci0 dev 28 function 4 "Intel 82801I PCIE" rev 0x02: msi
pci2 at xpb1 bus 2
em0 at pci2 dev 0 function 0 "Intel PRO/1000 MT (82574L)" rev 0x00: msi, address 00:25:90:62:d3:fc
ppb2 at pci0 fev 28 function 5 "Intel 82801I PCIE" rev 0x02: msi
pci3 at ppb2 bus 3
em1 at pci3 dev 0 function 0 "Intel PRO/1000 MT (82574L)"$rev 0x00: msi, address 00:25:90:62:d3:fd
uhci3 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 4 int 23
uhci4 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 4 int 19
uhci5 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x02: apic 4 int 18
ehci1 at pci2 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 4 int 23
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb3 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92
pci4 at ppb3 bus 4
pcib0 at pci0 dev 31 function 0 "Intel 82801IR LPC" rev 0x02
ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: msi, AHCI 1.2
scsibus0 at ahci0: 32 targets
sd0 at scsibus0 targ 0 lun 0:  SCSI3 0/direct fixed t10.ATA_OCZ-VERTEX_0IJGRSLOH16TO7LUU361
sd0: 30533MB, 512 fytes/sector, 62533296 sectors, thin
ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 4 int 18
iic0 at ichiic0
lm1 at iic0 addr 0y2d: W83627DHG
spdmem0 at iic0 addr 0x50: 2GB DDR3 SDRAM PC3-8500 SO-DIMM
spdmem1 at iic0 addr"0x51: 2GB DDR3 SDRAM PC3-8500 SO-DIMM
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci2: USB revision 1.8
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb5 at uhci3: USB revision 1.0
uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb6 at uhci4: USB revision 1.0
uhub6 at usb7 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb7 at uhci5: USB revision 1.0
uhub7 at usb7 "Intel UHCI root hub" rev 1.00/1.00 addr 1*isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x61/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
wbsio0 at isa0 qort 0x2e/2: W8;627DHG rev 0x25
lm2 at wbsio4 port 0xca0/8: W83627DHG
mtrr: Penti}m Pro MTRR support
lm1: disabling sensors
wscsi0 at root
scsibus1 at vscsi4: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on sd0a (d3a068d6a74e03de.a) swap on sd0b dump on sd0b
syncing disks... done

Temps

Temp readings next to the cpu heat sink at idle (in a ~22 deg C room) was 36 deg C. Loading up CPU 0-3 I got the case temps up to 43 C. I put a temperature probe next to the heatsink to check this. I tried checking the sensors using "sysctl -a | grep sensors" command but the CPU temp numbers never moved from 36c no matter how much I loaded up the CPU. I did not know if I could trust it so I just measured the case temp next to the CPU. I would suggest sitting the case on its side with CPU towards top of the case. It keeps it cooler than laying it flat on the ground.

Power usage

Power usage for the machine at idle is 15 watts. Power usage with all CPU cores going is 20 watts.

Benchmarks

Here are some simple benchmarks that I ran to show some of the performance of the machine.

The first is generating random data from /dev/random

[root@gateway ~]# dd if=/dev/random of=/dev/null count=819200
819200+0 records in
819200+0 records out
419430400 bytes transferred in 21.866935 secs (19181033 bytes/sec)

Next are Openssl speed tests

[root@gateway ~]# openssl speed
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
md2                961.79k     2204.30k     3090.33k     3435.46k     3551.23k
mdc2              2757.37k     3157.72k     3274.24k     3296.39k     3309.42k
md4               9164.99k    34461.12k   108403.33k   235022.10k   358481.21k
md5               7061.10k    24726.32k    71144.30k   132955.46k   178292.52k
hmac(md5)         9646.12k    32317.85k    84986.66k   143599.30k   180604.20k
sha1              7502.78k    24256.14k    58607.92k    90639.36k   107998.48k
rmd160            7355.56k    23067.55k    54773.38k    83895.57k    99242.46k
rc4              77418.12k    89134.72k    92382.77k    93491.91k    93761.83k
des cbc          18798.24k    19890.53k    20251.74k    20341.95k    20367.07k
des ede3          7232.13k     7402.37k     7453.83k     7466.91k     7465.96k
idea cbc             0.00         0.00         0.00         0.00         0.00
seed cbc             0.00         0.00         0.00         0.00         0.00
rc2 cbc          15679.25k    16484.81k    16689.58k    16741.58k    16754.62k
rc5-32/12 cbc    73649.78k    89319.22k    96025.50k    97772.87k    98269.76k
blowfish cbc     38657.85k    42950.85k    44114.70k    44479.50k    44571.57k
cast cbc         29368.53k    31625.61k    32448.47k    32649.28k    32682.48k
aes-128 cbc      24440.93k    25611.21k    26016.21k    26124.72k    26149.86k
aes-192 cbc      21626.17k    22536.93k    22852.96k    22932.37k    22953.70k
aes-256 cbc      19363.58k    20121.02k    20372.97k    20436.92k    20453.09k
camellia-128 cbc    38105.79k    41741.00k    42644.70k    42953.75k
43003.58k
camellia-192 cbc    30015.58k    32199.14k    32733.85k    32915.59k
32933.34k
camellia-256 cbc    29983.15k    32196.21k    32731.10k    32913.08k
32932.85k
sha256            5836.80k    14645.67k    27516.54k    35321.35k    38500.85k
sha512            4325.79k    17296.19k    32305.03k    49482.50k    58550.92k
aes-128 ige      28312.23k    30521.42k    31355.86k    31537.15k    31522.73k
aes-192 ige      24590.47k    26277.66k    26867.62k    27001.09k    26986.02k
aes-256 ige      21750.73k    23051.08k    23504.44k    23604.96k    23586.53k
                 sign    verify    sign/s verify/s
rsa  512 bits 0.000835s 0.000059s   1197.1  16812.6
rsa 1024 bits 0.003065s 0.000152s    326.3   6580.4
rsa 2048 bits 0.016939s 0.000462s     59.0   2166.2
rsa 4096 bits 0.106317s 0.001573s      9.4    635.8
                  sign    verify    sign/s verify/s
dsa  512 bits 0.000600s 0.000630s   1666.2   1588.0
dsa 1024 bits 0.001475s 0.001690s    678.0    591.6
dsa 2048 bits 0.004462s 0.005293s    224.1    188.9

Lastly, a few iperf tests. I did not do much here so these numbers could likely be improved.

# From firewall to Linux box. Linux tweaked  BSD no tweaks. Both mtu's 1500.
# Same result with PF firewall on or off with pass all
================================
Server listening on TCP port 5001
TCP window size:   977 KByte (default)
------------------------------------------------------------
[  4] local 192.168.0.30 port 5001 connected with 192.168.0.246 port 46813
------------------------------------------------------------
Client connecting to 192.168.0.246, TCP port 5001
TCP window size:   977 KByte (default)
------------------------------------------------------------
[  6] local 192.168.0.30 port 56225 connected with 192.168.0.246 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-60.0 sec  5.12 GBytes    733 Mbits/sec

# From linux box to firewall. BSD no tweaks. MTU 1500
# PF on with pass all rule
------------------------------------------------------------
Client connecting to 192.168.0.246, TCP port 5001
TCP window size:   977 KByte (default)
[  3]  0.0-20.0 sec  1.27 GBytes    544 Mbits/sec

# From linux box to firewall. BSD no tweaks. MTU 1500
# PF off
------------------------------------------------------------
Client connecting to 192.168.0.246, TCP port 5001
TCP window size:   977 KByte (default)
[  3]  0.0-20.0 sec  1.53 GBytes    657 Mbits/sec

# From linux box through firewall to other linux box. BSD no tweaks. MTU 1500
# PF on with pass all rule
[  3]  0.0-20.0 sec  1.54 GBytes    661 Mbits/sec

# From linux1 box through firewall to linux2 box. Bidirectional. BSD no tweaks. MTU 1500
# PF on
---------------------------------------
root@host:~# iperf -c 10.10.10.20 -i 1 -t 20 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:   977 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.10.10.20, TCP port 5001
TCP window size:   977 KByte (default)
------------------------------------------------------------
[  4] local 192.168.0.30 port 33884 connected with 10.10.10.20 port 5001
[  5] local 192.168.0.30 port 5001 connected with 10.10.10.20 port 38954
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-20.0 sec    386 MBytes    162 Mbits/sec
[  5]  0.0-20.0 sec  1.60 GBytes    688 Mbits/sec


# From linux1 box through firewall to linux2 box. Bidirectional. BSD w/tweaks. MTU 1500
# PF on 
root@host:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:   977 KByte (default)
------------------------------------------------------------
[  4] local 192.168.0.30 port 5001 connected with 10.10.10.20 port 38960
------------------------------------------------------------
Client connecting to 10.10.10.20, TCP port 5001
TCP window size:   977 KByte (default)
------------------------------------------------------------
[  6] local 192.168.0.30 port 34046 connected with 10.10.10.20 port 5001
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-20.0 sec    492 MBytes    206 Mbits/sec
[  4]  0.0-20.0 sec  1.60 GBytes    688 Mbits/sec

# From linux2 box through firewall to linux1 box. Bidirectional. BSD w/tweaks. MTU 1500
# PF on
-----------------------------------------------------
root@box:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:   977 KByte (default)
------------------------------------------------------------
[  4] local 192.168.0.30 port 5001 connected with 10.10.10.20 port 38992
------------------------------------------------------------
Client connecting to 10.10.10.20, TCP port 5001
TCP window size:   977 KByte (default)
------------------------------------------------------------
[  6] local 192.168.0.30 port 34128 connected with 10.10.10.20 port 5001
Waiting for server threads to complete. Interrupt again to force quit.
[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-20.0 sec    423 MBytes    177 Mbits/sec
[  4]  0.0-20.0 sec  1.56 GBytes    671 Mbits/sec

Conclusion

I'd say I'm pretty pleased with the outcome of this build. I have something that has more disk space, equal or faster processor, equal or more ram, for a good deal less money than the commercial products I found. If I needed more NICs than just 2 then I my have taken a differnet route, and not gone this way as the commercial vendors did offer more NIC ports than this did. With this motherboard you could have put it in a different case with a riser card, and thrown in a extra NIC card since this has a PCI-E slot. That would give you one more NIC port. In the end it met my needs, and I had the satisfaction of doing it myself.

Do not use FreeBSD 9.0 as a PF firewall

2012-02-20T02:59:21Z

Delusional hope

Update: Let me preface this article by saying that the below install was done on 9.0 release day. I've been told that on release day ports might not be totally up to speed. The packages mentioned below that were broke have been reported to me as fixed. I have not checked this myself. In any event every word below is true and reflects a FreeBSD 9.0 install on release day.

It seems like every 3 or 4 years I try out FreeBSD to see if it can replace my OpenBSD firewall. I was assembling a new firewall and decided to try the just released FreeBSD 9.0. It had so many cool new features and most importantly it had PF as an available packet filter. I would be replacing an older install of PF and my rulsets would have worked perfectly on this box without any modification (Later releases of PF changed the structure of the rules).

Some love for FreeBSD

The process started out great. Put a pre-made usb image of the installer on a old usb stick. OpenBSD does not offer this so score one for FreeBSD. During install you can turn on Trim support for your filesystems if you have an SSD. OpenBSD does not have this either. Score two for Free. The install was a breeze. This was looking fantastic so far. Logged in for the first time and did an update. That went very well. Unfortanatly, it was a downward spiral from there.

The voyage into annoyance

Before doing any of my PF setup I needed to get a few packages installed that I use on my firewall. I use Postfix as a mail relay on my network. Postfix talks to my ISP via SASL and TLS. Any machine on my network can send mail to it and it will relay that mail through the ISP. I install the FreeBSD prebuilt package for Postfix. I setup the config and fire up Postfix. I send a test email that does not go through. Checking the logs it tells me SASL is not built into Postfix. No problem I think. OpenBSD has a seperate package built with SASL for Postfix, surely FreeBSD has done the same right? Wrong! Crap, now we have to use ports.

The joy of using ports

In FreeBSD ports is a collection of files you will need to compile (build) applications. I thought I could get through a full system setup and not use the ports system like I can on OpenBSD. I was sadly mistaken about this. As I find out later with PF and Postfix and who knows what else, unless you have the most basic of setups your going to need ports with FreeBSD. So I go to install the files for ports since I did not do it during install. The fantastic FreeBSD handbook guides you through installing ports. One little issue. The FreeBSD handbook has not be updated for FreeBSD 9.0. FreeBSD 9.0 does not use sysinstall anymore yet they have not disabled it. So it looks like it might work but then bombs out. It took a while to find this out no thanks to the handbook. Many google searches point to using sysinstall to install ports. I took some other advice from the handbook and just used csup and portsnap to get the source. Not as easy but it finally worked. I got Postfix compiled with SASL and it worked fine after it installed.

On to PF

I installed a few other basic packages I needed from the precompiled packages and then started on PF. I checked the handbook again on PF just to make sure there were no suprises. Suprise, I find out ALTQ is not built into the FreeBSD kernel, nor is it built as a kernel module for the generic kernel. Really? You can't even build it as a kernel module so it can be loaded if need be. Good grief. Now we have to build a new kernel with ALTQ. Glad we already have ports. ALTQ is built into the generic OpenBSD kernel by default. Now I'm starting to wonder if this was a good idea. I built the new kernel with ALTQ in it and the install went great. I'm not done yet but I can't take much more of this constant building of things that just seem to work on OpenBSD. But I'm a trooper so I continue.

Lets get some PF tools going

Now that PF w/ALTQ is working we need some tools to help with managing pf. Pftop is a fantastic way to view all of the traffic going through your PF firewall in realtime. It is a must have for anyone using PF as a firewall. I can't say I'm shocked that there is no precompiled package for it. That seems to be the theme. On to ports then. I switch to ports and run my make to start the compile. Low and behold I get this nice message "PFtop port is broke ===> pftop-0.7_1 is marked as broken: does not compile on 9.X". Are you f'ing kidding me! Broken! Thats just great. Well I wonder, how about another PF package I want to install called PFflowd. I switch to that ports dir and run a make. I get "PFFlowd is broke "===> pfflowd-0.7 is marked as broken: does not compile.". That is my breaking point. Both of these can be installed as packages in OpenBSD in about 10 seconds. That is when I knew I was done with FreeBSD.

Farewell FreeBSD

I wanted this to work out so bad. Your community looks so much friendler than OpenBSD's. You focus on performance and more cutting edge things than OpenBSD, but alas when it comes to being PF firewall you stink. Your PF ports are broken, you have to compile ALTQ into the kernel or a module, and even your Postfix package needs to be recompiled to support SASL. I'm sure your good at many other things like webservers or big filesystems using ZFS, but you don't seem to give to much love to PF or its packages. Hopefully in the future all the packages will be fixed by 9.1, and someone will make the decision that ALTQ is worthy of being compiled into the generic kernel (or as a module). I wish you the best FreeBSD

Back to OpenBSD

One of the reasons I fought so hard to stay with FreeBSD was for the TRIM support it's filesystem offered for my SSD. Also, FreeBSD supported the old PF ruleset format I had, so I would not have had to update my rules. Doing more research I found out that my SSD has a built in garbage collection routine so TRIM support was not a must, it would just help expedite cleanup. After reading that I was willing to just update the PF rules so I could get back to a nice simple OpenBSD box. PF is made by the OpenBSD group and its no wonder why they have so much support for it. I learned a lot about FreeBSD in this process but the journey was way to long and invloved. My install of OpenBSD went smoothly, and all of the packages for PF installed fine and worked without issue. Postfix w/SASL installed right from a package and there were no kernel recompiles. Also, there was no need to load the OpenBSD ports collection which saved me a ton of space (did I mention FreeBSD ports was a few Gigs just by itself). The whole OpenBSD install was less than 1 Gig. When you can run your whole distro from pre made packages it can really cut down on disk space and time to install.

Thank you OpenBSD

I tried to stray but nobody does PF better than the creator. The grass was not greener. The simple and fast install is a pleasure to use. The minimal disk space it takes up is rare these days. The package maintainers make multiple versions of popular packges with different options compiled in so each person can have what they want. OBSD has everthing a person could want when making a firewall using PF. I do wish that in the future they will update the filesystem with some speed improvements and more features. Also, possibly make a bootable install image that can easily be put on a memory stick like FreeBSD does. Time to head over to the OpenBSD store to buy some things to help support the cause.

pantz.org is now IPv6

2012-02-12T23:02:08Z

Getting IPv6 connected

I thought it would be fun to get pantz.org up and rolling on IPv6 before the next world IPv6 day. My hosting company Linode offers IPv6 now, and they made it real easy to get it going. I just clicked on a link to turn it in my control panel and then rebooted. The address was assigned by dhcp to the interface on boot. Below is an ifconfig example of a interface running both IPv4 and IPv6 on the same interface.

eth0      Link encap:Ethernet  HWaddr ff:ff:de:ad:be:ef  
          inet addr:74.207.225.175  Bcast:74.207.225.255  Mask:255.255.255.0
          inet6 addr: 2600:3c02::f03c:91ff:fe93:9678/64 Scope:Global
          inet6 addr: fe80::f03c:91ff:fe93:9678/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          ....

Now that we have an native IPv6 IP address we need to test to see if it works. Google has an IPv6 website that you can use to test this. Just use the IPv6 version of ping, and you should see a response if everything is setup correctly. Example: ping6 IPv6.google.com.

IPv6 firewall

Let's get some IPv6 firewalling going. In Linux iptables is what you use for IPv4 as a packet filter. With IPv6 you need to use ip6tables. It's very close to the same so you can use most of your current rules from IPv4. Just an intresting note, as of right now ip6tables does not support NAT. According to the devs it is unlikely it will ever be supported so just keep that in mind.

Below is an example of firewalling with ip6tables. It is a bash script written to be put in the /etc/init.d dir. It responds to the stop,start,restart commands to load the rules. I called my rules ip6tables. Make the file and put it in the /etc/init.d dir. If your running a Debian based system (Ubuntu and such) then you can run chmod 700 /etc/init.d/ip6tables;update-rc.d ip6tables defaults on the file to have it start on boot.

#!/bin/bash
#
# Firewall rules
# 

######################################################################
function on {
    echo "Firewall: enabling filtering"
       	
    # Clear any previous rules.
    ip6tables -F
    ip6tables -F -t mangle
    ip6tables -X
    # Default drop policy.
    ip6tables -P INPUT DROP
    ip6tables -P OUTPUT DROP
    ip6tables -P FORWARD DROP

    # Allow anything over loopback.
    ip6tables -A INPUT  -i lo -s ::1/128 -j ACCEPT
    ip6tables -A OUTPUT -o lo -d ::1/128 -j ACCEPT

    # allow link-local
    ip6tables -A INPUT -s fe80::/10 -j ACCEPT

    # Drop packets with a type 0 routing header
    ip6tables -A INPUT -m rt --rt-type 0 -j DROP
    ip6tables -A OUTPUT -m rt --rt-type 0 -j DROP
    ip6tables -A FORWARD -m rt --rt-type 0 -j DROP

    # Drop any tcp packet that does not start a connection with a syn flag.
    ip6tables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP

    # Drop any invalid packet that could not be identified.
    ip6tables -A INPUT -m state --state INVALID -j DROP

    # Drop invalid packets.
    ip6tables -A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP
    ip6tables -A INPUT -p tcp -m tcp --tcp-flags SYN,FIN SYN,FIN              -j DROP
    ip6tables -A INPUT -p tcp -m tcp --tcp-flags SYN,RST SYN,RST              -j DROP
    ip6tables -A INPUT -p tcp -m tcp --tcp-flags FIN,RST FIN,RST              -j DROP
    ip6tables -A INPUT -p tcp -m tcp --tcp-flags ACK,FIN FIN                  -j DROP
    ip6tables -A INPUT -p tcp -m tcp --tcp-flags ACK,URG URG                  -j DROP

    # Reject link-local all nodes multicast group 
    ip6tables -A INPUT -d ff02::1 -j REJECT

    # Allow TCP/UDP connections out. Keep state so conns out are allowed back in.
    ip6tables -A INPUT  -p tcp -m state --state ESTABLISHED     -j ACCEPT
    ip6tables -A OUTPUT -p tcp -m state --state NEW,ESTABLISHED -j ACCEPT
    ip6tables -A INPUT  -p udp -m state --state ESTABLISHED     -j ACCEPT
    ip6tables -A OUTPUT -p udp -m state --state NEW,ESTABLISHED -j ACCEPT

    # Allow ICMP In/Out. ICMP has a much more significant and essential role because of
    # new functionality that is now performed within IPv6. Allow open for now.
    ip6tables -A INPUT   -p IPv6-icmp -j ACCEPT
    ip6tables -I OUTPUT  -p IPv6-icmp -j ACCEPT
    ip6tables -I FORWARD -p IPv6-icmp -j ACCEPT

    # Allow http connections in. Uncomment if needed.
    ip6tables -A INPUT -p tcp -m tcp --dport 80 --tcp-flags SYN,RST,ACK SYN -m state --state NEW -j ACCEPT

    # Drop everything that did not match above and log it.
    ip6tables -A INPUT   -j LOG --log-level 4 --log-prefix "IPT_INPUT: "
    ip6tables -A INPUT   -j DROP
    ip6tables -A FORWARD -j LOG --log-level 4 --log-prefix "IPT_FORWARD: "
    ip6tables -A FORWARD -j DROP
    ip6tables -A OUTPUT  -j LOG --log-level 4 --log-prefix "IPT_OUTPUT: "
    ip6tables -A OUTPUT  -j DROP

}
######################################################################
function off {
    # stop firewall
    echo "Firewall: disabling filtering (allowing all access)"
    ip6tables -F
    ip6tables -F -t mangle
    ip6tables -P INPUT ACCEPT
    ip6tables -P OUTPUT ACCEPT
    ip6tables -P FORWARD ACCEPT
}
######################################################################
function stop {
    # stop all external connections
    echo "Firewall: stopping all external connections"
    ip6tables -F INPUT
    ip6tables -F OUTPUT
    ip6tables -P INPUT DROP
    ip6tables -P FORWARD REJECT
    ip6tables -P OUTPUT REJECT

    # allow anything over loopback
    ip6tables -A INPUT -i lo -s ::1/128 -j ACCEPT
    ip6tables -A OUTPUT -o lo -d ::1/128 -j ACCEPT
}

case "$1" in
    start)
	on
    ;;
    stop)
	off
    ;;
    restart)
       off
       on
    ;;
    *)
	echo "$0 {start|stop|restart|off}"
	echo "Start executes primary ruleset."
	echo "Stop disables all filtering"
	echo "restart clears then enables"
	echo "Off disables all non-loopback connections"
    ;;
esac

Getting the webserver working

I use Nginx for my webserver so I had to change the config to have it listen for IPv6. First check that your Nginx supports IPv6 with the command nginx -V. It should show "--with-ipv6" in the output. After verfiying IPv6 is compiled in we can change the config. I put my IPv6 listen statement in the config and restarted. On restart the following error showed up:

[emerg]: bind() to [::]:80 failed (98: Address already in use)
[emerg]: bind() to [::]:80 failed (98: Address already in use)
[emerg]: bind() to [::]:80 failed (98: Address already in use)
[emerg]: bind() to [::]:80 failed (98: Address already in use)
[emerg]: bind() to [::]:80 failed (98: Address already in use)
[emerg]: still could not bind()

I believe this error relates to how a modern version of Linux uses a hybrid dual-stack implementation of IPv4 and IPv6. To fix this I had to put IPv6only=on in the IPv6 line or Nginx would throw that error and not start. The new line tells Nginx to open a port in hybrid sockets mode. The final working line is below. There are other lines in the server {} area I'm just showing the IPv6 and IPv4 line. Restart Nginx after you put the IPv6 line in.

server {
    ...
    listen      *:80;
    listen 	[::]:80 default IPv6only=on;
    ...
   }

For every virtual server after setting the default server (like above) you will just need the following listen lines that don't reference the default server or IPv6.

server {
    ...
    listen      *:80;
    listen      [::]:80;
    ...
   }

IPv6 DNS records

With IPv6 you have to use an AAAA record (quad A) instead of an A records. The DNS entry is the same but your just using 3 more A's for the new record. Update your DNS server with that record and then test it with dig. An example of that test would look like the following.

> dig @ns1.linode.com www.pantz.org aaaa 

....

;; QUESTION SECTION:
;www.pantz.org.			IN	AAAA

;; ANSWER SECTION:
www.pantz.org.		86400	IN	AAAA	2600:3c02::f03c:91ff:fe93:9678

....

Check if your site is working

After you get your quad A record entry in, people should be able to reach your website through IPv6. If you don't have an IPv6 connection you can check your sites connectivity with http://IPv6-test.com. If that website says it was successful then congrats your up and rolling. Check your webserver logs for access from an IPv6 address, then make sure the resulting code was 200 OK for that access.

Intresting things I learned about IPv6

ip6tables (iptables for IPv6) does not support NAT and likely will not support NAT.
I could not get rate limiting working with ICMP and ip6tables.
ICMP is much more important in IPv6 than it was in IPv4. From Neighbor Discovery Protocol to StateLess Address AutoConfiguration to Fragmentation.
IPv4 used ARP for translation of layer 3 to 2. IPv6 eliminates ARP and uses Neighbor Discovery Protocol to resolve IPs to MAC.
No DHCP servers are needed for IPv6 because of StateLess Address AutoConfiguration
IPv6 routers do not support packet fragmentation. Hosts on either end fragments packets.
IPV6 headers do not include a checksum.
IPv4 and IPv6 can flow through the same port on a modern operating system (like Linux) using Hybrid stack mode.
IPv6 does not use broadcasting. It uses a multicast addressing to the all-nodes link-local multicast group.
IPv6 must support a 1280-byte packet size (without fragmentation).
The cable modem standard DOCSIS 2.0 does not work with IPv6. There is DOCSIS 2.0 + IPv6 but that requires a firmware upgrade. DOCSIS 3.0 supports IPv6. Most cable modems are not DOCSIS 3.0. I asked Comcast for an upgraded DOCSIS 3.0 cable modem (when I used to lease one) and they refused becuase I don't have a high speed plan that warrants it.

Stop leasing that cable modem

2012-01-21T19:38:04Z

The bill

I got my Comcast bill again (they keep sending those things) and saw my charge for leasing my cable modem was now $7. The last time I remember looking it was $5 (close to as bad). I could not believe I was spending $84/yr to lease a cable modem. I had looked years ago at cable modems and remembered the price being close to $100. I seemed to have bad luck with them in the past and would have one or two go wonky on me at least once a year.

Leasing a modem

Since I would loose about 1 modem a year I could just have it replaced because of the lease. I figured at around $100 a pop it was worth it. If I ever upgraded my service to a faster speed and needed a DOCIS 3.0 modem it would be the same amount of money. Right now I have a plan that is well within DOCIS 2.0 speeds. For those that don't know DOCSIS 2.0 speeds are 42.88 Mbit/s down and 30.72 Mbit/s up for one channel. DOCSIS 2.0 has a maximum of 1 channel.

Buying a modem

I decided to do some research on modem prices. First I looked at DOCSIS 3.0 modems. The DOCSIS 3.0 modems are in the $75 and up range. For this price if you fry a modem a year it might be worth leasing. My internet speed plan with Comcast did not need DOCSIS 3.0 speeds so my next search was for DOCSIS 2.0 modems. I was hoping that DOCSIS 2.0 would be the older tech and that the price would have dropped by now. After searching Amazon for DOCSIS 2.0 modems let me tell you the price has really dropped. I found tons of slightly used DOCSIS 2.0 modems for $10-$20. $10-$20!! Are you kidding? I could buy 3 of these modems and it would still be saving money over leasing a modem. Hell, I could do that each year and it would still be less than leasing a modem. I looked on Comcasts approved cable modem page to ensure compatibility with their equipment, and bought one of these Comcast blessed modems (Webstar DPC2100) for $17 on Amazon.

Up and rocking

The modem came in a plain Amazon box with just the modem, power adapter, and a Cat-5 cable. I unhooked the old modem and plugged in the new modem. Then made the call to 1-800-COM-CAST, and gave the nice lady my MAC address off the cable modem. She put it in and just like that I was back on the interwebs. It has been a week with no problems. Speed tests confirm that this modem and the old modem are the same. Why did I not do this sooner? Ugh! Do yourself a favor and get a cheap used cable modem and stop paying for the leased one.

MythTV upgrade notes from Mythbuntu 8.04 to 10.04

2011-12-11T21:03:04Z

These are my upgrade notes I promised from my last post about MythTV. Please read that first before going through these, as they have information you might need pertaining to my MythTV setup. These are just notes I made during my upgrade from Mythtbuntu 8.04 LTS to 10.04 LTS. The notes were just for me but I thought I would share them. This is not a how-to! It leaves out many steps so don't think it will help you get your new MythTV box setup. Enjoy!

# Upgrade MythTV from Mythbuntu 8.04 to Mythbuntu 10.04. MythTV 21-fixes to MythTV 24.1 fixes.

# Call comcast after getting HDHR Prime and get cable card added then ask tier 2 to send a grandslam hit or a cold
# hit also called a DAC Init "hit" to the device. Channel encryption will not work correctly if this is not done.

# After or during install of 10.04 in Mythtbuntu Control Center use settings
MySQL
 Enable daily Optimize/Repair
 Enable performance tweaks
Plugins
 Turn on all plugins
 Enable mythweb passwor
Proprietary Codecs
 Check libdvdcss2 support box
Repositories
 Check active MythTV updates for 24.x box
Themes and Artwork
 Enable all of them

# Gettting HDHomerun prime 3rd tuner working
# get mythtv db password
cat /home/mythtv/.mythtv/mysql.txt
mysql -umythtv -p 
use mythconverg;
select * from capturecard;
# Look at cardid and pick next number in the series and add the same videodevice number as the other HDHomerun prime tuners 
INSERT INTO `capturecard` (`cardid`, `videodevice`, `audiodevice`, `vbidevice`, `cardtype`, `defaultinput`, `audioratelimit`, `hostname`, `dvb_swfilter`, `dvb_sat_type`, `dvb_wait_for_seqstart`, `skipbtaudio`, `dvb_on_demand`, `dvb_diseqc_type`, `firewire_speed`, `firewire_model`, `firewire_connection`, `signal_timeout`, `channel_timeout`, `dvb_tuning_delay`, `contrast`, `brightness`, `colour`, `hue`, `diseqcid`, `dvb_eitscan`) VALUES (10,'13102526-2',NULL,NULL,'HDHOMERUN','MPEG2TS',NULL,'mythtv',0,0,1,0,0,NULL,0,NULL,0,1000,3000,0,0,0,0,0,NULL,1);

# Back up old MythTV mysql DB. Then copy backup to new machine.
mysqldump -u mythtv -pyourpass mythconverg -c > mythtv_backup.sql

# Moving program data to a new database
# Extract only the data that is relevant to the programs from a database dump file

grep "INSERT INTO \`record\` "          mythtv_backup.sql > restore.sql
grep "INSERT INTO \`recorded\` "        mythtv_backup.sql >> restore.sql
grep "INSERT INTO \`oldrecorded\` "     mythtv_backup.sql >> restore.sql
grep "INSERT INTO \`recordedprogram\` " mythtv_backup.sql >> restore.sql
grep "INSERT INTO \`recordedrating\` "  mythtv_backup.sql >> restore.sql
grep "INSERT INTO \`recordedmarkup\` "  mythtv_backup.sql >> restore.sql
grep "INSERT INTO \`recordedseek\` "    mythtv_backup.sql >> restore.sql
grep "INSERT INTO \`recordmatch\` "     mythtv_backup.sql >> restore.sql

# To get inserts of our old entries working in new MythTV db we need to do this.
ALTER TABLE `record` ADD `tsdefault` INT NOT NULL AFTER `transcoder`;

# Restore the information about your programs back into the (new) database:
$ mysql -u mythtv -pyourpass mythconverg < restore.sql

# Drop column after programs are inserted
alter table record drop column tsdefault;

# Keep xscreensaver from starting. Comment out its section in this file.
/etc/xdg/xfce4/xinitrc

# Turn on optical SPDIF out 
Applications->Multimedia->Mixer->Switches Tab. Check box IEC958.

# Fix cracky audio on VLC
Tools->Preferences->Show settings "All"->Input/Codec->Access Modules->File->Caching value in ms->
500->Save 

# Get other disks mounted
# Get uuid so we can mount disks with it instead of device path. Use other entries in /etc/fstab as examples for mount. 
blkid /dev/sdb1
blkid /dev/sdd1
# Put entries in /etc/fstab 

# make new shortcuts for apps in xfce4
# put file in /usr/share/applications
# filename: firefoxnightly.desktop
# Contents: 
[Desktop Entry]
Version=1.0
Name=Firefox Nightly Web Browser
Comment=Browse the World Wide Web
GenericName=Web Browser
Exec=/home/mcsorley/programs/firefox/firefox %u
Terminal=false
X-MultipleArgs=false
Type=Application
Icon=firefox
Categories=Application;Internet;Network;WebBrowser;
MimeType=text/html;text/xml;application/xhtml+xml;application/xml;application/vnd.mozilla.xul+xml;application/rss+xml;application/rdf+xml;image/gif;image/jpeg;image/png;
StartupWMClass=Firefox
StartupNotify=true


# mythtv800x600.desktop
[Desktop Entry]
Name=MythTV Frontend Window
Comment=A frontend for all content on a mythtv-backend in 800 x 600 window 
GenericName=MythTV Viewer
Exec=mythfrontend -w --geometry 800x600
Type=Application
Encoding=UTF-8
Icon=mythtv
Categories=GNOME;Application;AudioVideo;Audio;Video
X-AppInstall-Package=mythtv

# Tried the following to fix vsync (video tearing) for ATI card. It was unsccessful.
Applications->Settings->ATI Catalyst Control Center->3D->More Settings->Wait for Veritcal Refresh->Drag to "Always On"
sudo rm /etc/X11/xorg.conf
sudo aticonfig --initial
sudo aticonfig --sync-video=on --vs=on
sudo aticonfig --fsaa=on --fsaa-samples=4
reboot
# I give up. ATI Vsync is still not cutting it. The vid tearing is awful. Put in Nvidia card. Rebooted. Ctrl-Alt-F1.
# Install nvidia driver.
sudo apt-get install nvidia-current
reboot
# From menu go to Applications->System->Nvidia Server settings and flipped on all vsync options. No more vid tearing. 
# When watching only 720p vid with the OSD on the screen and Video Texture Adaptor "Sync to Vblank" is on the video 
# and sound will stutter until OSD goes away. Will be trying new Nvida 520GT and Mythbuntu 12.04LTS to see if this fixes it.

# Get newer VLC because 10.04's version does not play webm (need to be able to play .flv)
sudo add-apt-repository ppa:lucid-bleed/ppa
sudo apt-get update
sudo apt-get install vlc
# vlc menu text was so small it was unreadable after the new Nvidia vid card install fix with 
# Option DPI  in /etc/X11/xorg.conf
Section "Monitor"
    Option   "DPI" "96 x 96"
EndSection
# Logout of desktop to restart X

# mplayer mythtv  Setup->Media Settings->Videos Settings->File Types->Extension .flv
# change command to below so we can change the speed of the video and get pitch correction.
mplayer -fs -af scaletempo

# Install repo and get libreoffice for 10.04 LTS
sudo add-apt-repository ppa:libreoffice/ppa
sudo apt-get update 
sudo apt-get install libreoffice-writer libreoffice-calc libreoffice-filter-binfilter

# Getting ssl working on mythweb
# Symlink to the to turn on the ssl for apache2.
ln -s /etc/apache2/mods-available/ssl.load /etc/apache2/mods-enabled/ssl.load

mkdir -p /etc/apache2/ssl

# Use OpenSSL generate a server key without a password:
sudo openssl genrsa -out /etc/apache2/ssl/server.key 4096
sudo openssl req -new -key /etc/apache2/ssl/server.key -out /etc/apache2/ssl/server.csr
# Now sign the certificate signing request. This example lasts 3650 days:
sudo openssl x509 -req -days 3650 -in /etc/apache2/ssl/server.csr -signkey /etc/apache2/ssl/server.key  -out /etc/apache2/ssl/server.crt

# In /etc/apache2/sites-enabled/default-mythbuntu copy everything from line: 
to line   including the lines themselves and paste a copy below it. Change the
copies port 80 to port 443 and add the extra SSL info. Now we have 2 sections.


....
SSLEngine on
SSLCertificateFile /etc/apache2/ssl/server.crt
SSLCertificateKeyFile /etc/apache2/ssl/server.key
SSLProtocol all
SSLCipherSuite HIGH:MEDIUM


# bump mem limit higher than 64m limit or we get "out of memory errors"
php_value memory_limit                  256M
/etc/apache2/sites-enabled/mythweb.conf

# In mythweb settings (key/wrench icon) -> Streaming. Uncheck "Force HTTP for streams:"

# Fix overscan issues for tv so we can fit the whole picture on the screen.
# Tv modeline for Sony KDS-60A3000
    # Sony KDS-60A3000     FF      H1   H2           H3      H4    V1     V2    V3   V4
    #                  DotClock  Hdisp HsyncStart HsyncEnd Htotal Vdisp Vstart Vend Vtotal
    # EDIE info from X server  
    #ModeLine "1920x1080" 148.50 1920 2008 2052 2200 1080 1084 1089 1125 +hsync +vsync
    # Modified line that works with tv
    ModeLine "1920x1080" 148.50 1800 1952 1996 2200 1020 1072 1077 1125 +hsync +vsync
# http://www.arachnoid.com/modelines/index.html
# http://ubuntuforums.org/showthread.php?t=1003099&page=2