Edit APT Sources

2. In the editor, add the Webmin source and save the file:
   deb http://download.webmin.com/download/repository sarge contrib

Add Webmin Source

 3. From the command prompt, download the Webmin source key:
    sudo wget http://www.webmin.com/jcameron-key.asc

Download Source Key

 4. Now, install the key:
    sudo apt-key add jcameron-key.asc

Install Webmin Source Key

 5. Update your APT repository:
    sudo apt-get update

Update the Source Repository

 6. Install Webmin and confirm:
    sudo apt-get install webmin

Install Webmin

 7. Once the install is finished, you are presented with a URL to access Webmin.  Type the URL into your browser and use your sudoer credentials to log in.  Note: If you do not have DNS setup, you may have to use the IP address in your URL instead.

Webmin Installation Complete

Webmin Main Screen

Call Widgets

If you have browsed my Contact Me page, you may have noticed that you can place a phone call to my Google Voice number.  This widget will ask for your name, phone number and whether you want to keep your number private.  When you click connect, Google Voice will dial both your phone and my designated phone and connect the call.  Both number are kept private throughout the entire process.

Read more about Call Widgets here.

Replacing Mobile Phone Voice Mail with Google Voice Mail

I currently have my mobile phone forwarding all missed calls to Google Voice Mail rather than the default Verizon Voice Mail.  I prefer this because I like to have my voice mails integrated with my Google Mail.  I also think the voice mail transcription feature is very helpful on my Blackberry.  I hardly ever listen to voicemails anymore.  I just read the transcription email Google sends me.  There are times though when the transcription doesn’t do a very good job and makes for a few giggles when you read the nonsense it sends you.

As you can see, the transcript feature can be bundles of laughs sometimes.

Read more about Google Voice Mail here.

Google Voice Blackberry App

The Google Voice Blackberry app integrates into my phone’s call features and address book.  I can choose to make a call or SMS a recipient via my Google Voice number instead of my mobile number.  When doing this, Google Voice keeps a nice little log of calls and SMS sent and received.  Placing a call this way also allows me to use some of the features that my mobile service doesn’t provide, like recording calls.

The Google Voice Blackberry App can be downloaded here.

Invites

I currently have two invites for Google Voice that I would like to hand out.  Please leave a comment on this post, making sure to fill in the e-mail address field correctly. I will draw names on 2/3/2010 to determine who I send invites to.

I searched for quite a while to find a quote that summed up my feelings about my recent experience with a full disaster recovery test.  I went into this test with measured confidence and was whipped back to reality within the first 2 hours.  The technology was not innocent and neither was I.

Encryption – Secured from external threats but what about your own stupidity?

Yes, backup encryption was the first thing to bite me in the ass during the test.  Note to self, when setting up the encryption pass phrases, check and double check that you documented them perfectly.  I think I may have mis-typed one of the pass phrases so the backup server restore procedure failed.  Luckily for me, I was able to remote in to our main office and copy over the encryption database so that I could proceed with the test.  In a real disaster, we would have been stopped dead in our tracks right here.  I now have to re-build the encryption restore procedure from scratch with new pass phrases, since I can’t know the correct pass phrases reliably in my old setup.  I am also keeping a copy of the encryption database in a safe location (just in case).

One Tape in, One Tape Out

We backup to one tape drive.  While this meets our needs on a daily basis (barely), restoring is another matter.  We spent almost 18 hours waiting for data to restore from tape.  Each job had to run serially since the DR site only had one tape drive as well.  We we’re able to restore most of the servers, however, some of the larger archive servers had to be skipped due to time constraints.  We are completely re-doing our backup and restore scheme in 2010 so our DR tape restore efficiency will be greatly improved.

Independent Disks, Not So Independent

Apparently, when a virtual disk in a VMware Infrastructure environment is set as Independent, Virtual Consolidated Backup (VCB) ignores the disk when backing up the virtual machine.  Of course, Netbackup does not report an error and shows a successful backup.  We found out halfway through the test that some of our machines would not fully restore due to this issue.  One of them ended up containing very important timekeeping information.  Post test, I have done a full visual inspection on all virtual disk configuration in our environment and have turned off the Independent Disk setting.  We are now getting proper backups of these problem virtual machines.

Not All of it Was Smoke, Fire and Ashes

Besides the 3 problems previously mentioned, this DR test was a success.  Most of our hosts were restored successfully and most of our procedures we’re validated.  Of course, the most important lesson learned and the purpose of this exercise was that we found out where our shortcomings we’re and are on the path of fixing them.  Personally, I am happy to have done this test because it exposed me to an aspect of business planning that I have not yet experienced.  Disaster Recovery felt like a giant mountain that I could never hope to scale to something a bit more manageable but still very formidable.

First off, I know I am late on my Disaster Recovery Test follow up post.  Bear with me as the holiday has limited the time I have available for blogging.  In the meantime, I have 8 Google Wave invites to dole out.  So, the first 8 people to comment on this post will get an invite!  Make sure you fill out your email address.

Here are some links to get you started with Wave once you get your invite:

Lifehacker article on running a group project on Wave

Gina Trapani’s Complete Wave Guide

Making your profile picture visible

Backups

A DR test is really a kick in the butt for backup reliability. When this project became a priority our backups were performing at an acceptable level for user error restores or the occasional server crash, but not even close for DR recoverability.

I have been tasked with getting the backups up to par so we can restore completely during the test. Now, to do this I needed to fix the small errors that cause backups to fail and tweak the windows to allow enough time for them to complete, but it really came down to knowing the performance impact of the processes that are run on each system and the backups themselves.

My company is one that actually has a higher overall resource need during non-work hours. We run so many automated processes at night that working backups around them becomes an art. In this case, a lot of communication with the process owners and developing some sort of schedule agreement is key. I met with the appropriate people and created a makeshift gannt chart to help with the visualization.  Our backups are now performing with an almost 100% completion rate.  There are still times when I need to manually re-run a job due to some scheduling conflict or system hiccup.

Workstation Disaster Recovery Image

I debated myself for quite a while on how I was going to accomplish this particular task.  We have so few computers at our company (>100) that a workstation image wasn’t something that I had created as of yet.  I needed an image that would work on all different types of HALs, have all of the required software and configure itself based on the computer’s hardware with little to no user interaction.

I ended up choosing Windows 7 simply because it was the easiest OS to achieve hardware independence, however, software compatibility was a concern.  I played tug of war with the operating system for several days but I was able to get all of our required software functional on the image.  Surprisingly, all it took was either running the installer as Administrator (even though I was already an Administrator) and/or running the installer in Windows XP compatibility mode.

The last bit, auto-configuring the OS, was probably the most fun.  Thanks to the wonderful tool, Automated Installation Kit(AIK), I was able to create an answer file that provided information such as Product Key, Registration Information and Domain Addition.  The only thing the user has to do is login and activate windows once the image is deployed.

Backup Server and Virtual Machine Restoration

Who/what backs up the backup server?  This is a valid question that can be easy to overlook, especially for those that are doing a DR test for the first time.  I have not seen all backup software under the sun but I believe the backup server can be backed up just like the client servers.  Local drives and system state can be snapshotted just like everything else.

In my case, however, things have to be done a little differently.  Symantec’s licensing model segregates every single little feature in Netbackup and we just so happen to not have the Bare Metal Restore license installed.  Instead of trying a OS build from scratch and then a restore of the system state over top, I just elected to do everything from scratch.  Everything except the catalog!

Netbackup stores all information about backups and policies in its catalog.  All Network Administrators should be taking a Catalog Backup along with all other backups.  After I get the new backup server installed, I will simply restore the catalog and will be ready to begin restoring my virtual machines.

Since we are running a VMware environment, we take advantage of Virtual Consolidated Backup to do our backups and restores of virtual machines.  VCB integrated pretty nicely into NetBackup and allows you to do the entire VM restore procedure right from the Restore utility.  NetBackup with VCB simply restores the virtual machine VMDK files to a mount disk and then fires up VMware Converter to move it into the Virtual Infrastructure Environment.

The Weekend From Hell

This weekend is our Disaster Recovery test.  I lovingly refer to it as the weekend of hell, of course!  My supervisor and I will be working from 4 PM Saturday until 4PM Monday to bring up our entire environment.  I will update next week with the results!

I feel no pressure at all

If you hadn’t noticed, I got the post up about Installing and Configuring WDS, so check it out if you are interested.

Step 1

Add the Windows Deployment Service Role using the Server Manager Tool.

Windows Deployment Services is the only server role that needs to be selected.

Choose both Deployment Server and Transport Server from the Role Services screen.

Step 2

Initialize the WDS server using the following command:

wdsutil /initialize-server /reminst:C:\RemoteInstall

Use the wdsutil command to initialize the server and set the Remote Install path.

Step 3

Configure the WDS Properties.  I only needed to change the settings in the “PXE Response Settings” and “Advanced” tabs.

Change the response policy to accept all client computers. You can further restrict this after WDS is confirmed working in this same tab.

Select "Authorize this Windows Deployment Services server in DHCP".

Apply the changes and click OK to complete.  At this point you may want to restart the WDS services or simply reboot the server.

Step 4

In order for the client computer to find the WDS server during a PXE boot, you may need to setup an ip-helper address in on your switch/router.  This enables the PXE broadcast to be seen by the WDS server if it is on a different subnet/VLAN.  On a Cisco device you can use:

config t
interface [VLAN Interface ID]
ip helper-address [IP of WDS server]
exit

Adios!

In the meantime, I will be remoting into my workstation at work to give the evil eye to my backups jobs to make sure they are running properly.  *Sigh* My work is never done.