Chase's Blog

Recipe: Slake's Chili

madsushi@gmail.com (Chase Christian) — Thu, 18 Jan 2024 00:03:00 -0700

THE CHILI

I’ve made The Chili recipe dozens of times. I’ve won multiple chili contests with it. I’ve never come home with leftovers. A double-batch fits into a large-sized dutch oven pretty nicely. It’s a real Texas chili.

However, it’s not my recipe. The original creator and namesake, Slake, takes the credit. Here’s his recipe, unedited, below:

SLAKE’S CHILI RECIPE

2 tablespoons olive oil
2 pounds sirloin steak, cut into 1-inch cubes (editor’s note: short ribs are also great here)
½ pound ground beef
12 ounces chorizo sausage, casing removed, cut into ½ cubes
1 large yellow onion, coarsely chopped
¼ cup chili powder
1 tablespoon garlic salt
2 teaspoons cumin
1 teaspoon dried basil
2 (14.5 oz) cans beef broth
2 (14.5 oz) cans whole tomatoes, drained
½ (12 oz) can Coca-Cola
½ (12 oz) can beer (your choice here, doesn’t have a huge impact what you use, just make sure it’s not too shitty)
1 cup cilantro, chopped
1 cinnamon stick
3 bay leaves
2-4 green jalapenos, slit lengthwise 3 times each (alternately, 1 habanero and 1 jalapeno)
1 tablespoon yellow cornmeal
Salt and pepper, to taste

Directions: Place oil in a large, heavy pot over medium heat. Brown the sirloin in batches. Remove to a bowl with a slotted spoon.

Add ground beef, chorizo and onions to the pot and brown. Make sure to break up the meat.

Return sirloin to the pot and stir in remaining ingredients, except for garnishes.

Bring to a boil, reduce heat, simmer for 3-5 hours (longer is better). Stir occasionally, breaking up tomatoes.

Before serving, discard cinnamon stick and bay leaves (and habanero if you used it, don’t want someone biting into that!). Garnish with cheese and sour cream, if desired.

Seriously though that chili is amazing. I usually eat it with some kind of carbohydrates, be it rice, pasta, or a baked potato. The longer you let it reduce the richer the flavors become and the less ‘sharp’ the spices are.

CHASE’S NOTES

I like to reduce the chili to a very thick consistency, and I find that a little bit goes a long way. My favorite preparation is to scoop out an avocado, fill it with chili, and top with sour cream and cheese. Also makes a great Frito Pie.

For the Coca-Cola, use Mexican Coke (real sugar) for the best effect. For the beer, I typically use Shiner or Guinness.

I’ve used a variety of meats for the “sirloin steak” portion, including short ribs, etc. Since you’re braising it so long, most beef cuts will work out pretty well here. I personally usually add more than the 2lbs quoted above, making it a little steak-ier.

I’m not a big jalepeno person, but I find that slicing them and removing them after does add a nice spiciness. I usually do 3 jalepenos, as someone who likes mild-to-medium spiciness.

Ivy

madsushi@gmail.com (Chase Christian) — Sun, 01 Aug 2021 00:14:00 +0000

My dog Ivy passed away due to an unknown illness TW: Details at the bottom of this post. on 07/27/2021. She was 7 years, 9 months old.

Writing this post over the past few days has been difficult for me. I start on one section, fall into scrolling through pictures and memories, and end up too upset to continue. Ivy was a constant in my life in Washington. Her presence was built into my rituals, into my mind. I still find myself checking for the “3rd dog” when I open the doors in the morning or close them at night. She was a comfort in times of deep sadness, and a light in times of great joy.

For those who have reached out or messaged me before this, thank you for your kindness, I have appreciated every word. I am through the worst of the grief and mourning now. I am focusing on celebrating the time that she was here. No more “sorry”s are needed.

Ivy Posts

My Two Dogs - About Ivy and Rex

My Three Dogs - About Ivy’s first litter and Lia

Puppies By Ivy - About Ivy’s litters

Guide this one, Kalahira, and she will be a companion to you as she was to me.

Ivy in 2013

Our first pic of Ivy, the day we picked her from the litter

Ivy on her way home with us. My work was closed for two weeks right when we picked up Ivy, so I got to spend a bunch of time with her when she was a pup

Ivy loved rolling over for belly rubs from a young age

Ivy’s first crate

Ivy in 2014

Ivy loved to bring home way-too-big sticks

Your browser does not support the video tag.

I’d play fetch with Ivy in the hall

We also played tug-of-war, where she’d turn into a blur

She helped me play video games

Ivy visited California with us, and got to check out the beach

Ivy’s first raincoat, for PacNW walks

Ivy’s first swim at Marymoor, biting the water

Your browser does not support the video tag.

Ivy loved rolling her back in the grass

We moved out of an apartment and into a house with a yard

Ivy and I chilling out

Ivy also loved rolling in mud

And playing fetch

Our first joint Halloween costume

My favorite Ivy memory is her head on my shoulder on a drive

She hated Christmas trees

But enjoyed naps

Ivy in 2015

We got Rex to give Ivy a buddy

They got along really well

They were best friends

Ivy showed Rex the ropes

Ivy was almost fully grown

Ivy taught Rex how to swim

And we took them to Montana

Rex also loved the mud

Another car ride

Ivy owning the foot of my recliner

Ivy in 2016 and 2017

Ivy making a face

We started our ritual of steak & eggs for dog birthdays

Your browser does not support the video tag.

Ivy and Rex were in sync

Ivy on the chair

Your browser does not support the video tag.

Ivy and Rex on the grass at our new house. You can also see Ivy using her patented “fall over on top of you” move here

Ivy on a haybail

Ivy finding the mud again

Ivy trying to beat the heat with the A/C

Ivy in 2018

Ivy had her first litter in 2018

And we kept her daughter, Lia

Ivy and Lia were close

Almost carbon copies

Although Lia could get on her nerves

Ivy’s birthday showed she had some white fur

Ivy in 2019

Mory took all the good photos of Ivy

Ivy also loved the snow, when we had it

Even after her litter, she was still a fighter

And a good mom

And a troublemaker

And still loved belly rubs

Ivy in 2020

Ivy had her second litter in 2020

With the singleton puppy, Buster

She took good care of Buster

And he was huge

After her second litter, her face lost color

But she was loved to smile anyway

Ivy relaxing

From her last birthday

Ivy in 2021

Ivy loved car rides the most

She was the queen of patience

She was a frequent guest on my office couch while working from home

Her and Lia still did the same pose

And loved rolling in the grass

Outside of the family, Ivy’s best friend was Jeff

She supervised the BBQ

And asked for some extra

Ivy would lay in the sun to recharge her gold

Ivy on the way to the vet, the last picture I took of her

Our last pic of Ivy, from our visit on July 26

Nicknames

Ivy
I.V.
Iverson
Allison
A.I.
Ivysaur
Ivysaurus
Ivybyevy
Bones
Tubba Bubba
Mama
Chocolate Chip Cookie
Cinnamon Sugar
Powdered Sugar
Fatty
Fat Butt
Lumpy
Rolly Polly
Potato Bug
Old Lady
Ives
Ives St Laurent
Boney Ives
The Queen
Loaf
Eevee
Baby Mama
Lazy Bones

Memories

If you have any pictures or memories of Ivy that you’d like to share, feel free to send them my way.

Illness Details

Ivy started medication for a thyroid imbalance in April 2021. She had been gaining weight and acting more lethargic in the prior months. After starting treatment, she rapidly began losing weight and gaining energy. After a check-up to test and adjust the dosage in June, there was an abnormal reading on the blood test for a protein. We scheduled a follow-up visit for July 12 to have an ultrasound and to check her blood again. The ultrasound showed no abnormalities, and Ivy seemed to be doing well (down 30 lbs from April/May), so we tried switching Ivy to a hydrolyzed protein (HP) diet.

In the week after the ultrasound, we noticed that Ivy was not wanting to eat much. Neither her old food or the new HP food was very attractive to her. We tried a few different types of canned food and direct food like chicken, but she wasn’t eating much. This was a big departure from the dog who would come running at the sound of anybody eating lunch. On July 20, we noticed that her stomach was also a bit distended, despite not eating much. The vet recommended that we start Ivy on steroids, which we did on July 22, to try to improve her appetite. Most dogs are very hungry when taking steroids. But, Ivy now refused to eat anything. I managed to get her to eat two bites of chicken, after quite a bit of convincing.

On July 24, Ivy’s belly was quite distended and she had not eaten anything in over 24 hours. She was also looking a little bit yellow, which suggested jaundice / icterus. I took her to our vet (open Saturday mornings) and then was referred to an emergency / specialist vet. Ivy was checked in on Saturday afternoon. They started Ivy on an IV and started monitoring her stats. Her vitals were all stable. On July 26, she had a repeat ultrasound and a bunch of other tests, all of which were inconclusive. Her liver and internals looked fine, but her bilirubin was still abnormal, her abdomen had transudative fluid, and she wouldn’t eat much at all. We visited Ivy that evening for about an hour, and she was about the same as when we had brought her in.

On July 27, Ivy’s vitals were stable. In the afternoon, Ivy had a needle aspiration of her liver. The hope was to narrow down the cause to either a GI issue or a liver issue, to guide the next steps of either a scope or biopsy. After the aspiration, at about 17:00, Ivy’s blood pressure and red blood cell count both started going down, and she was very lethargic. She had a transfusion of both red blood and plasma, which improved things for a short time, before her pressure and cells continued downward. She had a second round of transfusions, which did not help, and her breathing started getting labored. We drove straight to her. By the time we arrived at 21:55, she was nearing critical vital signs for heart and breathing rates. We had a minute to say good bye, as she was laying on the emergency table.

Our vet, the specialist vets, and the emergency vets did all that they could. We have some guesses, but will never know the final cause of her illness.

TMCon

madsushi@gmail.com (Chase Christian) — Thu, 18 Apr 2019 01:21:00 -0800

<Trademarked>

The guild I’ve led since 2007. Horde forever.

[HordeTM]

The clan I started to carry on the tradition into Destiny, and beyond.

TMCon

The LAN party we designed to bring us together.

BlizzCon

An early Lucille’s BBQ visit

The entire party room of Lucille’s

Our first house party

Those hallowed halls…

Every November, we meet at BlizzCon in Anaheim to celebrate our victories and to drink to our futures. We’ve been playing together for more than a decade, and BlizzCon was the obvious rallying call. We play Blizzard games together, everybody loves a swag bag, and there’s nothing more suspenseful than waiting for the opening ceremony to start. Chris Metzen yelling to the audience? Pure joy.

We took a vegetarian here… sorry Phil…

We had our yearly rituals: pastrami at The Hat on Thursday after picking up our bags, bagels and Red Bull and waiting in line on Friday morning, a guild dinner at Lucille’s BBQ on Friday night, and trying to find an open bar party on Saturday after closing ceremonies Shout out to the Twitch after-parties, I appreciate you. . We had a great time together every year.

There’s just one problem: Anaheim wasn’t big enough for us anymore.

As the Trademarked family grew in size, it became harder and hard to find accommodations in Anaheim. We were spread out across Disney-themed rental houses and overpriced hotel rooms. Code Enforcement was showing up any time we tried to congregate. We couldn’t even host a barbecue without the “neighbors” calling us in. Everybody in the entire damn town is a narc. We even had to drive to the next city over to get liquor at reasonable prices Shout out to the Total Wine in Brea, I appreciate you. .

Three hours of ping pong and various chips/dips

It was BlizzCon 2018’s muted celebration that finally broke me. We had to rent out a party room just to get our crew all together at the same time. Paying for craft beer bracelets, serving cold cuts, and hitting shuffle on my Daft Punk playlist was never what I imagined.

Planning

First, I want to set the record straight. There’s a reason that I don’t call it “SushiCon”, and that’s because Mory does way more than her share of the work. It’s TMCon because it’s put on by all of Trademarked for all of Trademarked, and it’s not about any one person. When I say “we” here, I mean Mory and myself, and probably mostly Mory.

Mory and I started planning TMCon right after BlizzCon 2018, in early November. We were sitting in the airport, waiting for our flight back to Seattle, and feeling like we hadn’t gotten to spend time with the people we came for. Housing and transportation restrictions ruined the community aspect. But if not BlizzCon, then what? What about just having people over to our house? Would that even work?

We ruled out trying to line up with events in Seattle like PAX Prime, where people would have to pay for tickets and flights would be full. Our house in Kent is also too far from Seattle to make it a cheap ride away. We ended up picking a regular weekend in March that coincided with Saint Patrick’s Day Improvement #1: We’re planning TMCon 2020 to fall on a natural 3-day-weekend, to allow people to stay longer. . Flights were cheap, and we could add some holiday theming to the event. March also avoided the “too hot” problem that can happen at summer LAN parties, but also was late enough that we didn’t have to worry about snow. We planned for four days, Friday through Monday, but Saturday and Sunday were by far the most-attended days. On Monday, we were reduced to just those few folks waiting on their flights.

I also made a page on our website for TMCon to be able to easily invite people!

Setup

We had no idea how many people would show up, so we posted signups in late December. We hoped that would give people enough time to buy flights and ask for time off of work. All told, we had over THIRTY attendees at the inaugural TMCon.

We broke our budget and planning up into broad categories:

Bedding
Food
Gear (PCs, tables and chairs, etc)
Swag/Branding
Other

Bedding

I’m an artist too

We measured every room in the house, and figured that we could host at least a dozen people. I originally planned for bunk beds I had a grand vision for barracks-style accommodations, but alas, it wasn’t meant to be. , but we found that high-quality air mattresses (a mix of twin and queen) were actually much more reasonable. We bought all of our bedding in bulk from Costco and Ross. I actually went to Costco for our normal grocery run, and I saw a display with jumbo pillows and blankets up front. I texted Mory and she had me fill a cart with them. We made some educated guesses about who would be staying over, and tried to assign rooms. We wanted a bed, sheet, blanket, pillow, case, and towel for every single sleeper. This had a high initial cost, but luckily, is 100% reusable for future years.

We also planned a few extra beds for folks that needed to “crash”, which worked out in our favor. Our maximum capacity was the 19 people that stayed over on Saturday night. The tankless water heater was able to keep up with the demand, and I think everybody had plenty of space for themselves. We stacked 4-5 of the beds on top of each other in the corner of the VR/upstairs media room, which worked great to convert it into usable space after everyone woke up.

Food

Hot dogs parallelize well

We planned out the meal schedule for three square meals a day: breakfast, lunch, and dinner. We had a snack and drink budget too, but we knew that meals would make up the bulk of our time and cost Improvement #2: We’re going to have more self-serve meals in 2020, to avoid any bottlenecks. .

We redid this like 100 times

We planned some of our signature meals, like Chris’ Famous Skewers, Little Mac’s BBQ, and my Smash Burgers Well, Kenji’s; I just cook ‘em. .

We updated the $100 game to be the $1000 game

We bought food in three batches: a preliminary dry food and drink run, a two-cart Costco trip to load up on meats, and then a final Costco and Safeway trek to get all of the vegetables and fresh foods. I started food prep a full week before, trimming and brining a prime brisket for corned beef. I bought a cheap used fridge off of LetGo from a family in Tacoma just to give us the extra drink and freezer space. I also might’ve had a few impulse snack buys on the way, too.

They were camping the spawn/doors

This turned into corned beef

Meals went mostly according to plan, although I think we were just a little short in a few places. We did have a cascading bread failure:

People used outdoor rolls instead of bread for sandwiches, because ?? reasons ??
People used brioche buns instead of outdoor rolls for hot dogs, since we ran out of outdoor rolls (see #1)
People used Hawaiian rolls instead of brioche buns for hamburgers, since we ran out of brioche buns (see #2)
We didn’t have enough Hawaiian rolls for the pulled pork (see #3)

Look at this beef that Chris got for skewers

Cooking with gas

We ended up pivoting on a couple of scheduled recipes We didn’t have the food schedule posted anywhere, so we kept fielding questions about it. It’ll be posted next year! , namely the chili. After seeing how much time was being consumed on preparing meals, I wanted something simpler than the all-day affair that Slake’s chili demands. I definitely underestimated the work it takes to cook for 25 people. At one point, I had my two sous-chefs, David and Scott, running all 4 burners on my stove at the same time. The food was great, it just took most of my time throughout the weekend.

Now we’re thinking with portals

I didn’t remember to get green beer

I also made a few steaks for folks after 2 AM, which is about the time when you want some meat to soak up the alcohol you’ve been drinking.

Gear

The main LAN area, 10 PCs deep

My garage was already full of computers and computer accessories from LAN parties past. I added a few more PCs and monitors to bring us up to 16 total stations. Jeff and Roman went over the top on helping me here. I wanted to be able to field at least a full Overwatch lobby, plus a few extra. We planned a few spots for people to bring their laptops or their own rigs too. I created a new wifi guest network with my Ubiquiti gear in like 30 seconds.

Amazon is the best

Like bedding, I knew this stuff would be reusable, so I bought a ton of keyboards, gaming mice and pads, and cheap headsets from Amazon. We had so many unique and random peripherals attached to each PC. As we drew up our floor plan, I saw that some 8-foot tables would fit perfectly in the computer area. I found a local rental company that brought us a stack of tables and chairs to accommodate everybody Improvement #3: We’ll be investing in our own tables/chairs for 2020. .

Shout out to Tents and Party Rents, LLC

For consoles, I reached out to my coworkers, who came through with a ton of great gear. We had fully kitted out Xboxes and XB1s, more Switches and pro controllers than I could count, and a full room-scale VR setup. I kept everything straight by putting a small piece of colored masking tape on each peripheral, one color per donor. I just bought a big pack of different colored rolls of tape from Amazon. We also borrowed some board games, and I bought a few party board games to round out my collection. Codenames was particular fun to play with a group like ours.

Room vs Room LAN battles

I had two console rooms set up: one on the main floor, and one upstairs. However, we also used the upstairs room for VR, since it was out of the way and had a good layout for it. Once we had the VR experts arrive and get the equipment set up, it was Superhot and Beat Saber the rest of the weekend. I wasn’t convinced originally that we should even have VR, but it was one of the stars of the event.

I had plenty of power and networking cables, save for being a single power strip short. My one and only Target run during TMCon was to grab that missing item so that we could get Artemis set up properly Improvement #4: We’ll have a more dedicated Artemis / collaborative gameplay room next year. . Random HDMI-DVI cable for the VR PC? Got it. 500’ box of CAT-5E to run my own cable lengths? Got it. Weird power cable for the Xbox? Got it. Enough USB and lightning chargers for a few dozen phones? No problem. Bad SATA cable in the PC? Check the bin.

Don’t trip

The early arrivers did the lion’s share of the setup work. Ryan and Chris were working together to set up PCs, just like we all did years ago. They imaged the PCs, hooked up the peripherals, tested configurations, set up networking, pre-installed games, and ran the cables. We only flipped a breaker twice the entire weekend!

One thing that didn’t work out great was music. I had a dedicated Spotify iPad and a shared TMCon playlist set up, and a bluetooth speaker to provide good quality sound, but nobody seemed interested in adding any music. I ended up just putting it on different Spotify “radio” stations each day and letting it ride. Not as useful as I thought. We also never got a full room vs room console match going, since VR dominated the interest in the second console room.

I also upgraded to gigabit internet. You know, just in case.

Swag/Branding

The official TMCon logo

I wanted to go big. Every year for BlizzCon, we bought everyone in the guild t-shirts. With the short timeline to the event and the drop-in/drop-out nature of the RSVPs, I didn’t want to commit to shirts. Instead, we went with a guild cup.

Mory designed the logo, and I started shopping around to short-order sites. In addition to the cups, we considered:

Pens
Bottle openers
Magnets
Water bottles
Umbrellas
Etc…

I also ordered a vinyl sign to mark the house, which I think turned out great. I also purposefully left the year off the sign, so we can reuse it next time!

Bacon, attendee #1!

We also wanted to capture some of the BlizzCon feel at TMCon, so I grabbed some red lanyards and badge holders. Mory worked up a template for the badges, and I printed them out on cardstock. We even made custom vendor and admin badges to give it some more flavor. Because we had many different groups all colliding at TMCon (coworkers, friends, family, WoW folks, Destiny boys, etc), I wanted people to be able to mingle. We also put the wifi password on the back of the badge, so we wouldn’t have to field questions about it.

We had these both posted and distributed throughout the house

I also printed up our House Rules to distribute to attendees. Since this was the first TMCon, I wanted to make sure that we set the stage right away. The rules were both simple and loose, but got the point across Improvement #4: There won’t be a ‘kids day’ at TMCon 2020, so drinking can start immediately. .

Complete with real hunted feathers

Witty also crafted us a custom Horde banner, that we’ll proudly display in our home!!!

Other

We made the specific decision for the inaugural TMCon to avoid any external activities. Driving people to and from the airport would be the only time we left the house. We’d focus on keeping people engaged and entertained, instead of trying to schedule events and wrangle cats and figure out payments. People were free to leave (and some did!), but the vast majority of folks just hung out all weekend with us.

We helped to coordinate a few flights, to make sure anybody that wanted to attend could come. I had been stashing my airline reward miles for just such an occasion anyway. We figured out the schedule of when everyone was arriving, and grouped people up into time slots to minimize the airport trips where we could.

TMCon 2019

The event itself was amazing. We had great late-night discussions until 4-5 AM every night, and we all rallied for breakfast by 10. We played video games together, side by side. We got to accuse each other of cheating at Counter-Strike and scream at our Helm team to JUST DOCK THE DAMN SHIP.

We tasted Rich’s generously donated beers and QTFOOTS’ curated whiskeys, we played a D&D one-shot full of mystery and betrayal, and we jammed the fuck out on Beat Saber. We even got some OG Halo in for good measure. My dogs got more pets than they’ve ever gotten in a single week.

TMCon 2019 was the party that I always wished we could have at BlizzCon. It was dozens of guildmates and allies playing together, sharing meals, and kicking ass. It was four days of friendship, built around games.

My wife, my sister, my coworkers, my former coworkers, new local friends, and my clanmates from another timezone were all at the same table.

We’ve been playing together for over a decade. Logging in every Thursday and Sunday to play WoW, grinding power in Destiny to hit the raid on Tuesday, strategizing our Overwatch teams, rerolling for sockets in Diablo, hotdropping in every battle royale with parachutes, and everything in between.

And we’re just getting started for next year.

Orange 2, baby.

Austin, TX - 2018

madsushi@gmail.com (Chase Christian) — Fri, 11 Jan 2019 01:21:00 -0800

I love BBQ.

In 2017, I started thinking of making a trip to all of the great BBQ cities in the US. Rather than trying to visit all of them in one go, I decided to dedicate a few days to each. The first city on my list was Austin, TX. I researched the best BBQ places in the city and surrounding area, tapping on knowledge from friends and coworkers who had lived in Austin. I had a list of 9 places to try, and planned a week to hit them all. One of the challenges was that many BBQ places are only open for lunch, as they’re out of meat by mid-afternoon.

Texas BBQ

Texas BBQ is simple: salt and pepper. There’s no sauce needed to complicate things Although I did have some great sauce. . A healthy dry rub with fresh-ground pepper and a several hours of indirect heat is all you need.

Brisket is the signature of Texas BBQ. If your brisket isn’t good, sorry. As I went about the task of categorizing and rating all of the BBQ that I ate, brisket always pulls the most weight. I tried the brisket everywhere that it was offered.

Texas BBQ also has a certain style. A lunch tray with butcher paper is how we were served at nearly every establishment. Pickles and white bread are always available. Everything is dished out by the pound, which makes it easy to gauge exactly what you’re in for.

la Barbecue

2027 E Cesar Chavez Austin, TX 78702

One of the few BBQ places owned by women, la Barbecue was my first stop. It’s located in Austin proper. One of the owners hails from the Mueller family line of Texas BBQ, but they do it differently at la Barbecue.

My recon had told me that the chipotle coleslaw was a signature item, and my sources were right. It was one of the best side of the whole trip, right off the bat.

The brisket here was great, with melty fat. This is why you get up early and make sure you’re in line at opening: you want the best possible brisket. Wait too long, and the fat will set again.

The best item on the plate, however, was the beef rib. I was surprised, because I hadn’t heard that they were known for their beef ribs.

It was cooked perfectly, had great flavor and bark, and we savored every single bite. It was our favorite beef rib we had on the entire trip.

Rating: 4th overall (#1 beef ribs)

Alamo Draft House

I’ve been hearing about Alamo Draft House from friends all over for years, and I finally got to visit one. I watched Mission Impossible: Fallout while Mory went shopping at some art stores. I love the custom trailers and intros, it was very cool. They also make a great dark and stormy.

Salt Lick BBQ

18300 Farm to Market Rd 1826 Driftwood, TX 78619

Salt Lick BBQ was easily the most controversial of the restaurants we visited. Half the people I asked said that I had to go, the other half tried to wave me off. We decided that it was worth the drive out to Driftwood to give it a shot, especially since they were open for dinner, which made it easy to schedule.

The brisket at Salt Lick was just OK. It didn’t have the great bark or melty fat of the top-tier places. The house-made sausage was good, but not great.

It would’ve been a disappointing trip if it weren’t for their pork ribs. They had an amazing sweet glaze that had me ordering an entire second plate. These ribs took the top spot for pork ribs on my trip. Perfectly cooked and absolutely delicious. Worth it.

We got a variety of food from Salt Lick, including sides and dessert. The pecan pie and cobbler were both delicious! However, this is Texas BBQ, so the only thing being rated is the meat.

Rating: 8th overall (#1 pork ribs)

Franklin BBQ

900 E. 11th Austin, TX 78702

Franklin BBQ is as close to a Mecca as there is for BBQ fans. It was the spot that made Austin the first city on my must-visit list. It’s one of the highest rated and acclaimed restaurants in the country, and the lines prove it. We got there bright and early and still waited in line for hours. We met a couple in line that had also traveled from out of state, just for a taste of the brisket. We were visiting after a fire had shut down the restaurant for months in 2017, but they were back and smoking in August, one year later.

Let’s just open with the brisket: it’s the best in the state. It lived up to my wildest expectations, and then some. The bark was crisp and flavorful, and the fat was the melt-in-your-mouth meat-butter that comes from a perfect cook. When I think back to this trip, the first flavor I remember was licking my lips at Franklin, and tasting the pepper.

The turkey here was also a sleeper hit. We tried the smoked turkey at several places, and Franklin’s was tops. It was savory and buttery in all the right ways, and it was one of Mory’s favorite items from the entire trip. Definitely worth getting a few slices to mix it up with the brisket.

The pulled pork was similarly delicious, with a light squeeze of sauce amping it up. It was my 2nd-favorite pulled pork on the trip, with no complaints about the flavor of the meat or the cook. It was great on a fresh bun.

The pork ribs were the only thing that was less-than-perfect. Not bad by any means, but simply not up to the level of pork ribs we found elsewhere. I’d double-down on more brisket or turkey next time instead.

Rating: TIE - 1st overall (#1 brisket, #1 turkey, #2 pulled pork)

Rudy’s Country Store

2451 S. Capital of Texas Hwy Austin, TX 78746

The only real “chain” BBQ place we visited was Rudy’s Country Store. Frequently colocated with gas stations, you can find Rudy’s all over the place in Austin. They were another easy choice for dinner, as they’re open long hours and won’t run out of meat. The prices here are also very cheap, comparatively.

The creamed corn at Rudy’s deserves its own paragraph. While they have a ton of sides, none are as delicious as the creamed corn. It was recommended to me by friends, but they didn’t hype it up enough. It’s exactly what you want creamed corn to be: buttery, and you can tell there are corn chunks in it.

Where Rudy’s fails is the brisket. It was easily the worst brisket we had on the trip, with zero bark and way too much smoke. I didn’t expect much from a place that sells brisket “all day” and no guarantee of freshness, but I’ve had leftover brisket better than this.

There were bright spots, though. The pulled pork was actually amazing, especially with Rudy’s special “sause”. This sauce is legit. I know it goes against the spirit of Texas BBQ, but that sauce elevated their pulled pork to #1 in my rankings.

The prime rib was also delicious, and it was another of Mory’s favorite items. I’m not sure that prime rib counts as BBQ, but it was great anyway.

Rating: 7th overall (#1 pulled pork)

Black’s Barbecue

215 North Main St Lockhart, Texas 78644

After spending our first couple of days in Austin, we ventured out to Lockhart The official Barbecue Capital of Texas to visit two of the oldest and most storied BBQ places in the area. Black’s Barbecue claims to be oldest major BBQ restaurant in Texas, and they might be. The Black family has run the place for over 80 years now.

We got to Black’s a few minutes before opening, and waited out front. We were the first customers of the day. They have a unique setup where you can dole out your own sides and then get the tally at the register.

The brisket from Black’s was just OK. It was the driest cut we ever got, and despite good flavor, didn’t have the cook quality that we saw elsewhere. This might be a style thing, and I missed it, but it didn’t meet what I was looking for.

The beef ribs here are absolutely massive. Be prepared for it to be well over 1 lb in size. This was our favorite item on the menu. The custom sausage blend was unique, and in the better half of sausages that we had. The turkey was unfortunately the worst we tried as well.

Somewhere had to be last, and despite high expectations, Black’s was the worst of the nine spots we visited. Don’t get me wrong, I’d happily eat there again. But when you’re comparing it at the altar of the best Texas BBQ… it falls a little short.

Rating: 9th overall

Kreuz Market

619 North Colorado Street Lockhart, Texas 78644

We had planned to wait a few hours before going to the next spot, but we were still pretty hungry after eating at Black’s. Kreuz Market is just around the corner in Lockhart.

Their aesthetic is simple: no plates (just butcher paper), no forks, no salad, and no vegetarians. It’s one of the few spots where you come right up to the pit to order your meat.

Kreuz was refreshingly good BBQ. I didn’t want the trip out to Lockhart to be for nothing, and Kreuz made it worth the drive.

Their brisket was very good, and fresh off the pit. It was hot and smoky and exactly what I needed. They also had a special here, “shoulder clod”, which is similar to a pulled pork. It was great!

The pork ribs here were the best dry pork ribs, and second only to Salt Lick’s sauced ribs overall. I really enjoyed them. They had a solid bite and some great flavor. The sausage was also savory and great.

They also had prime rib here, which Mory said was great, but not quite as good as Rudy’s, as surprising as I found that. All of the sides were tasty, although also all were spicy and had some kick.

Rating: 6th overall (#2 pork ribs, #3 sausage)

Terry Black’s BBQ

1003 Barton Springs Rd Austin, TX 78704

Despite the name, there’s no Terry running Terry Black’s BBQ. Terry Black, from the Black family line of Texas BBQ, became a CPA in Lockhart, with his office right across the street from the family BBQ joint. His twin sons, however, have carried on the family business in Austin. Despite some disagreements with their kin in Lockhart, the heritage is hard to deny. Their mother’s side of the family is also related to Smitty’s Market in Lockhart, the one place I didn’t get to visit on this trip.

After driving back to Austin, we were going to Terry Black’s for dinner. As I mentioned, many BBQ places don’t even do dinner, since they’re out of meat after lunch. Terry Black’s is much more of a traditional restaurant-style location than anywhere else we went, with a very different aesthetic. You still take your tray up to the butcher and order by the pound, but the rest was much more refined.

Of course, this had me worried. Who knew when this meat was cooked? Surely it can’t be the same quality as right off the pit, in the late morning, right?

My fears were unfounded. Terry Black’s has the best BBQ in Austin, outside of Aaron Franklin’s brisket.

Their brisket can go bite-for-bite with Aaron Franklin’s, only slipping to 2nd place due to Franklin serving it just a little bit fresher. I honestly can’t believe how dang delicious this brisket was as dinner time. It’s a miracle. Perfectly melty and with a bark to die for.

Their pork ribs had a great bark as well, which is one of their clear specialties. They had a better bite than anywhere but Kreuz. The jalapeño cheddar sausage was the best sausage of the trip, hands down, no question. I went back for more brisket and sausage after under-ordering my first time through. Their sweet BBQ sauce was another tasty surprise. Franklin had better turkey, but only by a small measure.

Also, every dang side was perfect. We finished every single bite on the tray.

The meal we had at Terry Black’s, during dinner, with nearly no waiting in line, and no driving, was spectacular. If anything, it’s the underappreciated gem of Austin. If I lived there, I’m sure I’d get up early and take my tourist friends to Franklin BBQ, bud I’d eat every week at Terry Black’s. Eating out on the patio on a nice summer evening was divine.

Rating: TIE - 1st overall (#1 sausage, #2 brisket, #3 pork rib, #3 turkey)

Bats at Congress Avenue Bridge

One of the reasons we picked August for our trip was the annual season of flights of the bats that live underneath Congress Avenue Bridge. We staked out a spot in the grass early, and waited until sundown to see millions of bats flying all over the place. It was a very cool experience, and only a couple of blocks from Terry Black’s. It got way more crowded as dusk approached.

Snow’s BBQ

516 Main Street Lexington, Texas 78947

All the way out in Lexington, Snow’s BBQ was a late addition to my list. A coworker recommended them at the last minute, and the reviews looked too good to be true. They’re only open on Saturdays, which meant rearranging the schedule and waking up real early to get out there in time, as they open bright and early.

It was worth the long drive. We got to see some more of the Texas landscape, and the drive was easy.

The line, however, was not. It was longest line we had to wait in anywhere, even Franklin BBQ. The upside is that there’s free beer while you’re waiting in line.

I can say that the Lone Star that was supplied was the most beer I’ve had before lunch just about ever. The line is even an event in itself. The pitmasters would come out and have people count off and tally up, and then they’d mark certain spots in line. There was a raffle where if they picked your number, you got shuttled right to the front of the line. No such luck for us.

After a few hours in the morning Texas sun, we got to the humble counter. We asked for one of everything.

Snow’s brisket is top-tier. A little lighter on smoke than the competition, but with tons of beefy flavor and tender fat. It was a refreshing change of pace, and great in its own right.

The pork shoulder steak was much smokier, and the contrast between the two made for interesting eating. I really enjoyed it, although I generally prefer pulled pork to a pork steak. Their sausage was also very smoky and had a nice crunchy texture on the exterior. The turkey had a great herby flavor, but was just a bit dry. My one regret is that we were literally two spots away from getting pork ribs, but instead we watched them run out for the day.

Snow’s was well worth the drive, especially for the brisket and unique sausage.

Rating: 3rd overall (#2 sausage, #3 brisket)

Louie Mueller’s BBQ

206 W. Second Street Taylor, TX 76574

The Cathedral of Smoke, Louie Mueller, was our final stop. It was Saturday, around lunch time, after an early start at Snow’s. College football was on every TV, with Texas in a tough match.

Louie Mueller is a classic, in both good ways and bad ways. The building is several decades old, and it was incredibly hot and humid inside. I wasn’t sure if we were being smoked, or the other way around. No beer in line here, but there were coolers with soda, at least. We were cheered on by our friend, Guy Fieri.

Louie Mueller knows how to cook brisket. It was incredibly moist, but with a coarse bark, which is tough to pull off. I really enjoyed it.

My notes for the trip had just one item on the list for Louie Mueller: beef rib. All of my friends had recommended it, and I wasn’t going to let them down. The beef rib here was Mory’s favorite, and worth the stop for sure.

The pulled pork at Mueller could’ve used more smoke for my tastes, but was delicious with some sauce. The turkey here was actually my favorite, and so I’m glad I added a few slices on a whim. We filled up on meat here, as it was our last chance before heading to the airport. As the Longhorns fell further behind, we decided it was time to leave.

Rating: 5th overall (#2 beef rib, #2 turkey)

Conclusion

We had an amazing time in Austin, with nine of the best meals of my life. I’d visit again in a heartbeat. Next up, Carolina.

My Three Dogs

madsushi@gmail.com (Chase Christian) — Thu, 22 Nov 2018 01:21:00 -0800

I have three dogs.

Their names are Ivy, Rex, and Lia.

Lia is Ivy’s daughter, and our youngest puppy. She loves bananas, and she loves to bring you things. You can see the pride in her eyes when she presents you with a gift. She has zero concept of personal space, and will happily hop right on top of Ivy or Rex on the couch.

At one point in February 2018, I had nine (9) dogs, which is a lot of dogs to have.

Where’d you get that dog? We’d LOVE a Golden Retriever like that!

It’s hard not to love a Golden. As our friends and our family met Ivy and Rex, several of them asked for our breeders’ contact info. They also let us know that if they were left alone with the dogs for long enough, we might not get them back. We heard stories of people who grew up with Goldens, and wanted that same experience for their kids.

We took all of these comments to heart. Ivy was not spayed young, and so we talked about her having a litter of puppies. Ivy is a beautiful and textbook Golden, her test results had all been positive, and her personality and looks had already won over everyone who had met her. I have a little bit of experience with dog breeding, as my mother had been a small dog breeder in my last few years at home. I know all about the whelping process, vaccines, necessary care, and the like. My wife also had experience with animal birth, but from the feline/kitty side.

We talked with a few of our family and friends who were the most serious about trying to kidnap Ivy or Rex, and confirmed that they’d be interested in a puppy. Our #1 goal was making sure that any and every puppy would have a loving home.

Finding a stud

The first step towards a litter of puppies was finding a suitable father, or stud. Rex is neutered, but more importantly, his coat is very thin and not fluffy. It doesn’t make us love him any less (in fact, he’s much easier to wash!) but he is not a great example of the breed. We searched the classifieds for stud ads, and found a family nearby with a very fluffy stud, Howdy.

After ensuring that Howdy and Ivy were not related (thanks to K9Data), we scheduled to for the two dogs to meet during Ivy’s next fertile cycle. When the time was right, in mid-December, we dropped Ivy off with Howdy for a weekend. Dogs have a very high fertilization rate, so our chances were good. We’d know in just a couple of weeks if Ivy would have puppies.

Pregnant with puppies

Dogs have a very short gestation period of only about two months. Ivy was growing fast, and so it was important to get her the proper nutrition in advance. We got an xray at our vet and found out that Ivy was expecting 7 puppies. That’s an average-sized litter for a Golden. Ivy was in a litter of 13 puppies when she was born!

We prepared an unused area of our house for Ivy’s personal puppy zone. It had a dedicated wall heater to keep it warm, and I put down layers of black landscaping plastic to protect the floor and baseboards.

I also built a whelping box for Ivy and her puppies, as that’s very important for her to feel secure that her puppies are safe.

Seven puppies

By the time Ivy’s litter was born, all of the puppies were claimed, and we already had a “waitlist” of people who wanted their names down for the hypothetical next litter.

A friend of mine gave me the idea of setting up a puppy cam to livestream the puppies. I thought this was a great idea, and bought a webcam and tripod from Amazon. I used a spare old laptop and set up a YouTube live stream of the puppy pen. We ended up streaming from 10 AM to 10 PM nearly every single day of puppyhood (about 8 weeks). All of the future puppy owners were able to watch their chosen puppy grow up and interact with Ivy! It also let me watch them obsessively at all hours, including checking in on them at night and while on break at work.

Of course, I also had to set up a website, Puppies by Ivy, to share updates and pictures of the puppies. I weighed each puppy every single day to track their weights and growth, to make sure nobody was falling behind, and charted everything in a Google Doc. The future puppy owners loved to see their puppies making progress and jockeying for the Biggest Dog title!

Work

Caring for puppies was a lot of work. My wife and I were running the laundry machine all day, every day, to keep enough clean bedding for the puppies. Poop cleanups had to happen on a regular schedule, as well as feedings, weigh-ins, and veterinary care. The workload only grew when the puppies began getting big enough to work together to escape the puppy zone, and so we had to work on reinforcing the pen and keeping them corralled. A breakout would take a long time to recover from: chasing down seven tiny puppies, and cleaning up any messes they made while on the run.

The little things

We wanted to give the puppies and future owners the best experience possible, so we tried to do all of the little things we could. We gave each puppy a velcro collar with a particular color, and built those into the stream overlay, so people could follow their particular puppy’s antics. We also had matching Wonder Walkers for each dog to keep the color scheme going. We took nice photos of each puppy individually once per week, so that you could see their growth and development week over week. We made sure all of the AKC paperwork and microchips were done ahead of time. We also hosted a couple of puppy meetup days, where the future owners could bring their family and friends to meet the puppies and help them get socialized.

Loving homes

We were able to accomplish our goal of giving every puppy a loving home. Three went to our families, two went to coworkers with older Goldens at home, and one went to a nearby friend You have no idea how much I wanted to make a “Three rings for the Elven-kings…” joke here. . We’ve been able to meet up with the puppies and owners and see how great they’re doing. The final puppy, Lia, stayed here with us.

Overall, it was a lot of work, but it was worth it. As my veterinarian told us when we asked him about Ivy having a litter: “The world can always use more Goldens.”

Found Disc Courtesy

madsushi@gmail.com (Chase Christian) — Sun, 31 Dec 2017 11:21:00 -0700

Everyone who plays disc golf will eventually lose one of their discs. A ravine will be too steep, a tree will be too tall, a thicket will be too thorny. Water is another popular resting place of lost discs. It happens. Traditional golfers lose balls too, but discs are more personal and also more expensive. It’s closer to losing a club than losing a ball.

Most disc golfers will write their name and phone number with a Sharpie on the underside of their discs. The hope is that if the disc is ever recovered, the finder will give them a call and they’ll be reunited with their disc. Maybe a disc falls out of a tree after a wind storm, a thicket gets cut back by a maintenance crew, or somebody goes diving to the bottom of the river. We imagine the stroke of fate that would be involved in our disc returning to us! The prodigal disc arrives at our door, hungry and cold, and back to our loving embrace!

Unfortunately, that’s rarely the case. Many discs are truly lost. The few that are found don’t always make it into the hands of someone altruistic. As anyone who has lost a marked disc will tell you, phone calls are rare. You have to learn to “lose” the disc as soon as it’s gone. If you do happen to get a call, and I’ve received a scarce few, it’s pure serendipity.

Here are my rules of found discs:

If you find a disc, you have to call the number. Not once, but twice. Leave voicemails. Wait 24 hours between calls. Texting is also pretty popular nowadays. I think one text and one call would work too. The key is to give the owner a fair shake. One missed call from a random number isn’t enough to act on, but two calls and voicemails is a pattern.

If you make contact with the owner, and they want their disc back, you’re not owed anything. Don’t expect a reward. Don’t ask for a reward. I won’t go out of my way to return a disc. If someone asks for me to drive it by their house on the other side of town, no thanks. If they want to meet at the course it was found at, sure, let’s plan a time. A bar or restaurant near the course works too.

If you’re the owner, and someone contacts you with a found disc, you have two options. If you don’t really care about that disc or have already replaced it, let the finder keep it. If you do care about the disc, for whatever reason (financial, sentinmental, etc), you can ask for it back. Try to find a time and place that’s convenient for the finder, as they’re doing you a favor. I like to offer the finder a gift: a beer (if we’re meeting somewhere with beers), a disc (I’ve always got a few spares or dupes), or a few bucks. Found discs can be an opportunity to meet a fellow disc golfer, and you can make friends if you set up a playthrough to give a found disc a reunion tour.

Let’s say you are unable to get in contact with the owner. My personal rule is that I will never sell a found disc. If I’m going to make it a part of my rotation, I’ll add my number to the bottom of the disc with a Sharpie. Both numbers will be on there, and if I lose the disc, it’s a 50/50 shot that I get a call first. If the owner lets me know they don’t want it anymore, I’ll black out their information and add my own. I don’t like the idea of trying to remove their information with acetone or whatever other chemical. The previous owners of a disc are part of its history.

One final note on losing discs: you’re going to lose discs. While it might seem heroic to strive and climb and crawl to save your disc, it can also be a big inconvenience to others in your party or on the course. If I can see my disc, but it’ll take a while to get to, I’ll try to finish the round with my party and come back on my own time to retrieve it. If I’m just searching for my disc, I will put a maximum time on how long before I’ll decide to move along and keep the round going. Nobody wants to be stuck on hole 6, waiting for you to search every inch of the brambles for your $15 disc.

Recipe: Shepherd's Pie

madsushi@gmail.com (Chase Christian) — Fri, 01 Dec 2017 16:03:00 -0700

One of my very favorite restaurants, Fresco Cafe, has daily specials. My friends and I used to call into their specials phone line every morning, a little after 10. You’d hear a recording of someone reading off their menu board. Towards the end, they’d list their daily special entree. They had a lot of great specials, like their barbecue pork tenderloin sandwich With both pork tenderloin and bacon on the same sandwich, we called it the “pork vendetta”. or their filet mignon enchiladas. But some days were different. On those rare and precious days, the entree would be shepherd’s pie.

Shredded beef braised with wine mixed with an assortment of vegetables, smothered with lightly browned mashed potatoes. Served in a ceramic dish, their shepherd’s pie would have the entire office making lunch plans. We’d reschedule meetings, plan out our paths to cross by Five Points around lunch, and try to find a big enough table.

MY RECIPE

After moving to Washington, my trips to Fresco became a lot less frequent. To fill that void, I started working on my own recipe for shepherd’s pie. I don’t eat lamb and my wife hates mushrooms, so many traditional recipes didn’t work. After a few months and many attempts, I ended up with a recipe that tastes close enough to Fresco’s but also stands on its own. It’s become a family favorite during the rainy season here That is to say, almost year-round. .

Shepherd’s pie consists of two main parts: the meat and the potatoes. First, the meat.

MEAT

I started with Alton Brown’s recipe as a base, and went from there.

1 cup onion, diced (I use a yellow onion)
1 cup celery, chopped (you can leave this out if you hate it)
1 cup carrots, chopped (I use baby carrots: no peeling)
2 cloves garlic, chopped
+++++++++++
2 lbs meat (I mix 1 lb ground sirloin and 1 lb sirloin steak cut into cubes, any beef mix works)
+++++++++++
2 tbsp flour
+++++++++++
2 tsp tomato paste (be very careful not to add too much)
1 cup broth (I actually use chicken broth here, more flavor)
0.5 tbsp worcestershire sauce
1 cup wine (red wine, I use merlot typically)
+++++++++++
2 tsp thyme, chopped
2 tsp rosemary, chopped (I usually get the “poultry pack” of fresh spices from Safeway/Albertsons)
+++++++++++
1 cup peas, frozen (you can leave these out… but why?)

I use a big stainless steel pan with straight sides. Add some oil to the pan, and set to medium-high heat.

Add the onion, celery, and carrots. Stir and cook until starting to brown, a couple of minutes. I usually give the onions an extra 30 seconds alone before adding the rest. Add the garlic, continue to stir.

Add in the meat. Add salt and pepper, but not too much. You’ll have time for flavor adjustment later. Brown the meat, stirring. Sprinkle the flour over the mixture.

Once you’re happy with the meat’s browning, it’s time to add the liquids. Put in the tomato paste, broth, worc sauce, and wine. Stir well. Once it’s combined, add in the thyme and rosemary.

Bring the entire mixture to a boil. Once you see the bubbles, reduce to medium-low and simmer. As it cooks, the mixture will reduce some. After 10-15 minutes, the meat should be breaking down and the vegetables should be soft.

Now is the time to adjust seasoning. I’ll add salt, pepper, thyme, or rosemary as needed between taste tests. Once you’re happy with the flavor, toss in the peas and then pour it into your 11”x7” glass pan.

POTATOES

I started with J. Kenji López-Alt’s recipe as a base, and went from there. You probably want to start the potatoes first when you’re making the recipe.

4 lbs potatoes, cubed (russets usually, but yukon golds work too)
2 cups half-and-half (or any cream)
1.5 sticks salted butter (Tillamook here)
+++++++++++
1 handful parmigiano reggiano cheese, shredded

I peel and cube the potatoes, and rinse once in cold water after cubing.

I fill a pot with water, and bring to a boil. I don’t salt the water.

I add in the potatoes, and cook for 15 minutes.

I drain the potatoes in a strainer, and only give a quick rinse with hot water. Just a quick spray from the hose faucet. Make sure you shake the potatoes dry afterwards.

Mash the potatoes, and add salt and pepper to taste. Avoid adding too much liquid, as you don’t want the potatoes to be runny. They’ll mix with the meat if they’re too runny. I make sure I like the taste with just salt & pepper, then I add the cheese in.

MEAT & POTATOES

With the meat on the bottom, add the potatoes carefully to the top of the meat. You want to avoid any mixing. I spoon the potatoes on gently, then use a spatula to spread evenly. Once you have a smooth surface, use a fork to make spikes and peaks for extra browning in the oven. Sprinkle with any remaining cheese for a little something extra.

Place the baking dish on a cookie sheet (in case of spillover) and cook for 20-25 minutes at 400°F.

Then, you eat it.

Comcast and DHCP

madsushi@gmail.com (Chase Christian) — Sun, 23 Apr 2017 15:03:00 -0700

Towards the end of March, my internet connection started acting flaky. I noticed it happening during WoW raids, and my wife caught Netflix and Spotify cutting out during the day. The internet would drop for a minute or so, and then come back online. It was never out for very long, and never in close succession.

I chalked it up to Comcast doing some maintenance work in our region. However, after two weeks of drops that seemed to be becoming more frequent, I decided to dig into the issue. Checking the logs on my cable modem, I never saw any loss of signal or connection, which ruled out any type of physical issue. Simple ping checks confirmed that it wasn’t an internal connectivity problem. The issue was somewhere between my firewall A Palo Alto Networks VM-100 and Comcast’s upstream router, at a logical level.

Logs don’t lie

The revelation came when my wife noticed a pattern in the disconects. Our Mumble voice chat server logs every time someone joins or leaves. She had been disconnected several times that day, and saw in the join/leave logs that each disconnect occurred about 1 hour apart. Example logs:

2:05pm - Mory has left the server

2:06pm - Mory has joined the server

3:06pm - Mory has left the server

3:07pm - Mory has joined the server

This gave me further confirmation that it was a logical issue, as physical problems will rarely manifest in such a timely fashion.

Armed with the knowledge of when the next outage should occur, I started monitoring all of my logs. I had my firewall’s log monitor open, I had pings running to a variety of sources, and I had Wireshark capturing my packets.

The next disconnect arrived exactly as predicted. I immediately noticed a flurry of activity in my firewall’s logs. As my pings started dropping, my firewall was negotiating its DHCP lease with Comcast’s DHCP server. The firewall was unable to renew the lease on that interface, it killed all active sessions and started the DHCP lease process from the beginning. This was the ultimate cause of the internet drops.

DHCPNAK

After doing some more investigation, I learned the details of Comcast’s DHCP process. My firewall would request an initial lease, and would receive the same public IP address from Comcast’s DHCP server (which happened to be in Oregon 76.96.94.198 ). The lease was for 7200 seconds, or 2 hours. DHCP clients will typically wait until their lease is halfway through Per RFC 2131 , and then they’ll try to refresh proactively.

My firewall would send the DHCPRENEW packet, to initiate the refresh. Comcast’s DHCP server would reply back with a DHCPNAK NAK means ‘negative acknowledgement’, as opposed to ACK for ‘acknowledgement’. , which indicates that the server did not accept the renewal request. That caused the refresh to fail, and my firewall to dump the lease and try a DHCPREQUEST from scratch. When my firewall fell back to requesting a “new” lease, the process succeeded. The strange part is that the Comcast DHCP server would respond to the DHCPREQUEST with the same IP from the original lease.

Unfortunately, it seemed like my firewall was acting properly, and Comcast’s DHCP server wasn’t respecting the renewal request. There’s no way to turn off DHCP renewal requests on my side (for good reason), and I didn’t have any visibility into Comcast’s server side. When one side of a protocol doesn’t work in the manner specified by the RFC, all bets are off.

Comcast Support

At this point, I tried calling Comcast tech support and explaining my issue. However, despite multiple attempts on the phone, I couldn’t get any help beyond a tech trying to reboot my modem to fix the problem. I tried explaining that the issue was occurring on a regular basis and that I had tracked down the cause. However, since my internet was always “working” at the time, there wasn’t anything they could do to help me. I could tell them the date and time the issue started, and the exact time it would happen next, but that didn’t matter since my internet was always working by the time I was talking to the tech. I received a similar response from their Twitter support account.

Luckily for me, I asked around, and a friend of a friend Thanks Bakka! happened to be a Comcast technician. After I talked with them, they explained that the 2 hour lease wasn’t typical, and was usually set before maintenances. Unfortunately, after looking into my account details, there wasn’t anything they could do. They did have one suggestion though: change the MAC address of my router to try getting a different public IP leased to me.

MAC Cycle

Now, many consumers won’t be able to do this, since their router won’t let them set the MAC address arbitrarily or they don’t have configurable WAN ports. I am using a Palo Alto in a virtual machine, so I swapped to a different vNIC (ethernet 3 instead of ethernet 1) and that did the trick. I received a new public IP address from the same DHCP server, also with a 2-hour lease. However, as the half-life of the lease approached, the renewal occurred and went through without a hitch. My issue was solved.

The challenging part of this issue was realizing that even though I had done the technical work to figure out exactly what was happening, when it would happen, how it would happen, what servers it was happening on, etc, it was impossible to reach out and talk technically with Comcast. If it wasn’t for a friend of a friend, I would still be in the dark. I remembered the XKCD comic about this issue, and realized it had happened to me.

How would a normal user troubleshoot this issue? They wouldn’t have access to the detailed firewall DHCP logs, their hardware wouldn’t have the capability to change its WAN MAC address, etc. I can imagine they would’ve been told to try getting a new router, which would’ve “fixed” the issue but also been a waste of money.

I was lucky to be able to troubleshoot and fix this issue, but it sucks that it can be so hard to have a technical conversation with a company providing a technical service. We should try to do better at both designing systems that users can troubleshoot by giving them information, and also providing channels for receiving technical feedback about the health of our systems.

Horizon Zero Dawn

madsushi@gmail.com (Chase Christian) — Tue, 07 Mar 2017 15:03:00 -0700

Platform: PS4

Genre: Action RPG with both tribal and future elements

One Sentence Description: “Western Monster Hunter“

Horizon Zero Dawn launched in March, but it’s already on my Best of 2017 list.

I am not typically a fan of open world RPGs. I like structured gameplay. I don’t like exploration. Somehow, HZD managed to pull me in anyway.

First, HZD is beautiful. The game looks crazy good. I can’t get over how clean and sharp it looks, especially as a console game. I have a suspicion that they’re stealing some processing power from the system reservations. Trying to load the PS4 menus during gameplay were very slow, and I frequently saw platform achievements fire minutes after I’d completed them. The cutscenes and facial animations were lacking in several cases. However, if that’s the #1 complaint, then you have a really awesome game on your hands.

The gameplay is stellar. It’s fast and fresh. You have a wide variety of weapons and tools, which give you free reign to approach combat as you would like. The talent trees also reward a variety of gameplay styles. I saw a YouTube video where a player defeated one of the game’s signature enemies using a completely different loadout and strategy from my encounter. There are many ways to skin the robo-cat, as it were.

I love the blurring of the action RPG and third-person shooter elements. There are plenty of shooters that have veered into RPG territory. Destiny, Borderlands, and The Division are a few examples. What sets HZD apart is the intricacy of the enemies and the combat systems. Simple enemies have multiple weak points and strategies you can attempt against them. Stealth is as rewarding as lobbing arrows into the fray. You can plan your battles with precision, without the gameplay being reduced to pumping rounds into a bullet-sponge. Watching chunks of your enemies being flung off in various directions is satisfying.

The idea of “shoot the boss a lot until it dies” is still a valid strategy, but it’s the worst strategy! You’re better off planting traps ahead of time, or unleashing an elemental assault, or destroying critical components with sharpshooter accuracy, or corraling the fauna under your control, or using environmental traps, or or or or or! The variety in approaches here results in rewarding and diverse gameplay. The way you fight your first Sawtooth won’t be anything like the way you fight your last one.

The only caveat here is that there is a lot of thumbskill required. While there are weapons that don’t require precise aiming, you’ll have a much harder time if you can’t land your arrow shots consistently.

The story is compelling. It starts with “who’s my mom?” and ends with massive implications and an emotional final cutscene. The cast of characters is small but strong. Rather than overwhelming you with a million NPCs, the developers trimmed it down to a more reasonable number of well-defined characters. The diverse cast Commander Zavala, what are you doing talking to me through my ~~Ghost~~ Focus??? helps underscore the transformation your character makes between the first and second acts. You’re moving from being an isolated hunter/gatherer to a pillar of knowledge of power within geopolitical powers. The story manages to weave together the current events with your progressive discovery of the past.

There’s so many fucking cutscenes. I honestly have no idea how they got all of these into the game. You’ll spend a lot of time watching as the story unfolds. Luckily, the story is good, so the cutscenes are worth it. But I also turned on subtitles so I could zoom through some of the slower dialog portions of the game. Like I mentioned above, some of the animations in these scenes aren’t great, but that’s a minor issue.

There are a few special dialog choices, called Flashpoints, where you can pick how you want to respond. There’s Compassion, the heart icon, which has you show mercy and pacifism. There’s Insight, the brain icon, which will allow you to find a clever solution. And finally, there’s Confront, the fist icon, which means you’re probably going to use violence. The system is reminiscent of the Paragon/Regenade system from Mass Effect, except that the Flashpoint choices have no lasting affects. I found that many of the Confront options were incredibly satisfying to use. The game doesn’t pull any punches with the dialog here.

The crafting/inventory system is so close to perfect. The biggest issue is that instead of crafting custom armor from the pristine scraps of your kills, you simply trade those scraps to a vendor instead. I feel like this takes out a ton of the impact out of gear acquisition. It’s just a vendor purchase instead of something you put together yourself. You do have to craft your inventory upgrades yourself, like your ammo pouches, and this feels great. You know that you need a Fox Skin for your next upgrade, so you go find one, and upgrade your bag immediately. Taking the Fox Skin to the vendor would cheapen the experience.

I also felt like the depth of gear was a bit shallow. There are really only 3 tiers of items in the game: green, blue, and purple. Outside of a few exceptions, you can just buy the best gear from the vendors and roll with that. The socket system adds a bit of additional pursuit, but by the end of the game, you’re swimming in modifications to install. I had all-purple gear for the entire back half of the game.

There’s plenty of collectibles strewn across the map, if you’re into that kind of things. I picked up a few but quickly lost interest.

All in all, this was an amazing game. I thought I would beat it in a day or two, and then move onto Breath of the Wild. However, I was hooked. If my Switch and BotW weren’t delayed from Amazon, I’m not sure which game I would’ve picked to play.

Games I Played In 2016

madsushi@gmail.com (Chase Christian) — Sat, 07 Jan 2017 00:03:00 -0700

I played around 20 games this year. That’s lower than my usual count. One big factor in the decline has been the age of my gaming PC. It’s over 7 years old now, and it shows. I need to rebuild it, but I’ve been punting that work to “next year” for a while. So, there are games I’ve actually avoided buying and playing because they won’t work well on my PC and I can’t (or don’t want to) play them on console.

This is my first year-in-review post for games, so I am going to focus on what I liked about the games and what my experience was like when I played them. I’m not a professional game reviewer, and I’m not going to score them on a scale of 1-10 in 5 different categories.

Anyway, let’s get started. The games are listed in roughly chronological order from when I bought/played them, but not exactly.

DOOM

Platform: PC

Genre: FPS…

One Sentence Description: “Doom 2016 feels like it had one design pillar: ‘is it speed metal?’” - Brendon Chung (@BlendoGames)

DOOM is easily the best game I played in 2016. There has been more than enough ink spilled praising DOOM, so I’ll stick to the three things that I tell anyone who will listen to me rant about DOOM:

#1 - The game is fast as hell. There’s no camping in the corner or carefully sneaking your way through a level. You’re going hard in the paint and you’re gonna get dirty. You never stop moving, you never stop shooting, and the demons never stop coming. Too many games coddle you with clean spawn waves and simple monster closets. DOOM throws enemy after enemy at you and challenges you to cover yourself in their giblets. Enemies that were once mini-bosses become trash mobs by the time you’re done.

#2 - You have complete control over your environment and arsenal. There’s a plethora of weapons and upgrades available, and there’s a dozen or more ways to solve every problem. Usually I get analysis paralysis in the face of tech trees and talent nodes, but DOOM makes it simply and everything gets the job done. You can duct tape a micro-missile launcher to your assault rifle or add explosive shells to your shotgun. Both options rule, and that’s important. You can get all over the map, climbing crates and jumping chasms and sprinting around the arena. The player agency in DOOM is unrivaled in any other shooter.

#3 - The game rewards you with fun. Watching demons explode into chunks is fun. The mini-challenge dungeons (Runes) are bite-sized fun. Ignoring the fucking NPCs that are yammering on about who knows what on your radio and stomping the shit out of McGuffins (and demons) is fun. One of DOOM’s signature weapons is the chainsaw. Do you know what happens when you kill something with the chainsaw? It explodes into a shower of ammunition like a fucking bullet piñata. Fun is baked into every single detail.

Rust

Platform: PC

Genre: Survival Shooter

One Sentence Description: “Minecraft plus Payday, but with dongs”

It’s 2 AM. Your crew is outside the wall, dressed in black and shotguns except one guy who is completely buck naked because he forgot his jacket and pants . The guards are asleep. Your crew farmed for a week to gather materials for these explosives. Your demolitions guy places the C4 on the wall, and counts down in Mumble. “3… 2… 1… breach”. An explosion!

But the wall still stands. It must be reinforced. You nod to one another, and everyone pulls out pickaxes and goes to work. After several minutes of furious clicking, the wall crumbles. You’re into the vault. Fan out to find the best chests and cupboards. Quickly smother the sleeping guards. Someone starts a 5-minute timer. Your firebug starts setting the base ablaze.

You spend the next few precious minutes performing inventory management, as you focus on taking only the best and most valuable things. New guns? Grab ‘em. Tons of potatoes? Leave ‘em. Black biker jacket? Take it, it’s your size. Your lookout calls “time’s up” and everyone funnels out of the tunnel and into the night.

You move as a column on your way back to your base. You talk about how tense the raid was, and swap details on the loot you scored. Adrenaline is pumping through your veins. Wait, what’s the smoke over there? As you crest the ridge, you see your base. Well, what’s left of it. The exterior walls look like swiss cheese, and someone’s built a giant phallic structure in your courtyard. Your only remaining possessions are what you have in your pockets. You pull your biker jacket tight, and get ready for revenge.

That’s Rust.

“Jump Check” is my favorite memory from Rust. There’s no UI in Rust. There’s also friendly fire. The combination of these two mechanics means that you’ll often find yourself looking at a player running across your screen, and you don’t know if they’re a friend or foe. So, you use jump check. Every time you see another player, you say “jump check” in Mumble. If the other player doesn’t immediately jump, you shoot the hell out of them. If you hear the jump check, you have to jump, no matter what you’re doing. Almost done with a 60-second crafting channel? JUMP. Or else.

XCOM 2

Platform: PC

Genre: Turn-based Tactical

One Sentence Description: “That’s more XCOM, baby!”

I never played the original XCOM games. XCOM: Enemy Unknown was my first experience with the series. I really enjoyed the tactical gameplay. It reminded me of a high tech or sci-fi Final Fantasy: Tactics. I like being able to plan at every level. You plan your base, you plan your crew, your plan your items, you plan the mission, you plan your abilities, you plan the whole dang thing. And then every shot misses and your entire team goes insane. That’s XCOM, baby.

XCOM 2 is truly the sequel to XCOM: Enemy Unknown. It’s got more tactics, more abilities, more items, etc. The gameplay is great and the campaign was fun to replay multiple times. Unfortunately, it also had a lot more bugs. At launch, there were tons of issues with frame rate, animation speeds, and gameplay problems. I ran into a particularly nasty bug related to carrying unconscious VIPs around that required me to rollback over an hour of gameplay to fix.

However, thanks to buying the game on PC, I was able to fix most of these issues myself with a couple of simple mods and .ini file tweaks. PC games might not have the same level of QA and certification that console games go through, but you can often take action and fix things on your own.

Despite these issues, XCOM 2 was a very fun and memorable game. The level design was great, the mechanics were fun, and I felt challenged at every step. The final battle is one that you’ll immediately want to talk to your friends about, to compare notes and war stories.

Hyper Light Drifter

Platform: PC

Genre: Isometric ARPG

One Sentence Description: “High-tech and low-tech Zelda“

Hyper Light Drifter is one of those games that I liked better on Kickstarter than Steam. I liked the idea, the art was beautiful, and I gave them my $25 in 2013. To give you an idea on how long ago this was, the campaign was advertising OUYA support. Then, I forgot entirely about it, outside of the 30 seconds or so I’d spend reading the semi-regular backer update emails.

Hyper Light Drifter, the idea, was a stylish game with engaging mechanics and a beautiful aesthetic. Hyper Light Drifter, the game, is arcane and difficult slog. Don’t get me wrong, the game itself is great, if you like that sort of game. It’s beautiful and engaging, but incredibly difficult and terse. I don’t like long frame-perfect combos and checkpoints that don’t refill your HP. If you’re a Dark Souls fan and you’re looking for a palate cleanser, this will do the trick.

Something I learned in 2016 is that I am not an explorer archetype. I don’t like esoteric games that have obtuse maps and zero dialog/UI/explanation/anything. I just never feel like pouring the time and effort into the game to really get the most out of it. It’s the same reason I’ve never finished more than 30% of Grand Theft Auto or Skyrim or The Witcher. I get it, you can do like 10,000 missions as Geralt. That’s really cool, it’s just not my jam.

Mighty No. 9

Platform: PC

Genre: 2D Platformer

One Sentence Description: “Shitty Mega Man”

My other 2013 Kickstarter dollars went to Mighty No. 9, which promised to be the off brand Mega Man that skirted as close as possible to getting sued by Capcom. The net result was a mess of a game that overemphasized a single mechanic, dash, and failed to deliver any actual fun. I played it for two hours and then uninstalled it and hid it in my Steam library. I would’ve asked for a refund, but instead wrote it off as an important lesson about early 00’s nostalgia and its failures. Now excuse my while I go watch the new Netflix seasons of Arrested Development and Gilmore Girls.

Fallout 4

Platform: PC

Genre: FPS RPG

One Sentence Description: “The world’s best sandbox”

I was sitting at the hotel restaurant, eating breakfast before another long day of E3 setup work. My friends and I were watching the Bethesda press conference on someone’s phone. When the Pip-Boy collector’s edition was revealed, I took out my own phone and went straight to Amazon.

Fallout 4 - Pip-Boy Edition - PC

Order Placed: June 14, 2015

Total: $131.39.

“Chase, you just said you’re not an explorer and that you hate games with massive unguided plots? WTF?”

It’s true, I never finished Fallout 4. I’m not even sure how far into the game I actually got. I’ll never know. What I do know is that I had a ton of fun taking Fallout’s sandbox for a joyride around the wasteland.

When I play games like Fallout, I spare no expense. Every cheat code is turned on, every way-over-the-top mod is installed, and I aim for maximum chaos. I equipped myself with the regalia and armaments of a post-apocalyptic god-king. I slew foes small and large across the blighted city of Boston. Lasers, power suits, and every single trait in the game. Steam says I’ve put 50 hours into Fallout 4, of which a solid 45 hours was spent building massive fuck-off bases with infinite resource and stomping every single living creature out of existence.

Fallout’s literally insane sandbox lends itself to tons of goofy fun. Watching your enemies be ripped into shreds in slomo V.A.T.S. is a real treat. Restrictions like “ammo” are fun, but spooling up your gatling gun and just letting it rip 24/7 is even more fun.

I will admit that I was underwhelmed with the physical Pip-Boy itself. It was really only a fancy wrist-holder for your iPhone and a huge security hole on your computer by letting wireless devices send commands to your PC.

As an aside, I also spent many hours playing the Fallout Shelter pre-game on iOS before Fallout proper was released. The game was fun while it lasted, with a very XCOM feel to it. However, it eventually devolved into the typical mobile game monetization trap, where spending cash was the only way to continue advancing at a reasonable pace.

Titanfall 2

Platform: PC

Genre: FPS / Mecha

One Sentence Description: “The most innovative FPS campaign that does Mega Man better than Mighty No. 9“

Titanfall 2’s campaign is the best FPS campaign I have played in years. They’re not afraid to add insanely fun and complex mechanics that only show up for a single campaign mission and never again. The story is rote, but compelling. The environments are stellar, including once memorable scene where you’re wallrunning on pieces of a house as it’s being constructed in a massive assembly line machine.

Respawn tripled down on movement modes as their niche, and it shows. You feel like you’re flying most of the time. Most FPS levels put you into a box and ask you to clear the box before you can proceed. You can skip a fair number of enemies in Titanfall 2 just by stunting on them. The freedom to roam around massive and engaging environments is a ton of fun, and the shooting itself is good enough to get the job done.

My favorite feature of Titanfall 2 is how they handle enemy titans in the campaign. Each enemy titan is named and they get their own anime-style cutscene. Once you defeat them, you gain their abilities and arsenal. Rather than force into a loadout between missions, Titanfall trusts the player to find the fun, and lets you swap between any titan loadout at any point. You can pause in the middle of a battle and swap to whatever abilities and guns sound cool at the time. That cool laser that the boss tried to use on you? One swap later, you’re blasting enemies to bits with the very same laser.

I normally don’t play multiplayer in random FPS games. But, since the original was multiplayer-only and I enjoyed it, I decided to give it a try here. It was worth it. There was a learning curve to the pace of the game, but the game modes (especially Amped Hardpoint) and maps were a lot of fun. I was cheesed off at some of the higher level abilities at first, like the anti-pilot turret and the shield wall. They’re really good, and starting players don’t have access to them. But you have to earn them yourself, there’s no way to buy your way to the top. I worked my way up to the “bullshit spells” eventually, and then made other newbies cry. The circle of salt.

Overwatch

Platform: PC

Genre: Team FPS

One Sentence Description: “The best game I absolutely hate to play”

Everything about Overwatch is awesome. The character design is tops. The art is sharp. The maps are creative. The gunplay is smooth. Everything is awesome.

Except for one thing: my temper. There’s something about team-based objective-focused games that brings out my competitive nature, and in a bad way. I always feel like my teammates aren’t doing their jobs. I know it’s my own character flaw, but playing Overwatch is a guaranteed way for me to end the night with an Alt-F4 and pissed off as hell. I yell at friends and pubs alike. So, despite my wife sinking hundreds of hours into the game, I refuse to play it anymore. I can’t play a game like Overwatch casually. A team deathmatch/slayer mode might fix that, but for now, I just get too upset at my teammates to make it worth playing.

Guitar Hero Live

Platform: PS4

Genre: Musical/Guitar

One Sentence Description: “The boys are back in town”

I’ll be honest, I wouldn’t have picked up Guitar Hero Live if I didn’t get a great discount on it through Activision. I loved the early Guitar Hero games, and I used to practice until I could finish every song on Expert difficulty. But Live required a brand new guitar, which seemed like a cash grab at the time. I was wrong.

The new guitar has a radically different layout, which makes it so much more fun to play. With the old guitars, you were constantly worrying about hand positioning, and making sure you could reach the top and bottom keys. Now, there are only 3 keys, so you’re not moving your hand constantly. Each key has 3 different variations: top, bottom, both. So you’ve actually got 9 total notes to work with, which makes it both easier and harder at the same time. It’s about technical skill instead of constant shifting up and down the neck. After playing with the new guitar for a few weeks, I won’t be going back.

The updated visual environment, replicating the actual experience of being a guitarist and a rock star Am I allowed to say “rock star” or is that trademarked??? is fun. There’s something magical about a roadie handing you your guitar and giving you a big thumbs up. The crowds are realistic and exciting to play for. The atmosphere really sells the experience.

Also, you get to play Eminem and Skrillex back-to-back in one playlist. It was such a treat to break out of the typical genres that Guitar Hero has focused on over the years. I was really delighted to get the opportunity to branch out and try some new beats. It feels pretty kickass to be jamming to Bangarang.

I actually like the new way they’re handling song ownership. Instead of buying and owning songs, you instead have the entire catalog at your disposal, and you pay for plays. You get a certain number free per day, and you can earn more, or you can buy a pass that unlocks as many plays as you want for a set time period. It’s great to be able to look over the vast list of songs and not feel nickle-and-dimed over every single song or pack. There’s downsides to it, but for my use case, it’s actually a pretty great model.

World of Warcraft: Legion

Platform: PC

Genre: MMORPG

One Sentence Description: “The premier Illidan simulator of 2016”

Every teenager that played Warcraft has wanted to be Illidan Stormrage, and Legion delivered. Demon hunters are incredibly fun to play, and I really enjoyed the leveling and story experience. Blizzard killed it with the new level scaling system, which lets you play any zone in any order. World quests and emissaries are ripped straight from Diablo 3, but that’s fine because they’re crazy fun and really make the world feel alive. Suramar is unlike anything Blizzard has done before, and it really paid off. The first two months of Legion were some of my favorite months I’ve spent playing WoW in the past few years.

Destiny: Rise of Iron

Platform: PS4

Genre: Shared World Shooter

One Sentence Description: “Yeah, I played another 300 hours of Destiny”

Call of Duty: Infinite Warfare

Platform: PS4

Genre: FPS

One Sentence Description: “CoD meets Star Fox, a fun couple of hours”

The Division

Platform: PC

Genre: MMORPG FPS

One Sentence Description: “I wanted FPS first/MMO second, but got the opposite”

Victoria, BC - 2016

madsushi@gmail.com (Chase Christian) — Sat, 25 Jun 2016 01:22:00 +0000

For my wife’s birthday this year, I shipped the ~~kids~~ dogs to my family Thanks Caitlin! and shipped our car (and us) to Canada.

Mory and I first visited Canada in 2011 on our Thanksgiving trip. She wanted a relaxing drive along the Pacific coast, I wanted a big plate of what the French call poutine. Our Civic took us from southern California to Vancouver, following the 101 and/or the ocean. We had a great time (and great poutine, thanks Fritz!) and knew that we’d be back again soon.

We’ve returned to Vancouver a few times since then. The drive is a bit shorter now that we’re living in the Seattle area. Vancouver is a great city, but I felt like we had already seen most of its attractions.

Surprise

I started planning the trip in February, long before Mory’s birthday in June. Our friends gave me great recommendations for places to go and things to do in Victoria. I kept the whole thing on the d/l, even though Mory hates surprises. When Mory was around, I even made fake-plans with my parents and family for that weekend, who were in-the-know and went along with the surprise.

When I woke Mory up on the Friday before her birthday, she had no idea why I was asking her to get dressed and get in the car. She thought I was going to work and she’d have a few more hours of sleep. Our bags were already packed Speaking as a man, it is hard to pack clothes for your wife. I have no idea what items go together to make up an outfit. I just kept stuffing her clothes in the suitcase until it was full. , I had the tickets and $200 CAD and our passports in hand, and my sister was on her way over to take care of the dogs.

Our first stop was 3 hours later, in Port Angeles. By now, the surprise was up, as pulling up to a ferry parking lot is a dead giveaway. My friend Missletoad recommended a great lunch spot in Port Angeles as we waited for the ferry to leave. I had an amazing sandwich and chili, loaded with smoked meat and delicious ingredients. We also visited a small candy boutique and got snacks for the trip.

The ferry ride was quiet. I played Battleheart on my iPad while Mory grabbed some of the sleep I’d deprived her of by waking her early.

Enter Victoria

Victoria hits you with a gorgeous view right off the bat. As the ferry pulls in, you’re surrounded by historical buildings and awesome landscaping.

We checked in at the Beaconsfield Inn, which is a great bed and breakfast. It was more expensive than a typical hotel, but I had been saving some free nights on Hotels.com for a special occasion and the USD:CAD rate was great at the time. It felt like staying in an antique British home, complete with sherry and crumpets served in the library in the afternoons. I had to change in our room the McClure room , as my pajamas/driving attire wouldn’t fly at our next step: the Fairmont Empress.

High Tea at the Fairmont Empress

I heard rave reviews of the high tea at the Fairmont Empress from at least a dozen friends. I didn’t buy it. My wife would get a tea, and I figured I’d grab a sandwich and a Coke How young and naive I was! Alas, Victoria is a Pepsi town. . The dress code said “Sophisticated, smart casual.” Whatever. I wore jeans.

I was wrong.

I was so wrong.

When our server offered to bring by the tea list, I was expecting a 4x10 sheet of teas with fancy names and fancier price tags. Here’s what was delivered:

When this tiny Violet Hold-esque menagerie of teas arrived, I realized that this wasn’t just a pot of tea and some crumpets. This was an experience.

Our server brought out two teapot warmers HOLY SHIT THAT’S WHAT TEA CANDLES ARE FOR?????? and set up our chosen teas. We went with the Empress Blend and the Earl Grey. They were both phenomenal.

When the tiered stand of tea treats arrived, I knew we were part of something special.

A selection of the best sandwiches I’ve ever had decorated the bottom tier. Make no mistake, these are the real power hitters in the lineup. I traded my salmon roll for Mory’s egg salad sandwich. I still think I won that deal.

The biscuits came with a strawberry and lavender jam, with the latter picked fresh from the Empress’ roof garden. The second condiment was clotted cream, which sounds somewhat gross but is actually incredibly delicious. It’s somewhere in between butter and whipped cream. Combined with the fruit, it made the biscuits a true delicacy.

The top layer was loaded with flavorful desserts, including shortbread cookies that dissolved with a mouthful of hot tea. Normally when I see an assortment of tiny dessert items, I’m wary. I’ve eaten one too many awful brownies or cheesecake morsels at cheap buffets. Each of these is handcrafted and equally delicious. The Passion Fruit Strawberry Pavlova was my favorite from the top tier.

Back to the teas: they were amazing. I had several cups of each, something near 10 cups total. I was mixing in different exotics sugars at various ratios like an alchemist.

I spent the entire tea service oscillating between “this is the best thing I’ve ever eaten” and “no THIS is the best thing I’ve ever eaten”. One thing is certain: it was the best meal I’ve ever had. They could triple the price and I’d do it again the next time I’m there. Step aside, poutine. There’s a new sheriff in Canada’s Best Food Town.

Day 1 - Downtown and Waterfront

After tea, we wandered out of the Empress, weaving past kids on a field trip and tourists taking selfies. Mory borrowed a camera to snap a few pics of some grateful Americans.

The government buildings across the street from the Empress are an amazing scene by themselves.

We explored all over downtown. We saw the Royal BC Museum, which we’d be visiting another day. We decided to walk down to the Fisherman’s Wharf for some seafood (well, for one of us). I grabbed some very Canadian supplies from a small corner store on the way: All Dressed chips and a Kinder Egg. As we neared the water, we watched seaplanes landing and taking off from the bay.

The Fisherman’s Wharf was a very cool spot. There are several attached houseboats, many of which are decorated in various artistic fashions. I bought an ice cream cone from a small shop on the wharf. There’s also a crew of seals that patrol the boardwalk, hoping for tourists to “drop” a bite of halibut or two. These seals are incredibly friendly, and will come right up to the edge of the water to take food from you. Nobody seemed to mind much.

We hopped a water taxi, which is basically Uber for boats. They took us back to the Empress in just a few minutes, dodging the other boaters and continuing line of seaplanes arriving and departing. Our ferrymaster told us the story of a mega-boat that was for sale in the harbor. Only a few million bucks, and the master cabin had never been slept in! We could be the talk of the marina! I told him we’d start saving our loonies.

We grabbed our car from the Empress’ parking lot, and headed up to Mt. Douglas. Graye had given me the great tip to head up there at dusk for an amazing panorama view of the island. It took some hiking, but he was right.

Finally, for a late dinner, we visited La Belle Patate just before closing. Graye said it was the most legit poutine in the city, even if not the cleanest. I know that a smoked meat burrito, high tea, and a tin of bacon poutine on the same day is an exercise in gluttony. I don’t care, I was on vacation.

I also tried a can of something called Spruce Beer, which tasted like I drank Pine-Sol. Hard pass.

Day 2 - Historic Victoria

After a warm breakfast at the Beaconsfield and another couple of hours of sleep, we arrived at our first stop for the day: Craigdarroch Castle.

It’s not really a castle, but it’s close enough. It was one of the most enjoyable experiences I’ve had in a historical pseudo-museum before. There were great artifacts from the various eras of the castle, information about Victoria’s storied past, and an easy-to-follow flow through the exhibits. I really enjoyed some of the political commentary from the era of the castle’s construction, where the coal baron was seen as an enemy of the common people. There were also a lot of stairs.

I bought Mory a tiny Canadian flag pin for her camera strap in the gift shop. I got myself a few things too.

After climbing all of those stairs in both directions, it was time for lunch. I had heard of a burger shop with local ingredients and good food Thanks again, Graye! . After eating at Big Wheel Burger, I can confirm that they serve great local burgers, along with killer fried pickles.

Now fed, it was time for the Royal BC Museum. We explored several different exhibits, most notably the Mammoth exhibit. There was a display that talked about a perfectly preserved baby mamoth that had been found in the Russian tundra, and how it was the only one, and how it advanced our understanding of their anatomy significantly. My wife and I both said “oh, cool” and then walked around the corner, where said mammoth corpse was right there on display in a glass box. We also learned a ton about Victoria’s history and the native peoples of the area.

For dinner that evening, I had made reservations for Zambri’s, a local Italian restaurant. The dinner was amazing, but the cocktails were even better. The ‘Spritz Hugo’ was one of our favorites.

Mory tried the cherry duck:

I stuck with the reliable pasta bolognese:

We also shared a panna cotta desert:

We turned in for an early night, since the next day would be even busier.

Day 3 - The Gardens Butterfly and Butchart

After another breakfast and post-breakfast nap at the Beaconsfield, we hopped on the highway and headed out of town towards the gardens. The Butterfly Gardens were our first stop.

The Butterfly Garden experience is walking into a greenhouse-turned-rainforest. The humidity is thick. The sky is blanketed with butterflies, and the exotic plants are covered with equally exotic animals like parrots and iguanas. The butterflies can (and will) land right on you. Mory took some great shots:

Even capturing the elusive Blue Morpho:

After returning back to the Pacific Northwest climate, we drove another mile down the road to the famous Butchart Gardens. They were even larger than I thought they would be. There were also many more tourists than I was hoping there would be. Every walking lane was clogged with people who I assume were taking literally their very first selfies and had no idea what buttons to push on their phone to make it happen. Also, the cafeteria food here sucked. My brother tells me that their afternoon tea is worth doing, so I’ll definitely be sampling it next time.

Mory had a bit more fun here than I did. I carried the camera bag (and often the camera), while she took pictures and got crazy gardening ideas. I also dropped my gelato onto the ground about 15 seconds after buying it.

We returned to Victoria proper in the afternoon, and hit up a few different liquor shops looking for Okanogan cider. If you haven’t had it before, you’re missing out. It’s a delicious Canadian cider and comes in many different fruit flavors, including my favorite, pear. We bought… a lot. More on that later.

We had dinner that evening at Red Fish, Blue Fish. The line was long, but the portions were huge and the food was great. Oh, and I guess the view was alright too.

I ended up getting an average steak from The Keg, where they not only do not serve Coke, they don’t even have Mountain Dew or Dr. Pepper as fallbacks. Get your shit together, Victoria.

That night, I tried to watch Game of Thrones on my iPad. I pay for HBO. I pay for HBO GO. Apparently you can’t use them in Canada. Luckily I have a proxy, but come on. I’m paying to watch the most pirated show of all time. Get your shit together, HBO/Canada.

Day 4 - Returning to America

Mory’s Birthday. We had to be up early to catch our ferry, as US Customs take much more processing time than their Canadian counterparts.

I knew that I had more than the duty-free amount of alcohol in the car. We had bought around 20 various 6-packs of Okanogan, along with a few 2-litres. I planned to pay the duty fees, as they were relatively modest, only about $0.10 per litre.

Here’s my conversation with the US border agent in Port Angeles:

CBP: “How long were you in Canada?”

CC: “4 days.”

CBP: “Did you bring back any tobacco, alcohol, firearms, or other restricted items?”

CC: “Yeah, we brought back some cider.”

CBP: “How much is ‘some’ cider?”

CC: “Uh, like 20 or so 6-packs.”

CBP: “… you were out of the country for how long?”

CC: “4 days.”

CBP: “… have a nice day, move along.”

And he waved us through without any inspection or duty fees to be paid. Nice. Here’s some of the haul:

We took a detour to Crescent Lake (West from Port Angeles), as Mory loved seeing it on our original Canadian trip and wanted to see it in the summer time.

We returned home to two very-excited-to-see-us-where-were-you-for-forever dogs and our own bed.

But I still dreamt about high tea that night.

Hugo, Let's Encrypt, and Caddy

madsushi@gmail.com (Chase Christian) — Sat, 14 May 2016 22:41:00 -0700

In February, I converted my site to the static site generator Hyde from WordPress. However, after using Hyde for a few weeks, I had some concerns. The setup was more complex than I liked, and the version of Hyde that I was using was already deprecated. I wanted writing to be simpler, and Hyde wasn’t meeting that goal.

Hugo

I researched various SSGs and settled on Hugo. It’s a nice and tidy single executable and is very easy to extend via shortcodes. It’s also still in active development, which is nice. Because I’d already converted all my content to Markdown, it was easy to switch generators. The structure of Hugo is easy to follow, and you can override any style/code/option by creating a copy of the file in your layouts folder.

Here’s how easy it was to implement sidenotes on Hugo, which took me a week with Hyde:

file: layouts/shortcodes/sidenotes.html
<label class="margin-toggle sn-number" for="{{ .Get 0 }}">
  <input class="margin-toggle" id="{{ .Get 0 }}" type="checkbox" />
  <span class="sn">{{ .Inner }}</span>
</label>

To use the shortcode, you just call it via {{% shortcode_name shortcode_parameters %}} syntax. To create a sidenote, I just wrap the text in the shortcode open/close: Here’s a sidenote {{% sidenote 00 %}}some inner text!{{% /sidenote %}}.

and Vine embeds:

file: layouts/shortcodes/vine.html
<iframe src="https://vine.co/v/{{ .Get 0 }}/embed/simple" width="600" height="600" frameborder="0"></iframe>
<script src="https://platform.vine.co/static/scripts/embed.js"></script>

I was able to get everything converted to Hugo in about two weeks. Ironically, I ended up starting with a theme named Hyde-Y and modifying it to the current style. I killed the scrolling headers but updated the sidenote CSS.

Let’s Encrypt

I have used StartSSL for many years, paying the $75 fee for their enhanced verification in order to get certificates for wildcard domain names. The enhanced verification is a pain if you don’t have any physical utility bills that reference your street address, which has been my situation for years as a renter. I was tired of faxing in my driver’s license and other documents just to sign up for another 2 years.

Let’s Encrypt is a free Certificate Authority that’s sponsored by some big tech companies, like Mozilla, Facebook, Cisco, and the EFF. They created a certificate request/renewal protocol called ACME and Let’s Encrypt is the first major CA to support ACME. There are several ACME clients for automating the request/renewal of certificates, including CertBot. CertBot was previously called the LetsEncrypt client and was provided directly by Let’s Encrypt. Now, the EFF is developing CertBot in order to prevent a monopoly/conflict of interest. The eventual goal of the entire project (Let’s Encrypt, ACME, etc) is to make certificates easy and ubiquitous, which includes many CAs supporting ACME for free certificates.

ACME has various ways of proving that you own the domain that you’re requesting a certificate for. HTTP-01 and DNS-01 are two of these challenges, which check for the existence of a code in a particular file on a website or DNS SRV record (respectively). I created an include file (letsencrypt.include) which I append to my nginx config files to automatically handle the serving of that code:

location /.well-known/acme-challenge {
    alias /etc/letsencrypt/webrootauth/.well-known/acme-challenge;
    location ~ /.well-known/acme-challenge/(.*) {
        add_header Content-Type application/jose+json;
    }

All the ACME requests to the /.well-known/acme-challenge subdirectory are automatically routed to the /etc/letsencrypt folder structure and the appropriate headers added.

Here’s the command that I ran to originally create the certificates:

./certbot-auto certonly  --webroot -w /etc/letsencrypt/webrootauth -d chasechristian.com -d www.chasechristian.com  -d armoryplus.com -d www.armoryplus.com -d dev.armoryplus.com -d hordetm.com -d madsushi.com -d www.madsushi.com -d tweets.chasechristian.com -d wiki.chasechristian.com -d mail.hordetm.com -d api.mdssh.com -d mdssh.com

I also added lines to my crontab to renew the certs automatically and to restart nginx to pick up the new certificate configuration:

0 */22 * * * /srv/letsencrypt/certbot-auto renew --rsa-key-size 4096 >> /var/log/letsencrypt/renew.log
30 3 * * 7 service nginx restart

Here’s my full nginx SSL configuration:

server {
    listen       23.***.***.***:444 ssl http2; ##more on the :444 later

ssl_certificate /etc/letsencrypt/live/****/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/****/privkey.pem;
ssl_session_timeout 5m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_ecdh_curve secp384r1; # Requires nginx >= 1.1.0
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off; # Requires nginx >= 1.5.9
ssl_dhparam sites/enabled/dhparams.pem;

HTTP/2 and Caddy

HTTP/2 requires SSL, which is part of why I went through the trouble of getting certs for everything. However, I found out that my specific server configuration (Ubuntu 14.04, nginx) won’t be supported for HTTP/2 by Chrome browsers after May 15, 2016. Due to the version of OpenSSL that ships with Ubuntu 14.04, I can’t get ALPN support, which Chrome will require for HTTP/2. I could compile nginx myself with the right version of OpenSSL, but that’s just a big hassle that I don’t want to deal with managing and updating. I also don’t want to hop to Ubuntu 16.04 just a month after its release. I was stuck trying to figure out how to get HTTP/2 support without major work.

Enter Caddy. Caddy is a web server written in Go, and is also a nice single-file drop-in like Hugo. It’s designed to be as simple as possible, which I appreciate. I wasn’t ready to cut my entire site over to Caddy as I have some pretty gnarly nginx configs for custom redirection, etc. Instead, I simply put Caddy in front of nginx. Caddy terminates SSL flawlessly (‘A’ rating from SSLLabs, HTTP/2 supported out of the box) and simply proxies everything to nginx. I put Caddy on :443 and moved nginx to :444.

Here’s my caddyfile:

:443

gzip

log / /var/log/caddy/access.log "{when} {proto} Request from {remote} type {method} to {path} proxy to {upstream} and status {status}" {
    rotate {
        size 100
        age 14
        keep 10
    }
}

errors {
    log /var/log/caddy/errors.log {
        size 100
        age 14
        keep 10
    }
}

tls /etc/letsencrypt/live/****/fullchain.pem /etc/letsencrypt/live/****/privkey.pem

header / {
    X-Frame-Options "SAMEORIGIN"
    X-Content-Type-Options "nosniff"
    Content-Security-Policy "default-src 'self' *.google-analytics.com vine.co disqus.com *.vine.co *.disqus.com; script-src 'self' static.wowhead.com *.google-analytics.com *.disqus.com *.googleapis.com *.vine.co data: 'unsafe-inline'; style-src 'unsafe-inline' 'self' *.googleapis.com *.disquscdn.com; font-src 'self' *.gstatic.com; img-src 'self' * data:;"
    Access-Control-Allow-Origin "{scheme}://{host}"
}

proxy / https://23.***.***.***:444 {
    insecure_skip_verify
    proxy_header Host {host}
    proxy_header X-Real-IP {remote}
    proxy_header X-Forwarded-For {remote}
    proxy_header X-Forwarded-Proto {scheme}
}

All requests are piped back to my nginx instance which handles serving the actual page. It’s been working great for a couple of days now!

I am hoping that I’m finished tweaking my site for a while, so that I can get back to focusing on building out my networking labs.

pip2pi - Python Package Caching

madsushi@gmail.com (Chase Christian) — Sat, 09 Apr 2016 11:03:00 -0700

I ran into an issue recently where I needed to be able to install Python packages on servers without internet access. I researched various options like devpi and other tools that mirror the PyPi servers. However, I didn’t want to download every single PyPi package, for both space and security reasons. Devpi says it has package whitelisting, but I couldn’t get it to work and the documentation was very lacking.

I found pip2pi to be the best solution. I feed a list of packages (or a requirements.txt file) to pip2pi and it downloads all the packages AND their dependencies into a folder. The best part is that the final folder layout matches the PyPi index layout. You can front-end the folder with nginx or Apache and use the -i parameter for pip to specify your server as the index. Now you can point pip at an internal server. Example:

pip install -i http://pythoncache/ requirements.txt

Pip2pi also ensures that every server gets the exact same code when they install packages, especially when not specifying a particular version like package == 0.8.

Consoles, Enterprise Firewalls, and UPnP

madsushi@gmail.com (Chase Christian) — Sun, 03 Apr 2016 22:40:00 +0000

Like many IT professionals, I have a home lab. The ability to test new technologies and ideas in the comfort of your own home without taking down a production environment Although the defition of “production” depends on who you ask, as the air time of Grey’s Anatomy counts in my house. is great. There’s a ton of enterprise tech that has free or very cheap licensing for home use, like the Juniper Firefly Perimeter (SRX) VM, vSphere Hypervisor (ESXi single-host), and several others. You can also get used servers and networking gear from a variety of sources to let you emulate an enterprise environment.

Unfortunately, gaming consoles don’t work well with enterprise configurations and hardware at home. Consoles like the PS4 or XBONE assume that your network follows an ubiquitous setup: a single modem/router/WAP provided by your ISP. UPnP is considered table stakes for routers in the home consumer market. Without UPnP, many games and applications don’t work without a lot of troubleshooting and tedious port forwarding. Peer-to-peer multiplayer (P2P) and voice chat are the two most common offenders.

However, there’s a way to get the best of both worlds: an enterprise firewall at the edge with cool features like logging, user-based policies, layer-7 inspection, and the rest, while still being able to play multiple consoles and games without any issues.

It’s called Double NAT.

Double NAT

First, credit for this design goes to my friend Jeff Ford, who introduced it to me as “the dirtiest thing [he’d] ever done”. I use it at home and it works.

Double NAT is usually a bad word for network engineers. However, I am a believer that you should use the best tool for the job, even if it’s an awful tool Ask me about the time I installed and used WINS on purpose in a darknet. . The key to the double NAT solution is to leverage the UPnP capabilities of your typical home router to automatically create port forwarding rules. Essentially, you put a home router (like my favorite, the Asus RT-AC66U) behind your enterprise firewall (like my Palo Alto VM100).

The home router’s Internet port sits on your LAN, with the enterprise firewall as its default gateway, as if you were an ISP. The router’s LAN ports are your custom console network, which should be a different RFC 1918 subnet from your actual LAN. The firewall feature on the router should be disabled, as well as web or admin access on the Internet port. It should be configured in “router” mode as opposed to a bridge or AP mode. I also turn off wifi, as my consoles are wired directly.

On the enterprise firewall, you be creating a NAT rule that sends all possible ports to the home router. Put it at the bottom of your NAT/PAT list so that your actual port forwarding takes precedence. Anything that you’re not manually creating a PAT entry for will be translated and sent to the home router.

The home router receives port forwarding requests from your consoles, and then configures its own PAT table for those ports. The home router thinks it is directly connected to the internet even though you’re using RFC 1918 space for its Internet interface. The router has no idea that you’re doing the translation in front of it with the enterprise firewall. When traffic comes in to the home router, it will use its port forwarding table to send traffic to the proper console.

Example

Enterprise Firewall untrust interface set to your public IP from your ISP
Enterprise Firewall trust interface is 172.16.1.1/16
Home Router internet interface is 172.16.1.100/16
Home Router internet default gateway is 172.16.1.1/16 (the firewall)
Home Router LAN interface is 192.168.1.1/24
Home Router is in router mode with firewalling turned off
Consoles are plugged into Home Router LAN ports
Console IPs are provided via DHCP by the Home Router
Enterprise Firewall has a 1:1 NAT rule at the bottom, mapping all incoming traffic to 172.16.1.100

Example code from Junos:

set security nat static rule-set console-nat from zone untrust
set security nat static rule-set console-nat rule rule1 match destination-address PUBLIC_IP/32
set security nat static rule-set console-nat rule rule1 then static-nat prefix 172.16.1.100/32

Example from Palo Alto Networks:

set rulebase nat rules OutboundConsoleRTR to Untrust
set rulebase nat rules OutboundConsoleRTR from Trust
set rulebase nat rules OutboundConsoleRTR source ConsoleRouterIP (172.16.1.100)
set rulebase nat rules OutboundConsoleRTR destination any
set rulebase nat rules OutboundConsoleRTR service any
set rulebase nat rules OutboundConsoleRTR nat-type ipv4
set rulebase nat rules OutboundConsoleRTR source-translation static-ip bi-directional yes
set rulebase nat rules OutboundConsoleRTR source-translation static-ip translated-address YOUR_PUBLIC_IP

My Two Dogs

madsushi@gmail.com (Chase Christian) — Mon, 07 Mar 2016 00:14:00 +0000

I have two dogs.

Their names are Ivy and Rexxar; they’re both purebred Golden Retrievers. They’re officially classified by the AKC as “Dark Gold”, which is a golden-brown hue that reminds me of a chocolate chip cookie that’s been left in the oven just a bit too long.

Ivy is two years old (plus some change) at the time of this writing. Rex recently hit the milestone of being in our care for a full year. They’re both young, active dogs that enjoy play-fighting in the backyard and immersing themselves in anything that even looks wet: mud puddles, rivers, more mud puddles.

Each of my dogs has their own personality and characteristics, which I try not to anthropomorphize Ran this one through Google immediately, spelling correct on the first try! but it’s hard sometimes. Ivy is a meeker and more fickle dog when it comes to food and strangers, but is ferociously passionate when it comes to things she loves: walks, friends, sleeping on the bed. Rexxar, or Rex for short, is fearless. He’s an “eat first, ask questions later” kind of dog that devours any food put within his tongue’s striking distance. He’ll meet anyone and everyone, saluting them with a violently wagging tail and perked-up ears. Let me tell you a bit more about each dog.

Ivy

Meet Ivy. AKA Iverson, AI, Ivysaur, Ivybyevy, St. Ives, and more.

Ivy’s birthday is October 27th, 2013 Sorry, not my password hint. . She was one of a dozen puppies in her litter. My wife, Mory, wanted a puppy for Christmas. My mom used to be a dog breeder, so I knew I wanted a dog with AKC paperwork. We checked local papers and websites until we found one family with a litter of puppies that would be turning 8 weeks old on Christmas. We drove an hour north to Mt Vernon and met the family. We were swarmed by a dozen unclaimed pups. Ivy seemed to be the most interested in us, as she nipped at our jacket zippers and cuddled up against our legs. They put a marked collar on her, and we waited. When we picked her up on Christmas Eve, we knew she was the one.

Ivy is a very friendly dog. She’s very gentle, even when playing with children and smaller dogs. She is very careful about where she walks, avoiding metal grates and taking the long way around obstacles. Swimming came naturally for her. A fellow Golden was paddling around in the river at Marymoor Park, and she hopped in to join them. She’s been teaching Rex how to swim lately. She loves to go on walks, and gets amped up if you even move towards the door.

Your browser does not support the video tag.

She is a cuddler, and will lay with you on the bed or sit on your lap for hours. She loves belly and ear rubs.

Our favorite thing about Ivy is her crazy tail. It’s huge and fluffly.

Rexxar

Meet Rex. AKA Rexy Boy, Rexareth Goldpaw, Rexington, Rox, and more.

Rex’s birthday is January 2nd, 2015. We wanted to get Ivy a dog companion to keep her company and help her exercise. There was a great breeder in Fall City, and we were on his waitlist for puppies. He called us out of the blue one Saturday. Someone had passed on a dog at the last minute, and he had a male Golden puppy ready to pick up immediately. We hit an ATM and drove out there that afternoon. Rex was digging a hole in the yard as his mom looked on. He was the last puppy of his litter to leave.

Rex is a fast eater and an even faster learner. Maybe it was due to our practice with Ivy in puppy classes, but Rex picked up tricks at an accelerated rate. He is not as careful as Ivy is. He’ll jump towards things that are too tall or too far away, and is always the first dog to get tangled in his leash. He uses his paws much more than Ivy does. High-fives for you and pawprints on doors.

Your browser does not support the video tag.

Rex sticks his muzzle all the way into the water when he drinks. His nose and whiskers get all wet, and then he drips that water all over the place as he walks away. I have no idea why. We should’ve known from the day we met Rex that he would be a digger. If we leave him alone for too long in the yard, he’ll be covered in mud and I’ll have a hole to fill. Mud puddle? He’s in it.

Rex is much more dependent on us than Ivy is. Ivy misses us while we’re gone and is happy when we’re back, but Rex panics and gets destructive when there’s nobody around. We’ve had to work strongly with him on crate training in order to have him calm enough when we want to go out somewhere without him. He has no sense of “personal space”, and will get right up to your face, nose-to-nose, if you let him. The backseat of our Civic has been converted to the Dog Zone™ for a long time. I try to take them with me whenever I can. They both love road trips and sticking their heads out of the windows. Their combo nickname is “Ivysaurus Rex” which is the prehistoric ancestor to all dogs.

Our favorite thing about Rex is a little prance that he does when he’s proud of himself. Whenever he’s returning a thrown ball or just completed a trick, he puffs out his chest and struts his stuff.

Matchmaking in Games

madsushi@gmail.com (Chase Christian) — Mon, 29 Feb 2016 23:57:00 +0000

When you’re designing a matchmaking system, there are 3 main variables:

Quality of match, usually related to skill and latency (QUAL) - this is good, most of the time
Wait time in queue (WAIT) - this is bad and players don’t like it, but it increases the QUAL, so it’s a necessary evil
Population size (POP)

The relationship between these elements forms the basis of matchmaking theory.

POP - The buck stops here

Increasing POP size will increase QUAL and decrease WAIT, assuming an even distribution of skill/connection within POP. A rising tide lifts all boats. Well, until you hit your server limits. In fact, depending on your matchmaking churn rates (how often players are ending matches and looking for new matches), you might need a huge population to support your goals. Joost has a great post about how common issues like skill matching, playlists, and churn determing your required population size.

Decreasing POP size will decrease QUAL and increase WAIT. You can alter your thresholds for each to adjust which of QUAL and WAIT is the most affected. Lower POP always results in either lower QUAL or increased WAIT, by definition, unless you have a perfectly balanced matchmaking pool of identical players in perfect team-sized groups (like a bracketed tournament). This never happens in real life. There are many tricks for handling matchmaking with a smaller population, but in general, higher POP = better experience for your players.

WAIT - But not too long

Increasing WAIT will improve QUAL for a fixed POP, but players don’t like WAIT, as it means net-fewer matches will be played and net-less fun will be had. And, at some point, the WAIT increase required to get QUAL where you want it is too high, e.g. making players wait an hour until finally some group of players logs in that makes a high QUAL match. WAIT varies by game/company/etc but typically is between 30 seconds and 5-10 minutes (especially for ranked/high skill games). In some games, this can go as high as 30-60 minutes. Increasing your WAIT too high can have the negative effect of lowering your POP as people get tired of waiting.

One variable that can affect the acceptable WAIT time is the length of a match. Players are willing to WAIT longer if matches are longer, as QUAL becomes more important. Many players are OK with playing a quick match where the QUAL is bad, but they’d rather add WAIT to increase QUAL as game length increases.

QUAL - Good enough

Your goal is to balance the QUAL thresholds (skill, latency, other [like groups vs singles]) with your WAIT targets, adjusted for your POP. As your POP goes down (other games, lulls in play, holidays, age of the game, etc), your QUAL targets must also adjust in order to meet your WAIT targets (unless those targets were very lax to begin with). You get to choose which elements of QUAL to focus on (skill, latency, etc) based on design and what players want. Something’s got to give, and the matchmaking system decides what gives first.

You also might need safety valves, namely breaking your QUAL targets when the WAIT is too long. A good example would be if six of the best players in the world queue up from Madagascar. You won’t find a good skill match and you won’t find a good latency match, so you can either punt (fail to find a game, kick them from the queue, etc) or you can force a game (with lower QUAL targets than you normally want). Typically you’ll want to use expanding filters, starting with say “within 50ms latency, within 50 points skill”, run that through your matchmaker for 30 seconds, then increase to 100/100 and run for another 30 seconds, etc, expanding your filters until you find a match. Being able to gather statistics about what percentage of matches are being made within each filter bracket helps you design and adjust the settings.

There are other variables that also influence your QUAL targets. If you start adding mechanics that punish players who leave games, you’re now responsible for ensuring the QUAL is at such a level that players don’t want to leave the game. So, by encouraging players to stay in a game via incentives or punishments, you’re entering into a contract with the player where you’re committing to a high QUAL value for each match. You’re telling the player that they need to stay in the match, and it becomes your responsibility to make staying in the match fun for the player (high QUAL). One of the most damaging things you can do is put players into a long, bad match with no recourse (fear of punishment/loss of rewards).

Another issue Games with split factions, like WoW, suffer from immediately portioning their POP into two, permanent buckets. The new Mercenary PvP queue seems to be an attempt to mitigate that issue. is when you fragment your POP into buckets, essentially creating smaller groups of POP, which as above, will decrease QUAL or increase WAIT. So every playlist/mode you create is fragmenting your POP and thus reducing your overall experience. This is why playlist-sprawl only works when POP is huge. Many games end up condensing down to just a few core playlists (maybe with a rotational schedule) in order to solve a dwindling population problem without lowering QUAL / increasing WAIT. Ranked vs unranked, hyper-sensitive latency requirements, pre-made vs solos, and many other issues can fragment even a massive playerbase beyond repair. Riot works very hard to keep their QUAL statistic maximized for both skill and latency, which can be tough.

Summary

At the end of the day, you’re always going to have blowouts and you’re never going to hit 100% of users matched in 10 seconds. Jeff Kaplan brought up the particularly tough issue of players leaving a match mid-game ruining matchmaking, as well as how skill can be variable based on a player changing their class or character or loadout mid-game. These are all outside of a matchmaking system’s control, and you have to accept some losses. It’s a balancing act of taking your current POP and figuring out what QUAL thresholds meet your goals.

Converting to Hyde

madsushi@gmail.com (Chase Christian) — Sun, 28 Feb 2016 15:22:00 +0000

I hosted my blog (chasechristian.com) via WordPress for several years, with the previous iteration running from 2012-2016. I’ve never been completely happy with WordPress WordPress was always “good enough” , due to issues with security, functionality, themes, and performance. I read a great post from Katie Zhu about NPR’s static site template and how they were able to serve a popular site with purely static pages on Amazon S3. If NPR could host their site with static pages, my site should be a breeze.

I had done some research into static site generators and other blog alternatives, including GitHub Pages/Jekyll, Octopress, Svbtle, Medium, and others. I made a list of features I wanted and tried to track down the right solution. Steve Losh migrated to the Hyde static site generator and comprehensively detailed the process on his blog. While I was more comfortable with PHP (and evaluated Pelican), I figured that this was a good opportunity to learn some more Python while following Steve’s guide. Much of the styling here comes from his examples.

Hyde

Steve’s migration to Hyde was six years ago, in 2010. In the time since then, the version of Hyde he used was deprecated (now called hyde-old) as well as the Aardvark Legs CSS suite. In fact, many of the Hyde documentation links you’ll find online either 404 or take you to a Chinese domain squatter. Luckily, Steve forked the version of Hyde he’s using on GitHub, so I was able to use the same code. That’s one of the benefits of a static site generator: as long as they output valid HTML, they’re never obsolete. With WordPress, it was scary to run more than a day or two without updates due to security concerns.

I setup a Python virtualenv for my Hyde site, created a set of folders, and got everything installed. There were some issues with getting the right versions of things installed. This server runs Ubuntu 14.04, and many of the packages were newer than they were when Steve set up his blog, and incompatible with what I was trying to do. I had to set up several of the requirements files with version information to grab the right copies of Django, etc. I set up some bash aliases to make things easier (generating pages, etc).

WP to Hyde

The next step was to convert my old WordPress posts to Markdown for use with Hyde. I used Boris Smus’ fork of exitwp, which he to convert his own blog to Hyde. I decided to jettison all of my old WordPress comments (RIP) as they’re no longer needed. I also punted on trying to convert the WordPress URL format to the Hyde URL format, and just created a very long list of nginx rewrite rules, like the following:

if ($request_uri ~ ^/278/wow/clearing-your-wow-creature-cache-automatically/$) {
        rewrite ^(.*)$ /blog/2013/02/clearing-your-wow-creature-cache-automatically/ permanent;
        break;
    }

I manually added Hyde headers to all of the posts, as there were only 33 or so of them.

{% hyde
    title: "Converting to Hyde"
    snip: "Static site generator"
    date: 2016-02-28
    categories: ["tech"]
%}

There was plenty of manual work to convert the stuff exported from exitwp into clean Markdown. I removed all the WordPress “more” buttons, completely rebuilt a bunch of tables from the ground up, and spent at least two hours redoing all of my WoW class colors article in pure HTML.

I did run into one platform issue Then I spent the better part of 8 hours with strace on it… , where Hyde was generating my articles out of order. When I’d browse the list of articles, the years would be mixed up. I found out that this was an issue due to how Python walks directories on different platforms, and opened a pull request to fix this issue. I would like to thank my friend Justin Cook for finding the fix for me!

I’m using Disqus for comments, FeedBurner for feeds, Google Analytics for stats, and nginx to serve up the data.

Sidenotes

The coolest blog feature on my list of requirements was *sidenotes* Hi mom! . Thomas Ptacek uses sidenotes to great effect on his blog. I found Waylan Limberg’s guide to writing Python Markdown Extensions and sampled exaroth’s Custom Span Class Markdown Extension for most of my code. I wrote an extension called SNExtension (for Sidenote Extension) that will automatically place the text between the marker characters into a span with class sidenote which then gets moved over to the right via CSS. I extended it to include the ability to number sidenotes as well as hiding them on small screens, as seen on via the Tufte-Jekyll style.

[>] - start sidenote
00-content - ## is the sidenote_id, followed by a dash, then the content
[/>] - end sidenote

I’ll get this up on GitHub sooner or later.

Style

I spent a long time browsing various fonts on Google Fonts before arriving at Fira Sans as my font of choice. I am also using Audiowide for my titles and headers. One of the cool things about separating content from layout is that I can change fonts globally with a snap of my fingers. I replaced the codehilite Markdown stuff with highlight.js as Boris Smus did as well. I find that it does a better job, especially with Windows shell scripts/batch files.

Updates

I also took the opportunity to update my nginx server (via the direct nginx repos) to support HTTP/2 and updated my SSL config. Adding Let’s Encrypt support is coming up shortly, as my certificate from StartSSL expires in May. I also pulled out CloudFlare temporarily, as it wasn’t handling SSL very well. I had hacked in CloudFlare support without actually delegating my DNS to CloudFlare (using an old hosting provider partner of theirs) but it’s time to either move my DNS there or find an alternative solution (like S3).

The history of WoW Class Colors

madsushi@gmail.com (Chase Christian) — Sun, 16 Aug 2015 08:42:55 +0000

In my time working on World of Warcraft sites and addons, I’ve gotten to dig into some of the arcane bits of the game, like class colors. These colors allow players to instantly identify another player’s class, and many players carry a strong bond to their class’ color. In fact, coloring is the primary way that players are able to identify official Blizzard representatives in-game and on the forums, via that beautiful Blizzard Blue:

#00B4FF

I was looking into information on the class color for the newly-announced Demon Hunter, and found a discrepancy in how the Druid class color was being reported. I dug deeper and found some interesting details about class colors that I’d like to share and document. I will do my best to avoid Rogue/Rouge jokes. No promises.

RGB

Every class color is represented by a simple RGB value, typically in hex, like #FFF569 (Rogue - brittle Rogues are anything but brittle yellow). There are 3 pairs of digits, each represents a different color. So Rogues can be broken down into FF-F5-69, with FF representing red, F5 representing green, and 69 representing blue. “F” is a high value in hex, while “0” is a low value. “00” means there is none of that color mixed into the final color, while FF means the maximum amount of that color is mixed in. Pure red would be FF-00-00, while simple magenta could be FF-00-FF. The values range from 00 to FF, which is a total span of 256 values, so 1 point of difference is only a 0.4% change in that value of that single color. Not a huge difference.

Can you tell these two blocks apart?

#FFF569

#FFF570

Canonical Source

One of the challenges I found when digging into class colors is defining the canonical source. What really defines the class colors themselves? When there’s conflicting values from various places, you have to figure out which is actually correct. I went straight to the source: the actual game code. Blizzard has obfuscated their UI code in recent years, leaving only encrypted blobs in your /Interface/AddOns folder. However, there are some advanced console commands you can still use to dump the source code of the WoW UI to a folder. I used ExportInterfaceFiles code to get the files from the client (latest, 6.2.0.20338), and then inspected the /FrameXML/Constants.lua file The data for class colors has been moved into /SharedXML/Util.lua at some point around Legion’s launch. .

RAID_CLASS_COLORS = {
  ["HUNTER"] = { r = 0.67, g = 0.83, b = 0.45, colorStr = "ffabd473" },
  ["WARLOCK"] = { r = 0.58, g = 0.51, b = 0.79, colorStr = "ff9482c9" },
  ["PRIEST"] = { r = 1.0, g = 1.0, b = 1.0, colorStr = "ffffffff" },
  ["PALADIN"] = { r = 0.96, g = 0.55, b = 0.73, colorStr = "fff58cba" },
  ["MAGE"] = { r = 0.41, g = 0.8, b = 0.94, colorStr = "ff69ccf0" },
  ["ROGUE"] = { r = 1.0, g = 0.96, b = 0.41, colorStr = "fffff569" },
  ["DRUID"] = { r = 1.0, g = 0.49, b = 0.04, colorStr = "ffff7d0a" },
  ["SHAMAN"] = { r = 0.0, g = 0.44, b = 0.87, colorStr = "ff0070de" },
  ["WARRIOR"] = { r = 0.78, g = 0.61, b = 0.43, colorStr = "ffc79c6e" },
  ["DEATHKNIGHT"] = { r = 0.77, g = 0.12 , b = 0.23, colorStr = "ffc41f3b" },
  ["MONK"] = { r = 0.0, g = 1.00 , b = 0.59, colorStr = "ff00ff96" },
};

We can see the colors plainly spelled out, with the first “ff” referring to the alpha (RGBA) or opacity of the color. The oldest available version of this file is from 2010, via Archive.org’s copy of wowprogramming.com (which is an amazing resource and book). The archived version matches the live version exactly, save for missing the new Monk class, and all of the values only being recorded in RGB percentage format (Rogues are 1.0/.96/.41, which converts to #FFF569 as expected). Luckily, the current file gives us both values, so it was easy to compare.

Here’s a trimmed list, straight from the canonical source, unchanging over a decade (with one exception - shamans):

Hunter - ABD473
Warlock - 9482C9
Priest - FFFFFF
Paladin - F58CBA
Mage - 69CCF0
Rogue - FFF569
Druid - FF7D0A
Shaman - 0070DE
Warrior - C79C6E
Death Knight - C41F3B
Monk - 00FF96
Demon Hunter - No game data

Shamans changed class colors from their former pink to their current blue when they became available to both factions in The Burning Crusade. When comparing this to my posted list, I noticed that the Monk value was not what I found previously

Left: In-game, Right: My posted color

#00FF96

#00FFBA

Which is a not-insignificant difference. Clearly the #00FF96 value is canonical, as it comes straight from the source code for the game itself. I actually referenced the “ExportInterfaceFiles” command in that very same class colors post, but had no idea about the code option for the command until recently.

WoWWiki and Shaman

One of the most-referenced sources of this data is WoWWiki’s class colors page. Unfortunately, despite going back to 2006, it has had its problems. For example, the Warlock class color was reported incorrectly due to a math error until late 2008. It does chronicle the evolution of Shaman from pink to blue, including what appears to be a swap in September 2010 from #2459FF to #0070DE.

#F58CBA

#2459FF

#0070DE

That would’ve occurred right before the Cataclysm expansion was released in December 2010. The user who makes the change noted that it had changed in the game files at that time as well. Our copy of Constants.lua from January 2010 from wowprogramming indicates that the colors were already set to #0070DE at least that early. It’s hard to say when exactly Shaman swapped, but it was definitely before January 2010. A Russian WoW wiki has Shaman listed with their new color as early as September 2009. August 2009 was the announcement of Cataclysm, which is the only major event in that time frame.

Battle.net CSS

While WoWWiki is second-hand information, we received a second source of direct information in late 2010. Blizzard launched the revamped Battle.net website, which included sections for each class. Browsing the Archive.org archives for Battle.net CSS files, the earliest references to class colors comes on December 6th, 2010, the day before Cataclysm launched. Let’s compare the CSS with our canonical list:

CLASS	CANON	BNET CSS	DIFF
Hunter	ABD473	AAD372	(-1/-1/-1)
Warlock	9482C9	9382C9	(-1/0/0)
Priest	FFFFFF	F0EBE0	(-15/-20/-31)
Paladin	F58CBA	F48CBA	(-1/0/0)
Mage	69CCF0	68CCEF	(-1/0/-1)
Rogue	FFF569	FFF468	(0/-1/-1)
Druid	FF7D0A	FF7C0A	(0/-1/0)
Shaman	2459FF*	2359FF	(-1/0/0)
Warrior	C79C6E	C69B6D	(-1/-1/-1)
Death Knight	C41F3B	C41E3B	(0/-1/0)
Monk	00FF96	00FFBA	(0/0/+36)

You’ll note that every single class’ color is altered slightly from the game file version. Shaman stand out strongly, as their CSS color is #2359FF, which is offset from their old color, #2459FF, which was changed to #0070DE in the game files over a year before the CSS was available. Fast-forward a year to October 23, 2011, and the class color CSS definitions are the same, so it wasn’t a launch day mistake. In fact, all of these colors are still slightly offset from the game values in the Battle.net CSS today (TYOOL 2015). These colors were picked on purpose… or were they?

The easiest change to explain is the move from pure white for Priests to an off-white color. Never Use White and Never Use Black are two common web design mantras. Most of the other tweaks are not noticeable, even when compared side-by-side.

Here’s a full table:

#ABD473

#AAD372

#9482C9

#9382C9

#FFFFFF

#F0EBE0

#F58CBA

#F48CBA

#69CCF0

#68CCEF

#FFF569

#FFF468

#FF7D0A

#FF7C0A

#2459FF

#2359FF

#C79C6E

#C69B6D

#C41F3B

#C41E3B

#00FF96

#00FFBA

Looking at today’s current Battle.net CSS file, we can see the addition of the Demon Hunter with #4DD827 (which some are calling Fel Green) and also the source of my false Monk information (#00FFBA, from earlier). Fel Green in action:

#4DD827

Demon Hunter Update - 8/21/2016

Demon Hunters have been released to the wild, and there has been an interesting development. While the main Battle.net CSS file stills references the #4DD827 color (.color-c12 { color: #4dd827 !important; } /* demon hunter */), there’s a different CSS file that controls the coloring on the forums. It matches the Battle.net CSS file exactly, except for Demon Hunters:

.Author-class.hunter{color:#aad372}
.Author-class.warlock{color:#9382c9}
.Author-class.priest{color:#f0ebe0}
.Author-class.paladin{color:#f48cba}
.Author-class.mage{color:#68ccef}
.Author-class.rogue{color:#fff468}
.Author-class.druid{color:#ff7c0a}
.Author-class.shaman{color:#2359ff}
.Author-class.warrior{color:#c69b6d}
.Author-class.death.knight{color:#c41e3b}
.Author-class.monk{color:#00ffba}
.Author-class.demon.hunter{color:#a330c9}

Via the CSS selector .Author-class.demon.hunter, we can see that there’s a new color for the DH class on the official forums (and in-game!):

#4DD827

#A330C9

It’s a fel purple (#A330C9)! Demon Hunters have shifted from green to purple, likely due to the overloading of green as a class color (hunter, monk).

Hunter

Monk

Fel Green

A decision was made to shift Demon Hunters over to purple, which they now share with Warlocks. I would’ve loved to have been a fly on the wall of that style meeting. Let’s take a look at the CSS and game files to see what other changes were made.

OLD WARLOCK CSS:     .color-c9,  .color-c9 a { color: #9382c9 !important; }
NEW WARLOCK CSS:     .Author-class.warlock{color:#9382c9}

So, no changes were made to Warlocks on the forums. However, the game files tell a different story (build 7.0.3.22423):

RAID_CLASS_COLORS = {
  ["HUNTER"] = { r = 0.67, g = 0.83, b = 0.45, colorStr = "ffabd473" },
  ["WARLOCK"] = { r = 0.53, g = 0.53, b = 0.93, colorStr = "ff8788ee" },
  ["PRIEST"] = { r = 1.0, g = 1.0, b = 1.0, colorStr = "ffffffff" },
  ["PALADIN"] = { r = 0.96, g = 0.55, b = 0.73, colorStr = "fff58cba" },
  ["MAGE"] = { r = 0.25, g = 0.78, b = 0.92, colorStr = "ff3fc7eb" },
  ["ROGUE"] = { r = 1.0, g = 0.96, b = 0.41, colorStr = "fffff569" },
  ["DRUID"] = { r = 1.0, g = 0.49, b = 0.04, colorStr = "ffff7d0a" },
  ["SHAMAN"] = { r = 0.0, g = 0.44, b = 0.87, colorStr = "ff0070de" },
  ["WARRIOR"] = { r = 0.78, g = 0.61, b = 0.43, colorStr = "ffc79c6e" },
  ["DEATHKNIGHT"] = { r = 0.77, g = 0.12 , b = 0.23, colorStr = "ffc41f3b" },
  ["MONK"] = { r = 0.0, g = 1.00 , b = 0.59, colorStr = "ff00ff96" },
  ["DEMONHUNTER"] = { r = 0.64, g = 0.19, b = 0.79, colorStr = "ffa330c9" },
};

Warlocks’ in-game color moved from #9482C9 to #8788EE, which is significantly more blue and somewhat less red. Previously, they had the same amount of blue (.79) as Demon Hunters do today:

["WARLOCK"] = { r = 0.58, g = 0.51, b = 0.79, colorStr = "ff9482c9" },
["WARLOCK"] = { r = 0.53, g = 0.53, b = 0.93, colorStr = "ff8788ee" },

#9482C9

#8788EE

Demon Hunters actually have their CSS and game files in sync, due to all 3 hex color values rounding up (see the Rounding section). This makes them the only class with matching in-game and CSS values:

Canonical Blue: .64 -- Hex A3/RGB 163 = 63.9% (round up to 64%)
Canonical Green: .19 -- Hex 30/RGB 48 = 18.8% (round up to 19%)
Canonical Red: .79 -- Hex C9/RGB 201 = 78.8% (round up to 79%)

An interesting note is that Mages also had their in-game color shifted in Legion, from #69CCF0 to #3FC7EB:

["MAGE"] = { r = 0.41, g = 0.8, b = 0.94, colorStr = "ff69ccf0" },
["MAGE"] = { r = 0.25, g = 0.78, b = 0.92, colorStr = "ff3fc7eb" },

#69CCF0

#3FC7EB

Again, their forum CSS color did not change. The Mage change is interesting, as it might be designed to further separate them from the newly-bluer Warlock. The Mage color mix saw a big decrease in red, which makes it less sky-blue, in my opinion. Here is new-Warlock vs old-Mage:

#8788EE

#69CCF0

Here’s a final comparison of all of the blues:

Old Mage

New Mage

Blizzard

Shaman

Old Shaman

Questions

It appears intentional that all of the class colors are slightly offset from their in-game representations… but why? Why are Shaman still using a class color that was changed in the client 5 years ago? Why just drop a couple of values by 1 point? It’s not distinguishable to most observers. Only the Monk color changes significantly, with a large blue boost. Where’s the beef?

Rounding, aka Off-By-One

Hunters, for example, are defined as: r = 0.67, g = 0.83, b = 0.45. That’s straight from the game files. The corresponding RGB values in decimal are 171 212 115, or ABD473. ABD473 technically maps to 67.1% / 83.1% / 45.1%, which is pretty close to the canonical definition. All percentages are slightly larger than the canonically defined values.

The CSS hunter is defined as: AAD372. All 3 RGB colors were decremented. That corresponds to decimal 170 211 114, or 66.66% / 82.7% / 44.7%. Those percentages actually do round up into 67/83/45 - the canonical definition. Is it possible whatever tool they’re using to generate the CSS is playing with Price Is Right rules, where it’s closest-without-going-over?

Game Hunter: .67 / .83 / .45 = ABD473 = 67.1% / 83.1% / 45.1%

CSS Hunter: AAD372 = 66.66% / 82.7% / 44.7%

If they’re rounding with CEIL instead of FLOOR or actually rounding, then AAD372 becomes the correct result to achieve the 67/83/45 split. Let’s look at one more example.

Warlocks are canonically defined: r = 0.58, g = 0.51, b = 0.79. The corresponding decimal values are 148 130 201, or 9482C9. 9482C9 maps to 58.039%, 50.9%, and 78.8%. So 50.9% (green) and 78.8% (blue) are both rounded up, while 58.039% (red) is rounded down.

When we look at the CSS warlock, it’s defined as: 9382C9. Interestingly, only the first color (red) is changed. Red is also the only color percentage that was rounded down in the canonical color, while the other two color percentages were rounded up (and unaltered in the CSS). 9382C9 = 57.6% red, which can be rounded up to 58, matching the canonical definition.

Game Warlock: .58 / .51 / .79 = 9482C9 = 58% / 50.9% / 78.8%

CSS Warlock: 9382C9 = 57.6% / 50.9% / 78.8%

We’ve found the issue. The rounding being used to convert simple percentages like 0.58 into the 256-value RGB space is different between the client code and the web/CSS code. That explains why almost every value is shifted by 0 or -1, and never +1.

The game values are all the closest possible to the original percentage specified, while the CSS values are all the closest without going over. So, in some cases, the CSS color values are slightly less accurate (assuming that the percentage values are canonical). Here’s an example from Death Knights:

Death Knight Canonical Value for Green: 0.12

Game Value: Hex 1F/RGB 31 = 0.1216 (31/255) for a differnce of 0.0016

CSS Value: Hex 1E/RGB 30 = 0.1176 (30/255) for a difference of 0.0024

So CSS is 0.008 further than game from true “0.12”

Priests are set to off-white for web best practices. Monks still see a significant change, and their color is much bluer on Battle.net than it is in the game for no discernible reason. And why are Shaman still using the ancient class color? That’s a mystery for a different time…

PS Vita Remote Play and SoftEther

madsushi@gmail.com (Chase Christian) — Tue, 13 Jan 2015 07:38:01 +0000

Sony’s latest gaming handheld, the Playstation Vita, has a great new feature: Remote Play. Your Playstation 4 turns on, starts playing a game, and then streams the video to your Vita. You control everything remotely via the Vita. It’s a great way to play PS4 games when you’re away from your console (or away from your house).

Because the game is actually being executed and rendered on your PS4, it needs to be on for this to work. But what happens if your PS4 isn’t on?

PS4 consoles will register themselves with a remote Sony service, which records their public IP address (which can change on most residential connections) and other information. Your PS4 is rarely “off” but rather “asleep” when you put it away. While asleep, it can be woken up by this remote Sony service, which then lets you connect to it via Remote Play (forwarded ports on your router/firewall). During the DDoS attack Sony experienced around Christmas 2014, this remote Sony service was down, so PS4s could not register themselves.

My own PS4 was happily asleep at home, but I could not turn it on remotely to access it. There are no PC-based tools to turn a PS4 on, no management software, etc. But there is another way to turn on a PS4: with a Vita. The Vita can send out a specially crafted broadcast packet, which will wake up the PS4 and allow you to connect. The issue is that you can’t easily send a broadcast packet from your Vita and have it hit your PS4 without being on the same network.

That’s where SoftEther comes in. SoftEther allows you to make a seamless layer 2 tunnel between endpoints, which lets broadcast traffic through. I installed SoftEther server on my home PC (remotely), installed SoftEther bridge on my laptop, and got to work. Because the Vita can only connect to the network via wifi, I used an extra USB NIC + wifi access point to tunnel my Vita through to my home network. The broadcast from the Vita woke up the PS4, and I was good to go!

The catch with trying to use SoftEther in this manner is that Windows is not very good at allowing you to make your device a wireless access point. If you set your wireless to be an access point instead of a client, it transparently adds lots of NAT / DHCP and won’t let you natively bridge your interfaces together. In order to get around this, I needed a real access point, with a very simple config, and then bridged my LAN interfaces together in Windows. If you see an IP range like: 192.168.137.0/24, you know that Windows is silently using NAT / DHCP in the background.

I also took a capture of the broadcast traffic that the Vita uses to wake up the PS4, and I plan to try replaying that PCAP file next time to see if the same packets will wake it up.

iOS Client Certificate Expiration - April 16, 2014

madsushi@gmail.com (Chase Christian) — Mon, 21 Apr 2014 18:16:53 +0000

All iOS devices (iPhone/iPad in particular) have built-in client certificates that are issued by Apple, that identify them as “official” Apple iPhones. Without one of these certs, a service isn’t able to verify whether the device connecting to them is actually an iPhone or not. Apps are also capable of checking this certificate to ensure they’re running on a proper iPhone and not an emulation. I believe the certificates are issued at either time of manufacturing or at time of activation.

Either way, a large number (all?) of iPhones and iPads all recently shared the same certificate expiration date: “April 16, 2014, 6:55:02 PM EST.”

Thanks to Twitter user @ryandolan123 for the screen cap.

When this certificate expired, any app or service that checked the client certificate for a valid iPhone failed. There were several affected services, including:

Aruba ClearPass
BBC iPlayer
Sky Go Player
Dish TV Player
and many others…

Most of these issues ended up being resolved by the app producers disabling client certificate checking. I am wondering how Apple is going to handle reissuing these certificates, or if it’s even a concern for them.

Update: “Apparently FaceTime has also been affected.”

More WoW API information

madsushi@gmail.com (Chase Christian) — Wed, 09 Apr 2014 19:06:13 +0000

I’ve recently been working more on the WoW API, specifically around hunter pet mapping. I have found several new discoveries from the last time I worked on the API.

Racial mapping

When retrieving character race data from the API, you simply receive a number. Here’s a table that maps that number to the respective race:

Alliance:

1 - human
3 - dwarf
4 - nightelf
7 - gnome
11 - draenei
22 - worgen
25 - panda-a

Horde:

2 - orc
5 - undead
6 - tauren
8 - troll
9 - goblin
10 - bloodelf
26 - panda-h

Neutral (temporary):

24 - panda-n

Parsing an entire realm’s characters

I am using the RealmPop method of parsing realm character data, namely:

How - To get the list of characters for a realm, first we record all the characters who posted to the auction house. Then we fetch and record their guild rosters. This should cover the majority of characters on a realm. To avoid getting listed, a character must never post to the auction house, and never belong to a guild where a guild member posts to the auction house.

So, how would you go about this? Realm auction data is packaged up into unique files every so often, and the URL of that package changes. There’s an API endpoint that will give you the latest URL of that data package.

http://us.battle.net/api/wow/auction/data/**$realm**

When you that URL, you’ll get back the file url for the auction data package. Then, you grab that data package. Here’s a recent Mal’Ganis package, for example:

http://us.battle.net/auction-data/e93144540f4aa366aefe80b13d2ebbb9/auctions.json

Once you have the auction data, you crawl through it and pull out the auction owners. The file is separated into Horde/Alliance/Neutral auction houses. I usually compile a list of auction owners from all 3 sets. I don’t record the faction data here, since it’s also contained (implicitly) in the race data I gather later. So now you just have a big list of all of the people that have current auctions posted. Then, you grab their guild information.

http://us.battle.net/api/wow/character/**$realm**/**$owner**?fields=guild

Now you turned your big list of auction owners into a big list of guilds! Finally, you then parse each guild’s roster:

http://us.battle.net/api/wow/guild/**$realm**/**$guild**?fields=members

RealmPop is probably done at this point, since they only really care about the basic member information. I wanted to dig deeper, so I parse out the list of members, select those that are hunters (class = 3), and then I grab their individual Armory information:

http://us.battle.net/api/wow/character/$realm/$member?fields=hunterPets,guild

Pet family mapping

Similar to the racial and class mapping, each pet family type is also mapped to a number. Here’s a table:

1 - Wolf
2 - Cat
3 - Spider
4 - Bear
5 - Boar
6 - Crocolisk
7 - Carrion Bird
8 - Crab
9 - Gorilla
11 - Raptor
12 - Tallstrider
20 - Scorpid
21 - Turtle
24 - Bat
25 - Hyena
26 - Bird of Prey
27 - Wind Serpent
30 - Dragonhawk
31 - Ravager
32 - Warp Stalker
33 - Sporebat
34 - Nether Ray
35 - Serpent
37 - Moth
38 - Chimaera
39 - Devilsaur
41 - Silithid
42 - Worm
43 - Rhino
44 - Wasp
45 - Core Hound
46 - Spirit Beast
50 - Fox
51 - Monkey
52 - Dog
53 - Beetle
55 - Shale Spider
125 - Crane
126 - Water Strider
127 - Porcupine
128 - Quilen
129 - Goat
130 - Basilisk
138 - Direhorn
59 - Silithid – (this specifically is the “brain bug” version)
66 - Wasp – (the new silithid wasp)
68 - Hydra – (very rare, only one known pet, now unobtainable, was actually retconned to ‘hydra’ in Cataclysm, was previously a ‘crocolisk’ type)

I’ll note that there are some additional mappings for other languages. For example, in the Spanish client, ‘Cat’ is translated to ‘Gato’, so a tamed cat-type will actually be named ‘Gato’. Now, some people also do funny things like name their wolves and scorpions ‘Gato’, but that’s unrelated.

CheckInstall - never run 'make install' again

madsushi@gmail.com (Chase Christian) — Sun, 23 Feb 2014 18:55:29 +0000

The first time I touched Linux was while I was working on setting up my own Minecraft server. I downloaded a free distro called MineOS. It ended up being really awful, and I eventually moved Minecraft to a Windows server (lovingly named Blocky). However, I was surprised at how functional Linux was. I came from a purely Microsoft upbringing.

I bought a Linode VPS to mess around with, and it’s that same instance that this site is hosted on today. I didn’t know much about Linux when I got started, and so I made a lot of mistakes. Experience is the best teacher, and so through those mistakes, I’ve learned a lot. One of my biggest mistakes was blindly installing any package I wanted to, especially via source and ‘make install’.

The problem with managing stuff from ‘make install’ is that it can be hard to uninstall and you have to keep the source around in case you want to reinstall, etc. There’s a solution: CheckInstall. When you want to install from source, you still run configure and make, but instead of running ‘make install’, you just run ‘checkinstall’. It will pull all of its info from the source and create a Debian package and install that instead. Now you can easily install/uninstall stuff via aptitude, and stuff installed from source will show as installed in aptitude and meet all dependencies! Seriously awesome, seriously easy.

WoW Addon/API Information

madsushi@gmail.com (Chase Christian) — Mon, 25 Nov 2013 09:36:19 +0000

I thought I would dump some WoW information here that I’ve learned from the Armory API and addons.

Classes in WoW are numbered 1-11, and each class has a specific color attached to it.

Warrior - color is #C79C6E - Tan or Brown
Paladin - color is #F58CBA - Pink
Hunter - color is #ABD473 - Green
Rogue - color is #FFF569 - Brittle Yellow
Priest - color is #FFFFFF - White
Death Knight - color is #C41F3B - Red or Crimson (isn't it interesting that DKs are #6????)
Shaman - color is #0070DE - Blue
Mage - color is #69CCF0 - Light Blue
Warlock - color is #9482C9 - Purple
Monk - color is #00FFBA - Jade Green
Druid - color is #FF7D0A - Orange
??? - color is #558A84 - No idea why this was in my CSS file

If you want to dump all of the icons from the WoW client in a convenient format, start WoW with the “-console” flag in the shortcut and run this command:

exportInterfaceFiles art

Note that it will take a while (minutes) to run. You will want to dump the icons again after any new patch to grab the updated icons. The names should always match those that you get back from the Armory API.

The gospel of WoW API documentation is the official Blizzard API page on Github. The data resources section is particularly awesome for grabbing lists of all achievements, all taletnts, etc. I use this to refresh my achievement comparison tools after every patch. Make sure to cache this in a database or file somewhere, since you don’t want to be pulling it constantly (it gets pretty big).

The Armory APIs are down during Tuesday maintenance (usually).

The guide to enterprise storage: disks

madsushi@gmail.com (Chase Christian) — Tue, 24 Sep 2013 08:01:22 +0000

Storage is one of my favorite things. I love everything about it: NAS, SAN, networks, SCSI, SSDs! However, most home users and amateurs never get the chance to do much with storage besides format it with Windows or maybe toss an SSD into their system. There’s a lot to learn with storage, and it’s been around long enough to have plenty of legacy cruft hanging around. Rather than trying to cover everything at once, I want to talk about storage in pieces. I am going to start with disks (spinning, hard disk drives) and work my way up the stack to SANs, and I’ll probably talk about SSDs separately.

The pursuit of 4K

While 4K televisions are the new hotness, 4K has been around in the hard drive world for a few years. When you go buy your 2TB hard drive from Western Digital on NewEgg, it’s probably partitioned into a ton of 4KB pieces called sectors. Drives with 4K sectors are said to be Advanced Format drives, while the old drives are just Old Drives. Prior to 2010 or so, drives were cut up into 512-byte sectors. If you do your math, you’ll see that 8x 512-byte sectors add up to 1x 4-kbyte sector, so there are 8x fewer sectors on a newer drive. This allows for less overhead to be wasted, and since the size of the average write size has grown, there isn’t much space savings.

When a hard drive goes to read some data, it reads it sector by sector. If you have 12KB of data to read, your hard drive has to go find the three different 4KB sectors that the data is stored in. The sector is the smallest unit of space that your hard drive can read or write at a time, and how fast it can find and read a sector is critical to determining the drive’s performance.

Alignment woes

One of the fading issues from the storage world is alignment, or rather, misalignment. The problem stems from older operating systems (Windows XP and Server 2003) only being programmed for 512-byte sector sizes. When Windows XP is formatting a drive, it blocks out the first 63 sectors in a row for the partition offset. Unfortunately, 63 is not divisible by 8 (31.5 KB instead of 32 KB), so when Windows XP or Server 2003 starts laying out the partition table, everything is off by 512 bytes.

Now, when Windows tries to read a single NTFS block, because the data is spread across multiple disk sectors, the hard drive has to read both sectors, doing twice the work. If you’re doing a large read or write operation, the penalty is N+1 (one extra read or write), while if you’re doing a lot of small operations, the penalty can grow to 2X (double the number of reads/writes required). Usually the penalty falls somewhere in the middle. I have typically observed a 30-40% drop in performance on a misaligned system, and as high as the worst-case (2N) with some workloads.

Misalignment is mostly a thing of the past. Nobody is installing Windows XP or 2003 from scratch, and anything after (and including) Vista or Server 2008 has migrated to a 1MB partition offset (which is easily divided into 4K sectors). If you’re not dealing with legacy operating systems, you’re probably fine. But, it’s still important to know for companies that are migrating from 2003, especially with VMs. To fix your VM, there are tools that can modify the VMDK or VHD file to stuff in some extra zeroes and ones to the front of the file to get everything back into alignment.

The quintessential input/output operation

You’ve probably heard the term “IOPS” at least once. It means input/output operations per second, but what’s an input/output operation anyway? The typical I/O request is a 4K read or write. When you’re navigating your OS menus, browsing the internet, or working on a file, your computer is doing a ton of 4K read/write operations. Checking permissions, writing cookies and registry values, remembering where you left the position of a particular window. Because 4K is the smallest operation possible, it tends to happen a lot.

How fast can your hard drive perform a single 4K operation? That’s a good question. Hard drive performance is tied very closely to the seek time of that particular drive. The seek time describes how fast your hard drive can find a random spot on the hard drive (any arbitrary 4K block). Let’s say you have a reasonable seek time (plus read latency) of 18 milliseconds (ms). That’s typical for a Western Digital Black: consumer-grade, but designed for enthusiasts. How many times can your hard drive seek in 1 full second? Well, 1000ms/(18ms/seek) = 55.5 seeks/second. So, if you tell your hard drive to go find 55 random 4K blocks, it will take about a second to pull that off. If you do the math, 55 blocks at 4KB per block is only 220 KB per second, which is just slightly faster than a T1 connection. That’s pretty slow when it comes to hard drive speeds! How is this possible?

Now, that’s assuming those blocks are distributed randomly. If all of the blocks are lined up in a row (by “in a row”, that means physically in a row on the actual platter), the hard drive can read those much faster. Much, much faster. Your average WD Black drive can put out at least 100 MB/s, or even higher at full blast. That’s equivalent to 25,000 of those 4K blocks per second, or almost 500 times faster than our last example. Because the seek time is reduced to near-zero, you can get great performance out of your drive. If you’re just telling the hard drive “go read these 1,000 blocks in a row”, that’s what’s called a sequential operation. If you tell a hard drive “go read these 1,000 blocks in random spots on the drive”, that’s a random operation.

Your hard drive will have very different performance characteristics when it comes down to sequential vs random transfers. That’s why it’s so important to know your workload’s transfer characteristics when sizing your storage. You’re never going to see 100% sequential or 100% random operations, so figuring out your mix is crucial. Applications like SQL are notorious for their focus on small block size random IO, while other applications like Exchange (2010 and above) are designed specifically for sequential ops.

The operating system can request data in larger than 4K sizes. For example, it could ask for a 4MB chunk, which is equivalent to a thousand 4K blocks. Your hard drive doesn’t need to seek to each block individually, it can grab the whole 4MB in one sweep (if there’s no fragmentation). I/O requests can range from 4K (smallest) to 8MB (largest). If you want to max your IOPS, you go 4K. If you want to max your bandwidth, you go 8MB. 32K is often seen as the “sweet spot” since most files (Word documents, pictures, etc) are somewhere around that size.

When someone is talking about IOPS, they’re almost always talking 4K performance, but it could be either random or sequential performance. 4K random is the worst case scenario, so you’ll see relatively small numbers, but they’re accurate as the minimum performance you’ll see from the drive. I like dealing with 4K random IOPS numbers, because ensures everyone is on the same page and that I’m looking at the worst possible performance. You want to size things for the worst case, not the best cast.

How many IOPS?

Obviously, every hard drive is different, at the tech is advancing quickly. But here’s my cheat sheet for IOPS calculations:

SATA drive / 7200 RPM (nearly all consumer drives) — 50-75 IOPS (13-20ms)
SAS drive / high-end SATA (Cheetah) / SFF drives 10000 RPM — 100-125 IOPS (8-12ms)
Enterprise SAS drives / 15000 RPM — 150-200 IOPS (5-6ms)

As a note, look at a SAN full of SATA drives. When your minimum disk latency is 13-20ms, and you tack on some network latency, you will never get back to the requesting system in under 20-25ms. That’s considered pretty poor performance. Remember that latency plays a part!

5ms = hearing the crunch of something you stepped on, speed of GOOD local disk
10ms = very quick response for a SAN, not much slower than local disk
15ms = acceptable
20ms = borderline, DBs (SQL) will start underperforming
25ms = unacceptable, VMware will get mad
30ms+ = users will start to notice their file shares are slow, etc

Humans can easily notice latency differences of 5ms and lower, so the difference between 10ms and 15ms is recognizable, and the difference between 5ms and 10ms is monumental.

How many IOPS does something use? Your average desktop user (VDI or physical) is somewhere between 10-20 IOPS. I have run setups of ~100 VM servers on about 2,000 IOPS during the day (20 IOPS/server). Obviously this depends on the workload: a single server can push 2,000 IOPS if it is busy enough (but not commonly).

$$$ MONEY $$$

If 15k RPM SAS drives are so great, why don’t we use them for everything? Because they’re really expensive.

Based on some LIST NUMBERS that are at least a year old now (you should be able to get 60%+ off list EVERY TIME), let me break down some stats for you. The best IOPS/$ (performance per dollar) is a SAS disk, at around 70 IOPS/$1,000 being a common stat. You should expect around 3,500 IOPS out of a 15k RPM system that is priced at $50k. On the low end, SATA drives only offer about 15 IOPS/$1,000. That same $50k system would only furnish 750 IOPS, which is very low. Note that the difference is somewhere around 5X, that is, you get 5X the performance per dollar with faster disk.

Now, when it comes to GB/$ (storage per dollar), the role is reversed. A system full of 2TB or 3TB drives will give you about 700 GB/$1,000, or 35TB raw for about $50k. A system full of 450 GB SAS drives will yield only 200 GB/$1,000, or only 10TB raw. The slower drives offer a 3X improvement in storage per dollar. Obviously you need to know what you’re buying storage for.

However, there is an often overlooked stat, which is IOPS/GB, which determines how many IOPS you get for every GB of storage. Having 500 TB of storage is pointless if you don’t have the IOPS to back up your workload. You need to know the rough IOPS/GB to make sure you’re buying the right disk speed AND size. When you have 1-4TB SATA drives available, for example, you need to know which one meets your performance needs AND your storage needs. Buying 10 of the 4TB drives will yield one quarter of the performance of 40 of the 1TB drives, but the same amount of storage.

As a quick aside, don’t buy the bullshit about “NL-SAS” drives. They’re either 7200 RPM or they’re not. A 7200 RPM drive, even in a nice NL-SAS wrapper, is still only capable of so many IOPS based on the physical limitations. Definitely don’t pay 10k RPM prices for them!

10k RPM drives tend to be the SFF, small form factor - 2.5” instead of 3.5”, drives. These are usually only required if they fit the IOPS/GB requirement nicely, or if there’s a particular need for the extra density (storage per rack unit) they provide. They also tend to use less power, which could be important in certain environments. You’re usually better off just buying more fast drives for storage or more big drives for performance, though.

AHCI and NCQ and…

While I could talk about desktop storage at length, let’s keep this focused on enterprise storage. It’s par for the course to assume every drive you use will be communicated with via the SCSI protocol with all of the basics like NCQ baked in. These are optimizations that can make the drive performance much faster. NCQ (usually called TCQ with SCSI) allows the drive to queue up a few I/O requests to find the best path across the disks to get all of the blocks, so that it takes the most efficient route instead of zooming about randomly due to a FIFO input queue. Let’s assume that everyone is sourcing their disks from the same places (they are) and have the same tech baked in (they do) and that per-disk performance across vendors is roughly equal (it is). Now, there are a lot of things you can do with those disks that are very cool, particularly in regards to free space allocation and fragmentation, but that’s a file system discussion.

SAS vs SATA, and controllers

What’s the difference between a SATA drive and a SAS drive? What are SATA and SAS anyway? Where does SCSI play in?

SCSI is a protocol for a computer to talk to a hard drive. SCSI is old, but also very good. You want your computer (or SAN) to be able to talk to your disks via SCSI, to get the best performance and most options. SATA is another protocol that does the same thing, but not as well. In general, SCSI>SATA. SAS is the newest form of SCSI. It stands for serial attached SCSI, which can use serial-style physical connectors (SATA connectors, NOT protocol) to connect to SCSI/SAS drives.

You can plug a SATA drive into a SAS card, but you can’t plug a SAS drive into a SATA card. Only the very lowest quality storage solutions will still be using SATA controllers or true SATA drives. Every solution worth buying will be using SAS, and when we talk about “SATA drives”, those are usually slower 7200 RPM drives that have a SAS controller tacked on (NL-SAS). So, they’re capable of talking SAS (thanks to the tacked-on controller) but still have the slower physical characteristics of a SATA drive. Everything is talking SAS/SCSI, typically.

When we’re talking about a “SAS drive”, what that means is that the drive itself is capable of talking the SAS/SCSI protocol. The big difference advantage of SCSI as a protocol is that the endpoint (drive) has intelligence built-in and can help offload and optimize some of the storage requests. That’s why enterprise storage drives are more expensive: they’re capable of greater physical speeds (twice the RPM), they have built-in controllers and software (SCSI), and they need to be rated for extremely high durability (MTBF).

Up until a couple of years ago, the SAS and SATA protocols were both capable of moving up to 3Gbps of data per port. Often, SATA controllers would share that 3Gbps across multiple drives, while SAS controllers could typically provide full bandwidth to each hard drive. Newer versions of both protocols (SAS 3 and SATA 3) can increase that limit to 6Gbps. While only SSDs are capable of even coming close to capping out either of those limits (spinning hard drives still can’t cap 3Gbps), these speeds do play an important part in your storage configuration.

It is common for a shelf of storage (say 24 disks) to be attached to your SAN via a SAS cable. Those 24 disks are now all sharing that same 3Gbps/6Gbps connection to the controller, which means you could be limiting your throughput. What’s the point in having 10Gbps or more in network bandwidth to your SAN when your controller can’t even access the disk that fast? The solution is to use multiple SAS ports to aggregate your bandwidth, and to use SSDs and other controller caching options to allow for additional data to be served without hitting the disk.

NetApp, for example, sells the DS4243 shelf and the DS4246 shelf. The shelves are identical in almost every way (both hold 24x 3.5” drives), but the DS4243 is only connected at 3Gbps while the DS4246 is connected at 6Gbps. The first digit (4) refers to the number of rack units (U) the shelf consumes, the second and third digits (24) refer to the number of hard drives it can hold, and the final digit is the speed at which it is connected. So the DS2246 is a 2U shelf that holds 24 disks (2.5” SFF) and is connected at 6Gbps.

There also used to be FC drives, which were attached to your controller via fiber channel (but still talking SCSI the whole time). These are pretty much deprecated now. It was still SCSI commands back and forth, but using FC as a transport. FC could reach speeds of 1Gbps, 2Gbps, or 4Gbps, but had some other limitations. SAS has all but replaced FC-connected disk in the enterprise. FC for targets, however, is still alive and well.

Questions?

Let me know if you have any questions or comments on disks, or if there’s anything you think I should add. The next article will probably be on file systems/caching/writing, with protocols to follow.

Paired failover

madsushi@gmail.com (Chase Christian) — Tue, 24 Sep 2013 06:09:23 +0000

When you’re designing an infrastructure for a service or application that requires high uptime, you have to plan for certain levels of resiliency and redundancy. There are a lot of considerations that come into play. Every decision is balanced against the cost, the benefit, and the uptime requirements. You have to ensure that you have high availability at every level of your tech stack, so there’s no point in buying two firewalls and two switches if your server only has a single NIC. Do you have separate internet connections, from different carriers, coming in over different physical paths? Does each host connect to redundant network fabrics? Are key services like DNS and authentication highly available?

There are a lot of different models for high availability. For example, Cisco ASA firewalls are typically deployed in an Active/Passive configuration. A trio of Juniper EX switches in a Virtual Chassis are all passing traffic at the same time, with the higher level roles being consolidated to one master device at a time. Each device has its own failover scenarios and requirements. One of the things to avoid when combining all of these different mechanisms is a paired failover.

A paired failover, at least by my definition, is when one failover occurs, other dependent devices are required to failover as well. A good example would be having two internet connections, with each one plugged into a different firewall. If an internet connection failover had to occur, you would also have to have a firewall failover occur at the same time. The internet connection and the firewall form a paired failover group, where one’s failover requires the other to failover too.

I hated paired failovers. In the ideal network design, each object in a path is redundant, and a failover is transparent to the devices both upstream and downstream of it. There are a few good ways to accomplish this:

Single-image systems (think Cisco’s VCS or “stacking” or Juniper’s Virtual Chassis)
LACP and VRRP in both directions (Cisco’s VPC handles this well)
Quick failover routing protocol
Active/Passive failover with state replication (Cisco ASAs fit the bill – still need LACP for link redundancy)

The key feature is that the identity of the device (the IP address or the link) is transparent/virtual and can move between devices. While using a routing protocol to handle failover between devices might fit the bill, it depends on the protocol and the speed of the failover. BGP is great for this, since it can support dual-active scenarios. I also particularly like single-image solutions or configurations where the state information is shared between redundant devices to further minimize the impact of a failure.

How do you end up with a paired failover? It’s an easy trap to fall into.

Imagine you have two firewalls (lets say Juniper SRX boxes) and a couple of Juniper switches. You obviously want networking redundancy, so you put the switches into a virtual chassis. Done, right? Nope, you still have to make sure everything is properly multihomed. Are your servers capable of LACP or another bonding protocol to the upstream switch? Both of your firewalls will need to connect to both of your switches (4 cables), otherwise a switch failover would also require the firewall to failover at the same time. Every device in the chain needs to have redundant connections to the other devices, in both directions. But what about the internet connections to the SRX boxes? Do you just have a single WAN switch out in front? Are they separate links? How would your IP space failover between the devices? You’ll probably want BGP to ensure your IPs can float. But what if your BGP addresses float to the non-active firewall, causing asymmetric routing? As you can see, there are a lot of considerations at play, and you have to be careful to address every possible failure scenario.

Playing Spotify music via Mumble

madsushi@gmail.com (Chase Christian) — Thu, 04 Jul 2013 06:49:45 +0000

After years of chatting in Mumble while playing games, I wanted a way to provide a soundtrack. I polled my friends and came up with the following criteria:

Collaborative playlist
Wide selection of music
High-quality sound
Suppressible
Run from my normal workstation (Win8) alongside my normal copy of Mumble

After ruling out several options, the solution was clear: Virtual Audio Cable + Fidelify + Mumble portable mode.

First, you’ll probably need to buy Virtual Audio Cable. It’s worth every penny. I originally thought of just injecting a music stream into my microphone via VAC, but then individual listeners couldn’t selectively mute just the music without muting me too (suppressible). You end up just creating a single line in VAC (with volume control enabled) that Fidelify outputs to and Mumble listens on.

Spotify removed all of their options for specifying which audio device to play sound on. Fidelify is an alternative Spotify client that allows you to specify your audio device. Pick the VAC line you created for your output and you’re good to go. I created an Open Playlist on Spotify and shared it with all of my friends, so they can add music to the playlist to suit their tastes.

Finally, you have to launch Mumble in portable mode. This will ensure that your second copy of Mumble has its own settings and configuration. Create a second Windows account on your PC. Then, run Mumble with the following command:

runas /user:spotify "c:\Program Files (x86)\Mumble\mumble.exe -m -s"

That will launch a second copy of Mumble as the user “spotify” or whatever you named your second account. On Windows 7, I could just launch this from a shortcut. On Windows 8, I actually have to open up a command prompt and type this in every time. Log in to your server with this Spotify Mumble account, and put your Spotify account into a special protected channel. Link that special channel with your main channel in Mumble to bridge the audio. Set the Spotify Mumble program’s Audio Input to your VAC line, and set the Audio Output to something else and set the output volume to zero (so you don’t hear everything twice). Set the input transmission type to continuous and adjust the quality until you have good sound and high quality. Remember that this can soak up bandwidth on your Mumble server, if that is a concern.

Any time I want to globally mute the music, I can either pause Fidelify or simply unlink the channel in Mumble, then relink it when I’m done.

One big issue is skipping songs, which is still a manual process for the Fidelify operator. Fidelify also has been waiting for an update for a couple of years now, and has a few quirks/bugs (like not being very random on random and crashing sometimes).

Squid3 with SSL support, on Ubuntu 13.04

madsushi@gmail.com (Chase Christian) — Sun, 16 Jun 2013 05:42:10 +0000

A few years ago, I spent many hours getting Squid with SSL working on Ubuntu 10.10. The core issue is that the Ubuntu-provided binary for Squid does not include SSL support. Thus, you have to compile Squid yourself, from source. Unfortunately, it’s not as simple as you might think, due to a lot of inconsistent information and errors that you’ll run into. I set up Squid on my Linode VPS, to act as a transparent proxy, to allow me to get my web traffic past firewall filters and other proxies. I also wanted all of the data in transit to be encrypted, which is done via SSL. Chrome would redirect all of my web traffic to Squid via an encrypted SSL tunnel, and Squid would go out to the internet and fetch my pages for me. I had everything working great for a long time.

I thought I was still good to go, until a recent update of a seemingly unrelated module caused my Squid authentication mechanism, ncsa, to start segfaulting any time I tried to log in. This is the story of how I got Squid3 working again, via a rebuild.

Download the newest source code

My first mistake was trying to use the source code provided by Ubuntu, via “apt-get source squid3”. The version of Squid3 that is provided by Ubuntu didn’t have the ncsa fix in it, so I spent a lot of time compiling source for absolutely nothing.

Head straight to the official Squid website and grab the latest release build (3.3.5 in my case) and download the tar.gz file via wget.

No debian/rules file?

Because the Squid source code directly from their site doesn’t have the nice debian/rules file built-in, you have to pass all of the variables to the configure script. I took bits from the Squid compiling guide and other sources. Here’s what my final command looked like:

./configure --prefix=/usr --localstatedir=/var --libexecdir=${prefix}/lib/squid3 --srcdir=. --datadir=${prefix}/share/squid3 --sysconfdir=/etc/squid3 --with-default-user=proxy --with-logdir=/var/log --with-pidfile=/var/run/squid3.pid --enable-inline --enable-async-io=8 --enable-storeio="ufs,aufs,diskd" --enable-removal-policies="lru,heap" --enable-delay-pools --enable-cache-digests --enable-underscores --enable-icap-client --enable-follow-x-forwarded-for --enable-basic-auth-helpers="LDAP,MSNT,NCSA,PAM,SASL,SMB,YP,DB,POP3,getpwnam,squid_radius_auth,multi-domain-NTLM" --enable-ntlm-auth-helpers="smb_lm," --enable-digest-auth-helpers="ldap,password" --enable-negotiate-auth-helpers="squid_kerb_auth" --enable-external-acl-helpers="ip_user,ldap_group,session,unix_group,wbinfo_group" --enable-arp-acl --enable-esi **--enable-ssl** --enable-zph-qos --enable-wccpv2 --disable-translation --with-logdir=/var/log/squid3 --with-filedescriptors=65536 --with-large-files --with-default-user=proxy

So, that’s quite the long string. It pulls the Ubuntu/Debian defaults from the Squid page, then adds in several of the parameters that are in the debian/rules file that Ubuntu provides when you download the source straight from their PPA. It also includes the essential –enable-ssl flag, which is what gives Squid the capabilities to do SSL.

Configure/Make All/Make Install

Just follow along with the basic Squid compiling instructions and you’ll be fine. You’ll have to build your own service or /etc/init.d/ files for controlling squid, and you’ll also need to set it up to start at system boot. I’ll leave the /etc/squid/squid.conf settings for another day, they’re pretty simple. Look for “https_port” and “ncsa_users” to find some of the common guides.

Your network must be unimpeachable

madsushi@gmail.com (Chase Christian) — Mon, 13 May 2013 07:28:39 +0000

When someone asks me for advice on computer parts, I tell them that there’s two components that you can’t afford to skimp on: power supply and motherboard. If you buy an off-brand video card, it will be pretty easy to figure out if it is causing any problems. If your hard drive dies, which all hard drives will, you’ll know immediately. The same isn’t true of a PSU or mobo. If one of them is being flaky, they can cause all sorts of issues that are tough and expensive to troubleshoot. They’re central components that everything else in the computer rely upon.

I feel the same way about networks. Everything in IT relies on the network. End-users connect to their applications over the corporate LAN, servers connect to their SANs via storage networks, and customers connect to web services via the WAN. If you are uncertain about your network’s stability, doubt will creep into your mind every time there’s a minor issue. You’ll start spending your time chasing network ghosts instead of focusing on building and maintaining your infrastructure. The network is the heart of your environment, and buying cheap equipment or running with a bad configuration can undermine its capabilities. Your network must be unimpeachable.

Nobody has an infinite budget, and networking is seen as “just a bunch of ports” instead of the core service that every devices relies upon. Networking is so crucial that it’s often taken for granted, as everyone assumes that it’s already in place, which makes it hard to see the value that it provides. Why would you spend thousands of dollars on expensive Juniper or Cisco equipment when you could spend hundreds of dollars on a Netgear or Linksys with all of the same features? The value of an enterprise-grade switch isn’t purely that it checks all of the important feature boxes, but rather that those features work in a consistent, dependable, and repeatable way.

A Netgear switch might say that it supports LACP and LAG groups, but you’ll find yourself wondering whether it really supports dynamic LACP and what your selection of traffic hashing algorithms will look like and if it will interconnect with your existing network gracefully and so on. When you buy an enterprise-grade switch, you know that it’ s going to pass all of the traffic you can throw at it, that it will have the logging and monitoring required to detect issues proactively, and that you won’t have to worry about half-implemented features or compatibility.

When it’s 4:30pm on a Friday and you get the call that your VoIP phones are down, do you really want to be wondering whether your trunk port forgot its VLANs again or whether a cheap switch got into a weird state and needs a reboot?

If networking is the heart of an infrastructure, then routers and firewalls are the heart of networking. When a SonicWall or Watchguard claims to have feature-parity with a Juniper or Palo Alto for a quarter the price, how could any rational decision-maker go with the more expensive options? The manufacturers of the second-tier networking gear do everything in their power to make the products appear capable and competitive. It’s not until you’re hours deep in troubleshooting a web app that randomly disconnects or a adjusting a VPN that just won’t negotiate or trying to build a NAT rule that the box can’t do that you realize that feature parity on paper is much easier to achieve than feature parity in practice.

I have a VPS through Linode (which is currently serving this post to you) that runs in a data center in Fremont. When I’m neck-deep in code and web server configuration, I don’t want to wonder if it’s the network that’s causing my site to have issues. I don’t want to wonder if a router is silently dropping every 15th packet, I don’t want to wonder if an over-aggressive ACL is rejecting my SYN-ACK packets, and I don’t want to wonder if maybe there’s a duplex mismatch issue somewhere that’s causing slowness. The network needs to work, or else I’m always going to be suspicious that it’s failing me in some way. A simple issue now requires digging into switch logs and doing packet captures. I love running Wireshark as much as the next guy and I’ve got my gigabit port-mirroring device for grabbing captures, but I’d much rather solve application issues without worrying that the pipe somewhere down the line is messing something up.

802.1Q trunking between network vendors

madsushi@gmail.com (Chase Christian) — Sun, 14 Apr 2013 06:15:49 +0000

When you’re going to link two switches from different vendors together, you have to make sure that your VLAN mappings match up on both sides. Unfortunately, each switching vendor does things their own way, so you might have to adjust your configuration to get everything working properly.

I am going to focus on discussing trunking between switches. At the most basic level, this is just a cable running between two switches. What’s important is the configuration of the interfaces.

Native VLAN

Every switch port has a native or default VLAN Technically, Juniper lets you remove the native VLAN if you like, and just passes untagged control packets (but not untagged data packets) to the CPU. You have to manually specify a native VLAN on the interface if trunking to Cisco, etc. Of note: you exclude the native-vlan from the members list on that interface. Thanks Jeff! . Cisco calls this the native VLAN, others call it the default VLAN, Extreme Networks refers to it as the untagged VLAN, etc. The key is that this VLAN represents all of the traffic on the switch that doesn’t have an 802.1Q tag on it. You might make the assumption that you could plug a Cisco port with a native VLAN of 10 into an Extreme port with an untagged VLAN of 20 and that would bridge both networks together, without either switch knowing that there was a mismatch. Unfortunately, higher-level STP protocols will actually catch the mismatch and shut the port down unless you specifically tell STP to ignore the error. So, native VLANs should be match where possible. Because of this, I don’t recommend ever using a switch’s default VLAN, but rather resetting your “default” VLAN to 10 or 20 or something to ensure compatibility later on. Alternatively, you can create a new “native-only” VLAN that specifically serves as the native VLAN for trunk negotiation.

Cisco calls it the native VLAN, and the default VLAN ID is 1.

Juniper calls it the default VLAN, and the default VLAN ID is 0.

Extreme calls it the untagged VLAN, and the default VLAN ID is 1.

HP calls it the untagged VLAN, and the default VLAN ID is 1.

Dell is weird. They call it the untagged VLAN or the PVID, and the default VLAN ID is 1. On a PowerConnect interface in “trunk” mode, VLAN 1 is tagged! You have to set the interface mode to “general” and then mark VLAN 1 as untagged.

In general, your native VLAN needs to match on both sides. Even if you plan on trunking everything other VLAN, switches often won’t negotiate a link without a matching native VLAN. Sometimes it will work, sometimes it won’t. Don’t leave it up to chance.

Trunking

When you configure a trunk in Cisco, it essentially adds every VLAN to that interface as tagged VLANs. If you want to connect two Cisco together, you can just set both interfaces as trunks and plug a cable in. However, most of the time we want to control which VLANs are being passed between the switches. We should always use 802.1Q for our trunk encapsulation, since it’s universally supported.

Note that many vendors use the word “trunk” to describe a set of aggregated interfaces (like LACP or etherchannel), which can confuse things.

In the VLAN-centric world, you configure your desired VLANs as tagged so that they’ll be passed between the switches over a single link. Here’s an example:

configure vlan native add port 5 untagged
configure vlan voice add port 5 tagged
configure vlan storage add port 5 tagged
configure vlan data add port 5 tagged

In the interface-centric world, you configured the allowed VLANs that can cross the link. Don’t forget to include your designated native VLAN in your list of allowed VLANs! Here’s an example:

interface gi-0/5
switchport trunk allowed vlan add 1,5,12-13

You need to make sure that your VLAN IDs match up on both sides, as if a switch sees a packet with an 802.1Q tag that it doesn’t recognize or have on the allowed list, the packet will be dropped.

Interface-centric vs VLAN-centric configuration

madsushi@gmail.com (Chase Christian) — Sun, 14 Apr 2013 05:33:06 +0000

In the world of network switches, it’s hard to escape the shadow of Cisco. Cisco owns roughly 70% of the switching market, with something like 80% of installed switches and over 60% of all new switch sales. Cisco engineers are plentiful, and the CCNA/CCNP/CCIE certifications are considered some of the best in the industry. Because of how much momentum Cisco has behind them, it forces other vendors to emulate their product in order to emulate their success.

There are two basic approaches to configuring VLANs: interface-centric and VLAN-centric. Cisco switches are all interface-centric, which means that most of the industry follows suit. However, there are a few manufacturers out there that believe VLAN-centric configuration is the better solution.

Interface-centric config

I’ll start by talking about the incumbent. Interface-centric VLANs all revolve around the individual interfaces. Let’s talk about a hypothetical voice/VoIP VLAN, named “voice” and with the VLAN ID (tag) 225.

In an interface-centric world, you add the VLAN to each interface. Here’s an example:

interface gi1/5
switchport access vlan 225
exit

interface gi1/6
switchport trunk allowed vlan add 225
exit

If you created a new voice VLAN on an edge switch and wanted to enable users to plug their phones into any port, you would have to configure each and every port on the switch. You can typically edit multiple interfaces (or an interface range) at once to make the job easier. The VLAN membership is contained in the configuration of each interface. Your worst case scenario is creating a new VLAN and being forced to add that VLAN to each port individually. If you can configure your multiple interfaces at once, this isn’t so bad.

VLAN-centric config

In a VLAN-centric configuration paradigm, all VLAN membership configuration is done at the VLAN level, not at the interface level. You add each interface to the VLAN. Let’s go back to our old voice VLAN example.

create vlan voice
configure vlan voice add ports gi1/5 untagged
configure vlan voice add ports gi1/6 tagged

As you can see, you’re not going into each interface’s configuration to set the VLAN. This make it very easy to run a command like “show vlan” to show you all of the member ports in a particular VLAN. If you make a new VLAN, you can add all of the switch’s ports to the VLAN with one command. Your worst case scenario is adding multiple VLANs to the same interface, because you would have to go into each VLAN individually, rather than just adding all of the VLANs at once to the interface. There’s not really any good ways to mitigate this. Extreme Networks and HP are the only pure VLAN-centric switching manufacturers I’ve worked with.

Junos

Juniper’s Junos does things a bit differently, and actually allows you to configure your VLANs either way. You can focus on an interface-centric configuration if your network engineers work with Cisco gear, or you can switch to a VLAN-centric design if you have a bunch of ex-Extreme Networks engineers working for you. The key is to be very careful to avoid overlapping both approaches on the same switch, which can have unintended results. Pick one design or the other and stick with it. I have heard that Juniper recommends configuring the untagged/access ports as VLAN-centric, and the tagged VLANs under each interface.

The guide to fiber

madsushi@gmail.com (Chase Christian) — Sat, 13 Apr 2013 00:52:45 +0000

There are a lot of names and acronyms that get thrown around when it comes to fiber, so I thought I would sum up what I know on the topic. I will mention that I have had no formal fiber/optical training, this is just what I’ve picked up in the field. I’m not an expert here by any means!!

Edit 07/15/2015 - RAS did a great talk on optical networking.

Edit 03/02/2017 - Svein gave me some tips and corrections. Thanks!

First: why use fiber? Because copper cables (CAT5/6 etc) can only carry data 300ft. That’s not very far. Fiber can go for miles.

Fiber

There are two types of fiber: single-mode (SM) and multi-mode (MM). Generally speaking, single-mode is goes further, but is more expensive.

Single-mode goes further (especially at high speeds), while multi-mode goes shorter (REALLY short at high speeds). Of course, single-mode is much more expensive than multi-mode.

Also: while you shouldn’t look directly into EITHER type of fiber (shine it onto your palm or a piece of paper), looking into single-mode fiber will probably blind you AND it won’t show any light since it’s way too small to see. Also, NEVER TOUCH THE END OF A FIBER CABLE. Your oils/dirt can damage the contact and can ruin the cable.

Need to test a multi-mode fiber cable? Shine a flashlight down one end (colored flashlights or LEDs are perfect) and put your hand/paper in front of the other end and see if you see the tiny spot of color. There are also IR detector cards that you can shine the light onto.

Fiber is very tiny strands of glass-like filaments. You CAN break them if you bend the cable too far. DON’T snap your fiber cable, and throw it away immediately if you think you broke it. Don’t step on it, wind it too tight, etc. Be careful if you break or cut fiber, as the filaments are like very nasty splinters. Wash your hands!

Physically, the difference comes down to the size of the fiber core.

Single-mode fiber is typically 8.3 µm to 10 µm (micrometer or micron) in diameter, where 9 µm is the ideal. This is very small.

Single-mode can carry data at 10Gbps up to 50 miles with commercial equipment
Single-mode can carry data at 1Gbps a very, very long way
Single-mode fiber doesn’t have any different “qualities” or “sizes” - SMF is SMF is SMF

Multi-mode fiber is typically between 50 to 62.5 µm in diameter. This is not as small. The smaller (50 vs 62.5), the better.

Here’s a table I’m stealing from Wikipedia:

Fiber Quality	1 Gb	10 Gb	40 Gb	100 Gb
OM1 (62.5/125)	275m (SX)	33m (SR)	Not supported	Not supported
OM2 (50/125)	550m (SX)	82m (SR)	Not supported	Not supported
OM3 (50/125)	550m (SX)	300m (SR)	100m	100m
OM4 (50/125)	1000m (SX)	550m (SR)	150m	150m

The old “military-grade” multi-mode (50 µm) is just as good as the laser multi-mode, EXCEPT at 10+Gbps speeds
Old multi-mode (62.5 µm) can carry 1Gbps about 275 meters or 10Gbps about 33 meters - fine for racks
Multi-mode comes in 4 flavors: OM1 (62.5 µm), OM2 (50 µm), OM3 (50 µm laser), and OM4 (50 µm laser-enhanced)
The second number (125) is the size of the cladding around the fiber, ignore this.

So, because single-mode fiber is more expensive, it tends to only be used in situations where the distance travelled is too far for multi-mode. One common way it’s phrased: “multi-mode inside the building, single-mode between the buildings”.

The reason that OM3 and OM4 are “laser” is because they’re compatible with laser optics (40/100 Gbps), as opposed to the LED-only optics of OM1 and OM2. The laser optics are way better, as you’ll see later.

In order to make the single-mode and multi-mode fiber easily distinguishable, there is a color code:

yellow = single-mode

orange = multi-mode

aqua = laser/laser-enhanced multi-mode

The key to understand is that these are not only physically different CABLES, but also different SIGNALS. With copper networking, you can use a CAT5 or a CAT6 or a CAT7 cable with the same NIC. With fiber networking, ALL PIECES HAVE TO MATCH.

If you want to do single-mode fiber, you have to have:

Single-mode fiber all the way
Single-mode optics of the same quality on both sides

If you want to do multi-mode fiber, you have to have:

Multi-mode fiber of the same quality/size all the way
Multi-mode optics of the same quality on both sides

There is no mixing/matching.

Optics

Now, we know all about the fiber cables themselves. Optics are the ports on your switch/device where you plug in the fiber cables. More powerful optics = faster/longer distances. You have to match your optics on both sides (of course). You also have to get the right optics for your fiber type.

Note that the multi-mode optics don’t ever have a designated cable size that they work with. The same SX optic will work with OM1/OM2/OM3/OM4 cable just fine. The only exception is the newer laser multi-mode optics, which only work with OM3/OM4, as listed above.

These ports are often described based on their SIZE.

GBIC - Gigabit interface converter

These are the really big ports. We don’t run into these anymore, and they only run 1Gbps.

SFP - Small Form-factor Pluggable Transceiver

These are pretty common. Again, they only go to 1Gbps. Commonly (and mistakenly) referred to as “mini-GBIC”.

For multi-mode fiber, you use an SX optic.
For single-mode fiber, you use an LX or EX optic for 1310nm, or ZX or EZX optic for 1550nm. I haven’t seen any other options.

So, if you see an “SX” optic, you know immediately it’s a multi-mode optic in the SFP form factor with 1Gbps capabilities.

XFP - 10 Gigabit Small Form Factor Pluggable

The “X” is for 10, as in 10Gbps

This is the old 10Gbps standard, don’t see it in any new deployments, really was only used for a little while to replace XENPACK (the GBIC of 10Gbps)

Sort of like HD-DVD… it lost the war. SFP+ ports are smaller and consume less power, so SFP+ has taken over as the standard.

Doesn’t fit into an SFP/SFP+ port and isn’t backwards-compatible at all

SFP+ - enhanced small form-factor pluggable

These are the new 10Gbps options for SFP-sized slots. They look the same as SFP! They go to 10Gbps.

For multi-mode fiber, you use an SR optic. (SR is commonly called “short reach”)
For single-mode fiber, you use an LR optic. (LR is commonly called “long reach”)
For single-mode fiber at LONG range, you would use an ER optic, which is the extra-long distance (or “extended reach”).
SFP+ slots are almost always backwards-compatible with SFP modules

QSFP+ - quad small form-factor pluggable

These are used for 40Gbps speeds, are are NOT the same size as SFP/SFP+. You will need special ports on your switch for QSFP+. These can also often be broken out into multiple SFP+ connections, with a breakout cable. This gives you 4 different connections with only one physical switch port, at the cost of some cabling mess. The different optic types are also called SR4, LR4, and ER4, similar to the SFP+ naming.

Multi-mode:

SX - SFP - 1Gbps

SR - SFP+ - 10Gbps

SR4 - QSFP+ - 40Gbps

Single-mode:

LX - SFP - 1Gbps

EX - SFP - 1Gbps long-range

LR - SFP+ - 10Gbps

ER - SFP+ - 10Gbps long-range

ZR - SFP+ - 10Gbps ultra-long-range (80km rated)

LR4 - QSFP+ - 40Gbps

ER4 - QSFP+ - 40Gbps long-range

Connectors

ST - “Stick and Twist” - also called BFOC

This is the “old school” style and is used on about 50% of patch panels that I run into. Almost exclusively used for patch panels, never for switch/device. You have to twist and push the cable to lock/unlock it, which always makes it a huge pain.

FC - “fiber connector”

My friend Marshall says he’s seen this but I rarely see these in the field.

LC - “small connector”

The real name for this connector is Lucent Connector, but that’s not what I call it. The joke is that LC = Small while SC = Large.

It’s backwards… but if you can remember that, you’re golden.

LC is the most typical cable that switches/devices use. So your device/switch is almost always LC.

SC - “large connector”

The real name for this connector is Subscriber Connector. This is the replacement for the ST-type connector. This is usually what you’ll see on your patch panels or drops.

So, for cabling from a fiber patch panel to a switch, you need an SC-LC cable. From switch to switch, you want an LC-LC cable. From switch to server, you want an LC-LC cable. There are always exceptions, but these are the standards.

These connectors are used for BOTH multi-mode and single-mode fiber of all types/qualities. So the connectors don’t tell you ANYTHING!!! What matters is the fiber inside.

Examples

10Gbps - We wanted to do 10Gbps backbone cabling for a client. They had distances of over 300 meters for some runs. What fiber cable would we use?

Because the distances were 300+ meters and the new OM4 multi-mode fiber wasn’t out yet, we had to go with single-mode fiber. We had their fiber guy run SM fiber in the building. What optics would we use?

Because we needed 10Gbps speed and single-mode capabilities, we used LR optics, which came in the SFP+ form factor. What connectors would we need?

Because we were going switch-to-patch panel, we used LC-SC (or LC-ST if using different patch panels) connectors.

Solution: SM fiber + LR optics + LC-SC connectors

1Gbps - You want to set up 1Gbps cabling for your IDF switches. Your max distance is 350 meters. What fiber cable would you use?

Because we need to get to 350 meters, OM1 multi-mode won’t reach. We would use OM2 multi-mode, since it’s the cheapest and reaches 550 meters at 1Gbps speeds. What optics would we use?

Because we are using multi-mode cabling and only need 1Gbps speed, we would use SX optics which come in the regular SFP form factor. They’re cheap and do 1Gbps over multi-mode. What connectors would we need?

Because we’re going switch-to-patch panel or switch-to-switch, we’ll need either LC-SC connectors or LC-LC connectors, depending on how it’s cabled.

Solution: MM fiber (OM2) + SX optics + LC-LC connectors

HP Moonshot - low power, nothing else

madsushi@gmail.com (Chase Christian) — Wed, 10 Apr 2013 06:02:40 +0000

Here’s the pitch:

It’s a vertical-mounted blade chassis that’s loaded with low-end blades.

45 blades per enclosure
1 CPU (Atom - these are awful) and 8GB of RAM (single stick) per blade
1 hard drive per blade – SATA or SSD
2x 1Gb ports per blade
4.3U per enclosure
45 blades per 4.3U or 10.5 blades per U, which means ~11 proc and 88 GB of RAM per U
Low density compared to a c3000/7000 or DL360s
10Gb uplinks from enclosure
Only requires 2x 1200W (redundant) PSUs, which is less than the 4-6x 1200W PSUs that are commonly needed for a c3000 chassis with 8 blades.
Moonshot is a POWER PLAY (no hockey pun intended) - HP says 89% less power.
Did I mention no support for Windows? Only officially supports Linux distros at this time.

My guess is that this box is aimed squarely at companies like GoDaddy and Rackspace, where they’re carving out virtual instances for clients and they want to cut datacenter power utilization as much as possible. I wouldn’t want to be the guy buying 45 VMware licenses. Unless your workload can easily be sharded onto a lot of servers, this design probably isn’t for you.

My backpack (with bonus car trunk action)

madsushi@gmail.com (Chase Christian) — Sat, 30 Mar 2013 05:11:41 +0000

My backpack is heavy. Every time a colleague tries to pick it up, they’re surprised and dismayed by the weight of the thing. I tend to be overprepared, and my backpack is the natural extension of that mindset. I decided to crack it open to catalog its contents.

I won my very sturdy OGIO backpack in a company contest during my first few months. It’s about 6 years old, has tons of pockets, and includes the very old “Juniper Your Net” logo from Juniper Networks.

I’ll have to see if Bill Kelly can get me an updated version.

Sorry for the awful picture quality. Well, not sorry enough to get some more lamps or ask my wife to use her Nikon D7000, but sorry enough to apologize here on my blog.

Two NetApp DB9-to-RJ45 console adapters.
A classic “baby blue” Cisco DB9-to-RJ45 console cable.
My kneepads, which have saved me thousands of dollars worth of pain and discomfort on datacenter floors. $10 on Amazon.
Some really, really sharp scissors.
A hex screwdrive, a flexy-neck ratchet screwdriver, and a regular orange screwdriver. The orange one came from my first Riverbed install. It used to have Riverbed branding all over it, but it’s been so well-used that the logos have worn off. What a great tool.
A boxknife. Because I open lots and lots of boxes.
My laptop’s power supply.
A Logitech G400 (the MX518 successor)
An extra battery pack for my laptop. My aging HP laptop (5+ years old now) only gets about 30m of charge by itself, this pack adds another 45m.
Two cheap Bic pens from the 100-pack on my desk, a Logitech pen (red), and a random clicky pen I must’ve picked up.
My small screwdriver set. The only standard-issue item in my pack, CIO gives a pack of these to every new engineer.
A pair of nail clippers.
A Cisco USB-to-ethernet adapter, gigabit. Yes, I know gigabit is overkill for a USB2 adapter, so sue me.
A Cisco USB-to-wifi adapter, with N capabilities.
An Apple USB-to-lightning cable, which I have on reserve in case I need to charge my phone.
Exactly three RJ11 heads that somehow got lost.
A roll of yellow electrical tape, which I use for impromptu labeling and marking.
A collection of Sharpies.
Two null-modem serial adapters.
A Juniper-style DB9-to-RJ45 (female) adapater.
A serial gender changer.
My USB serial port, which is attached to my official Extreme Networks-branded serial cable.
My null modem serial cable, clearly marked “NM” on both sides of both ends.
My straight-through serial cable, clearly marked “ST” on both sides of both ends. This was a replacement cable. My original ST came from a former engineer, who gave me her prized ST cable when she left. I had to leave it in Madison, WI for many unfortunate reasons. I still use it (remotely) from time to time.
A very long NetApp serial cable.
A pack of CAT5E cables: a 1” yellow cable, a 3” white cable, a 10” black cable, and a 12” black cable.

I also have a pouch on top that holds my business cards and a few misc items: my Square, a NetApp flash drive loaded with software, and a 16GB flash drive.

Why stop at my backpack? I also keep several “go bags” ready for specific tasks. My phone bag is one of my favorites. When I’m not designing a complex enterprise virtualization/networking/storage environment, I’m running and punching analog lines and making test calls to 911.

The “go bag” itself, a dual-sized zippy bag.
Exactly four RJ45 heads. I hate making RJ45 cables, since they transfer a digital signal and are heavily relied upon.
Exactly forty-five RJ11 heads (I counted). Yes, these are just jiggling around loosely.
My favorite wire strippers.
A handful of jumpers for 66 blocks. These particular ones are nice because you can grab them with your fingers since they have a little handle.
A regular RJ11 cable.
Miscellaneous pieces of wire/RJ11 for patching in analog lines and the like. Took about 100’ off of a spool somewhere, and I’ve been using it ever since.
A set of RJ11 splices, to put two pieces of RJ11 together. Sort of like wire nuts, but better.
My basic punch-down tool.
My cable crimper, with a fuzzy velcro lock. Why the velcro? Because I once cut my index finger bad enough on its razor and bled all over my backpack while rummaging around for a pen.
A wire stripper/cutter.
No clue, came with the set.
My Fluke punch-down tool.
My Fluke butt set. Phone guys are drooling at this point.

There are a few more trick items in my phone gear repertoire…

This is a spare ShoreTel amphenol + patch-panel combo, often called a “harmonica” cable.

This is my EnGenius PoE injector. Key fact: it’s gigabit. Usually paired with:

This is my secret sauce. It’s a gigabit switch with a mirrored port -and- PoE passthrough. I am able to easily Wireshark the traffic from a PoE VoIP phone!!!!! Seriously this is one of the most baller tools in my kit. Took me a long, long time to find a reasonably priced device that allowed me to do mirroring and PoE passthrough, both at gigabit speeds.

Here are some bonus picks of the stuff in the trunk of my car. A big tangle of cables:

My crossover and T1 and special cable converters.

My secret weapons at company picnics.

My power driver/drill.

My awesome wire snips.

My fake USB CD-ROM / HDD enclosure. So great. It emulates a CD-ROM and presents the ISO you preloaded into the _ISOs folder.

Some more miscellaneous stuff.

Over ten regular (C13) power cables.
Some of the C14 power cables.
My “every possible option” screwdriver set.
A really awful label maker that I never use.
A traditional SATA/IDE to USB kit.
Lots of Fry’s brand AA batters.
Another regular phone cable.
Some cage screws
A book.
My needle-nose pliers for phone work.
Another well-marked ST serial cable.
A well-marked NM serial cable.

Hopefully you have enjoyed this trip through a lot of my stuff. I didn’t even get to show you my 300’ ethernet cable (used for when someone forgets to put a drop in the CEO’s new office).

The IT laptop

madsushi@gmail.com (Chase Christian) — Tue, 19 Mar 2013 05:11:53 +0000

I want the following laptop, which will never exist.

15” screen with a 16:10 ratio and a vertical resolution of 1200px or greater. Retina-like PPI would be nice, but unlikely.
A DB9 serial port. Infinite points if it actually had a serial cable wound up inside that I could pull out and plug in and then retract when done.
Two gigabit ethernet NICs. Yes, two.
A trackpoint.
A touchpad, with buttons.
A keyboard that includes all critical keys, like Ins/Del, Home/End, PgUp/PgDn.
Configurable up to 16GB of RAM.
Latest Intel processors.
Dedicated GPU with reasonable gaming power, see nVidia 6xxM-series.
VGA, DVI, and HDMI ports.
The latest and greatest tech, like USB3, SATA 6Gbps, etc.
No optical drive. It’s unnecessary in today’s world, especially with a Zalman optical-emulating hard drive enclosure.
Two hard drive bays. I don’t really need this, but it would be nice to have a nice 256GB SSD and also some raw space for tossing random ISOs.
Four USB ports. Yes, four.
Three hours of battery life. I don’t expect this thing to run without juice, but just long enough to get some work done during a reasonable car ride.
Weight and size are negotiable. I don’t need it to be thin and light, I expect it to save me from carrying around tons of dongles and cables and extras.

I would pay a lot of money for this laptop. Unfortunately for me, the laptop manufacturer’s designs are diverging in the opposite direction. The only manufacturer making high-res screens with powerful GPUs is Apple, who doesn’t even include a single ethernet port on their laptops. The opposite side is Lenovo, who has the technical oomph but is desperate to emulate Apple’s success by dropping buttons and keys as fast as they can but without replicating the screen resolution. Creating a laptop of my own, especially with such a custom set of requirements, is impossible. I can build a desktop to any possible spec and standard that I like, but we’re locked into the laptop models offered by today’s manufacturers. Unfortunately, those manufacturers are angling to take a larger share of the consumer market, not the enterprise market, which is working against me.

Saving HTTP basic auth passwords with Chrome

madsushi@gmail.com (Chase Christian) — Sun, 17 Mar 2013 22:59:14 +0000

If you’ve ever tried to save your credentials for an HTTP basic auth form in Chrome, you’ve run into problems. Chrome won’t let you save the credentials natively. You might read a few articles about mixed results or the feature working in previous Chrome builds, but that’s not the way to solve your problem.

The easiest solution is to open up Firefox or IE, go to your HTTP basic auth site, log in, and save the password in FF/IE. Once the password is saved in that browser, go back to Chrome, and tell it to import the saved passwords from the other browser. It will properly import the credentials and will auto-populate the HTTP basic auth popup. It takes some extra work, and it certainly shouldn’t be so difficult, but this trick will get you where you need to go.

Palo Alto Networks - Using a dynamic public IP address

madsushi@gmail.com (Chase Christian) — Sun, 24 Feb 2013 19:41:43 +0000

I use a Palo Alto Networks VM-100 at home as my primary firewall. I previously used a Juniper SSG5, but the VM-100 has more horsepower and way more options. I’m using a my home ESXi server (a whitebox) to run the VM-100, and it’s very fast. The only issue I run into is that I am on a residential internet connection, so I have a dynamic public IP address that my ISP can change at any time.

In order to configure inbound NAT policies when you have a dynamic public IP address, you’ll have to use some tricks with dynamic DNS and address objects to make it work. I wish there was a cleaner way to do this, but I haven’t found anything. The new version of PANOS has some features where it can poll an XML server for IP addresses to add to an address object, but the Palo Alto’s XML export API doesn’t match the required XML syntax.

Configure your public interface

Your first step will be to use the Interfaces section under the Network tab to configure your public interface. It will probably be in the Untrust Layer 3 zone. In the interface properties, you want to go to the IPv4 tab, and then set the Type to DHCP Client and ensure that both boxes are checked. The interface will now automatically get a public IP address from your ISP, and will create the proper route in your routing table.

Install & configure dynamic DNS updater

Dyn has a service called DynDNS, where their software will automatically probe your public IP address, and will then update a public DNS record with your new IP address. The Juniper SSG5 used to be able to do this on its own, as it had a DynDNS agent built-in. The Palo Alto Networks firewalls don’t have this feature, so you’ll have to install the software from Dyn onto any of your home PCs or servers to facilitate this. DynDNS Pro is the cheapest service, which is $20/year for users. I had an account with Dyn before they started charging, so my account has been grandfathered in for free. If you’re just getting started with them, you’ll unfortunately have to pay the yearly fee. There are alternative options, like No-IP, which still offer free dynamic DNS services. Any dynamic DNS service will do, but I would recommend sticking with Dyn if you already have a free account with them, or trying out No-IP if you want a free solution.

Depending on which dynamic DNS service you use, you’ll get a custom domain name that will match your public IP address. For example, I could use “chasefirewall.dyndns.org” as my dynamic domain name that maps to my home IP address. Once you have picked a service and configured their updater software, write down the dynamic domain name that you picked (or were assigned) and move on to the next step.

Create a FQDN address object

Navigate to the Addresses section under the Objects tab to create a new FQDN address object. You want to create an object that will be a stand-in for your public IP address, and we’ll tell the object to use a DNS lookup to your dynamic domain name to grab the right address.

Now, in order for this to work properly, your Palo Alto Networks firewall needs to be able to do DNS lookups to resolve your dynamic domain name to the proper IP address. Make sure you’ve configured your firewall’s management interface with DNS servers and a default gateway.

Note that you can (and probably should in 99% of cases) use your Trust interface for your default gateway, which will send your management interface’s traffic through your firewall and NAT policies to the internet.

The firewall will refresh the object regularly, so the object should stay relatively accurate. Even though your address may be dynamic from your ISP, the IP itself tends not to change that often.

Create your NAT and security policies

When creating your policies, you always reference the object that we created as the Destination Address in both the NAT and security policies.

NAT Policy:

Security Policy:

As you can see, I use my new Untrust-IP object in both policies. Because we used an FQDN Address object type, the IP address will be regularly updated and our traffic will always be allowed in.

If you try to cheat by setting the Destination Address in your NAT policy to “Any”, you’ll get a commit error.

Hopefully you find this guide useful and you’re able to set up some inbound NAT/security policies for your Palo Alto Networks firewall even though you don’t have a static IP address from your ISP.

Clearing your WoW creature cache automatically

madsushi@gmail.com (Chase Christian) — Mon, 18 Feb 2013 05:39:00 +0000

Tired of having to clear your WoW creature cache for NPCScan manually? I wrote a little script to do the same job.

First, make a batch file. You can just open up Notepad and make a new document. Here’s the command you want to run:

@ECHO OFF
del "C:\Program Files(x96)\WoW\Cache\WDB\enUS\creaturecache.wdb"
"C:\Program Files(x96)\WoW\wow.exe"
exit

Save this file as clearcache.bat and put it somewhere you’ll remember, like your C:\ drive or your Desktop. If you have trouble making it yourself, you can clearcache, which assumes your WoW is installed at the default location. Download it to your C:\ drive and unzip it, so that it’s in the folder C:\clearcache and unzipped. Now, when you run this batch file, it will clear your cache then launch WoW, so it’s done for you every time! You can also move the batch file to your Desktop if you like and click it there.

If you want to pin this to your Taskbar, you’ll have to use a custom shortcut, since you can’t pin batch files OR shortcuts to batch files. First, make a shortcut to your batch file (right-click and Create Shortcut). Then go into the shortcut properties and in the Target field, add a cmd /C before your command, so it looks like this:

cmd /C "C:\clearcache\clearcache.bat"

Then you’ll be able to pin it properly. I also made clearcache with both the batch file and the modified shortcut ready. You might need to modify the files to point to YOUR specific WoW installation.

Trying out Citrix XenClient

madsushi@gmail.com (Chase Christian) — Sun, 10 Feb 2013 20:49:26 +0000

I was very excited when Citrix announced XenClient, their client hypervisor, in 2010. The idea is that it’s a Type 1 hypervisor (bare metal) as opposed to being run on top of Windows or another OS. It’s based on their Xen/XenServer architecture, and the idea was that you could run multiple operating systems on your laptop or desktop at once. It would be a significant improvement over our current two options: dual booting or Type 2 hypervisors (Virtual Box, VMware Workstation).

Unfortunately, their version 1.0 software was awful and didn’t support most devices. I have an older laptop that I inherited from another employee, and it’s over 5 years old now. I wasn’t on their hardware compatibility list (HCL), and so I couldn’t give it a spin. I have always wanted to be able to replace Cygwin with a real Linux environment on my desktop, but I find that VMware Workstation is simply too clunky and slow for what I want. Now, in 2013, I heard XenClient mentioned again and thought I’d give it another shot. Surely Citrix has made several improvements since their 1.0 days, right?

Citrix and the naming disaster

Citrix has a serious, institutional issue with product naming. They constantly rename and rebrand core products, incorporate different suffixes and affixes (see: Xen), and have different versions of a product bear the same name. As a Citrix partner, it has been very frustrating at times, because it becomes very difficult to search for documentation and communicate effectively when the product names are shifting constantly or don’t represent the actual product. Anyone who has tried to research the Citrix Access Gateway and its various incarnations will know exactly what I mean.

It wouldn’t be the Citrix we know and love if this pattern didn’t continue to their XenClient platform. The whole issue is exacerbated by superannuated references on their website that reference old names, incorrectly name new products, and when all of their links always end up at the same page. From all of my research, it looks like there are 3 different versions of XenClient, although the naming structure is almost identical.

XenClient Enterprise 4.5 (XCE)

Locked-down and restricted version of XenClient (no SSH or root access)
Highly compatible with many, many laptops (including mine)
Designed for use with a managed Synchronizer server, but not required
Free for individual use
Due to restrictions in place, no options for Mac OS X and poor Linux support
None of the hotkeys for XenClient seemed to work in XCE

XenClient XT 3.0

Security-focused version of XenClient
NOT the successor to XenClient 2.1
I didn’t actually try this one due to bad reviews

XenClient 2.1

This is the original evolution from the XenClient 1.0 platform
Poor support for older laptops and nVidia chipsets (both apply to my laptop)
Tons of features/options, including special flags for Mac OS X and Linux support
All of the cool hotkeys, SSH access, shell/root access

If you’re following along, there’s a 2.1 version, a 3.0 version, and a 4.5 version, all listed under XenClient, but that are all totally different products and unrelated to each other.

I ended up installing XCE 4.5, assuming that was the latest build of XenClient. Of course, this was the locked-down enterprise version that didn’t have any of the options I wanted. After some reading, I found that Citrix acquired Virtual Computer and merged it with XenClient to make XCE. The original XenClient was still stuck on version 2.1, which had been released over a year ago. I tried installing XenClient 2.1, but found that my laptop was still not compatible and I could only get it to run by disabling several features and turning off graphics acceleration. I actually couldn’t even install it via DVD (since my laptop’s optical drive is PATA not SATA), nor via USB thumb drive or USB CD-ROM (since it disables legacy USB support during install). I had to install the whole thing via HTTP using IIS from my Windows desktop!

It’s clear that XenClient Enterprise is where Citrix is taking XenClient in general, and it makes sense. Their customers want locked-down hypervisors where they can control the images that are deployed, and that’s what XCE brings to the table. The old XenClient, which let advanced users tinker with a Type 1 hypervisor on their laptop, was just a tech toy that has now evolved into a marketable product. Unless I buy a new laptop specifically with XenClient support, I’m out of luck. Even then, who knows if XenClient 2.1 will ever see another update? Maybe I’ll try again in 3 years and see if there have been any improvements.

Palo Alto Networks - Understanding NAT and Security Policies

madsushi@gmail.com (Chase Christian) — Sun, 10 Feb 2013 20:04:24 +0000

When creating your NAT Policies and Security Policies on a Palo Alto Networks firewall, you have understand how the Palo Alto runs the packet through its various filters. I found a great Palo Alto document that goes into the details, and I’ve broken down some of the concepts here.

NAT policies are always applied to the original, unmodified packet

For example, if you have a packet that arrives at the firewall with:

Source IP: 192.168.1.10 (your private)

Destination IP: 8.8.8.8

then your NAT policy must have those IP addresses listed. Similarly, for incoming traffic, say from:

Source IP: 8.8.8.8

Destination IP: 206.125.122.101 (your public)

then you must have those IP addresses in the NAT policy. Notice that for incoming traffic from the internet, you use your public IP address in the Destination field, not the private IP address.

NAT policies always match the original packet’s zone and IP information

For that same incoming packet, the policy will read:

Source Zone: Untrust

Destination Zone: Untrust

While this seems counterintuitive at first, you have to realize that the original packet has a public IP address for both its source and destination IP addresses, and public IP addresses all live in your Untrust zone. The NAT policy elements must all reference the original packet, so don’t think about where you want the packet to go, but rather where it came from.

Security policies are similar, as they also reference the original packet’s IP information before any NAT has been applied. So, for an inbound security policy, you would use:

Source IP: 8.8.8.8

Destination IP: 206.125.122.101

just like in the NAT policy. However, in security policies, you have to reference the translated destination zones. Assuming that you’re translating that public IP address to a private IP address, your security policy would use:

Source Zone: Untrust

Destination Zone: Trust

since the translated IP (which you’re NOT referencing in the policy) is in the Trust zone. For some environments, your Destination Zone could be a DMZ instead.

Here’s a cheat sheet:

Outgoing Traffic

NAT Policy:

Source Zone: Trust
Destination Zone: Untrust
Source IP: Original Private IP
Destination IP: Original Destination IP

Security Policy

Source Zone: Trust
Destination Zone: Untrust
Source IP: Original Private IP
Destination IP: Original Destination IP

Incoming Traffic

NAT Policy

Source Zone: Untrust
Destination Zone: Untrust
Source IP: Original Public IP
Destination IP: Original Public IP

Security Policy

Source Zone: Untrust
Destination Zone: Trust
Source IP: Original Public IP
Destination IP: Original Public IP

Hairpin Traffic

(internal clients accessing internal resources via public IPs)

NAT Policy

Source Zone: Trust
Destination Zone: Untrust
Source IP: Original Private IP
Destination IP: Original Public IP
Translated Source IP: Your Untrust Interface/Public IP
Translated Destination IP: Translated Private IP

Security Policy

Source Zone: Trust
Destination Zone: Trust/DMZ
Source IP: Original Private IP
Destination IP: Original Public IP

Palo Alto Networks – User-ID agent configuration

madsushi@gmail.com (Chase Christian) — Sun, 03 Feb 2013 03:59:23 +0000

The Palo Alto Networks firewall can detect the Active Directory names of users on a network and match those names against security policies. A User-ID agent will check the Active Directory domain controllers for Event Log entries that are generated that contain user names and their client IP addresses. Using this information, you can make rules that filter traffic based on particular users (or groups), allowing you to fine-tune your policies.

What you’ll need:

The User-ID agent software from the PAN support site (optional, recommended)
A Windows server to install the User-ID agent on (optional, recommended)
A domain admin account for polling the domain controllers (or a service account)

Install the User-ID agent

Download and install the User-ID agent on a Windows server. I prefer using an agent on a server rather than using the built-in firewall agent, as it reduces the usage on the firewall and offloads it to the agent.

You’ll want to edit the settings to fill in your Active Directory administrator account. You can leave the rest of the settings at their defaults. Next, visit the Discovery tab to select the domain controllers you want to poll.

If you hit the auto discover button, it will find all of your domain controllers. Make sure to add any that it missed, or to delete any that you don’t want to poll. That should be all the config you need on the agent.

Configure the User-ID Agent

The next step is to add a new User-ID Agent in the User-ID Agents tab in the User Identification section under the Device tab.

Port 5007 is the default User-ID agent port, so make sure to punch that in. You shouldn’t need a pre-shared key, but I filled one in anyway. Now you can use Active Directory user names in your policies, or you can set up an LDAP server profile to do group-based mapping.

Palo Alto Networks - LDAP and Group Mapping config guide

madsushi@gmail.com (Chase Christian) — Sat, 02 Feb 2013 06:14:02 +0000

In order to configure your Palo Alto Networks firewall to do filtering based on Active Directory (LDAP) user groups, you have to configured the firewall to poll your domain controllers for group membership information. The User-ID agents only identify the user names of your users, but in order to sort them into groups, you have to configure Group Mapping.

What you’ll need:

The name and IP address of your domain controllers (and the domain)
A Bind DN and password of an AD administrator (CN, OU, DC, etc)
The Active Directory groups you want to add to Group Mapping

Create LDAP Profile

The first step is to go to the LDAP Server Profiles section under the Device tab. We’ll be Adding a new LDAP Server Profile.

For the server column, just fill in the name of the server. Obviously you put the IP address into the address column. Finally, pick your LDAP port, which is 389 by default.

For domain, you want the NETBIOS name of your domain, NOT the FQDN. If you put the FQDN in here, you’ll get a mismatch from your User-ID agent and nothing will work. Select ‘active-directory’ for the LDAP type, and then fill in the base with your base domain LDAP string.

The Bind DN is the user account that the firewall will try authenticating with. If you set your view to Advanced in ADUC, you can go to the Attribute Editor tab of the user object and just copy the DN right from Active Directory. Enter in the user’s password. I suggest using a service account for this.

Create a Group Mapping

You’ll now be navigating to the Group Mapping Settings tab, which is the User Identification section, under the Device tab. We’ll be making a new mapping.

First, select the server profile that you just created. The update interval is the time between group refreshes, in seconds, so set it to something like 60 seconds. Finally, change the default object class to ‘user’ instead of ‘person’. Leave everything else at the default, then hit the group include list tab at the top.

Navigate the directory on the left and select the groups you’d like to map. You need to map any groups that you’ll be using in policies. In my case, I picked the “pan_nofacebook” group in order to block Facebook for certain users. If you’re not able to navigate the LDAP tree on the left, check your server profile for any errors.

Create a Group-based Policy

****Now that we’ve created our group mappings, we can use these Active Directory groups in our security policies. I whipped up a quick URL Filtering Profile to block social networking sites.

Head over to the Security section under the Policies tab, and we’ll put all the pieces together.

Create your policy from your Trust to Untrust zones, and select the Active Directory group in the source user section.

The groups you’ve selected for mapping with automatically show up when you go to add a source user. Now, go to the Actions tab.

I’ve selected my URL Filtering profile of ‘nofacebook’ which will block social networking for the users, while allowing all other traffic through. This policy will only apply to users that are in the ‘pan_nofacebook’ group that have their User-ID mapped by the User-ID agents. The final product results in targeted URL filtering application.

Palo Alto Networks - Active Directory authentication (via Kerberos) configuration

madsushi@gmail.com (Chase Christian) — Sat, 02 Feb 2013 02:53:57 +0000

In order to use your Active Directory accounts to log on to your Palo Alto Networks firewall, you have to configure the firewall to poll your domain controllers via Kerberos. Assuming that you’re running PANOS 5 or higher, the Kerberos agent is built-in and very easy to configure for access.

What you’ll need:

The IP addresses and host names of the domain controllers you want to query (pick at least 2)
The domain name of the domain your accounts are in
A list of the Active Directory accounts you want to grant access to

Make a new Kerberos Server Profile

The first step is to visit the Kerberos Server Profiles section under the Device tab. We’ll want to Add a new Kerberos Server Profile.

Make sure that the realm and domain match your Active Directory domain. In most cases, these should match.

When adding servers to the list, the server column is for the IP address, while the host column is for the DNS name. You don’t need to specify a port number if you’re using regular Active Directory domain controllers for authentication.

Make a new Authentication Profile

Now we’ll be visiting the Authentication Profile section under the Device tab. We’ll want to Add a new Authentication Profile.

As you can see, I selected the authentication method of Kerberos, and I chose the Server Profile that we made in the last step. I leave the Allow List with the default ‘all’ group active, since we’ll be restricting access later. This doesn’t give everyone access to the firewall, it just gives everyone access to the authentication method.

Add your administrators

Finally, we’ll be using the Administrators section under the Device tab to Add a new Administrator.

The name that you use here is the user’s Active Directory account name. For example, my account is often ‘ccadmin’ or something similar. Choose the Authentication Profile that you just created to allow the user to log on via Active Directory. Finally, choose the role/access you want to give this user. The ‘Superuser’ role is the highest level of permissions and has the ability to add and remove other administrators. The ‘Device Admin’ role can do anything on the box EXCEPT manage other administrators. You can also use the Role Based access to define more granular permissions and roles.

Log on with your new account

Now you’ll be able to use your simple username and password to manage your firewall.

Texas is football country

madsushi@gmail.com (Chase Christian) — Wed, 30 Jan 2013 06:19:08 +0000

I was in the Dallas-Fort Worth, Texas area for business last week. It was my first visit to the state, and so my expectations were largely set by what I had seen in the media. I was planning to eat steak every day, and for everyone to wear boots and say “ya’ll” often. Country music would be the de facto soundtrack, and I’d be surrounded by pickup trucks. I even thought about buying a cowboy hat.

While I did manage one steak per day and drank my fair share of sweet tea, most of my other Texas tropes proved to be unfounded. People wore sneakers, listened to classic rock, and drove Maximas and Corollas. What I did notice was an abundance of college stickers on the backs of those Maximas and Corollas. I had landed in Texas A&M country, from the looks of things.

Where I’m from in southern Oregon, you can’t go 100 yards without seeing a bright yellow “O” sticker on the back of a Tacoma or an S-10. Oregon Ducks fans are proud to display their team’s logo everywhere. Oregon fans have a customary greeting when passing each other in the street: “Go Ducks”. Perhaps a strong fist symbol is gestured as well, particularly on a Monday after a big win. It was Oregon country, and you knew it the moment you entered town.

In Santa Barbara, I feel alone in my college football fandom. None of my colleagues follow any college teams. I don’t see USC or UCLA stickers on the bumpers of the BMWs and Jettas that line State St. If I want to talk about a great play or an amazing game, I have to call my family or friends in other states. After nearly 6 years here, I’ve just come to assume that nobody else cares about college football as much as I do.

Wearing my Oregon Ducks jacket in Texas, I expected to be ignored, just like I am in Santa Barbara. Everyone here just assumes I just like the color green. What I found was that everyone in Texas is not only a college football fan, but that they’re also up-to-date on their information and love to chat about their favorite teams. The coat was a sign to them that I was a fellow follower.

I had over a dozen people stop me in stores, restaurants, my hotel, everywhere, just to talk about Chip Kelly’s recent departure from the Ducks, to give me their thoughts on the OC Mark Helfrich’s promotion to HC, and to brag about how Johnny Football was right to spurn Oregon to lead Texas A&M to its rightful SEC glory.

Their questions were eager: Had I seen how badly TAMU had crushed Oklahoma? Had I seen the pictures of Manziel living it up after their bowl game? Had I prepared myself mentally to see the Ducks face off vs TAMU in the natty next year?

Yes, in fact, I have.

I was around my types of fans once again. My wife went to the restroom at the airport, and even just two short minutes later, someone had already approached me to discuss whether Helfrich would be able to keep Kelly’s furious pace and discipline at Oregon for seasons to come.

I enjoyed getting to talk football in Texas, as opposed to being limited to just talking tech at work or gaming with friends. I hope to visit Texas again someday. I’ll just have to make sure it’s during a season where TAMU and UO don’t face off, as I want to make sure there’s no hard feelings.

Snapshots are not backups

madsushi@gmail.com (Chase Christian) — Thu, 27 Dec 2012 06:58:11 +0000

Maintaining backups of business-critical data is one of IT’s most crucial roles in an organization. While other departments are tasked with the creation or generation of that data, IT’s job is to make sure that data is available and preserved at all times. Businesses executives decide the company’s appropriate RPOs and RTOs, and it’s up to IT to make that design a reality.

Snapshots are a widely available tool that seems to promise incredibly near-line RPOs and nearly instantaneous RTOs. Every major SAN vendor has a snapshot technology available, and even operating systems like VMware and Microsoft have their snapshot stories. The truth is that snapshots aren’t really backups, and they cannot be relied upon as a backup.

What’s a backup anyway?

In order for a tool to be a true backup solution, the backed-up data must be:

On separate offsite media
Completely self-contained copy
Immutable

If you back your SQL databases up to simple *.bak files on the same drive as the database, you haven’t created a backup, but rather you’ve created an archive. An archive is a copy of your data as it exists at a point in time that doesn’t meet all of the requirements to be a true backup. Archives can be very useful for quick restores and solving minor issues, but only backups truly protect your company’s data. Backups can be sent to external media, like portable hard drives and tape, or remote storage, like a NAS, SAN, or server at another site. A server’s RAID array does not count, as a backup must be able to survive a total failure of the source, not just a single hard drive failure.

A true backup must not be reliant on the original data. VMware snapshots and Microsoft shadow copies both rely on the original source information in order to restore to that point in time. A new copy of the data isn’t created. The original file is preserved and required, which means that if it is lost, then the snapshots are worthless. You must be able to restore all of your data with nothing but the backup itself.

If you’re using a mirroring strategy like NetApp’s SnapMirror technology, you’ve created a highly-available copy of your data, but not a backup. Any corruption on the source data is replicated at a block-level to your destination, which means that you are always at risk of losing the second copy of the data to an error or attack. The destination side of a backup must be completely immutable, so that any corruption or errors on the source can’t affect the destination.

Snapshots are awesome

While snapshots aren’t true backups, they do provide a lot of value. They can allow administrators to quickly fix minor issues and restore data in most scenarios. They can also be used for testing, development, and cloning. Snapshots reduce the amount of labor needed to manage an environment and are generally good enough for daily use. The issue is simply that they can’t be relied upon in a major disaster, and any true backup strategy must go beyond snapshots.

SnapVault is the best of both worlds

NetApp’s SnapVault technology allows companies to leverage snapshots for both quick management and backup purposes. A SnapVault relationship between NetApp filers creates a separate copy of the source data on the destination filer. The destination copy is completely self-contained and isn’t reliant on the source filer for anything after synchronization. The destination copy is completely immutable and uses its own file table and metadata to ensure that corruption in the source is prevented from affecting backups on the destination. The destination can be regularly updated via automatic replication.

By meeting all of the requirements of a true backup solution, SnapVault allows companies to be sure that their data is secure. Without a solution that includes all of the requirements of a true backup, a company’s data is at risk of being lost.

Telnet isn't the problem

madsushi@gmail.com (Chase Christian) — Fri, 03 Aug 2012 17:43:29 +0000

I have been receiving a lot of feedback regarding my comments on the Windows telnet client from my article on Server 2012:

As a note for anyone at Microsoft reading this: not including **telnet** in the default installation is an egregious offense.

Between Twitter replies, emails, and comments here and on Hacker News, there are a lot of people who agree with Microsoft’s decision to leave telnet out of the default Windows installation in recent versions. I vigorously disagree with this notion, although I haven’t yet provided any facts to defend that opinion. I plan to rectify that today.

1 - Everyone else does it

Including telnet by default is not that great of a request. In fact, Windows used to include telnet by default, up until the Vista/Server 2008 wave. Other popular operating systems, such as Mac OSX, Ubuntu, BSD, and SUSE all include telnet by default. The only OS I could find that didn’t include telnet in the default distro was CentOS/RedHat. Have you ever heard of anyone uninstalling the telnet client from their Mac OSX installation because it was unsecure? Ubuntu is one of the most widely used server operating systems on the web, and comes with telnet pre-installed on the server distro. Nobody has a problem with these big players coming with telnet by default, why would Windows be any different?

2 - Unencrypted traffic is OK

If you have a problem with a server sending unencrypted traffic, then what about the **ftp **and wget clients that are included in most operating systems? Even CentOS has the unencrypted ftp client available in the default distro. The ftp and wget commands both send usernames and passwords in plain-text, in addition to all of the other packets they send/receive. Many of the protocols we use every day are unencrypted, such as FTP, HTTP, DNS, and DHCP. The key is that an educated user can still use unencrypted services if they understand the limitations and precautions that need to be taken. We are a long way away from a 100% encrypted internet.

3 - Telnet is outbound only

The telnet client does not represent a significant security risk because it does not listen for requests, but rather only initiates outbound requests. There’s no risk of outside entities attacking your telnet client. You choose when and who to initiate traffic to. There are no open ports, there are no always-running daemons.

The default FTP and HTTP tools have access to your hard drive and your file system, which would allow a man-in-the-middle to alter the file that you are downloading in-transit. FTP and HTTP protocols have been widely used to deploy viruses and other malware. These are unsecure protocols that are far more dangerous than telnet, and yet their presence is considered the default state. Telnet is a pure communication channel, and so it is not possible to be served a virus or other bad code through a MITM session.

4 - Telnet is useful

Telnet is a powerful tool for testing a variety of functionality. Telnet is crucial for the validation and troubleshooting of SMTP servers. Telnet allows you to test and confirm open ports and firewall settings. As long as you’re aware of the risks (unencrypted traffic), telnet is a tool that should be in everyone’s repertoire and on every server.

If you blocked all unencrypted traffic, the internet would be in shambles. Most email would stop flowing (SMTP), the majority of sites would go dark (HTTP), and file servers everywhere would disappear (FTP). You shouldn’t be using telnet to connect to your secure network devices, but you also shouldn’t be using HTTP to connect to your secure banking website. Telnet is a useful for plenty of other situations. When FTP or HTTP tools are included on every operating system, why leave out the as-secure and yet often more-useful telnet?

5 - Don’t blame the tool

We’ve all heard the stories of misguided administrators using telnet to connect to their secure network devices or very important data sources. We should be replacing telnet with SSH wherever possible, especially if there is any confidential data or passwords involved. Telnet is not even an acceptable management tool inside a firewalled network, because you never know if your network has been compromised. I am not advocating the use of telnet for purposes that do not suit it.

The fact is that there are use cases for telnet that make sense and that will not be replaced by encrypted protocols in the near future. A completely encrypted worldwide SMTP network is still years away, and until then, telnet is a valuable tool when working with mail. When working with other unencrypted services, telnet is a simple and effective tool for confirming firewall settings and doing basic testing. It’s not for managing your switches or servers, but rather as a basic networking tool like netcat or ping.

Copying WoW macro and addon settings during a realm transfer

madsushi@gmail.com (Chase Christian) — Wed, 01 Aug 2012 07:58:35 +0000

My guild is transferring from Azgalor-US to Mal’Ganis-US, so I figured this would be a good time to work on a project for copying my macro and addon settings between server profiles. You’ll need to know your new server and character names up front.

Your account information is stored in:

C:\Program Files (x86)\World of Warcraft\WTF\Account

Pick the folder for the account that contains the character you’d like to transfer. I have a total of 30 account folders in my current WoW install, which means I’ve logged into 30 accounts from my PC!

1 - Make a Backup

First, make a backup of the folder. Make a copy on your desktop or in your My Documents folder to ensure we can revert back if necessary.

2 - Prepare the Character Folder

Now, open up your account folder. You should see some server names listed as well as some other things:

There are a few different sections here. Basically, all of the files at this level are safe, as none of them are specific to your server or characters. What we’re concerned with are is the SavedVariables and Realm/Character folders. The files in these specific server folders and the generic SavedVariables folder all contain references to your old character name and your old realm name. We need to go through and re-do all of that in order to have everything carry over smoothly. We’re going to do the character folder first.

Navigate down to the character folder that you’ll be transferring. This puts me in folder:

C:\Program Files (x86)\WoW\WTF\Account~~\Azgalor\Cure

We’re going to use PowerShell for this next bit, so hopefully you’re using Windows!

Here’s the command:

gci -recurse | ? {$_.Attributes -ne “Directory”} | % {(gc $_.FullName) -replace “OLDNAME”,”NEWNAME” | sc -path $_.FullName }

Basically, this command is going to go through and rename every instance of “OLDNAME” with “NEWNAME”. We’ll need to run this once or twice. If you have the same toon name on the destination server, then you just need to change your old realm name to your new realm name and everything is peachy. If you are changing your name too, you’ll need to run it again to edit the name. Here’s what I ran exactly:

C:\Program Files (x86)\WoW\WTF\Account~~\Azgalor\Cure> gci -recurse | ? {$_.Attributes -ne “Directory”} | % {(gc $_FullName) -replace “Cure”,“Radsushi” | sc -path $_.FullName }

C:\Program Files (x86)\WoW\WTF\Account~~\Azgalor\Cure> gci -recurse | ? {$_.Attributes -ne “Directory”} | % {(gc $_.FullName) -replace “Azgalor”,“Mal’Ganis” | sc -path $_.FullName }

So only two commands here. Each one will take a bit to run, but that’s fine. Once you’re done, we have PREPARED THE CHARACTER FOLDER!

3 - Prepare the SavedVariables Folder

Navigate to your account’s global SavedVariables folder. This puts me in folder:

C:\Program Files (x86)\WoW\WTF\Account\~~\SavedVariables\

Here’s the command:

gci -recurse | ? {$_.Attributes -ne “Directory”} | % {(gc $_.FullName) -replace “OLDCHAR - OLDREALM”,”NEWCHAR - NEWREALM” | sc -path $_.FullName }

This time, we’re trying to find something like “Cure - Azgalor” and we want to replace it with “Radsushi - Mal’Ganis”. This won’t fix ALL of your addons, but it should fix most of them. The reason that we don’t run the other commands for this folder is that we don’t want to overwrite the values for your other characters that might be staying behind on the old server.

4 - Copy the New Folder

Now we need to copy the character folder to your new realm. If you don’t already have a folder for your new realm, create one. Then, copy your old character folder from your old realm (like \Azgalor\Cure) to your new realm (\Mal’Ganis\Radsushi):

Now you’re done! Kick back and fix the few addons that might’ve screwed up (Postal is one of the few that doesn’t like this process much).

My time with server 2012 and metro

madsushi@gmail.com (Chase Christian) — Sun, 29 Jul 2012 09:15:40 +0000

Over the past week, I spent my free time installing the new suite of Microsoft products onto my home VMware environment. I like to be ahead of the curve when it comes to new technology and software. I was an early adopter of Vista (unfortunately) and Windows 7 (just to escape Vista). Once I read about the preview release of Exchange 2013, I had to give it a spin.

The installation

I ran into a few issues getting the new Server 2012 operating system to work on my fully-updated ESXi 5u1 environment. Windows Server would start installing, but then stall after the first reboot. I was able to fix the issue by adding these lines to the *.vmx files for my new VMs via SSH/vi:

hypervisor.cpuid.v0 = FALSE mce.enable = TRUE vmGenCounter.enable = FALSE

Installation was fast onto my SSD-based datastore. I obtained my evaluation copies from Microsoft directly. I would recommend at least 4 cores for Exchange 2013 on Server 2012, as it was hitting 100% CPU usage on my original 2-core VM. A fresh Server 2012 install takes up about 12GB of space without any roles or features installed (except for telnet). As a note for anyone at Microsoft reading this: not including telnet in the default installation is an egregious offense.

I set up one Server 2012 server as a domain controller with DNS, another Server 2012 server as my Exchange 2013 member server, and a Windows 8 workstation with Outlook 2013 as a client. I used all IPv6 for my networking, and used my favorite StartSSL wildcard certificate to set up SSL. I also used my Ubuntu-based Linode VPS to relay SMTP for me, since port 25 is blocked by my residential ISP.

With the dcpromo command completely removed in Server 2012, I had to use the Roles/Features wizard (the Roles wizard and Features wizard are now combined) to add DNS and Active Directory Domain Services to my DC. I added a few necessary DNS records and then added my Exchange 2013 and Windows 8 VM to the domain. Finally, I installed Exchange 2013 onto my Server 2012 box, which required a couple of restarts and the manual uninstallation of the latest Visual C++ runtimes.

Issue 1: Hot corners don’t work

While configuring my domain and my Exchange server, I ran into my first major issue with Server 2012: hot corners. By moving my mouse to the top-right corner of the screen, I’m able to activate the hot corner, which brings up the new menu:

The issue with the hot corner design is that it doesn’t work in environments where the corners aren’t fixed. If you’re actually sitting at a physical server with a single monitor attached to it, hot corners are easy to use. Unfortunately, it’s 2012, and I can’t remember the last time I actually had to plug a monitor into a server. When I’m connected to my server via iLO, or RDP, or the VMware console, I can’t easily move my mouse into the corner of the screen, because my mouse can leave the window entirely. I am almost tempted to uninstall the VMware Tools just to lock my cursor inside the window.

The supposed answer to this issue is the Windows key, which drops you right into the new menu without using your mouse. The Windows key works, if I wanted to use my keyboard. Being able to navigate Windows with just a mouse is now incredibly difficult. This is a feature that was present in every preceding version of Window Server. I don’t understand why the addition of the hot corner has to preclude the existence of the Start button. Why not provide both options? The hot corners require more mouse movement and precision than a Start menu, and provides fewer benefits.

Hot corners were not designed with a mouse in mind, but I can guarantee you that 99.99% of all servers are not managed via touch. Server OS design needs to be mouse-centric, and the focus on touch negatively affects every admin. The fact that there is no “Classic” option makes it clear that Metro is here to stay, regardless of how negative its impact will be. Server 2008 brought a lot of UI changes with it, but they weren’t forced on us and I know a lot of admins that preferred Classic mode. Server 2012 brings changes that are even more dramatic with no chance to opt-out.

Issue 2: Side-to-side scrolling

In the new metro menus, Microsoft has replaced the typical up-down scrolling system with side-to-side scrolling, for no apparent reason.

Scrolling has been up-down since its origin, and even the physical wheels on our mouse match this design. We have PgUp and PgDown buttons on our keyboards, not PgLeft and PgRight. There are several menus that scroll side-to-side in Server 2012, and it’s disconcerting. In addition, using the scroll wheel here is painfully slow.

Perk 1: A dramatically improved System Manager

The new System Manager in Server 2012 is dramatically improved. My favorite feature is the landing page, which provides you with quick access to some of the most common administrative tasks:

I love the plethora of information that’s easily available here. It makes it very easy to spot-check servers to ensure everything is set up properly. In addition, the new System Manager is clearly designed to manage multiple servers quickly and efficiently. It’s easy to add and maintain servers:

I am pretty impressed by how easy it was to use a single System Manager to manage multiple servers. To make it even easier, RSAT options are now available without any downloads necessary:

Perk 2: Built-in IP address management (IPAM)

I have been looking for a good IPAM solution for years, and Microsoft answered the call. Their new IPAM modules allow you to plug into your existing Windows DHCP and DNS servers to manage all of your subnets. You can easily provision new scopes and globally document statically assigned IPs without relying on a Google Docs spreadsheet.

Issue 3: The “show password” button

I can’t think of any good reason for this button to exist, and I have to assume that it’s at least as insecure as including telnet by default:

Perk 3: The new file management options

The new copy/paste dialogs and the shortcuts in the file manager are welcome improvements from the older versions of Windows:

These features are especially useful for consolidating folders filled with various pictures.

Issue 4: Where did Network Settings go?

I was pretty upset when I couldn’t click on my networking icon in the system tray to change my IP address. The new Network and Sharing Center is completely useless, but yet it adds another hoop that I have to jump through in order to get to my adapter settings. Metro takes it a step further and now clicking on the system tray icon brings you a menu that gives you nothing of value:

In order to get to anything useful, I have to hit a hot corner, go to Settings, type in “network”, then select View Network Connections:

Conclusion

Luckily for admins everywhere, the Metro UI that we see in Server 2012 is only a fraction of what Windows 8 users will be facing. While I’m not happy about the lack of a Start menu and the clunky hot corner/search style, it’s much better than it could’ve been.

Reference points

madsushi@gmail.com (Chase Christian) — Tue, 24 Apr 2012 06:44:19 +0000

I can never remember when things happened in the past. Unless there was something specific about the date or year, I simply can’t recall the details. How many years ago did I buy my car? How long have I lived in this apartment?

The only way that I can figure out when something happened is by tying it to a reference point: one of the few dates that I actually remember. Some of the reference points are obvious due to their significance, while others appear minor even though they’ve lodged themselves in my mind.

I remember that I turned 13 during my first year at Rogue Community College. If an event happened before that, then I know we’re looking at pre-2000 years.
I remember that I started my current job on May 16th, because that was payday. Julie came around the office with paychecks, and told me that if I stuck around for two more weeks, I could get one too.
I remember that I worked for Three Rivers Hospital in Grants Pass from 10/03 to 10/06, because it was exactly three years.
I remember that I married Mory on 6/6/10, because if we had married 4 years earlier, it would’ve been 6/6/06.
I remember that the Omen came out 6/6/06 because it was the only movie I remember that exploited the date. The Dreamcast also came out on 9/9/99, which is also a reference point.
I remember that I bought my car on July 4th, because I was going to Fry’s for their Independence Day sale when we stopped by the Honda dealership.
I remember that I was never at home after I turned 16, so most of my memories with friends in Oregon were from 2003-2007.

If it weren’t for my handful of reference points, I wouldn’t know when anything happened. I just try to find the closest reference point to whatever I’m trying to remember, and then do some math to figure out when it actually happened. I have tried keeping a journal in the past, but I never kept it updated and it never worked the way I wanted it to. At work, my inbox is my timeline, allowing me to go back to the past. I like Facebook’s idea with timeline, but again, I don’t update there and so it’s not very valuable to me. Twitter only contains my fleeting thoughts and jokes, which makes it a poor guide to the past.

I guess I’m stuck with my reference points.

Why I am learning Linux

madsushi@gmail.com (Chase Christian) — Sat, 31 Mar 2012 21:56:51 +0000

I made the decision about a year ago to start learning Linux.

It’s pretty commonplace in the IT world already, and becoming more prevalent every day. VMWare’s domination over the virtualization world grows each year, *nix-based firewalls are now in the Gartner leaders quadrant, even many SANs today are based on *nix technology. If your career is IT, there’s no excuse to exclude Linux from your repertoire of skills. I hate saying “I can’t do that” when someone asks me, and Linux skills was a gaping hole on my resumé.

I was surprised to find that there’s so much in common between Linux and Windows. While people are often put off by the number of Linux distributions (distros or flavors), it’s no different from the various Windows builds, like Server 2000/2003/2008 or Windows XP/Vista/7. There’s a similar architecture in place, and each build or version is unique in its own way. Both Linux and Windows are designed to run on the same hardware, with a CPU, RAM, and hard drives. Because everything is based off of the same hardware model, there’s really nothing unique about either OS. Both have ways to install programs easily, run updates, and manage files.

The command line

Windows admins are often hesitant to learn Linux due to it’s command line-based interface. However, as Powershell and Windows Server Core continue to expand, we’re going to find ourselves at the command line more often. At some point, you have to bite the bullet and go back to the old days of cd‘ing your way around.

It doesn’t need to be user-friendly

I don’t use Linux for my “user” OS at all. My work laptop and my home PC both run Windows 7, and I’m unlikely to ever swap. There’s simply too many programs and too much hardware that’s incompatible with Linux, and it’s not worth the trouble. The Linux GUIs are inconsistent and constantly changing. The true value of Linux comes from its server capabilities.

I constantly see guides advising people to “switch to Linux” for a few weeks, such as Ubuntu’s ability to actually dual-boot off the same partition as Windows. No. You are doing yourself a disservice as an IT professional by not knowing the world’s most popular client OS, Windows. Windows’ hold on the client market is unlikely to budge any time soon. But, with Linux servers rapidly gaining market share in the enterprise IT world, you need to at least have a working knowledge of the technology to stay relevant.

By learning Linux over the past several months, I have already solved several client requests/issues that I would’ve previously been unable to address. Expanding my capabilities has allowed me to tackle more issues, which directly results in a more satisfied client and a financial win for my company.

Break it, then fix it

I began my Linux education by starting a project to run my own Mumble server. I had been paying money out of my pocket every month to my friend Antiarc for Mumble services, but I thought it would be a fun experiment to try running it on my own. I bought a small VPS from Linode on Antiarc’s advice, and installed my first build of Ubuntu server. I spent weeks poring over Linode’s library of Linux documentation and the internet, struggling with all of the challenges that come from learning a completely different OS and environment. After many long nights of struggling, I finally got my Mumble server up and running.

In the months since then, I now run a website, a wiki, my Mumble server, a mail server (Postfix/Dovecot/Squirrelmail), a proxy server (Squid), and my DNS servers for all of my domains on my Ubuntu servers. Between my two geographically-separate Linux servers, I’m getting thousands of different requests a day. I am always looking for the next Linux project I want to tackle. The amazing thing is that I’m doing all of this on servers with 256MB of RAM, which is a feat I’d have a hard time replicating on Windows.

Test post from iPhone

madsushi@gmail.com (Chase Christian) — Mon, 26 Mar 2012 20:02:25 +0000

Going to lunch with Bjoern and Marshall.

Bjoern is a crazy driver.

We’re headed to Panda Express.