The project started as something I was spearheading myself, and has since taken on a life of it’s own.  It’s now an official project with larger involvement, which means that it slows down a bit   I hope to have some more details in early January as we flesh out the new architecture we’ll be using for the system.

Developers on my team have spent the last month prototyping a new reservation and scheduling system that will allow us to control more devices from a central point within our automation.  This will include things like L1/L2/L3 switches, our own product, and various test harnesses from IXIA, Spirent, ANUE, and others.  Part of this framework will be used in this new harness.   What we foresee right now is a mixed harness – automated build testing during the day using our customized framework, and more complex customer scenarios during the night, using IXIA Test Conductor.

We have set aside two brand-new IXIA Acceleron-NP cards for this project – a huge amount of test capacity (more than 7 times what we originally had dedicated to the project) and re-tooled some of the ANUE and Spirent equipment involved.  We also may be adding some larger 10GB switches to support the different devices we’re planning on testing.

More details in a few weeks!

I wanted to post more of a question than anything.  At my work, and in several consulting related situations, I interact with Spirent equipment for various testing needs.  In the last year, it has become progressively more difficult to get responses and information out of Spirent for a number of reasons.  This has ranged from support calls, to software fixes and questions, to sales inquiries.

Are others seeing this?

Spirent has a lot of equipment and customers, and I have been contacted by a number of existing Spirent customers with similar concerns over the last year.  Most people do not have the demands and requirements (at least those I talk to) that my company does for performance test vendors, and as such, haven’t run into any of the problems we’re experiencing, or at least not to the degree we are.  Another way of saying this – we feel the pain more than most, faster than most.

I wanted to see if others are experiencing similar issues, or if there are things I am not aware of that should factor into this opinion.

It’s always hard to find folks who have test experience specifically related to performance, so I’m hoping readers of the blog who might know someone looking can pass this info along.  Thanks!

This time around, we’re focusing on two main use cases for this harness:

• Quick baseline tests between builds of product during the development process.
• Complex scenario testing that can be automated

For the first one, Test Conductor has a lot of great features that we’ve used in the past to help.  The most important being the pass/fail criteria that allows you to specify percentages of tolerance against a baseline.  It’s really quite slick and has saved us a lot of time – first you build your tests and regressions, and set the pass/fail criteria to have a baseline tolerance for a particular value or set of values.  Then, you run your regression and tell it to collect baseline values.  Once that is done, you are ready to run against something and see if the performance has varied.  Simply start the regression and run it normally, and the pass/fail criteria will evaluate your baseline and give you the result.

It’s a super simple way to check if performance has changed without complex analysis, graphs, reports, or worse – manual examination.  And it doesn’t have what many other solutions have – hard coded values and engineer input.  In our use case, we simply reset the baseline on the next build after we’ve validated things are OK, and get ready for the next build coming down the line.

We’ve designed a pretty focused set of tests that covers key features so we can try to get coverage on as many areas as possible.  It’s not the kitchen sink, just a quick validation point.  This enables our test engineers to wait for the result, which only takes a short while, before moving on to more detailed testing through our other large automation systems.

The second use case is one we’re hoping to take more advantage of with Test Conductor – complex scenarios that can be automated.  Why do I say “can be automated”?   Lots of scenarios are complex, and many of them are so convoluted that to automate them would take more time than it’s worth.  We test many of these already and have great solutions for them.  What I’m looking for is new ideas, new IXIA tests from customer visits, and complex existing configurations that the various other devices we have in the harness, such as the Apcon, can assist with.

So far I’ve come across a lot of features in Test Conductor that are going to make this far easier than I thought it would be.  In particular the functionality in Test Composer is going to be key to making all of this happen.  So far we’ve created procedures that allow us to gather basic information from our DUTs, switches, and other devices, review the information and make decisions on what to do within a test or regression – all in just a day or two.  And, without coding in TCL or some other language.  Key features we’re using so far include DUT Configurations, Procedures, Response Maps, and more.  As we get specific examples tested and better documented, we’ll post a couple here.

A lot of the press release is focused on Service Providers and wireless markets, which is an area that Shenick focuses on more than many of the other players.  Direct from the press release:

“NGN and LTE services and applications bring with them increased quality of service (QoS) and quality of experience (QoE) demands. Per-flow test and measurement is the only mechanism to guarantee performance at these granular levels enabling providers to pinpoint application and individual user issues accurately”

I think it’s great that this support has been added on both ends so that users of iTest can take advantage of existing or new Shenick equipment.  I hope there will be more details about how deep the integration goes in the coming weeks.

One interesting thing is the fact that both Fanfare and Shenick are part of the TesLA Alliance – I wonder how much the proposed TesLA standards have influenced this solution, and if we’ll see more of this in the coming months since the standards are getting closer to being approved…

Performance Harness Diagram

Part of my documentation is diagram of all of the equipment.  Some of this will include physical connections for those items that are static, but there will be a large portion of the equipment that is dynamic and changing based on the use of the Apcon Intellapatch switch we have in the mix.

In diagramming this stuff, I’ve found, to no big surprise, that vendors still don’t keep up with Visio templates.  What I was surprised about is how many vendors don’t keep up!

IXIA doesn’t have any of their newer chassis in the templates they make available through a 3rd party.  ANUE doesn’t have any templates I can find at all (hence the orange box in my diagram to the left).

Of those that had templates that were semi current, APCON’s came with fully populated chassis instead of the individual shapes so you could add the specific cards you want.  I’m sure you could unlock the shape, deconstruct some of it, and then go from there.  They do include a number of different cards in the template, just not undocked from the chassis.  F5 has some nice updated templates, but they seem more marketing than anything – 3D shadows and all.

Dell was the only vendor to have accurate and well thought out templates with the right shapes, sizes, combos, views (both front and back for everything) along with just about everything else you could think of.

Not that I use Visio constantly, and want pretty shapes, but I would have thought this would be pretty straightforward for vendors to keep updated.  Oh well.  Back to diagram-land…

• Apcon ports and connections
• Dell switch ports
• F5 BIG-IP ports and cabling
• All versions of software/firmware

So far, the majority of this documentation is within the systems themselves.  For Apcon, this was a little difficult to find in the web UI, but once there, it took a few minutes to input all of the port names.

The Dell switch is a known quantity as we’ve used these many times before for other projects.  That doesn’t make it any easier, as the syntax is slightly off from any of the others out there.  For instance, port names are unnecessarily long, and there are extraneous words you have to use to switch to a specific ethernet interface.  They’ve also chosen some really weird abbreviations for the 10GB ports – x – which is really hard to remember.  Regardless of their unique config, they are very good performing switches.  There was a Tolly evaluation of the Dell PowerConnect 6248 a while ago that really pushed us towards using these for many projects since they are so well performing, and for not a lot of money.

I’ve also started creating a high level network diagram which I hope to post here shortly to show how everything is connected.  Of course, the hope is that the Apcon and Test Conductor will allow me to make this a constantly changing network setup based on the various topology goals I have in mind, so the diagram will just be a starting point.

Finally, I’ve created a single link to see all the articles on the harness, which can be found here.  More info tomorrow…

• IXIA Test Conductor – central orchestrator, test and regression management
• IXIA X16 chassis with 8 ALM cards – generates L4-L7 traffic using the IxLoad application
• IXIA 400T chassis with 1 TXS card – generates L2-L3 traffic using the IxAutomate application
• ANUE GEM Network Emulator – used to emulate WAN conditions
• APCON IntellaPatch L1 switch – for changing port configurations and connecting/disconnecting devices
• Dell PowerEdge 1950 server - running CentOS 5.3 with a special Chelsio CX4 10GB network card and tuned software – more on this later
• Dell PowerConnect 6248 stacked switches – three of these provide all of the switching I need
• F5 BIG-IP 3600s – the devices I’m testing

So far that’s my equipment list.  I have a few other items I will likely be adding as the days go on, and may be able to scrounge up some additional interesting scenarios.

I will be writing here about the progress I’ve made with each device and test scenario, and discussing a bit about each device and the things I’ve found to be interesting.  Off we go!

We have a small amount of 24 port 1GB copper cards – the majority of the cards are the RX-Bi48T which is a 48 port MRJ-21 connector style card.  It requires a lot of additional cabling and breakout boxes, but it’s worth it to get more out of your switch.

The Foundry architecture for the 1GB 24 port cards only allows 20 of the 24 ports to operate at line rate, but we’ve planned our architecture not to be a problem.  The 48 port cards only allow 40 of the 48 ports.  Again, we’ve planned around this limitation.

In addition to the cards above, we  use the 4 port 10GB card for our larger devices, and as interconnects between the two RX’es, and many smaller switches such as the Force10 S50 and Foundry FESX-448.

We are moving away from Foundry equipment, however, due to a lot of failures over the last year and the need for a more port-dense 10GB switch for our larger products.  The Force10 C300 is a very attractive option – it’s significantly less expensive that the RX, and many of the other switches out there, and performs better in many cases – line rate in the case of the C300.  We have several already in-house and have been very impressed.

In all cases, the moment we get a new switch, we do many L2/L3 tests with various IXIA and Spirent devices to make sure we understand the limitations.  In the last few years, we haven’t found any issues with the new vendors and switches we’ve come across – at least none of the serious line rate issues we found in the past, including with the RX-series chassis.

Looking forward to spending more time with Force10 products in the near future.