Seamless Transitions...

I'd add that the Intel Cluster Ready architecture applies in the cloud as well.  In fact, using cloud images based on Intel Cluster Ready architecture allows more seamless transitions between clusters running in the cloud to physical clusters running locally.  As the video points out, cloud provides an entry point into using HPC or even a try-before-buy approach to HPC clusters.  Cloud bursting benefits as well, since the application that runs on the architecture locally would see the same architecture in the cloud.

The common architecture enables the applications to better understand and execute on top of the computing environment without being tied to the exact details of the solution.  In other words, you can get better mileage with a common architecture in both the cloud and local clusters.

Ever since the shift to multi-core processors began, I have always had a nagging question - Does more cores mean less nodes? I have wrestled with this question and finally realized that there is no simple answer. I should preface this post by stating that I am talking about HPC and not the general market place where multi-core and virtualization are all the rage.

To understand why I ask this, consider that back when Linux clustering began, the single core Pentium Pro and Dec Alpha where the two processors of choice. Large clusters were maybe 64 or 128 nodes (often tower cases), which translated into 64 or 128 cores. Today you can easily pack 128 cores into 16 nodes - not even a full rack chassis. Given this trend, is the node count for clusters getting smaller?

From my anecdotal evidence, I seem to notice several trends. First, the HPC market, after a downturn, seems to be growing again and is projected to reach $11.7 billion by 2012 (IDC). This revenue is up from a $9.6 billion figure for 2008. Thus, the hunger for nodes is increasing and not deceasing, which begs a further question. Are node counts increasing or are more people buying clusters? (i.e. instead of a few people buying larger clusters, are there a lot of people buying smaller manageable clusters.) For the marketing types out there, maybe you know the answer. If not, that is a good question to ask. Leave a comment and give us a clue.

Second, from my experience there is plenty of HPC work to go around. Whether nodes are in a large data center cluster or in a small local blade system, it seems the cores are busy. Perhaps we are seeing the rise of the Closet Cluster?

The use of virtualization and multi-core processors has made cloud computing an option for many users. The ability to buy cloud time as you need it and not purchase hardware is certainly attractive from a financial standpoint. The concept is not new and has its roots in time shared mainframes and grid computing. One might assume the the vast amount of computing resources in clouds may make them ideal candidates for HPC clustering. Unfortunately, it is not as simple as collecting cores.

One of the issues facing clouds is I/O. Basically, I/O is often not predictable or repeatable. From a storage standpoint read and write times can be fast, but not always fast. In terms of messages between servers, most clouds do not support high performance interconnects and similarly make no guarantees as to latency or bandwidth consistency. While grids paid attention to certain HPC performance guarantees in terms of I/O, clouds, in order to offer ease of use, have declined such guarantees. Unless a cloud has been specifically designed for HPC, the user cannot expect consistent and/or high performance. There are two papers which discuss this very idea. The first paper looks at Benchmarking Amazon EC2 for High-performance Scientific Computing and the second paper asks, Can Cloud Computing Reach The TOP500?. Both papers conclude that the cloud is not mature enough for HPC applications.

The limitations of the cloud become more apparent when one looks a little deeper at HPC applications. First, many applications rely on user space communication (i.e. high performance MPI programs transfer data directly from one node to another without using kernel services.) Such a close to the wire operation runs counter to the virtualization model. Secondly, as reported in the first paper (above), the performance of OpenMP applications was reduced by 7-21% when running in the EC2 cloud.

Recently Penguin Computing began offering POD (Penguin on Demand) for HPC cloud computing. The POD cloud offers both Ethernet and InfiniBand connections between nodes thus providing a dedicated high performance computing environment. This service can be considered a specialized HPC cloud.

There are some other other important issues to consider with cloud computing -- security and reliability. When data leaves your domain over the Internet it is virtually impossible to guarantee 100% security. If your organization can live with this situation, using the cloud may be an option. If on the other hand, you need to keep a tight reign on your data, then you may not want to be injecting it into the cloud. The other issue is reliability. If your day to day operations are based on using a cloud, then a contingency plan is a must. Interruptions in Internet traffic due to congestion or hardware failures can be common in some areas. In addition, the cloud provider may have issues (even go out of business) and thus not meet the service requirements.

I believe the cloud is an interesting model, but it is not a real solution for HPC (in its current form). My issue with clouds is that they are often categorized as "grid like" and then are somehow (incorrectly) considered "HPC like." Cloud offers utility computing like grid promised, but has pushed the application layer further away from the hardware. HPC practitioners spend a lot of time making sure the application is as close to the hardware as possible. At this point in time, HPC in the cloud is more of a curiosity than a solution. When examining HPC benchmarks it becomes clear that clouds are not the best means to provide HPC cycles. Whether efforts like POD can meet the HPC users needs in the cloud is still unknown.

To be fair, there are some HPC applications that lend themselves to clouds quite well. (i.e. those that do not require predictable I/O) Folding@home and Seti@home are two good examples. These applications could easily run in a cloud (in a sense they do run in the Internet cloud). Keep in mind they have been designed to work in a robust distributed fashion and are not virtualized. Clouds can be enticing and even enabling for some applications, but remember a collection of servers (in the cloud or in a rack) does not a cluster make.

Many of the HPC people I know are what you would call rugged individualists. The have been around since the beginning and were responsible for moving the market/community along when commodity HPC was less than fashionable. This group consists mostly of developers, implementers, and administrators. Many of these people developed, by way of discussion on the Beowulf Mailing List, the best practices used today. The Beowulf Mailing list is a true resource if there ever was one. A newbie can ask a question and get polite (and lengthy) answers by list members. The list holds a large amount of open community knowledge because all the HPC plumbing is open source. Unhindered discussions can take place at any level between any number of people.

There is also the false notion that open source software is "free as in beer." This idea is not quite true because software unlike toasters have a usage cost. Once you install, configure, and study any software you have already made an "investment" in the package. Continued use furthers this investment. The size of the investment is up to you. And, because the software is open, in theory you can fix any problem. Thus, you have the choice to decide how much time you can invest in a particular software package before it becomes "expensive" to you. At some point, the cost (or dilution) of your time may come into play. Spending weeks fine tuning a single application at the expense of other responsibilities is probably not going to play out very well.

In terms of support responsibilities, many clusters have minimal issues once they are configured correctly. There are, of course, hardware failures, but in general, once everything is booted, things often work quite well. There are two areas that need attention, however. The first is software updates. Updates are needed for several reasons including security, bug fix, or feature updates. These types of updates are usually easy to manage unless they have dependencies, which means there may be a whole raft of packages that need updating. If you don't get the dependencies right, then there can be problems with the entire cluster.

The other issue is local integration. This is what I consider the "last mile problem" for clusters. Very often local file system issues need to be worked out and managed in addition to creating job submission policies. There is usually some end-user assistance needed as well as questions on how to compile and submit jobs to the queue. Of course, "Why is my job sitting in the queue?" is probably the question that gets asked the most.

If your job includes time for installing, integrating, and updating software and you happen to be one of those rugged individualist HPC people, then you probably have no interest in professional support. If on the other hand, you are new to clustering (or Linux) and already have many responsibilities, then you may want to consider using professional support services. As with all open source software, the choice is yours. In terms of commercial software, or commercial support of open software, there are many options. In any case, purchasing support for business critical applications is always a good idea. As is the use of a Intel Cluster Ready (ICR) solution. By adhering to the ICR specification support is much easier -- for you and/or a vendor. That is, a reference platform allows you and your vendors to work from the same page. Without a common framework, support vendors and others in your organization may have to decipher/debug how you configured the cluster.

In conclusion, cluster support can be commercial or it can be institutional. In either case, there is a cost. If you do it on your own, it will cost time and if you hire a consultant or company, it will cost money. To supplement either effort, there is a large amount of information on the web that can be useful when identifying and solving problems. Support is an important part of any successful HPC cluster,  just ask the old-timers. They figured it out, so you don't have to.

I think I do not have to state that setting up an HPC cluster sometimes is an adventure of compiling and installing different program versions and libraries. Nevertheless, when having all up and running, the Intel Cluster Checker is a really nice tool to verify that all services needed for your HPC cluster are set up correctly and you haven't forgotten one of those tiny details during the installation of your cluster.

My last experience with the Intel Cluster Checker and the newly installed HPC cluster were a bit different from the ones before. The first steps ran smoothly and everything seemed to work fine, with me having learned from my previous first Intel Cluster Checker experience. The tests were successful, I altered the benchmark thresholds and all tests ended with a "passed". But in the end, when checking the dmidecode output I got a "failed" on this test. First I thought this was due to some BIOS specific mismatches, which you can exclude. But when I had a closer look on the output file of the Intel Cluster Checker I saw that there seemed to be installed different RAM modules on the compute nodes.  Hmm...the sizes of the memories looked fine, all were 2GB and I double checked the product numbers of the DIMMs that the Intel Cluster Checker reported. Here we got two different types of memories installed. I was astonished on the one hand as I would not have checked this without the Intel Cluster Checker and on the other hand I hardly could believe that there were two different types of memory DIMMs installed. Well, the servers I had installed were Intel Nehalem based and I remembered the days when AMD started with the CPU built-in memory controller and the problems with memories that arose during those times...

I took a closer look at the compute nodes and opened the chassis of  those which were affected. And indeed, there were different types of memory modules built in, but they nearly looked the same and also had the same product number written down on them. After some investigation with our purchasing department I found out that those memory types have been mixed up. Unfortunately it was not possible to pin down the real cause for this confusion.

The DIMMs were swapped to the correct ones in the end and I ran the Intel Cluster Checker again. This time all tests were passed and the results were sent to Intel to verify the Cluster Ready Certificate, which now proudly resides beside the other ICRs we have scored.

The Solid State Drive (SSD) has had a break-out year. Unlike traditional mechanical hard disk drives that use spinning platters with movable read/write heads, SSDs have no moving parts. The SSDs is made entirely out of a special type of flash memory -- the same kind of NAND flash memory found in thumb-drives and memory sticks. Overall, SSDs are faster, quieter, more energy efficient, but less dense than the traditional spinning platter drive.

The staple of the storage industry has been the mechanical drives, where I/O rates are limited to the mechanical properties of a drive. Unlike other semiconductor trends, the sustained write rate has barely doubled from 50 to 90 MB/second over the past 8-10 years. All this is about to change as the use of flash memory will allow stroage to take advantage of a semiconductor growth curve similar to that of processors and memory. (i.e. mechanical drives are limited by the physical motion of spinning disks).

Perhaps the most important feature offered by SSDs is the read and write performance. The IOPS (I/Os per second) rate for an SSD is usually two to five times that of a traditional mechanical hard drive. When reading, performance is mostly constant because the seek time is virtually instantaneous and does not depend on the physical location of the data on a platter. As a result, file fragmentation has almost no impact on read performance. In addition, because there are no moving parts, SSDs use as little as one-fifth the power of a mechanical drive. Another interesting feature is the SSD failure mode. Most SSD failures tend to happen when writing. In contrast, mechanical drives tend to have most failures when reading. Thus, once data is written, it is more likely it can be read from a failed SSD.

SSDs do suffer from “degradation” over time that results in reduced performance and limited lifetimes (i.e. there are a limited amount of read/write cycles avaliable for NAND memory). Vendors have taken this into account and include wear leveling algorithms in SSDs that spread write access evenly over the entire device.

If you are interested in the exploring SSDs, there are some key points to consider. First, SSDs are not the best solution in every case. Currently, their capacity is much less than that of traditional mechanical drives and as such may not be suitable for some of the large HPC data sets. In addition, the cost is per MB is higher and are more susceptible to data loss from energy and power surges.

Second, the very fast read times offered by SSDs has made them a good candidate for improving OLTP (Online Transaction Processing) systems where frequently read tables and indexes can be accelerated. Check both the read and write IOPS, as there is usually a big difference between these values. Typically, the read speed is 10 times the write speed resulting in asymmetric performance. In terms of clusters, using SSDs for NFS mounts or read-only data may be helpful.

Third, because SSDs are new, questions still remain about how much of that speed they can deliver for the long haul (due to degradation). Typically, an SSD will show an initial decrease in performance and then level off. Even with a performance drop over time, SSD drives are almost always faster than traditional hard drives. The JEDEC standards organization plans to publish two standards by the end of this year for SSD endurance metrics.

Finally, pay particular attention to “write endurance,” this number should pertain to random writes. For instance, an Intel® X25-E Extreme 64 GB SATA Solid-State Drive is rated for 2 petabytes of lifetime random writes.

In terms of software, one difficulty facing the industry are the legacy assumptions built into file systems. These assumptions will need to be challenged in order to take advantage of SSD technology. For instance user applications and file systems will need to account for the asymmetric read/write performance of SSDs. Many computer applications rely on synchronous patterns of read/write operations, wherein a given write or update must be completed and the write confirmed before additional application read requests can be issued. With SSDs this process may need to be reconsidered.

There is no doubt that SSDs are the future of storage. Indeed, SSDs are even changing the way we compute. For example, CAE (Computer Aided Engineering) applications can use the speed advantage of SSDs in their out-of-core algorithms. The power of semiconductor manufacturing technology combined the speed of NAND flash memory are about to make the storage market stand still!

The thirst for better file system technology is not new to the Unix/Linux world. There is a rich history of trying to optimize the balance between the storage system and the process improvements in a demanding user environment. The efforts to build a better file system are quite numerous and are built on the efforts of many people. For example Kirk McKusicks’ original Berkeley Fast File System improved on the original V7 release. Steven Tweedie’s ext3 took ideas from the database logs and Margo Seltzer’s LFS and added them to Linux’s implementation of UFS -- Ted Tsao’s ext2. In the mean time, DEC released Megasafe, SGI released XFS and Sun released ZFS all to the wild. And now Oracle has developed Btrfs for Linux.

So why do we as users care? There better be a good reason to change a file system because a new file system usually means converting to and trusting a new format with your data. Thus, any new format or change must provide a compelling reason or solve a big problem. Otherwise what is "good enough and works" is often better than that which is "new and fancy."

If you follow the details of file systems development, then this quick update may not be of interest to you. For the rest of us, who just choose whatever file system the installer offers, you may want to read further because changes are afoot.

If you are like me, you probably are running Linux with the ext3 file system. There is nothing wrong with ext3 as it is stable, robust, and a standard Linux file systems. And, one other thing, it is old. Even if you are running the newer ext4, you are still running a 30-year old file format that is more than a little short on features.

There are those that believe ext4 is going to be the end of the line and a switch over to Btrfs is very likely. Btrfs (pronounced "butter-F-S") is being developed by Chris Mason at Oracle. It is an open source project that has recently been added to the Linux kernel (as of 2.6.29) as experimental code.

Btrfs is based on several new ideas including b-trees (binary trees, which is where the btr comes from in Btrfs) and "copy-on-write" or COW. While, I won't go into the details, b-trees and COW allow for some new features that would be difficult in the ext* line of file systems. (If you want to learn more about the technical details of Btrfs, see A short history of btrfs on lwn.net.) Some of the new features include file-system snapshots, check-summing, online defragmentation, compression, extents, resizing, and more. In particular, Btrfs allows one thing that has been difficult to achieve in the past -- optimizing both access time and disk space.

The fact that Oracle sponsors Btrfs has lead to some concern. Recently, Oracle purchased Sun Microsystems which has been developing the ZFS file system for many years. ZFS is similar to Btrfs (it uses COW) and provides many of the same features, but it is very different in its internal implementation. ZFS will also "run" under Linux using Fuse. Mason and other have assured the community that Btrfs is important to Oracle and they will continue development. In addition, the open source nature of Btrfs ensures that it cannot be "taken away" now or in the future.

There is plenty more to consider and I suggest reading Linux Don't Need No Stinkin' ZFS: BTRFS Intro & Benchmarks by my friend Jeff Layton. The consensus seems to be that Btrfs is destined to become the default Linux file systems within two years. ZFS on the other hand must overcome some licensing issues before it can even make it into the Linux kernel for testing. Your next Linux install may offer a new and better (or "btr") file system than in the past.

Welcome to the world of industrial HPC. Today, we will consider a small part of those large trucks that spend most of their day crisscrossing our highways. At highway speeds, anything that moves through the air has an aerodynamic cost. Pushing a big box takes more energy than a round ball, which is why better aerodynamics means less energy and lower costs. Almost all trucks have some kind of mudflaps to prevent road dirt and debris from hitting the truck. Midsized truck maker Kenworth wondered how much it costs to move those mudflaps though the air. To answer the question, they turned to HPC where they were able to determine  trimming and tapering the mudflaps can cut about $400 from a typical trucks annual gas bill. This amount adds up quickly when you have a fleet of 1000 trucks. And, based on the mudflap success, Kenworth has started using HPC to help increase the efficiency of their truck designs, thus saving customers even more money. You can read more about these efforts in Heavy-duty Computing from Fortune Magazine.

If mudflaps don't pique your interest, but saving money with HPC does, then you may be interested learn more about The Council on Competitiveness (CoC). Who or what is the CoC? They are a group of corporate CEOs, university presidents, and labor leaders committed to enhanced U.S. competitiveness in the global economy. One of their main focus areas is HPC. That is correct. Not only can HPC dock bio-molecules, design jets, and find oil, it can also help many companies save money and be more competitive. The CoC is a nonpartisan, nongovernmental organization based in Washington, D.C. The Council shapes the debate on competitiveness by bringing together business, labor, academic and government leaders to evaluate economic challenges and opportunities. The High Performance Computing Initiative is intended to stimulate and facilitate wider usage of HPC across the private sector to propel productivity, innovation, and competitiveness. Click the link to find out more how other companies have cashed in on HPC. Yours could be next.

First, always use the required and supplied "provisioning system" tools to update a cluster. It may be relatively easy to update all the nodes in a cluster using a couple RPMs and something like pdsh to script the installation of the update - but don't be tempted.  This manual or brute-force method bypasses the software that manages the image on each server. Provisioning systems may reimage nodes after a crash or maybe a node is replaced or added to the cluster. If software updates are applied manually (outside of the provisioning system) then the reimaged node will be inconsistent with the rest of the system. An admin would need to remember all the manual changes and apply them again. It's much better to let the provisioning software worry about that. That is one the reasons ICR required a provisioning system!

Once updates are applied, it's a good idea to verify the cluster is working as it did before the update was installed and it remains compliant with the Intel® Cluster Ready architecture. Many if not most software updates will behave well, but verification helps ensure an update didn't alter or remove a key system component that may lead to application failures. Intel® Cluster Checker provides an easy way to check the compliance after an update. By using the command-line --compliance option, the tool will verify the interface defined by the architecture still exists as before. It's an easy way to check that the update hasn't had any ill effect on the architecture interface used by ICR applications.

Finally, there may be needed updates to the Intel Cluster Checker configuration files to reflect the updated software. For example, if a newer version of the Intel C compiler is installed, the Intel Cluster Checker configuration file should be updated to utilize the newer version. Running the tool would then verify the new installation is functioning on all nodes. It's also valuable to update the list of packages that are expected on each node. The packages test verifies the RPMs installed on each node matches a predetermined list. Using the --packages command-line option will create new package lists based on the current installation (use it after all updates are complete). Save the original list file and set the configuration file to use the updated list. For more information on using the tool, see the Intel Cluster Checker Users Guide.