One of the biggest problems with increasing the capacity of a software-based messaging platform by adding brokers is the communication that inevitably needs to happen between brokers. When clients or applications connected to one broker need to communicate with an application that’s connected to another broker, you end up sacrificing some of each broker’s capacity to that inter-broker traffic, leaving less available for client communications. In fact, this problem increases exponentially as you add more brokers, so the return you get by adding each new broker is less and less.

The next challenge that IT personnel face when horizontally scaling software-based message brokers is that of capacity planning. With discrete message brokers serving unique sets of clients and instances of applications, each broker must meet the demands placed on it by clients and applications alike with its own available capacity. The complexity of assuring adequate system performance makes load balancing across message brokers much more difficult than load balancing across application instances, and in fact doing so amplifies the inter-broker communications problem described above. This means a given broker can be overwhelmed with traffic (focused overload) while another broker sits idly by with stranded capacity going to waste.

Software-based message brokers can only handle a few thousand messages a second, so capacity problems come up quickly. Solace gives you all the capacity you need (over 200,000 messages per second per appliance) to meet the need of a few high-volume applications, or lots of low volume applications, all in one compact device with unified administration.

In this 10-minute video, I explain why horizontally scaling message brokers doesn’t work as well as horizontally scaling other assets like applications, and why our high-capacity appliance provides you with a better solution.

This was all it took to customize the included JMS Endpoint to work with a Solace appliance, which we make available in the public Internet as a messaging service:

With the Solace JMS endpoint configured, the rest was “one handed programming.” Dragging and dropping icons and connecting endpoints from the pallete was quick and easy, and before I knew it  I was running a REST API for publishing messages to Solace topics and queues.

After that, I wanted to explore further and make a more “cloudy” demo so I picked up on the available Cloud Connectors which include integration components for Amazon, Salesforce.com, Facebook, and Google.

A quick drag and drop and I had a streaming Twitter application that consumes all @SolaceDemo tweets, transforms them into both XML and JSON formatted messages and publishes them to a topic name hierarchy that allows the subscriber to pick which format of messages they want to receive based on topic wildcard subscriptions.

Adding topic subscriptions to attract these messages to queues for persistence is easy using the SolAdmin GUI so I created a queue for xml tweets (xmlQ) and another for the JSON versions (jsonQ) and configured the appropriate topic subscriptions for each.

All that’s left to do is to wire up an analytics engine to pore over the tweet-stream and figure out by sentiment analysis what people are thinking and saying about Solace messaging.  A quick search on MuleSoft’s APIhub shows there are at least 42 ways to do that too!

If you’re new to SolAdmin, it’s a management application that can be used to access all management features of Solace appliances. It is designed for both advanced administrators already familiar with Solace and those just getting started.

This video demonstrates the monitoring of subscriber applications receiving messages from a Solace messaging appliance. My goal was to show how easy it is to take a typical problem of an application that is not correctly receiving messages and to show both how the appliance provides detailed statistics and debug information that help to diagnose the problem and how SolAdmin makes it very easy to access these details. Here’s a brief summary of what I covered:

In general applications having issues receiving messages can be divided into these categories:

1. The application’s network connectivity to the message broker has issues.
2. The application is simply consuming messages too slowly. I.e. the application is back pressuring the message broker and needs to be streamlined.
3. The application may be hung.
4. The message broker may be overloaded.

Initially when debugging this issue it might not be immediately obvious which of these reasons is being seen and furthermore it is often different teams that are responsible to fix the issue. So being able to determine the real reason the application is misbehaving is very useful.

To demonstrate this, the video starts with a look at a typical publish subscribe message flow with a single publisher sending messages to the Solace appliance. The messages are then fanned out to several subscribing applications. SolAdmin is used to explore the details of the receiving clients and plot the real time message flow towards these clients and look at the TCP/IP details of each client.

Then a client is connected over a simulated lossy wide area network (WAN) and I drill down into the TCP signature of this client and how to use SolAdmin to observe and diagnose the network issue. Finally a simulated slow subscriber is connected and the video goes over the client statistics available to differentiate slow subscribers vs hung applications.

More practically, imagine using the onboard GPS to define geocoded zones that dynamically adjust the height of the Active Air Suspension to avoid scraping speed bumps and ditches that would otherwise fail to clear the front spoiler or the underlying battery pack.

I would love to have the Android app Torque Pro running on the console and streaming real-time telemetry from the onboard computer to a cloud hosted application for post-race — sorry post commute — analytics. I’ll let you know how that goes in a later post.

I’m still waiting for Tesla to release the native SDK for this platform, but in the meantime there’s a lot you can do via the embedded browser which I’ve discovered does support HTML5 WebSockets. For fun I tried a few of the Solace Web Messaging demos and as you can see in this video they work fine.

What this says to me is that the WebSocket  protocol is essentially becoming ubiquitous and you can add cars to the ever expanding list of the Internet of Things.

I recently presented at QCon in San Francisco on the topic of “Distributed Data Fabrics and Hardware WAN Optimization“. If you have never been to a QCon event, note that it is a geekfest and the attendees have zero tolerance for marketing hype. Developers and architects come to QCon to learn, share, and socialize amongst their peers, not to listen to sales pitches. I like the vibe at QCon and think it’s very much in line with the kind of content that works on a tech blog, rather than a corporate blog, so I’d like to expand the audience of my thoughts here.

My talk was about the Long Fat Network (LFN, or sometimes “elephant”) problem and how to distinguish between different ways of solving this problem at the network/transport level (layer 3) versus the application/messaging level (layer 7).

The basic problem of WAN optimization is that a WAN is not just a slow LAN. TCP/IP based apps that run great on a LAN often suck wind on the WAN and the blame often incorrectly goes to the lack of bandwidth. The expensive way to find out this is wrong is to pay for a 10 Gigabit WAN link and find out that throughput is actually a function of the latency, and loss, in addition to bandwidth.

Check out one of the many throughput calculators and you will quickly see that a 10 Gigabit WAN link with 100ms round trip latency and 0.5% packet loss yields just 1.65 Mbps of maximum effective throughput per TCP flow. This is all an artifact of the TCP sliding window and the fact that over long distances the speed of light becomes a significant limit. The further complication is that unlike a LAN, packets get dropped and need to be retransmitted. Even small loss rates stall the advancement of the the TCP windows and kill your throughput.

Why do you care? Databases and data grids are increasingly becoming replicated and distributed across multiple datacenters for redundancy or active/active partitioning and localization. The trend to “big data” is also expanding to sources that span remote datacenters, and even mobile apps and sensors, which all adds more WAN into the mix.

Enter Riverbed and Silverpeak, and a host of other WANop hardware and software vendors that apply compression, forward error correction, and other techniques to speed up the flow of data across the WAN. These solutions are typically not application aware. They operate on TCP packets without understanding what they contain. That can still yield impressive results but it really depends on the use case. Replicating encrypted data or pre-compresses binary data will reduce or eliminate the performance gains that a layer 3 WAN optimization tool can provide.

Some WANop solutions try to go further up the stack and optimize for popular application protocols like NFS or Microsoft’s CIFS file sharing protocols. However the vast majority of application specific WAN traffic still goes unoptimized. This is the target for distributed messaging solutions such as those provided by Solace.

Consider the following example of distributed database replication using a CDC tool like GoldenGate or DataMirror. In the simplest case you might have just two datacenters and want to replicate all updates from one site to the other for disaster recovery. This is a great use case for traditional WANop solutions because the data is point to point and all of the updates at one site go to the other. There is no value in looking at the contents of the updates or filtering based on the values of the data because everything needs to flow across a single WAN link between two sites.

Now consider a more distributed data grid, where some data (reference data) needs to flow between 3 or more global sites and other local data needs to be selectively replicated, or in some cases not replicated at all. This is a great fit for “layer 7″ or application aware WAN Optimization because it is necessary to look into the contents of the data and selectively filter and route the updates.

WAN optimization and application-aware WAN Optimization both add value and can even be used together, and you’ll be most successful when you use the right tool for each job. Network optimization is a broad brush that can speed up all your network traffic, while application-specific WAN optimization accelerate the traffic that you care most about to an even higher level.

First it’s important to introduce what constitutes large scale in the log processing space and why there is a need for solutions like Kafka. These are some log processing metrics quoted in the paper.

-          LinkedIn: generates 100’s of GB of new data in the form of 1 Billion messages per day

-          China Mobile: generates 5-8 TB of Call Detail Records (CDRs)  per day

-          Facebook: generates 6TB of user activity data per day

These huge volumes of data are generated across thousands of source systems distributed in multiple data centers, so it’s no easy feat to reliably capture the torrent of data, consolidate it and route it to the various analytics engines and back-end applications that feed on log data as the raw material for search, fraud detection, targeted advertising, and social networking.

Sounds like a good job for a message broker, either in the form of open source like RabbitMQ and Apache ActiveMQ, or a commercial product like IBM WebSphere MQ or TIBCO EMS.

The four main reasons outlined in the paper for why traditional enterprise messaging offering are not a fit for LinkedIn’s use case are:

1. Feature bloat and the overkill of unnecessarily strong message delivery guarantees
2. Not focused on throughput
3. Weak clustering/partitioning support
4. Poor message spool handling under heavy load

This is when I started to get really intrigued because the items in the list above are the very same ones that I use to qualify a fit for a Solace Messaging Appliance. Log processing, and the similar application in sensor networks, are becoming common and repeatable use cases.

Let’s look quickly at how the four needs are addressed in a hardware implementation:

1. Bloat – Reliable message delivery with batching – Solace has two simple tiers of delivery quality (best effort, and guaranteed) with configurable sliding windows, ACKs, and vectored send (i.e. batching) for high throughput in a LAN or WAN environment (where the RTT will kill most JMS implementations). Dedicated PCI cards handle the different qualities of service so that traffic from one does not impact the performance of the other. If guaranteed delivery is not required, the extra card is not installed and there is no “feature bloat” or unnecessary code paths in the broker.
2. Throughput – Hardware offload is perfect for maximizing throughput. Offloading the Linux kernel from interrupts for disk, network, memory, context switching, is how the Solace 3260 achieves 10/40 Gbps wirespeed throughput on a single box.
3. Clustering – Interconnected nodes with no master hub and with dynamic but efficient message routing and balancing protocols are built in. FPGAs and Network Processors (like those found in F5 Load Balancers and Cisco Routers) can be reprogrammed to do high speed inter-node Layer 7 routing between the nodes in a network of message brokers without impacting the performance of each node to handle client producers or consumers. By dedicating the balancing to bare metal silicon, it is possible to do continuous balancing at higher speed than periodic balancing done in software.
4. Message spooling -  Solace appliances accelerate the persistence of  messages in a novel tiered non-volatile message spool without suffering from the hit of slow spinning hard disk I/O for every message. Flash and SSDs don’t help if you treat them like hard disks and write every message because the constant writing creates longevity problems very quickly. Solace uses RAM backed by Super-Capacitors and RDMA memory replication within clustered nodes to allow huge queues to build up and then efficiently dequeue at a later time. This enables the “snake eating a pig” use cases where messages can bulge temporarily in the message spool until downstream processing kicks in to digest the data. This capability makes for an excellent buffer or shock absorber for log data streams flowing into batch oriented big data frameworks like Hadoop.

So now let’s look at the actual performance. The benchmark testing in the Kafka paper matches with the proof of concept testing I have seen with ActiveMQ and RabbitMQ. Basically anything with durable, persistent, clustered operation puts a heavy load on general purpose Linux servers and yields no more than 10,000 – 20,000 msg/sec in or out. The Kafka numbers are impressive and proof that the optimizations and tradeoffs made are effective.

Kafka Perf test

-          2 Linux servers with 8 cores, 1GigE, 16GB RAM

-          200 bytes message size

-          Tested both batches of 1 or 50 messages per batch

-          Pub rate of 50,000 msg/sec (batches of 1)

-          Pub rate of 400,000 msg/sec (batches of 50)

• Said another way this is 8,000 batches/sec @ 10 KB batch size (50*200 bytes)
• Said another way this is ~0.7 Gbps wire speed throughput

-          Sub rate of 22,000 msg/sec

Solace Perf test

-          1 Solace 3260 Appliance w/ Fedora Core Linux control plane with 8 cores, 1GigE, 16GB RAM

• Solace Topic Routing PCI Card
• Solace Assured Delivery PCI Card
• Solace 10 GigE Network Acceleration PCI Card

-          200 bytes message size

-          Pub rate of 206,400 msg/sec (batches of 1)

-          Pub rate of 2,670,000 msg/sec (batches of 50)

• Said another way this is 53,400 batches/sec @ 10KB batch size (50*200 bytes)
• Said another way this is ~4.4 Gbps wire speed throughput

-          Sub rate of 206,400 msg/sec

So by most metrics the Solace 3260 appliance is 6.7 – 9.4 times higher throughput than Kafka even with the higher quality of service provided with the once-and-only-once-in-order “assured delivery”.  Lessening the restriction on guaranteed delivery some more would yield a further 25 times increase in throughput (to ~11 Million msg/sec, in+out, batches of 1). While it is true that the Solace benchmark uses 10 Gbps Ethernet rather than the 1 Gbps Ethernet used in the Kafka test, the network bandwidth was not the bottleneck in either test so the performance difference can be attributed to the acceleration provided by the Solace Appliance and associated Solace PCI cards offloading the Linux kernel.

So to summarize, the architecture of Kafka does an excellent job of optimizing generic message oriented middleware into a lean and powerful log processing framework. The lessons learned and approaches used are equally applicable in firmware, rather than software, and can yield yet another leap in overall throughput. Features such as compression, replication, and stream processing can be efficiently added inline in firmware. The benefit of the modular approach to a hardware based implementation makes it possible to add back in extra features and functions and enable a common shared infrastructure for generic messaging as well as log processing.

The Raspberry Pi Model B is much more than the Arduino microcontroller devices popular with the Maker Faire crowd. It’s a complete computer with Ethernet, USB Ports, SD Card slot, 256MB RAM, 1080p HDMI video, all running a standard Debian “squeeze” Linux distribution running on it’s ARM-based SoC. Add 4 AA batteries, a WiFi dongle, and some custom software and you have a very capable wireless sensor node in a box (box not included). The waitlist for these tiny embeddable computers is growing and an ecosystem of complimentary add-ons has sprung to life overnight.  Everything from cases, to cameras, to the inevitable Arduino to Rasperry Pi bridge. With the one-two punch of the sub-$50 price point of the Pi and the emerging $200 price point of the full blown touch screen tablets (Nexus 7, Kindle Fire, et al.) we should finally be at the tipping point where we will get an explosion of devices connected to the Internet of Things.

Real-time message passing between embedded devices seems like a perfect fit for an M2M messaging appliance like the Solace 3260. Devices like the Pi enable customer kiosks, point of sale terminals, medical monitors, remote sensors, and remote control networks via intelligent customer premise equipment. For this reason some of us have been experimenting with running the Solace messaging APIs on this incredible little machine and I am happy to report that it works like a charm. Now we just have to work our way through that waitlist a bit longer so we can test 100,000 of these puppies connected simultaneously.

1 ) Federate: Get them talking to one another so that they benefit from the network effect that returns greater value as more endpoints are inter-connected

2) Accelerate: Get rid of the road bumps and bottlenecks slowing down the real-time flow of information. Modern data movement is much lower latency, globally distributed, with active/active  replication into in-memory datagrids and distributed cache.

3) Rejuvenate: Remodel a few key flagship implementations to take advantage of the new world of streaming big data, real-time analytics, mobile sensors, M2M, etc.

Easier said than done you’re saying, and of course you are right. However, it’s not the technology that is holding us back anymore. The world of the message broker has never been more open and interoperable. AMQP, Websockets, JMS, REST, Web Services, XML, JSON, Virtualization and Cloud Computing all combine to make the task of building a multi-tenent multi-protocol federated message bus a reality.

The following is a 12 min video demonstration of a set of off the shelf components easily tied together to show seamless real-time communications between IBM WebSphere MQ, a RabbitMQ AMQP 0-9-1 Broker, any Java Messaging Service (JMS) server, and a Web Streaming application tunneling through standard HTTP. A Solace Systems 3260 Appliance and Layer 7 Technologies SecureSpan Gateway are the key components integrating the other systems together.

The server-side of the demo uses an early access version of the Solace Node.js module. The client side is written with the generally available JavaScript Solace Web Streaming client libraries running on an Android  mobile phone. Communications between the client and server is all via HTTP Streaming through a  Solace Messaging Appliance.

If you want to see the text, the video is best viewed full screen in 720p HD quality.  I hope you enjoy these quick videos. Email us at info@solacesystems.com if you have ideas for future video demos or technical tutorials that you would like to see.

Enterprise Messaging in healthcare has been principally leveraged as a mechanism for information sharing either in the form of system synchronization or real-time information distribution.  From a patient’s point of view, information sharing is particularly important. For example, a patient who visited different hospitals and specialists within a system using messaging based synchronization gets the benefit of collated and correlated clinical, medical and social information, so diagnosis can be more accurate and overall care can be of much higher quality.  Additional benefits can include reduction or elimination of repetitive tests and procedures.  Collectively these benefits can have a dramatic affect in accelerated diagnosis and overall cost reduction.

There are many ways of architecting a national healthcare system to achieve information sharing. In the UK, the backbone of providing patient information and shared services at the national level is called the Spine [1] and there are many applications from Local Service Providers (LSPs) connected to the Spine. The Spine is a part of the NHS National Program for IT (NPfIT), which aims provide “a single, centrally-mandated electronic care record for patients and to connect 30,000 GP(s) to 300 hospitals, providing secure and audited access to these records by authorised health professionals” [2]. British Telecom is responsible for implementing the Spine, as well as the LSPs for the London cluster and South of England cluster.

To connect local applications to the Spine, the NHS IT authority – Connecting for Health (CfH), employed a loosely-coupled architecture based on messaging. LSP applications need to send different types of messages to the Spine requesting information and services. The Spine then returns information and services in the form of messages.  This project, spanning more than 10 years, has been claimed to be the largest civilian IT project in the world.

Message Handling Service

With such a big project and complex system in the Spine, where exactly in the architecture does a messaging appliance like Solace play a part?

Figure 1: MHS Interactions

To talk to the Spine, all applications need to do it via a Message Handling Service (MHS), of which its specification is defined by CfH. In fact, all senders and receivers of the Spine messages must implement an MHS. MHS is responsible to move messages between applications that request external services and applications that implement such services. For example, if an application from an LSP needs to find demographic information of a patient (such as name, address, date of birth, NHS number), it will need to send a message via the LSP’s MHS node to the Spine MHS node, and the Spine’s MHS node will route the message to its Person Demographic Service (PDS) to

process the request. The Spine will then return the patient demographic data to the LSP system via the Spine MHS. See Figure 1 for the MHS interactions, where LRS stands for Legitimate Relationship Service that is to make sure the requestor has the authorization to retrieve the patient data.

MHS is currently implemented as a software component that supports both HL7 and non-HL7 XML messages, and operates under either Web Service (synchronous) or ebXML (asynchronous, reliable) mode. HL7 messaging is very complex but hugely important for healthcare systems to talk to each other. It is used not only for conveying data but also invoking actions. However, message format is not what we should concern here. If Solace appliances are to be used to play a part in MHS, all HL7 or non-HL7 messages will be wrapped in message payloads. The question is, how much does the Solace SDK cover MHS functionalities?

Let’s look at the main tasks that an MHS is supposed to perform.

Figure 2: MHS Functional Components

• MHS Service Interface.  The abstract interface between the MHS and a system that is sending or receiving HL7 (via HL7 processor) or non-HL7 messages.
• Header processing. The creation and processingof (ebXML or web service) header elements.  This also includes the parsing of header elements, the interaction with contract properties, and the passing of elements to and from the message service interface.
• Message packaging.  All messages, HL7 or non-HL7, ebXML or web service, need to be wrapped in SOAP envelopes. This part is responsible for the building of the ebXML and web service envelopes, including the ebXML SOAP attachment structure.
• Routing.  Responsible making any routing and address mapping decisions required.
• Reliable messaging.  Responsible for the delivery and acknowledgement of reliable messages.  This component deals with persistence, retries, error notification, and acknowledgement of messages requiring reliable delivery.
• Transport Binding.  The abstract interface between the MHS and the various protocol stacks.
• The HL7 Processor is a logical component that may be external to MHS. It is responsible for marshalling and unmarshalling HL7 messages.

Figure 3: MHS with Solace SD

Based on what an MHS normally do, we can architect Solace into MHS, and partially replace the function of Routing, Reliable Messaging and Transport Binding.

Summary

There are still many details to be considered for the architecture to work efficiently and across other services that the Spine provides. For example,

1. How to utilise Solace’s topic routing strength to help the Spine to fan out to thousands of local applications and service requestors effortlessly?
2. The Spine is currently using SeeBeyond eGate with heavy customisation as the messaging gateway called TMS (Transaction Messaging Service). How efficient is TMS to receive thousands of guaranteed messages per second and fan out to the service providers on one hand and response to service requestors on the other?
3. Some Spine services such as PDS, LRS and PSIS (Personal Spine Information Service, delivers detailed patient clinical data), use HTTP but some others such SDS (Spine Directory Service, locating hospitals, doctors and nurses etc.) and SBS (Spine Security Broker) are just TCP/IP based protocol. How can Solace integrate with those services so Solace can provide a unified messaging platform for the Spine?

If the above questions are addressed, topic-based messaging platform will ease the development of MHS across thousands of healthcare systems, cut down the mounting cost of NHS IT program, and provide a solid scalable platform for better care for patients.

Let’s leave the discussion open and I will further elaborate this messaging solution in Part 2 of this blog. Watch this space.

References

[1] http://www.connectingforhealth.nhs.uk/systemsandservices/spine

[2] http://en.wikipedia.org/wiki/NHS_Connecting_for_Health

[3] http://www.hl7.org.uk/repository/mim.asp