Enterprise Service Bus (ESB) Magic

JMS Redelivery Configuration

2009-03-04T07:03:00.000-08:00

I recently had to help a client tune JMS redelivery on rollback. The application would post a message to a Queue with persistence, and a set of MDB's would pull these messages and tries to send them over to another point using Web Services, SMS, Email etc. However, if the sending fails, the requirement is to retry the operation by rolling back the read, so that the redelivery count could be used to discard messages on a threshold, or place them into a dead letter queue for manual action etc.

However, if a redelivery time is not specified, the messages keeps rollingback, and immidiately redelievred, which is of no use whatsoever! Though some JMS providers lets you easily configure this (e.g. WebLogic), I came across a JIRA for ActiveMQ, and a set of links that suggested incapability for JBoss.

It will also be impossible to write a MessageSelector, that is expected to pull only messages with say: (JMS_JBOSS_REDELIVERY_COUNT < 3) OR (JMSRedelivered IS FALSE)etc.

The correct trick, is to set the following properties, when placing the message on the Queue, and let the subscribers simply pull the messages out without any selectors etc..

message.setIntProperty("JMS_JBOSS_REDELIVERY_DELAY", 10000);
message.setIntProperty("JMS_JBOSS_REDELIVERY_LIMIT", 3);

The above correctly delays redeliveries to 10 seconds, with a maximum of 3 redeliveries, until the message is placed into the dead letter queue.

Logger for Axis2 now available too

2009-03-01T11:29:00.000-08:00

I have just uploaded the Logger for Apache Axis2 to the utilities here. See the sample client found here on how to integrate this with a typical Axis2 client. Basically you will need to copy the deployable MAR file into your Axis2 repository/modules directory. Configuration of the module will be via your axis2.xml in the form shown below:

...

<module ref="logger"/>
<moduleConfig name="logger">
<parameter name="rootPath">/tmp/x/y</parameter>
<parameter name="identifier">cfg1</parameter>
<parameter name="enabled">true</parameter>
<parameter name="enableJMX">true</parameter>
</moduleConfig>

...

Connecting via JMX/JConsole allows you to enable/diable the Logger on demand, and can easily be modified to report statistics, response time of the backend service etc.. Let me know how you like this, and what would make this even more helpful

Logging all messages passing through Axis

2009-02-25T12:47:00.000-08:00

Recently I came across a client who required that all web service calls should be easily logged to a specific path, and grouped by date, and possibly other information. The SOAP stack used was Apache Axis (not Axis2), and thus I wrote a handler that does this..

The source code is in Google [http://code.google.com/p/adroitlogic-utils/] and released under the Apache License v2, so that anyone could use it and modify it as required, and possibly enhance it and send back patches too. The binary for Axis 1.x can be downloaded from [http://adroitlogic-utils.googlecode.com/files/axis-logger-0.1-beta-1.zip]

It can be managed via JMX, and left disabled by default if required, to help troubleshooting problems as they arise. For more details see [http://code.google.com/p/adroitlogic-utils/wiki/AxisLogger] for configuration and usage

Basically, once installed and enabled, this handler would intercept your Axis client side web service calls and save them into a pre-configured directory ordered by date, and other information. This would help comply with requirements for many industries where each request made to external services and the responses received should be logged into files for archival - e.g. airline industry, credit card/payment transactions etc

I hope to include the Axis2 version of this same handler shortly..

Chaining messages with the Apache Synapse ESB

2009-02-22T22:17:00.000-08:00

I've just written a sample that illustrates message chaining with the Apache Synapse ESB here http://adroitlogic.org/knowledge-base-synapse/11-message-chaining-with-synapse.html. This same idea could be used in many different ways to achieve many scenarios, where web service composition etc, is required.

Ohnoh - Ohloh got it all wrong!

2008-11-01T12:12:00.000-07:00

Well its been almost 8 months since I first reported the inability of Ohloh to correctly analyze a codebase of a project.. In-fact, Ohloh knew about this and they stated "This affects a lot of projects, and I desperately want to get this fixed, but it will take some time" [1].. So I checked back in 4 months since there was no progress, and got the attention of Jason - the chief geek at Ohloh. But unfortunately "this problem simply doesn't rank as high as driving & monetizing ohloh adoption" [1].. and there were some "workarounds" which got my attention! However, after many trials by Jason and me, we couldn't get a historical repo created for Synapse, probably because "Apache Svn repo is one single GIANT repo" [1].. but then again.. this is an Apache project! where many open source projects that Ohloh tracks, lives and thrives.

I then questioned the reason for Ohloh to use the "--stop-on-copy" option.. to maybe try to fix the "real" problem, where it "really" occurs.. but no reply from Ohloh.. but some other guy got worked up at this point and said "Just waiting for an answer is enough - The Ohloh boys and girls will work on your issue once they have fixed more important things" [1].., which really got me laughing :)! - You've got to read this yourself here [1].. Then finally, having worked with many open source projects, I thought Ohloh code would/could also be open source, and maybe I could help them fix it!.. but after suggesting that maybe I could help, about a month has now passed, but still no replies from Ohloh.. so I have given up now!

Going back to cool Statsvn, we can see that the Apache Synapse project lived from September 2005 - for over three years! While .. Ohnoh!!.. Ohloh thinks its not even an year old and even shows a yellow triangular warning - suggesting that this could "indicate an immature or volatile project".. I am not exactly sure to which that seems to apply now..

[1] Thread of discussion on Ohloh - https://www.ohloh.net/forums/3491/topics/1326
[2] The correct code statistics for Apache Synapse can be found at [http://people.apache.org/~asankha/synapse/statsvn/]

Webinar on integration with the Open Source WSO2 ESB

2008-06-30T02:53:00.000-07:00

Tomorrow Paul, Ruwan and I will be taking users through some interesting integration scenarios based on some real customer deployments, during the "Example integration scenarios with WSO2 ESB 1.7" webinar!

Be sure to register here for it, and the other upcoming webinars on the leading open source enterprise service bus (ESB)

http://wso2.com/about/news/esb-webinar-series-july-08/

WSO2 Enterprise Service Bus (ESB) Webinar series

2008-06-26T11:20:00.000-07:00

The WSO2 Enterprise Service Bus (ESB) team will be conducting a series of webinars, that will introduce some of the new features of the WSO2 ESB 1.7, common integration scenarios, the support for Financial messaging with FIX and complex event processing

Be sure to register! More details from: http://wso2.com/about/news/esb-webinar-series-july-08/

Improved handling of large messages using constant memory, via connection throttling

2008-06-24T02:01:00.000-07:00

I recently updated the Apache Synapse ESB http/s transport module, and replaced the use of OS Pipes and buffers we had, with shared buffers from the Apache HttpCore/NIO library. Oleg Kalnichevski suggested this approach over an year back, and has helped us immensely over the years, in our efforts to build the fastest open source Enterprise Service Bus around!

This approach uses a constant size shared buffer to read a message from the NIO reactor, into the application processing threads, and signals the channel (i.e. TCP socket) that we are not willing to accept any more data - when this buffer fills up - effectively throttling the connection. This allows us to handle very large messages without requiring loads of heap memory. Now we can handle extremely high concurrency rates, as well as large messages - without dropping any!

Read the latest performance numbers for Apache Synapse ESB / WSO2 ESB here: http://wso2.org/library/3740

WSO2 Enterprise Service Bus (ESB) Performance Testing - Round 3

2008-06-23T20:51:00.000-07:00

We have just concluded the third round of Enterprise Service Bus (ESB) performance testing here at WSO2!

Its great to hear that even BEA / Oracle was very much interested about the performance numbers we demonstrated last year, and wanted to beat our numbers with their latest release of AquaLogic Service Bus 3.0 on Weblogic 10. I think this demonstrates a level of trust by the Enterprise Service Bus (ESB) vendors and users alike, on the tests we conducted. Several users re-ran some of the tests themselves, during the past year; while some vendors promised to re-run the tests and publish their response, but have so far failed to talk about the results they obtained. Ross Mason from Mulesource has been wanting to publish performance test results for Mule for the last 2 years! The fact that we made the configurations/tuning and the tools used openly available, and asked for help from the vendors to optimally configure their ESB's for the scenarios, and reporting back any problems we encountered adds to the level of openness we demonstrated. Additionally this round of testing shows that even the WSO2 Enterprise Service Bus (ESB) / Apache Synapse Enterprise Service Bus (ESB) has room for improvement against a proprietary ESB that we benchmarked against, but also shows a very clear lead against the other open source alternatives.

It should be noted that the advantages of the WSO2 Enterprise Service Bus (ESB) does not lie only on its low resource usage foot print, excellent performance or scalability or its free and open source Apache License v2.0 alone; but in its ease of use, the ability to define a configuration graphically, and the fact that it ships over 55 working samples and documentation that demonstrates various features to help users become effective from day 1. Add to this the excellent level of free support that users always talk about, and compare this to some of the mailing lists for other open source ESB's where you hardly would get a reply to a technically challenging question.

Let me just highlight some of the observations and conclusions here

- Mule CE 2.0.1 couldn't handle the cases where we used a concurrency level of 80; while other ESB's scaled to support to over 2500 concurrent connections. This was after tuning the maximum active thread count to 100 from its default value, which limited Mule to a very few concurrent connections.

- A proprietary version of an open source ESB had the same problem described above. However, we cannot name this ESB due to its restrictive license which states that we cannot : “..publicly disseminate performance information or analysis (including, without limitation, benchmarks) from any source relating to the Software.”. One of my friends commented that the above line is a “Nice way to "performance tune" your slow app :P”

- Mule CE 2.0.1 also dropped 1% of ALL requests it received

- Apache ServiceMix 3.2.1 failed to forward the incoming SOAPAction for proxy services, and this was now a known issue (I would consider this a blocker, and would suggest that ServiceMix folks follow up a 3.2.2 release just to fix this critical issue)

- A proprietary ESB we benchmarked and beat last year, did some major improvements to their performance, and did 1.6~1.9 times better than us for some of the scenarios

- The WSO2 Enterprise Service Bus (ESB) / Apache Synapse Enterprise Service Bus (ESB) shows a clear lead and dominates the open source ESB space

Read about it all here: http://wso2.org/library/3740 and run the benchmarks yourself!

Get ready for the 3rd round of performance testing!

2008-06-12T12:33:00.000-07:00

Its been almost an year since we last conducted performance benchmarks on the leading open source and commercial ESB's. We published the results from the last performance tests at:
http://wso2.org/library/1721
http://wso2.org/library/2259

For this round, I will be using a new set of scenarios, where I would measure the performance of each implementation for 1K, 5K, 10K, 100K and 500K requests/responses, for the simple proxy, content based routing and XSLT transformation scenarios. I would also introduce an artificial delay at the backend service implementation to determine how well the ESB's operate when waiting for the actual services to respond.

For this round of testing, we will compare:
Apache Synapse / WSO2 ESB 1.7
Mule 1.5.1 EE
Mule 2.0.1 CE
Apache ServiceMix 3.2.1/4.x?
JBossESB 4.3GA
OpenESB v2

Stay tuned for more information as we actually conduct the tests.. and post results!

A look back at the last year..

2008-06-12T12:31:00.000-07:00

Its been exactly one year since the release of the WSO2 Enterprise Service Bus v1.0, in June 2007!

Yesterday we announced the general availability of the WSO2 Enterprise Service Bus (ESB) v1.7, which adds dramatic improvements to performance, as well as support for many new features. The past year has been very exciting for us, and we have been able to impress many happy users on the community, as well as win many production deployments world wide.

Let me highlight just some of the many users of Apache Synapse and the WSO2 ESB during the last year. The ability to easily support Hessian messages in an extremely short period of time, won us a great new client from Germany! They are now actively involved with the projects, and even help us perform load/regression testing as well as testing some of the most complex features such as sticky load balancing in clustered deployments, general clustering high availability and failover and advanced JMX management etc. We have also been chosen over the leading commercial ESB after a one week bakeoff conducted on-site, by one of the largest integrated managed care organizations in the US. In addition, a leading mobile resource management company in the US uses the technology to communicate with thousands of mobile devices, and to integrate existing JMS based services with partners using SOAP Web Services. The underlying Apache Synapse ESB which powers the WSO2 ESB, has been deployed in production by one of the largest auto makers in the world, and will soon be used in production by one of the largest armies in the world as well! Synapse has been able to gather a large community from the US, Europe, Latin America and Asia/China. Its even been embedded already into one of the leading network appliances for SOA, built by a world leader in telecommunications, with others to follow shortly! Its also being used by one of the largest banks operational mostly in Asia, Middle East and Africa with presence in over 70 countries worldwide. Many governments including the Government of Sri Lanka where I live, has standardised on the use of Apache Synapse as "the" ESB. These are all a few of the real users of the software we've built over the last year

Performance of the core engine has always surpassed the leading commercial alternatives, as well as other open source options such as Mule and ServiceMix.(See http://wso2.org/library/1721 and http://wso2.org/library/2259) In fact when we published the results of the performance tests carried out, it surely has created a controversy!

However, we will soon be conducting another round performance testing, so that one could easily compare the latest versions of the WSO2 ESB, Apache Synapse, Mule EE, Mule CE, ServiceMix, JBoss ESB and OpenESB, along with the leading proprietary ESB. We will share all configurations, tools and artifacts used, so that one could simply re-run the tests themselves if needed. Thus lately I have been looking into OpenESB to create some of the scenarios with it, so keep a lookout for my observations!

Since many users from the financial markets has been interested in Apache Synapse and the WSO2 ESB, with the latest releases of these, we have added support the Financial Information eXchange (FIX) protocol and AMQP messaging. This is in addition to the existing support for other transports such as non-blocking http, https, JMS, Mail (pop3, imap, smtp), File systems (ftp, sftp, zip, jar, tar, file, cifs, webdav.. etc). The latest WSO2 ESB v1.7 also adds enhanced support for JMX management, and graceful shutdown of instances, and enhanced support for clustered deployments with support for sticky load balancing, throttling and caching. It also embeds a version of the WSO2 Registry, and provides the ability to connect to an external instance as well.

Apache Synapse / WSO2 ESB Community Site (beta)

2008-03-10T09:52:00.001-07:00

It is with great pleasure that we announce the launch of the Apache Synapse and the WSO2 ESB user community site - http://esbsite.org!

We hope that this will help us build a vibrant community around our users and developers, and help our community to share their experiences, extensions, articles, case studies, screen casts, patches and many more artifacts and resources related to these open source ESB's.

You can sign up as a member, and begin to upload your custom extensions, or article and code snippets etc, and rate content uploaded by fellow users. In the future, the Synpase developer community would also publish some of the extensions over the community site, which will allow users to update their installations with new functionality. The community site will also feature patches for released versions of the ESB's as well as link to articles and screen casts and FAQ's of interest.

Mediators

EJB Callout Mediator
URLRewrite Mediator
Java Rule (JSR-94) Mediator
Smooks Transform Mediator
IM Mediator

Article and Screencast summaries

Apache Synapse / WSO2 ESB Performance against Mule and Service Mix [Article]
Apache Synapse / WSO2 ESB Performance against the leading commercial ESB [Article]
HMO Case Study using the WSO2 ESB [Case study]
Exposing EJB and WS Repositories using Service Aggregation, Utilizing WSO2's Open Source ESB [Case study]
Stepping into Apache Synapse [Article]
Bridging from HTTP and WS-Security to JMS with the WSO2 ESB [Screencast]
Simple Proxy Services with the WSO2 ESB [Screencast]

Sample XSLTs

Identity transformation with a namespace change

Sample XPaths

Selecting over both SOAP 1.1 and 1.2

Sample Configurations

Using Clone and Aggregate mediators and using XSLT's for selecting the highest quote

FAQ for Apache Synapse and WSO2 ESB

Patches for Apache Synapse and WSO2 ESB

Enjoy!
the WSO2 ESB development team

Trying out a ride on a Mule and a Camel for a difference...

2008-02-12T07:31:00.000-08:00

After the recent discussions on the Apache Synapse mailing list by a Camel contributor, I thought of looking at both Apache ServiceMix/Camel and Mule.. My first impressions of both are that they are very much closer to the 'developer' level users, who would want to 'code' into their ESB's, instead of using simple configuration. Also, its simply obvious that both of these lacks adequate samples to show most of the features they claim to support. Sometime back I tried one of the leading commercial ESB's, and I must say that the samples, concepts and the documentation it provided, with a simple to understand GUI for management and administration was really good. But then again, it comes with a rather expensive price tag, and there are free and open source ESB's today, that ships with the Apache License v2.0 - with the same level of documentation, GUI management and even more samples!

So today, I had a look at one of the samples that Mule ships - which is called the Loan Broker sample (http://mule.mulesource.org/display/MULE/LoanBroker+ESB). Although a nice image on the description section says that this example accepts a REST request from a client, it seems like the sample doesn't really show that.. but makes a blind in-vm call. Running the example, it seems like the client, the ESB, the Credit Agency and the Bank etc was all in one mess.. and I soon found our it really was :-)..

The full example contains of 41 Java classes! and you actually define Java classes to hold messages which are supposed to be REST or SOAP or JMS - Text/Binary? Take a look at the org.mule.examples.loanbroker.messages.CustomerQuoteRequest where everything starts off.. Wow! and this is to supposedly carry a REST call made by a client?

Anyway getting on the Mule was quite difficult.. but I guess at the same time the Camel was quite a challenge too.. When I downloaded Camel, it was just a couple of mega bytes and it in-fact impressed me. But after I extracted the release, and looked for what I could do, it asked me to run Maven (with quite a few errors I had to troubleshoot and overcome) and downloaded many more megabytes of "stuff"! Running the Mule samples, I realized that Mule does the same..!

One of the main barriers of entry to using Camel and Mule was the lack of adequate samples, and having to understand complex concepts and languages to even do something simple! Luckily Apache Synapse and the WSO2 ESB ships over 50 working samples that can be started and studied easily! In the days ahead I will update you with more information on these open source ESB's and how you could implement the same samples with Apache Synapse/WSO2 ESB much easily!