Claus Ibsen (@davsclaus) riding the Apache Camel

Apache Camel 3.11 What's New

2021-07-04T20:57:00.004+02:00

Apache Camel 3.11 has just been released.

This is a LTS release which will be supported for 1 year with regular patch and security releases.

This blog post first details the noteworthy changes since the last 3.10 release from last month.

For readers that are upgrading from the last 3.7 LTS release then we have added a summary section that highlights all the important new features and changes (3.7 to 3.11).

At first what did we do since the 3.10 release.

So what's in this release since 3.10

This release introduces a set of new features and noticeable improvements that we will cover in this blog post.

Kamelets

Kamelets is a higher level building blocks that we keep innovating and improve over the coming releases.

For Camel 3.11 we worked on making Kamelets universal across the various runtimes such as standalone, Karaf, Spring Boot, and Quarkus.

We added a new camel-kamelet-main component that is intended for developers to try out or develop custom Kamelets. This module runs standalone which is intentional as we want to ensure Kamelets are not tied to a specific runtime (or the cloud on Kubernetes) but are truly universal in any environment where you can use Camel.

You can find an example with camel-kamelet-main at https://github.com/apache/camel-examples/tree/main/examples/kamelet-main

The YAML DSL has improved error reporting when parsing to better report to Camel end users where the problem is.

Common Source timestamp

We added a `getSourceTimestamp` API on `Message` to get hold of the timestamp from the source of the message.

The idea is to have a common API across all the Camel components that has a timestamp of the event (such as JMS, Kafka, AWS, File/FTP etc).

Cloud component

The Camel AWS, Azure, and HuaweiCloud components have had various bug fixes and smaller improvements.

Quarkus

This release is the baseline for Quarkus 2 support which is to follow shortly after this release with a new Camel Quarkus release.

Spring Boot

We have upgraded to latest Spring Boot 2.5.1 release.

No OSGi code in main project

We had about six remaining Camel components which had some special OSGi Java source code.

The OSGi code has been ported over to the Camel Karaf project.

Better Java 16 support

Although Java 16 is not officially supported, we did improve a few Camel components to make them work with Java 16.

The official support is Java 11 (primary) and Java 8 (secondary).

New components

This release has a number of new components, data formats and languages:

camel-huaweicloud-functiongraph - To call serverless functions on Huawei Cloud
camel-huaweicloud-iam - To securely manage users on Huawei Cloud
camel-kamelet-main - Main to run Kamelet standalone
camel-resourceresolver-github - Resource resolver to load files from GitHub

Upgrading

Make sure to read the upgrade guide if you are upgrading from a previous Camel version.

Release Notes

You can find more information about this release in the release notes, with a list of JIRA tickets resolved in the release.

Summary of changes since the last 3.7 LTS release

It is 6 months since the last 3.7 LTS release, and here is a high level summary of the most significant changes we have done:

Optimized core (faster startup and quicker routing engine)
Modularized core (even smaller core)
Reduced Object Allocations (lower memory footprint)
Reflection free (Native compilation friendly)
Optimized toD EIP for messaging based components
Better startup and shutdown logging
Java Flight Recorder
Routes loader (Java, XML, YAML, Groovy, JavaScript, and Kotlin)
YAML DSL
Kamelets
17 new components
Support for Spring Boot 2.5 and Quarkus 2.0

There are many other great new features and improvements that you can find detailed in each of the Whats New blog posts:

Webinar: Integrate systems in the age of Quarkus and Camel

2021-05-29T11:15:00.006+02:00

A few days ago I presented a webinar where he covered all the latest innovations with Apache Camel with focus on Camel Quarkus, Camel K, and Kamelets. This trio is a powerful combination that takes Camel to another level, which allows non developers and IT professionals, to manage and bind systems together without any Camel knowledge.

Kamelets being the Apache Camel solution for an app store experience with integration software.

The webinar is online on youtube and the slides is here.

Apache Camel 3.9 - No more saw tooth JVM garbage collection

2021-03-16T14:07:00.001+01:00

We continue our effort to optimize Apache Camel. This is blog post part 7 which covers are latest effort on dramatically reducing the object allocations caused by Camel while routing messages.

The good news is that we have overachieved and was able to reduce object allocations to ZERO!!! - so no more JVM memory usage graphs with saw tooth (note: in real world use-cases there will always be user data causing object allocations - but I wanted to have a click-bait blog title).

To help identify potential areas of improvement in the core Camel, we put together a small performance application, which has only a single route triggered by a timer every producing 1000 msg/sec. These messages are routed to 10 different log endpoints (logging turned off). This allows us to focus on the internals of Camel only and what code paths is executed and what objects are being allocated and in-use by the internal routing engine. There are no message data (body or headers), or network communication etc.

Running the example (JVM heap size set to max 32mb) for 10 minutes profiled by JFR and browsed in JDK mission control we can see the dramatic difference.

In Camel 3.8 597mb of objects is allocated by Camel in total.

And in Camel 3.9 that is ZERO.

How did we get to zero?

That is a long journey that started about a year ago, and we have gradually optimised Camel which I have blogged about in the 6 parts preceding this post.

All this work is like pealing an onion, layer after layer. As one layer has been optimised, then the profiler reveals another layer, and so on. This time we could identify 5 areas for improvements:

consumers
core EIP patterns
internal routing processor
error handler
exchange and message

The consumers are the source of incoming messages into Apache Camel. And so that is a great place to start. It's the consumers that allocate a new exchange, populate the exchange with message data such as body and headers.

After that it's the internal routing engine that routes the exchange via EIP patterns. And here we identified several spots where we could eliminate object allocations, or reduce allocations when some features are not in use etc. Error handling is one of the most complex part in the core Camel, and it uses objects to keep state in case of exceptions to handle redeliveries and whatnot. We were able to split the error handling into two tasks that operate either as a simplified or complex task. In the core EIP patterns we were able to optimize code that reduces object allocations.

The 5th area we optimized is the exchange object. EIPs and the Camel routing engine store state per exchange on the exchange instance itself as exchange properties. That data is stored in a Map which means for each entry both a key is allocated in the java.util.Map. We optimized this to use an internal object array where each key is hardcoded as an index entry in the array. That means read/write is very fast and simple as its just an array index.

And then we ..... cheated ... instead of allocating new objects (via new constructor) we recycle existing objects from the previous exchange to the next. In other words we are using a sort of object pooling - this feature is called exchange pooling in Camel.

Exchange Pooling

The diagram above with ZERO object allocation is in fact with exchange pooling enabled. If exchange pooling is turned off (default), then the diagram should have been as below:

As you can see there is saw-tooth graph. However the total object allocation is gone down from 597mb to 492mb (18% reduction).

Awesome this is fantastic. And yes indeed it is. However when using anything there are both pros and cons, and so with object pooling. There is tiny tiny overhead of Camel to manage the object pools, and to "scrub" objects before they can be recused. That is a possibly a very very tiny CPU overhead compared to the JVM allocate and initialise new objects; instead of pool reuse. The biggest con is object leaks .. if objects are no returned back in the pool. Therefore you can turn on statistics which will report a WARN if a leak is detected when you stop Camel. The objects must be manually returned back into the pool, which we have coded in all the Camel components, and of course in core Camel. Now object leaks in this situation is not severe as you just have a situation as if there are no pooling, the JVM will create a new object - so the object allocations goes up, but its not severe like a database pool leaking TCP network connections.

Upcoming work

There are a few very complex EIP patterns and Camel component which does not yet support object pooling. We have this on the roadmap for Camel 3.10.

Camel 3.9 is planned for release in March 2021.

Apache Camel 3.8 and Java Flight Recorder

2021-01-20T12:35:00.002+01:00

In the upcoming Apache Camel 3.8 release we have a new Camel component to integrate with Java Flight Recorder.

Camel is now capable of capturing "work steps" during startup that can be recorded with Java Flight Recorder. This can be used to better diagnose and find where your Camel applications may be slow to startup, for example due to a misbehaving component or custom user code.

The screenshot below shows a recording that has captured a Camel application that takes about 3 seconds to startup. Its a very tiny application so we expected it to be faster.

If we sort the events by duration in the JDK mission control, we can see that there are 4 events that take over 2 seconds.

The sequence is a sequence of the following step (sub step):

Initializing context -> Initializing routes -> Creating route (route2) -> Creating Bean processor (bean1)

What we can see is that the step with the highest depth is "Creating Bean processor" which takes about 2 seconds. This is the culprit of the bottleneck.

If we check the Camel route for where bean1 is in use, its in route2 at:

from("direct:slow")
.to("log:slow?level=OFF")
.bean(MyBean.class, "hello");

Here we can see the bean is using MyBean class, which we can then look at next:

public MyBean() {

// force slow startup

try {

LOG.warn("Forcing 2 sec delay to have slow startup");

Thread.sleep(2000);

} catch (Exception e) {

// ignore

}

Ah okay here is the problem. The bean is sleeping for 2 seconds. Yes of course this is a made up example, but it does affect the recording and allow us to find it via the JDK mission control tool.

We also offer a logging recorder where you can "see" some of the same information as in JDK mission control. However when using JDK mission control, you have the entire JFR recording that also captures alot of JVM information about CPU and memory use and whatnot.

To use Java Flight Recorder with Camel, all you have to do is to add camel-jfr on the classpath. Then Camel will auto-detect this and enable it. You can configure the recorder with various options which will be documented as part of the common options.

But for quickly finding startup bottlenecks for Camel applications then the logging recorder is a good start. The screenshot below shows the logging output, and as you can see from the red square we have identified where the "2 second" problem is.

The logging recorder comes out of the box in camel-core, and you can just use it by configuring:

camel.main.startup-recorder = logging

If you are using Camel Main, Camel Quarkus etc. And for Spring Boot, you can enable it with

camel.springboot.startup-recorder = logging

You can also set a custom recorder, or one of the out of the box implementation via Java code:

camelContext.adapt(ExtendedCamelContext.class)
.setStartupStepRecorder(...);

You can try this example (camel-example-flight-recorder) from the Camel Examples git repository. From command line you can run

mvn camel:run

And Camel will automatic capture a JFR recording and save to disk. The output of the file is shown in the log, which you can then open from JDK mission control.

Apache Camel 3.7 (LTS) Released - The fastest Camel ever

2020-12-21T10:00:00.001+01:00

The Apache Camel 3.7 was released some days ago.

This is a LTS release which means we will provide patch releases for one year. The next planned LTS release is 3.10 scheduled towards summer 2021.

So what's in this release

This release introduces a set of new features and noticeable improvements that we will cover in this blog post.

Pre compiled languages

We continued our avenue of making Camel faster and smaller. This time we focused on the built-in Simple scripting language.

First we added the jOOR language. jOOR is a small Java tool for performing runtime compilation of Java source code in-memory. It has some limitations but generally works well for small scripting code (requires Java 11 onwards).

Then we worked on compiled simple.

Compiled Simple

The csimple language is parsed into regular Java source code and compiled together with all the other source code, or compiled once during bootstrap via jOOR.

In a nutshell, compiled simple language excels over simple language when using dynamic Object-Graph Navigation Language (OGNL) method calls.

For example profiling the following simple expression

<simple>${exchangeProperty.user.getName} != null && ${exchangeProperty.user.getAge} > 11</simple>

with the equivalent csimple expression:

<csimple>${exchangeProperty.user} != null &&

${exchangeProperty.user.getName()} != null &&

${exchangeProperty.user.getAge()} > 11</csimple>

yields a dramatic 100 times performance improvement in reduced cpu usage as shown in the screenshot:

For more information about the compiled simple language and further break down of performance improvements then read my recent blog post introducing the csimple language.

We have provided two small examples that demonstrate csimple as pre compiled and as runtime compiled during bootstrap.

You can find these two examples from the official Apache Camel examples repository at:

Optimized core

We have continued the effort to optimize camel-core. This time a number of smaller improvements in various areas such as replacing regular expressions with regular Java code when regular expressions were overkill (regexp take up sizeable heap memory).

The direct component has been enhanced to avoid synchronisation when the producer calls the consumer.

We also enhanced the internals of the event notifier separating startup/stop events from routing events, gaining a small performance improvement during routing.

We also reduced the number of objects used during routing which reduced the memory usage.

Another significant win was to bulk together all the type converters from the core, into two classes (source generated). This avoids registering individually each type converter into the type converter registry which saves 20kb of heap memory.

If you are more curious about how we did these optimisations and with some performance numbers, then read another of my recent blog posts.

Optimized components startup

The camel core has been optimized in Camel 3 to be small, slim, and fast on startup. This benefits Camel Quarkus which can do built time optimizations that take advantage of the optimized camel core.

We have continued this effort in the Camel components where whenever possible initialization is moved ahead to an earlier phase during startup, that allows enhanced built time optimizations. As there are a lot of Camel components then this work will progress over the next couple of Camel releases.

Separating Model and EIP processors

In this release we untangled model, reifier and processors.

This is a great achievement which allows us to take this even further with design time vs runtime.

Model -> Reifier -> Processor

(startup) (startup) (runtime)

The model is the structure of the DSL which you can think of as _design time_ specifying your Camel routes. The model is executed once during startup and via the reifier (factory) the runtime EIP processors is created. After this work is done, the model is essentially not needed anymore.

By separating this into different JARs (camel-core-model, camel-core-reifier, camel-core-processor) then we ensure they are separated and this allows us to better do built time optimizations and dead code elimination via Quarkus and/or GraalVM.

This brings up to lightweight mode.

Lightweight mode

We started an experiment earlier with a lightweight mode. With the separation of the model from the processors, then we have a great step forward, which allowed us to make the lightweight mode available for end users to turn on.

In lightweight mode Camel removes all references to the model after startup which causes the JVM to be able to garbage collect all model objects and unload classes, freeing up memory.

After this it's no longer possible to dynamically add new Camel routes. The lightweight mode is intended for microservice/serverless architectures, with a closed world assumption.

Autowiring components

The Camel components is now capable of autowiring by type. For example the AWS SQS components can automatically lookup in the registry if there is a single instance of SqsClient, and then pre configure itself.

We have marked up in the Camel documentation which component options supports this by showing Autowired in bold in the description.

Salesforce fixes

Our recent Camel committer Jeremy Ross did great work to improve and fix bugs in the camel-salesforce component. We expect more to come from him.

VertX Kafka Component

A new Kafka component has been developed that uses the Vert.X Kafka Java Client which allows us to use all of its features, and also its robustness and stability.

The camel-vertx-kafka component is intended to be (more) feature complete with the existing camel-kafka component. We will continue this work for the next couple of Camel releases.

DataSonnet

The new camel-datasonnet component, is to be used for data transformation using the DataSonnet.

DataSonnet is an open source JSON-centric, template-based data transformation standard built to rival proprietary options available in the market.

Spring Boot

We have upgraded to Spring Boot 2.4.

New components

This release has 7 new components, data formats or languages:

AtlasMap: Transforms the message using an [AtlasMap](https://www.atlasmap.io/) transformation
Kubernetes Custom Resources: Perform operations on Kubernetes Custom Resources and get notified on Deployment changes
Vert.X Kafka: Sent and receive messages to/from an Apache Kafka broker using vert.x Kafka client
JSON JSON-B: Marshal POJOs to JSON and back using JSON-B
CSimple: Evaluate a compile simple expression language
DataSonnet: To use DataSonnet scripts in Camel expressions or predicates
jOOR: Evaluate a jOOR (Java compiled once at runtime) expression language

Upgrading

Make sure to read the upgrade guide if you are upgrading to this release from a previous Camel version.

More details

The previous LTS release was Camel 3.4. We have blog posts for what's new in Camel 3.5 and Camel 3.6 you may want to read to cover all news between the two LTS releases.

Release Notes

You can find more information about this release in the release notes, with a list of JIRA tickets resolved in the release.

Apache Camel 3.7 - Compiled Simple Language (Part 6)

2020-12-09T14:29:00.005+01:00

I have previously blogged about the optimizations we are doing in the Apache Camel core. The first 3 blogs (part1, part2, part3) were a while back leading up to the 3.4 LTS release.

We have done more work (part4, part5) and this (part 6) that will be included in the next Camel 3.7 LTS release (to be released this month).

This time we worked on a new variation of the Camel simple language, called csimple.

Compiled Simple (csimple)

The csimple language is parsed into regular Java source code and compiled together with all the other source code, or compiled once during bootstrap via the camel-csimple-joor module.

To better understand why we created csimple then you can read on about the difference between simple and csimple (in the section further below). But first let me show you some numbers.

I profiled a Camel application that processes 1 million messages, which are triggered in-memory via a timer, and calls a bean to select a random User object that contains user information. The message is then multicasted and processed concurrently by 10 threads, which does some content based routing based on information on the User object.

The Camel route is from a Spring XML file, and then a few Java beans to represent the User object and the bean to select a random user.

The application is profiled running with simple and csimple language until all messages has been processed.

The main focus is the difference between the following simple and csimple expression (XML DSL)

<simple>${exchangeProperty.user.getName} != null &&
${exchangeProperty.user.getAge} > 11
</simple>

<csimple>${exchangeProperty.user} != null &&
${exchangeProperty.user.getName()} != null &&
${exchangeProperty.user.getAge()} > 11
</csimple>

At first glance they may look identical, but the csimple language has an additional not null check whether the user object exists or not. You may think that the csimple language contains type information but it does actually not. We have "cheated" by using an alias (a feature in csimple) which can be configured in camel-csimple.properties file as shown:

# import our user so csimple language can use the shorthand classname

import org.example.User;

# alias to make it shorter to type this

exchangeProperty.user = exchangePropertyAs('user', User.class)

Here we can see the alias is referring to the exchangePropertyAs function that takes the property name as first input, and then the class name as 2nd input. And because we have a Java import statement in the top of the properties file, we can type the local classname User.class instead of org.example.User.

The csimple script gets parsed into the following Java source code, which is then compiled by the regular Java compiler together with the rest of the application source code:

@Override

public Object evaluate(CamelContext context, Exchange exchange, Message message, Object body) throws Exception {

return isNotEqualTo(exchange, exchangePropertyAs(exchange, "user", User.class), null) && isNotEqualTo(exchange, exchangePropertyAs(exchange, "user", User.class).getName(), null) && isGreaterThan(exchange, exchangePropertyAs(exchange, "user", User.class).getAge(), 11);

}

Performance numbers

Okay lets get back to the performance numbers. The raw data is presented below as screenshot and table.

CPU usage

simple 814815 millis
csimple 7854 millis

Memory usage

simple 123 objects      5328 bytes
bean              3171 objects   177680 bytes

csimple          3 objects    792 bytes

As we can see the cpu usages is dramatically reduced by a factor of 100 (one hundred).

The memory usage is also reduced. The simple language uses OGNL expression with the bean language and hence we should calculate the combined usage which then is roughly 3294 objects taking up about 183kb of heap memory. (the bean language has introspection cache and other things). The csimple language is very very tiny with just 3 objects taking up 792 bytes of heap memory. The memory usage is dramatically reduced by a factor of 231.

The memory screenshot includes simple language for both runs, the reason is that there are some basic simple expressions in the route which was not changed to csimple. Only the script that performed the most complex expression with OGNL on the User object.

So all together is a very dramatic reduction in both cpu and memory. How can this be?

Very low footprint, why?

The low footprint is because of mainly two reasons

The script is compiled as Java code by the Java compiler either at build time or during bootstrap.

2)
The script is not using bean language / bean introspection with reflection for OGNL paths. However this requires the script to include type information so the Java compiler knows the types to compile the OGNL paths as regular java method calls. This is the main driver of the reduced footprint on both memory and cpu. Basic scripts such as ${header.zipCode} != null would have similar footprint. However csimple with pre compiled would have lower footprint as the script is pre parsed which otherwise would have to happen during bootstrap to generate the Java source code for the Java compiler to do an in-memory compilation; which will impact the startup performance.

Are they any limitations?

Yes the csimple language is not a 100% replacement for simple (we will continue to improve the feature parity). In the Camel 3.7 release csimple is in preview mode and have the following limitations

- nested functions is currently not supported

- null safe operator is not supported

And for OGNL paths, then as previously mentioned, then csimple requires to be type safe, by including the types of the objects.

Difference between simple and csimple

The simple language is a dynamic expression language which is runtime parsed into a set of Camel Expressions or Predicates.

The csimple language is parsed into regular Java source code and compiled together with all the other source code, or compiled once during bootstrap via the camel-csimple-joor module.

The simple langauge is generally very lightweight and fast, however for some use-cases with dynamic method calls via OGNL paths, then the simple language does runtime introspection and reflection calls. This has an overhead on performance, and was one of the reasons why csimple was created.

The csimple language requires to be typesafe and method calls via OGNL paths requires to know the type during parsing. This means for csimple languages expressions you would need to provide the class type in the script, where as simple introspects this at runtime.

In other words the simple language is using duck typing (if it looks like a duck, and quacks like a duck, then it is a duck) and csimple is using Java type (typesafety). If there is a type error then simple will report this at runtime, and with csimple there will be a Java compilation error.

Any examples for me to try?

We have provide two small examples that demonstrate csimple as pre compiled and as runtime compiled during bootstrap. You can find these two examples from the official Apache Camel examples repository at:

Whats Next

We want to implement the missing feature for nested functions and the null safe operator. We are also working on camel-quarkus to make csimple optimized for Quarkus and GraalVM. This effort is already started and Camel 3.7 will come with the first work in this area.

We also want to work on speeding up the runtime compilation to be able to do batch compilation. Currently each csimple script is compiled sequentially.

And we want to take a look at if we can make runtime compilation work better with Spring Boot in its tar jar mode.

However at first enjoy csimple in the upcoming Camel 3.7 LTS release and as always we want your feedback and love contributions.

Apache Camel 3.7 - More camel-core optimizations coming (Part 5)

2020-11-04T14:34:00.003+01:00

I have previously blogged about the optimziations we are doing in the Apache Camel core. The first 3 blogs (part1, part2, part3) were a while back leading up to the 3.4 LTS release.

Now we have done some more work (part4) and this part 5 that is coming up in the next 3.7 LTS release.

This time we have mainly been focusing on reducing the footprint of Camel after bootstrapping.

separating design time model vs runtime processors

We have continued the modularisation and have separated the design time vs runtime parts in the core.

The route model is now in camel-core-model, and the runtime EIP implementations are in camel-core-processor. And in between we have camel-core-reifier which transforms the model into processors. This separation plays a significant part of continued effort of making Camel even smaller, such as Quarkus and GraalVM runtimes. This will essentially allow us to pre build from the route model the runtime processors, and then at runtime not even include the model and reifier classes. There is still some work to get to the finish line, but Camel 3.7 is a major step forward.

Lightweight mode

For runtimes that are not Quarkus or GraalVM based, then we have a new lightweight switch in camel-main / CamelContext that can be turned on, that does some internal optimizations by null'ing the model and reifiers from the runtime processors, which then aids the JVM garbage collector, so it can reduce memory.

The following two screenshots shows the camel-example-main-tiny running with Camel 3.4.4 and 3.7.0 (lightweight mode).

The top screenshots shows that there are 731 Camel objects and that they take up about 58kb of memory in the heap. Now in Camel 3.7 this has been significant improved as there are only 340 objects and they take up 25kb. This means Camel has shrunk to half the size in memory footprint. The memory referred to here is the memory used by Camel to bootstrap and start itself.

Improved type converter

The type converter system have been optimized to be quicker for common convertions, and as well reducing the number of method calls during a conversion. The core converters are now bulked together and source code generated into 1 class per maven artefacts (2 in total for core). Before this there was about 10 different converter classes. This helps reduce the footprint of the type converter registry significantly. Now all the core converters are referenced by just 2 classes. Before each converter method was registered via a double keyed map which leads to a map containing 200 converters (takes up 20kb of heap memory). And on top of that then each converter method is invoked via lambda call that the JVM compiles into a separate class (so there was also 100 extra classes loaded). So all together this greatly reduces the footprint by lowering the heap memory usage by 20kb and 100 less classes loaded. And if you are using any of the XML components then camel-core-jaxb/camel-core-jaxp comes with 100 converters as well, which means if they were loaded then that would double the memory footprint. And the type converter is now also faster as the lookup of the converter is quicker and uses just Java primitives, where as before new objects was created as keys to lookup in the internal registry map. So there is also less garbage for GC.

The following screenshot shows the same example as before with Camel 3.7.0 (no lightweight mode):

Here we can see that there are 346 objects and they take up 28kb of memory. That is only +6 objects and +3kb more memory. So the lightweight mode did not in this example reduce as much memory. That is because there is only 1 route and 4 different models in use. If the application has more routes and more EIP models and reifiers, then more memory would be reduced.

The cost is however that the application cannot dynamically add new routes (via the model). The lightweight mode should be used with caution and it is only for "static Camel applications".

Optimized direct component

The direct component has been optimized to wire up its producers and consumers during startup. Before the direct producer would for each message then lookup its associated consumer before sending the message to the consumer. We have moved this to an earlier phase. And then in case a consumer is stopped / removed / or re-created etc then the producer is capable of invalidating its old wiring and re-associate to the new consumer. This improved avoids internal lookup and thread synchronisation.

Optimized event notifier

Another performance improvement was to divide event notifiers into lifecycle vs messaging events. As lifecycle events such as staritng and stopping Camel does not affect what happens during routing messages. We use this knowledge to avoid checking for runtime events, in the routing engine, if there isn't any custom lifecycle added to Camel.

Parsing without regular expressions

Some parts in Camel uses regular expressions to parse, such as a string to long time converter, so you could convert 10m30s (as 10 minutes and 30 seconds). This was using several regular expressions. However when they get compiled by JVM they objects in memory are rather large. There are other places internal in Camel that regular expression was used for more basic parsing. So we rewrote those using plain basic Java and this reduced the memory and has better performance. As using regular expression in the first place was a bit overkill.

BootstrapCloseable

Naming in IT is hard, so we named a new marker interface BootstrapCloseable, after java.io.Closeable. We use this to mark services and other entities in Camel as something that are only used during bootstrapping Camel. Then after Camel has been started, we will then invoke those so they can free up resources and memory.

More to come

There will be more to come for Camel 3.7. But at this point we wanted to share the good news that Camel is on a diet and have shrunk to half the size for its memory footprint (startup).

We have plans to look at a compiled simple language which would greatly improve performance for some simple expressions that are dynamic and use method calls. One area of inspiration is the new camel-joor language which gets compiled to Java during bootstrap.

Apache Camel 3.6 - More camel-core optimizations coming (Part 4)

2020-10-09T11:32:00.000+02:00

I have previously blogged about the optimziations we are doing in the Apache Camel core. The first 3 blogs (part1, part2, part3) were a while back leading up to the 3.4 LTS release.

Now we have done some more work that is coming in Camel 3.6 leading up to the next 3.7 LTS release.

To speedup startup we switched to a new uuid generator. The old (classic) generator was inherited from Apache ActiveMQ which needed to ensure its ids were unique in a network of brokers, and therefore to ensure this the generator was using the hostname as prefix in the id. This required on startup to do a network access to obtain this information which costs a little time. Also depending on networks this can be more restrictive and delay the startup. The new generator is a pure in-memory fast generator that was used by Camel K and Camel Quarkus.

We also identified a few other spots during route initialization. For example one small change was to avoid doing some regular expression masking on route endpoints which wasn't necessary anymore.

Now the bigger improvements are in the following areas

Avoid throwing exceptions

We identified on spring runtimes that Camel would query the spring bean registry for known beans by id, which the Spring framework would throw a NoSuchBeanDefinitionException if the bean is not present. As Camel does a bit of optional bean discovery during bootstrap, we found a way to avoid this which prevents this.

Singleton languages

Another related problem is that in Camel 3 due to the modularization then some of the languages (bean, simple, and others) have been changed from being a singleton to prototype scoped. This is in fact one of the biggest problems and we had a Camel user report a problem with thread contention in a high concurrent use-case would race for resolving languages (they are prototype scoped). So you would have this problem, and because the language resolver would query the registry first then Spring would throw that no such bean exception, and then Camel would resolve the language via its own classpath resolver. So all together this cost performance. We can see this in the screenshots from the profiler in the following.

The top screenshot is using Camel 3.5 and the bottom 3.6. In the top we can see the threads are blocked in Camels resolveLanguage method. And in 3.6 then its actually the log4j logger that is blocking for writing to the log file. Both applications are using the same Camel application and have been running for about 8 minutes.

Reduce object allocations

The next screenshots are showing a sample of the object allocations.

With Camel 3.5 we are average about 1000 obj/sec and with 3.6 we are down to about a 1/3th.

One of the improvements to help reduce the object allocations was how parameters to languages was changed from using a Map to a plain object array. The Map takes up more memory and object allocations than a single fixed object array.

Do as much init as possible

Another performance improvement that aids during runtime was that we moved as much we could from the evaluation to the initialization phase in the Camel languages (simple, bean, etc.). We did this by introducing the init phase and ensuring CamelContext was carried around in the interns so we can use the context during the init phase, where its really needed. This ensures the runtime evaluation is as fast as possible.

Other smaller optimizations

We also improved the simple language to be a bit smarter in its binary operators (such as header.foo > 100). Now the simple language has stronger types for numeric and boolean types during its parsing, which allows us to know better from the right and left hand side of the binary operator to do type coercion so the types are comparable by the JVM. Before we may end up with falling back to converting to string types on both sides. And there is more to come, I have some ideas how to work on a compiled simple language.

The screenshots below shows a chart with the CPU, object allocations and thrown exceptions.

As we can see this summarise what was mentioned was done to optimize. The number of exceptions has been reduced to 0 at runtime. There is about 3500 thrown during bootstrap (that is Java JAXB which is used for loading the spring XML file with the Camel routes used for the sample application). We do have a fast XML loader in Camel that is not using JAXB.

Another improvement we did was to build a source code generator for a new UriFactory which allows each component to quickly build dynamic endpoint URIs from a Map of parameters. The previous solution was to use RuntimeCamelCatalog that was more generic and required loading component metadata from json descriptor files. A few components use this to optimize the toD (such as http components). By this change we avoid the runtime catalog as dependency (reduce JAR size) and the source code generated uri factory is much faster (its speedy plain Java). However the sample application used for this blog did not use toD nor the UriFactory.

Apache Camel 3.6 is scheduled for release later this month of October. It's going to be the fastest Camel ever ;)

BarcelonaJUG talk on Tuesday July 28th about Camel 3 in the era of Kubernetes and Serverless

2020-07-27T16:39:00.000+02:00

Just back from PTO and myself and Andrea Cosentino are on the spot tomorrow where we have been invited by Barcelona JUG to give a talk about the Camel 3 and its latest innovation around Kubernetes and Serverless (and Kafka).

It's a free and online event, in English, tomorrow Tuesday 28th at 19.00 CEST
https://www.meetup.com/es-ES/BarcelonaJUG/events/271746564/

The session runs for 45 min with Q and A at the end.

PS: Just updated my platform to use latest Camel K 1.1.0 release, so hope the demo gods are on our side tomorrow.

Apache Camel K 1.0 is here - Why should you care

2020-06-10T14:51:00.000+02:00

Yesterday we released Apache Camel K 1.0 and it was announced on social media and on the Camel website.

So what is Camel K and why should you care? That is a great question and I want to help answer this by referring to great minds.

Hugo Guerrero posted the following tweet

That is a powerful statement from Hugo, where he highlights the groundbreaking innovation from Camel K, that gives developers the tools and means to build Java based services that are both serverless and functional and runs using standard Kubernetes building blocks.

Camel K is the biggest innovation in Apache Camel for the last 10 years. So fill your cup with coffee or tea, and sit back and enjoy the next 10 minutes read.

I give the floor to Nicola Ferraro (co-creator of Camel K) whom have allowed me to re-post his blog post from the announcement yesterday.

Apache Camel K has made a lot of progress since its inception and we're now proud to announce the 1.0 release. We've been working hard in the past months to add more awesome features to Camel K, but also to improve stability and performance. This post contains a list of cool stuff that you'll find in the 1.0 GA release.

First of all, if you're living under a rock and it's the first time you hear about Camel K,
you can read some introductory blog posts here (1 - introducing Camel K) (2 - camel k on knative)
or look at the Apache Camel website that contains a Camel K section
with a lot of material that is automatically generated from the Github repository

User experience

Camel K development style is minimalistic: you need just to write a single file with your integration routes and you can immediately run them on any Kubernetes cluster. This way of defining things is common to many FaaS platforms (although Camel K is not a proper FaaS platform, but a lightweight integration platform) and it's technically difficult to provide IDE support, such as code completion and other utilities, to developers.

But now we've it. The integration tooling team has created some cool extensions for VS Code that make the development experience with Camel K even more exciting.
You don't need to remember the Camel DSL syntax, the IDE will give you suggestions and error highlighting.

Code completion works with Java code, but it's not only limited to it: you also have suggestions and documentation out of the box when writing the Camel URIs and property files.
And you also have many options to run integrations and interact with them, all integrated in the IDE.

Just install the VS Code Extension Pack for Apache Camel to have all these features available.

Getting started tutorials

Good tools are fundamental to have a great development experience with Camel K, but then you need to learn what you can do with such a great power.
We've created a new repository in the Apache organization that hosts getting started examples: the camel-k-examples repository.

So far we've added guides that drive you through:

- 01 Basic: Learn the basics of Camel K and some interesting use cases
- 02 Serverless APIs: How to design a serverless (i.e. auto-scaling, scaling to zero) API and run it in a few minutes

The basic quickstart is also available online, so you can have a look at how camel k works without installing anything on your laptop.

More tutorials are expected to come in the following months. You are also welcome if you want to help us by contributing your own. They are based on the VSCode Didact project, that provides an
awesome user experience.

If you are looking for Camel K code samples that you can just pick and run using the CLI, the examples directory of the Camel K main repository contains a lot of them. You can also run them directly from Github:

kamel run https://raw.githubusercontent.com/apache/camel-k/master/examples/Sample.java

You can find ready-to-use examples written in different languages (e.g. XML, JavaScript and others).

Serverless

Serverless is the most important area where we're focusing the new developments in Apache Camel K, although, you should remember, you can have a wonderful Camel K experience even without serverless features. To enable the serverless profile in Camel K, you just need to have Knative installed.

In recent releases, we have added support for the most recent advancements in Knative, for example, Camel K is very well integrated with the Knative event broker and you can easily produce or consume events from it.

With 2 lines of code you can transfer events (e.g. generated by IoT devices) from your MQTT broker to the mesh:

bridge.groovy

from('paho:mytopic?brokerUrl=tcp://broker-address:1883&clientId=knative-bridge')
.to('knative:event/device-event')

No kidding, you just need to write those two lines of code in a file and run it with kamel run bridge.groovy to push data into the Knative broker.

And you can also scale the Integration out (Integration is a Kubernetes custom resource, kubectl get integrations to see all of them)
to have a higher throughput. Scaling here is manual because the source of events is a MQTT broker (but we've plans to put auto-scaling also in this scenario

The Camel K embedded auto-scaling feature works really well when you want to react to some Knative events:

listener.groovy

from('knative:event/device-event')
.to('http://myhost/webhook/random-id')

This integration is configured to receive all events with `type=device-event` and scales automatically with the load because it is materialized into a Knative Serving Service and automatically subscribed to the Eventing Broker via a Trigger.

It then receives a CloudEvent when your IoT devices produce something and scales down to zero if there's no data coming. You just need to create it (as before, just kamel run listener.groovy), all the remaining configuration is done automatically by the Camel K operator.

We've added much more features for having a better integration with the Knative ecosystem and we've also fixed some compatibility and performance issues that were present in previous versions. The user experience is now much smoother.

If you are a Knative YAML developer (!), instead of using Camel K directly, you also have the option to use Knative Camel Sources which are part of the Knative release. They are wrappers for Camel K integrations that are compatible with all the tools used by Knative developers (such as the kn CLI or the OpenShift serverless console).
Sources in Knative can only push data into the various Knative endpoints, but not the other way around (i.e. they cannot be used to publish data from Knative to the outside).
In Camel K you don't have this limitation: the Route is the fundamental building block of a Camel integration and you can do whatever you want with it.

Fast startup and low memory

We cannot say we're serverless without mentioning the work that we've been doing in improving the performance of Camel K integrations.

Starting from Camel 3.3.0 which is the default version used by Camel K 1.0.0, you can benefit from all improvements that have been made directly in the Camel core to make it much more lightweight. More in depth details of the Camel core improvements can be found the following blog series that highlights what has been changed in the 3.x Camel timeline to reduce memory footprint and speedup the startup time, which is fundamental when running integrations in a serverless environment: part 1, part 2 part 3, part 4.

But improvements are not only limited to the Camel core: we're doing much more. Several months ago we've started a new subproject of Apache Camel named Camel Quarkus with the goal of seamlessly running integrations on top of the Quarkus framework. As you probably know, Quarkus is able to reduce the memory footprint of Java applications and improve the startup time, because it moves much startup logic to the build phase. And Quarkus applications can also be compiled to a native binary, allowing a dramatic improvements in startup performance and very low memory footprint.

In Camel K 1.0.0 we support Camel Quarkus in JVM mode. A goal is to have also the in-cluster native compilation soon (for some DSL languages, such as YAML), in one of next releases!

To use Quarkus as underlying runtime, you just need to enable the Quarkus trait when running an integration:

kamel run myintegration.groovy -t quarkus.enabled=true

Quarkus is expected to be the default underlying runtime in the next release, and support for Standalone mode (via camel-main) will be deprecated and removed. This means that you won't need to enable Quarkus manually in the next releases, but you still need to do it in 1.0.

Fast build time

Every application running on Kubernetes needs to be packaged in a container image, but in Camel K you only provide the integration DSL and the operator does what it takes to run it, including building images directly in the cluster.

The operator manages a pool of reusable container images and if you redeploy your integration code, it does try to reuse existing images from the pool rather than building a new one at each change, because it takes some time to build a new one. It was 1 minute at the beginning...

But Kubernetes is moving so fast that you cannot solve a problem once and forget about it, you need to take care of it continuously. It happened that some of our third party dependencies that we used for doing builds in "vanilla Kube" has slowly degraded in performance up to a point where Camel K user experience was highly affected.

We decided to work harder on the build system in order to dramatically improve (again!) the build phase of Camel K integrations.

Build time can be be now measured in seconds in dev environments such as Minikube. A bunch of seconds, most of the times. This is more than a simple improvement!

Better CLI

The 'kamel' CLI is the main tool we provide to developers to run integrations. It's not a mandatory requirement: at the end, an Integration is a Kubernetes custom resources and you can manage it with any Kubernetes standard tool (e.g. kubectl). But the kamel CLI adds a lot of value for integration developers.

For example, if you're a Camel Java developer it's not super easy to remember the boilerplate that you have to write in order to instantiate a Camel route builder. Now you don't have to remember that:

kamel init Handler.java

You get a Java file with all the boilerplate written for you and you just have to write your integration routes.

It works also with all other languages: Groovy, XML, YAML, Kotlin and JavaScript.
For example you can write:

kamel init foo.js

This way you get a simple route written in JavaScript.

It's not just that. Often Camel K developers need to add a lot of command line options to configure the final behavior of their integration. For example, you may want to add a custom library with the `-d` option or configure a trait with `-t`. E.g.:

kamel run -d mvn:org.my:lib:1.0.0 -d mvn:org.my:otherlib:2.0.0 -t quarkus.enabled=true Handler.java

Sometimes the number of command line parameters you've to add can become too many. For this reason we've added the possibility to specify them as modeline options in the integration file (done by adding a comment line with `camel-k:` as prefix).

Handler.java

// camel-k: dependency=mvn:org.my:lib:1.0.0 dependency=mvn:org.my:otherlib:2.0.0 trait=quarkus.enabled=true

// ...
// your routes here

Once the options are written in the file, you can run the routes with just:

// simply this, additional args are read from the file
kamel run Handler.java

The other options are taken automatically from the file modeline. The CLI also displays the full command to let you know what's running.

This kind of configuration is extremely useful in CI/CD scenarios because it allows you to have self-contained integration files and you don't need to change the pipeline to setup additional options. If you're curious about the CI/CD configurations, you can follow the tutorial about Tekton pipelines to have more information.

Monitoring and Tracing

Ok, you've finished level 1 of Camel K development and you want to make serious things. You're in a very good position because Camel K provides a lot of useful tools to add visibility on what your integration routes are doing.

Let's suppose you've a Prometheus instance in your namespace and you want to publish your integration metrics:

kamel run Routes.java -t prometheus.enabled=true

That's it. No need to setup services a labels to enable scraping. A default prometheus configuration file is also provided for the integration, with sensible defaults. Of course you also have the option to provide your own configuration for advanced use cases.

Now, let's suppose you want to see what your routes are doing and trace the execution flow of an integration. What you need to do is to install an opentracing compatible application in the namespace, such as Jaeger, and run the integration as:

kamel run Routes.java -t prometheus.enabled=true -t tracing.enabled=true

That's it again. The Camel K operator will add the camel-opentracing library and connect it to the Jaeger collector that is available in the namespace. Here again, advanced use cases are supported.

Master routes

Good old Camel users know why and when master routes are useful, but for those who are not familiar with the term, I'm going to provide a brief explanation.

Whenever you have an integration route that must be running, at any point in time, in at most one single Camel instance, you need to use a master route. Master routes can be declared by simply prefixing the consumer endpoint by the 'master' keyword and a name that will be used to create a named lock, e.g.

from('master:mylock:telegram:bots')
.to('log:info')

It can be used to print all messages that are sent to your Telegram bot. Since the Telegram API support a single consumer only, you can guard the route with a master prefix to have the guarantee that there will be at most only one consumer at any given time.

If you're wondering how there can be two instances running of you deploy one, well, think just to when you change your code and need to do a rolling update: for some time there'll be two pods running in parallel. In some cases, you may decide to scale your service out but keep only one instance of a particular route among all the pods of your service. Or you may want to embed a master route in a Knative autoscaling service: in this case, the service can scale autonomously based on the load, but there'll be only one telegram consumer at any time.

Master routes work out of the box in Camel K, you just need to put a prefix in your endpoint uri. A leader election protocol based on Kubernetes APIs resource locks will be automatically configured for you!

CronJobs

All complex enough systems contain several scheduled jobs. This is especially true for that part of the system that handles integration with the outside.

Ideally, if you need to execute a quick periodic task, say, every two seconds, you would startup an integration with a route based on timer to execute the periodic task. E.g.

from("timer:task?period=2000")
.to(this, "businessLogic")

But if the period between two executions, instead of 2 seconds ("2000" in the Camel URI, which is measured in milliseconds) is 2 minutes ("120000") or 2 hours ("7200000")?

You can see that keeping a container with a JVM running for a task that should be executed once every two minutes may be overkill (it is overkill for sure when the period is 2 hours). We live in a time where resources such as memory and CPU are really valuable.

So the Camel K operator automatically handles this situation by deploying your integration not as a Kubernetes deployment, but as a Kubernetes CronJob. This saves a lot of resources, especially when the period between executions is high. When it's time to run your integration code, a container starts, triggers the execution and then gracefully terminates. Everything is handled automatically by Camel K and Kubernetes.

There are cases when you don't want this feature to be enabled, for example, when your code makes use of in memory caches that is better to keep between executions. In these cases, you can safely turn off the feature by passing the flag `-t cron.enabled=false` to the `kamel run` command.

The Cron feature does not only work with the `timer` component. We've also added a cron component since Camel 3.1 that works really well in combination with the cron trait.

So you can also write the cron expression in the route directly:

from("cron:job?schedule=0/5+*+*+*+?")
.to(this, "businessLogic")

In this case, a new pod with a JVM is started every 5 minutes to execute your scheduled task. For the remaining 4+ minutes you don't use any resource.

Transparency

Camel K does a lot of work for you when you run your integration code in the cluster and it's possible that you put some errors in the code that can block the deployment process. We've added a lot of visibility on the deployment process that now communicates with the users via Kubernetes events that are printed to the console when you use the CLI.

This way you're always notified of problems in the code and you can better understand what to fix to make your integration run.

How to try Camel K 1.0

The first step is to go to the Apache Camel K release page, download the kamel CLI for your OS and put it in your system path.

Installation is done usually using the `kamel install` command, but, depending on the kind of Kubernetes cluster you're using, you may need to execute additional configuration steps.
The Camel K documentation contains a section about installing it on various types of Kubernetes clusters.

If you have trouble or you need to install it on a particular cluster that is not listed, just reach out in the Gitter chat and we'll do our best to help you.

Future

We've reached version 1.0.0 and this is a great milestone for us. But we are not going to stop now: we've big plans for the future and we'll continue to develop awesome new features.

We need your help to improve Camel K and we love contributions!

Join us on:

- Gitter: https://gitter.im/apache/camel-k
- GitHub: https://github.com/apache/camel-k

Tech Talk tomorrow (friday) - What's new with Apache Camel 3?

2020-06-04T10:16:00.001+02:00

This blog is with short noice. Tomorrow on Friday June 5th there will be a DevNation Tech Talk with Andrea Cosentino and myself presenting Apache Camel 3.

Date: June 5, 2020
Time: 13:00 UTC / 15:00 CET / 9:00 AM EDT

Apache Camel is a leading open source integration framework that has been around for more than a decade.

With the release of Apache Camel 3, the Camel family has been extended to include a full range of projects that are tailored to popular platforms including Spring Boot, Quarkus, Kafka, Kubernetes, and others; creating an ecosystem.

Join this webinar to learn what’s new in Camel 3 and about Camel projects:

Latest features in Camel 3
Quick demos of Camel 3, Camel Quarkus, Camel K, and Camel Kafka Connector
Present insights into what's coming next

Registration required at following link.

Free book on Knative covering Camel K and Kafka and upcoming webinar with live demos

2020-04-24T09:38:00.000+02:00

I want to say my congratulations to two of my fellow red hatters, Burr Sutter & Kamesh Sampath whom have published a new book on Knative - Knative Cookbook.

Blog post announcement: book launch.

The book is around 150 pages and is a cookbook style, so it's a great learner book and with step by step instructions to try out first hand.

The book has a chapter on Apache Camel K which is fantastic. They show you how to get started with Camel K and then continue to add Knative into the mix and how Camel K easily adapts to Knative being present in the platform.

Kamesh will show all the greatness of Knative together with Kafka and Kamel, in his upcoming webinar: 4K Kubernetes with Knative, Kafka, and Kamel scheduled at April 30th. So this is a great opportunity to hear first hand from the author, and see live demos, and just relax with a cup of coffee/tea or maybe even a cold beer. I surely will do that, maybe all 3 ... or skip the tea ;)

So I suggest to go download the free book, and register for the webinar.

How to quickly run 100 Camels with Apache Camel, Quarkus and GraalVM

2020-04-14T14:41:00.002+02:00

Today I continue me practice on youtube and recorded a 10 minute video on creating a new Camel and Quarkus project that includes Rest and HTTP services with health checks and metrics out of the box.

Then comparing the memory usage of running the example in JVM mode vs native compiled with GraalVM. Then showing for the finale how to quickly run 100 instances of the example each on their own TCP port and how quick Camel are to startup and service the first requests faster than you can type and click.

For this demo I am using Java 11, Apache Camel 3.2.0, Quarkus 1.3.2 and GaalVM CE 20.0.0. You can find the source code for the example at camel-quarkus github with instructions how to try for yourself.

We are working on reducing the binary image size for Camel 3.3, by eliminating more classes that GraalVM includes that are not necessary. And we also have an experiment with an alternative lightweight CamelContext that are non dynamic at runtime which can improve this further. And then GraalVM and Quarkus will of course also keep innovative and make it smaller and faster.

Best Practices for Middleware and Integration Architecture Modernization with Apache Camel

2020-03-25T09:36:00.000+01:00

Yesterday I gave the following virtual talk at the Stockholm Meetup.

Best Practices for Middleware and Integration Architecture Modernization with Apache Camel

What are important considerations when modernizing middleware and moving towards serverless and/or cloud native integration architectures? How can we make the most of flexible technologies such as Camel K, Kafka, Quarkus and OpenShift. Claus is working as project lead on Apache Camel and has extensive experience from open source product development.

I thank the organizers Forefront Consulting for inviting me. Unfortunately there was a glitch with the talk yesterday. As I could not be in person then the talk was pre-recorded and was cut half way. So I promised to post a blog today and upload the talk to youtube.

The title and abstract of the talk was somewhat given to me, and so was the session length of 30 minutes. As I am so heavily invested in Apache Camel, then I focused the talk about Camel and its evolution over the last 10 years as introduction and then using the latest innovations from Camel K, Camel Quarkus and Camel Kafka Connectors as the meat of the talk, and with 3 demos.

The talk can be watched on youtube and the slides are here.

Apache Camel 3.1 - Fast loading of XML routes

2020-03-22T18:45:00.001+01:00

A feature that was added to Camel 3.1 is the ability to load XML routes much faster. This is part of the overall work we are doing on making Camel much smaller and faster.

You may say ewww XML. But frankly there are many users of Camel that have built applications with XML for defining routes. In Camel 2.x then you would have to use Spring or OSGi Blueprint for XML routes which both are becoming heavy in modern cloud native world.

In Camel 3 we have a standalone mode for Camel called camel-main. We use camel-main as a common way to bootstrap and configure Camel for standalone, camel-k, camel-quarkus, and for most parts of camel-spring-boot as well. This ensures an unified and consistent developer experience across those runtimes.

Okay this is probably a topic for another blog post to dive into camel-main as a great runtime for quickly running ... just Camel.

So what I wanted to say in this blog post is that we have made it possible to loading XML routes much quicker and with a lot less overhead. In Camel 2.x, and for Spring XML and Blueprint XML they rely on JAXP and JAXB which ... are heavy.

So what we have done for Camel 3.1 is to source code generate a XML parser based on the Camel DSL. This means anything we do changes to the DSL then the parser is re-generated. The parser just uses standard Java so there are no additional 3rd party library dependencies.

For loading XML routes in Camel we now have 2 parsers in the following JARs

- camel-xml-jaxb (traditional JAXB based as in Camel 2.x)
- camel-xml-io (new fast and lightweight source code generated parsers)

The example camel-example-main-xml is setup to use the new parser. But you can try for yourself and switch to the jaxb parser by changing the JAR dependency.

Lets see some numbers (note this is just a quick test on my laptop to run this example with the 2 XML parsers).

camel-xml-jaxb: Loaded 1 (808 millis) additional Camel XML routes from: routes/*.xml
camel-xml-io: Loaded 1 (76 millis) additional Camel XML routes from: routes/*.xml

So the new parser is about 10 times faster (76 vs 808 millis).

By profiling the JVM we can see that there is a lot less classes loaded as well: 4734 vs 3892. And on top of that JAXB leaves more objects and classes around in the JVM that may or may not easily be garbage collected, and would also be using more cpu and memory during its parsing.

And then on GraalVM then the new parser would be much quicker as you can avoid having the entire JAXB and JAXP API and implementation on the classpath and for the GraalVM compiler to crunch and compile. And speaking of GraalVM then we are working on some great improvements in the upcoming Camel 3.2 that should help reduce the image size and compilation, and allow to do more dead code elimination and whatnot to make Camel even more awesome. That's yet another topic for another blog post, so stay tuned.

Apache Camel 3.2 - Reflection free configuration of Camel

2020-03-22T10:37:00.003+01:00

At the Apache Camel project we are working towards the next upcoming Apache Camel 3.2.0 release, which is planned for next month.

One of the ares we have worked hard on in Camel 3 is to make it smaller and faster. And one aspect of this is also configuration management. You can fully configure Camel in many ways and according to the 12 factor principles, to keep configuration separated from the application. A popular way to configure is to use properties files (eg application.properties) or in Kubernetes you can configure from config maps or environment variables as well.

So we have gradually over Camel 3.0, 3.1 and now 3.2 made configuration faster. With the latest work we are now fully reflection free.

Camel is capable of reporting when reflection based configuration are being used. Which can be configured with:

# bean introspection to log reflection based configuration
camel.main.beanIntrospectionExtendedStatistics=true
camel.main.beanIntrospectionLoggingLevel=INFO

We have prepared the camel-example-main-tiny to report this. The numbers for Camel 3.0, 3.1, and 3.2 are as follows:

Camel 3.0: BeanIntrospection invoked: 12 times
Camel 3.1: Stopping BeanIntrospection which was invoked: 11 times
Camel 3.2: Stopping BeanIntrospection which was invoked: 0 times

What this means is that you can fully configure all your Camel endpoints, components, routes, EIPs, data formats, languages, camel main, camel context, and whatnot, in declarative properties files etc and then at runtime all of this ends up invoking the actual setter methods on all these instances (ie just direct java method calls, no java.lang.reflect).

This is possible because we source code generate configurer classes based on what options are present. And these configurer classes are reflection free. There can be many options so it would be impossible to implement this by hand, see for example the kafka endpoint configurer.

And btw another feature coming in Camel 3.2 is that we made all of the components options available for configuration, before we didn't include nested configuration options. And if you do not like configuring in properties files, then we have type-safe component-dsl and endpoint-dsl as well.

Upcoming Webinar - What's new in Apache Camel 3

2020-02-27T11:17:00.000+01:00

On March 3rd, at 3pm CET (Central European Timezone) I will co-host, together with Andrea Cosentino, a webinar session for 1 hour, where we cover all the great new features that are in the Apache Camel v3 release.

Andrea and I will cover in more details the high level goals of Apache Camel 3, and focus on the key elements about making Camel smaller, lighter, and faster for the cloud native world. So you will find details about what we have done internally to make this happen.

We also cover and introduce Camel K, Camel Quarkus and Camel Kafka Connector where we have 4 live demos ready for you. And finally we present the roadmap for the upcoming releases.

At the end we have Q&A session where we will assist and answer as many questions you may have.

The webinar is free to attend, but requires registration, as the webinar is run by professional media company.

More details and registration here.

PS: Yes we will cover details up till the latest Camel 3.1 release which is going to be released today ;)

Apache Camel 3.1 - More camel-core optimizations coming (Part 3)

2020-02-12T15:05:00.004+01:00

I have previously blogged about the optimizations we are doing in the next Camel 3.1 release

Today I wanted to give a short update on the latest development we have done, as we are closing down on being ready to build and release Camel 3.1 as early as end of this week or the following.

Since part 2, we managed to find additional 10% reduction on object allocations during routing.

We have also continued the effort of configuring Camel via source code generated configurers that performs direct Java method calls vs using java bean reflections. Now all components, data formats, languages, and EIP patterns is complete. Only in more advanced use-cases where configuration is based on nested complex objects that are dynamically configured would be outside the scope of the source code configures and Camel fallback to use reflection.

We also found a way to optimize property placeholder resolution on EIPs to avoid using source code generated configurers which means that there are 200 classes less to load on the classpath, and about 90kb of memory is saved. This is great as these classes and memory were only used during bootstrap of Camel, and now they are all gone.

We also managed to further modulaize camel-core, so JAXB and XML routes are optional.
Even for XML routes (not Spring or Blueprint as they have their own DOM XML parser) we have created an alternative, fast and light-weight pull based parser. The camel-example-main-xml is using this and by comparing JAXB vs Camel XML then its 6x faster (approx 1500 millis vs 250) and loads 700 classes less than JAXB.

However for non XML users (eg using Java DSL) then JAXB can be avoided on the classpath at all, and you can have tiny Camel applications, such as camel-example-main-tiny with the following dependency tree (bold are Camel JARs; the example uses the bean and timer components)

[INFO] org.apache.camel.example:camel-example-main-tiny:jar:3.1.0-SNAPSHOT
[INFO] +- org.apache.camel:camel-main:jar:3.1.0-SNAPSHOT:compile
[INFO] | +- org.apache.camel:camel-api:jar:3.1.0-SNAPSHOT:compile
[INFO] | +- org.apache.camel:camel-base:jar:3.1.0-SNAPSHOT:compile
[INFO] | +- org.apache.camel:camel-core-engine:jar:3.1.0-SNAPSHOT:compile
[INFO] | +- org.apache.camel:camel-management-api:jar:3.1.0-SNAPSHOT:compile
[INFO] | +- org.apache.camel:camel-support:jar:3.1.0-SNAPSHOT:compile
[INFO] | \- org.apache.camel:camel-util:jar:3.1.0-SNAPSHOT:compile
[INFO] +- org.apache.camel:camel-bean:jar:3.1.0-SNAPSHOT:compile
[INFO] +- org.apache.camel:camel-timer:jar:3.1.0-SNAPSHOT:compile
[INFO] +- org.apache.logging.log4j:log4j-api:jar:2.13.0:compile
[INFO] +- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] \- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] \- org.slf4j:slf4j-api:jar:1.7.30:compile

I ran this example with the profiler and configured it to use 10MB as max heap (-Xmx10M) and as the summary shows this can easily be done. About 5mb is used in the heap.

There has also been a few other minor improvements to avoid using Camel 2.x based type converter scanning by default. This reduces a scan on the classpath.

Okay its time to end this blog series and finish up the last bits so we can get Camel 3.1 released.

Apache Camel 3.1 - More camel-core optimizations coming (Part 2)

2020-01-30T12:34:00.000+01:00

I have previously blogged about the optimizations we are doing in the next Camel 3.1 release (part 1).

Today I wanted to post a status update on the progress we have made since, about 4 weeks later.

We have focused on optimizing camel-core in three areas:

unnecessary object allocations
unnecessary method calls
improve performance

In other words we are making Camel create less objects, calling fewer methods, and improving the performance during routing.

To help identify these issues in camel-core we were using a simple Camel route:

from timer:foo

to log:foo

And other times we focused on longer routes:

from timer:foo

to log:foo1

to log:foo2

to log:foo3

...

to log:fooN

Or the focus on the bean component:

from timer:foo

to bean:foo

And so on. We also added an option to the timer component to not include metadata so the message dont contain any body, headers or exchange properties. This allowed us to focus on the pure routing engine and its overhead.

So all together this has helped identify many smaller points for improvements that collectively gains a great win.

tl:dr - Show me the numbers

Okay let's post some numbers first and then follow up with details what has been done.

Object Allocations - (5 minute sampling)

Camel 2.25 2.9 M objects created

Camel 3.0 55 M objects created

Camel 3.1 1.8 M objects created

Okay we have to admit that Camel 3.0 has an issue with excessive object allocations during routing. There are no memory leaks but it creates a lot of unnecessary objects. And I will get into details below why.

However what is interesting is the gain between Camel 2.25 and 3.1 (40% less objects created).

Method Calls - (5 minute sampling)

Camel 2.25 139 different Camel methods in use

Camel 3.0 167 different Camel methods in use

Camel 3.1 84 different Camel methods in use

The table above lists the number of methods from Camel that Camel calls during routing. The data does not include all the methods from the JDK. As we cannot optimize those, but we can optimize the Camel source code.

As you can see from the table we have improvement. Camel 3.1 uses less than half of 3.0, and 40% less than Camel 2.2.5.

Camel 3.0

Okay so Camel 3.0 has a problem with using too much memory. A big reason is the new reactive executor which now executes each step in the routing via event looping, by handing over tasks to a queue and having workers that execute the tasks. So this handoff now requires creating additional objects and storing tasks in queue etc.

Some of the biggest wins was to avoid creating TRACE logging message which unfortunately was always created regardless if TRACE logging level was enabled. Another big win was to avoid creating toString representation of the route processes with child elements. Instead Camel now only output the id of the process which is a fast operation and dont allocate new objects.

Another problem was new code that are using java.util.stream. This is both a blessing and a curse (mostly a curse for fast code). So by using plain for loops, if structures, and avoiding java.util.stream in the critical parts of core routing engine we reduces object allocations.

Camel 3 is also highly modularised, and for example in Camel 2.x we had all classes in the same classpath and could use instanceof checks. So in Camel 3 we had some code that performed poorly doing these kind of checks (java util streams again).

Another problem was the reactive executor which was using a LinkedList as its queue. So if you have tasks going into the queue and workers processing them in the same pace, so the queue is empty/drained, then LinkedList performs poorly as it allocates/deallocates the object constantly. By switching to a ArrayQueue which has a pre-allocated size of 16 then there is always room in the queue for tasks and no allocation/deallocation happens.

There are many more optimisations but those mentioned above where likely the biggest problems. Then a lot of smaller optimisations gained a lot combined.

Many smaller optimisations

The UUID generator of Camel is using a bit of string concat which costs. We have reduced the need for generating UUIDs in the message and unit of work so we only generate 1 per exchange.

The internal advices in the Camel routing engine (advice = before/after AOP). Some of these advices has state which they need to carry over from before to after, which means an object needs to be stored. Before we allocated an array for all advices even for those whom do not have state and thus storing a null. Now we only allocate the array with the exact number of advices that has state. (very small win, eg object[6] vs object[2] etc, but this happens per step in the Camel route, so it all adds up.). Another win was to avoid doing an AOP around UnitOfWork if it was not necessary from the internal routing processor. This avoids additional method calls and to allocate a callback object for the after task. As all of this happens for each step in the routing then its a good improvement.

Some of the most used EIPs has been optimized. For example allows you to send the message to an endpoint using a different MEP (but this is rarely used). Now the EIP detects this and avoids creating a callback object for restoring the MEP. The pipeline EIP (eg when you do to -> to -> to) also has a little improvement to use an index counter instead of java.util.Iterator, as the latter allocates an extra object

Camel also has a StopWatch that used a java.util.Date to store the time. This was optimized to use a long value.

Another improvement is the event notification. We now pre-calculate if its in use and avoid calling it all together for events related to routing messages. BTW in Camel 3.0 the event notifier was refactored to use Java 8 Supplier's and many fancy APIs but all of that created a lot of overhead. In Camel 3.1 we have restored the notifier to be like before in Camel 2.x and with additional optimisations.

So let me end this blog by saying that .... awesome. Camel 3.1 will use less memory, execute faster by not calling as many methods (mind that we may have had to move some code which was required to be called but doing this in a different way to avoid calling too many methods).

One of the bigger changes in terms of touched source code was to switch from using an instance based logger in ServiceSupport (base class for many things in Camel), to use a static logger instance. This means that there will be less Logger objects created and it's also better practice.

Better performance

Other improvements is that we have moved some of the internal state that Camel kept as exchange properties to fields on the Exchange directly. This avoids storing a key/value in the properties map, but we can use primitives like boolean, int etc. This also performs better as its faster to get a boolean via a getter than to lookup the value in a Map via a key.

In fact in Camel 3.1 then during regular routing then Camel doesnt lookup any such state from exchange properties which means there is no method calls. There are still some state that are stored as exchange properties (some of those may be improved in the future, however most of these states are only used infrequently). What we have optimized is the state that are always checked and used during routing.

Exchange getProperty(5 minute sampling)

Camel 2.25 572598 getPropety(String)

Camel 2.25 161502 getPropety(String, Object)

Camel 2.25 161502 getPropety(String, Object, Class)

Camel 2.25 141962 getPropeties()

Camel 3.0 574944 getProperty(String)

Camel 3.0 167904 getPropety(String, Object)

Camel 3.0 167904 getPropety(String, Object, Class)

Camel 3.0 91584 getPropeties()

Camel 3.1 0 getProperty(String)

Camel 3.1 0 getPropety(String, Object)

Camel 3.1 0 getPropety(String, Object, Class)

Camel 3.1 0 getPropeties()

As you can see Camel 2.25 and 3.0 lookup this state a lot. And in Camel 3.1 we have optimized this tremendously and there are no lookup at all - as said the state is stored on the Exchange as primitive types which the JDK can inline and execute really fast.

The screenshot below shows Camel 2.25 vs 3.1. (The screenshot for 3.1 is slightly outdated as it was from yesterday and we have optimised Camel since). See screenshot below:

Okay there are many other smaller optimizations and I am working on one currently as I write this blog. Okay let me end this blog, and save details for part 3.

Apache Camel 3.1 - More camel-core optimizations coming

2020-01-03T20:27:00.003+01:00

Hope all is good and you had a safe entry into 2020.

The Camel team is already busy working on the next Camel 3.1 version. One of the goals is to continue optimize camel-core, and this time we have had some time to look into finding some hot spots in the routing engine.

One of the aspects we have looked at is also the object allocations that occurs per message that Camel routes. The JVM itself is great at allocation objects and garbage collecting them when they are no longer in use. However there are room for improvements if you can identify a number of objects that is unnecessary per EIP in the route.

So today I found several of these by just running a basic Camel route that is

from(timer:foo?period=1")

.to("log:foo");

Which basically routes 1000 messages per second. And prints each message to the log.

One of the bigger culprits in object allocations turned out to be human logging for the reactive executor which logs at TRACE level. So by avoiding this we can reduce a great deal of allocations, and string building for logging messages.

Other aspects we have optimised is the to EIP (the most used EIP) which is now smarter in its startup to avoid creating caches that was not necessary. And this goes together with areas where we now lazy creates some features in Camel that were very rarely in use that would otherwise also setup and create some caches.

We also identified as part of the Camel 3 work, then the LRUCache was not pre warmed up as early as before, which meant Camel would startup a bit slower than it otherwise are capable of. So by moving this warmup to an earlier phase then Camel can startup faster by doing concurrent work on startup until the LRUCache is warmed up (its caffeine cache that needs this).

The log component has also been optimised to reduce its object allocations which building the logging message.

So all together a great day and if we compare startup up a Camel 3.0.0 vs 3.1.0-SNAPSHOT with the Camel route as shown above, then we have an awesome reducing in object allocations per second (thanks to YourKit for profiler).

The profile says that in Camel 3.0.0 then Camel would roughly generate about 22.000 objects per second (routing 1000 messages). And that has been reduced to about 6.000 objects per second in Camel 3.1. That is fantastic, and is almost a 4x reduction.

Apache Camel 3 - Whats New Top 10

2019-12-04T12:01:00.002+01:00

Apache Camel 3 was released last thursday November 28th 2019, which also happens to be the day of the US Thanksgiving. This was not intentionally but we can say its a big thanks from us to the community with a brand new major version of Camel - this does not come by often. In fact it's 10 years since Camel 2 hit the streets. So this 3rd generation is long overdue.

This blog post highlights the noteworthy new features and improvements in Camel v3.

1) CAMEL IS NOW A FAMILY OF PROJECTS
Apache Camel, is now a family of projects (3 at this time of writing):

Camel 3: Integration Framework Swiss knife of integration
Camel K: Lightweight Serverless Integration Platform Camel on Kubernetes & Knative
Camel Quarkus: Camel extensions for Quarkus Optimised JVM & Native compiled Java (GraalVM)

The Camel code-base is very large, and we have setup sub-projects for new innovative projects using Camel. The first sub-project was to run Camel as cloud-native on Kubernetes in a serverless manner which became Camel K. Then Camel Quarkus came to make Java and Camel with very fast startup and very small memory footprint primary for container based deployments.

2) NEW WEBSITE
A major goal for Camel 3 was to finally revamp the old aging website to use modern technologies and be able to auto-generate content from the source code. This has taken years to get to this point as we have built tools over the last many Camel 2.x releases that could take us closer. At end of 2019 then the Camel community and others stepped up and provided the new art-work, logo, and look and feel for the new website - thank you very much!.

For Camel 3.x we will continue to improve the website and the documentation. This is much easier for us to do, and also for people to contribute changes as its just a regular github PR to provide updates. We love contributions.

Zoran had some fun with the new look and feel and he added a little gem; if you stare at the front page, then you should see a little animation of the curved bezel ;)

3) JAVA 11
Camel 3 is the first official release that supports Java 11. Java 8 will still be supported for the first number of 3.x releases, but is expected to be dropped later in 2020. However we wanted to provide Java 8 support to help migrate Camel 2.x users whom may be restricted to Java 8 for some time to come.

4) MODULARIZED CAMEL-CORE
The camel-core has been modularized from 1 JAR to 33 JARs. The core functionality has been splitup into:

camel-api
camel-base
camel-caffeine-lrucache
camel-cloud
camel-core
camel-core-engine
camel-core-osgi
camel-core-xml
camel-endpointdsl
camel-headersmap
camel-jaxp
camel-main
camel-management-api
camel-management-impl
camel-support
camel-util
camel-util-json

For Camel end users then only a few JARs is relevant.

camel-api contains the public API for Camel (eg interfaces such as CamelContext, Endpoint, Exchange, Message, and so on).

camel-support contains the base classes and RouteBuilder which you would use to build Camel routes and applications. This JAR is also contains necessary base classes for building custom Camel components, and other kinds of plugins.

The components that resided in camel-core has also be externalized into individual components:

camel-bean
camel-log
camel-stub
camel-browse
camel-mock
camel-timer
camel-controlbus
camel-properties
camel-validator
camel-dataformat
camel-ref
camel-vm
camel-direct
camel-rest
camel-xpath
camel-directvm
camel-saga
camel-xslt
camel-file
camel-scheduler
camel-zip-deflater
camel-language
camel-seda

Camel end users can then pick and choose exactly only what they need, or keep using everything.

Therefore we have camel-core and camel-core-engine as two starting dependencies. You can use camel-core which gives you all the JARs which is similar to Camel 2.x. When you use camel-core-engine you get the minimum set of JARs that makes a functional Camel.

camel-core contains 33 JARs and camel-core-engine contains 12 JARs.

5) FASTER STARTUP AND LOWER FOOTPRINT
We have reduced the size of core Camel and the number of classes loaded. For example in Camel 2 about 5200 classes was loaded, which has been reduced to about 4300 loaded classes in Camel 3.

We have also done many smaller optimizations in the core, to reduce the number of allocated Java objects, and speeup initialization and other means. We have used JVM profiling tools to assist and find the bottlenecks.

Another area of improvement is to reduce Java reflections. In Camel 2 then all the configuration of Camel components, endpoints, and routes are reflection based. In Camel 3 we have source code generated Java code for configuration that allows us to use direct Java calls instead of reflections.

Another similar area is Camel’s type converters which in Camel 2 are Java reflection based (you could build custom type converts that were not reflection based). In Camel 3 we also generate Java source code which means that type converting is direct Java calls at runtime.

We have also moved initialization logic to earlier phases when it was possible. For example there is a new build phase which allows Camel to do special initialization during building your project (this requires Camel Quarkus).

All this optimization improves the startup performance of Camel and reduces the memory overhead. With Camel Quarkus you can natively compile your Camel application and make it startup in 30 milli seconds and consume only 10mb of memory (RSS) with a full blown HTTP REST server and health-checks and metrics.

There are still a few items on the agenda that we want to work on in Camel 3.x to further optimize Camel core.

6) TYPE SAFE ENDPOINT DSL
Camel end users whom has configured endpoints using URI strings, would all have experienced the problem when you make a configuration mistake in the endpoint, which then makes Camel fail on startup.

In Camel 3, we have a new type-safe DSL for endpoints which you can use in Java routes. You can continue to use the classic URI strings, but if you want to try the endpoint DSL, then you need to add camel-endpointdsl to your classpath. Then you should extend EndpointRouteBuilder instead of RouteBuilder to access the endpoint DSL.

Here is a basic example without and with the endpoint DSL:

from("timer:click?period=3000&fixedRate=true")
.to("seda:foo?blockWhenFull=true");

from(timer("click").period(3000).fixedRate(true))
.to(seda("foo").blockWhenFull(true));

You can also find a little example in the source code.

7) REACTIVE ROUTING ENGINE
The routing engine in Camel has internally been reactive’fied and all EIPs has been retrofitted to work in a reactive manner. However this is internal only, and the Camel API for both end users and component developers are based on existing callback behavior.

We will later introduce and work on a client-side facing reactive API after we have jumped to Java 11 as minimum version (then we can support Java 9 flowable API).

Camel already have integration with reactive frameworks such as Vert.X, RxJava and Reactor Core in the dedicated Camel components.

8) CAMEL MAIN
We have introduced camel-main as a standalone JAR that makes it easier to run just Camel. There are a couple of examples with the source code that demonstrates how to do that.

We also use camel-main to have common code to configure and bootstrap Camel for standalone, Spring Boot, Camel K, and Camel Quarkus. This allows us to share the same code, and configuration options.

9) CAMEL MICROPROFILE
Camel 3 now integrates better with Eclipse Microprofile and we have Camel components for Microprofile configuration, metrics, health checks, and fault tolerance (on the way).

More components to come in upcoming Camel releases. These microprofile components are also used by Camel Quarkus.

10) MISCELLANEOUS IMPROVEMENTS
Camel 3 now supports JUnit 5 for unit tests, with the test components that have -junit5 as suffix.

The Camel Registry is now also writeable, so you can add beans to the registry at runtime, or from unit tests etc.

You can also configure endpoints (producer) to lazy start. By default Camel works in a fail-fast mode, which means that Camel components that fails to connect to external systems during startup may cause the route to fail on startup. For Camel 3 you can now configure these endpoints to lazy start, which means the route will startup and they will first fail when a message is routed to the endpoint.

Camel also allows to configure your routes to be supervised during startup, which allows Camel to more intelligently start routes in a more safe manner, by restarting routes that failed.

11) MIGRATING TO CAMEL 3
We have of course cleaned up the code base, such as removing all deprecated APIs and components. We have also adjusted some APIs to make them easier to use from end users, and more Java 8 lambda friendly.

Internally we have also adjusted the route model, to make it easier to extend into new DSLs; and there is a YAML DSL on the way which was initiated in Camel K.

In terms of backwards compatibility then Camel 3 is mostly compatibility for regular Camel applications. However if you are using some of the more advanced features and other plugins in Camel then migration is needed. Also custom components must be migrated and recompiled. There are other adjustments such as Spring Boot users must use org.apache.camel.springboot as groupId instead of org.apache.camel etc. All details can be seen in the migration guide.

Good luck with your migration if you decide to continue your Camel journey. And for new users to Camel then good luck getting onboard.

12) BONUS: NEW COMPONENTS
There are 30 net new components in Camel 3, such as more stuff for Amazon AWS, and with GraphQL, and also worthwhile to mention is integration with Debezium, which is a change data capture project to grab change events from databases.

Upcoming Tech Event in Copenhagen - What does the system integration toolbox anno 2019 looks like

2019-10-07T11:10:00.002+02:00

On October 24th 2019, I am presenting at a full day tech event in Copenhagen.

My session is the first of the day, with a total of 90 minutes covering all the latest about Apache Camel 3, Camel K, and Camel Quarkus. I will have up to date slides, and demos ready for the talk.

The event also hosts other speakers such as Jeppe Cramon whom will talk about EDA and microservices. And I have been told that we should look forward to Syed Shaaf's session where he has a big tech demo with many moving parts.

In the afternoon you can choose to listen to breakout sessions or attend a full 2,5 hours hands-on workshop. The breakout sessions will among others cover use-cases with Apache Camel and Apache Kafka from the real world.

You can find more details about the event, the location, and how to register here.

Hope to see you there.

Cloud-native integration with Apache Camel on Kubernetes

Cloud-native applications of the future will consist of hybrid workloads: stateful applications, batch jobs, microservices, and functions, wrapped as Linux containers and deployed via Kubernetes on any cloud.

In this session, we will explore key challenges with function interactions and coordination, addressing these problems using Enterprise Integration Patterns (EIP) and modern approaches with the latest innovations from the Apache Camel community:

Apache Camel 3
Camel K
Camel Quarkus

Apache Camel is the Swiss army knife of integration, and the most powerful integration framework. In this session you will hear about the latest features in the brand new 3rd generation.

Camel K, is a lightweight integration platform that enables Enterprise Integration Patterns to be used natively on any Kubernetes cluster. When used in combination with Knative, a framework that adds serverless building blocks to Kubernetes, and the subatomic execution environment of Quarkus, Camel K can mix serverless features such as auto-scaling, scaling to zero, and event-based communication with the outstanding integration capabilities of Apache Camel.

We will show how Camel K works. We'll also use examples to demonstrate how Camel K makes it easier to connect to cloud services or enterprise applications using some of the 300 components that Camel provides.

Upcoming Webinar - Integration Patterns in a Serverless World

2019-10-04T09:59:00.001+02:00

Next Thursday, October 10th, I am giving a 35 minutes webinar about Serverless Integration (yeah it covers Apache Camel, Camel K, Knative and Quarkus).

The webinar is part of a full day event (virtual event) where there are keynotes and a parallel tracks with 12 breakout sessions.

My talk is focused purely on the upstream work we do at Apache Camel with the next generation Camel 3.

The abstract for my talk

Cloud-native applications of the future will consist of hybrid workloads: stateful applications, batch jobs, microservices, and functions, wrapped as Linux containers and deployed via Kubernetes on any cloud.

In this session, we'll explore the key challenges with function interactions and coordination, addressing these problems using classic integration patterns and modern approaches with the latest innovation from the Apache Camel community: Camel K, a lightweight integration platform that enables enterprise integration patterns to be used natively on any Kubernetes cluster.

When used in combination with Knative, a framework that adds serverless building blocks to Kubernetes, and the subatomic execution environment of Quarkus, Camel K can mix serverless features such as auto-scaling, scaling to zero, and event-based communication with the outstanding integration capabilities of Apache Camel.

We will show how Camel K works. We'll also use examples to demonstrate how Camel K makes it easier to connect cloud services or enterprise applications using some of the 250+ components that Camel provides.

Registration

The event is free to attend, and you can signup using this link.

Date: October 10th 2019
Time: 5 pm CEST (event starts with keynotes)

Apache Camel 3 is only 2 months away

2019-10-02T10:34:00.003+02:00

The Camel team is busy working on the last set of work for Apache Camel 3. Today the 2nd release candidate was built and published on a staging repository for early adapters to give it a try.

As I am busy myself then I just wanted to write a short blog post to keep the community posted that Apache Camel 3 is on the way, and that we expect it to be released by end of this year (sometime in November or December).

The latest deadline is to release Camel 3.0 before December 19th 2019 as it was exactly on that day 1 year ago, we switched the master branch to become the work branch for Apache Camel 3. That means the total development time for Camel 3 would be 1 year.

Here is a illustration that highlights the timeline of Camel 3:

That is not all Apache Camel, is now a family of 3 projects (at this moment). So working on Camel 3 is not all we do. Camel K and Camel Quarkus are very promising for cloud-native integration (microservices and serverless).

These projects have their own lifecycle. Will will post more details about these projects, and what's new in Camel 3 in the following months leading up to the final release of Camel 3. So stay tuned.

PS: If you are migrating Camel 2.x applications to Camel 3, then read the migration guide.

New website for Apache Camel

2019-08-21T06:20:00.001+02:00

Yesterday we published the new website for Apache Camel.

It's really fantastic to finally see a new website. And now with a modern tech for the website we can much easier update and keep it more modern for the future, and also it should help encourage users to contribute to its content to help build a better user guide.

Huge thanks to all involved.