Angle of Sight

MongoWave: Persistence on Google FedOne Wave Server with mongoDB

Anthony — Wed, 03 Mar 2010 06:15:02 +0000

Purpose

My company, SESI, has been working on applying advanced collaboration and analytical techniques to complex-situations to aid decision-making. As part of that effort I have been quite involved with the Google Wave Protocol project. I wanted to share my preliminary success implementing persistence on FedOne with MongoDB. In my previous post on the topic of persistence I explored where you could add persistence. As I followed the recommendations of that post, it is probably a good primer for this entry. This post will be more focused on the mechanics of my implementation. I will discuss my current progress, why I chose mongoDB, my design goals, how I implemented the design in code, the hurdles I encountered, and my next steps.

Current Progress

I have tested persistence locally and through federation with the console client, QWaveClient, and wavesandbox.com. I can create multiple waves, add and remove participants, and host conversations between multiple wavesandbox, QWave, and console clients. When I restart my server the clients’ waves are loaded automatically. When I open a wave, the conversation is brought to the latest version (even if the client was not online when the last updates were made). I was pretty excited when the QWave client restored correctly, because I tested solely on the console client that is part of FedOne and had no knowledge of how QWave was implemented. Many thanks to Torben Weis for his excellent work on QWave (and not just because it worked with my persistence).

I have NOT constructed or run any JUnit tests. My persistence mechanisms are focused on storing information in the current implementation of FedOne and I have not expended any effort into correctly handling the commit-notice. That is to say I persist information when I receive a wavelet-update as FedOne does not currently have an algorithm in place to store pending deltas in a separate container from applied deltas that have been acknowledged by a commit-notice (see this post for details). This isn’t an attempt to blame FedOne, rather to say that is not where my focus has been. And as a side note, I’m hoping the Google team decides to do away with the notion of a separate commit-notice.

Why MongoDB

As defined on their website, MongoDB (from “humongous”) is a scalable, high-performance, open source, schema-free, document-oriented database. I admittedly like the sound and thought of ‘MongoWave’ as in huge wave, but that was not part of the decision to go in that direction. My primary reasons for using mongoDB were the flexibility of schema-free storage (no tables), a JSON sytle syntax for storing and retrieving information (converted to and stored in BSON), embedded map-reduce, auto-sharding for scalability, good documentation, easy setup, drivers in many languages (Java, C++, Ruby, Python, Perl), and an active expanding community. I do not want to start a holy war here or engage in the “why didn’t you go with or have you tried ?”. Suffice to say this is what I feel mongoDB offers me and that is why I am using it.

For more in-depth info on mongoDB, I recommend:

Interactive Online Shell Tutorial - 5 minute hands-on demonstration of mongoDB by Kyle Banker. I also recommend his other posts on mongoDB.
Case Study on Switching to mongoDB - Very insightful and well-written article on why a company went from MySQL to MongoDB and the issues involved in their migration by David Mytton.
Getting Started - mongoDB quick start reference.

Design Goals

My overarching goals in building persistence into the FedOne solution were

Minimal disruption to code base. FedOne is an open source project that is rapidly changing. I did not want my code to be littered throughout such that it became tedious to re-implement persistence anytime a new release was made available.
Flexibility to modify what information I persist and where I persist/restore that information. As FedOne matures, I expect there to be more information that needs to be persisted (e.g. - user management/access controls). I also expect there to be refactoring due to optimizations, updates to the protocol, and other progress in general. In consideration of this, I wanted to be able to easily modify the information I persist and update the location in code of my persistence mechanisms.
Loose coupling and clear separation of responsibilities. Just good design practices here.
Ability to swap out different implementations of storage solution. This applies at any level of the persistence package and is not solely related to the use of mongoDB.

Implementation

To achieve my design goals, I decided to incorporate the following tools and techniques:

Google Guice for dependency injection and Aspect Oriented Programming (AOP)
Seperation of code into AOPInterceptor, AOPImplementation, PersistenceManager, and WaveStore
MongoDB/Schema-free database

I have previosly written about Wave’s use of Guice. Another capability of Guice outside of dependency injection is the ability to provide annotations on methods that will act as interceptors. Whenever an annotated method is invoked, Guice will proxy the call and forward execution to a segment of code I have defined. I don’t want to go over the benefits of AOP in general, so I’ll just point to the wiki article and the Guice AOP site.

The reasons I used AOP was to avoid introducing a lot of cross-cutting concerns directly into the code base. With the FedOne code base changing rapidly I did not want to go through great effort restoring my persistence everytime a significant release was made. By using AOP I can just annotate the method and all the work is done in a separate set of classes defined by me. Lastly, I gain great flexibility using AOP. If I decide at some point that it is more efficient to save or restore information at another location, I simply move my annotation and adjust my code for the new set of parameters (which may not be any different since I am probably still storing the same information). In this way there is no refactoring of the main code base. FedOne will continue to function without regard to my update.

I provide an activity diagram below to show how the internals of my persistence solution interact. What’s important at a high-level is the way the components have been separated to allow for localized changes. I am using Guice for dependency injection, so everything is written to interfaces with the subclasses being applied in a separate Module class. This allows me to easily change out an implementation for another in the case of optimizations, changes in course, testing, etc. For example, my persistence modules will still invoke the WaveStore interface if I later decide to switch to Couchdb for a comparison. I simply change WaveStoreMongo to WaveStoreCouch in my Guice binding without the need to update any of my source code.

A breakdown of responsibilities of each component is as follows:

AOPIntercepter - invoked by Guice when an binded annotation is encountered. Responsible for interpreting annotation and forwarding to correct method of AOPImplementation. This module only has knowledge of the annotation and methods available on the AOPImplementation. It has no knowledge and takes no action with regards to Wave structures.
AOPImplementation - acts as a bridge between the method invocation passed from the interceptor and the PersistenceManager. It will extract parameters, invoke the method, and enact the correct method within the PersistenceManager (not necessarily in that order). It has no knowledge of how the information extracted is used and therefore no knowledge of the underlying WaveStore.
PersistenceManager - prepares the data passed in for storing or as a key for retrieving values from the WaveStore. It has no knowledge of its caller (AOPImplementation) and has no upward dependency. This means that I can easily choose not to use AOP and instead make direct invocations within code by calling the PersistenceManager. This class is responsible for understanding the wavelet data and its internal structures so that it can be stored efficiently.
WaveStore - acts as a store for serialized information. It has no knowledge of Wave structures, AOP, or the PersistenceManager. This keeps the interface simple and immune to changes in Wave structures. Any serialization that needs to be done prior to storing information is the responsibility of the PersistenceManager.

The following activity diagram illustrates a general interaction (click image to see in full-size):

The following is a dependency graph produced by using GraphViz with Guice. You will see the dependencies exist only on generic interfaces such that the implementations may be substituted at any level. Click on the image to see in full-size.

Challenges

I did experience a couple of challenges while implementing persistence. Recall that one of my goals was minimal disruption of the FedOne code base. The framework I established allows for this by and large, but there were still a couple of changes that needed to be made to the FedOne core.

If you read my previous post on the topic of persistence you will recall that I suggested much of the work of storing and restoring wavelet data could be accomplished in the WaveletContainerImpl class. Perhaps my largest hurdle in implementing persistence within my framework was intercepting the method annotations I placed in this class with Guice. Guice AOP has a restriction that states instances that support method interception must be created by Guice by an @Inject-annotated or no-argument constructor. Both the LocalWaveletConatinerImpl and RemoteWaveletContainerImpl (which both extend WaveletContainerImpl) are created by a Factory method in fedone.waveserver.WaveServerModule as:

[code]

bind(LocalWaveletContainer.Factory.class).to(LocalWaveletContainerFactory.class)

.in(Singleton.class);

private static class LocalWaveletContainerFactory implements LocalWaveletContainer.Factory {

@Override

public LocalWaveletContainer create(WaveletName waveletName) {

return new LocalWaveletContainerImpl(waveletName);

}

invokes

public LocalWaveletContainerImpl(WaveletName waveletName) {

super(waveletName);

}

bind(RemoteWaveletContainer.Factory.class).to(RemoteWaveletContainerFactory.class)

.in(Singleton.class);

private static class RemoteWaveletContainerFactory implements RemoteWaveletContainer.Factory {

@Override

public RemoteWaveletContainer create(WaveletName waveletName) {

return new RemoteWaveletContainerImpl(waveletName);

}

invokes

public RemoteWaveletContainerImpl(WaveletName waveletName) {

super(waveletName);

state = State.LOADING;

}

[/code]

Even though the Factory class is binded by Guice, you see that the actual LocalWaveletContainerImpl and RemoteWaveletContainerImpl are not created by Guice, thereby negating the possibility of AOP within these classes. After a little bit of research I realized that Guice does provide support for creating objects through Factory methods via its @AssistedInject construct. Using this method the code above became:

[code]

bind(LocalWaveletContainer.Factory.class).toProvider(FactoryProvider.newFactory(

LocalWaveletContainer.Factory.class, LocalWaveletContainerImpl.class))

.in(Singleton.class);

invokes

@Inject

public LocalWaveletContainerImpl(@Assisted WaveletName waveletName)

{

super(waveletName);

}

bind(RemoteWaveletContainer.Factory.class).toProvider(FactoryProvider.newFactory(

RemoteWaveletContainer.Factory.class, RemoteWaveletContainerImpl.class))

.in(Singleton.class);

invokes

@Inject

public RemoteWaveletContainerImpl(@Assisted WaveletName waveletName) {

super(waveletName);

state = State.LOADING;

}

[/code]

As you can see we reduced the amount of boilerplate code by removing the private Factory classes that contained the create() method. I did not have to modify the code that called the create method in WaveServerImpl [wc = localWaveletContainerFactory.create(waveletName);].

Of course, what is most important to me is that Guice now creates the RemoteWaveletContainerImpl and LocalWaveletContainerImpl classes thereby allowing me to intercept methods through AOP. Not only does Guice now allow me to intercept the local and remote classes, but I can also add my annotations to the super class WaveletContainerImpl where the actual commit and restore of wavelet deltas occurs! Yes, I think this is super cool and one reason I am a huge fan of Guice.

Another issue was deciding exactly what type of information to persist. I had to decide whether I wanted to store only the deltas or attempt also to store the substantive data (e.g. - WaveletData). I made the decision to only store processual information such that when I load wavelet deltas, I must then apply the underlying operations to seed my data containers. I am quite interested in knowing if Google also stores their data objects along with the deltas.

The last significant roadblock was determining all the sections of code that implicitly functioned without regard to persistence. For example, the fedone.waveserver.ClientFrontendImpl.PerWavelet and fedone.waveserver.UserManager are in-memory objects used by the ClientFrontendImpl to keep track of the index wave. When the server is restarted these objects are empty. One issue that arises from this is if a remote participant updates a wave hosted by our newly started wave server the delta will be applied and the callback ClientFrontendImpl.waveletUpdate() will be invoked. An IllegalStateException will now occur because the expected version that is pulled from the perWavelet object will have a version of 0 while the start version of the delta sequence will be non-zero. From this point the local participants will have a different version from the latest version and will never be able to update the wave again.

I don’t want to delve too far into this topic, but I have serious reservations about the way the index wave is handled, that there are no recovery mechanisms if versions are ever out of sync, and that in the current implementation of FedOne participants must specify an end version (even though the spec indicates it is optional) negating the possibility of restoring with the latest deltas.

Next Steps

My goal was to establish a flexible framework for persistence just as much as it was to actually have the ability to store and retrieve information in my wave server. I have many questions regarding the most efficient storage of wavelets and I do not claim that I have chosen the optimal places to store/retrieve information. My hope is that through the framework any modifications I need to make will be clearly defined and concise. For now, I want to do more testing and get feedback on from Google and the community on my strategy for persistence.

Google Wave’s Federation Protocol Under the Hood, Part 5 Purpose [Updated 4/3/2010] This is the fifth and final post...
Google Wave’s Federation Protocol Under the Hood, Part 3 Purpose [Updated 4/3/2010] This is the third post in...
Google Wave’s Federation Protocol Under the Hood, Part 4 Purpose [Updated 4/3/2010] This is the fourth post in...

Related posts brought to you by Yet Another Related Posts Plugin.

Google Wave’s Federation Protocol Under the Hood, Part 5

Anthony — Mon, 22 Feb 2010 19:42:29 +0000

Purpose [Updated 4/3/2010]

This is the fifth and final post in the series dedicated to introducing some of the internals of the FedOne protocol. As you probably realize, FedOne does not store wavelets to a persistent state. The goals of this post are to illustrate where the in-memory store takes place and highlight the information that should be captured in order to restore the state of a wave upon the restart of a server.

The documentation on the Fed protocol states that local AND remote wavelets are all stored in the wave server’s persistent wave store. In addition to persisting hosted and remote wavelets, the wave server must also maintain additional information related to its clients (covered later). I have provided the following graph to illustrate the structure of the in-memory wavelet storage containers:

The process of retrieving a wavelet container is initiated in fedone.waveserver.WaveServerImpl.getOrCreateLocalWavelet() and fedone.waveserver.WaveServerImpl.getOrCreateRemoteWavelet(). These methods call localWaveletContainerFactory.create(waveletName) and remoteWaveletContainerFactory.create(waveletName) respectively, which are defined by the fedone.waveserver.ServerModule() Guice Module. This module simply calls the constructors of the fedone.waveserver.LocalWaveletContainerImpl and fedone.waveserver.RemoteWaveletContainerImpl classes. Each of these constructors in turn calls the super constructor of the fedone.waveserver.WaveletContainerImpl class.

Most of the work of persistence can be narrowed down to the fedone.waveserver.WaveletContainerImpl class. Specifically, I suggest checking the persistent store as part of the constructor process, such that if the passed WaveletName is contained in the store, the applied and transformed deltas will be retrieved to seed the Wavelet Container. Wavelet operations can be commited to the store in the commitAppliedDelta() method. Updates in these sections will apply to the fedone.waveserver.LocalWaveletContainerImpl and fedone.waveserver.RemoteWaveletContainerImpl, which both call the fedone.waveserver.WaveletContainerImpl.commitAppliedDelta(). It will be necessary to update these two classes to only invoke the commit method when a commit-notice is received. It will then be necessary to maintain a set of pending deltas that correspond to wavelet updates that will be dequeued when the corresponding commit-notices are sent (note that this is one strategy of many possibilities).

It should be noted that in FedOne the RemoteWaveletContainerImpl maintains a set of pending deltas. This set is populated when it receives a waveletUpdate from a remote Host server. However, commitAppliedDelta() is invoked prior to receiving the the actual commit-notice and the newly applied delta is removed from the queue of pending deltas. When the commit-notice is received by wave.federation.xmpp.XmppFederationRemote.update() it is passed through to the client via fedone.waveserver.ClientFrontendImpl.waveletCommitted(). Obviously, commit-notice must be handled differently in production. This is necessary as the wavesandbox currently supports the sending and queuing of hosted wavelets prior to committing and sending the commit-notice.

The FedOne Host server currently has no mechanisms in place to first queue deltas then commit them at some later time. When a delta is submitted to the LocalWaveletContainer, it is immediately committed through the call to commitAppliedDelta() (please note that it is not actually committed, but it is treated as such and a commit notice is sent to Remote servers as soon as the deltas are applied). The reason wavesandbox queues updates is to optimize latency, however, this does cause a number of issues that are out of the scope of this post. To read more about commit-notice and its implications on the wave server, I suggest following this thread on the google group.

To properly restore the WaveServer at startup, it is also necessary to add persistence to the implementation of the wave.crypto.CertPathStore. The wave.crypto.DefaultCertPathStore notes that this default implementation is in-memory only and will lose certificate chains when the server is shut down.

Finally, we must provide some persistence for the client. Specifically, we must store information related to the Index Wave. The Index Wave is a lookup that contains all waves a participant has access to. Conceptually, the Index Wave can be thought of as follows:

When a user connects to the client, a call is made to fedone.waveserver.ClientFrontendImpl.openRequest() with the Index Wave ID (specified in fedone.common.CommonConstants.INDEX_WAVE_ID). This will request history on all wavelets associated with the Index Wave for the user. As illustrated in the above graphic, the Wavelet ID of the Index Wave is actually a reference to a Wave ID to which the user is a participant.

In all, the following information should be persisted: The Wavelet identification, Applied Wavelet Deltas, Transformed Wavelet Deltas, Certificate Store, and Index Wave information. These can either be stored in-place (for instance, as a ProtoBuf wrapped object) or the data can be extracted and saved in its native format.

Objects decomposed into native representations (see wave.protocol.common.proto for ProtoBuf based objects)

The following is a breakdown of the ProtocolBuffer objects used in FedOne. The syntax is /: { }.

ProtocolAppliedWaveletDelta: {

signed_original_delta/ProtocolSignedDelta: {

delta/bytes(ProtocolWaveletDelta): {

hashed_version/ProtocolHashedVersion: {

version/int64

,history_hash/bytes

} //end ProtocolHashVersion

,author/String

,operation/ProtocolWaveletOperation: {

add_participant/String

,remove_participant/String

,mutate_document/MutateDocument (see reference for details)

,no_op/bool

} //end ProtocolWaveletOperation

,address_path:String

} //end of ProtocolWaveletDelta

,signature/ProtocolSignature: {

signature_bytes/bytes

,signer_id/bytes(ProtocolSignerInfo): {

/*The certificates present in a ProtocolSignerInfo are encoded in PkiPath format, and then hashed into signer_id using the hash algorithm indicated in the ProtocolSignerInfo*/

hash_algorithm/enum

,domain:String

,certificate:bytes

} //end of ProtocolSignerInfo

,signature_algorithm/enum

} //end ProtocolSignature

} //end ProtocolSignedDelta

,hashed_version_applied_at/ProtocolHashVersion: {

version/int64

,history_hash/bytes

} //end ProtocolHashVersion

,operations_applied/int32

,application_timestamp/int64

} //end ProtocolAppliedWaveletDelta

ProtocolWaveletDelta for transformedProtocolDelta. (The number of operations applied as referenced in the AppliedDelta above reference the operations contained in this transformedProtocolDelta):

{ //same as above. repeated here for ease of reference

hashed_version/ProtocolHashedVersion: {

version/int64

,history_hash/bytes

} //end ProtocolHashVersion

,author/String

,operation/ProtocolWaveletOperation: {

add_participant/String

,remove_participant/String

,mutate_document/MutateDocument (see reference for details)

,no_op/bool

} //end ProtocolWaveletOperation

,address_path:String

} //end of ProtocolWaveletDelta

WaveletName: {

waveId/WaveId: {

domain/String

,id/String

} //end WaveId

,waveletId/WaveletId: {

domain/String

,id/String

} //end WaveletId

} //end WaveletName

{

public SignerInfo get(byte[]);

public void put(ProtocolSignerInfo);

}

Wrap Up

I hope this series has been a useful companion to the FedOne documentation. When you really dig into the code as a potential contributer you realize that there is a very complex (yet elegant) architecture under the hood. There was much not covered in this series like the implementation of the OT algorithms or the current (soon to be replaced) client-server protocol. This will not be my last post about the wave protocol, but I figured the 5 posts was enough for an introduction/overview.

Further Investigation

The best place to go for more in-depth information is the code itself. I would also HIGHLY recommend joining the Google User Groups for the FedOne project. There is a wealth of information being shared there on a daily basis.

A few links if you’re just getting started:

Installation Instructions [http://code.google.com/p/wave-protocol/wiki/Installation]

Installation Instructions for Windows users [http://jamespurser.com.au/blog/How_To_-_Install_WRS_On_Windows]

Wave Protocol Google Group [http://groups.google.com/group/wave-protocol/]

Wave Protocol Code Google Group (more focused on code reviews and repository updates) [http://groups.google.com/group/wave-protocol-code-discuss]

MongoWave: Persistence on Google FedOne Wave Server with mongoDB Purpose My company, SESI, has been working on applying...
Google Wave’s Federation Protocol Under the Hood, Part 4 Purpose [Updated 4/3/2010] This is the fourth post in...
Google Wave’s Federation Protocol Under the Hood, Part 3 Purpose [Updated 4/3/2010] This is the third post in...

Related posts brought to you by Yet Another Related Posts Plugin.

Google Wave’s Federation Protocol Under the Hood, Part 4

Anthony — Tue, 16 Feb 2010 17:26:17 +0000

Purpose [Updated 4/3/2010]

This is the fourth post in the series dedicated to exploring the internals of the FedOne architecture. This post is more a reference that will quickly guide you to the location in code that corresponds to the receipt and execution of an XMPP packet (e.g. History Request IQ Get Packet or a Wavelet Update Message Event). The messages processed are those supported by the FedOne codebase and is not based off of the documentation (not saying anything against the documentation, which I referenced many times. For instance to see examples of successful Packet transmissions).

FedOne was recently updated to introduce more robust error handling mechanisms. You will now find a set of files in the package org.waveprotocol.wave.federation that are dedicated to federation errors. The error codes supported can be found in org.waveprotocol.wave.federation.federation_error.protodevel (note this is the specification, if you want to see the actual .java file, go to proto_src/org.waveprotocol.wave.federation.FederationErrorProto). Per the documentation, the error codes map to XMPP error stanzas as specified in RFC 3920 (9.3.3). There is also a wrapper FederationErrors.java to make it easy to create the FederationError objects. You will see these used throughout the communication code as a parameter to org.waveprotcol.wave.federation.xmpp.PacketCallaback responses (also used to communicate success). For an example of its use see wave.federation.xmpp.processHistoryRequest().

Communication Interfaces

XMPP communication within the federation protocol occurs in the org.waveprotocol.wave.federation.xmpp package. All elements related specifically to XMPP (e.g. IQ packets) are read in here and stripped out into neutral formats (well, let’s count ProtoBuf as neutral). The resulting data is then passed to the wave server for processing. For each operation, I will list the file/method of the XMPP filter along with the file/method within the wave server that processes the data. Please recall that I am focusing on Server to Server communication, however, I will try to list the mirrored methods of the wave server that would be used by the client (e.g. - instead of just listing the server’s response to an incoming submitRequest, I will attempt to show where a local client would initiate its own submitRequest).

As a preliminary note, all incoming packets are received by receivePacket() in org.waveprotocol.wave.federation.xmpp.XmppManager

HistoryRequest - received from Federation remote to request wavelet operations from the hosting wave providers.
Packet Type	IQ Get Packet
XMPP Filter	wave.federation.xmpp.XmppFederationHost processHistoryRequest() - collect history request data from XMPP IQ packet and pass to Wavelet provider for processing.
WaveServer Handler	fedone.waveserver.WaveServerImpl.requestHistory() - retrieves info for a signer of an applied delta and returns IQ result to foreign remote via fedone.federation.xmpp.XmppFederationHost. HistoryResponsePacketListener.onSuccess(). Note: onFailure currently just logs an error, but does not actually send error back to foreign remote.
WaveServer Mirror	fedone.waveserver.WaveServerImpl.requestHistory() - gets the transformed history for a wavelet and returns to caller. Note this call is overloaded, so it is not the same method as above.

SignerRequest - received from Federation remote to request certificates for wavelets where the signer is unknown. Request sent to host wave providers.
Packet Type	IQ Get Packet
XMPP Filter	wave.federation.xmpp.XmppFederationHost. processGetSignerRequest() - collect signer request data from XMPP IQ packet and pass to Wavelet provider for processing.
WaveServer Handler	fedone.waveserver.WaveServerImpl.getDeltaSignerInfo() - retrieves delta history for a given wavelet and returns info to listener via wave.federation.xmpp.XmppFederationHost. DeltaSignerInfoPacketListener.onSuccess() or .onFailure().

DiscoveryInfoRequest (Disco) - received from foreign source to get info on what we support. Will send back message saying we are Wave Component.
Packet Type	IQ Get Packet
XMPP Filter	wave.federation.xmpp.XmppDisco.processDiscoInfoGet() - handles discovery info get. Will send back message identifying server as a Wave Server.
WaveServer Handler	N/A

DiscoveryItemRequest (Disco) - received from foreign source to get disco items. No defined use within Wave.
Packet Type	IQ Get Packet
XMPP Filter	wave.federation.xmpp.XmppDisco. processDiscoItemsGet() - handles discovery item get. Will send back copy of message in response.
WaveServer Handler	N/A

errorCallback - when the WaveServer sends a packet, it has the option of specifying an error callback in the case the host or remote responds with an error.
Packet Type	IQ Error Packet
XMPP Filter	wave.federatoin.xmpp.XmppManager.response() - on an error packet, checks to see if there is a callback defined.
WaveServer Handler	N/A

MessageReceived - received from Federation remote to acknowledge an update.
Packet Type	Message Received Packet
XMPP Filter	wave.federatoin.xmpp.XmppManager.response() - handles Wavelet Update Response by sending packet to callback handler.
WaveServer Handler	N/A

MessageEvent (update) - received from Federation Host to Federation Remote to update a wavelet.
Packet Type	Message Event Packet
XMPP Filter	wave.federation.xmpp.XmppFederationRemote.update() - Parses update message. Creates a latched callback that is invoked when all deltas of all wavelet updates are applied. Also parses final commit notice if one is given.
WaveServer Handler	fedone.waveserver.WaveServerImpl.listenerForDomain(). waveletDeltaUpdate() - get and check state of wavelet. Place deltas in ProtoBuf and get/create remoteWavelet to update. fedone.waveserver.WaveServerImpl.listenerForDomain(). waveletCommitUpdate() - notify client listener that update has been persisted and they can release any resources held for update (of course since there is no persistence we’re just telling lies) fedone.waveserver.RemoteWaveletContainerImpl.update() - pre-fetch all the signers and run the internalUpdate.

Message Ping - provide evidence of available service.
Packet Type	Message Ping Packet
XMPP Filter	wave.federation.xmpp.XmppManager.processMessage() - adds receipt to response and sends back to requestor.
WaveServer Handler	N/A

DiscoveryInfoResult (Disco) - handles disco info result for remote JID and triggers callback.
Packet Type	IQ Result Packet
XMPP Filter	wave.federation.xmpp.XmppManager.response() - forwards to callback hander defined in PacketCallback.run() of wave.federation.xmpp.RemoteDisco. requestDiscoInfo() - receives response from a disco info request to a specific target JID (supports wave) and processes in wave.federation.xmpp.RemoteDisco.finish() - if we find the remote JID, execute the callback functions that were set in wave.federation.xmpp.RemoteDisco.discoverRmoteJID().
WaveServer Handler	N/A

DiscoveryItemResult (Disco) - handles disco item response from remote JID and triggers callback.
Packet Type	IQ Result Packet
XMPP Filter	wave.federation.xmpp.XmppManager.response() - forwards to callback hander defined in PacketCallback.run() of wave.federation.xmpp.RemoteDisco.startDisco() that calls wave.federation.xmpp.RemoteDisco.processDiscoItemsResult() - process disco item results executing requestDiscoInfo() to see if any are wave servers.
WaveServer Handler	N/A

HistoryResult - foreign Federation Host sending history response.
Packet Type	IQ Result Packet
XMPP Filter	wave.federation.xmpp.XmppFederationRemote .processHistoryRequest() - receives history request response packet, decodes it and calls wave server. [this is defined as the PacketCallback in wave.federation.xmpp.XmppFederationRemote. requestHistory()]
WaveServer Handler	fedone.waveserver.RemoteWaveletContainerImpl.internalUpdate() [onSuccess() method defined in federationProvider.requestHistory() call] - converts deltas into ProtoBuf format and calls fedone.waveserver.RemoteWaveletContainerImpl.update() - pre-fetches signer info and calls fedone.waveserver.RemoteWaveletContainerImpl.internalUpdate() [note, first reference to this function above was not actually direct call. What was called was the onSuccess() method that was defined as a parameter in the requestHistory() method, so there is no recursion here] - applies deltas received from foreign Federation Host.

SubmitResponse - handles response from a submit request
Packet Type	IQ Result Packet
XMPP Filter	wave.federation.xmpp.XmppFederationRemote. processSubmitResponse() receives submit response response packet, decodes it and calls wave server callback. [defined in PacketCallback.run() of wave.federation.xmpp.XmppFederationRemote.submitRequest()]
WaveServer Handler	not really implemented in FedOne or by the WaveServer. The responsibility of handling a submit response does not necessarily fall to the Federation Remote since it does not host the wave. We should store the applied deltas, but the client will ultimately handle this response. This is done at a very basic level in the FedOne at fedone.waveclient.common.ClientBackend. sendAndAwaitWaveletDelta().

PostSignerResponse - ACK from foreign Host server in response to certificate chain sent to authenticate wave delta. This must be done first time before sending any wave delta.
Packet Type	IQ Get Packet
XMPP Filter	wave.federation.xmpp.XmppFederationHost. WaveletFederationProvider. DeltaSignerInfoResponseListenter() - receives post signer response packet, decodes it and calls wave server callback.
WaveServer Handler	fedone.waveserver.WaveServerImpl.submitDelta() [callback defined as parameter in postAllSignerInfo()* method] - logs success and then uses Federation Remote to submit request.*

SubmitRequest - handles a submit request from a foreign Remote server
Packet Type	IQ Set Packet
XMPP Filter	wave.federation.xmpp.XmppFederationHost. processSubmitRequest() - receives submit request packet, decodes it and calls wave server callback.
WaveServer Handler	fedone.waveserver.WaveServerImpl.submitRequest() - ensures delta is properly signed and meets preconditions to be applied, then submits delta calling: fedone.waveserver.WaveServerImpl.submitDelta() - submits delta to local or remote wavelets (remote first sends signatures then delta) , broadcast updates to Federation Hosts and Client. Returns result via listener.
WaveServer Mirror	fedone.waveserver.WaveServerImpl.submitRequest() - signs the delta and calls: fedone.waveserver.WaveServerImpl.submitDelta() - submits delta to local or remote wavelets and returns result to listener

PostSignerRequest - Signer info from foreign Remote to be stored prior to receiving and applying deltas.
Packet Type	IQ Set Packet
XMPP Filter	wave.federation.xmpp.XmppFederationHost. processPostSignerRequest() - receives post signer request packet, decodes it and calls wave server.
WaveServer Handler	fedone.waveserver.WaveServerImpl.postSignerInfo() - stores signer info with CertificateManager.

Next In Series

The next post will be focused on Persistence. I will highlight where the local in-memory store occurs and hopefully point out the correct areas to implement persistence if you so choose.

Google Wave’s Federation Protocol Under the Hood, Part 5 Purpose [Updated 4/3/2010] This is the fifth and final post...
MongoWave: Persistence on Google FedOne Wave Server with mongoDB Purpose My company, SESI, has been working on applying...
Google Wave’s Federation Protocol Under the Hood, Part 3 Purpose [Updated 4/3/2010] This is the third post in...

Related posts brought to you by Yet Another Related Posts Plugin.

Google Wave’s Federation Protocol Under the Hood, Part 3

Anthony — Fri, 12 Feb 2010 16:18:59 +0000

Purpose [Updated 4/3/2010]

This is the third post in the series dedicated to the design of FedOne. I am taking a quick aside to show the dependency object graph as managed by Guice. I found it quite interesting to view a picture of how many of the components were associated. There are dependencies that are not captured by Guice and are therefore not present in the graph, however, I would suggest that the areas where dependency injection was used are most likely suitable for alternative implementations (if you decide to develop your own wave server).

Dependency Object Graph

It became apparent upon creating the Activity Diagram that it would be helpful to illustrate the dependencies as resolved by Guice. Guice 2.0 has a Grapher module that will produce a .dot file that can be visualized with GraphViz. Read Instructions here. However, Grapher is an extension that is NOT bundled with Wave or the pre-compiled Guice libraries, so you will have to download Guice src and build Grapher (located in extension/grapher).

Following is the complete image (click image to open new window at full-size) ran on my Wave Server.

Legend

Nodes:

Dashed Nodes are Interfaces or Abstract Classes
Black backgrounds denote Implementation types
Gray backgrounds denote Implementation Instances

Edges:

Solid edges represent dependencies (think the parameters of a constructor)
Dashed edges represent bindings from types to their implementations (think subclasses or implementation of an interface)
Double arrows represent that the binding or dependency is to a Provider (Provider is a Guice construct)

I will focus on one aspect of the object graph to demonstrate how Guice is utilized and why viewing the dependencies may not be straightforward at first glance.

The following image illustrates the instantiation of the ProtocolWaveClient interface. I will only traverse to the 3rd dependency at ClientFrontEndImpl.

In the run() method of fedone.ServerMain you will find:

[code]

Injector injector = Guice.createInjector(new ServerModule(), flags);

…

ProtocolWaveClientRpc.Interface rpcImpl = injector.getInstance(

ProtocolWaveClientRpc.Interface.class);

[/code]

The injector.getInstance(ProtocolWaveClientRpc.Interface.class) requests Guice to find the associated dependency, which we can see from the graph is satisfied by WaveClientRpcImpl. To see how this connection is made, we will examine the first line in the run() method that created the injector [Guice.createInjector(new ServerModule(), flags)].

In the configure() method of fedone.ServerModule(), we find:

[code]

// Receive updates from the outside world, and push them into our local Wave Server.

bind(WaveletFederationListener.Factory.class).annotatedWith(FederationRemoteBridge.class)

.to(WaveServer.class);

…

install(new WaveServerModule());

bind(String.class).annotatedWith(Names.named(”privateKey”)).toInstance(”");

…

[/code]

Since we do not find a binding associated with ProtocolWaveClientRpc in this configure() method, we will step into the configure() method of fedone.waveserver.WaveServerModule() (note we are in a different package). Here we find:

[code]

bind(CertPathStore.class).to(DefaultCertPathStore.class).in(Singleton.class);

…

bind(ClientFrontend.class).to(ClientFrontendImpl.class).in(Singleton.class);

bind(ProtocolWaveClientRpc.Interface.class).to(WaveClientRpcImpl.class).in(Singleton.class);

…

[/code]

Now we see the bind statement that creates the actual dependency to the WaveClientRpcImpl class (which btw is a Singleton class). The resolution does not end here since stepping into the constructor of the fedone.waveserver.WaveClientRpcImpl class reveals a dependency on ClientFrontend:

[code]

/**

* Constructor.

* @param frontend ClientFrontend that consumes the operations.

@Inject

public WaveClientRpcImpl(ClientFrontend frontend) {

this.frontend = frontend;

}

[/code]

Since we see the @Inject annotation, we know that Guice will provide this value as well. Since we are still in the path of the same Injector that was originally created I will look back into either the flag, ServerModule, or WaveServerModule modules. Since ClientFrontend is a class in fedone.waveserver, we expect and do indeed find the binding in the configure() method of fedone.waveserver.WaveServerModule(). This is denoted above in the excerpt of that class [bind(ClientFrontend.class).to(ClientFrontendImpl.class).in(Singleton.class);]. I will stop here, but you will also note from the graph or if you explore ClientFrontendImpl that there is another dependency within the constructor that calls for a WaveletProvider, which Guice must also resolve.

I can hear some of you now saying “OMG, there are so many levels of indirection. Why would you ever do this?”. First I would point out that our dependencies exist regardless of using Guice. Usually these dependencies are just littered throughout the code making it more difficult to locate and switch to different implementations. Guice takes these bindings and places them in defined areas that, once you are familiar with Guice and the codebase, can be located fairly easily. This makes testing more efficient as bindings can be switched without much effort. As you have seen, object graphs can be created which will illustrate dependencies within your code managed by Guice giving you a better overall understanding of the structure of your project. Guice is also typesafe such that at compile time you will know if a binding was left out or applied incorrectly.

If you are curious where the command line Flags are used within Wave, just take a look at the object graph. Since they are instances they will be located in gray nodes and will tell you the java File and line number where it was resolved as well as the value passed in. For instance, the ComponentPacketTransport uses several.

Next in Series

The next post investigates the XMPP to WaveServer communication methods. All methods related to incoming or outgoing packets through XMPP will be covered as implemented in code (as opposed to what is documented).

MongoWave: Persistence on Google FedOne Wave Server with mongoDB Purpose My company, SESI, has been working on applying...
Google Wave’s Federation Protocol Under the Hood, Part 1 Purpose [Updated 4/3/2010] This post is the first in...
Google Wave’s Federation Protocol Under the Hood, Part 2 Purpose [Updated 4/3/2010] This post is the second in a...

Related posts brought to you by Yet Another Related Posts Plugin.

Google Wave’s Federation Protocol Under the Hood, Part 2

Anthony — Thu, 11 Feb 2010 15:42:37 +0000

Purpose [Updated 4/3/2010]

This post is the second in a series aimed at understanding the internals of the FedOne protocol. I will focus on the startup process in this post. Specifically, the classes and packages involved coupled with how they interact. If you are not familiar with Guice, I recommend reading the first post in this series, which covers pre-requisites to understanding the FedOne codebase.

Startup Process

When you startup your Wave server, you may use a command that resembles the following:

java -jar dist/fedone-server-0.2.jar –client_frontend_hostname=127.0.0.1 –client_frontend_port=5222 –xmpp_component_name=wave –xmpp_server_hostname=%XMPP_SERVER_HOSTNAME% –xmpp_server_ip=%XMPP_SERVER_IP% –xmpp_server_port=5275 –xmpp_server_secret=%XMPP_SERVER_SECRET% –xmpp_server_ping=”acmewave.com” –certificate_private_key=%PRIVATE_KEY_FILENAME% –certificate_files=%CERTIFICATE_FILENAME_LIST% –certificate_domain=%CERTIFICATE_DOMAIN_NAME% –waveserver_disable_verification=true –waveserver_disable_signer_verification=true –xmpp_jid=wave.%WAVE_SERVER_DOMAIN_NAME% –xmpp_server_description=”FedOne“

This startup command maps to ServerMain.java within the package org.waveprotocol.wave.examples.fedone (until otherwise specified, all files I list are housed within this package). The flags that are passed via the command are specified in FlagSettings.java. There you will see the annotation @Flag (specified in Flag.java). The main() method of ServerMain.java calls FlagBinder.java to read in the Flags and validate them against the allowable values as specified in FlagSettings.java. FlagBinder then returns an injectable Guice module which is used by the Injector in the run() method.

The above describes how the command line arguments are propagated throughout FedOne. The following Activity Diagram illustrates the complete startup process (click image to see full-size).

Next In Series The next post will cover some of the dependencies in the FedOne code (those managed by Guice). I will show a diagram of the dependencies in the GraphViz tool using a data file produced by the Grapher extension of Guice.

Google Wave’s Federation Protocol Under the Hood, Part 3 Purpose [Updated 4/3/2010] This is the third post in...
Google Wave’s Federation Protocol Under the Hood, Part 5 Purpose [Updated 4/3/2010] This is the fifth and final post...
Google Wave’s Federation Protocol Under the Hood, Part 4 Purpose [Updated 4/3/2010] This is the fourth post in...

Related posts brought to you by Yet Another Related Posts Plugin.

Google Wave’s Federation Protocol Under the Hood, Part 1

Anthony — Tue, 09 Feb 2010 23:59:08 +0000

Purpose [Updated 4/3/2010]

This post is the first in a series intended to cover the basics of how the FedOne project as released by Google is designed and implemented in code. I could not find a guide that covered the internal architecture and namespaces, which left me sifting through approximately 200 files in 30 packages to gain an understanding that allowed me to modify and contribute code. In addition, I was not familiar with some of the Google technologies employed (e.g. - Guice and ProtoBuf). Since we are exploring the use of Wave at my company, I thought it would be helpful to our engineers to write up an overview. Hopefully, as you read this you will find it useful as well.

I do not intend to write an exposition on every package or file. There will be many topics left uncovered or as I like to think of it many opportunities for others to expand upon this post :). I plan to highlight the topics that will give those new to the project/code a push in the right direction. My current intention in this series is to cover prerequisites to examining the code, the startup process, dependencies as captured by Guice, the XMPP to WaveServer communication, and Persistence. Other topics such as the Client-Server protocol or the OT algorithm itself will not be addressed (at least at this time).

Audience

This post is targeted specifically to developers and technical managers. No effort has been made to abstract the technical implementations or figures into high-level concepts. Furthermore, I assume the reader has an understanding of Google Wave in general and more specifically the Google Wave FedOne project.

For an introduction to Google Wave, I recommend:

About Wave [http://wave.google.com/help/wave/about.html]

Google Wave Wikipedia Entry [http://en.wikipedia.org/wiki/Google_Wave]

For an introduction to the FedOne Project see:

Wave Federation Home [http://www.waveprotocol.org/]

Prerequisites

An understanding of a couple of Google technologies used will make it much easier to follow the FedOne code base. These are Guice and Protocol Buffers.

Guice

You will quickly get lost in the code without an understanding of Guice and how it is used. Guice is a lightweight dependency injection technology used and open sourced by Google. It allows for the decoupling of dependencies in code that are created when direct calls are made to constructors of external Classes. The Guice technology has many benefits and applications, but we’ll focus on the ease of switching out implementations of an algorithm that are present in different subclasses (think Template pattern as defined by the Gang of Four). What you need to know at a base level (and I posit you need more) is that when you see an @Inject annotation, the parameters that follow will be looked up and supplied by Guice at or prior to runtime. Guice does this by defining bindings in a separate Module from an Abstract class to a subclass. A Guice injector is then created from that Module and the specified dependency (Subclass) is injected when an Instance of the Base Class is requested. Clear as mud? Yes, you really need to read about Guice separately from this discussion.

The reason I even broached the topic of Guice is because while dependency injection is an incredibly useful technique it does have the downside that it can often be difficult to track the class/values that are actually being applied at runtime. Using an example from the Guice site, a constructor may have a BillingService class that is implemented by RealCreditCardBillingService and FakeCreditCardBillingService. Since the binding to one of the subclasses is specified in a separate file and is constructed by Guice, you may have difficulty when you’re testing to realize that RealCreditCardBilling is still being used. If you set up your Guice bindings in your own project, you’ll probably be familiar with where these settings are located, but if you’re looking at another Project for the first time that has 200 files over 30 packages it can be difficult. For instance, the ComponentPacketTransport.java within org.waveprotocol.wave.federation.xmpp package contains @Named annnotations for parameters that are supplied by FlagBinder.java in org.waveprotocol.wave.examples.fedone. There is a Graph extension that allows you to view the dependency linkings, however, that is not released within FedOne. I will show an output of this tool on FedOne in a later post to this series.

To read more on Guice, I recommend you watch the video and read the user guide. There are other uses within FedOne (e.g. - scoping) that I did not cover above.

Protocol Buffers (often referred to as protobufs)

Protobufs are used by Google to serialize structured data for storage and transmission. The data structure you wish to emulate in Java (or C++ and Python) is specified in a .proto file, which the Proto compiler then uses to generate code allowing an object with that structure to be constructed in Java. The Proto compiler also adds a number of other methods such as getter, setter, and exist methods. The benefit of the ProtoBufer is the simplicity, size and speed of the resulting code. To use an example from the Protobuf site, accessing data from XML may require the developer to write person.getElementsByTagName(“name”)->item(0)->innerText() while ProtoBuf would access the same data with person.name(). As a result of the amount of code generated by the Proto Complier it is much easier to understand the structure of an object by locating the .proto file. The methods generated by the compiler can be found by researching the ProtoBuf documentation. ProtoBufs are used early and often in FedOne, so an understanding of them will be of great assistance.

The Authority of Perspectives

Anthony — Thu, 04 Feb 2010 16:17:26 +0000

This post is based on a response I made to a Knowledge Management listserv several months ago. Since that time the notion of authoritative data sources and validity of information has continued to surface in the course of my work. My view on the subject has been refined by discussions with Dr. Andres Sousa-Poza and Sam Kovacic at the National Center for System of Systems Engineering at Old Dominion University. The content of my original post was as follows:

I believe that too often, particularly in the Federal government, there is too much emphasis placed on SMEs and authoritative sources of information. I believe a culture shift is in order that will go from “authoritative” to “value-added”. You won’t find many people who consider themselves experts in a given category, but that does not mean they do not have valuable viewpoints on a topic. Out of the 700 or so people on this listserv, there are probably a large number that could add value to this subject, but will not because they do not feel they can offer expert advice.

This is akin to offering Consumer Reports to the community as the authoritative source of information on products and providing access to the people that actually performed the reviews. Certainly, their insight is valuable and they can tell you that the battery life on the portable DVD player is actually only an hour as opposed to the 2 hours listed in the technical specifications. However, I find the most value by going to Amazon.com and reading regular user reviews on a product. There I might find that another parent bought the product for their child on long car drives, but the buttons were left exposed so the child constantly turned the DVD player off (based on this I bought a Sony that had a Hold button :) ). You will also discover issues like how customer support responds to problems and be able to surmise the true value of a product based on finding trends in the comment threads. The great majority of these people would not consider themselves experts, but do feel they have a perspective that may add value. Sites like Amazon.com and Barnes and Noble encourage this type of user participation. I have not seen the same encouragement from the Federal government.

I have modified my view on the subject, although the sentiment of my original post remains the same. The key remains capturing the perspectives that contribute valuable information. However, it was inaccurate for me to make a one to one correlation between authoritative sources and SMEs. Expertise can be localized such that the authoritative source on an issue is not a SME.

For instance, a solider that has been on the ground in Afghanistan for a year may have more knowledge of the prevailing conditions than a PhD who has studied the society. I consider both perspectives to fall within the “expert” category. Both individuals have relevant perspectives and should be considered as capable of providing valuable information. However, it must be recognized that these two views will be more appropriate in different situations.

A SME point of view will be most effective in situations that require strategy based on broad generalities that are valid over some defined segment. The local perspective is much more effective in providing context specific advice. For instance, a SME could best advise on how the US Military should approach civilians in Afghanistan given their culture, while a local figure would be more knowledgeable on the most likely hideouts for terrorists in a region. To facilitate informed decision-making it is vital to understand when to shift between these perspectives as most problems have an element of both.

An additional issue arises when trying to identify a local perspective. SMEs can usually be identified by a number of factors to include experience, education, speaking engagements, and published works. Locating an individual that has “lived-in” a related situation or within a target environment and has a viewpoint that is in line with an overarching strategy is much more difficult. A local perspective may also extend beyond the views of a single individual, but rather be composed of a group of individuals that exhibit certain patterns. As local perspectives are either solicited or derived from a crowd, there is a massive amount of information that must be filtered.

Social Media is largely responsible for bringing many of these issues to the forefront. The crowd now has a voice that extends as far or beyond traditional communication outlets, which are more accessible to SMEs. Also, the volume of user-generated content enabled by this technology far exceeds that which was previously available. As a result, organizations within the public and private sector along with private citizens are engaged in processes to aggregate, search and analyze the flow of information to reveal interesting patterns and drive decision-making in near real-time.

While Social Media has enormous potential to drive business, the success of any organization at making informed decisions in the future will be limited by the credibility they assign to the local perspective. The technology to filter relevant content and identify reliable patterns has and will continue to evolve at a rapid pace. The greater challenge for most organizations will be the cultural shift required to embrace and rely on this new type of “authoritative” information.

Do You See What I See? When I conceived of this blog, I thought I would...

Related posts brought to you by Yet Another Related Posts Plugin.

Do You See What I See?

Anthony — Fri, 20 Nov 2009 05:11:15 +0000

When I conceived of this blog, I thought I would explore “solutions”. That is, how technology could be applied to solve challenges faced in the government and commercial sectors. I have recently started a new job that focuses on assisting clients with decision-making within complex environments. While I still intend to focus on applied technology, my new position has introduced me to the philosophical underpinnings that play a heavy role in decision-making. Paramount to what I am researching now is how to account for perspectives. An individual’s perspective will affect how the problem is framed, the approach taken, and will ultimately define how success is measured.

Given the importance of perspective, it’s probably a good idea to tell you a little bit about mine. As a former business owner, I tend to look at the value of technology as measured by its utility to the end user in conjunction with its costs to the organization. Keep in mind that cost is not always monetary. Cost could be defined as resources required, time to deploy, maintenance schedules, impact on culture, or any number of other factors. The key point being that technology does not exist in a vacuum and should serve a defined need. While this is a belief that I hold dear, I must be candid. I am a Software Architect by trade so I do still appreciate, and sometimes even marvel at, a well-designed innovate piece of software. To put it plainly, there’s definitely some geek in me and I do enjoy the occasional shiny object.

With this in mind, my blog will share my thoughts and experiences in regards to business strategy, software architecture, and technologies that revolve around collaboration-based solutions. I will also catalog what I learn in regards to Philosophy and how we are able to apply it to software to enhance decision-making capabilities. I hope that through this channel I will engage you enough that you will share your experiences and ideas with me. I look forward to speaking with you.

The Authority of Perspectives This post is based on a response I made to...
About Me BarCamp Style Who are you? - Husband | Father |...

Related posts brought to you by Yet Another Related Posts Plugin.