Message Paging

The biggest problem with millions of messages is memory.  Gibraltar Analyst is a 32-bit process which means it’s going to fail if there’s more than about 1.5GB of RAM allocated at any time.  Of course, we want it to use the least memory possible but we can’t exceed 1.5GB.  After some memory profiling it was clear that displaying the messages was more memory intensive than loading the session data itself.

We sampled a set of data from our own sessions and customer-provided samples and determined that over about 250,000 messages the value of browsing and dynamically filtering more messages is more than offset by the performance impact.  In Gibraltar 2.5 any session with more than 500,000 messages will get split into pages of 250,000 each.

We’ve added a status bar to the bottom of the log messages grid to provide additional navigation context as you work with log messages, including paging information:

You can jump between pages, and there’s an overlap of messages so if you are working on an issue right at a boundary between pages you don’t have to keep hopping back & forth.

How Many Messages?

We’ve always shown you how many messages were recorded for a session, but not necessarily what you have access to.  Why would there be a difference?  The main reason is due to data pruning in the agent when a session runs for a very long time or generates a lot of data.  If the session hits the pruning rules you define it won’t all be there.  Now we show how many messages there are at any time:

And update that if you start filtering to reflect the filter:

Anyone got the Time?

Many times you only need the approximate time and absolute order of log messages to follow what’s going on, but some times (like when understanding the performance of your application or understanding the exact timing of external events) you need as much detail as possible.

Just pick any set of messages and the total duration of the selection is displayed in the status bar (unless it’s zero).  The accuracy of the time is still limited to the resolution of the source clock , so you’re unlikely to see values less than 16 milliseconds.

You can also switch the grid to display the full time in milliseconds instead of using the default time format for the current culture (this feature was added in 2.1.1).

Chart Many Metrics

If you’ve been using the Gibraltar Agent for PostSharp, or done’ your own event metrics, you may have run into an annoyance that it can generate a lot of distinct metrics.  For example, we use the Gibraltar Agent for PostSharp to inject metrics into the Gibraltar Hub – and we get a distinct metric for each web service command and database call.

When we want to chart these to, say, find the top 10 time consuming database calls it’s annoying to have to add them one-by-one to the chart.  It’s slower and it causes the chart to keep recalculating after adding each metric.

Now you can drag & drop a whole category of metrics into the chart and add them in one go.  It’s faster, less error-prone, and lets you get answers fast to common questions.  Now the choice of whether to record everything under one metric or make many metrics isn’t driven by these limitations.  We have customers that generate thousands of metrics and this optimization as well as a few others related to it really improve the experience.

This actually demonstrated some other performance challenges when we created a few extreme data sets – with two million events in thousands of metrics and then start dynamically manipulating the chart, well, things can slow down.  We’re confident if it’s usable at these sizes, it’ll be fast enough for your situation.

Organizing the Session List

Like most grids in Gibraltar you can do your own grouping, sorting and filtering through the context menu.  But, we’ve add in a set of canned views that you can easily select through that will quickly reorganize the content.  Beta users particularly liked to view By Added Date and By Application, which work great to quickly get an overview on what’s going on.

It’s then natural to have more fun filtering and sorting, but if you want to get back to the view you started with just click the button again and you’re back to baseline.

When you group a set of messages we now show a breakdown of the sessions in each group:

If you collapse all of the sessions to groups since you can then get an overview of what’s going on by each category.

Try it out:  This is very effective when you combine it with the built in filtering of the tree view to get you in the neighborhood of the data you’re interested in, then you can easily browse to get a sense of what’s going on even when there are a large number of sessions.

Previewing Sessions

We’ve upgraded the session detail view to now show the results of analyzing a session’s messages.  You can select the Errors by Class or Errors by Category to see all of the unique critical, error, and warning messages in the session without opening the session.

This saves a lot of time both when you’re just browsing to see what happened in a session and when you are trying to “look sideways” – see what errors happened around another error.  To do this, display the Errors by Class Session summary, click on an error and then select a session with that error to see all of the other errors & warnings that happened in the same session.   This is also really popular when combined with one of the defect tracking summary views for the same purpose.

If you see something worth digging into, go ahead and open the session by double-clicking it.

Deferred Refresh

We use Gibraltar day in & day out, but we’re often not running it that long at a time because we’re starting & stopping it for development.  In working with a new large client we noticed an annoyance caused by the view automatically refreshing whenever new data arrived from the Hub.  If data is arriving often enough the pause while the view refreshed really got in the way of the task the user was trying to perform, like sorting & grouping the set of sessions.

Imagine that a new session is arriving every 15 seconds, which in a larger deployment isn’t unreasonable.  It takes about 1-4+ seconds to refresh the view depending on the speed of the system and how many sessions there are, what views are up, etc.   Just about when you start getting back into reading through the data to find what you want, the UI pauses again to refresh.

To address this, we’ve added an automatic deferred refresh where the current repository view won’t update to reflect new data as long as it has focus.  So if you’re working around in one of the summary views, the session list, or the details it will suspend the data refresh.  If you click off the node it’ll automatically update or if you have another window active or another part of the UI active.

When new data does appear, we’ve added a button that will show up to let you know.  We deliberately placed it in a “white” area of the user interface so its appearance is conspicuous even out of the corner of your eye while working in the view.  If you want, you can press it and get the latest data or ignore it and at the next best opportunity the UI will update automatically.

This is another case where we’re trying to make Analyst just “do the right thing” without you having to think about it.  We did a few rounds of usability testing with a select group of high-volume users and made some adjustments, and we’re confident you’re going to like it.

Other Improvements

We added a dedicated Remove button to the sessions list.  While this was always an option on the context menu, this is fewer clicks and is a frequent feature in many workflows when managing a large number of sessions.  We did some usability testing with this to ensure people wouldn’t hit it by mistake and made some spacing and format adjustments to improve it.

The default session summary view (Sessions by Timeframe) now defaults to the All row so the sessions grid shows the list of all sessions by default.  This was based on feedback from the beta community who commented they were always selecting that row so they could take advantage of the new session grid view features.

We’ve also improved some data caching behind the scenes which will make the user interface more responsive when switching between nodes in many cases, particularly for large numbers of sessions.

Keep that Feedback Coming

Many of the items above came from our real world experience and observation of customer patters, but a few are direct from customer requests.  If something doesn’t feel right to you please let us know – don’t worry if you’re not sure what would be better or how we should improve it.  We are driven to make sure Gibraltar is an easy, effective tool that just works for you – providing insight you can use to create better outcomes for your team & your customers.  We’re always listening – just shoot your thoughts to support or any of us and we’ll get a dialog going.

Gibraltar 2.5 includes built-in integration with two issue tracking systems and provides hooks for integrating with others.  This integration extends Gibraltar to automatically create issues for new problems as they occur and make it easy to drill into all of the Gibraltar data related to any issue.

Check out our release notes for technical details or watch videos of how Gibraltar integrates with FogBugz (by Fog Creek) and Gemini (by CounterSoft) to get an overview of our integration.

Update:  Check out the article on Closed Loop Software Development over at CounterSoft which talks about this integration. It’s a really easy way to get going, even if you’re a one developer shop!

The Full Scoop

While validating the new Gibraltar Add-In API we wanted to pick a real-world, hard problem that would push the edges of the integration capability.  We decided to pick up an idea that had always been on our to do list:  Integration with a defect tracking system.  We built the original integration for our internal use with our in-house system.  After working with that for a few weeks we were clear that this was a must-have feature that became a cornerstone of the Gibraltar 2.5 release.

Integrate with Your Processes

The primary goal of the issue tracking integration was to integrate Gibraltar and the data it collects into your existing workflow.  We realized that no matter how great we might make tracking and managing errors within Gibraltar it wouldn’t be as good as just being able to use your existing workflow.  Why?  Because you would still need your existing system, so why have two separate workflows?  Instead, you want to be able to look in your issue tracking system and say this is all there is without having to manually key and translate.

As a baseline, consider what would happen if you just configured email notification to flow from Gibraltar to your issue tracking system.  This would create a new issue each time a qualifying event happened.  In a production environment the number of issues would quickly get out of hand – one recurring problem and you’d be spending your time closing duplicate cases instead of working the important ones.  Instead you need to be sure:

• Each new issues creates a new case
• New sessions with the same problem just add more information to the open case
• You have some way of ignoring errors that don’t represent a defect.

The Gibraltar integration delivers on this by scanning sessions as they come in and mapping errors into your issue tracking system.  This integration ensures:

• Errors with a new unique signature will open new issues.
• Errors that match an existing signature will add additional information to the existing issue.
• If an error reoccurs in a new version after the related issue has been closed it will be reopened, unless you’ve set that issue to be ignored.
• Gibraltar sessions and issues will be related so you can easily see all of the sessions in Gibraltar for a particular issue.

All of this is possible because Gibraltar queries the issue tracking system while analyzing sessions to determine the right action to take.

Unexpectedly Awesome

When we first set this integration up ourselves we carefully sandboxed it off so it wouldn’t interfere with our existing approach to our CEIP.  Several weeks in, the development team started pulling items out of the sandbox and into our support queue and defect backlog because they were spot on.  Soon we were hooked – this was the way it should to be for everyone.   We took our original implementation and included it in the release as a source code sample and set out to create in-the-box integration for some great systems.

This is one of those cases in software where the system is more valuable than the sum of its parts:

• Real user errors are accurately and completely conveyed into your issue tracking system without any manual effort.
• You manage issues the same way you always have, including whatever reporting you’re already doing.
• You see all of the Gibraltar data for an issue with just a few clicks, no searching for the data by looking at messages until you find the right one.

Gibraltar adds value and reinforces your investment in issue tracking without requiring you to change how you work.

Translating between Gibraltar and Issue Tracking

In Gibraltar each session is identified by a distinct Product, Application and Version, like Gibraltar Analyst 2.5.0.742 or Gibraltar Hub 2.1.0.594.  These are usually very specific.  On the other hand in your issue tracking system you usually have more general categorization, like Gibraltar Current Development for the active branch and Gibraltar Support for shipped versions.   Not only are the two lists different, but you need enough mapping information to put new issues into the right workflow which may require selecting a priority, area, category, or whatever your issue tracking system requires.
Once you’ve enabled support for your issue tracking system you can configure how to map Gibraltar sessions into it from within Analyst. When a session is being analyzed it’s compared against the list of mappings until it matches one. To help minimize the amount of configuration necessary you can specify:

• Just a Product
• Just a Product & Application
• A Product, Application, and partial or complete version.

In the example at right, all sessions for Gibraltar Analyst with the major/minor version 2.2 would match.  You can do more interesting wild card version matching to help fit your exact situation, see our documentation on configuring relationships.

The idea is to configure a session to get into the right initial state in your workflow.  From there it should flow through just like if you keyed the defect yourself.

Details you Can’t Ignore

As more sessions come in they will be checked and the appropriate issues updated.  A special case for this is when you’ve closed the related issue.  Gibraltar has to decide whether to ignore the new data or reopen the issue.  To work this out it focuses on the version of the new session:

• If it’s the same or older as the latest version previously seen on the issue then the new data is ignored.
• If it’s for a newer version the defect is reopened

The idea is that Gibraltar recognizes there’s a lag between when you declare an issue resolved and when that fix gets deployed to users.  As long as it’s only occurring in the same set of versions it showed up in previously it’s still reasonable that the issue is closed.  If it shows up in a newer version then it probably hasn’t been fixed and certainly should be rechecked.

Until you have to Ignore Them

In any system there’s a set of errors that aren’t defects.  Temporary network connectivity issues, user errors, server timeouts…  They are still errors but they don’t call for any action on your part.  To handle this, Gibraltar supports marking an issue to be permanently ignored.  The exact mechanism depends on the particulars of the issue tracking system, but it’s selected to make it natural and obvious.

When you first enable automatic integration these noise issues will get opened once, but all you have to do is mark them to be permanently ignored and they’ll never resurface again.  Within a short period of time all of the common noise items will be suppressed and every new item will be gold.

Researching and Resolving Issues

Opening issues automatically is just one direction of integration.  Once the issue exists you want to be able to see all of the related sessions in Gibraltar effortlessly.  Within Analyst you can click on a dedicated node in the tree to see all of the open issues with Gibraltar sessions and drill down to the list of all related sessions.

Pick a session and you can open it or just preview all of the errors and warnings within it.  In this view you can quickly spot trends in the sessions:  perhaps they’re all in the same timezone or same operating system, or have another error that’s the true culprit.

Imagine that – you receive an email notification that there’s a new issue for you to look at.  You double-click the Analyst icon on your system tray and Analyst pops up.  Click on the issue tracking node and the most recent issue is on the top.  In seconds you’re seeing the whole picture of the error:  Related exception(s), actions they took leading up to the problem, performance and memory data.  You’ll get the whole picture quickly.

The dedicated issue tracking display supports a number of built-in views that are designed to match each system’s typical grouping & sorting to help you find what you want faster.

Creating Issues by Hand

You don’t have to just rely on the automatic issue management feature:  When you’re viewing a session you can select a range of messages and make a new issue from them on the fly.

Each of the messages you’ve selected are summarized with the last message in full detail.  This lets you show events leading up to a problem and then the big problem itself.

Gibraltar will still search your issue tracking system to see if there’s an existing case matching the last message you selected so it can handle adding new data to an existing issue or creating a new one.  It also checks your mapping configuration to pick the best defaults for the new case in your issue tracking system.

You can add whatever additional text you feel is useful to describe why you’re creating the issue and override some of the initial defect options like category or priority, depending on the system.

Create issues for suspicious items in our data – warnings, or just messages that don’t make sense.  Regardless of how you create the issue the rest of the integration works the same.

Work as a Team

Issue tracking systems are fundamentally about tracking & sharing knowledge on the state of your environment with a team of people.  Gibraltar fits right into this by:

• Sharing configuration with your whole team: Only one person needs to go in and update the mapping setup when you deploy a new version or application and the whole team is kept current.
• Centralizing Analysis: You can have the Hub perform all of the session analysis so no one need run Analyst for you to get new issues created & updated.

You can perform all of the configuration within Analyst and it flows back and forth to the Hub automatically, no need to get IT involved on the server when you need to adjust something.

An Army of One

If you’re using just Analyst stand alone you can still take advantage of everything we’ve covered – you just need to leave Analyst running to automatically have sessions analyzed (otherwise they’ll wait for you to run it the next time).  Fortunately we’ve made that a pretty good experience in this release.

Get Started Today

We’ve designed the integration to work without requiring anything be installed into your issue tracking system:  Just enable the integration feature and go. You can get it started with just one member of your team and see how it goes, or charge in full-bore by configuring automatic integration with the Hub and sharing the configuration with your whole team.  Download Gibraltar and get started!  If you don’t have a defect tracking system, FogBugz will provide a free hosted instance for 1 or 2 users and Gemini offers a free license for up to three users.

Check out our release notes for technical details or watch videos of how Gibraltar integrates with FogBugz (by Fog Creek) and Gemini (by CounterSoft) to get an overview of our integration.

Quick Takeaway

With Gibraltar 2.5 we’ve added email notifications to the Hub server.  You can receive text or HTML-formatted messages when new sessions arrive at the hub or detailed emails when errors are reported.  You can read about it in the online documentation.   The messages are pretty and we’ve worked hard to make sure you won’t get flooded when something goes wrong.

The Full Story

With Gibraltar 2.5 we’ve introduced a new server-based email notification system that can be easily configured by your team.  Our original goal was to satisfy a common Hub user request to give them some way of knowing that there was new data worth looking at so they’d know when to power up Analyst, retrieve updates and dig in.  As we scoped out the mission of Gibraltar 2.5 we saw another essential characteristic:  Email is the most pervasive process management tool in the world.  By providing the right email messages from the server, we could integrate with the defacto processes of a large number of customers.

• About a third of Gibraltar customers don’t have a dedicated issue tracking system.  They informally use email and spreadsheets to track what’s important.
• Most issue tracking systems can intake issues via email, so email notifications become a default integration option for these systems.

As we went through the design process for notifications, the key mission we kept laying out is that they had to be good enough to let you know if you needed to leave a meeting with your boss to go dig into an issue.  Think about it:  You’re sitting there in a meeting and your phone vibrates with an email,  You steal a quick glance and need to see enough to decide no biggie, it can wait or uh oh – better make my apologies and get back to my desk.  This meant it had to be readable on mobile mail clients (iPhone, Android, etc.) as well as your full email client (Outlook, GMail, etc.).

What Kind of Notifications?

We decided there needed to be two types of notifications:

1. New Session Notification: Alerts you that an interesting application session has been received with a summary of the overall session.
2. New Error / Warning Notification: Alerts you that an error or warning has been received.  You get an email for each unique error or warning in a session.

Example Session Summary Notification

Sometimes you just need to know if there’s new data available and if so a very quick summary of what it contains.  This is primarily interesting for users that don’t receive a lot of sessions, for example where sessions are only submitted when an end-customer has a runtime problem with the application or you’re primarily supporting one or two web sites.

For this scenario you can select Send a Session Available Notice and get few emails that are fairly high level.  You can see from the example that you get a summary of the session including what exact application & version it related to and the most commonly used information about the computer where it was running.  After that, each unique message that matched your notification threshold is summarized.  If it occurs many times the total number of occurrences is listed along with the timestamp of the first occurrence.  This format is designed to handle scenarios where a single error (like being unable to open a database connection) results in a storm of other errors (e.g. record not found) you can see both instead of the first error being lost in the storm.

Example Log Message Notification

The other scenario is to see deeper drill-in information on each specific error as they happen.  This is needed if you want to integrate with an external issue tracking system (since it needs something specific and somewhat unique to build an issue from) or you need to be able to take action on a problem quickly, possibly without even starting up Gibraltar Analyst.

For this scenario you can select Send Details on Qualifying Messages and get an email for each error individually.  The full details of the log message will be written out in an HTML format optimized for quick visual scanning.

If the log message that triggered the notification includes exception details the exception(s) will be written out to the detail with the call stack for each in order from outermost to innermost.

At the end of the message  a session summary is included with the most common information about the application & computer to help you perform quick triage.

Safety first – Notification without Noise

One thing we were worried about from the start was flooding prevention. You don’t want to be bombarded with 1300 emails because a web server went crazy and started screaming about a connection problem or your server got hit by a DoS attack.   With session notifications this is easy because you’ll just get one notification per session which significantly reduces the rate they can happen.  With log messages it’s a different story.

What we did was leverage an existing idea built into Gibraltar of grouping messages by a fingerprint – typically the class & caption or the category & caption.   In our experience this tends to be a fair grouping of “sameness” – The captions tend to only include a modest amount of really variable content (like file names & user names) but are unique enough particularly when paired with the location the event happened.  By grouping messages in this way recurring errors that might otherwise flood you are all pulled together and carried as a single alert.  We display the first occurrence in full detail then just let you know how many more times it happened in the same session.

Getting Just the Alerts You Want

It was clear to us from the start that notification couldn’t be as simple as “always send an email when there’s an error”, it was going to have to be more contextually sensitive or it’d be turned off.  We’re highly opposed to features that look great in a demo but not in the real world (e.g. demoware) so we knew it had to be easy to configure to just “do the right thing”.  We also work with enough real world customers to know that it had to be configurable from Analyst.  In many shops the Hub server is run by IT who treats every change to its configuration as a change request.  That barrier, no matter how well meaning, would create enough friction to make notifications less nimble in the real world.

To configure notifications, you just select Notification Options from within Analyst.  As long as you have a Hub connection you can then view & edit all of the notifications and changes will take effect on the server as soon as you click OK.

We wanted the notification rules to be as simple to understand as possible, but we wanted to hit all of the reasonable user scenarios we could.  The basic model used for a notification rule is to define the matching sessions, indicate what type of notification you want, and then where you want it sent. You can pick a single entry from the address book which in turn forwards to a single email address. If you want to sent it to a group of people, just put in the address of the right email distribution list. The address book makes it easy to update a set of destinations in one place, as well as disable HTML formatting for emails if necessary. For example you could create a single address book entry for your issue tracking system with HTML turned off and then one for yourself with HTML turned on.

Finding the Right Sessions

You can match sessions by specifying just the required elements:  Product, Product & Application, Product & Application & Version and separately Computer.  For Version you can match exactly, less than, or greater.

We considered more elaborate version matching capabilities but as we walked through user scenarios we just couldn’t justify the added complexity of the user experience.  In a typical environment either:

1. Version is irrelevant: If you have a web site and aren’t tracking version, then it doesn’t matter at all.  In this case Computer is more important to separate out environments.
2. Production & Development Versions: If you are tracking versions there are usually older version(s) that are in production and then after a certain point all newer versions are development/QA/experimental.  You then are looking to either match production versions or development versions which just requires less than and greater than.

We knew from our customer conversations that some folks were very concerned about warnings, and most were just concerned about errors and crash events (critical messages in Gibraltar).  You can configure how bad something has to be to qualify in the same editor.

The Server that Cried Wolf

There are often computers that you never want to hear from – typically development systems or internal test systems.  You want to be able to run the production code in the production configuration – so they’ll still submit sessions to your Hub – but not want to be bothered when there’s a problem.

To solve this, you can add the computer to the list of suppression rules and that will prevent any related session from generating a notification even if it matches a rule.  This simple approach will get rid of most of the noise alerts.

Avoiding Duplicates

When the server is processing a session it calculates the messages it needs to send based on all of the rules and then sends each unique message to the set of qualifying recipients.  This effectively merges all of the rules together that would cause the same message to be generated.  It then eliminates duplicate destinations by email address (so if two address book entries use the same email address and both qualify for the message only one entry is added).  Finally, since a single message with all of the addresses is submitted to the mail server the mail server can eliminate duplicates once it expands address lists.

All of this is designed to prevent you from receiving multiple copies of the same notification even if you are obliquely included due to distribution lists.

Another way you may get duplicate messages is due to the same session being provided to the hub multiple times.  This can happen for a few reasons, most commonly because it’s being submitted incrementally as problems are happening but the session hasn’t ended yet.

To prevent the notification engine from reprocessing messages it has already examined the Hub server keeps track of exactly what set of messages have previously been checked and ignores them when the session is resubmitted.  This way only new messages are processed.  If you read the resulting alerts carefully you can see when this happens because the notification subject will say that an updated session is available or the error detail will describe “new” occurrences of an error.

Broad Compatibility

Since notifications are primarily about getting information out to other systems and for use in situations when you may not be able to run Analyst it’s more important that they look OK wherever you are than that they look awesome in any particular viewer.  If you’ve ever worked on HTML formatting an email you know the differences between email clients are stunning (and really appalling - You’ll be partying like it’s 1999).  We did a range of testing of each of our HTML formats using:

• Outlook 2003, 2007, 2010 (It’s what we use along with most of our customers)
• Outlook Web Access 2007, 2010 (Also what we use)
• Gmail (second most popular mail client our customers use)
• iPhone Mail

Just between the first three we had major problems to work around – each ignored font sizing or coloring when declared in different ways for example.  Ultimately we found a combination that worked reasonably in all of them.

When we generate each email the HTML version is technically an attachment to the main text version.  This way a client that doesn’t understand HTML formatting will see the text-optimized version and continue unaware of what it’s not seeing.

Just the Text Please

If you are sending emails to an address you know you never want the HTML version for (either because it just doesn’t display well or it just isn’t necessary because you always want it as text) you can disable the HTML version from being generated & attached by clearing the HTML Formatting option in the address book.  The Text versions do more-or-less fixed width text formatting to attempt to create tables of items where necessary and may be parseable if you are feeling frisky.

Get Notifications Today

If you’re using the Gibraltar Hub Service, just download & install the latest version of Gibraltar Analyst and you can start receiving notifications today.  You don’t need to do anything to your deployed application – this change only affects the Hub & Analyst.

If you’re using your own private Hub you’ll need to first upgrade it to the latest Gibraltar Hub before you can configure notifications.  You’ll also need to configure a mail server for your private hub to send mail through – complete details are in the user documentation.  If you or your IT department have any questions just contact our support department via email, we’re here to help.