This issue was discovered while analysing a hung ColdFusion server for a client.  The thread dump showed many threads stuck in the TemplateClassLoader which were waiting on a lock.

The following shows three of the threads.  The two that caused the deadlock and a third that is essentially a victim – any further CFM requests to the server will fall in to this victim category.

WSDL Request – note that 0×10d69910 is locked, also that it is waiting on 0×0f970e48

CFM Request – note it is waiting to lock 0×10d69910, which is locked by the WSDL thread

Finally our victim – this thread isn’t blocked in the traditional sense.  It’s waiting another thread, which has a lock on 0×0f970e48 to “notify” it

It is this victim thread that is interesting.  It’s waiting for another thread to acquire a lock on 0×0f970e48 and call .notify() on the lock object.  Unusually, no thread has a lock 0×0f970e48.  Under normal circumstances it would be the CFM thread (above) that would obtain the lock on 0×0f970e48 and release all the “victim” threads - but it’s deadlocked so it can’t.

Not shown is the thread dump of Web Service requests that were also present.  After investigation these proved to be victims of the deadlock, not contributors.

The key insight in the problem was the realisation that a garbage collection on the permanent generation had occurred shortly before the crash.  To have garbage collection details logged add the following flags to your jvm.config file   -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution

The first step in troubleshooting this situation was to replicate the problem locally.  The first attempt was to simply use JMeter to place some load on the application, requesting CFM pages, WSDL files and Web Service requests.  This proved to be fruitless.  In an attempt to cause a permanent generation collection, as was implicated in the server logs,  I manually called the Garbage Collector while the JMeter load was running.  This too was futile; I could not force a collection of the permanent generation. To monitor the various JVM memory spaces I had JProfiler running

I remembered that clearing the Template Cache caused ColdFusion to reload class files, even if you are not using the Trusted Cache feature.  We can see this occurring in JProfiler – the following is a graph of the size of the permanent generation, while requesting the same CFM file and clearing the template cache.  You can see that each time the template cache is cleared, the used space in the permanent generation increases.  The drop at the end is a Garbage Collection – this shows that the TemplateClassLoader is releasing classes from the permanent generation allowing them to be collected.

To test this, I again ran the same JMeter load as before but this time manually cleared the Template Cache in the ColdFusion Administrator.  Bingo!  Each and every time the server would hang.

The next step was to make a synthetic example – one that did not rely on confidential client code.  The primary reason was to have code I can publish so others in the community could reproduce this bug.  It has also allowed me to further investigate the causes of this issue.

I started by making a simplistic replica of the client’s code.  The main points where that the Web Service and CFM files share common code.  The Web Service has an Application.cfc file with the common code loaded in the OnRequestStart method.  The CFM files have an Application.cfc file with an OnRequest method.

Did you know that OnRequestStart gets called when you request a WSDL?

After a reasonable amount of investigation, the common code between the Web Service and CFM had NOTHING to do with the issue.  Nor did OnRequestStart or anything to do with Applaction.cfc.  It simply comes down to a WSDL request and a CFM request – even if both are completely unrelated – occurring at the same time.

So we know this occurs when the Template Cache is cleared.  We can assume that it can occur at server start-up, although this would be much harder to reproduce.  It is likely it that it will occur in common usage such as when a WSDL is requested for the first time and a new CFM page request occurs at the first time.

But what about our client and the link to permanent generation collection?  The permanent generation collection itself is not a cause, but it is a symptom.  The TempleClassLoader uses Soft References in its data cache – this lets the TempleClassLoader have data in memory, but that data can be Garbage Collected.  What occurred was exactly that.  The TempleClassLoaders cache was garbage collected, causing it to reload classes — this freed classes in the permanent generation which where then garbage collected — the event that I saw initially.

Attached to this post is a zip file containing code that will reproduce the problem.  clearTemplateCache.cfm will clear the template cache using the Admin API – you will need to edit this file and put in your Administrator password.  This file makes it easy to include in a JMeter test.  I’ve included the JMeter test I used, the hostname I used is cfdeadlock, if you use something else change the host name in the “HTTP Request Defaults” element.  Clearing the template cache is set to start 5 seconds after the test starts.

The tests I ran had only 3 threads (concurrent requests).  One CFM, one WSDL and one to clear the Template Cache.  This issue is not related to load.  I have no idea if this issue is present in any other version of ColdFusion – I’ll leave that as an exercise for the community.

Enjoy.

Deadlock Demo Files

As a result of the superior quantity and quality of sites entered, the 14th Annual Webby Awards recognized sites and teams that demonstrated a standard of excellence.

Of the over 8000 entries submitted to the 14th Annual Webby Awards, fewer than 15% were distinguished as an Official Honoree. This honor signifies an outstanding caliber of work.

Dreamworld’s website was one of 12 sites chosen as honorees in the competitions’ Tourism category. Izilla is proud to have worked with Cru and Dreamworld to develop the site and implement it in izilla’s Cognition CMS

Izilla seek a highly motivated, reliable under-graduate to join our Web UI Design and Development team.

It is anticipated that applicants will be studying IT or Graphic Design related subjects

Company

Izilla is one of Australia’s most experienced web solutions companies. Izilla services a growing list of clients nationally across Government, Manufacturing, Engineering, Financial Services and Retail industries.

Position

You will work closely with our Senior UI Developer and Creative Director. The role is primarily UI Development with scope to move into graphic design. While commercial experience is not essential, Ideally you will have several years experience with website Design/UI builds - tinkering with your own site or those of friends, associates etc.

A knowledge of and experience with CSS, XHTML, W3C compliance, Accessibility and JQuery will be highly regarded as will familiarity with Adobe web and design products.

A traineeship may be offered to the successful candidate after a probabtionary period.

Salary

$35K - $45K

Apply Here

http://www.wired.com/underwire/2010/01/i-love-local-commercials/all/1

http://ilovelocalcommercials.com/

We seek a hard working team player to join our web development team as a Developer / Analyst in the company’s Newcastle office

Company

Izilla is one of Australia’s most experienced web solutions companies. Izilla services a growing list of clients nationally across Government, Manufacturing, Engineering, Financial Services and Retail industries.

Position

We are seeking a highly motivated Web Analyst Programmer with good analysis and problem-solving skills. The position will provide an equal-mix of analysis and coding, with a small amount of account management.

Duties and Responsibilities

• Technical development of .net web applications
• Business and technical analysis
• Preparation of Technical and Functional documentation
• Assistance with account management duties

Experience / Qualifications Required

• 3-5 years commercial software development experience with C#.
• Database development
• Web development.
• Technical documentation.

Desired:

• Tertiary qualifications in Computer Science, Software Engineering or similar.
• Microsoft certification

Salary is commensurate with skills and experience in the range of $70k - 80K.

Apply online now

Read the full article in the Adobe Fireworks Developer Center.

As a Fireworks and web standards evangelist, I decided to work with the Fireworks team to help produce an enhanced version of the CSS Export feature, which empowers you to create better, standards-compliant web pages.

Read the full article in the Adobe Fireworks Developer Center.

The Relative Font Size Reference Chart is a simple, customisable table chart that calculates common, relative font sizes in both percentages and ems.

Why is this important? Well, for IE’s font resizing to work, fonts need to be set using a relative size unit.

It becomes indispensable as your stylesheets become more complex, where trying to calculate nested font-sizes can leave you more than confused.

As the common browsers have a built-in default base font size of 16px, we can use this figure to calculate which value to resize an element by to achieve our desired font size.

To use the Reference Chart, find the base font size along the top row, and then find the cell that correlates to your target font size from the left.

For example, to achieve 12px on the body, find the Base Size - 16px, then Target Size - 12px, which gives you 0.75em.

If you then wished to set h1’s to be 20px, the Base Size is 12px, Target Size is 20px, which gives you 1.667em. Too easy!

The Chart also allows you to alter the size range from 1-50px, and choose which unit (% or em) to display the results in.

• View the Relative Font Size Reference Chart
• Download as a zip file to run locally
• Download as an AIR app - requires the Adobe AIR runtime

Related Links

• dekePod Podcast Series | Design — digitalmedia.oreilly.com
• dekeOnline

Related Links

• Kevin Kelly: Author, editor, publisher
• EG 08 - The Entertainment Gathering

Source

• TED.com - Kevin Kelly: Predicting the next 5,000 days of the web