ColdFusion TNT

The 1-Day ColdFusion TNT Audit is the fastest possible way to begin to identify, prioritze and fix the causes of ColdFusion performance and stability problems. Here’s what we deliver in just one day of effort:

• A review of your SeeFusion configuration, in particular query monitoring and database logging settings.
• A review of your JVM configuration, in particular memory utilization settings.
• A review of your ColdFusion log files, stack traces, and any other diagnostic information you’ve gathered.
• A review of your ColdFusion Administrator settings.
• An analysis of your SeeFusion metrics.
• A written report detailing and prioritizing your system’s bottlenecks, complete with extensive data in tables and charts sorted by a variety of methods.
• Fixes for the most severe causes of performance/stability problems (time permitting).

ColdFusion TNT

If your ColdFusion systems are experiencing performance or stability problems, contact us today.

Creating a Maintenance Plan

In SQL Server Management Studio, expand the Management folder, right-click Maintenance Plans and select New Maintenance Plan…

Give the Maintenance Plan a name and click OK. The new empty Maintenance Plan will open in Design Mode. All of the available tasks are in the left hand column Toolbox under Maintenance Plan Tasks.

To add any task to the maintenance plan, simply drag the task from the Toolbox into the design frame. Once the task is in the design frame, double-click to edit.

Reorganize Index Task

The reorganize index task updates table and view indexes so the search order is more efficient. The task dialogue is pretty simple. Use the Database(s) drop down to select one or more databases. Then use the Object drop down to select whether Tables, Views, or Both should have their indexes reorganized. For this example, I’m going to reorganize both Tables and Views. When done, click OK.

Back Up Database Task

The Back Up Database Task creates a Full, Differential, or Transaction Log backup. In this example, I’m going to create a Full backup which will create a *.bak file.  This task dialogue is also pretty simple. Set the Backup type to Full and then select the Database(s) to backup. Jump down to the bottom third of the dialogue and select the Create a backup file for every database radio button. Then browse to the folder where the backup files are to be saved. When done, click OK.

Join the Reorganize Index Task to the Backup Database Task by dragging the green arrow between the two. Once the two tasks are joined, right-click the arrow and select Completion.

Shrink Database Task

The Shrink Database Task should come after the back up task because the shrink cannot be executed on a database until a Full backup has been created. In the task dialogue, select the Database(s) and just accept the defaults. When done, click OK. If you’ve noticed your log files getting out of hand, the shrink task will bring the file size down. In a recent test, my active log file went from over 2GB to just over 200MB.

Join the Back Up Database Task to the Shrink Database Task, and set the type to Completion.

Maintenance Cleanup Task

As the last task in the Maintenance Plan, I’m going to cleanup the backup directory by removing any *.bak files that are older than 2 weeks. If you’re doing this type of maintenance, you’ll want to make sure you’ve got some sort of server backup that keeps a longer history of your *.bak files off-server.

Open the task dialogue and skip down to the Search folder and delete files based on extension.  Navigate to the backup folder that was set in the Back Up Database Task. Then set the file extension to bak. Note the extension is simply bak. You don’t need to include a . (dot). Skip to the bottom of the dialogue and set the age of the deleted files to 2 weeks. When done, click OK.

Join the Shrink Database Task to the Maintenance Cleanup Task, and set the type to Success.

Execute the Maintenance Plan

Save the maintenance plan. From the Object Explorer, right click the new plan and select Execute to test it.  If everything runs without error, you can set the plan to run on a schedule.

On the design mode screen in the Subplan grid, click on the calendar icon and set the schedule for executing the plan. When done, click OK.

That’s it. The plan will now run as a scheduled task. Not only does the plan backup the database, but it’s also reorganizing the indexes, shrinking the database and cleaning up  the backup directory. Best of all, once it’s setup, the plan executes without your intervention and at a time that is best for your system.

Here’s how to use these scripts:

1. Copy the files to solr installation directory. i.e /Applications/ColdFusion9/solr or /ColdFusion9/solr. Change the scripts to have execute permission.
2. Open cf-init-solr.sh and cf-solr-startup and be sure that CF_DIR is set to your coldfusion9 install directory.

From there, run:

sudo ./cf-init-solr install
to install the scripts.

sudo ./cf-init-solr uninstall
to remove the scripts.

All done!

Download the scripts now.

There are still a few $30 tickets left, so register now before the prices go up!

As we all know, businesses and their supporting web applications are constantly changing and adapting. With any luck, change is in the direction of growth, and adaptation simply becomes a component necessary to keep up with that change. As businesses and web applications grow over time and exposure, so does the data. A tool or process in a web application that has been humming along and performing well over the years is subject to turning on a dime and making your life miserable as additional data and concurrent usage flood your application.

I recently fixed a classic example of this for a customer of ours. This particular customer runs (among other things) a pretty well utilized job posting and searching application which sits atop ColdFusion 8 and a SQL Server 2005 database. The site is averaging about 6,000,000 page views a month, so performance gains can be expected to have a broad impact when you factor them by that level of utilization. The customer was reporting all sorts of performance problems and slow running pages within their application but was at a bit of a loss in terms of specifically identifying the problem areas and addressing them. That’s when they called Webapper.

I never begin a performance analysis engagement like this for a customer without first installing SeeFusion. It’s quite simply one of the best tools I’ve ever worked with when it comes to quickly identifying the problem areas and usage trends of a ColdFusion application.

After a day of monitoring the application performance with SeeFusion, it became quickly evident that a very critical page in their application needed some tuning – their “Browse Jobs” page. I could see via the SeeFusion logging repository that this page was continually running long (anywhere from 20-45 seconds), and was accessed very frequently in comparison to other pages on the site. So, I started looking into why this page was causing such headaches for our customer.

The page delivers some pretty simple functionality. It first shows a total number of active jobs available in the application. Then, just beneath that, it generates a list of all 50 states, a count of the jobs listings available in each state, and a parameter-driven link to a second-level page that then displays the jobs for that state. Pretty simple concept. When I accessed the page, the average execution time I was getting was around 24 seconds (24,000 ms) per request. This clearly needed to change.

The first place I typically start when evaluating a page or process like this is with the SQL operations. No matter how clean and optimized your CFML code may be, if your SQL calls are lackluster in the performance department, then it bleeds into everything else. You can’t build a great house on a tattered foundation, right? So I always begin with tidying up the SQL performance before moving on to any potential CFML optimizations.

I could see this page was running 3 varieties of SQL operations. First, a simple parameterless query to the tblJobPostings database table in order to determine the overall number of active job postings available to the entire site. Second, another parameterless query to the tblStates table in order to get a listing of all states. And third, a loop through the query of all 50 states in order to run a by-state query returning the count of jobs in each particular state. Put them all together and you have a total of 53 queries. Now, I immediately jumped out of my seat at the idea of running the by-state job counts from within a loop. But, I stayed focused on first task of SQL optimization before getting sidetracked – I’d get to that part later.

The query running within the loop of all states to get each by-state job count looked something like this.

Seems like a pretty simple query, but looks can be so deceiving when it comes to SQL performance if you’re not looking under the hood. I could already see in the ColdFusion debugging output that the 50 instances of this seemingly simple query were taking about 400-500ms each to execute. Multiply that average execution time by 50 states, and you can quickly see why the page was taking an average of 24,000 ms to render. So, I cut-n-paste an example of this query from the ColdFusion debugging output and brought it over into Query Analyzer in SQL Server Management Studio. I paste the query into Query Analyzer, enabled the “Include Actual Execution Plan” option, and executed the query.

I could immediately see some problems in the execution plan. SQL Server was performing a pretty costly index scan operation and then combining those results with a key lookup operation. This meant that SQL Server was utilizing an available index to match part of the selection criteria presented in the query, but was then having to perform additional work and filtering in order to evaluate other columns that weren’t a part of the index being used. Immediately, I took a look at the indexes on the tblJobPostings table. There was a clustered primary key index and a few single-column indexes, but not one multi-column index. This immediately explained the execution plan I was staring at. So, I created a new index on the table that included state, jobCode, expireDate, and jobPostingID in that order, essentially to match the conditions being applied in the WHERE clause and then including the columns referenced in the SELECT clause. With this approach to indexing, SQL Server should now be able to get all the information it needs for this query from the new index without even directly accessing the table itself.

With this new table index in place, I returned to the web browser and reloaded the page.

Execution time: 12,400 ms

I had already effectively doubled the performance throughput of the page simply by creating that index. But, 12 seconds was still far from being an acceptable load time. So I moved on to step 2 – reorganizing the CFML code.

Experience has taught me that running <cfquery>s from within a <cfloop> is almost never a good idea. Not only can it be a system memory hog, but it’s often a shortcut for not understanding how to get that same data in a single database call. When I had first glanced at this code, I had already made a mental note that I would need to re-write this into a single database call, grouping the job counts by state, and organizing it in a way that I could later reference individual job counts by simply indicating a state.

Re-writing the query was the easy part.

This addressed the issue of getting all the counts in a single database call, but didn’t directly address the problem of how I would go about referencing the job counts from within the <cfloop> through all states. But this is a technique I use all the time, so I quickly moved onward. I had decided that converting this query into a structure would be my best option. I would create the structure with keys for each state abbreviation and a value of the respective job count for that state. The code looks something like this.

The result of this code is a structure that visually might look something like this:

With the data organized this way, I can use the following bit of code to get the specific job count of a given state.

At this point, I was ready to move ahead with adapting the code within the <cfloop> to utilize this new data structure. Here’s a summary of original code.

But with the new single grouped query to get job totals, and the conversion of that query to a structure, I could re-write the code to look something like this.

It’s worth noting that the structKeyExists() call is critical here since the possibility exists that there may be 0 active jobs for one or more states that were separately returned from the states table in qStates. For example, if “TX” exists in the states table, but there are no active jobs in tbjJobPostings for state “TX”, then there will not be a record in the qJobCountsByState query for “TX”. That means a reference to stcJobCountsByState["TX"] would return an error indicating that the specified key does not exist. The structKeyExists() method call addresses this issue.

With this new method of deriving by-state job counts in place, I returned once again to the web browser to see how my work was paying off in the performance department. I clicked the refresh button in my web browser and scrolled to the debugging output.

Execution time: 2,120 ms

Alright, well now we’re getting somewhere. Typical human factors studies of web users would indicate that 2 seconds is reasonable when it comes to perceived wait times for a given page in a web application. However, I knew I could do better than this. I knew that the list of all states wasn’t likely to change very often (if ever), and that the job postings on the site were only updated once a night by a scheduled import process. This means that during the business day when the site is being access by the end users, the data being utilized to render this page isn’t changing. While I’m typically an advocate of always writing your own caching methods in ColdFusion for better control and management, this seemed like an ideal opportunity to take the easier path and simply cache the database queries on this page using the cachedWithin argument of the <cfquery> tag. So, I returned to the code and altered it by adding a 1 hour cache directive to the 3 remaining <cfquery> tags.

With this change in place, ColdFusion will handle all the logic associated with caching and refreshing the cached data from these queries every hour. One request per hour will run the queries against the database, but any other requests in that same hour will be fetching the cached query object from ColdFusion’s in-memory cache. Once again, I returned to the web browser, clicked my refresh button, and scrolled to the debugging output.

Execution time: 24 ms

Bingo! The page rendered almost faster than I could release the browser refresh button. I could see from the debugging output that the page was now referencing the 3 queries from the cache. Granted, once an hour a single request will actually have to run them against the database, but even then the execution time will only take about 2 seconds.

So, from an original page that started off with an average execution time of 24 seconds (24,000 ms), I had worked through a series of optimizations and come to rest on a new average execution time of 24 ms. In the span of roughly 30 minutes, I had created a 1000x increase in the performance throughput of this commonly requested area of the application.

If you have an area of your existing web application that seems to be suffering from some performance challenges, I’d encourage you to have another look and see what you can do about it. Review your SQL statement performance, optimize your database indexes, tweak your application logic, and apply caching where you can afford to do so. If you don’t feel comfortable doing it on your own or simply would prefer to have someone else do the work for you, then pickup the phone, call Webapper, and put our expertise in this area to work for you. The result just might be a 1000x increase in performance.

Foundations of ColdFusion Request Settings

To begin, let’s look at how the ColdFusion settings were configured in earlier versions of application server. With the release of ColdFusion MX 6.0 through versions 6.1 and 7, all editions of the ColdFusion server had one setting for Request Limits. This was referred to as the “Simultaneous Requests” setting. This single setting throttled the number of running requests to be processed concurrently. Should the running pool be fully occupied by requests that are processing but haven’t yet completed, the J2EE server underlying ColdFusion will hold requests in a queued request thread pool that are to be fed to the running request pool.

What’s the Best Setting?

As with most articles you may read on the topic, there is no magic setting for the Simultaneous Requests value that would automatically produce the best throughput. Rather, determining the best value is done as an iterative process by carrying out real world load testing during application development and functional testing in the software development life cycle.

CPU-intensive or Heavy I/O?

ColdFusion applications tend to have a heterogeneous mix of templates that require intense CPU utilization, such as when performing string parsing or calculations in loops, or templates that require little system resources, such as those that are database intensive and spend time waiting on results or those that are I/O intensive such as applications that perform lots of network transactions with CFFTP or CFLDAP. Generally speaking, the Simultaneous Requests value will be higher for I/O-centric applications, and lower for CPU-intensive applications. The default value for the Simultaneous Request setting in ColdFusion MX 7 and earlier was set as 8 by default after installation. Few server administrators changed that value, for better or worse, but it was thought to be a reasonable starting point with the generic guideline of 3-5 per CPU.

One approach to determining the best value for the setting is to perform load testing, perhaps with tools such as the freely available and robust Apache JMeter, or commercially available and highly scriptable products like Paessler Web Stress Tool or Borland Silk Performer. The testing should be carried out on a platform and configuration as close as possible to how it will be run in production. As you perform testing while tracking request throughput, repeat the test iteratively while modulating the Simultaneous Request setting from low to high, then graph the results to find the setting for highest throughput on the curve. Very likely, you will notice CPU utilization will reach a high of 60-80% during the trial that produced the best results. When the CPU is consistently running at higher values, the CPU maybe thrashing which is another way of saying that it is being used inefficiently during the heavy context switching of frequently changing which cpu threads are being executed. Throughput typically declines as the CPU begins to “redline”. Tuning ColdFusion request limits to moderately utilize the CPU as a baseline allows room for occasional bursts of traffic (which can be simulated with ramp testing). An example of a burst might be when users first start the application at 9am during the start of their work day, or when the application is popularized through Digg or other social networking sites.

How Not to Tune The Request Limits

The Simultaneous Request setting has often been misunderstood as the total number of end users to the application that can be concurrently supported. When administrators tune ColdFusion based on that incorrect premise, they will sometimes scale up the value to be the total number persons they expect to be using the application at the same time, and as such they might set the Simultaneous Requests value to an unusually high value such as 100 or even 500. In the majority of all situations, values this high will negatively affect server throughput by thrashing the CPU during heavy context switching, as alluded to earlier.

Types of Requests

So which type of application requests are throttled by the Simultaneous Requests setting? Up to and including ColdFusion 7, all request types are throttled through the same request pool, including ColdFusion CFM template requests, CFC template requests that arrive via HTTP requests such as AJAX applications, Web Service Requests for exposed endpoints, and Flash Remoting Requests such as those from Flash or Flex presentation layers.

Where Do I Find The Setting?

The ColdFusion Administrator page titled Setting exposes this configuration option. The Simultaneous Requests value is directly tied to the JRun activeHandlerThreads value in the JRunProxyService section of it’s corresponding jrun.xml file. For ColdFusion installations deployed to other J2EE containers such as WebSphere or Weblogic, the Simultaneous Request setting does not exist in the ColdFusion Administrator. Refer to the J2EE server documentation for request tuning settings. Commonly on those other J2EE servers, the ColdFusion server must compete with other J2EE applications having additional incoming requests for JSP and Servlets directly. Your tuning strategy will have to consider the total blend of requests across all applications running on that J2EE server.

Current and Future Versions of ColdFusion

With the release of ColdFusion 8 Enterprise Edition a new means of request tuning was introduced by a CF Admin page of the same title, Request Tuning. The ColdFusion 8 Administrator exposes 4 separate ColdFusion pools for granular request tuning. These pools discretely throttle CFM requests, CFC requests, Flash Remoting requests, and Web Service requests. These request pools are downstream of the underlying J2EE server request pool. Note that the CFC throttle refers to CFCs accessed via HTTP directly, from AJAX applications for example.

ColdFusion 8 introduced 4 types of request pools that throttle independently. These pools are downstream of the JRun thread pool.

ColdFusion 8 Standard Edition, however, does not expose these 4 discrete ColdFusion pools, but rather it behaves exactly the same as ColdFusion 7 where there is a single Simultaneous Request setting.

Advantages and Disadvantages of ColdFusion Request Pools

The advantage of ColdFusion 8 Enterprise Edition is that if your application has different entry points such as an AJAX view for smart phones, a Flex view for desktop browsers, and a typical CFM/HTML view for clients not supporting Flash or JavaScript, then you can tune accordingly. If a link to the AJAX view of your application suddenly rises to the top on Digg and receives a burst of traffic, the other Flex view and CFM view may continue to be available without noticeable disruption.

The reason I say may continue to be available is because, again, the ColdFusion request pools are secondary to the underlying J2EE server request pool. In this example, if the AJAX view linked from Digg is wildly popular, then you may receive so many requests that the CFC thread pool becomes full and queues, and then the JRun thread pool begins to queue. Once the JRun thread pool is queuing, then the CFM and Flex views to the application cannot be processed while they’re waiting in the JRun queue behind all the AJAXian CFC requests, even though the discrete CFM and Flash Remoting pool types are empty.

Think of it as trying to get into a fast food restaurant during lunch hour in the city. If there’s a drink line, a sandwich line, and a buffet line inside and all you want is a Coke. If the buffet line is backed up and stretching out the door then you can’t get in to fill your cup even if the drink line is empty.

Come On In!

ColdFusion 8 recognized the potential for the individual request pools to become full, making the ColdFusion Administrator inaccessible. A (partial) solution to this problem is that any request starting with /CFIDE in the script name is permitted to bypass the ColdFusion request pools. This means that if the CFM template pool is full occupied and queuing, a request for the ColdFusion Administrator will still be accessible without queuing. Once again, however, this backdoor access for /CFIDE remains subject to the J2EE server pool and any queuing that may occur there.

A similar weakness exists regarding the ColdFusion Server Monitor should be noted. The UI to the Server Monitor is built from Flex and runs as a Flash application, and the data that is displayed in the UI is obtained via Flash Remoting requests, as observed by its posts to /flex2gateway servlet mapping. Requests to /flex2gateway do not participate in the /CFIDE bypass. Therefore, when the server is under load and the Flash Remoting pool becomes full and queues, then the Server Monitor is no longer able to retrieve data. Additionally, once the Server Monitor is running, if the JRun server is queuing, then again the Server Monitor may not be able to retrieve any data.

The ColdFusion Server Monitor requires a connection through the JRun thread pool to access the CF Flash Remoting

This weakness in the Server Monitor is where Webapper’s SeeFusion monitoring tool shines. SeeFusion is installed as a Servlet Filter and listens on its own port. Requests to the SeeFusion monitoring interface are not subject to the restrictions of the ColdFusion request pools or the J2EE server request pools and is always available regardless of request volume.

Multiserver Considerations

Some admins may choose to install ColdFusion in the Multiserver Configuration to take advantage of JRun’s failover and load balancing abilities, or to take advantage of application isolation since each ColdFusion server instance runs in its own JVM. When tuning the request pools, keep in mind during testing that the additive nature involved of the various request pools. If it was previously determined that a CFM Template Limit should be 8 on single instance of ColdFusion, then if you add more server instances then obviously they cannot all be set to 8. Instead the application should be retested for performance while tuning the various instances at the same time for balance and aggregate throughput. In fact, since each JVM requires some CPU just to manage its heap and thread pools, you may find that the number of total ColdFusion Simultaneous Requests per CPU decrease further. That is to say if you had one instance optimally set to a limit of 8 template requests, and you added another server instance for failover benefits, it would seem logical that you would set the template request limit from 8 down to 4 for each instance, but with the added overhead of an additional JVM you may end up discovering that the optimal throughput occurs with a limit of 3 template requests for each ColdFusion instance.

Other Differences Between ColdFusion 8 and Earlier

Recall that in ColdFusion MX 7 and earlier the Simultaneous Request setting was directly tied to the JRun activeHandlerThreads setting in jrun.xml. If you set ColdFusion to 5, then the whole JRun server had a request limit of 5. With ColdFusion 8, the request tuning pools are subsets of the JRun thread pool, with the default values of CFM/10, CFC/5, Flash Remoting/10, Web Service/5. (These values are held in the ColdFusion configuration file neo-runtime.xml.) The total default value is then 20 ColdFusion requests, and the JRun Master Request Limit for running JRun threads is set to a default of 50 where that value directly ties to the JRun activeHandlerThreads setting in the jrun.xml config file.

Notice that in ColdFusion 7 the total size for the JRun thread pool activeHandlerThreads has a default of 8, and that the ColdFusion 8 default JRun activeHandlerThreads setting is 50, about a 6-fold increase. The overall JRun defaults were increased in ColdFusion 8 based on the performance improvements observed with the upgrade to JVM 1.6 and the other internal ColdFusion server performance enhancements. Its wise to be aware of the difference in these default pool sizes because an upgrade without load testing may result in inadequate default request tuning settings.

As mentioned earlier, on other J2EE server types such as WebSphere, the ColdFusion 7 Simultaneous Request setting is not available, however with ColdFusion 8 the individual CFM/CFC/FR/WS pools are accessible for ColdFusion Enterprise on those non-Adobe J2EE servers.

CFThread

The use of CFThread in ColdFusion 8 applications requires its own discussion apart from what’s covered in this article. Notice that CFThread requests do not arrive via HTTP Requests from end users, and therefore are not subject to the ColdFusion Request Limits. The ColdFusion Administrator provides a separate setting for CFThread. In Enterprise Edition the default value for CFThread limit is 10 but may be increased upwards as needed. In Standard Edition the hard limit for CFThread is 10. It has been suggested in a comment on Ben Nadel’s blog that for applications spawning high volumes of CFThreads that you may need to adjust the jrun.xml JRunSchedulerService setting for activeHandlerThreads, not to be confused with the setting of the same name in the JRunProxyService section. I haven’t yet validated that statement, so you may want to do so yourself. If you begin to tune the JRunSchedulerService settings, be aware that the maxHandlerThreads should not be larger than the activeHandlerThreads setting for the JRunSchedulerService because those threads do not die when idle, so that type of request pool does not shrink, but can only grow until the max limit.

Watch it!

With the addition of these new ColdFusion thread pools in ColdFusion 8, the commandline utility cfstat has been enhanced with 12 new columns to report the new values (including running, queued, and timedout). To see these columns in ColdFusion 8 cfstat, use the -x switch for eXtended metrics, as in cfstat -x 1 to output extended data every second.

Here cfstat reveals 22 running requests total, but only 10 cfm and 10 cfc requests. The extra 2 requests were from accessing the ColdFusion Administrator which bypasses the new ColdFusion request limit pools.

To the Future

In July 2009 Adobe released a public Beta version of ColdFusion Server 9. The Request Tuning settings remains unchanged from as they were in ColdFusion 8 at this time, so this article still applies as of this writing.

Other Facets of Server Performance

Request Tuning is just one aspect of performance tuning considerations. Continue reading the Webapper blog for posts having overviews and recommendations of JVM tuning, and architectural suggestions.

Many thanks for the technical review provided by:

• Andy Allan
• Mike Brunt

SeeDSN is utility that I’ve been using internally with our SeeFusion customers for years in order to assist with configuring JDBC wrapped datasources that can be monitored via SeeFusion.  The utility was originally developed for Webapper by Daryl Banttari, but it was never officially released to the public – until today.  Recently I was on location with a Webapper customer in New Mexico assisting with installation and configuration of SeeFusion when the lead technical resource, Robert Aguayo, suggested the idea of integrating SeeDSN directly into the ColdFusion Administrator interface.  It was Robert’s suggestion that motivated me to revisit the utility, clean up some of the technical aspects, and publicly release it as a ColdFusion Administrator extension for SeeFusion users/customers.

What is SeeDSN?

SeeDSN is a web-based administrative utility for use with SeeFusion.  Using SeeDSN, administrators of ColdFusion instances configured with SeeFusion can easily wrap/unwrap datasources with the JDBC wrapper required for reporting query information within SeeFusion.

Main Features

• One-click functionality of backing up an existing ColdFusion datasource before applying the JDBC wrapper required for proper reporting to SeeFusion.
• One-click functionality of restoring ColdFusion datasources to its original unwrapped configuration and removing the JDBC wrapper.
• Form-driven configuration of additional JDBC connection string arguments relevant to datasources wrapped for use with SeeFusion.
• Ability to verify datasources in the same manner that they are verified from the ColdFusion Administrator Datasources page.
• Integrates SeeDSN and SeeFusion directly with your existing ColdFusion Administrator interface.

Requirements

• ColdFusion 7+
• SeeFusion

Installation

First, you’ll need to download the distribution here (http://www.seefusion.com/seedsn_v1.1.zip).  Once you’ve downloaded it, you’ll need to open it up and follow the steps below.  These directions can also be found in the readme.txt file bundled with the distribution.

1. Copy the “seedsn” folder in this archive directly to your /CFIDE/Administrator/ folder.�
   This means that when you’re finished, you should have a new “seedsn“ folder in your
   /CFIDE/Administrator/ folder (i.e. /CFIDE/Administrator/seedsn/).
2. If you’re using ColdFusion 8+, move “/CFIDE/Administrator/seedsn/custommenu.xml” to “/CFIDE/Administrator/custommenu.xml“.
3. If you’re using ColdFusion 7.*, move “/CFIDE/Administrator/seedsn/extensionscustom.cfm” to “/CFIDE/Administrator/extensionscustom.cfm“.
4. Edit “/CFIDE/Administrator/seedsn/Application.cfm” and replace the value of the “request.urlSeeFusionRedirect” variable with the fully-qualified URL to your existing SeeFusion Flex dashboard interface.

That’s it!  You’ve just installed SeeDSN, and now you can access it directly from your ColdFusion Administrator.

Using SeeDSN

Once you have completed the steps outlined above, you’ll be able to access both SeeDSN and SeeFusion directly from within your ColdFusion Administrator interface.

Simply fire up your preferred web browser and log in to your existing ColdFusion Administrator.  Once you’ve logged in, you should notice a new group of menu options towards the bottom of the left-side navigation menu.

ColdFusion 8+
You should see a new navigation menu section labeled “SEEFUSION” which, when expanded, contains two links.
- “Flex Dashboard” will link to the SeeFusion Flex dashboard as indicate by the URL you supplied during step 3 above.
- “Datasource Wrapper” will link to the SeeDSN JDBC wrapping utility that will assist you with your datasource wrapping needs.

ColdFusion 7.*
You should see a new navigation menu section labeled “CUSTOM EXTENSIONS” which, when expanded, contains two links.
- “SeeFusion Flex Dashboard” will link to the SeeFusion Flex dashboard as indicate by the URL you supplied during step 3 above.
- “SeeFusion Datasource Wrapper” will link to the SeeDSN JDBC wrapping utility that will assist you with your datasource wrapping needs.

We’re very excited to announce yet another big notch in our Web application engineering belt – Perry Woodin joined Webapper early this month. Whereas our recent hire of Steven Erat has contributed greatly to our ColdFusion Tuning & Troubleshooting services, Perry brings an altogether different set of skills to our Application Development services, and will be leading development projects.

Having worked with ColdFusion since version 2.0,  he co-authored “Discovering Fusebox 4 with ColdFusion” while working on a range of business and commercial web applications ranging from eCommerce apps to specialized internal systems for everything from art galleries to industrial manufacturers. Additionally, he has significant Flex development under his belt (his latest application made Laptop Magazine’s Top 50 Web Applications of 2008), which we’ll be putting to good use in very short order.

We couldn’t be happier about adding Perry’s talents to our team of Web application engineers. Welcome Perry!

Ali Daniali has plane tickets in hand now, and we’ll lend a hand setting up the spaces donated by 360|Flex so the coding can begin on Monday, May 18. Coding will continue 24 hours/day until we’re done, and there will be copious amounts of caffeine and food. So bring your laptop and plenty of sleep before the conference, and help make a difference for The Leukemia & Lymphoma Society. Webapper is honored to be pitching in with some dollars to support such an incredible effort in giving back.

Before I go too far into this topic, I should point out that this threat really only exists if you happen to be running ColdFusion on a Windows server.  For those of you who are running on any sort of *NIX-based server, you can move on – nothing to see here.

If you’re still with me at this point, then I need to do a little explaining.

First off, you should be aware of a known issue with ColdFusion on Windows servers as documented in this Adobe TechNote article.  This TechNote basically explains that there’s a bug with certain JVM versions under the hood of ColdFusion that results in the potential for the createUUID() function to actually increase the speed of the system clock within Windows.  Now, this acceleration of the Windows system clock is only likely to occur in situations where createUUID() is “heavily utilized”.  The tricky part here is that I’m not entirely certain how “heavily utilized” is classified as referenced in the TechNote.  Later in this article, however, I will give you an example of a particular customer scenario that was creating this server-crashing behavior as a point of reference.  For now, just note that heavy usage of the createUUID() function might cause the Windows system clock to speed ahead a matter of seconds or even minutes within the span of an hour or two depending the volume with which the function calls are made.

Now, the second vital bit of of information comes down to the inner mechanics of how the createUUID() function actually works.  The function more or less generates a unique hash/GUID based on a combination of the server machine ID, a cryptographically strong random number, and the present date and time.  Now, we all know that calls to the createUUID() function on the same server will all have the same machine ID input value.  And even though the chance is extremely slim, fact remains there’s a fraction of a fraction of a chance that two calls to generate a “cryptographically strong random number” could result in returning the same value.  That’s where the importance of the present date and time consideration made by createUUID() becomes important.  If you think about how quickly many ColdFusion templates execute (quite often literally within milliseconds), you can quickly see how a series of calls to simply get the current server date and time (even if to the precision of milliseconds) might return the exact same date and time.  For this very reason, the createUUID() method makes a call during its execution to another internal method – uniqueTOD().  The purpose of uniqueTOD() is to return a Unique Time Of Day – as the name suggests.  But as we already discussed, it’s possible that extremely rapid calls to return the system date and time may not be separated by more than a millisecond, resulting in a return of the same date and time.  So, in order to ensure that this doesn’t take place, the uniqueTOD() function has been designed as a “synchronized” method.  In the Java world, that just means that the method is single-threaded – it can only be run one invocation at a time, and any attempts to invoke it while it’s already in the process of running will be blocked until execution of the running invocation completes.  In addition to this, the uniqueTOD() also includes a Thread.sleep(1) statement which effectively causes the thread to sleep for 1 millisecond by designating a specific date and time at which to wake and resume execution.  The combination of these two concepts ensures that dates and times returned by the uniqueTOD() function will always be separated by at least 1 millisecond.

If you need proof that this takes place, take the following bit of sample code, toss it on your server, enable server debugging output, point your browser at it, and take a look at the execution time spent.

<cfloop from="1" to="1000" index="i">
    <cfset x = createUUID()>
</cfloop>

When you run this code, you’ll quickly notice that the execution time takes roughly 1 second or 1,000 milliseconds in direct correlation with the to argument value of 1,000 for the cfloop.  This correlates directly to the 1ms sleep time being incurred with each of the 1,000 calls within the loop to createUUID().

So now that we understand all those mechanics, it’s time for me to unveil where this might become a server killing problem.  There’s a handy bit of operating system housekeeping that most of us take completely for granted these days – automatic internet-based time synchronization.  Consider the following scenario with me.

• Your server starts off at 1:00:00.000pm with a date and time that are synchronized to an internet time server of some sort.
• Your ColdFusion application makes “heavy utilization” of the createUUID() function over the course of 2 hours, gradually advancing your server clock by 200 seconds or so as a result of the behavior/bugs I’ve explained above.
• Your ColdFusion server is in the middle of a series of high-volume calls to the createUUID() function.
• Your server makes a call to the configured internet time server, realizes the system clock is currently running about 200 seconds fast, and resets the system clock appropriately to 3:00:00.000pm.
• As the system clock was turned back to the proper time, a single internal call to the uniqueTOD() function had already assigned a “wake time” of 3:03:20.001pm (based on having previously been 200 seconds ahead).

In the scenario outlined above, you now have a createUUID() call waiting on an internal uniqueTOD() call that won’t actually complete execution until 3:03:20.001pm.  Prior to the call to synchronize the system clock with the internet time server,  this would have only been 1 millisecond away.  But now that the server clock has been corrected, we’re suddenly 3 minutes, 20 seconds, and 1 millisecond away from that “wake time”.  And what makes matters worse is that the waiting uniqueTOD() invocation is also synchronized.  This means that all other invocations to uniqueTOD() or to any other functions that include the use of uniqueTOD() must now sit and wait until the running/waiting invocation “wakes” and completes at 3:03:20.001pm.

Wait a minute.  What’s that I hear in the background?  Oh, yes… that’s the sound of thread activity grinding to a halt on your production server as threads are suddenly being blocked for 3 minutes, 20 seconds and 1 millisecond until that uniqueTOD() invocation finally wakes and completes in order to generate that UUID that was requested.

I actually tracked this situation down for a Webapper customer who had been experiencing frequent yet unpredictable “mystery crashes” of their ColdFusion instances.  Our customer was very technically proficient and had already looked under most of the “rocks” we would typically turn over in this sort of situation.  During our initial kick-off call with the customer, we were all in consensus that this was likely a result of some sort of database transaction locking and blocking that we’d need to track down.  After a little investigation and profiling on the servers in question, I realized that was simply a red herring.  With the help of SeeFusion, we were able to snag stack traces from the server as the crashes were actually taking place and quickly determine that the entire system was hanging up WAY before ever getting to any database activity.  In fact, the application seemed to be hung on the first line of their Application.cfm file – the <cfapplication> tag.

That’s when it hit me.  The customer’s application was architected in such a way that there were very high-volume rapid-succession web service calls being made to a ColdFusion component.  By high-volume rapid-succession, I specifically mean 2 or 3 calls every 50ms.  That component, by design, was including the root Application.cfm file.  And the <cfapplication> call was hanging because the clientmanagement argument had been set to “true” and the customer had enabled the “Use UUID for CFTOKEN” checkbox in the ColdFusion Administrator.  Each web service call was resulting in the assignment of a new CFID/CFTOKEN pair – an entirely separate blog-worthy issue.  Suddenly, the whole situation made perfect sense – well, at least in theory.  The real test, of course, would be to have the customer uncheck the “Use UUID for CFTOKEN” setting and see if the “mystery crashes” came to an end.  I’m happy to report that they haven’t had a system hang since disabling that ColdFusion setting.

So, if you have any code in your server that might be creating opportunities for high-volume utilization of the createUUID() function, then you just might be one internet clock synchronization cycle away from a complete ColdFusion meltdown.