Blogging Techstacks

Another Bling Release: v0.4.1 - So Long, Technorati

2009-11-07T21:51:08-05:00

I'm releasing a new version of Bling. The only change is the removal of the Technorati ping, since this doesn't work any more.

According to their developer page, all (all?) previously published developer APIs ceased being available on October 25th, 2009. Now I am sure that many of you running blogs out there were able to get some useful and interesting data from these APIs and maybe some of you are even bummed that they are gone but there is at least one bitter semi-professional blogger contributing posts on this site who still hasn't been able to attract Technorat's spider, let alone the attention of their support staff.

Back when this site first went live, the first thing I learned as a budding new blogger was to get listed on Technorati. **Everyone** who had a blog needed to be listed on Technorati. Tagging your pages with Technorati-specific links was one of the first things you were supposed to do. Heck, people even developed lots of cool widgets and applications that would create those tags for you. Technorati had ping buttons available everywhere. Then something happened. Sometime within the last 18 months or so, Technorati started experiencing lots and lots of problems. Blog claims wouldn't work. Searching would fail. You couldn't even load your profile at times. What I suspect that happened was that instead of confessing to twitter-like outages and documenting every little thing on some Technorati status site, Technorati stayed online by ignoring the small sites. Support requests would go unanswered. Pings, although accepted, would never result in an actual crawl by their searchbot. Then a couple months ago, a new Technorati launches. Pings don't work. APIs are deprecated and then retired wholesale. The new site does...something.

If anyone hits this page via search engine query and you are trying to find out if Technorati works any more, the answer from me is "No. It doesn't work. It never really worked for me. My site profile never updated. I never saw their crawler unless I deleted and recreated my blog claim. I would almost state that for all I know, my Technorati listing is actually hurting me but I won't because I really have no clue what they do. I'm not even sure why I'm still tagging my pages with technorati tags. Why am I doing this?"

WasaLive and BlogBuzzMachine may have to go to in the near future. The two of them worked great until I released v0.3 and I guess they weren't ready for the 10's (yes, I meant "tens") of new bloggers running xmlrpc pings with this script. Although truthfully, I need to add a debug option to the script in order to display the raw output for failed pings to really see why they are failing.

The new version is available on the downloads page. The main bling page has been updated as well. The full source is available in the download (which mostly includes lots and lots of comments). The source, without the 60+ lines of comments documenting change history at the beginning of the script, is below:

#!/usr/bin/env groovy

import groovy.net.xmlrpc.*
import groovy.util.slurpersupport.GPathResult

// You'll want to make this next section your own
def blogTitle = "YOUR_BLOG_TITLE_HERE"
def blogURL = "YOUR_BLOG_URL_HERE"

// Set up a map (hash) of popular rpc endpoints
// It is too bad for my syntax highlighter, but some of 
// the blogs containing periods in their names required
// placing them in quotes. 

def trackbacks = [
 Google:'http://blogsearch.google.com/ping/RPC2',
 Weblogs:'http://rpc.weblogs.com/RPC2',
 FeedBurner:'http://ping.feedburner.com/',
 Moreover:'http://api.moreover.com/RPC2',
 Syndic8:'http://ping.syndic8.com/xmlrpc.php' ,
 BlogRolling:'http://rpc.blogrolling.com/pinger/',
 NewsGator:'http://services.newsgator.com/ngws/xmlrpcping.aspx',
 Bloglines:'http://www.bloglines.com/ping',
 'Blo.gs':'http://ping.blo.gs/',
 BlogCatalog:'http://rpc.blogcatalog.com/',
 PubSub:'http://xping.pubsub.com/ping/',
 'MyBlog.jp':'http://ping.myblog.jp/',
 Goo:'http://blog.goo.ne.jp/XMLRPC',
 BlogPeople:'http://www.blogpeople.net/servlet/weblogUpdates',
 Twingly:'http://rpc.twingly.com/',
 Spinn3r:'http://rpc.spinn3r.com/open/RPC2',
 PostRank:'http://api.postrank.com/v2/ping',
 WasaLive:'http://www.wasalive.com/ping/',
 BlogBuzzMachine:'http://rpc.blogbuzzmachine.com/RPC2',
 IceRocket:'http://rpc.icerocket.com:10080/',
 FeedBlitz:'http://www.feedblitz.com/f/f.fbz?XmlPing',
 ]

// Set up canned responses to make the outputted responses nicer.
// Previously, the output used the literal response from the endpoint
// which did not look all that nice in a terminal window.
def weal = "Thanks for the ping!"
def woe = "PING ATTEMPT FAILED."

// IceRocket is separate because their ping method call
// is different from everyone else's--go figure...


// Here is the section responsible for iterating through each ping
// url in the trackbacks map.

println "====PING RESULTS===="

trackbacks.each {
 try{
  def url = it.value
  def proxy = new XMLRPCServerProxy(url)
  response = proxy.weblogUpdates.ping(blogTitle, blogURL)
   response.data instanceof GPathResult

  if (!response.flerror)
    println " ${it.key}".padRight(25) + "${weal}"
   else
    println " ${it.key}".padRight(25) + "${woe}"

  }catch(ConnectException ex) {
   println " ${it.key}".padRight(25) + "${woe}"   
  }catch(IOException ex){
   println " ${it.key}".padRight(25) + "${woe}"
  }
}
println "===================="

Testing for Trace and Track

2009-11-04T09:52:03-05:00

I am modifying cryptonark to report on more commonly-reported PCI findings than just SSL cipher strength. Another popular issue reported during PCI Scans is the "Web Server HTTP Trace/Track Method Support Cross-Site Tracing Vulnerability". The wording of this particular error has been a small thorn in my side mainly because "Trace/Track" can, and has, been interpreted as meaning "trace and track", "trace or track", or "trace and/or track". The modifications I've made to this script proved to be so useful, I thought breaking it out and sharing it might be useful to others as well.

The truth is, with most modern web servers and application servers, this vulnerability notification really means: "trace is enabled unless you're running a fairly old version of IIS--like IIS4--then it is **probably** track".

The script below uses Perl 5.10 and LWP::UserAgent. I've already got Net::SSLeay and Crypt::SSLeay installed, so I did not need to declare or install them in order for LWP to be able to connect to ssl-encrypted web sites. The script is a little bit cooler than others that I have written because not only does it attempt an HTTP Trace request and report back success, failure, or other information but it also tests for HTTP Track. The script right now only assumes that you'll be testing against a well known ssl port, unfortunately, so your web server will need to be running ssl on port 443. Otherwise, the port is assumed to be an HTTP port. It should help you satisfy those security and compliance administrators who demand proof that both methods are disabled, (even though TRACK does not really exist anywhere any more).

#!/usr/bin/perl 
#===============================================================================
#
#   FILE: test4trace.pl
#
#  USAGE: ./test4trace.pl <host> <port>
#
# DESCRIPTION: The "UI" for this script may not
#    make sense but this is a sub
#    routine of a future version of my
#    cryptonark script that tests for
#    the existence of the TRACE method
#    on a web site. That's why,
#    syntactically, this script follows
#    cryptonark's syntax and version
#    numbering.
#
#  OPTIONS: ---
# REQUIREMENTS: Perl 5.10 and LWP::UserAgent
#   BUGS: None Found Yet
#  NOTES: ---
#  AUTHOR: Chris Mahns
#  COMPANY: Blogging Techstacks
#  VERSION: 0.3
#  CREATED: 11/03/2009 21:45:30
#  REVISION: ---
#===============================================================================

use strict;
use warnings;
use feature ':5.10';

use LWP::UserAgent;

my $host = $ARGV[0];
my $port = $ARGV[1];
my $scheme;

my $help = "Usage: $0 <hostname> <port>";

if ( !@ARGV ) {
 print $help . "\n";
 exit 0;
}

if ( $port == 443 ) {
 $scheme = 'https';
}
else {
 $scheme = 'http';
}

sub test_for_trace {
 my $url = "$scheme://$host/";

 my $method = "TRACE";
 my $ua  = LWP::UserAgent->new;
 $ua->timeout(10);
 $ua->env_proxy;
 $ua->agent('test4trace-pci-auditor/v0.3');
 
 my $request = HTTP::Request->new( $method => $url );
 $request->header(Header0 => "TRACE");
 $request->header(Header1 => "Test");

 my $response = $ua->request($request);

 given ( $response->code ) {
  when (200) {
   say "======this is what you sent======";
   say $response->content;
   say "=================================";
   say $method, " is enabled and working.";
  }
  when (301) {
   say "Redirect present. Retry request against ",
    $response->header('Location');
  }
  when (302) {
   say "Redirect present. Retry request against ",
    $response->header('Location');
  }
  when (307) {
   say "Redirect present. Retry request against ",
    $response->header('Location');
  }
  when (403) {
   say $response->status_line;
   say $method, " is forbidden.";
  }
  when (404) {
   say $response->status_line;
   say "This is an unexpected response";
  }
  when (405) {
   say $response->status_line;
   say $method, " is not permitted.";
  }
  when (501) {
   say $response->status_line;
   say $method, " is not implemented.";
  }
  default {
   say $response->status_line;
  }
 }
}

sub test_for_track {
 my $test = 'This is an HTTP TRACE test.';
 my $url = "$scheme://$host/";

 my $method = "TRACK";
 my $ua  = LWP::UserAgent->new;
 $ua->timeout(10);
 $ua->env_proxy;
 $ua->agent('test4trace-pci-auditor/v0.3');
 
 my $request = HTTP::Request->new( $method => $url );
 $request->header(Header0 => "TRACK");
 $request->header(Header1 => "Test");

 my $response = $ua->request($request);

 given ( $response->code ) {
  when (200) {
   say "======this is what you sent======";
   say $response->content;
   say "=================================";
   say $method, " is enabled and working.";
  }
  when (301) {
   say "Redirect present. Retry request against ",
    $response->header('Location');
  }
  when (302) {
   say "Redirect present. Retry request against ",
    $response->header('Location');
  }
  when (307) {
   say "Redirect present. Retry request against ",
    $response->header('Location');
  }
  when (403) {
   say $response->status_line;
   say $method, " is forbidden.";
  }
  when (404) {
   say $response->status_line;
   say "This is an unexpected response";
  }
  when (405) {
   say $response->status_line;
   say $method, " is not permitted.";
  }
  when (501) {
   say $response->status_line;
   say $method, " is not implemented.";
  }
  default {
   say $response->status_line;
  }
 }
}
say "First we test for Trace...";
sleep 2;
test_for_trace();

say "\nNow we test for Track...";
sleep 2;
test_for_track();

Bling v0.4 Released - Blog Ping Tool

2009-10-28T06:23:05-04:00

I'm releasing an updated version of bling today. Written in groovy, the changes and additions made in this version are as follows:

Added 2 Additional Services: BlogBuzzMachine and FeedBlitz.
IceRocket is no longer managed separately since using the same XML-RPC calls that everything else uses also now seems to work for IceRocket. Right now, the section of the script that contained the separate IceRocket calls has been commented out and will be removed entirely in the net release.
As mentioned above in the version 3.0 section, WeBlogALot has been removed. In the time since version 3.0 was released, I never got a successful ping.
The spooky and primitive output format utilizing whitespace and tabs has been replaced with padded output. Thanks again to Mr.Haki for publishing his post about Padding Strings in Groovy.
Finally, fixed a potential bug in the !response.flerror section, where !response was actually set as !resp.

With this script, after modifying it to include your Blog Title and Blog URL, you can ping 22 blog directories, news aggregators, and other blog services. This script is a bit different from all the other posts out there in that I've actually tested each endpoint and the ones included in bling seem to work.

The main bling page provides all the documentation and the main downloads page has been updated. Source is below.

#!/usr/bin/env groovy

import groovy.net.xmlrpc.*
import groovy.util.slurpersupport.GPathResult

// You'll want to make this next section your own
def blogTitle = "YOUR_BLOG_TITLE_HERE"
def blogURL = "YOUR_BLOG_URL_HERE"

// Set up a map (hash) of popular rpc endpoints
// It is too bad for my syntax highlighter, but some of 
// the blogs containing periods in their names necessitated
// placing them in quotes. 

def trackbacks = [
 Google:'http://blogsearch.google.com/ping/RPC2',
 Weblogs:'http://rpc.weblogs.com/RPC2',
 FeedBurner:'http://ping.feedburner.com/',
 Moreover:'http://api.moreover.com/RPC2',
 Syndic8:'http://ping.syndic8.com/xmlrpc.php' ,
 BlogRolling:'http://rpc.blogrolling.com/pinger/',
 Technorati:'http://rpc.technorati.com/rpc/ping' ,
 NewsGator:'http://services.newsgator.com/ngws/xmlrpcping.aspx',
 Bloglines:'http://www.bloglines.com/ping',
 'Blo.gs':'http://ping.blo.gs/',
 BlogCatalog:'http://rpc.blogcatalog.com/',
 PubSub:'http://xping.pubsub.com/ping/',
 'MyBlog.jp':'http://ping.myblog.jp/',
 Goo:'http://blog.goo.ne.jp/XMLRPC',
 BlogPeople:'http://www.blogpeople.net/servlet/weblogUpdates',
 Twingly:'http://rpc.twingly.com/',
 Spinn3r:'http://rpc.spinn3r.com/open/RPC2',
 PostRank:'http://api.postrank.com/v2/ping',
 WasaLive:'http://www.wasalive.com/ping/',
 BlogBuzzMachine:'http://rpc.blogbuzzmachine.com/RPC2',
 IceRocket:'http://rpc.icerocket.com:10080/',
 FeedBlitz:'http://www.feedblitz.com/f/f.fbz?XmlPing',
 ]

// Set up canned responses to make the outputted responses nicer.
// Previously, the output used the literal response from the endpoint
// which did not look all that nice in a terminal window.
def weal = "Thanks for the ping!"
def woe = "PING ATTEMPT FAILED."

// Here is the section responsible for iterating through each ping
// url in the trackbacks map.

println "====PING RESULTS===="
trackbacks.each {
 try{
 def url = it.value
 def proxy = new XMLRPCServerProxy(url)
 response = proxy.weblogUpdates.ping(blogTitle, blogURL)
 response.data instanceof GPathResult

 if (!response.flerror)
 println " ${it.key}".padRight(25) + "${weal}"
 else
 println " ${it.key}".padRight(25) + "${woe}"

 }catch(ConnectException ex) {
 println " ${it.key}".padRight(25) + "${woe}" 
 }catch(IOException ex){
 println " ${it.key}".padRight(25) + "${woe}"
 }
}
println "===================="

SpringSource Releases tc Server Developer Edition Preview

2009-10-21T23:08:33-04:00

I just downloaded it myself and watched the demo and tc Server Developer Edition actually looks pretty cool. It integrates tomcat 6.0.20 and an application health and application monitoring tool called Spring Insight, which lets you view in real time a bevy of application and server-related statistics. Personally, I think developers are really going to love this but this product is not meant for production deployments.

The entry on the SpringSource Team Blog provides an introduction to tc Server Developer Edition. There is also a short screencast that provides an introduction to the bundled Spring Insight tool.

Currently a preview release, the cost to get access to SpringSource tc Server Developer Edition is some basic registration information. Download it when you can. If you are in operations and/or infrastructure, this will also give you an introduction to the tc Server product and provide some intel on how the product differs from a standard tomcat distribution.

I still have no idea how much SpringSource charges for tc Server. If anyone out there uses it, let me know and also let me know if you have any opinions on it that we can share.

Sun Publishes Glassfish vs. JBoss Comparison Guide

2009-10-20T16:58:50-04:00

Poor Redhat. First SpringSource declares war and now Sun is joining in. Today Sun made available a 14 page Glassfish vs. JBoss Comparison Guide, (Sun login required to download). Check it out. It actually makes for pretty interesting reading and if I was eager to see Redhat/JBoss's response to SpringSource, they've now got me besides myself with excitement waiting to hear how they respond to both SpringSource and Sun! It's going to be epic!!

Help Wanted: Generating Remote Thread Dumps for Large Numbers of Servers

2009-10-08T20:31:37-04:00

A recent commenter on this site posted a request for assistance on generating thread dumps remotely, quickly, across a large number (30+) of servers, whenever an incident occurs. I thought it would be a good idea to solicit ideas from anyone reading this site for ideas on how you might be doing it.

When a failure occurs within their web application, he must remotely generate thread dumps across 34 load-balanced jboss systems and restore service as quickly as possible. Since I've been experimenting a lot with Jmx4Perl and we are talking specifically about JBoss servers, scripting something in Jmx4Perl that would iterate through a list of servers would certainly be worth a try since Jmx4Perl has built-in methods that would help. Is anyone out there doing something similar today? JMX-Console is too slow for his purposes and depending upon the failure, both jmx-console and jmx4perl might not be all that usable, (if for example, the http connector is dead).

I would love to hear what other people are doing managing large clusters of JBoss systems, especially in the area of remotely generating thread dumps. Please brag about it in the comments! :) Let's assume, for the solution's sake, that we do not know 'why' he has to remotely generate thread dumps across this many systems.

Apache HTTP Server v2.2.14 Released

2009-10-05T20:08:25-04:00

The Apache HTTP Server project has released version 2.2.14. The release fixes three security vulnerabilities and 11 bugs and renders thousands of web sites out of PCI Compliance. (ok, ok, technically you are not out of compliance instantaneously)

Apache on Solaris and mod_proxy_ftp users seem to be directly impacted by the vulnerabilities fixed in this release. The CHANGELOG goes into more detail on the fixes and updates addressed in this release. Downloads are at the usual location.

New HOWTO Article on Setting Up JSessionID Persistence on a BigIP

2009-10-04T18:30:57-04:00

This article serves more as a reminder for me than anything else, since I'm always forgetting the last step. However, if it serves useful to anyone else out there...great!

One of the first things I wanted to tackle when we got our spiffy new BigIPs in several years ago was persisting user sessions to tomcat or jboss application servers using iRules. There are lots of examples of persisting traffic based on JSESSIONID, ASPSESSIONID, etc., but one thing that the budding application traffic manager may not necessarily realize from browsing CodeShare on F5's DevCentral site (login required) is that you don't (always) simply code an irule and attach it to your VIP. There are a couple steps necessary and this new HOWTO article outlines those steps.

Go to HOWTO: Set Up JSESSIONID-Based Persistence on a BigIP now.

Groovy 1.6.5 is out

2009-10-02T11:07:13-04:00

Announced earlier today on the mailing lists, Groovy 1.6.5 is out. Source and binaries are available at the codehaus download site and release notes are available as well.

There are no new features but there are quite a few bug fixes.

Tomcat Management: Basic Tomcat Interrogation Using Jmx4Perl

2009-09-28T23:18:11-04:00

Many of us have scores, hundreds, or more tomcat servers sitting on our network and sometimes (often, for me) the naming conventions we might use for our servers are not necessarily descriptive enough for us to figure out what applications that server might be running.

I first started writing about the goodness that is Jmx4Perl a couple weeks ago. I've been playing with it off and on ever since and I have not lost my initial favorable impressions. I thought it would be a useful learning exercise writing some basic script that I would use on a semi-regular basis at work—I do not necessarily care to write something that I won't ever use again once I've been able to successfully execute it. Since I have plenty of tomcat servers on my network now, things have gotten to the point that if someone in our NOC reports an issue with server297, a simple running of the following script is enough to freshen my memory as to what applications this server is hosting. Heck, it may even turn out to be something that will grow into a full server interrogation script in the future, like cryptonark did.

Below is a basic Jmx4Perl server interrogation script I'm calling sooth.pl. It's designed so that you execute sooth.pl $host $port and it will tell you the Tomcat version string and will print out all the webapps it finds. The script calls a couple of the actual mbeans and attributes, to demonstrate syntax when no suitable alias exists. I'd like to thank Roland Huß for publishing such a useful module that lowers the learning curve for folks like me who want to take advantage of the utility that JMX provides but who don't necessarily have the time to invest in doing it the "Java-way" right now.

#!/usr/bin/env perl 
#===============================================================================
#
#         FILE:  sooth.pl
#
#        USAGE:  ./sooth.pl $host $port  
#
#  DESCRIPTION:  Demonstrates Jmx4Perl.
#
# REQUIREMENTS:  JMX::Jmx4Perl
#        NOTES:  ---
#       AUTHOR:  Chris Mahns 
#      COMPANY:  techstacks.com
#      VERSION:  0.1
#      CREATED:  09/06/2009
#===============================================================================

use strict;
use warnings;
use feature ':5.10';

use JMX::Jmx4Perl;
use JMX::Jmx4Perl::Alias;

my $host = $ARGV[0];
my $port = $ARGV[1];

my $usage = "Usage:  $0 <host> <port>";

if ( !@ARGV ) {
  say $usage;
  exit 0;
}

my $jmxconn = JMX::Jmx4Perl->new(url => "http://$host:$port/j4p");
my $version = $jmxconn->get_attribute( {
  mbean => "Catalina:type=Server",
  attribute => "serverInfo",
  } );

my $contexts = $jmxconn->get_attribute( {
  mbean => "Catalina:port=8080,type=Mapper",
  attribute => "contextNames",
  } );

say "";
say "$host is running: $version";
say "";
say "The following webapp contexts were found";

foreach (@$contexts) {
say $_;
}