Running a site like Codehaus presents many challenges; some are solved easily; some are not.

One of the recurring themes is the necessity of dealing with spam.  Since I doubt that any of the moronic spammers that hit our site read our tech articles, this article shouldn’t open us up to additional risk.

Spam hits almost every one of our services

* email – 100k spam per day – we block with a variety of techniques

* web – referrer spam – we ignore / block prolific spiders

* confluence – profile spam; comment spam

In this post we’ll look at Confluence spam and how we deal with it; and how YOU as a Confluence user can help ensure that the site remains spam free.

Like most internet systems we use defence in depth against spam.

The techniques shown below can be applied to almost any 3rd party (or inhouse application) without modification to the core product.

Defence 1: IP addresses

As per the Atlassian guidelines for JIRA – http://svn.atlassian.com/svn/public/contrib/jira/spamfighting/spammers/blockspammers.rc – we reprocess the block file using a basic Ruby script to convert the IP tables blocklist into an HTTPD blocklist.

The spammers.conf is then “included” in any of our sites that needs to be protected from spam – which is most of them.

See update-blocklist

We also have a second list of spammers that have harassed our sites. We also include comments as to when and why a block is added, in case we unintentionally block legitimate users when we block Class A/B/C networks.

order allow,deny
deny from 91.212.226.0/24 #spammed ActiveIO 20070105
deny from 89.120.0.0/16
...
allow from all

Defence 2: Prevent spam user names

Since prevention is far better than a cure, we then prevent zombie machines from creating users with typical spam names. This is a never ending battle of course – but since we are not a high priority target (we hope) – we can get away with pretty simple defences.

Since Confluence does not support a blacklist for names, we use an insert trigger on the “USERS” table in the PostgreSQL database.  This trigger prevents the most obvious of spam accounts from even registering.  It does cause an ungainly error 500 in the web interface – but most spam bots won’t be put off by that.

user_insert_trigger

The above trigger has been pruned to keep it brief – we have more rules in our live triggers.

Defence 3: Destroy spam user names

As with all things; a static defence is usually defeated over time. We supplement our user name trigger with a script to lookup usernames that match a variety of more vague rules.

The use of broader rules in this phase is intentional – removing accounts is less problematic than blocking users who may trigger a spam rule by accident.

e.g. we need to watch out for things like “cialis” matching “specialist” – if the user has authored content, then it doesn’t really matter though (as they won’t be removed in the script below)

This will take a while as it scans all the profile content

SELECT DISTINCT C.username
  FROM CONTENT C, BODYCONTENT BC
 WHERE C.content_status = 'current'
   AND C.contenttype = 'USERINFO'
   AND C.username NOT IN ('good-user') -- the whitelist
   AND BC.contentid = C.contentid
   AND LENGTH(BC.body) > 100
   AND (
     BC.body LIKE '%porn%' OR
     BC.body LIKE '%cialis%' OR
     BC.body LIKE '%incest%' OR
     BC.body LIKE '%viagra%'
    );

The result is a list of users who may / may not be spammers (but on the balance of probabilities they are):

• doctormexico
• parishiltonsuck
• from-paris-hilton-bff-naked
• paris-hilton-video
• wild-things-clip-denise-richards

Defence 4: Dealing with the junk users

We then use an automated script with all the user IDs that have been gathered in our previous layers and do a top level user removal.

This works well because only users that have not authored content will have their accounts removed.  This can inconvenience the occasional user with an odd name that hasn’t authored content, but on the balance of probabilities it achieves the end-goal.

The script can be found at https://svn.rubyhaus.org/confluence4r/trunk/examples/remove-users.

The script is coded in Ruby (because that’s how we roll); and uses our opensource confluence4r library for interfacing with Confluence.

Simply feed it a set of parameters like:

./examples/remove-users http://confluence.example.com admin password spammerlist.txt

The script will connect to Confluence, authenticate, and then attempt to remove all the listed users.

It would be trivial to write an XML-RPC client that does the same thing in fewer lines of code; but the script shown above is similar to a lot of other internal management scripts.

Defence 5: Whitelist group

Within Confluence we have a group CAPTCHA-FREE – they are not subjected to a CAPTCHA when adding comments. All users registered by Xircles are added to this group; and there is an interface inside Xircles to add extra users.

We also have a group “codehaus-whitelist”; users in the codehaus-whitelist are not affected by various anti-spam activities. This is required for the odd occasion where a user matches a rule; but is not a spammer. We can quickly add these users to the whitelist so that our process remains clean and thoughtfree.

Defence 6: Proactive Users

If you see spam in Confluence / JIRA, please let support know as we can’t be everywhere at once.

Just send the offending URL and any other information as required.

Defence 7: The Future

We frequently have to revisit our approaches – spammers are persistent lot – and Confluence / JIRA are likely to attract specialised attention from them in the future.

Tweet This Post Delicious

Just a quick post today (as opposed to no post at all!)

Yesterday we deployed the latest version of Confluence to http://docs.codehaus.org/ (this was preceded by an upgrade to the latest stable PostgreSQL as well).

The result of this is that we are now running the latest and greatest Confluence (3.0.1) – and you now have access to the features contained within.

New Features

If you’re interested in the new features in Confluence 3; then head on over to Atlassian’s documentation.

• What’s new – http://www.atlassian.com/software/confluence/whats-new.jsp
• Learn more – http://www.atlassian.com/software/confluence/learn.jsp
• Documentation – http://confluence.atlassian.com/display/DOC/Confluence+Documentation+Home

Activity Streams

Activity streams are only partially enabled at present. We are looking at ways of integrating activity streams across the Codehaus platform (i.e. covering all our platform – and not just the Atlassian parts).

Support

The usual story applies to support – head on over to http://codehaus.org/support/ for more information on support channels.

Tweet This Post Delicious

If you’re just looking for the URL, here it is – http://bamboo.ci.codehaus.org/

History

In the very early days (pre 2004), there was a server hogshead (in fact there still is). Hogshead provided all services for Codehaus; although there were only a handful of projects and none of them had a core requirement for continuous integration.

In the 2005-2006 era, beaver (at Sentex) was provisioned, and a handful of projects did unauthorised CI style builds on the machine.  As this took out core services, the users doing builds were summarily executed.

Let us not speak of the hard drive crash of May 2006.

With the migration to managed hosting at Contegix, we no longer had infrastructure that could safely run end-user jobs. SimulaLabs provided a machine called cheddar to perform CI builds. A variety of CI tools were used on this machine continuum, the Andy Pols precursor to Bamboo (whose name eludes me at present) and finally early versions of Bamboo.

Then cheddar was relocated to a new data centre without warning; and ultimately switched off without warning.

This was untenable and Contegix came to our aid and helped us with codehaus04 (finally we have sensible names!).

codehaus04

codehaus04 is a 4 core machine (Intel(R) Xeon(R) CPU @ 1.86GHz) with 4G of physical memory, 130G of disk and access to the Codehaus internal network for moving data around.

Bamboo

Bamboo can be viewed at http://bamboo.ci.codehaus.org/

We can’t keep up with the Bamboo documentation, so it is recommended that you take a look at the Atlassian Bamboo documentation.

Access

If you require access to create a plan; and you are a project despot; simply raise a JIRA requesting access for your project. All despots are typically granted rights to create new plans.

If you are not a despot; contact one of your despots to get the plan created. They can then grant you management rights for the plan.

Configuring Bamboo for Codehaus eccentricities

Please see the following related articles for more information on configuring Bamboo to work with Codehaus infrastructure:

• Continuous snapshots
• Sonar at the Haus

Git Support

If you have an existing build you wish to switch to Git; then clone the build or create a new build. Do not change the source repository – there is something odd inside Bamboo / Git that will cause your build to go haywire and repeatedly build (and email).

.

The default configuration for Bamboo is pretty straightforward; just follow the prompts and ping support if you have any questions.

Hudson

Hudson will likely be viewed at http://hudson.ci.codehaus.org/ (HOWEVER IT IS NOT AVAILABLE YET!!!)

Hudson support will be coming soon. Stay tuned for more information!

Tweet This Post Delicious

The only people who are likely to have issue are the Subversion users (previously Maven users had issues, but the CA for this certificate has existed in the JDK since at least 1.4)

Cert Fingerprint: bb:2b:d4:07:3f:d1:d0:f3:3e:ac:e3:53:cb:04:0c:b2:e5:d0:6a:9a
CA SHA1 Fingerprint: D2 32 09 AD 23 D3 14 23 21 74 E4 0D 7F 9D 62 13 97 86 63 3A
CA MD5 Fingerprint: 67 CB 9D C0 13 24 8A 82 9B B2 17 1E D1 1B EC D4

OpenSSL

The primary issue with some certificates is that they use newer root authorities that are sometimes not present in OpenSSL. We did attempt to finagle a reduced cost deal with a particularly large existing CA but they weren’t even interested enough to reply to our emails. C’est la vie.

As the CA we use (Equifax) isn’t in the default OpenSSL libraries on Mac OSX Leopard; it is safe to assume it won’t exist on some other systems either (it does exist on RHEL 4+ though).

Subversion

The following is a conversation you will have with svn when you first attempt to connect to our Subversion services (these are the live fingerprints for the new certificates; you can verify this using your browser to https://svn.codehaus.org/)

# svn co https://ssltest.codehaus.org/...

Error validating server certificate for 'https://svn.codehaus.org':
- The certificate is not issued by a trusted authority. Use the
fingerprint to validate the certificate manually!
Certificate information:
- Hostname: *.codehaus.org
- Valid: from Tue, 30 Jun 2009 16:51:47 GMT until Fri, 02 Jul 2010 02:52:13 GMT
- Issuer: Equifax Secure Certificate Authority, Equifax, US
- Fingerprint: bb:2b:d4:07:3f:d1:d0:f3:3e:ac:e3:53:cb:04:0c:b2:e5:d0:6a:9a

(R)eject, accept (t)emporarily or accept (p)ermanently? p

Maven / Java

This CA has been available since JDK 1.4; so should pose no significant issue to Maven / Java users.

You can check your JDK / JRE using the following commands:

keytool -list -keystore $JRE_HOME/lib/security/cacerts | grep -i 67:CB:9D:C0:13:24:8A:82:9B:B2:17:1E:D1:1B:EC:D4
Enter keystore password:  changeit
Certificate fingerprint (MD5): 67:CB:9D:C0:13:24:8A:82:9B:B2:17:1E:D1:1B:EC:D4

Migration

The migration will be starting tonight; and be progressively rolled out across all services.

We are standardising our internal SSL configuration files at the same time; so if you notice a service running expired SSL certificates; shoot us an email and we’ll get it sorted!

The systems that will be affected include:

• https://jira.codehaus.org/
• https://svn.codehaus.org/
• https://docs.codehaus.org/
• https://xircles.codehaus.org/
• https://dav.codehaus.org/

Tweet This Post Delicious

If you have configured  a build in Bamboo, then you would know that using continuous integration (CI) can be a great way to build packages for users who want to use the latest and greatest, but don’t have the time nor the patience to build these packages.

Maven Configuration

In order to make Maven deploy your snapshots to the appropriate area on the Bamboo server, you will need to configure the snapshot repository using your pom.xml

Please review the pom.xml fragment found at profile.xml

Please note that any process is free to write / remove files under /opt/snapshot.ci.codehaus.org/ – so please be respectfui of other projects. If there is too much contention we will be forced down the totalitarian path – and we don’t want this.

Deploy

You can then deploy your artifacts using the deploy plugin.

maven deploy:deploy

Cleanup

At some point; the archive will become fat and unwieldy. We will implement a pruning mechanism to remove old artifacts at that point.

The upshot of this – don’t rely on versioned SNAPSHOTS being this repository for ever.  Also note that this applies as a general rule to artifacts help in the main snapshot repository – http://snapshots.repository.codehaus.org/

Issues?

As always contact support with any issues.

(We’ve just pieced this altogether from the support requests we’ve handled. If there is any nuance we’ve messed up, let us know!)

distributionManagement

Tweet This Post Delicious

Sonar is a “code quality management platform” – it draws together a variety of code quality metrics into a consolidated portal.

Sonar enables you to collect, analyze and report metrics on source code. Sonar not only offers consolidated reporting on and across projects throughout time, but it becomes the central place to manage code quality.

At present this is only accessible to projects building on the Bamboo server (Hudson will be established soon).

Our friends at SonarSource are managing the Sonar instance for us; but you should still raise all service requests through our normal channels – we will delegate and monitor the requests over to SonarSource.

Sonar Reports

(the bottom two images are the maven images shown below – ignore them; my Wordpress-fu is too weak and I’m too lazy to manually script only the images I want at the moment)

Accessing Sonar

Head on over to the Codehaus Sonar at http://sonar.ci.codehaus.org/

Getting Started

To manage the quality of your project with Sonar, create a new plan in bamboo, fill plan details and source repository tabs.
Refer to Analyzing Java Projects section to setup an analysis, but here is some guidance : on the builder tab, choose maven as a builder in all cases.

If your project is a maven project use the following goal to run the analysis:

clean install sonar:sonar -Dmaven.test.failure.ignore=true

If your project is a non-maven, you should create a small pom.xml file as described here. Then use the goal “sonar:sonar” to run the analysis :

Save the plan and run it.

Go to http://sonar.ci.codehaus.org/ to see the result of the analysis.

FAQ

How can I get assistance in running a Sonar analysis on my project ?

Please create a ticket in Jira, using Sonar as a component.

Who can I ask questions about Sonar ?

You can ask questions on the user mailing list.

How can I use my own quality rules (profiles) and setup alerts on my project ?

At present this capability can only be managed by our Sonar administrator.

Please create a ticket in Jira, using Sonar as a component.

Where do I find more information about Sonar ?

• Sonar web site : http://sonar.codehaus.org
• Sonar wiki : http://docs.codehaus.org/display/SONAR/Documentation
• Sonar plugin library : http://docs.codehaus.org/display/SONAR/Sonar+Plugin+Library/

Tweet This Post Delicious

What’s the plan?

Well the theory goes that we will have all manner of blogging on here; but realistically it will be mostly written by me with a smattering of third party articles.

Is this an ego blog?

I hope not; I hope to have this primarily as the focal point for ongoing discussion with the Codehaus community; and in an ancillary role as a place to make notes about all the weird and wonderful things we discover in our travels.

How is this different to all the other communications channels?

So we’ve got RSS feeds, twitter, an unused “Get Satisfaction” account, JIRA, Confluence and a variety of other informal means of disseminating information.

While each of those have their own special focus, the longer (and more detailed) articles will end up here.

Will you put non-tech stuff on here?

No, you won’t be exposed to stories about cycling, chocolate eating or photos of cats doing stupid things. Unless of course these items have a direct impact on Codehaus / technology.  If I break my arm (again) riding my bike – that will be relevant as it impacts support. If I end up in a diabetic coma from eating chocolate – relevant to support. If a cat gets into the servers – relevant (and I’ll be asking them to lock the cage to the servers too).

So stay tuned!

Tweet This Post Delicious