Cozened Cognizance

Development Database Setup

2010-11-18T08:20:00.003-06:00

Hello Everyone,

If you're working on an application of any significant complexity, one of the tasks developers must deal with are database set up scripts. One trend that I've noticed over the years is that projects where developers maintain a set of scripts that setup the databases from scratch tend to go more smoothly. To be clear, this process includes tearing down any existing database, rebuilding the database to the most recent release version, loading good test data, and finally running all change scripts for the next release. Though those last two steps can be reversible depending on the project.

Example the first. I have a system. Yaay! I need to make a change to the database. It's a relatively complex change involving creating a new table including keys and indexes, copying data from an existing table to the new one, and removing columns on the old table. Booo! After I make all these changes and have the scripts in place, I need to test that I did everything correctly. On the plus side, I have a thorough rebuild process in place. I can click a button and the whole system is rebuilt, I find problems and fix them myself and I know without bothering any other developers that the scripts are good. Yaay!

Example the second. I have a system. Yaay! I need to make a change to the database. It's a relatively complex... yada yada yada... I need to test that I did everything correctly Booo! Unfortunately, databases are not torn down and rebuilt as part of the setup process, they are only modified. What do I do to easily test my script. I can undo the changes by hand and rerun the scripts to make sure, or I can check in the changes, hope for the best and let the next developer find any problems and repeat the cycle with yet another developer. Which one do you think ends up happening more often? Booo!

This may not be a huge issue that comes up a lot, but when I see emails going out a few times a month with issues lasting a couple of days because things are done the latter, it reminds me of how many things that developers already need to keep track of and this is just another disturbance. In the end the amount of work to make the changes is the same, but having a complete rebuild is a way to minimize disturbances and keep developers developing.

If starting a new project, I highly recommend the complete rebuild approach because the hardest possible task would be overcoming momentum and switching from one process to the other.

Time to Leave Sprint?

2010-08-28T23:28:00.002-05:00

Not many people are fans of their cell phone companies lately. Most people I know are getting upset at AT&T. Their phones, mostly iPhones, are dropping their calls left and right. Verizon's customer service doesn't seem to be up to par. And Sprint, that's the one I get to complain about, I'm getting more and more upset with their phones.

I just picked up an EVO 4G. On the surface, this is a great phone. Easy to set up and use every app I've tried so far. I then noticed that there's a bunch of apps that come prepackaged on the phone that I just have no interest in. NASCAR? What's that? Voice mail? I want to use Google Voice thanks much. Uninstall prepackaged apps? Good luck.

Yes there are people on the web that say they have done it. I tend to believe them as there are videos documenting the process. But it is a process that is not for the computer illiterate; sometimes consisting of multiple updates, flashes, backups and recoveries. And in my case, while I'm not afraid of linux or ADB, all of the instructions I have found simply end with my phone displaying a disheartening message of "Update Fail!"

One post on the web asked the question, "Whose phone is this?" We probably all agreed that the phones weren't ours when we signed those contracts to indicate that we had read said contracts even though in my case the sheaf of papers was barely on the counter long enough for me to dot the 'i' in my last name. Even if that is the case, shouldn't the phone feel like it belongs to me? If it doesn't, why would I want to use it? Meh, I'll probably forget about it eventually... until I go looking to use a nearby app... and remember all the folks that were able to remove that app... and then their phones got faster... and had longer battery life... grumble grumble... damn Sprint...

I suppose all the phone companies are being forced, at gun point of course, to cut corners somewhere to be competitive. In the case of Sprint, apparently that means selling those corner seats out to advertisers. I suppose it did get me talking about NASCAR... Ah well, might as well get in the spirit. "Make that telco your bitch Ricky Bobby!"

IQueryable and Repositories

2010-06-17T09:33:00.006-05:00

One approach for coding repositories that I have seen a couple of times recently is that of exposing IQueryable collections through the repository interfaces. On the one hand this allows an extremely convenient way of exposing a flexible data source to your business layer or controllers. However, it also breaks the Single Responsibility Principle thus creating less reusable and testable solutions.

By allowing your business layer to define queries within its own methods, those methods are taking on two responsibilities, performing data access and performing logic on the results of that data operation. It's all well and good to complain about not hitting some high ideal, but what affect does this have in the real world?

First, if you happen to reuse the same data query in multiple parts of your application, you would have multiple places that you would then need to make changes and test. DRY this isn't.

Second is the testability of this code. In the terms of this afore mentioned white paper, you would need to write your tests for your business layer or controller in such a way that the state is correct as well as the interactions, for each method. As the paper says, testing the state aspect requires a larger set of data. Why would you want to write that extra code for every method in your business layer / controller that uses the same query? If you don't test each one, how do you know they work? How do you know when they break?

The simple solution is to keep your queries in the repositories behind the repository interfaces. Test the state of each one thoroughly and then you only need to test the interactions in your business layer. Yes it requires an extra function call in your business layer, another method in your repository interface and another function declaration in the repository. And while that might be a lot of words, it's only four or five more lines of code for each query (including brackets on their own line in C#.) And if you reuse your queries, you get those extra lines back anyway.

I will certainly concede that no one paradigm fits all of the software problems out there. Maybe what I suggest is overkill. Maybe it's another example of a programmer too set in his ways. But by exposing IQueryable collections through the repository interface, we're ignoring a number of the principles of coding that have been making our lives easier and our code better for a while now.

[Edit]

After a second read through of the article, K. specifically states that enumerables could be returned instead of queryables. Not that this is the only publication I have read where queryables were suggested. Let's just say that now all four of my regular readers know where I stand on the subject.

UnitOfWork

2010-06-17T08:36:00.007-05:00

This white paper by K. Scott Allen is a great read and was my formal introduction to the Unit-Of-Work concept.

For a while, the recommended approach for providing data contexts was to create the context and store it in a session of some sort. Each call to your server would provide a single data context for all of the logic needed to render a new page or perform some sort of data operation.

UnitOfWork might not be a name that explains exactly what the object is (a data context container). But, it does describe what the object does. It frames a the data context within a single logical unit of work. The distinction being that it may take many logical routines to render a single web page.

One advantage to the technique is that you are committing data changes at the end of each logical routine. This means that many of the data changes for that routine can all be committed at once using fewer network and database resources over a shorter period of time.

Another advantage is that you are holding the data necessary for each operation in memory for only as long as it is needed. When the operation is done and the data is likely no longer needed, the memory is flushed and reused for other operations.

Lastly (that I'll mention anyway), it separates your data context scope from session scope. The session will likely be different in web, windows and service applications. This means that your session across different applications may comprise a different number of business methods which can have unforeseen affects on performance and functionality. Instead, you can tie the scope of your data context to each business method as those make good candidates for a unit of work. This in turn means that your routines will perform more predictably across different types of applications.

Backgrounds, Google? Really?

2010-06-10T08:55:00.008-05:00

If you got to this post, you're probably wondering how to turn off Google backgrounds. I for one am pretty sure that if there isn't already a setting, there will be one shortly. I can't be the only person that want's my plain white, pleasant and unobtrusive background back. I'm also far too lazy to go look for that setting given that a link to it doesn't exist on the front page.

That being the case, I created some 800 x 600 solid color backgrounds and uploaded them to Picasa. 800 x 600 by the way is the minimum resolution that can be used. Yes, I do like the fact that Google is hosting my solution to my Google problem. Download early, download often. Here's the links:

I recommend the dark gray though the white does give a good 'White Album' feel. Obviously, you can also easily create your own image using MS Paint or any simple image editor if you prefer, say, color. Click on the 'Change background image' link in the lower left hand of the Google front page and upload the file from your computer.

Fine, this didn't take much imagination or skill. But I was so dumbstruck when I saw the page, it was all I could do to listen to various suggestions from coworkers.

[Edit]

In retrospect, this may have just been really good marketing. Google did seem to have a fix in place fairly quickly. I'm sure ours wasn't the only office where the new background had everyone talking for a few minutes. And who knows how many bloggers out there were duped into creating buzz about it.

hrmmmm

Milyli Update

2010-04-14T13:42:00.005-05:00

They say that in order for a blog to be successful, you need to write about what your audience cares about. Since I've started writing, I've only had one person ask me to give updates on how our little company / experiment is going. Then again, I've only had one person leave comments about the coding tips I post here so I guess either topic is as valid as the other at the moment. Which one will I talk about today?

That's right, it's company news. Our little company has grown to 6 whole, full time employees. That's right pretty soon, they'll out number us founders and then who knows what variants of heaven or hell they'll unleash on us. Whichever it is, I'm sure things are just going to keep being interesting for a while.

We're also considering leasing out the rest of the floor in the building we're currently in. It's a big step, but we've realized that there really is no substitute for closed off separated meeting places and work areas. Keep the loud side loud and the quiet side quite. We currently lack that separation and every new familyli member brings that fact into further relief.

We're in the home stretch of a nice long project as well. I can't talk much about it, which is probably why I haven't posted here much. Oh the mystery is killing you. I know. Hopefully though, I'll soon be able to take up posting on all the various interests I have, thus ensuring that my blog continues to wallow in obscurity.

Until then...

Google Applications

2010-02-10T07:35:00.003-06:00

I logged into GMail today and got an invitation to activate Buzz, so I did. I even posted a quick little message about my immediate impressions. And a fine bit of useless drivel it turned out to be if I do say so myself.

There was one thought that stuck me as interesting though. The realization cemented in my mind that there is bit of a pattern Google has to creating their solutions. They build or sometimes buy software that solves specific and difficult problems in easy to use ways that they provide for free. And with advertising much less obtrusive than other applications. They have many such pieces of software at this point. Now, they are finding the best possible ways to integrate them as elegantly as thy can. Networking applications to solve social networking, if you will.

I know, I'm late in realizing this. But I never accused myself of being overly watchful of other companies' strategies. Many people will argue that the one point of entry provided by Facebook makes social networking more convenient. My problem is that the utilities provided there just don't provide the functionality I want in photo management, calendars, email and event planning.

MVVC, Silverlight and Staying Grounded

2010-02-04T22:29:00.005-06:00

I was reading some Visual Studio Magazine today and there was an article on MVVC gushing over how great it is. I know it's not exactly new. I also know that the pattern, when combined with the trend toward data binding is a powerful combination. But it's surprising that many 'architects' apparently swear by this solution and insist on writing whole applications of every type without deviation.

There are many in-the-trenches sorts that believe that while it is useful in certain situations, it actually causes more work and creates less maintainable code much of the time as well. I ran across many differing opinions all in the same day. Probably the most sane of which were saying that it's a good architecture for mid size, data processing applications; too much for small tools and not flexible enough for large, interaction rich utilities.

As with all layers of an application, there appears to be different problems to solve in the UI. The different paradigms are good at specific problems of certain sizes, but no one is best suited for all tasks of every caliber. From messaging between disparate parts of a large application, to showing data in a small application, to interacting with complex UI's and business logic, it seems like you need to choose the solution(s) that will best fit the problem(s) at hand. This might not help you out much in choosing the best model to use on your current project, but it may be ammunition against those of the one-solution-fits-all mind set.

Maybe it all comes back to the fact that a text box and button with some in line SQL in the code behind is a perfectly acceptable way to modify some simple data under certain circumstances...

I also read up a bunch on Silverlight 4 today. I guess there's a slew of new technology being released along with .NET 4 and VS 2010. The general summary is that it makes Silverlight a viable line-of-business application platform. But, there seems to be a lot of bells and whistles that are more web 2.0'ish rather than business requirements. Maybe there's some overlap there, or maybe there will be when businesses are ready for it. That being said, it's probably worth looking into the specifics on the WCF changes as those were the hardest challenges that I remember having to overcome with the platform in a business application. The whole printing thing is probably pretty nice too.

Milyli Update

2010-01-22T15:44:00.006-06:00

It would seem it's been a while since my last post. As Google appears to be the largest source of my traffic, I don't think there's all that many loyal followers on RSS that need apologizin' to. And I wouldn't want to assume that my words are so engrossing that the few regular readers I do have even realized I hadn't written.

So just read this thing.

The past few months Milyli has been growing. In addition to the three founders we have hired our first full time employee, we are on the look out for others, we have two part time employees and we just moved into a real office. Fine, it's a big loft space that we use as an office, but it has desks, meeting tables, shelves and colorful network wire running every which way. It's more of an office than the house that we moved out of.

Probably one of the best articles I have read that sums up the experience is this one by Paul Graham. I direct you in particular to the section at the end sub titled The Super Pattern. It's worth a read, but the gist of the article is that so many people have written about the process of starting up a business that there should really be no surprises about what experiences you're in for.

What you probably will be surprised about however is the intensity of those experiences. Yes, you will have to do all of the things that everybody always says you need to in order to start up a business. But the peaks and valleys of this here roller coaster are far more extreme than I had anticipated.

ASP.NET PasswordRecovery Control and GMail

2009-11-06T09:20:00.000-06:00

I recently had need to whip up a quick ASP.NET web site with authentication. Nothing fancy, so the default ASP.NET Membership provider tables and account controls worked fine. The one problem I ran into was that I needed to route the email the MembershipProvider and PasswordRecovery control create through GMail. You see, GMail uses secure connections and the control doesn't seem to offer built in support. But it isn't too hard to get it working.

Here's the approach I ended up with.

protected override void OnInit(EventArgs e)

{

base.OnInit(e);

PasswordRecoveryControl.SendingMail += PasswordRecoveryControl_SendingMail;

}

void PasswordRecoveryControl_SendingMail(object sender, MailMessageEventArgs e)

{

// Create network credentials from SMTP configuration settings.

var config = WebConfigurationManager.OpenWebConfiguration(HttpContext.Current.Request.ApplicationPath);

var settings = (MailSettingsSectionGroup)config.GetSectionGroup("system.net/mailSettings");

var credentials = new NetworkCredential(settings.Smtp.Network.UserName, settings.Smtp.Network.Password);

// Send message over secure connection.

var mailClient = new SmtpClient(settings.Smtp.Network.Host, settings.Smtp.Network.Port);

mailClient.EnableSsl = true;

mailClient.UseDefaultCredentials = false;

mailClient.Credentials = credentials;

mailClient.Send(e.Message);

// Prevent control from sending message via default implementation.

e.Cancel = true;

}

Canceling the event still results in success as far as the control is concerned. This means the SuccessText and SuccessUrl properties on the control behave the way you want them to if not quite the way you might expect them to.

Inversion of Control

2009-09-22T08:38:00.002-05:00

I've been a fan of IoC frameworks for a little while now. In particular, I have been using StructureMap. I knew that these frameworks were useful, but I didn't realize what I was missing until recently. I would look at some of the proponents of the frameworks and their example code. This code would have formal code Interfaces defined for almost every object and interaction in the system. That always seemed like overkill to me. I would think, "You are only ever going to have one class that fulfills each of those sets of functionality. Why bother with all the extra coding for the interfaces?" It didn't make sense... until now.

I started working on a new pet project in my spare time. It's a program with a significant number of moving parts that all have different sets of functionality. Following the single responsibility principle, I've broken down each of those sets of functionality into objects that each have their own domain of control. Then I started writing unit tests. And that's when the realization hit me. There may only be one production implementation of most of the classes. But combine those interfaces that define concise functionality with a good mock library and writing unit tests with few lines of code becomes a piece of cake. I'm talking well tested code with tests that are at most twenty lines of code each; sometimes more, but most often less.

"Yes, IoC frameworks are cool for plugins and they make unit testing easier," I would say to myself. But it turns out that I didn't fully grasp the full extent and usefulness of this paradigm. That doesn't necessarily mean I'm all the way there yet either. But, I wanted to write something about this because it really was an eye opening experience; the difference between knowing and grokking. Also, I am not all that unique a person. And since I wasn't putting all this together from reading documentation and blogs, I felt that perhaps there are others out there that might benefit from a different point of view on the subject as well.

ASP.NET MVC and ReverseDOS

2009-08-27T18:39:00.006-05:00

I've blogged a bit about ReverseDOS before. It's a neat utility that allows you to lock up and/or deny HTML requests from content spammers. You know, you have a web site users are allowed to create content on, comments in particular, and the Online Consortium of Gambling Websites has targeted your site as a nice place to do some free advertising. ReverseDOS very easily allows you to tie up their resources and ignore their offerings.

The point of this post is that there is a trick to get ReverseDOS to show custom errors when using ASP.NET MVC. I forget where the original information came from to get this done. It's already out on the web somewhere, I just don't remember where I pulled the pieces from. Just in case it isn't easy to find, I'll show you how I got it done.

First, configure ReverseDOS just like the instructions tell you to. The one ReverseDOS configuration setting that you need to verify is that endRequest = false.

Next, in your Global.asax file, you need to add a new event handler.

protected void Application_PreRequestHandlerExecute()

{

var error = Context.Items["ReverseDOS_Exception"] as HttpException;

if (null != error)

{

ServeError("/Error/AccessDenied", 403);

}

Obviously, change the path to your own error page. The ServerError() method is an example of how to render a different ASP.NET MVC action to the response stream from outside of the MVC framework. This is basically the same as supplying a different view to render when returning from an action. The method looks something like this:

private void ServeError(string path, int statusCode)

{

var url = new StringBuilder();

url.Append(VirtualPathUtility.ToAbsolute("~"));

url.Append(path);

Server.ClearError();

Context.Response.Clear();

string originalPath = Request.Path;

Context.RewritePath(url.ToString(), false);

IHttpHandler httpHandler = new MvcHttpHandler();

httpHandler.ProcessRequest(HttpContext.Current);

Context.RewritePath(originalPath, false);

Response.StatusCode = statusCode;

Context.ApplicationInstance.CompleteRequest();

}

Granted, there are probably other ways of handling this. In my admittedly limited experience however, this gives me the most control when setting up ReverseDOS along side custom error handling in ASP.NET MVC. It allows me to serve my own error pages with proper http response codes without redirects and without relying on IIS features.

Happy coding to you.

A Solution To My Repository Dilemma

2009-08-27T17:28:00.005-05:00

I woke up this morning and realized I had a better solution to my repository problem. Create instances of your business logic that accept repository instances in their constructors. Pass the models through the business logic to the repositories for simple data operations. Perform business logic and relationship management on the models when needed before the data is stored.

It's still not ideal though. First, there's the potential for a lot of pass-throughs. I can live with that I suppose. Second, you end up losing a lot in the way of encapsulation in that the business methods that operate on the data are not contained in the same objects as the data they operate on. But it's not like there isn't precedence for effectively relegating your business models to what amount to DTOs or data contracts. Think web services. You're just not passing those contracts over the wire. Plus, mapping between models tailored to the application layer they are used in is becoming an accepted practice. At least it is in the ASP.NET MVC world which is what prompted me to write about this anyway.

As I said, it may not be ideal, but I think I like it better than where I ended up at in my last post. The code becomes more testable and you don't need to re-implement relationship management in every version of the repositories that you create.

One Small Problem With Repositories

2009-08-02T09:57:00.007-05:00

It used to be that I wrote applications with a hard separation between layers; presentation calls business, business calls data, and other systems were wrapped in abstraction as needed. This worked well enough and many large enterprise applications have been build this way.

One technique frequently implemented in this paradigm is that the entry points for the persistence routines of domain objects are encoded in the objects themselves. While not perfect, one great upshot of this approach was that there was only one place where the relationships between business objects was defined. E.g., if an Order is saved, the Order enforces that all of itsLineItems are also saved. The code is written once and tested once. If new repositories are needed for a new database, write the code, swap them out and as long as your business layer tests still pass, you know your relationship persistence is in tact. The biggest problem with this paradigm though is that it leads to difficult to write tests for oh so many different reasons.

As test driven development is the way to go, I haven't written applications like that for a couple of years. I have been sharpening my saw and forcing myself to build more testable code. In ASP.NET MVC, the prevailing winds seem to push in the direction that your controllers should be injected with repository instances and those instances should be used for data management. This works exceptionally well from the stand point that I can pass repository mocks and stubs to my controller when testing actions. This is just a better way to write code in general.

But one part of all this newer-fangeld code crafting still nags at me. If the repository contains the relationship logic, that logic and all of the tests that enforce it travel with that implementation. If I need to write a new data layer for a new database, I need to rebuild that logic and the tests for it and maintain multiple pieces of code that ultimately enforce the same rules. In almost all cases, this is not a real world problem. Most people never port their applications to multiple databases and so the interfaces are relegated to testing purposes only. Most of us only ever have one set of repository code to test. I'll live with it by ignoring it.

The perfectionist in me still wants to do something about it. I've thought about crazy schemes like dividing the repository functionality into multiple components, CUD/R if you will, with methods targeted to the domain model and UI respectively. I've thought about abstractions and composites and delegates, oh my! But in the end, every solution solves some problems and creates others. In the end, I suppose that knowing when good enough is good enough and being able to identify when it isn't is the way to handle this situation.

WCF and IIS 7 Error

2009-07-28T10:35:00.003-05:00

I ran across this little nugget of frustration today when trying to hit my WCF services hosted by IIS 7 on a computer with a new build:

Error 403.3

The page you are requesting cannot be served because of the extension configuration...

Turns out that the .svc extension was never registered with IIS.

Simple problem, simple solution. With administrative privelages run:

C:\Windows\Microsoft.NET\Framework\v3.0\Windows Communication Foundation\servicemodelreg -i

or for 64 bit OS's

C:\Windows\Microsoft.NET\Framework64\v3.0\Windows Communication Foundation\servicemodelreg -i

Just another post to bump the solution up Google a bit.

Rhino Mocks Out Ref Parameters

2009-07-25T14:00:00.003-05:00

I started playing with Rhino Mocks recently. Yes, I'm late to the game. And yes, that game is great. If you've made it to this post I'm obviously preaching to the choir, so let's get to it.

I ran across a need to mock up an out parameter today. For the context of this article that would be a ByRef parameter to all you VBers. What to do? Well, it turns out to be a very simple task.

I have several repository methods in a little project of mine that read paged data. I like to return the total number of records in the same query from the database and so I use an out parameter in my repository methods. The methods look something like this:

public IList<RecordModel> ReadPaged(int pageNumber, int pageSize, out int recordCount);

This project of mine is written for ASP.NET MVC which, like every other ASP.NET MVC project, contains controllers. My controllers makes calls on the repositories and push the models returned into the ViewDataDictionary. The goal then is to create a stub for the IRecordRepository.ReadPaged() method that passes a content count, among other values, back to the controller so that the value may be pushed into the ViewDataDictionary and then validated in a unit test. That's a lot of talk for one small method call:

recordRepositoryStub

.Stub(m => m.ReadPaged(pageNumber, pageSize, out recordCount))

.OutRef(250)

.Return(records);

That call to OutRef(250) is what takes care of the out parameter. The method accepts a parameter array of objects. When the stub method is called by the controller, it populates its arguments into the out parameters in the order in which they are declared in the method signature.

Goal accomplished.

IoC and Structure Map

2009-07-06T09:23:00.004-05:00

I started playing with StructureMap a couple of days ago. It is a pretty decent framework for IoC and dependency injection. It works well with ASP.NET MVC and makes it possible to write some good, testable code. That's not to say that there isn't anything better out there, just that this gets the job done well and so I didn't need to look for anything better. The features that I like most are the registry configuration system, and the dependency injection approach.

One way to configure StructureMap is to subclass the Registry class. Methods in the class allow you to create mappings between interfaces or abstract classes and the concrete types that will be used during run time. The simplest approach is to create one to one mappings for each interface / concrete type pair. Nothing you couldn't do with a simple text or configuration file of some sort. However, there are more powerful methods available that will automatically map all types against the interfaces they implement within a given assembly or namespace. You can even map multiple types against the same interface and use a key to get the correct implementation for a given situation at run time. Score one for the IoC framework.

As for creating the mapped instances, there are a couple different ways to get that done. The simpler approach is just a factory method that accepts the interface name and optionally a key and returns the configured implementation. Again, this is nothing that you can't do very quickly and easily with the Activator object in .NET. Once again, a more powerful technique exists and that is to request an instance of an object that accepts mapped types in its constructor. E.g. Suppose I have a class SiteController with a constructor defined as void SiteController(ISiteMapper mapper, ISiteRepository repository) and that both the interfaces have mapped concrete classes configured. I could request an instance of SiteController from the StructureMap ObjectFactory and it would create the SiteController container with the appropriate dependencies passed to the constructor. This is another nice convenience that good IoC frameworks provide you with.

Martin Fowler said something I like about IoC:

"When these containers talk about how they are so useful because they implement 'Inversion of Control' I end up very puzzled. Inversion of control is a common characteristic of frameworks, so saying that these lightweight containers are special because they use inversion of control is like saying my car is special because it has wheels."

My take on this quote is that IoC is nothing more than good object oriented design that takes advantage of polymorphism and factory patterns. It's a technique that has been around since the first OO languages and isn't some new advance in software design. That being said, StructureMap and other frameworks like it do offer some great utilities that simplify the task of writing the code.

State of Milyli

2009-06-11T10:17:00.004-05:00

It's been a little while since I wrote about our little company Milyli. Since that was one of the original purposes of starting this blog, I figured I'd give a little update. Things are going well, but it's a lot of hard work.

As for myself, being mostly in charge of product development can be a little frustrating working almost alone. It's not that I'm the only person working to make the company and product a success, not by a long shot. But I am effectively the only developer working on the code. Granted, I get to develop the application whichever way I want. But for a not small piece of software, that's not as ideal as it might sound. I really miss being able to bounce ideas off of other developers that have a good sense of how the application works. The small increments of work are also not as satisfying when you realize that no work has been done on the rest of the application.

Besides the application, I did a bunch of project work early on and it looks like I'm about to get another load. I enjoy working on the consulting side. The projects are different and there's always a chance to learn a new trick of some sort. But it is frustrating to see so much of our already limited time not being spent on our product.

As for my two partners, they've been doing the bulk of the bill paying for the past number of months. They're each feeling the pain and glory in their own ways.

One of them is working with technology older than even that used at our last place of employment. The old code is plenty crufty and has more than its share of best practice failings. That and interacting with and managing the client seems to be wearing him a bit thin of late.

My other partner spends his time schizophrenically jumping between business development, market analysis, office manager, project manager and consultant. Basically filling in wherever needs filling. I am not envious at all of that role.

All I can do is thank them and keep my head down and keep making software of all sorts work better. What's important to remember is that the Dip is worth getting through. I'm pretty sure we haven't seen the bottom yet and if by some miracle we have, that will just be a pleasant little surprise.

There have been a number of high points as well. We get to make the environment the way we like it including eating lunch together everyday that we are in the office. We have started to see some preliminary design ideas for the look of our application. They look great and I can't wait to start implementing. We have signed on some work that should pay our bills for a while. We have other new, exciting projects coming in that look like they will fit great with our skills and experience. And we keep meeting new people to talk about their problems and solutions even when we aren't necessarily the best folks to tackle the resulting projects.

And that's where it all stands right now; things are going well, but it's a lot of hard work.

Create SharePoint Job Timer Definitions

2009-06-10T17:34:00.009-05:00

So I realized recently that one of my MOSS applications did not have all of the timer jobs defined for it that it should have. You know, you go to Central Admin -> Operations -> Job Timer Definitions and they just aren't all there. Which ones should be there by default? Here's a good list. Just how do you go about getting some of those jobs in there? There's a bunch of tricks and supported approaches.

The publishing tasks (Scheduled Approval, Scheduled Page Review, etc.) can be created in one shot by creating a new web site collection using the publishing portal template under the existing application and deleting it. The timer definitions stay, the services start running and everyone is happy. Yaaaay!

Some of the definitions will be created if you set the appropriate properties using stsadm.
E.g., stsadm -o setproperty -pn job-immediate-alerts -pv "Every 5 minutes between 0 and 59" -url http://your-site.com
Once again the timer job shows up and most of the time your alerts start flowing. Hooray for our side!

But there are a bunch of definitions that there don't appear to be tools to create on their own. Sure you can back up the database, blow away the application, create a new application with the correct template and same name, restore the database, and fix the issues (there's always some). I didn't want to go through all that, though I'm not sure that what I figured out was less work. What I ended up doing was to write some crafty SQL. Well, maybe it was just plain old mundane and obvious SQL, but that's what I did. All the answers lie in the Objects table in your config database.

First find the id of your web application. This is going to be the parent id of the timer record you will insert. The following SQL worked pretty well for me.

SELECT * FROM Objects WHERE Properties LIKE '%SPWebApplication%'

Next, query the table and look for existing definitions of the jobs you are missing. If you don't already have an application that contains them, you can create one. The Name column of the records contains the stsadm property for each timer job. Here's a list of a few. And here's some SQL to help you out.

SELECT * FROM Objects WHERE o.Name LIKE 'job-%'

Find a record for the job you are interested in. You will need two pieces of information: the ClassId and the Properties contents. The classId tells the system which timer job to create and the Properties contain the frequency of the timer job and a couple of other easy to change settings.

Last, run a simple insert statement like the following...

INSERT Objects(Id, ClassId, ParentId, Name, Status, Properties)

VALUES (NEWID(), '[JobClassId]', '[ParentApplicationId]', '[JobPropertyName]', 0,'[PropertyContents]')

A quick refresh of the Timer Job Definitions page will show the new job ready to run. You may need to restart the Windows SharePoint Services Timer service before the new jobs are picked up though. Also, the Timer Job Status list will not show the new jobs until after the first time they are run. Be patient for those weekly tasks.

Again with the disclaimer: I saw in some posts that Microsoft doesn't take to kindly to messing around with the SharePoint databases. If you want continued support, tweak at your own risk. This should really only be used as a last resort in any case as there are usually other options. I just happened across someone asking this question and decided to figure out how to do it.

SharePoint Alerts Not Sending

2009-06-10T17:21:00.006-05:00

There are all sorts of reasons why your alerts might not be working in SharePoint. There are a whole slew of causes as to why with the most common being a backup and restore to a new server, farm, environment, etc. Googling will provide you with a bunch of options for the common problems pretty quickly.

None of those really worked for me. I did eventually solve my problem though. One post pointed me at the EventBatches table in the content database. This table contains two pieces of information, the last event time and the last event id processed. It turned out that the id in the EventBatches table was FAR greater than the last id in the EventCache table. This was probably the result of some less than optimal backup and restore voodoo that had gone on previously. Regardless, I set the id in the EventBatches table equal to the last EventCache id and alerts started flowing freely from that point on.

I wasn't able to find anyone that just came flat out and said that, so there it is.

Disclaimer: I saw in some posts that Microsoft doesn't take to kindly to messing around with the SharePoint databases. If you want continued support, tweak at your own risk.

Comments By Google

2009-05-30T17:20:00.003-05:00

A couple of posts back I wrote about some techniques one can use to fight comment spam on their web site. Just recently, I ran into another technique, one that is all the rage in so many other business aspects: outsource it.

It turns out that Google has an interesting little service called Friend Connect. The features include comments, ratings, authentication and moderation tools to name a few. I haven't set it up yet nor read all there is to read, but I'd be willing to bet they also throw some of their great spam filtering technology built for GMail at the comments as well. An API exists so that developers can make use of the information tracked by Google to provide a more interactive experience. Think privileges determined by karma and such. And with a reasonably trustworthy third party writing a large chunk of your code, developers can focus on all the aspects of the site that deliver content instead.

The biggest downside to this approach, and it can be a biggie in some cases, is that all that social data is going to be housed in a remote database. That's probably OK for a great number of sites out there, but for some application, that may just be asking too much.

Another downside is that the look and feel is going to be limited unless you code all of your own controls against their API. That probably still does save some time. More importantly it means that you don't need to worry about the more demanding aspects of authentication. But once again, it means that outsourcing may not yield all the time saving benefits so many people think it will.

Middle Mouse Button Broken

2009-05-26T21:46:00.002-05:00

I had a frustrating problem with my mouse tonight. Basically, the middle mouse button stopped working on my Razer Lachesis. I tried searching for a resolution and all I found were suggestions on how to clean the mouse. I'm too lazy for that.

I found that a simple way to test the buttons is to remap the button(s) in question to a keystroke; mouse button 3 to the number 3 in this case. I opened up notepad, clicked, and lo and behold, a 3 every time I click. What gives?

I started to switch the mouse button function back to 'windows button 3' and... what gives again? It's gone. As I switched through the other mouse profiles though I noticed that button 3 was almost always set to 'universal scroll'. "Let's try that," I thought to myself. I applied the new setting. Don't forget to apply the new setting! And look, the mouse is as good as new.

As near as I can figure it, I actually set the mouse button to 'windows button 3' at some point in some previous Windows version or application. The interesting thing is that the profiles are stored in the mouse, not the driver software. That's my guess at least as I removed the driver software and restarted the computer at one point and my middle mouse button was still pumping out 3's. Combine that with the fact that the signal is evidently a bit different between Windows 7 and whatever program I set it on, and a little bit of profile changing butter fingers on my part and voila - apparently broken button.

So, it turns out that my only real complaint about my nice, year old Lachesis is that it is too customizable for my clumsy self. Aside from that, this is one heck of a mouse.

Figured I'd share my experience in case some one else runs into this problem.

MOSS Search Access Denied

2009-05-26T12:48:00.003-05:00

So I have been running into an issue on Office Share Point where the search service ends with an access denied error when it runs.

"Access is denied. Check that the Default Content Access Account has access to this content, or add a crawl rule to crawl this content."

The fix for this in our situation turned out to come from Microsoft KB 896861. A lot of solutions focus on making sure account permissions are set up correctly. However, this lesser known issue is caused by a security feature in IIS that prevents reflection attacks. The feature gets in the way of the shared services provider when it tries to crawl the site in a single server environment. The recommended solution is to map a specific hostname to the loopback address. Check out the KB for the details.

I originally found this article on SharePoint Blogs. Hopefully, promoting this solution will help someone else out there.

Techniques to Fight Comment Spam

2009-05-19T10:33:00.006-05:00

The following post is a list of techniques that I have run across that attempt to deal with the problem of comment spam on sites. This is a followup to my last post titled, "Preventing Comment Spam."

Requiring authentication is generally seen as a fairly effective approach to preventing comment spam. However, the disadvantages are frequently enough to dissuade implementation on many sites. One problem is that authentication is a feature requiring resources and expertise that not all development shops have enough of. Another issue is that authentication results in a barrier to leaving comments on a site that casual visitors will probably not bother overcoming. In addition, the challenge is not that great for technically adept spammers if the payoff is access to a large user base.

Building upon authentication is the idea of karma. Forcing users to build karma based on quality of participation before they can take certain actions is usually a hurdle that is too high for most spammers to deal with. Unfortunately, depending on your user base, it can be an equally high hurdle to legitimate participation.

Moderation is another technique that comes up often. It is generally regarded as the only foolproof approach. Simply put, every post made to the site is screened by a human being. The downside of course is that if your site is heavily trafficked by spammers, weeding quickly becomes a task that takes up all of your time.

Filtering is another approach that can best be described as automated moderation. As an example, I found ReverseDOS. This is an easy to setup ASP.NET HttpModule that reads all of the content of a request and determines whether or not the request is a spam attempt based on rules that you define. The rules can include checking all or only a portion of the request against a set of regular expressions and can be turned on or off for each directory within a site.

Another suggestion along these lines was to create a central repository for tracking spam. Sites could query the repository which would try to determine if the submitted content was spam based on past submissions, user feedback and a bit of good natured artificial intelligence. Regardless of the technique, the idea of filtering is to cut down the number of spam comments to an amount manageable by other means.

Reverse Turing tests like CAPTCHA can sometimes be used to increase the difficulty of posting spam. The problem is that the effectiveness of the most common implementation, retyping words presented as an image, wanes as image recognition tools get better and better. The images must get more warped in order to prevent automated scanning, but that makes it more difficult for legitimate users as well. E.g., Google's captcha for new emails is so difficult to read at times, that I only get one out of four correct.

Throttling can be used in order to prevent any user from posting too many times. Limits can be set on the number of items that can be created over a span of time or making sure that no user posts multiple comments back to back in a single thread. The challenge here lies in identifying users. If no authentication is used, relying on IP address is inconsistent at best and runs the risk of blocking legitimate users.

In the end, no single approach is probably good enough to stop spam. The pet project I am currently working on has been built with a mix of most of the techniques above. I combined a bunch of existing frameworks with a little bit of custom code so it wasn't too much work. At times I worry that I may have spent too much time on this aspect of the site. Then again, the whole site was started as a learning endeavor. If nothing else, I gained some knowledge and will have the tools in place to respond quickly if spammers begin to target the site.

Preventing Comment Spam

2009-05-19T09:29:00.008-05:00

I have taken up the coding challenge of dealing with comment spam. As with most topics that I write about, I am by no means an expert. But I have done a lot of reading recently, and here are some of my observations.

There is no majority consensus on the single best approach to prevent spam filtering. Everyone agrees something must be done, but few people agree on which one method is the most effective.

The tactic that most people do agree on is that multiple techniques are necessary in order to achieve the desired levels of spam reduction, ease of maintenance and usability for visitors. The business of spam is based on the idea that by getting a lot of content in front of a lot of users it is likely that enough people will respond to make a profit. Countering spam is a process of making it difficult enough for spammers to post to your site that their time is better spent elsewhere. The challenge lies in creating a system that is easy enough for your users to participate in that is at the same time complex or smart enough to discourage spammers.

The combination of most effective tools to employ varies depending on the site being targeted. Your breadth of content and comment topics, user quantity and quality, and a host of other variables will determine the different tools that will achieve the best results fighting spam. The larger and wider ranging each of those dimensions is, the smarter your techniques will need to become. At some point, the easiest to implement approach may be to screen submissions by hand.

There seems to be a general feeling that if enough sites take steps to reduce spam, the web can be made a better place for everyone. Spam will probably never go away. If it does, it is likely that the infrastructure of the web was changed for the worse for everyone in some way. But the idea is to make it difficult enough that spammers would make more money performing constructive services instead of annoying ones.

For a bit more on techniques used to prevent comment spam, check out this followup post.