Eric J. Smith

CodeSmith Generator 6.0–Status, Website and Future

ejsmith — Thu, 10 Nov 2011 21:48:35 GMT

We are on the home stretch of Generator 6.0 and things have come together really nicely. This is going to be a great release! We have been mostly working on performance, memory utilization and bug fixes, and we are doing really good on all...(read more)

CodeSmith Generator 6.0 Beta Survey

ejsmith — Thu, 08 Sep 2011 21:22:20 GMT

We are currently working on dogfooding the new template editor by updating and building some new templates. Along the way, we are trying to find and fix as many pain points as we possibly can. We are also working on profiling and memory usage reductions...(read more)

CodeSmith Generator 6.0 – Editor in Action

ejsmith — Thu, 14 Jul 2011 14:30:00 GMT

I thought the best way to show our progress so far would be to show you a quick video of the new editor in action. So without further adieu, here is the new CodeSmith Generator 6.0 editor:

In general, I think our editing experience is already about 10 times better than it was before. Not even counting the fact that it’s integrated right into Visual Studio now. We still have some work to do, but we will be releasing a beta version soon.

Thanks again,
Eric J. Smith

CodeSmith Generator 6.0 Alpha

ejsmith — Fri, 13 May 2011 22:07:00 GMT

We have been making great progress on CodeSmith Generator 6.0 and would love to get some feedback on our new template engine, template runner UI and new property grid (more on this soon). As I mentioned in a previous post, we have completely re-written the template engine so that we can implement a greatly improved editor / IntelliSense experience as well as make it easy to make future changes to our template syntax. We think we have these core pieces working pretty good and would like to get some early feedback on them while we continue to work on the new editor / Visual Studio integration.

Download CodeSmith Generator 6.0 Alpha

Please be aware of a few things with this build:

This is not a full version of CodeSmith Generator. It does not include Visual Studio integration or the CodeSmith Studio template editor.
It is very likely to have bugs.
It’s a known issue that startup performance when starting CodeSmith.exe needs work.
That being said, it is simply a zip file and you can use it without effecting your currently installed version of CodeSmith Generator.

We would love to hear any feedback that you might have. Especially if you have problems running your existing templates. Also, please be aware that if you have assemblies that you are referencing from your templates, you may need to recompile them against the 6.0 assemblies to get your template to work.

CodeSmith Generator 6.0–Status and IntelliSense

ejsmith — Wed, 23 Mar 2011 16:29:00 GMT

Work on Generator 6.0 is progressing pretty nicely. We are code complete on our new parser, new compiler, new caching mechanism, out of process template execution, and WPF based property grid. There is a lot to talk about there, but I have also started working on implementing IntelliSense and I wanted to talk about that today.

Better working IntelliSense has been our biggest request probably from day one. People tend to take IntelliSense for granted because Microsoft has done a great job with it in Visual Studio and it almost becomes transparent. We just use it and expect it to work without thinking about what goes into it. I was just as guilty of this as everyone else until I started working on implementing it in CodeSmith Studio. Think about it for a second, you have a collection of text files and you need to make sense of them… what goes into that? Here is a simplified list of the steps and parts required for implementing IntelliSense.

Tokenize – the first step is to take that text file and create a series of tokens from it.
- This sounds easy, but in and of itself it can be pretty hard because of ambiguities in languages where certain things have different meanings depending on their context.
- In our case, things are even more complicated by the fact that we are mixing random languages together.
- Also, keep in mind that this needs to be happening in real time as the user types without slowing the editor down.
Parse – next we read the tokens in, make sense of them, validate them and construct an abstract syntax tree along the way so that we can have some sort of structure that can be more easily understood, traversed and transformed.
- This is expensive to build and needs to happen in a background thread and constantly be updated as the user is typing.
Gather Type Information – we need a database of reflected and parsed type information from any referenced assemblies and source files so that we can use that information in resolving and knowing what members are available for any given type.
- It is expensive to reflect over every single type in every single referenced assembly. So you need to cache this information in a structure that makes it really fast to load and search.
Gather Context – when an IntelliSense request is made, we need to figure out where we are and provide as much information about our surroundings as possible.
- Due to the fact that active typing is going on, we have to use a combination of token scanning and looking at our most recent AST to provide this information.
- Also due to active typing, we have to work with an AST that has information that is slightly out of date and we will need to translate that out of date information to the current source.
Resolve – now we need to take all of the information that has been gathered and use it to figure out what we are looking at and what options are available to show in the IntelliSense list.
- There are a lot of rules that come into play here like member scope and precedence for extension methods to name a couple.

Another thing I would like to address is that many people in the past have said to us… "just use Microsoft's IntelliSense". Unfortunately, it's not that easy. Microsoft does not expose their IntelliSense services and, even if they did, we most likely would not be able to use them because they would be very much geared around their use cases which would make them unusable for ours.

As you can see, there is a LOT of work that goes into providing that handy little drop down menu that magically pops up and makes it so that you don't have to memorize an entire framework library. We are completely re-writing every single one of those pieces in CodeSmith Generator 6.0 and our end goal is to make it so good that you completely take us for granted because it just magically works.

CodeSmith Generator 6.0 Template Parser Progress Report - Part 2

ejsmith — Wed, 02 Feb 2011 18:55:58 GMT

In this post I’m going to talk about how the new template parser in CodeSmith Generator 6.0 creates an AST (Abstract Syntax Tree) and then uses the visitor pattern to iterate over the nodes in the AST and do transformations.

The grammar rules I talked about in my previous post are used to both validate the syntax of the template code and to create typed AST nodes. Here is a very simple template:

<%@ CodeTemplate Language="C#" %>
<%= DateTime.Now %>

The resulting AST from the parse looks like this:

DirectiveNode Name=CodeTemplate
AttributeNode Language=C#
ExpressionNode
ValueNode DateTime.Now

Each node in the AST implements a visitor pattern and I’m actually using a visitor to generate the AST visualization above. The visitor recursively goes through each node in the hierarchy and calls the Visit method for that specific node type. So I end up with a visitor class that looks like this:

public class VisualizeTemplateVisitor : TemplateAstVisitorBase
{
    …
    public override void Visit(DirectiveNode node)
    {
        _builder.Append("DirectiveNode Name=");
        _builder.AppendLine(node.Value.Trim());
    }
    public override void Visit(AttributeNode node)
    {
        _builder.Append(" AttributeNode ");
        _builder.Append(node.Name.Trim());
        _builder.Append("=");
        _builder.AppendLine(Format(node.Value));
    }
    …
}

I simply call the Accept method on the visitor and pass it the root node that I want to visit and it creates the output above:

string nodeVisualization = VisualizeTemplateVisitor.Accept(templateNode)

This pattern is used extensively in the new parser. Here are some of the visitors I have so far in the new parser:

CollapseWhitespaceVisitor – used to traverse the AST and remove whitespace and line break nodes from lines that only contain template code. This visitor actually modifies the AST while it is visiting it.
PopulateParseResultVisitor – used to gather information about the template like what sub-templates are referenced, what include files are referenced, what assembly references are declared, what language the template is using (ie. C#, VB, JS) and many other things.
GenerateCompileUnitVisitor – used to generate a CodeDOM compile unit from the AST that can be fed into the .NET compiler and create an assembly.

By creating an AST and using the visitor pattern, I’m able to do any sort of transformation that I want by simply creating a new visitor and plugging it into the template compilation pipeline. As you can imagine, this ends up being a much cleaner, simpler and extensible solution than what we had before. We may even expose the compilation pipeline with MEF in CodeSmith Generator 6.0 so that anyone could make changes to the generated code or implement new template directives.

CodeSmith Generator 6.0 Template Parser Progress Report - Part 1

ejsmith — Wed, 05 Jan 2011 00:00:00 GMT

I am currently working on the new parser for CodeSmith Generator 6.0. When I wrote the original template parser back in 2004, I really had no idea what I was doing. :-) I had no idea what an AST was or what an LL(*) parser was, etc. That being said, I think I did an OK job and it has certainly served it’s purpose, but it can’t be easily changed and things are more complicated than they need to be.

I have been working on our new parser for a few weeks now and I have a pretty good start. Here is a list of things that we want to get out of the new parser:

Structure - Generate an AST as an intermediate step so that we can do transformations using a Visitor pattern and even LINQ queries.
Grammar - Create a grammar for the syntax so that we can give grammar errors.
Testability - Make it easy to unit test.
Maintainability - Make it easy to implement changes and add new features.

I decided to make use of the parser framework that Actipro includes with their SyntaxEditor control. It is an LL(*) parser framework that lets you define the grammar right in C#. Here is what a sample grammar production looks like:

codeExpression.Production = @expressionStart + @expressionText["expr"] + @expressionEnd
> Ast<ExpressionNode>()
.SetProperty(n => n.Value, AstFrom("expr"));

This code is defining the grammar rule that makes up a code expression <%= SomeValue %> in a template. It says that there will be an expression start token followed by some expression text and then an expression end token. It is also constructing an AST node for the expression. The parser framework has really helped get our new parser of the ground quickly and I highly recommend it.

The cool thing about defining a grammar is that if there are errors in the template, you will get much more intuitive error information about what the parser saw and what it was expecting to see. Previously, we pretty much just parsed the template, generated code from it and tried to compile it. Here is an example of how we will be providing much better errors in Generator 6.0. Here is a template that has a missing % at the end of the first line:

<%@ CodeTemplate Language="C#" >
<%= DateTime.Now %>

Here is the error that Generator 5.3 gives:
error CodeSmith0104: You must specify a valid template language.

And here is the error message that the new parser gives:
Line 2, Char 3: %> expected.

In my next post I will talk about the AST and how we are using the visitor pattern to do transformations on the parsed template content.

P.S. If you haven’t figured it out by now, I’m really bad about writing to my blog. Rather than break out the random excuse generator, I’m just going to start posting. Happy New Year!

CodeSmith Insight Launch Party

ejsmith — Thu, 30 Sep 2010 22:11:12 GMT

CodeSmith Tools is excited to announce the upcoming release of our new product, CodeSmith Insight, and we would like to invite you to come celebrate! Join us at the Addison Conference Center‎ on Tuesday, October 12th, from 6pm to 9pm, for the:

CodeSmith Insight Launch Party

Prizes and Giveaways
There will be over $1,000 in prizes, including: Apple iPad, Crucial C300 128GB SSD, a new Microsoft Xbox 360 Slim, Nintendo Wii, and more! Plus all attendees will receive a free CodeSmith Insight T-Shirt, as well as a 50% Lifetime Discount for their CodeSmith Insight subscription!

Games, Competition, and Food
This will be a Retro Game Party. There will be a series of old school arcade machines, including: Tetris, Streetfighter 2, Pac-Man, Mortal Kombat, NBA Jam, Metal Slug, Galaga, and more! High scores will be kept throughout the evening, and at the end of the party the leaders will battle it out for the aforementioned prizes! Also, there will be free pizza and drinks for all attendees.

What is CodeSmith Insight?
CodeSmith Insight is a help desk software with advanced application integration. This means that Insight can handle your email and user feedback, as well as your application errors and crash reports, all from a single convenient location. Insight is useful to anyone who makes software.

Please RSVP at our party website!

Be sure to visit our website and RSVP if you are coming, as there are a limited number of spots and they are filling up fast! http://party.codesmithtools.com/

We hope to see you there,
The CodeSmith Tools Team

PLINQO at the Dallas .NET Users Group

Eric J. Smith's Weblog — Tue, 07 Jul 2009 19:23:25 GMT

This Thursday July 9th, Shannon Davidson and Tom DuPont will be talking about PLINQO at the Dallas .NET Users Group at Microsoft's Las Colinas Office in Irving, Texas. If you are in the area, we would love for you to stop by and see what PLINQO is...(read more)

The NuSoft Framework

ejsmith — Thu, 20 Sep 2007 17:41:00 GMT

Really nice set of new templates: http://www.codeplex.com/nusoftframework

"The NuSoft Framework is an extensible business and data layer framework. It is built as a set of CodeSmith templates. The templates generate a strongly typed C# domain model based off your SQL 2005 database schemas. Our code framework is a great way to get a custom development project off the ground on the right foot."

Tips & Tricks: Merge Strategies

shannon — Sun, 10 Jun 2007 04:08:00 GMT

Using merge strategies in CodeSmith enables you to combine both user and machine generated code in a single file. You can also regenerate the machine generated code while still preserving your custom code modifications.

CodeSmith comes with two types of merge strategies out of the box:

· InsertRegion - Allows you to insert a region of generated code into an otherwise custom code file. You can modify the code outside of the target region and your changes will be preserved during regeneration while overwriting the contents of the target region.

Public Class InsertRegionSample

Public Sub SomeCustomMethod()

' This is my custom code that I want to

' preserve. I can make changes to it and

' my changes will not be overwritten.

End Sub

#Region "Sample Generated Region"

' This region will be overwritten during each

' template execution.

#End Region

End Class

· PreserveRegions – Allows you to designate multiple regions inside of the file as custom code regions. You can modify the code inside of these regions and your changes will be preserved during regeneration while the rest of the file will be regenerated. Any regions whose name matches against the supplied regular expression will be preserved during regeneration.

Public Class PreserveRegionsSample

#Region "Custom Region 1"

' This is my custom code that I want to preserve.

' I can make changes to it and my changes will

' not be overwritten.

#End Region

Public Sub SomeGeneratedMethod()

' This section and all other regions that do not

' match the region name regular expression that you

' supply to the merge strategy will be overwritten

' during each template execution. In this sample,

' the regular expression being used matches any region

' who's name starts with "Custom".

End Sub

#Region "Custom Region 2"

' The contents of this region will also be preserved

' during generation.

#End Region

End Class

· IMergeStrategy – Allows you to implement your own merging logic.

The attached samples show an example of using both kinds of merge strategies as well as a CodeSmith Project file to execute them. Simply, right click on the MergeSamples.csp file and select Generate Outputs to regenerate. You can also see the merge settings being used by right-clicking the MergeSamples.csp, selecting Manage Outputs, and then clicking edit on each output to see the settings.

CodeSmith 4.0 Released

Eric J. Smith's Weblog — Wed, 08 Nov 2006 19:11:00 GMT

CodeSmith 4.0 has been officially released today at the Dev Connections conference in Las Vegas! This is a huge release for us and has a bunch of great new features as well as countless minor enhancements and bug fixes all based on customer feedback....(read more)

CodeSmith 4.0 Public Beta Released

Eric J. Smith's Weblog — Fri, 20 Oct 2006 00:53:00 GMT

CodeSmith 4.0 Beta 2 has been released and is now available to the public. Here are just a few the new capabilities in CodeSmith 4.0: CodeSmith Projects (.csp) - This feature makes automating your code generation process really easy and consistent whether...(read more)

CodeSmith $5,000 cash give away at Dev Connections

Eric J. Smith's Weblog — Sat, 07 Oct 2006 21:57:00 GMT

CodeSmith will have a booth at Dev Connections in Las Vegas next month and we thought it would be a lot of fun to give away some gambling money! If you are coming to Dev Connections, just make sure to stop by our booth and signup. If you aren't coming...(read more)

XML-based Code Generation with CodeSmith

ejsmith — Mon, 22 May 2006 20:23:00 GMT

Mike Gunderloy has just posted a new article about XML-based Code Generation with CodeSmith over at Developer.com. Mike does a great job of explaining the new XmlProperty feature in CodeSmith and how powerful it can be in using XML as metadata for your templates.