Getting in the flow: According to wikipedia, flow is the mental state of operation in which a person performing an activity is fully immersed in a feeling of energized focus, full involvement, and enjoyment in the process of the activity. In essence, flow is characterized by complete absorption in what one does. Focus, immersion, being concentrated, involvement …  being everyday in the flow by coding hours and hours, contribute a lot to a solid positive state of mind.  These are moments where one can completely forget about minor everyday pesky stuff, but also forget for a while more serious problem in life everyone has to face. Being in the flow is the condition to solve challenging problems and to create beautiful pieces of engineering. Being in the flow can lead to addiction but it is not addiction. This is essential to control when to check-in in the flow and when check-out, making sure to not be disturbed meantime.

Being creative: Being a software engineer is one of the most mainstream way of being paid for being creative. Often, writing software is deemed to be an artistic activity. A programmer has to be humble, because this is a kind of art not understood by the mass. But being humble is a chance to become wiser and increase self-confidence. Also, knowing you are going to be creative for a while, is an excellent motivation to overcome the first step effort to jump in the flow. But the truth is that for every passionate programmer, there is a background thread in the mind in charge of creativity (often running at sleep-time), that makes it so that in the morning the envy to create what you have in mind is too strong.

Becoming an expert: It is common to hear that a programmer must know numerous technologies, that its skill is to learn how to learn new technologies. On this I disagree because what makes me really happy is to master completely a technology and exercise daily my expertise. I used to master all the tricky details of C++ and COM and it was fun. Before that I used to master some assembly level programming and it was fun, I was not even getting paid for that. In 2002/2003 and then  2005 I wrote two editions of a 1.000 pages book on .NET and C# and writing it has been one of the most blessed moment in my career. Since then I capitalize on this knowledge every single hour of coding, letting me focus my thoughts on problems to solve, and not on all the non-trivial things a complex platform like .NET is actually performing under my feet. Of course I am constantly discovering new details about surrounding technologies, like functional programming paradigms through the prism of functional paradigm introduced in .NET languages. But I know what is my core knowledge, both in terms of technologies and in term of program design skills. And as long as I won’t be forced to change my core skills, relying on my expertise to express my creativity and making a living on top of it is a source of personal achievement.

Meet inspiring people sharing the same passion: I imagine meeting peers is a source of happiness for every expert in something. This is also why investing in a solid programming particular set of skills is a positive thing. Not only respect from others programmers arise, but it lets have great exchange with smart people as passionate as you. The importance of flow, underlined above, also comes with the disadvantage of being often alone with your thoughts. Most programmers enjoy working alone anyway, but for those who need a bit of more social activity, having an expertise in something is also a great way to become partly a teacher (in professional or academic spheres), partly an architect and contribute to important decisions, partly a team-leader and be responsible for project progressing, partly a consultant, and be able to share your knowledge in a pleasant social environment. I put the word partly in italic because if your social activity make it so that you are not writing code meant to be run in production anymore, you shouldn’t consider you as a programmer anymore and you’ll loose a great deal of the points I am mentioning here. If you need social interactions all day long, programming is not for you.

Being involved in something that make sense: Here also my position might be a bit different of what is widely accepted. I agree that for juniors, it is important to multiply the opportunities to work in several different teams and companies, to get an idea of what they like and what they don’t, to be influenced by several inspiring mentors. But once you become seniors, working on the same application in the long term, where you feel well programming in,  polishing it days after days, seeing its evolution across years, maintaining it in a clean state by adopting modern paradigms like automatic tests, DbC or relentless refactoring, having your word to say about strategic decisions, personally I found this being a great source of daily happiness and a great motive to involve myself! In addition, working hard to achieving important milestones regularly, is an excellent way to give a sense to your professional career, which is (much) more the exception than the rule.

Work wherever, whenever you like: A 2KG laptop with the proper tools set installed, a few hours of electricity every 24 hours, this is all what a programmer needs to do his job well. A descent internet connection is often appreciated, and there are today only few points on earth where internet is not available at all somehow. Programming might be probably the less demanding working activity in terms of time and space requirement. Getting in a flow in a 12 hours plane flying across the planet, scheduling half a year to live and work in a paradise tropical island, avoiding traffic jam by staying at home for work (in your pijama), delaying programming in the night or early morning to take care of the kids and their education, all this is not only possible but pretty common actually. Most serious software companies let some of their skilled engineers work wherever they prefer. Did you know that many of the great minds behind Visual Studio didn’t actually moved with their family to Seattle, but still live in their country, sometime far away from the US?

Make a decent living doing something you like: Last but not least, everywhere, skilled programmers get paid above the average salary in their countries, and if we take the example of a developing country like India, good programmers get a much much higher income than the average. On top of that, a skilled programmer have pretty close to zero chances to remain unemployed for a long time. This also means that if you don’t like your current position, it is easy to find a new better suited one. This situation is made possible because less than two decades ago, the modern civilization realized that IT is the condition for its development. It is a fact that many of the richest persons in the world were originally programmers and every motivated programmers has the potential to create its own ISV business and become a millionaire. A programmer can also decide to make more money by coding for the financial industry, or even bet on a startup and potential makes millions in a few years.

All the next big things will consume even more IT, this includes human genome analysis for the mass, the entire medicine that will be deeply impacted by that, more prevalent portable devices, more sophisticated entertainments, augmented reality, robots and automated machines to do surgery, to assist the increasing growing number of old people, artificial intelligence in the long term, and certainly everything nobody hasn’t imagined yet. After all 20 years ago, nobody anticipated the impact of Google. 10 years ago nobody anticipated the impact of Facebook and smart phones.

Ok enough getting in the flow of writing my passion for programming, I have some code to write before the weekend

For example, if you look at our Web API sample, you’ll see there’s a bunch of friction if you just try to get Web Api working from scratch.

• You need to add using statements for each namespace you want to use. This is a lot more painful than one might think when you don’t have intellisense.
• You need to configure web API, this involves creating a host, defining default routes etc. Adding lots of object creation and such starts to make the script pretty hairy. Not impossible, but painful when there’s no template.
• You need to teach Web Api how to resolve controllers in script by implementing a custom controller resolver.

Now pull in the Web Api script pack (scriptcs –install scriptcs.webapi) using the Require<WebApi>() function and your boiler plate code evaporates to this:

The script pack does all of the following to make the experience better:

• Removes the need for using statements for common namespaces. The script pack provides those which is why you don’t have to add the web api namespaces in your example above.
• Adds dll and nuget package references that bring the dependencies the framework needs.
• Removes general boilerplate code. In the previous sample you need to create a host, define routes etc as I mentioned. In this case the script pack creates the host for you and configure with the default routes. You can customize if you need to.
• Provide APIs to fill gaps that prevent the framework from working well in script / supporting dynamically emitted assemblies. The Web Api script pack brings in and configures a custom controller resolver for you.

We’re just getting started with the work we’ve done with script packs, but they are a really nice extensibility point and really take advantage of nuget as a delivery mechanism. The community has been rising to the occasion and building out quite the gallery as well.

There’s some great posts about script packs covering topics like how to build them or even use them from the REPL that you should really check out.

• http://blog.martindoms.com/2013/05/14/building-scriptcs-script-pack/
• http://scottksmith.com/blog/2013/05/14/deeper-dive-into-scriptcs/

Have fun exploring the new world of scripting in C# with scriptcs!

1. Standardizing the look and feel. It is far more consistent and maintainable to set attributes (including a css class) and event handlers in one centralized place.
2. Simplify the script. Often a part of the logic the script will follow is already known server side. Instead of writing everything out in javascript rendering the intended statements leads to a leaner client. There is an example in my previous post on building post-data.
3. Decoupling the libraries used. At the moment we are using the Telerik MVC suite. In my previous post I described how our html helpers build standardized Telerik components for our views. In the not to far future we want to switch to the Telerik Kendo suite. Having wrapped up the library dependency in our Html helper will make this switch a lot easier to implement.

What has evolved is the way we work with the model. In MVC the implementation of the controller and the view is clear. When it comes to the implementation of the model there are almost as many different styles of implementation as there are programmers. In general the model can bring any data to the view you can imagine. Not only the source of the data varies, from plain sql to a C# property, also the use of the model’s data varies. It can be a customers name from the database. Or it can be the string representation of some html attribute needed for a fancy picker. Here data and presentation start to get mixed up. Our extensions needed information for the Html-Id. The original Html helper had a custom constructor to get that specific data from the model into the helper. Which required to create our own htmlhelper when starting the view and use that one instead of the standard @html. As seen in the eposHtml in the previous story. It would be cleaner if our extension methods could be satisfied with the default html helper. It would also be cleaner to keep a better separation between ‘real’ data and presentation.

The model is available in every HtmlHelper extension method.

It’s a property of the ViewData.

In our case we needed something to give the control an unique Id. The Id method builds that Id. Previously we passed the Id-base in the constructor, which lead to the custom helper. A far more elegant solution is using a very basic IOC-DI pattern. As implemented By the Id method

The method queries the model first for the IProvideCompositeId interface, in case the model does not implement that it is queried for the IprovideId interface. Resulting in a string which can be safely used in an HtmlId. (A negative number would lead to a ‘–’ in the string, which is not accepted in an Html Id).

These interfaces are very straightforward

In case the model is going to be used in a view requiring unique Id’s the model has to implement one of these interfaces.

Working this way:

• We can use our custom html extensions in the default html helper
• Specific data from the model is available inside our extensions
• The model and the view do not get entangled

The code is no big deal. I know. But the model is something whose horizons are still not in sight.

As its name suggests this rule matches fields assigned from multiple methods. Bugs due to corrupted state are often the consequence of fields anarchically assigned.

On many code bases tested, I found the threshold of 4 methods writer to be the best balanced number between too many false positives, and a decent amount of suspicious matches to look after at code-reviewing time. Of course this threshold is easily customizable.

Several parameters can be taken account.

• Is the field static? in which case it can be pretty serious if many methods assign it.
• Does the field type is a value or a reference type? In the case of a reference type, such situation might reveal a potential NullReferenceException scenario luring.
• Is the field assigned outside from its class. Since another rule prevents fields to be visible from outside its class, this case has been eliminated.
• How many assigner methods of the parent class are visible outside the class? How many  methods calling directly or indirectly an assigner method are visible outside the class?
• Can a particular class instance be accessed from multiple threads? in which case the situation sounds pretty alarming.

Not all cases are taken account since static analysis have limits and for code review warning purposes the rule algorithm must be kept easily understandable. Here is the rule code and comments that contain insight about how to generally fix such issue.

I’d be curious to read your suggestions and comments on that.

And here is what it looks like when we run the rules on the NUnit code base:

Before you go further, if you are wondering what all the scriptcs hype is about please check out Scott Hanselman’s great post and his new Tekpub video.

Last few days I’ve been working on a new REPL experience for scriptcs and now it’s in! REPLs are nothing new to dynamic languages but they have not really been available in C# with one exception, Mono has a great REPL. Roslyn introduces the Roslyn interactive window in VS which is also a REPL which runs in the editor.

This REPL is different than both in that it is specific for scriptcs, and like the rest of the scriptcs experience, there’ s no IDE required. Basically it combines the goodness of scriptcs (nuget) with an interactive experience. You can just install some nuget packages and type code which instantly executes. For example imagine just pulling in HttpClient and then just doing some http requests!

Below you can see I am installing the mongo nuget package. Then running scriptcs by itself and typing in some simple code to work with the package.

And then we have pretty error handling.

Thanks to the Roslyn team’s efforts layering a REPL on top was AMAZINGLY easy. The REPL part itself was like 15 lines of code! And thanks to the nuget team for even making this a remote possibility through having an awesome package ecosystem. I look forward to ALSO seeing the mono version soon

If you wanna try it, you can grab the latest from github at http://github.com/scriptcs/scriptcs in the dev branch.

It will also be on our chocolately nightly builds tomorrow: http://www.myget.org/F/scriptcsnightly. It should be on our public feed very soon there after.

Practical ReST – the book

I’ve been talking for years about writing a brain dump of what I know of ReST. The project has started, you can pledge to buy your copy today (in various editions) and get content as early as next month!

The response has been amazing, with three quarters of the minimum funding pledged in 48 hours. Of course, the more contributors the more time I can allocate to the book, the quicker the book will be in your hands, so don’t hesitate!

Writing a book in itself is a big challenge, and the scope is very ambitious. But it’s also a foray into self-publishing, and I expect it to be an amazing opportunity for learning.

Practical ReST – the class

I’ve been giving a ReST class for a while, but I’m rewriting it so it follows the same structure as the book, and make it open to people with many technological backgrounds.

First date should be in May in London, and I expect Warsaw, Vilnius and other locations to pop up this year.

If you want to see a workshop organised in your city (or your company), pop me an email.

I’m also launching a referral program *and* group rebates, so if you’re a training company and you want to add the course to your books, you can too.

Hanselminutes podcast

Scott was nice enough to have me on his show this week, talking about Practical ReST. It’s published and ready to be listened to, and while I was half-way through an asthma attack and having microphone problems, Scott saves the day by being a great host.

Cancer Research UK

You may remember that I asked for your contributions to help me raise money for cancer research UK. I can now announced that we raised more than 700£, and that both my objectives of non-drinking were achieved. I cannot thank you all enough for your generosity.

serialseb.com

It had to happen. I now have my own web site. You cannot imagine how difficult it is for me to put work and effort in talking about what I did rather than what I want to do. But you will find on there a link to all my previous talks, with videos and slides (where applicable), as well as where you’ll find me next, my future talks and where classes will be delivered next.

What next?

ReST will be my focus for the next few months. I have a couple of other projects up my sleeve, but this is enough announcements for now, don’t you think?

If you haven’t seen it, it looks exactly like this:

The reason this is happening is because by default we don’t allow errors to propagate back to the user. This is in order to prevent you publishing to production and leaking details of your system.

Let’s assume I just published a simple express app, but I made an error. Below you can see that instead of requiring winston, I required win$ton.

When I publish, I’ll see the page cannot be displayed error.

iisnode.yml to the rescue

To find out what is going on, you can enable displaying errors using our YAML configuration file otherwise known as iisnode.yml. If you use our azure-cli tool, and create a site in a directory where there is a node app, then we automatically create it for you. If not, you can create one really easy with any text editor. The entries you need to put in the file / update are below.

Once you save that file, go republish to Windows Azure. If the file did not exist and this is an Azure Website, do an “azure site restart”. if the file already existed, you won’t need to do anything.

Now with debugger errors enabled, when I see the following:

Bingo!

Don’t go to production this way, but DO use tail for websites.

Make sure to disable debugger errors when you go to production. You can leave logging enabled however, because another command you can still use is “azure site log tail” which will stream logs directly to your console realtime. Also if you are a cloud service, you can remote in to view the logs written locally.

Using “azure site log tail” watch what happens when I refresh the page.

azure-log tail has many more uses. You can use it to capture any log output that your applications give you in a realtime manner. For example here’s a screenshot of me tailing an app using socket.io and Service Bus in Azure.

Yes it is awesome!

PS: Today “tail” works only for node apps, but that is going to change VERY shortly.

If you are using our azure-cli then you might have though of creating automation scripts to package up common tasks. A while ago I posted on how you can achieve some script-ability just using bash tools. Basically the techniques I listed there involve piping text from command to command and using tools like grep and awk to parse the results in order to feed to the next command.

It turns out there’s another way, and one that is much closer to the kind of fidelity you an achieve with Powershell. Sound interesting? keep reading. It turns out the majority (like 99%) of our cmds can return JSON objects if you apply the –json switch.

For example if I do “azure site list –json” here’s what I get.

That means that I can take that result and parse it into a JSON object which I can then easily manipulate. Potentially I could even pipe results in from one cmd to the other. I could even imagine being able to define cmds as JSON objects so that I could script out my tasks in a more object friendly fashion. I could then wrap that all up in a nice pretty box to be used by myself or others.

And that is what azure-scripty is about. (Kudos to @JpScripter for reminding me I need to post.)

azure-scripty gives you a JSON oriented API that you can use within a node script for automation tasks. It lets you combine the power of raw node with the capabilities of our azure-cli. One thing that is also really nice is it leverages the knowledge you already have of the CLI. As it says on the github page, “if you know the CLI you know scripty.”

azure-scripty offers you the following:

• Works anywhere node works i.e. Windows, Linux, Mac, Nodecopter
• Author using pure string command or JSON object command styles.
• Uses the standard node callback model
• Batching multiple commands together.
• Piping results from one command to the other

Here’s a few examples to give you an idea:

String command style

This will list out my websites in json format.

Object command style

This sample creates a new mobile service using the object oriented style. Notice you can pass fixed position args, with all other named args being by convention.

Batching

This sample shows creating a web site and a mobile service using the object oriented style.

Piping

One of my favorites. This example shows using piping to pipe the list of sites into the stop command. Notice that :Name, that’s plugging in the Name parameter from the returned site.

Get it and please give feedback

1. npm install azure-scripty
2. go check the README.
3. Start scripting!

Looking forward to your feedback and your contributions!

One thing we can agree on, there is a massive lack of maturity in this business. At one time or another, including yours truly, has been guilty of an off-color joke. Most of the times, at community gatherings like conferences; we are typically amongst friends and feel comfortable that we can be ourselves, letting our guard down. At the end of the day, we must rely on the fact that in this business, we have to have a thick skin and that we are adults. That in no way is meant to justify offensive comments and that somebody like Adria didn’t have a right to be offended. She did have a right to be offended. And to that, it’s not for anybody else to question it. It’s enough for anybody to say they were offended – and others should respect that. That said, does it mean the offended have an open ended license to exact whatever retribution they want on the offenders? No…. In these community gatherings, there is a level of privacy, within the community cocoon; we have a reasonable expectation to.

We see the t-shirt shirt = “Fork you”. We see on open source projects: “Fork me on GitHub”. Doesn’t take a rocket scientist to see the double entendre here. If one is to where that shirt in public, does that give anybody who is offended license to snap a picture, tweet it, and then hope that some misfortune befalls that person? I don’t think such license is granted.

Consider this YouTube video from Ms. Richard’s: http://www.youtube.com/watch?v=nMpJSKbsmgQ. There’s a saying in the law that in matters of equity, you have to have “Clean Hands.” Is it permissible for her to be offended as she was when she makes a video that could be taken as offensive by others? At about 3:03 into the video, she talks about an incident that was done to her – that could very well be taken as offensive, and she let it pass. In this case, she decided to not let something pass…something that WAS NOT said directly to her. If I was prosecuting this case, and I was going to depose Ms. Richards, this is a line of questioning I would surly explore to great details. What was her real motivation here?

This gets to my first question:

Was there a reasonable expectation of privacy by these two gentlemen?

I believe there was. They were having a private conversation, which Ms. Richards overheard. Yes, this was a public place. Did she have the right to express her indignation at those comments to these two gentlemen? Absolutely yes. In fact, that would have been the more impactful thing to do. That would be a proportional response. Reporting the incident to the PyCon staff is not appropriate. They are not there to police what people say. Note, these two guys didn’t say the comment to her. She thrust herself into their conversation. By the same token, the two guys allowed her to confront them. She didn’t. Instead, she took their picture and posted a tweet. And from there, SHE took what was a private thing and made it public. She made the situation worse. In that regard, she violated the privacy rights of these two men, notwithstanding their comments. Speaking of those comments, only her, the two guys and maybe one or two others know what the precise words were. The first result, at least one of the guys lost his job.

Next question..
Does the guy who got fired have a cause of action against Ms. Richards?

This is the part you want to pay attention to as you should think very carefully about the consequences of tweeting something that is private and further, identifying specific people where harm could come to them. First, did these two guys have a right to say what they said, no matter how stupid it was? Yes, they had a right. There is such a thing as a First Amendment last I checked. It’s also not illegal to be stupid. Did the guy who got fired, his employer, have a right to fire him? If he works in an employment at-will state, the answer is yes. Is he totally out of luck? No.. He can sue Ms. Richards for Tortious Interference with an Employment Relationship. She knew and intended a certain level of a negative outcome to these two gentlemen. That’s malice. While his employer may be on good ground to fire him, he can nevertheless sue Ms. Richards to recover damages. There may also be a cause of action on defamation grounds. These two guys are being labeled as sexists when all that may have happened was a joke, off color as it may have been, that offended this one person. That too may be the basis of a cause of action.

Next question..
Is there any other entity that could be liable?

As SendGrid has already admitted, Ms. Richard’s was there in her capacity as a representative as a SendGrid Employee. Ms. Richard’s did not do SendGrid any favors when she tweeted that she had SendGrid’s full support – a claim that SendGrid didn’t dispute. In the law, there is the legal doctrine of “Respondeat Superior.” It effectively means that the master is liable for the actions of his agent. Ms. Richards has a large Twitter following and that is due in least in part to her former job as a developer evangelist for SendGrid. There’s a strong argument to be made that SendGrid is liable, at least in part, for Ms. Richard’s actions – regardless of the fact that they terminated her services.

This whole episode illustrates again the importance of how we react emotionally and the legal consequences therein. If I’m the gentlemen who lost his job, I’m exploring my legal options – and there are legal options. Above all, this is a teachable moment. If you are offended at something somebody does, consider the context. 99.99% of the time, these matters are best handled privately. Takes more courage and is more impactful when you communicate your disdain to the offending party directly. If your goal is to change behavior that will more likely have the desired effect. Sure, you may be told to eff-off. That’s a risk…a risk worth taking.

How about conference organizers?

I understand that PyCon has modified their code of condut… Didn’t know that such a thing existed. I would always suggest to any conference organizer to NOT get into the business of policing their attendees. Rather, I’d rely on the venue staff for that. Obviously, if somebody is disruptive, boot their tails out. These guys didn’t appear to be disruptive at all for the record – not that it really matters in this case.

A word about the “Brogrammer” culture..

You probably know about it. If not, look it up on our favorite search engine. The fact is, guys are often not the sharpest knives in the drawer socially and are often in need of an increase in maturity level. Women have a tough enough time in this business. I know several women that do a lot of work in getting more girls interested in software development. Sad as it is, Adria Richards did more to set that effort back than the two guys she was offended by. That’s a perfect example of irony!! It’s really a shame. Ms. Richards looks to be a pretty sharp individual – one who appears to have better common sense than she displayed here. Learn from it folks.

As soon as it shipped we started to get reports of issues using our SDK and CLI. We’ve investigated this and uncovered that there are some things we need to fix. In the meanwhile, node 0.10.x is not currently supported. We’d ask everyone using our SDK and CLI to stick to node 0.8.x for the time being.

If you are using our Mac or Windows installers you should be fine as they each include private copies of node with working versions.

We are on top of this and will be working on this in our next sprint. We’ll keep you posted and let you know as soon as we have it working which we expect to be very soon.

Thanks for your understanding and sorry for any inconvenience this has caused.

Nate soon after posted:

It’s funny, you never see someone with the title “Master Craftsman” at a startup. I wonder why?

— Nate Kohari (@nkohari) March 5, 2013

then

I take it all back. I would strongly encourage startups to use TDD while developing any products that compete with ours.

— Nate Kohari (@nkohari) March 5, 2013

I wanted to write a few comments about TDD in startups. Good code is the least of the risks in a startup. Sorry but worrying about technical debt making us go slower when we have a two month runway and likely will pivot four times to quote Bob.

Captain Sulu when the Klingon power moon of Praxis exploded and a young Lieutenant asked whether they should notify Star-Fleet: “Are you kidding?” ARE YOU KIDDING?

One of the biggest mistakes in my career was building something appropriate…

It was just after Hurricane Katrina. I was living in a hotel. An acquaintance asked me if we could hack together this business idea they had for a trading system. He had the knowledge but not the know how. I said sure, hell I was living in a hotel!

In less than two weeks we had an algorithmic trading system. It was a monstrosity of a source base. It was literally a winforms app connected directly to the stock market. UI interactions happened off events directly from the feed! Everything was in code behinds (including the algos!) Due to the nature of the protocol if anything failed during the day and crashed the app (say bad parsing of a string?) the day for the trader was over as they could not restart.

But after two weeks we put it in front of a trader who started using it. We made about 70-80k$ the first month. We had blundered into the pit of success. A few months later I moved up with the company. We decided that we were going to “do things right”. While keeping the original version running and limping along as stable as we could keep it while adding just a few features.

We ended up with a redundant multi-user architecture nine months or so later, it was really quite a beautiful system. If a client/server crashed, no big deal just sign it back on, multiple clients? no problem. We moved from a third party provider to a direct exchange link (faster and more information!). We had > 95% code coverage on our core stuff, integration suites including a fake stock exchange that actually sent packets over UDP so we could force various problems with retry reconnects etc/errors. We were very stable and had a proper clean architecture.

In fact you could say that we were dealing with what Bob describes in:

As time passes your estimates will grow. You’ll find it harder and harder to add new features. You will find more and more bugs accumulating. You’ll start to parse the bugs into critical and acceptable (as if any bug is acceptable!) You’ll create modules that are so fragile you won’t trust yourself, or anyone else, to modify them; so you’ll work around them. You’ll build a festering pile of code that, with every passing week, requires more and more effort just to keep running. Forward progress will slow and falter. It may even reverse as each release becomes buggier and buggier, and less and less stable. Catastrophes will become more and more common as errors, that should never have happened, create corruptions and damage that take huge traunches of time to repair.

We had built a production prototype and were suffering all the pain described by Bob. We were paying down our debt in an “intelligent” way much the way many companies that start with production prototypes do.

However this is still a naive viewpoint. What really mattered was that after our nine months of beautiful architecture and coding work we were making approximately 10k/month more than what our stupid production prototype made for all of its shortcomings.

We would have been better off making 30 new production prototypes of different strategies and “throwing shit at the wall” to see what worked than spending any time beyond a bit of stabilization of the first. How many new business opportunities would we have found?

There are some lessons here.

1) If we had started with a nine month project it never would have been done

2) A Production Prototype is common as a Minimum Viable Product. Yes testing, engineering, or properly architecting will likely slow you down on a production prototype.

3) Even if you succeed you are often better to stabilize your Production Prototype than to “build it right”. Be very careful about taking the “build it right” point of view.

4) Context is important!

Never underestimate the value of working software.

BUT……

Where are the apps? Granted, Words with Friends is finally in store. The new ESPN App is nice. But what about Taxi Magic and Pandora? How about a better FaceBook and Twitter app? Instagram? I was just at an event where somebody was showing Vine – a really cool iOS app. I looked for it on the Windows App Store…. How about Instapaper???

I’ve taken the plunge to give the Windows Phone a fair shake. I love the UI. What is needed are apps…apps people want – which will drive the platform. Whether Microsoft wants to admit it, the Apple AppStore is the store by which the Windows Store is measured. There’s A LOT of noise on the Windows Store. Note – quantity is not quality. If there is a top app in iOS, Windows needs to have that app. Microsoft needs to be focused on those vendors to get these apps built.

Another nit is the difference between Windows 8 and Windows Phone. It would be great if those two worlds could come closer together from a development standpoint. On the plus side, the process by which apps are certified for the store is simple. The Windows App Certification Kit is a nice utility and easy to use. If your app fails, there is clear guidance on what to fix.

None of this is to say there are not useful apps on the Windows/Windows Phone Stores – there are. And when it comes to things like Live Tiles, Microsoft nailed that one. A very useful feature – which makes the negative things especially frustrating to deal with.

At the end of the day, like many, I want to see Windows 8 succeed. I think and believe there to be a great amount of potential. But to succeed, the public needs to see the apps that are available in other ecosystems. Until that day comes, Windows 8, and in particular Windows Phone, will continue to be third in a 3 horse race – which shouldn’t be the case.

Disclaimer: I am not the first person to create a tool for C# scripting. There are a long list of projects from too many people to name, and I respect all of them A big part of why I did this was also to mess with Roslyn.

Learnings from node

For the past 18 months I’ve been living mostly in text editors and script getting fully absorbed in node.js. Recently I started to think a lot about my C# development experience and if there are things I could bring back from my learning’s with node / JavaScript

• No projects, just script- One of the things I love about node.js is you don’t need a project. You can just jump in a folder start creating js files and go to town.
• No IDE requirement, you can just use a text editor.
• Packages over assemblies – In node, when you want to get something you use npm to download the packages.It’s super simple. You just have your app and your local node_modules folder and you are good to go.
• No compilation – This is a big one. With node, I just run node.exe and my app and it works. I don’t have to first create an executable to run, I just run.

On first glance it might seem like this is just oil and water with C#.

Thanks to Roslyn and Nuget however I no longer think that’s the case.

What is Roslyn?

Roslyn is a compiler as a service. It gives you an API that lets you compile and analyze code on the fly. Additionally it gives you hooks to dynamically refactor and generate code. Beyond that it also includes really powerful scripting support allowing you to author applications and components as loose C# scripts. You don’t need a class if you don’t want one and you don’t need to compile. And you don’t need to use Visual Studio, you can host Roslyn in your own applications and components and it’s even got nuget packages.. You can read more on Roslyn at MSDN or in a nice post from Filip on Roslyn and Web API

Roslyn is a breath of fresh air in terms of it’s simplicity. If you have worked with the existing horrendous mechanisms for doing this like the Codedom you are going to love Roslyn. You can literally be up and running with Roslyn having it compile scripts for you in a matter of minutes.

How does Roslyn help?

I can cross off three bullets

• No projects, just script- One of the things I love about node.js is you don’t need a project. You can just jump in a folder start creating js files and go to town.
• No IDE requirement, you can just use a text editor.
• Packages over assemblies – In node, when you want to get something you use npm to download the packages.It’s super simple. You just have your app and your local node_modules folder and you are good to go.
• No compilation – This is a big one. With node, I just run node.exe and my app and it works. I don’t have to first create an executable to run, I just run.

Now yes it does compile, but that’s all taken care of for me, I don’t have to think about it.

Packages vs Assemblies

For the last bullet I looked to nuget. The nuget command line provides an easy way to install packages right from the command line into the same folder as a set of scripts. However, the challenge is how to make those scripts see those packages? Today Nuget works great with a csproj and within Visual Studio or using MS Build. But what if you don’t have Visual Studio or a csproj? Well when you install packages using the cmd line you get folders for each package that at some level actually do contain binaries. So, if you can find those binaries and get them loaded up into Rosyln, you are golden. Yes this doesn’t get rid of assemblies, but I don’t have to manage them individually, I just install packages.

• Packages over assemblies – In node, when you want to get something you use npm to download the packages.It’s super simple. You just have your app and your local node_modules folder and you are good to go.

Roslyn + Nuget = scriptcs

Having crossed off all my bullets, I then decided to spike a little with Roslyn and out came Scriptcs: https://github.com/glennblock/scriptcs

scriptcs is a little command line tool which lets you have the following work flow.

• You author your “app” as a .csx file or csharp script in your favorite editor. There’s no project.
• You add dependencies to your app by simply installing nuget packages from the command line. By convention the dlls within those packages will be loaded up. No need for references as the idea is if it’s there you want it.
• You add GAC references or other dll references (not nuget packages) by using the “r:” Roslyn syntax, i.e.”r: System.Net”.
• You run your app using “scriptcs” passing the csx file.

For example here is a snippet of setting up a web api host.

Back to reality, though the future looks bright.

At this point I would say this is just an experiment. I am not claiming you can build an enterprise app with this or anything like that. I am using this more as an exploration to understand what a c# script experience from outside VS could look like.

From here there’s a lot more places to go. One would be allowing you to have includes of other csx files as you probably don’t want your whole app in a single file! Beyond that Roslyn offers a ton of rewriting, AST analysis or even auto-refactoring capabilities that I am sure can be leveraged in scriptcs but are not.

Seeing this definitely makes me excited about where things can go in the future. I’d love to hear your thoughts.

• explore two folders containing older and newer versions of assemblies to compare
• create on-the-fly two NDepend projets and analyze them
• compare the two older/newer code base snapshots, which lead to the creation of a ICompareContext object
• use this compare-context object to explore code diff and changes

The power offered by a compare-context object is barely exposed here. We only reports 3 lists of assemblies added/removed/changed, which can already be a useful information. But much more can be done with a compare-context object in hands. I wrote recently about how Code Query LINQ can be used to write rules to detect code regression issue. This post explains how code-diff can be used to detect regression, including API Breaking Changes, loss in code quality, recent code not well covered by tests… All CQLinq queries/rules exposed are based on NDepend.CodeModel API. And thus, they can all be integrated and combined into a program that starts like the one above, and then uses a compare-context object to explore code diff.

A subtle points is that when one writes a CQLinq rule involving code diff, one doesn’t deal with a compare-context object. This is an astute we use to make CQLinq rules/queries as fluent, concise and elegant as possible. Indeed, a CQLinq pre-compiling phase takes care to transform calls to extension methods, into calls involving the compare-context. Hence the following CQLinq rule that reports methods that were complex, and that became even more complex….

…is actually transformed into this expression, where the usage of the compare-context is made obvious.

Thus the logic of this query could be easily integrated into our program this way:

In the real-world, for companies that have build servers and that care for the quality of code delivered, this ability to write custom programs that can be included into the build process chain to focus on the quality of new or refactored code can certainly be a nice option.

But one can also see value in writing a custom-tooling program to be ran on the developer desktop. Such program could also use the special method TryCompareSourceWith() that can open a source file diff tool on the right location of two versions of the same source file (the source file diff tool is the one configured in NDepend > Tools > Options > Source Files Compare Tool). For example the open-source PowerTool Code Review Methods Changed uses this TryCompareSourceWith() method to show diff in code during a code review session. Certainly a more sophisticated program could help reviewing code diff, for example with the ability to assign some users to approve some diff, and persist in a DB such approval or disapproval actions.

Enjoy!

The projections we have used so far have used a method called when(). This method allows you to match functions back to types of events. Up until now that has been a single match but you can also use more than one.

fromStream('test').when({
                           Event1: f1,
                           Event2: f2
                        });

This defines that every time an event of type Event1 is seen the function f1 should be called with that event and function f2 for events of type Event2. This is a very useful construct when trying to build out projections that require the ability to handle many different types of events.

There are also some special matches defined.

$any: $any will match all events to your function. This is useful for example when you want to build an index for all events. We will get into how this works later but you can imagine if I wanted to build an index based upon the user that created the event (stream per user) then the function would want to look at all events in the system.

It is important to remember that as of now $any cannot be under in conjunction with other filters.

$init: $init gets called before any other handler. The job of $init is to return the initial state that will be passed to the rest of your functions. In the intermission post this handler was used to set up initial state so the other handlers did not have to. The usage can also be seen in looking at the post from the Projections 3

fromStream('$stats-127.0.0.1:2113').
    when({
        "$stats-collected" : function(s,e) {
              var currentCpu = e.body["sys-cpu"];
              if(currentCpu > 40) {
                   if(!s.count) s.count = 0;
                   s.count += 1;
                   if(s.count >= 3)
                        emit("heavycpu", "heavyCpuFound", {"level" : currentCpu,
                                                           "count" : s.count});
              }
              else
                   s.count = 0;
         }
    });

In this projection the line if(!s.count) s.count = 0 is being used to initialize the state if its the first time into the function. This could also be implemented as

fromStream('$stats-127.0.0.1:2113').
    when({
        "$init" : function(s,e) { return {"count":0},
        "$stats-collected" : function(s,e) {
              var currentCpu = e.body["sys-cpu"];
              if(currentCpu > 40) {
                   s.count += 1;
                   if(s.count >= 3)
                        emit("heavycpu", "heavyCpuFound", {"level" : currentCpu,
                                                           "count" : s.count});
              }
              else
                   s.count = 0;
         }
    });

The two will work in the same way. In our next post we will start looking at how indexing works in the event store.

I have updated from now through the summer most of my schedule on the “where am I page”. I have not included everything but most is up there.

One big change we made is that we are going to spend a month in Australia. She seems to have this crazy idea of driving through the outback with kangaroos jumping everywhere (might be the movie!). While there I will stop in Sydney to teach the new class and Perth to go through the CQRS + ES class.

But the next six months looks pretty crazy! Hopefully we can make the best of it.

fromStream('$stats-127.0.0.1:2113').
    when({
        "$stats-collected" : function(s,e) {
              var currentCpu = e.body["sys-cpu"];
              if(currentCpu > 40) {
                   emit("heavycpu", "heavyCpuFound", {"level" : currentCpu})
              }
         }
    });

This is a very common type of scenario we will find in event based systems. We can describe this as

“When this event happens and this information is on the event, trigger a new event to a different stream.”

Very often however its not just one event that will cause something to trigger. This is why the state variable exists. Very often we want to handle a question that is more akin to:

“When this event happens, then this event happens, then this event happens trigger an event to a different stream”.

Let’s try to change our problem from Projections 2 into one like this. I am only interested in highcpu scenarios where the cpu is over 40% for more that 3 samplings in a row. A single one could just be a fluke that happened. In order to do this type of query we will have to use our state variable to tie together multiple function calls.

fromStream('$stats-127.0.0.1:2113').
    when({
        "$stats-collected" : function(s,e) {
              var currentCpu = e.body["sys-cpu"];
              if(currentCpu > 40) {
                   if(!s.count) s.count = 0;
                   s.count += 1;
                   if(s.count >= 3)
                        emit("heavycpu", "heavyCpuFound", {"level" : currentCpu,
                                                           "count" : s.count});
              }
              else
                   s.count = 0;
         }
    });

Note: if you are trying this at home you may want to change how often statistics are sampled. You can set this with –stats-period-sec=SECONDS.

Now we use our state that gets passed from call to call to correlate multiple events together. If we get three or more samples with a CPU usage greater than 40% in a row then we will produce a message to the heavycpu stream that looks like:

{
  "eventStreamId": "heavycpu",
  "eventNumber": 3,
  "eventType": "heavyCpuFound",
  "data": {
    "level": 41.896265,
    "count": 6
  },
  "metadata": {
    "streams": {
      "$stats-127.0.0.1:2113": 8
    }
  }
}

This is a very powerful paradigm as the state variable allows me to bring state from one call to the next allowing me to correlate multiple events together. Another example of this might be I am looking for users in twitter that said the word “coffee” and “happy” within 5 minutes of mentioning “starbucks”. This query would be implemented in the same one as the one we just tried.

As food for thought. Could I now write another projection off of “heavycpu” that then looked for items with 5 measurements>80 and counts >10? You wouldn’t probably do this in practice as you could put that logic in the first projection but you can compose projections as well!

In our next post we will look at having multiple types of events.

I’ve been working with Windows 8 and building Windows Store Apps more and more. Building native Windows Apps with HTML/JS/CSS is, in a word, awesome. I’ve mentioned before how the OS is a first class citizen in the development space. Having standard API’s in the dev environment to handle common tasks like searching and sharing is another nice feature in Windows 8 Development. I wanted to take a quick moment to talk about the sharing contract. You can download a full working example of how to implement the sharing contract here.

This example does a great job of dealing with situations where you can and want to share. But what about times when sharing is not appropriate? A good user experience means the user not having to guess as to whether encountered functionality is intended or a bug. By default, when the share contract is not implemented you get a pretty blunt message back: “This app can’t share.”

Looks pretty, but it’s not all that informative.  Fortunately, it’s very simple to provide your users with enough information so they know what they see is what you, as the app developer intended. The following code is a basic and typical share contract data request function:

  function dataRequested(e) {
        var request = e.request;

        // Title is required
        var dataPackageTitle = document.querySelector("article .item-title").textContent
        if ((typeof dataPackageTitle === "string") && (dataPackageTitle !== "")) {
            var dataPackageText = document.querySelector("article .item-content").innerHTML
            if ((typeof dataPackageText === "string") && (dataPackageText !== "")) {
                request.data.properties.title = dataPackageTitle;
                request.data.setText(dataPackageText);
            } else {
                request.failWithDisplayText("A problem occurred with this sharing request");
            }
        } else {
            request.failWithDisplayText("A problem occurred with this sharing request");
        }
    }

This still doesn’t do everything we need. For example, using the Grid Template as an example, this code would only work on the item detail page. If you happen to invoke the Share Contract from the Group Items or Group Details page, you will either get an error or the necessary data won’t be found. In most cases, it’s an individual item you are looking to share.  With that in mind, consider adding code like this to your common share data request function:

if (WinJS.Navigation.history.current.location != "/pages/itemDetail/itemDetail.html") {
   request.failWithDisplayText("Please note, you must navigate to an individual item in order to share.");
   return;
}

Now, when the user wants to share something in a context that does not account for that functionality, the user will know two things:
1. It’s not intended to be used in this context and;
2. The user knows where the sharing context is

This yields much better results:

To start with there is a very special stream inside of the event store. This stream represents statistics measurements that are happening internally. You can control how often they are taken via config. To find this stream in your system you can assuming you are bringing up a brand new node look at the “streams” tab when going to whatever port you set for http.

Hint: as projections are experimental as of the last release you need to enable them on the command line or in configuration when bringing up the Event Store. The command line is –run-projections.

For me (the default) stream for statistics is $stats-127.0.0.1:2113. If you want to see statistics data you can point your browser to 127.0.0.1:2113/streams/$stats-127.0.0.1:2113 and view the data in the stream. You should see something that looks like this:

If you click on one of the events you should be able to see the actual data from a statistics event entry. If you want to save some time you can see it on my gist. This is a json encoding of what the statistics measurement looks like. We are going to write a basic projection against that stream.

fromStream('$stats-127.0.0.1:2113').
    when({
        "$stats-collected" : function(s,e) {
              var currentCpu = e.body["sys-cpu"];
              if(currentCpu > 40) {
                   emit("heavycpu", "heavyCpuFound", {"level" : currentCpu})
              }
         }
    });

If you want to test this projection. Go to new projection and paste it in. Give it a name and select “emit enabled” and for mode put “continuous”. We will discuss in a later post what these things mean. The UI around this is currently being changed as well, we see its not the most intuitive.

This is a very simple projection. Its not very interesting. We will get to doing more interesting ones shortly. What it does is it listens to your statistics stream. This is setup when it says “fromStream” this is says “listen to all events in stream s”. It then defines a function that will be passed all $stats-collected events which happen to be the ones we saw in the statistics stream above.

The function declared checks the “sys-cpu” field of the event. If the cpu is higher than 40% it emits a new event out to another stream called “heavycpu”. If you are running the projection you can bring up your CPU usage then try navigating to the stream 127.0.0.1:2113/streams/heavycpu. You will see an event there of the form.

EventStreamId: heavycpu, EventNumber: 2, EventType: heavyCpuFound, Data: {
  "level": 40.9823952
}, Metadata: {
  "streams": {
    "$stats-127.0.0.1:2113": 49
  }
}

This is a very basic projection that is emitting a new event based on some criteria that is found. This is a very common pattern in event based systems (SEP). In the next post we will introduce state into our projection to look at how we can alert not just off a single event but off some group of events that are correlated which is another very common pattern in projections (CEP).

Over at http://geteventstore we have 1.0′ed the Event Store as an Event Store (eg storing/retreiving events/multinode version/etc) but in the process we did not release projections. They are still marked as experimental. Projections are however getting close to ready so its time to write a bit about them as the official documentation takes shape.

Through the series we will get fairly deep into their usage but for many the best way to understand them to understand the underlying theory so let’s start there.

Functions

When we talk about a projection off of an event stream basically what we are describing is running a series of functions over the stream. We could as our simplest possible projection have a projection that runs through all of the events in the system (in order) passing current state from one function to the next. The simplest of these could be to count how many events there are.

var count = function(state, event) {
return state + 1;
}

Then to run it across all of the events (let’s imagine we had three) in our system we would end up with a pattern like:

var result = count(count(count(count, 0), event1), event2), event3)

This operation is known in the functional world as a higher order function -> left-fold. It is a very powerful construct that is useful in solving many problems. When we talk about Event Sourcing, current state is a left-fold of previous beahviours.

Projections at their heart allow for the specializing of a generalized function. Their underlying model is that of a left-fold. Looking at the above left fold there are a couple of pieces that we could possibly change. The generic version of this function would look something like:

var result = transform(f(f(f(initial()), e), e), e)

Let’s discuss briefly what the three main parts of this function are.

f(state, event) => state – is the function that is run over the series of events.

transform(state) => result – is a function that can transform the state to the form of result you want to receive

initial() => state – returns the initial state you want passed into your left-fold.

Taking one example (don’t worry we will go through more in later posts!): f(state, event) => state – gets specialized through a pattern match:

var f1 = function(state,event) {}

when({
$any : f //runs for all
})

or

var f1 = function(state,event) {}
var f2 = function(state,event) {}

when({
Something : f1, //match all Something
SomethingElse f2: //match all SomethingElse
})

To see the differences between the two, let’s imagine that we had a stream of events.

Something
SomethingElse
Something

The first would call as:

f1(f1(f1(nil, Something), SomethingElse),Something)

The second would call as

f1(f2(f1(nil, Something), SomethingElse),Something)

If we were to change the stream to:

Something
SomethingElse
SomethingElse

The second would end up as

f2(f2(f1(nil, Something), SomethingElse),SomethingElse)

Again don’t worry too much about the details. We will have a whole post on “when” and pattern matching

Event Selection

The second part of how projections work theoretically is the controlling of which events get passed to which definition. In our first example we passed all events in the system through a single left-fold. While many projections due in fact work exactly this way many are only interested in certain events not all events in the entire event store.

fromAll() would read all of the events in the entire event store. fromStream(‘stream’) tells projections to take whatever definition you have and run it against only the events within the stream ‘stream’. Where as fromStreams(‘s1′, ‘s2′, ‘s3′) joins the events from three streams and then runs the left-fold of the resulting stream.

Don’t worry if you don’t completely “get” projections after this post, there is a lot to them and we will be delving into a lot more detail and use cases of them. Overall however projections have a relatively simple underlying theory associated with them. Hopefully by understanding the underlying theory it will make the rest of the posts in the series easier to understand.