The first thing to note is that database servers are complex beasts (don’t underestimate them) and each and every database provider has its own strengths but also its own caveats.

Sooner or later you’ll face a behavior that may seem counter-intuitive and there’s when you must read a lot of documentation. In this case, MySQL Performance Blog and MySQL documentation have been of great help, as well as the now unmantained-but-still-useful Hack MySQL site, which provides great tips very well explained.

Second. As always, doing an 80% optimization consumes 20% of the time and most of it is quite straightforward. EXPLAIN, review results, create index, EXPLAIN again. This way you can do a good job just knowing some indexing basics. Your indexes may be a bit fat, but will do the job.

Then you’ll need to be patient, creating indexes is not always fast, in fact the number, size and heterogeneusy of the columns involved will increase the time it takes to create them. While you wait, the best is to read good books about MySQL or relational databases like: Relational Database Index Design and the Optimizers.

You’ll need to be organized. Creating indexes randomly just looking a couple of slow SELECTs is something, but you must look at the big picture: make an inventory of all the application queries, no matter how unimportant look like because if they attack a big table with a WHERE or an ORDER BY they will perform bad. Fact. This list will help you organize the indexes and see which ones are used and which not and where.

Learn as well what the configuration parameters mean, MySQL has a bunch of them. Just increasing numbers will not always solve your bottlenecks. If you see something strange, there might be a parameter misconfigured and causing bad results.

Sometimes you’ll find yourself frustrated by something that you can’t understand. I found that almost anything in this field has an explanation, even a reasonable one.

Finally, don’t overlook other thinks. Database performance is important but there may be a lot of room to improve in your code, don’t blame MySQL as the root of all problems!

Creating Glimpse plugins is really easy (just implement IGlimpsePlugin and decorate it with GlimpsePluginAttribute), implement the methods and reference the plug-in assembly in the project where you are using Glimpse. MEF will do the gluing and the JSON.NET will do its best to deserialize the results in a human readable way.

To use this plug-in, just compile it and reference the assembly in your Glimpse-enabled project. I you have questions, suggestions or bug reports, you can use Github’s own issue reporting system, drop me a line o fork the project and make your improvements.

So, I hope you like it.

Link: miniprofiler-glimpse-plugin [github.org]

I trusted Time Machine, and actually it seemed a good program and a very convenient way of making and managing backups but… it does not matter how pretty is if when you need to restore a backup it gives you strange errors.

The whole story is that I wanted to upgrade my Mac hard drive to a bigger one and reading Time Machine documentation, seems that I can use the OSX Installation disk to restore a backup to a new fresh drive. Well, that’s just theory but when doing as instructed I got an “Error -22″ after about an hour restoring… epic fail. Good that I did keep the original drive untouched.

Then I tried donationware Carbon Copy Cloner to create a bootable clone backup of my working hard drive. After a couple of hours I had an external USB that mimicked my original drive. I replaced the hard drive, created a new partition and cloned again the external drive backup to the new one. Now I’m writing this on my upgraded Mac.

So, thanks to Carbon Copy Cloner from Bombich I was able to upgrade without problems. Great job!

First of all, my topology is the following: a client, in this case for testing purposes, my machine, which is running an ASP.NET web application and a NServiceBus host that processes all the messages. This has been configured following the official documentation and looking at the samples (check out the full NServiceBus source code, you’ll need it). It’s a bit messy but with some trial and error you can make sense of it. There are some obscure properties that have to be googled to know what they mean.

Processing the messages is not trivial for the bus host. Some of them attack the database and need to use MSDTC to manage the transactions. This means that MSDTC must be up and running without problems.

One trick I learned after a lot of researching is the Runner.exe tool from NServiceBus. This comes with NServiceBus source code (and I think that also with paid version) and ends up in build\tools\MsmqUtils if you build it with build.bat or build-net4.bat. Just open a command line as administrator and run:

Runner.exe /i

This will install MSDTC and configure it properly. I don’t think you’ll need anything else but check the MSDTC configuration in Component Services -> Computers -> My Computer -> Distributed Transaction Coordinator -> Local DTC. Right click to go to properties and ensure everything is configured like in this Troubleshooting DTC post.

At this point, I recommend you to disable the Windows Firewall on both machines, at least for domain network, just to be sure it’s not interfering and breaking the tests, we will solve this later.

Then, DTCPing is your friend. This small tool will check the DTC connectivity between two machines. It’s a bit awkward. You have to start the tool in both machines, then put the name of the other and click ping on each one. This will make a test and if everything goes well, you’ll see a detailed log without errors:

++++++++++++Validating Remote Computer Name++++++++++++
Please refer to following log file for details:
	C:\tools\DTCPing\LOCALMACHINE2928.log
Invoking RPC method on REMOTEMACHINE
RPC test is successful
++++++++++++RPC test completed+++++++++++++++
++++++++++++Start DTC Binding Test +++++++++++++
Trying Bind to REMOTEMACHINE
Received reverse bind call from REMOTEMACHINE
Binding success: LOCALMACHINE-->REMOTEMACHINE
++++++++++++DTC Binding Test END+++++++++++++
Please send following LOG to Microsoft for analysis:
	Partner LOG: REMOTEMACHINE5928.log
	My LOG: LOCALMACHINE2928.log
++++++++++++Start Reverse Bind Test+++++++++++++
Received Bind call from REMOTEMACHINE
Trying Reverse Bind to REMOTEMACHINE
Reverse Binding success: LOCALMACHINE-->REMOTEMACHINE
++++++++++++Reverse Bind Test ENDED++++++++++

Here started my nightmare, I got the damned: RPC error 5 (Access denied). After hours googling for the problem I found this post about MSDTC problems (I was looking for RPC problems, which actually was the root cause). The cause is that remote RPC is disabled in Windows workstations by default.

I opened for the N-th time the registry and added the RPC key to HKLM\Software\Policies\Microsoft\Windows NT and the RestrictRemoteClients DWORD value to 0. Rebooted without much hope. Magically, DTCPing worked and life seemed better and brighter.

Then, the Windows Firewall must be configured. In Windows Server 2008 (or Windows 7, it’s the same interface), go to Control Panel -> System and Security -> Windows Firewall (or just execute firewall.cpl at the command prompt). Go to Advanced Configuration to open a new window. Go to inbound rules and look for the Distributed Transaction Coordinator. If they are disabled, enable them. On the workstation the firewall can be enabled. Beware that DTCPing may not work, because the Firewall is only allowing connections to/from svchost.exe and msdtc.exe (if you want it to work, add a rule for that process).

Looking around I found fail2ban which is a tool that does som regex matches on the server logs (sshd, httpd, authd, …) and takes proper actions, like banning the offending IP.

I then installed fail2ban in my Debian box:

> apt-get install fail2ban

Then, I took a look at /etc/fail2ban/jail.conf and found some entries for Apache but none for nginx, my server of choice, so I decided to create a jail.local to add some nginx stuff (this is recommended in Debian to allow hassle-free upgrades).

I started copying the Apache sections of the default fail2ban as the log files in my case use the same format that allows me to use Awstats easily. Then, I modified my log routes to point to the nginx ones and using Apache rules, if they don’t work I’ll tune them later.

[nginx]

enabled = true
port    = http,https
filter  = apache-auth
logpath = /var/log/nginx*/*error.log
maxretry = 6

[nginx-noscript]

enabled = false
port    = http,https
filter  = apache-noscript
logpath = /var/log/nginx*/*error.log
maxretry = 6

[nginx-overflows]

enabled = false
port    = http,https
filter  = apache-overflows
logpath = /var/log/nginx*/*error.log
maxretry = 2

Although this is ok, the bots I see don’t leave a trace in error.log but in access.log so I took a look at the filter.d folder where an interesting apache-badbots.conf was present. Then, I found the default fail2ban documentation in /usr/share/doc/fail2ban where there’s an usage example of the badbots script. I added I to my jail.local:

[apache-badbots]

enabled  = true
port    = http,http
filter   = apache-badbots
logpath  = /var/log/nginx*/*access.log
bantime  = 172800
maxretry = 1

Finally, I added this to the top of the file, to send mails to myself when a rule matches and a host is banned.

[DEFAULT]

action = %(action_mwl)s

Finally, restart the service and start receiving mails:

> sudo /etc/init.d/fail2ban restart

I’m sure this will need further adjustments, but it’s a beginning in my bot fighting war. I’ll make some updates as I find interesting results.

I tried to repair Visual Studio 2010 and told me that it had troubles with vs_setup.msi… strange. Looking around I found a post that led me to an explanation of the msiinv.exe tool.

It took me a while to get the tool as the author’s site is broken, but Stackoverflow came on rescue.

Then, I ran the tool:

msiinv -p > msiinv.txt

Then opened the file and… surprise!

Microsoft Visual Studio 2010 Professional - ESN
    Product code:    {725041D1-9F45-30D3-A78E-DF08C4E1A297}
    Product state:    (1) The product is advertised, but not installed.
    Package code:    {9E58363D-E4A9-49D3-82B7-42584AFCA9E7}
    AssignmentType:    1
    Language:    3082
        Package:    vs_setup.msi
    0 patch packages.

Strange. I tried to fix it with MSI:

msiexec /f {725041D1-9F45-30D3-A78E-DF08C4E1A297}

It asked me for the vs_setup.msi, it’s in the root of the Visual Studio 2010 DVD in case it’s not found automatically. Magically it worked and now it’s installed:

Microsoft Visual Studio 2010 Professional - ESN
    Product code:    {725041D1-9F45-30D3-A78E-DF08C4E1A297}
    Product state:    (5) Installed.
    Package code:    {9E58363D-E4A9-49D3-82B7-42584AFCA9E7}
    Version:    10.0.30319
    AssignmentType:    1
    Publisher:    Microsoft Corporation
    Language:    3082
       Installed from: D:\VS2010\
    Package:    vs_setup.msi
    Help link:    http://go.microsoft.com/fwlink/?LinkId=133405
    Local package:    C:\Windows\Installer\3f3bc84.msi
    Install date:    2011\01\14
    0 patch packages.

Great! Now the ASP.NET MVC3 RTM setup package worked like a charm.

I don’t know exactly why but I can’t install the MVC 3 package provided by Web Platform Installer nor the standalone one, so I uncompressed it and tried to figure out how to install it manually.

The problem I have is with vs10-kb2465236-x86.exe that gives me a “failed with 0x8007066a” error. It seems to be for enabling Razor syntax but the prerequisites are Visual Studio 2010 Ultimate ENU. I have Visual Studio 2010 Professional ESN and that may be the reason or may not.

Anyway, I found the way to install the package manually, although the Razor Intellisense stuff is not available in my setup.

First you have to uncompress the AspNetMVC3Setup.exe into a folder and then install the following files (I got the sequence from the parameterinfo.xml file, which describes the installation process):

• VS10-KB2465236-x86.exe (it fails in my case, I don’t install it)
• AspNetWebPages.msi – ASP.NET Web Pages
• AspNetWebPagesVS2010Tools.msi – ASP.NET Web Pages Visual Studio 2010 Tools
• AspNetMVC3.msi – ASP.NET MVC 3 Runtime
• AspNetMVC3VS2010Tools.msi – ASP.NET MVC 3 Visual Studio 2010 Tools
• NuGet.msi – NuGet

And that’s all. In case you have Web Developer Express instead of full Visual Studio, you have ton install the tools specifically for that edition, AspNetWebPagesVWD2010Tools.msi instead of AspNetWebPagesVS2010Tools.msi and AspNetMVC3VWD2010Tools.msi instead of AspNetMVC3VS2010Tools.msi.

Good luck!

Recently I tried (again, first time I found some trouble) to create an MsBuild script to make a web publication to a folder an then zip that folder in order to make a simple distributable with all the things needed to set up our web project that anyone in the organization could download and install, instead of having to create that zip myself or publishing through Visual Studio.

Finally, I got the script and seems to work using MsBuildCommunityTasks and some community knowledge.

The first part deals with publishing the site:

  <PropertyGroup>
    <OutputFolder>$(MSBuildProjectDirectory)\publish</OutputFolder>
    <TempFolder>$(MSBuildProjectDirectory)\temp\</TempFolder>
  </PropertyGroup>
  <Target Name="PublishSite">
    <Message Text="Publishing in folder: $(OutputFolder)" />
    <RemoveDir Directories="$(OutputFolder);$(TempFolder)" ContinueOnError="true" />
    <MSBuild
      Projects="src\Adapting.Web\Adapting.Web.csproj"
      Targets="ResolveReferences;_WPPCopyWebApplication"
      Properties="Configuration=Release;WebProjectOutputDir=$(OutputFolder);OutDir=$(TempFolder)"
      StopOnFirstFailure="true" />
    <RemoveDir Directories="$(TempFolder)" ContinueOnError="true" />
  </Target>

I define some variables and then I call the _WPPCopyWebApplication which is the important target (in .NET 2.0 was _CopyWebApplication). This should be the same as publishing a web site to a folder using the wizard.

Then, I get the files in the OutputFolder and make a zip with them:

  <Target Name="PackageSite" DependsOnTargets="PublishSite">
    <ItemGroup>
      <PublishFiles Include="$(OutputFolder)\**" />
    </ItemGroup>
    <Message Text="Packaging Site" />
    <Zip
      Files="@(PublishFiles)"
      ZipFileName="release-web.zip"
      WorkingDirectory="$(OutputFolder)"
      />
  </Target>

I simplified this a little bit because I have as well a target that generates the build number from SVN and changes the AssemblyInfo.cs and uses this numbers to name the zip.

In TeamCity, you can create a new configuration with *.zip as artifacts, that points to your SCM (Subversion in my case) and the most important thing is that uses the MsBuild runner targeting PackageSite.

The only thing that was that if I put the <ItemGroup> section out of the <Target>, works on the command line but not inside TeamCity.

That’s all, I hope this helps.

Sources:

• How to publish a web site with MsBuild
• Using MsBuild to create a deployment zip

The thing is that the code does like this:

public IncludeHandlingSectionHandler()
{
    _types = new Dictionary<IncludeType, IIncludeTypeSettings>
    {
        { IncludeType.Css, Css },
        { IncludeType.Js, Js }
    };
}

Where this is the constructor and .Css and .Js properties are marked with [ConfigurationProperty], so in theory seems correct.

In the tests, there’s an XML file with the section and it’s loaded using the DeserializeSection method of the ConfigurationSection base class. And this seems correct, everything goes ok and the dictionary is populated correctly with the data from the XML file.

But it’s wrong. If you add the section to the app.config (or web.config) file and instead of using DeserializeSection you use ConfigurationManager.GetSection(sectionName), the constructor is called before populating the Css and Js properties so the dictionary is not created properly but as those properties return a default implementation in case there’s no configuration, it seems to work, but the configured values are never loaded.

This way, tests pass but it’s impossible to configure the library via web.config. Anyway, in my opinion the problem is in .NET Framework because internally, GetSection should use DeserializeSection to be consistent.

Finally I found in the excellent pinvoke.net site a very nice and simple example using a simple call to iphlpapi.dll’s SendARP method. The trick was to use ARP to get the MAC from the IP.

Note that this has it’s limitations, as I will only work if the remote machine is in the same network and the packet does not need to go through a router. I was enough for our needs.

Here’s the code I adapted from the example:

public static string GetMAC(IPAddress ipAddr)
{
var mac = new byte[6];
var len = (uint)mac.Length;

var result = SendARP((int)ipAddr.Address, 0, mac, ref len);

if (result != 0)
{
return "MAC not found!";
}

var str = new string[(int)len];

for (var i = 0; i < len; i++)
{
str[i] = mac[i].ToString("x2");
}

return string.Join(":", str);
}

[DllImport("iphlpapi.dll", ExactSpelling = true)]
private static extern int SendARP(int destIP, int srcIP, byte[] pMacAddr, ref uint phyAddrLen);

Enjoy!