CodeBlog.TheG2.NET

Samsung Epic 4G CyanogenMod Notes and Links

noreply@blogger.com (Greg Bray) — Wed, 13 Feb 2013 05:39:00 +0000

Up until a few months ago I very much enjoyed having the Samsung Epic 4G as my personal phone. It is the best Android phone with full QWERTY keyboard that I have ever used, but the hardware is a bit dated and the WiMax speeds were not as fast as LTE, so I decided to upgrade to the Galaxy Note II on Verizon. Samsung stopped updating the software on the Epic 4G a while ago, but the CyanogenMod community has done a great job at supporting the phone and even has Android Jelly Bean 4.1 ROMs with full functionality. I still have friends with the Epic 4G so I thought I would post all the links I used for loading custom ROMs.

Most recent versions (As of January 2013)

Android 4.1.2 using CyanogenMod 10-20130107 nightly build
Rooted with Clockworkmod version 6.0.2.7
Video guide for Rooting Epic 4G (Kudos to QBKing77 for his great demos)
Video guide for Installing CyanogenMod 10 ROMs
EpicCM blog (CyanogenMod team working on Epic4G)
Epic 4G Nightly CM Builds

Sprint has Steps to activate phone, which includes instructions for activating a used phone. These may not work in CM 10, but I usually put a temporary boot ROM on the SD card that will boot into the stock FC09 ROM

Temporary Stock Boot Instructions:

Boot into CWM5 on your Epic 4G
Install zip from sdcard > choose zip from sdcard > multiboot > FC09 > boot_FC09.zip
WAIT. It will take a while to boot. You might see a "com.android.phone is not responding", do not worry, it is harmless. Wait until Media Scanning is complete before doing anything.
Follow instructions for activating phone or calibrating tilt sensor

You can also revert the Epic 4G to original locked boot loader and stock ROM if you have any issues with the custom ROMs or have to take it in for service/support.

Hope this helps any Epic 4G users out there. If Samsung ever decides to make another Galaxy S phone with a keyboard definitely will consider buying it, but for now I am enjoying the Galaxy Note II with flygrip.

Looking for a .NET programmer in Salt Lake City, Utah? You should hire me!

noreply@blogger.com (Greg Bray) — Wed, 25 May 2011 07:32:00 +0000

UPDATE 6/01/2011: I have accepted a great position at GE healthcare that interestingly has very little to do with .NET and everything to do with making better software.

UPDATE 5/15/2013: Although I am still working for GE (Now part of a joint venture with Microsoft) I am also actively looking for new challenges in Software Engineering or Developer Operations.

My 30 second over-hyped elevator pitch:
Most people don’t have the time to read through a full resume nowadays, so I’ll keep things short and to the point. You should hire me! Why? Because I am smart and I get things done. The last time I was in the job market I turned a 3 week software testing project into a 6 year pursuit helping to convert a small ISV into a cloud driven heavyweight. I don’t always code in .NET, but it is definitely my platform of choice. Life is too short to worry about pointers and memory leaks. If I wanted to do that I’d go back to working in assembly language on an 8086 like God intended. I can automate just about anything using a combination of BASH, PowerShell, AutoIT, VBA, WatiN, MSAA, a bunch of HWNDs, a packet sniffer, a few servos, and a roll of duct tape. I work well as part of the A-Team or sent out solo in the wild like Rambo. In short: I am the most hirable man in the world! :-P Keep coding my friends!

I live my life like an open book, indexed by Google. I have a thirst for knowledge and a passion for technology. I pay great attention to detail and strive for nothing less than perfection. A lot of my past projects are internal or backend systems, but below is a sample of what has kept me busy over the last decade or so.

Personal Coding projects:
PhraseMeme Scanner, WinFone Alpha (Windows Phone, 2010-2011)
Silverlight port of ZXing Barcode Scanning library (Silverlight, C#, 2010-2011)
Texas Holdem Monte Carlo Simulator (Silverlight, Windows Mobile, 2009)
Resolver one: Texas Holdem, Pivot Charts/Auto filter, Web page automation, WPF Charts, Mad Lib generator (IronPython spreadsheet, 2009)
GnuMap P2P Mapping project (Visual Basic, Web Spider, aiSee, 2002)

Samples of documents I have created:
High level overview graph (Visio 2007)
Detailed technical overview chart (Visio 2009)
In depth technical documentation (Word 2008)
Short section of web based training material (PowerPoint 2010)
Samples of feedback reports w/ synopsis (Acrobat 2006)
Memo to customers re: Software Upgrade (Word 2010)
Custom Reporting Tools using SOAP web service (C#, ClickOnce, Access, Excel 2009)
Raw Documentation Files (2006 – 2011, I created the majority of the documents on here)

Screencasts and videos of project I have created:
PhraseMeme scanner demo video (3 mins, Camtasia, PowerPoint 2010)
WinFone Alpha demo video (3 mins, Camtasia, PowerPoint 2010)
Texas Holdem Monte Carlo Simulator (5 mins, Camtasia, PowerPoint 2009)

Coding Competitions:
PhraseMeme Scanner received good rankings in Windows Phone 7 Competition sponsored by Red Gate (2010)
Texas Holdem app received honorable mention in 2009 INETA Code Challenge and 2009 Telerik Silverlight contest
Monte Carlo spreadsheet was the winner of $4,000 prize in April 2009 round of Resolver One Challenge

InvalidOperationException from IsolatedStorageSettings.Save on Windows Phone 7

noreply@blogger.com (Greg Bray) — Tue, 18 Jan 2011 03:29:00 +0000

I just had my first failed test from the marketplace certification for PhraseMeme Scanner v1.2. You can view the full error report here, but the important part is:

Comments: The application terminates unexpectedly when the user turns on/off one of the setting while trying to scroll to

another page.

Steps to reproduce:

1. Launch the application.

2. Scroll over to the setting menu.

3. At the same time, press and scroll on any of the settings.

4. Notice the application terminates unexpectedly when the user turns on/off one of the setting while trying to scroll to

another page.

The settings page uses ToggleSwitches from the Silverlight for Windows Phone Toolkit with a simple two-way binding to a values stored in an IsolatedStorageSettings class, so I was a bit confused where the error was coming from. Using Visual Studio I found that the IsolatedStorageSettings.Save method was throwing an InvalidOperationException whenever the user changed a setting and navigated way from the page at the same time.

The MSDN documentation doesn’t mention that the method can throw this error, but my guess is that either the value is in an indeterminate state or there is a conflict between the save method and the two-way binding. The error could have been there all along, but in this version I had moved all the IO calls to background threads to increase performance where as previously the exception was captured by a try/catch block on the calling method.

Luckily it was an easy fix: I added a try/catch statement to the save code block and inserted a delay to let whatever processing it was conflicting with finish before it tried to save the settings. Below is the new code that can be used for saving settings stored in an IsolatedStorageSettings class:

/// <summary>

/// Save settings to isolated storage. Will run using background thread to prevent blocking UI thread.

/// </summary>

public static void Save()

{

ThreadPool.QueueUserWorkItem(func => {//Save items using background thread.

try

{

//NOTE: may throw InvalidOperationException exception if user is in process of changing settings and navigating away from page

System.Threading.Thread.Sleep(200);

issSettings.Save();

}

catch (Exception ex)

{

Debug.WriteLine("Error saving settings: " + ex);

}

});

}

Hope that helps someone prevent a failed submission to the marketplace.

Enable Blogger Mobile Templates Support for WP7 and Opera Mobile

noreply@blogger.com (Greg Bray) — Mon, 20 Dec 2010 23:23:00 +0000

Blogger recently released a new feature to their Blogger in draft platform that allows the website to automatically detect mobile browsers and render the page using a small-screen friendly template. This fixes a HUGE issue I have been working on for the last few weeks ever since I released the PhraseMeme Scanner application, namely publishing content that will be consumed by both large and small screen devices. Previously I was planning on having two version of the website with one being optimized for mobile and one for desktop, but now I can store all the content on Blogger and use the mobile templates to render for small-screen devices.

The Show mobile template currently works with the default iPhone and Android browsers, but it does not work with the Opera Mobile, Pocket IE, or Windows Phone 7 Internet Explorer browsers. I assume there are other mobile browsers that also don’t work, but there is a simple way to add support for any browser by using JavaScript to redirect users. Here is the code that I wrote to detect mobile browsers and redirect accordingly:

The templates work by appending a m=1 parameter to the blog URL or (m=0 to force full webpage on mobile devices), so all that the above code has to do is look for that parameter and add it if a mobile device is detected. You can add the above script to your blog using either the blog template or the HTML/Javascript Gadget. Adding it to the blog template will cause it to run before the page is fully loaded, but you should save your current template first to make sure you don’t mess anything up;

I added a feature request for Blogger to support more mobile browsers, but for now the above script should let you start using this amazing feature.

Windows Phone Barcode Library (Silverlight C# ZXing Port)

noreply@blogger.com (Greg Bray) — Thu, 09 Dec 2010 01:37:00 +0000

I have been working on a Barcode Scanning application for Windows Phone 7 for about 3 months now, but none of my efforts would have been possible without the prior work done by Google on the open source ZXing (“Zebra Crossing”) barcode scanning library. This library is released under the Apache License v2.0 and has spurred development of many different barcode scanners on many different platforms. The idea for PhraseMeme Scanner came when I saw there weren’t any good scanners available for Windows Phone 7 and then stumbled across the partial C# port of ZXing. Through a bit of testing I was able to modify the library to support Silverlight and Window Phone 7, and decided I would help push the library into a well maintained and supported state in the Silverlight world.

While the primary purpose for updating the library is to build the "Best barcode scanning app available" I also want to give back to the community and help others create great windows phone or Silverlight apps. ZXing has their own Google Code project, but that site is mainly devoted to Android or Java development, so I decided to store the code on Codeplex to get it some .Net love. This also means that I can use Mecurial instead of SVN/CVS, which so far has been a great experience and should be very helpful if anyone else decides to contribute patches or new features to the library.

So without further ado, please check out the Windows Phone 7 Silverlight ZXing Barcode Scanning Library based on the csharp ZXing port created by Suraj Supekar at revision 1202 in the SVN repository. Let me know what you think, and happy coding!

Update 3/11/2011: It looks like the project is getting a lot of interest from developers with over 6,000 page views and 750 total downloads in the last few months. This is probably due to the various tweets on twitter and a blog post from Rudi Grobler about reading barcodes on WP7 using the Silverlight ZXing port. That blog post was mentioned on This Week in Channel 9, which totally made my day even if it was only talked about for 40 seconds and never even mentioned my name. Still… pretty wicked cool

VPS/VM vs Dedicated vs Cloud Servers: Hosting options and cost comparisons

noreply@blogger.com (Greg Bray) — Tue, 05 Oct 2010 04:58:00 +0000

I usually talk about Code on this blog, but eventually that code is going to have to run somewhere so I thought I would outline some of the different server options that are available. While there are free or low cost options for hosting simple or small websites, if you are using ASP.NET, PHP, or Java to create a dynamic website or if you are looking for a file or application server then you are going to have to pay for the resources that you use (bandwidth, hardware, software, etc). The price/performance ratio can vary greatly depending on what type of server you purchase. There are a lot of different configurations, but here are some of my recommendations.

First, what not to do:

While in certain cases it may be appropriate, my recommendation is to steer clear of two specific hosting options: Shared web hosting and Do-It-Yourself hosting. Shared hosting refers to the $5 a month plans offered by Godaddy, Yahoo, and thousands of “resellers” that use a control panel like Plesk or cPanel to divide up a single server into hundreds of small websites. These servers are often over-sold and under-powered, and you will only get FTP and control panel access, so you will have to rely on your hosting provider to make any advanced configuration changes or perform troubleshooting steps for you. If you cannot afford any of the other options then Shared Hosting might be a good way to start, but keep in mind that you get what you pay for and will probably out-grow it very quickly.

As for Do-It-Yourself hosting, this is simply a case of economies of scale: You wouldn’t start a wheat farm just to make a sandwich, so why build and maintain your own servers? Unless you are a corporation with a full IT department, it is not in your best interest to setup and maintain your own web or application servers outside of your personal testing/development environment. Servers are more or less a commodity nowadays, so while you could setup a computer in a closet of your office and use it to host your company website, you then have to worry about what happens when the power goes out or when your Internet connection goes down. Setting up and maintaining a reliable network with redundant paths and uninterrupted power takes a considerable amount of time and money. I highly suggest that you outsource this effort to a 3rd party and focus your energy on improving your website or application instead. And in terms of running your own Email and Calendaring server, you really, REALLY should look at paying Google, Microsoft, or someone else to provide these services for you if at all possible.

Virtual Private Server / Virtual Machine:

It use to be that when looking for a hosting provider you were either stuck with shared hosting or a dedicated server. Now there are many other options, most of which are based on server virtualization. While it may sound similar to Shared Hosting, VPS or VM hosting options can provide guaranteed resources and an isolated execution environment in which you have full control of the server. The hosting provider will purchase high-end server hardware and then use a hypervisor to divide that server up into multiple virtual machines for different customers (usually 2-24 depending on the size of the VMs). Each VM gets a certain percentage of the CPU, Memory, and Disk Space and will function as its own server. From my experience VPS or VM hosting provides the best bang for your buck at around $20-$120 a month (see Choosing a Hosting Provider below).

Dedicated Servers:

Some applications require significant horsepower, in which case a dedicated server is usually a requirement. You can purchase standardized “cookie-cutter” servers from most hosting providers, or you can also get a custom quote if you need to have a specific type of RAID setup or maximize RAM size or CPU speeds. Until recently I have primarily worked with dedicated servers that usually run anywhere from $100 to $1000 a month, so they are only viable in business environments that can justify the cost of the server. You also may have the option to rent, lease or purchase the server, but if you are purchasing you must factor in the time and costs for replacing failed hardware components in the future. Also if you work with sensitive information such as credit cards or medical records then you may require a dedicated server to meet Payment Card Industry (PCI), Health Insurance Portability and Accountability Act (HIPAA) or various other federal regulations.

Cloud IaaS and PaaS Servers:

Cloud hosting is the new hot term, but a lot of the time it means the same thing as VPS or VM Hosting except you pay by the hour instead of by the month. There are typically 3 types of “Cloud Computing”: IaaS, PaaS, and SaaS. For the most part Infrastructure as a Service (IaaS) is exactly the same as VPS or VM hosting just with a fancy API and pay-by-the-hour terms. Software as a Service (SaaS) doesn’t really have servers in the sense that you usually are riding on top of another application (Ex: Google Apps or Salesforce). Platform as a Service (PaaS) still uses Virtual Machines, but the platform is designed to help you scale your application horizontally across multiple nodes. Amazon Web Services (AWS) has some PaaS products, but their EC2 product fits more into the IaaS category (you have to scale-out your code to work on multiple servers on your own). Windows Azure, Rackspace Cloud Sites and Google App Engine are examples of PaaS, but they each have different programing models so you have to build your application around their platform. PaaS is designed to scale to meet your traffic needs and use multiple server nodes to provide better reliability, but this also means you will play a higher cost if you are running a 24x7x365 application. For instance, running two small web roles in Windows Azure to create a high availability website would cost $0.12*2*24*30 = $172.80 a month.

Choosing a Hosting Provider:

So now that you have an idea of the different types of hosting, all you have to do is find a hosting provider that offers a solution that meets your needs. I have worked with ASP.NET websites of various sizes for the last 5-10 years and can highly recommend the following companies based on my experience with them:

Superb.net: Offers all types of hosting ranging from shared to dedicated to virtual machines and custom quotes. Their VPS product is very affordable starting at $35 per month for a Win 2003 server or $20 per month for a Linux server. They also have great rates on dedicated servers starting at around $70-$80 a month. I had a server with them for about 10 years and they have always had great support and great prices.
AccuWebHosting.com: Recently I moved my Windows server over to a VPS hosted at AccuWebHosting. They are not as large as Superb.net, but they offer some of the the best rates on VPS servers starting at $23 per month for Windows or $16.50 per month for Linux. While not as powerful as a dedicated server the VPS servers work great for light web or application servers.
Rackspace.com: Offers fully managed dedicated servers and colocation services with “Fanatical Support”. They are a premium provider and cost a bit more than their competitors (starting around $500-$600 a month for a mid-range server), but if your application is mission critical you can feel comfortable turning over the keys to them and they will make sure your servers remain online and operational. Also Rackspace is the only vendor I have ever been able to get to sign a HIPAA compliant Business Associate Agreement (BAA).
Rackspacecloud.com: Originally called Mosso it was rebranded in July 2009 and offers Cloud CDN, Website, and Server solutions. They have managed servers starting at less than $100 a month, so if you want “Fanatical Support” without the high costs this might be a good place to start.
Amazon EC2: EC2 is the most mature of the online cloud providers and offers the most pricing options including Spot and Reserved instances. If you envision scaling out to hundreds of servers it is well worth your time to check out EC2. Their Simple Storage Service (S3) is also one of the best storage providers as it offers two price points depending on the level of durability that you require.
Windows Azure: Designed for high reliability and easier development, Azure is worth looking into if you are starting a new Microsoft .NET project that will require multiple servers. They are the new kid on the block, having been officially released in February 2010, but their use of Roles instead of servers means an application can be easily scaled up or down as needed. Also SQL Azure is a great way to create a hosted MS SQL database, as you pay $10 per GB per month, which combined with $0.15/GB/Month for blob storage means you should be able to store all your data and images/files for relatively cheap and support a large number of simultaneous users. Check out an introduction to Windows Azure for more information and watch the Cloud Cover show on Channel 9 for great coverage.

I hope this helps as you look at designing or scaling out your application or website. If you have questions I highly suggest asking them on one of the StackExchange websites such as ServerFault.com or Webmasters.StackExchange.com. There are usually lots of great people there willing to help answer questions.

Localization and Globalization in WP7 Applications Using Resource Files

noreply@blogger.com (Greg Bray) — Sun, 19 Sep 2010 23:51:00 +0000

I’ve been watching the progress of Windows Phone 7 for a while now and recently started developing a few applications that I hope to release to the marketplace when it opens later this year. One of the apps I have been working on uses .NET Resource files to allow label and button text to be translated into other languages. While the idea is simple the implementation was a bit more complicated than what I was expecting, so I thought I would post some details here to help guide others.

First off, if you are not familiar with .NET Resource files, I suggest watching this video on Globalization and Localization from the Windows Phone 7 in 7 training series, which gives a great overview of what each of the terms mean. In my case I was most interested in localization using separate resource files for each language, which was not covered in the video but is covered in the MSDN Windows Phone documentation linked to at the end of the video. This documentation was very helpful for setting up the resource files and creating a LocalizedStrings class, but after that they only provided one method of setting up the data binding for XAML objects. They also did not elaborate on some of the other options you can use when setting up localization resource files.

After a few tries and a bit of Google-fu I was able to get things setup how I wanted by adding these additional steps:

I created all the Resource files and the LocalizedStrings class in a Localization folder to keep things organized. It use to be that the .RESX files had to be in a special Resource folder in Visual Studio, but with 2010 they can be stored anywhere you want. If you have a large application you would probably want to have multiple folders with their own set of resource files (ex: one for each UI page or section of your application). In this case you would still have a single LocalizedStrings class, but inside that class there would be public properties for multiple resource files (ex: MainPageResources, SettingsPageResources, etc…) instead of just one Localizedresources property.
The above articles specifically reference the en-US and es-ES cultures, but .NET also supports using just en or es for culture-invariant language specification, which is usually better if all you need is localization without currency or date/time format conversions. You can in fact create a general fr file that includes French translations and use that same file to support both France (fr-FR) and Canada (fr-CA) cultural variants. See MSDN for more information about the Hierarchical Organization of Resource Files and see this list for all of the specific languages and cultures supported by Silverlight.
The MSDN article shows you how to setup the files and create the LocalizedStrings class, but they then assume that you know how to use that class for data binding. Visual Studio 2010 and Silverlight handle data binding differently than Winforms, and it gets even more confusing since XAML also has it’s own definition of Resources that are different then the .NET resources we just created. Silverlight also uses the term Resource to refer to files that use the the Build Action of "Content”, as these files get wrapped up into the .XAP file similar to how files with Build Action of "Resource” get embedded into the .Dll assembly (ex: loading an image from content or resource files). I found that instead of using the Text="{Binding Path=resourceFile.resourceName, Source={StaticResource Localizedresources }}" XAML syntax it was easier to use the following steps:

Open your primary XAML page (usually MainPage.xaml) in the Visual Studio designer
Open the properties for the PhoneApplicationPage and set the DataContext to be Application.Resources –> LocalizedStrings. NOTE: if you already are using a DataContext object, then you should integrate the LocalizedStrings class into that object so that it has localization support.
Once the Page’s DataContext has been set you can change the data binding for any control on the page by simply selecting the property (ex: text, checked, etc), selecting “Apply Data Binding…”, and setting the Path to Localizedresources.BtnText or whatever the name of the desired resource value is.

If you want to change the text in the Application Bar you have to jump through a few hoop because it is not actually a Silverlight control and does not support data binding. Instead you have to create or update the Application Bar at runtime.

I hope this helps!

Excel 2010 Regular Expressions: Manually installing Morefunc 5 Add-in

noreply@blogger.com (Greg Bray) — Sat, 11 Sep 2010 00:26:00 +0000

I came across a need to use regular expressions in a spreadsheet today. Usually in this case I would look at using a more powerful tool such as Resolver One, which I have added into my toolbox and used for many side projects, but in this case I was stuck using Excel for legacy reasons. Sadly even the latest version of Excel 2010 does not have built in support for Regular Expressions so you either have to drop into VBA code and write your own Regex functions or install an add-on that has pre-built functions. This blog post brought me to a nice free add-on for Excel called Morefunc which has a bunch of Regex functions, but it doesn’t work right out of the box with Excel 2010. After installing it I had to follow these instructions to manually add the Morefunc.xll, Morefunc11.xla, and Morefunc12.xlam files from the “C:\Program Files\Morefunc” folder. Once I did that and allowed macros on the spreadsheet then the new functions showed up under the Morefunc category and under the Functions section of the Formula ribbon group. Since I didn’t see these instructions posted anywhere else I figured I would post them here so that others could find them.

Enjoy!

Powershell Backup All Databases Locally and to Amazon S3

noreply@blogger.com (Greg Bray) — Mon, 14 Jun 2010 18:13:00 +0000

Previously I had posted a Powershell script that would use 7Zip to compress and encrypt a folder and then send it to Amazon S3. This script could have many uses, but in my case it was designed to fulfill the need for off site storage of compressed database backups. While it did all the heavy lifting of getting the data encrypted, compressed and uploaded to S3 I left the creation of the database backup folder as an exercise for the user to complete.

I now have a working system that I would like to share in case anyone was unable to create the backup folder using Powershell. There were a few hiccups that I found, such as not being able to use the Start-Transcript method in a SQL Agent Job and having my MSDB data file triple in size after 3 months of backups, but after working out those issues the script provides a very cost effective method of creating both local and offsite database backups with 1 week or 1 month retention. Currently I have it running on a server with about 100 databases ranging in size from 7MB to 700MB with daily full backups and hourly differential backups being sent to Amazon S3 and stored with 1 month retention for about $5 a month. The total raw backup size is 2.2GB per full backup set and 100-400MB for each differential backup set, but the script gets about 10:1 compression so the daily storage on S3 is about 500MB total for all of the full and differential backups.

You will need to modify the settings in both the psBackupAllDBtoAmazonS3.ps1 and ps7ZiptoAmazonS3.ps1 files to get this to work on your server, but once you do you can sit back and let the scripts do it’s magic. You can set it to run as a scheduled task, but we chose to run it as an SQL Agent Job so that our server monitoring software would notify us if there were any issues with the script.

#Description: Powershell 2.0 script to backup all databases in MS SQL Server to the local disk and then zip and encrypt the folder using 7zip and then send it to Amazon S3 for offsite storage.
#Author: Greg Bray (6/14/2010)
#Website: http://codeblog.theg2.net/2010/06/powershell-backup-all-databases-locally.html
#License: free for personal or commercial use under the Creative Commons Attribution 3.0 United States License.
#Prerequisite: You must first install the latest version of 7Zip from http://www.7-zip.org/ (licensed under the GNU LGPL)
#Prerequisite: You must also install the Amazon Web Services SDK for .NET from http://aws.amazon.com/sdkfornet/ (licensed under the Apache License, Version 2.0)
#Prerequisite: You must also download the ps7ZiptoAmazonS3.ps1 from http://codeblog.theg2.net/2010/02/powershell-7-zip-amazon-s3-upload.html
#Prerequisite: You must set the settings listed below as well as the settings listed in the ps7ZiptoAmazonS3.ps1 file
#Recommended: I recommend using CloudBerry S3 Explorer for managing Amazon S3 files. http://cloudberrylab.com/default.aspx?page=cloudberry-explorer-amazon-s3
#Recommended: I recommend using http://PowerGui.org Script editor if you want to change or debug this script.

<#Example calling from Run, a batch file, SQL Agent Job, or the task Scheduler: NOTE: may need to run "Set-ExecutionPolicy RemoteSigned" in powershell to allow unsigned local scripts to execute
powershell.exe -Command "& 'c:\Backup\psBackupAllDBtoAmazonS3.ps1'"
#>

#Script will backup all active databases on the server except those listed on an exclusion list. 
#Performs Differential backup except when $bFullBackup is true (currently set to do full backup when run between 00:00 and 01:00)
#Uses a 1 week local retention and either a 1 week or 1 month offsite retention on Amazon S3 depending on the definition of $S3FilePath
#Offsite files will be compressed and encrypted using 7zip. Local files will not be compressed or encrypted.
#There also will be a transcript log created that can be used to monitor the script. NOTE: If run using SQL Agent Job Start-Transcript will not work.
$strLogfile = "c:\Backup\Backup Log.rtf"
if (($host.Name -notmatch '^PowerGUI') -and ($host.name -ne "Windows PowerShell ISE Host")) {#PowerGUI and ISE Hosts do not support transcriptions yet :-(
    #Start-Transcript $strLogfile -Append | Out-Null #Append output to log file. Disabled as SQL Agent Jobs don't work with powershell transcripts
    #NOTE: Transcripts don't work with SQL Agent Jobs either... See http://stackoverflow.com/questions/2383605/powershell-transcript-is-empty-when-running-script-from-sql-agent-job-in-2005-sql
}

#Settings
$strBackupPathBase = "c:\Backup" #Base path on server where to store backups. NOTE: Do not include trailing slash.
$arrDBsToExclude = @("tempdb", "master") #DBs not included in the backup set. NOTE: Master database does not support differential backups, but will be included in full backups (see $strBackupQuery definition below)
$dtStart = [DateTime]::Now #capture start date to track total time
$bFullBackup = ($dtStart.Hour -eq 0) #Perform full backup instead of differential when run between 00:00 and 01:00
$strDateTimeStamp = $dtStart.ToString("yyyy-MM-dd_HHmm").ToUpper() #Timestamp appended to all backup files. NOTE: this is not the actual backup time, but when the script starts running
$strBackupPath = "{0}\{2}-{1:ddd}\{1:HH}" -f $strBackupPathBase, $dtStart, $dtStart.DayOfWeek.value__ #Full path where files will be stored locally (1 week retention). If folder exists it will be deleted
#See $S3FilePath and DB Connection string definitions below. Also update encryption settings in ps7ZiptoAmazonS3.ps1
Write-Host ("`r`n`r`n{0} - Starting SQL Server Differential backup with these parameters:`r`n`tBackupPath='{1}'`r`n`tTimestamp={2}`r`n`tFullBackup={3}`r`n`tExclude Databases: ${arrDBsToExclude}" -f [DateTime]::Now, $strBackupPath,$strDateTimeStamp,$bFullBackup)

#Delete backup directory if it already exists. This creates a 1 week local retention for the backup files
if([System.IO.Directory]::Exists($strBackupPath)){
    Remove-Item $strBackupPath -Force -Recurse
}
New-Item $strBackupPath -type directory | Out-Null #Create directory again

#Setup database connection 
$conn = new-object System.Data.SqlClient.SqlConnection
$conn.ConnectionString = 'Data Source=localhost;Initial Catalog=master;User ID=sa;Password=SAPASSWORD' #NOTE: use ' instead of " to prevent issue with $ character
if($conn.State -ne [System.Data.ConnectionState]::Open){ $conn.Open();} #Open connection if not already open

#Get a list of all databases
$cmd = new-object System.Data.SqlClient.SqlCommand
$cmd.Connection=$conn
$cmd.CommandType = [System.Data.CommandType]::Text
$cmd.CommandText= "SELECT name FROM sys.databases"
$sqlReader = $cmd.ExecuteReader()
$lstDatabases = New-Object System.Collections.ArrayList
while ($sqlReader.Read() -eq $true){
    [void]$lstDatabases.Add($sqlReader.GetString(0)) #Add database name to list
}
$sqlReader.Close()

#Backup all databases to disk
$strBKType = "DIFF"
$intBackups = 0
$strBackupQuery = "BACKUP DATABASE [{0}] TO DISK = '{1}' WITH DIFFERENTIAL, INIT" #0: Database name, 1: Backup file path
if($bFullBackup){# Perform full backup instead of differential
    $strBackupQuery = "BACKUP DATABASE [{0}] TO DISK = '{1}' WITH INIT" #0: Database name, 1: Backup file path
    $arrDBsToExclude = @() + ($arrDBsToExclude | where {$_ -notmatch "master"}) #make sure master database is included in the full backup
    $strBKType = "FULL"
}
foreach($strDatabase in $lstDatabases){
    if($arrDBsToExclude -notcontains $strDatabase){
        #write-Host ("`r`n{0} - Backing up database: {1}" -f [DateTime]::Now, $strDatabase)
        $strFilename = "{0}\{1}_{2}_{3}.bak" -f $strBackupPath, $strDatabase, $strBKType, $strDateTimeStamp
    
        $cmd = new-object System.Data.SqlClient.SqlCommand
        $cmd.Connection=$conn
        $cmd.CommandType = [System.Data.CommandType]::Text
        $cmd.CommandText=  $strBackupQuery -f $strDatabase, $strFilename
    
        $intBackups += 1
        $intReturn = $Null
        $intReturn = $cmd.ExecuteNonQuery()
        if ($intReturn -eq $Null) {
                write-Host ("{0} - Database {1} backup failed ({2} of {3}). Retry after 10 seconds." -f [DateTime]::Now, $strDatabase,$intBackups,$lstDatabases.Count)
            Start-Sleep -s 10
            $intReturn = $cmd.ExecuteNonQuery()
            if ($intReturn -eq $Null) {
                    write-Host ("{0} - Database {1} backup failed ({2} of {3}). Script aborted." -f [DateTime]::Now, $strDatabase,$intBackups,$lstDatabases.Count)
                exit 2 #Quit script if there is an error
            }
        }
        #write-Host ("{0} - Database {1} backed finished successfully ({2} of {3})" -f [DateTime]::Now, $strDatabase,$intBackups,$lstDatabases.Count)
    }
}

#Perform basic monthly db maintinece here. Will execute on first day of the month when the full backup is created
if($bFullBackup -and ($dtStart.Day -eq 0)){
    #Clean up MSDB backup history logs to prevent excessive growth of MSDB. See http://technet.microsoft.com/en-us/library/ms188328.aspx
    write-Host ("{0} - Start server maintinence: sp_delete_backuphistory on MSDB" -f [DateTime]::Now)
    $cmd = new-object System.Data.SqlClient.SqlCommand
    $cmd.Connection=$conn
    $cmd.CommandType = [System.Data.CommandType]::Text
    $cmd.CommandText= 'use msdb; Declare @date datetime; Set @date = dateadd (mm, -1, getdate()); Exec sp_delete_backuphistory @date' 
    $Null = $cmd.ExecuteNonQuery()
}

if($conn.State -eq [System.Data.ConnectionState]::Open){ $conn.Close();} #Close database connection

#Call powershell script to encrypt data and send it to Amazon S3. NOTE: there are other settings configured in the script file.
#$S3FilePath = "DB2/{2}-{0:ddd}/Hour_{0:HH}_{1}_Backup.7z" -f $dtStart,$strBKType,$dtStart.DayOfWeek.value__ #S3 path using DB2 bucket and 1 week retention
$S3FilePath = "DB2/{0:dd}/Hour_{0:HH}_{1}_Backup.7z" -f $dtStart,$strBKType #S3 path using DB2 bucket and 1 month retention
& 'C:\Backup\ps7ZiptoAmazonS3.ps1' "$strBackupPath" "$S3FilePath"

$dtEnd = [DateTime]::Now
$tsTotal = $dtEnd.Subtract($dtStart) #Get total processing time
write-Host ("`r`n`r`n{0} - Finished SQL Server Backup. Total processing time: {1}`r`n`tBackup Path:{5}`r`n`tS3 Storage Path:{7}`r`n`tFull Backup:{6}`r`n`tTotal databases backed up: {2} of {3}`r`n`t{4} Exclusions: - ${arrDBsToExclude} " -f $dtEnd,$tsTotal,$intBackups,$lstDatabases.Count,$arrDBsToExclude.Count, $strBackupPath, $bFullBackup, $S3FilePath)
if (($host.Name -notmatch '^PowerGUI') -and ($host.name -ne "Windows PowerShell ISE Host")){
    #Stop-Transcript #Disabled for use with SQL Job.
}
exit $LASTEXITCODE #Make sure to pass exitcode from ps7ZiptoAmazonS3 to host for error tracking

Lap Around .NET 4 with Scott Hanselman

noreply@blogger.com (Greg Bray) — Fri, 16 Apr 2010 20:39:00 +0000

Things were slow at the office today, so I used the free time to catch up on some of the recorded sessions from DevDays 2010. Scott Hanselman’s overview of the new features in .NET 4 is one of the best presentation I have seen about .NET 4 so far. How many presentations include 14,000 generations of monkeys trying to write Shakespeare? Here is the synopsis:

In this session, Scott Hanselman gives a deep and broad tour of the .NET 4 release, with a focus on making your development experience easier. See lots of demos (and very few slides) showcasing the key new features in the .NET Framework 4 including MEF, improvements in ASP.NET, threading, multi-core and parallel extensions, additions to the base classes, changes and additions to the CLR and DLR, what's new for the languages (Visual Basic and C#), and of course, what's new in Windows Presentation Foundation and System.Web. Come and see how all these new features and capabilities improve your overall .NET experience!

Enjoy!

TED Fullscreen Video Multiple Monitor Userscript: Yay!

noreply@blogger.com (Greg Bray) — Sun, 14 Mar 2010 07:35:00 +0000

I like to think that I can hack out JavaScript as good as any web developer, but occasionally I still fall into rookie mistakes. For instance, I totally forgot to test my last userscript for fixing Slashdot articles while zooming in anything but my default browser (Google Chrome). In theory it should have worked with Greasemonkey, but when I tried testing it today using Firefox it failed miserably. Luckily with a few code changes I was able to update the script and get it to work with both Firefox and Google Chrome as well as fix a huge bug when visiting Slashdot’s main page as a non-registered user (another thing I probably should have tested!).

Considering that it was my first attempt at a user script I am not too upset and will chalk it up to experience. Tonight my plans for the evening fell apart, so I decided to try and fix a big pet peeve of mine: no multiple monitor support for video players on certain websites. Maybe I am the only one that thinks this is a problem, but I hate it when websites have a Flash or Silverlight video player embedded onto their page and don’t have an option to enlarge the video or open it in a pop-out window. Most people do fine using the fullscreen mode built into the player, but both Silverlight and Flash fail at fullscreen with multiple monitors since they will exit fullscreen as soon as you try and click on something outside the video (for security purposes apparently). This means you can either stop multi-tasking and only watch the video or you have to settle for a small video player that is absolutely tiny on anything larger than 1024 resolution width. For instance: here is a video that a friend brought to my attention as viewed on a 1600x900 LCD TV.

That is a lot of white space going to waste, and while the fullscreen mode works I often watch videos on the LCD TV while browsing the web on a projector connected to the same computer and end up switching in and out of fullscreen mode multiple times. I’ve had this same problem before with Channel9 and fixed it using a bookmarklet, but that actually requires clicking a link in my favorites every time I loaded a video, which was less than ideal. Now that Chrome supports userscripts I am working on converting some of the bookmarklets that I have into userscripts, and since I already have a working solution for Channel9 I thought that I would try creating one for TED.com next.

So if you ever watch TED videos on a machine with multiple monitors then today is your lucky day. This userscript will enable multiple monitor fullscreen video for any page that uses the TED.Com flash player. Videos that are embedded from YouTube currently are not supported, but YouTube already lets you increase the width of the player, open it in a pop-out window, or even use the browser’s zoom feature to resize the video so that shouldn’t be a problem. And this time I did do some basic testing in both Firefox and Chrome so hopefully it will work as advertised.

Enjoy!

Slashdot Zoom Fix: a Greasemonkey and Chrome compatible userscript

noreply@blogger.com (Greg Bray) — Fri, 05 Mar 2010 23:40:00 +0000

One of my top ten visited websites is probably Http://www.slashdot.org who’s headline is “News for nerds, stuff that matters”. The site usually posts over 20 stories a day covering technology, math, physics, and computer related topics, which help me keep informed and entertained. One problem that I often have is that the text size is very hard to read when using a high resolution monitor or when using the projector that I have setup on a media center computer at home. Usually this is easy to solve, since one of the 11 buttons on my mouse is mapped to the CTRL button and can be combined with the scroll wheel to zoom in and out on web pages quickly. However in this case there are boxes on the right hand side of the screen that also enlarge when the page is zoomed, causing the text width to be smaller and more difficult to read. Here is an article on Slashdot normal and zoomed:

Normal

Zoomed

The site is designed using CSS and for some reason the advertisement and “Interviews” box will get bigger when the page is zoomed, shrinking the width of the textbox significantly. If you are a registered user Slashdot will not display the ad but will still have a side panel on the right that grows larger when you try and zoom in. I initially created a bookmarklet that could be used to remove the side panel by clicking on a link in my favorites to inject some javascript into the webpage to alter the CSS classes, but that still required that I manually click the button every time that I visit he website.

I planned on making a Google Chrome Extension, but recently Chrome announce that they now will automatically convert Greasemonkey scripts into Chrome Extensions. This means you can write a cross-browser compatible script that can run in Firefox, Chrome or any other browser that supports Greasemonkey scripts. It only took a few minutes to convert the bookmarklet into a userscript, and I posted it on userscripts.org so that anyone can use it. The script will automatically remove the sidebar so that zooming works much better. You can even change the script to automatically zoom a specified amount, however I left this off by default since the right amount highly depends on the width of your screen. Here are the results:

Normal

Zoomed

Hopefully someone else will find this useful. I have a few other bookmarklets that I use and will probably convert into userscripts soon, as well as a few new ones that I want to work on when I get some free time.

Powershell, 7-Zip, Amazon S3 Upload Script with AES-256 Encryption

noreply@blogger.com (Greg Bray) — Fri, 19 Feb 2010 23:08:00 +0000

I recently was tasked with finding a way to store some backup files from our server in a secure and reliable off-site location. After talking with our hosting provider, who wanted around $600 a month for off-site tape rotation, we decided to look at using Amazon Simple Storage Service (Amazon S3) to store the files in the cloud instead. We needed a way to automate the upload process and make sure that the data was encrypted, so I spent a few days working on a Powershell script (using the excellent PowerGui Script Editor) that uses 7-zip to create a .7z archive with AES-256 encryption and then send it up to Amazon S3 using the Amazon Web Services SDK for .NET. Here is the script:

#Description: Powershell 2.0 script to zip and encrypt a folder using 7zip and then send it to Amazon S3 #Author: Greg Bray (2/19/2010) #Website: http://codeblog.theg2.net/2010/02/powershell-7-zip-amazon-s3-upload.html #License: free for personal or commercial use under the Creative Commons Attribution 3.0 United States License. #Prerequisite: You must first install the latest version of 7Zip from http://www.7-zip.org/ (licensed under the GNU LGPL) #Prerequisite: You must also install the Amazon Web Services SDK for .NET from http://aws.amazon.com/sdkfornet/ (licensed under the Apache License, Version 2.0) #Recommended: I recommend using CloudBerry S3 Explorer for managing Amazon S3 files. http://cloudberrylab.com/default.aspx?page=cloudberry-explorer-amazon-s3 #Recommended: I recommend using http://PowerGui.org Script editor if you want to change or debug this script. #Usage: ps7ZiptoAmazonS3 'BackupFolderPath' 'S3ObjectKey' 'TranscriptLogFile' # BackupFolderPath = Folder to backup (do not include a trailing slash) # S3ObjectKey = Object key where to store file on Amazon S3. Note: If object already exists on S3 it will be overwritten. # TranscriptLogFile = Optional log file where the transcript will be appended. #NOTE: you also will need to set the Zip and Encrypt settings, Amazon S3 settings, and 7Zip/AWSSDK file paths below <#Example calling from another powershell script (with log file): & 'c:\Backups\ps7ZiptoAmazonS3.ps1' 'c:\Backups\DataFolder' 'S3Backups/DataFolder.7z' 'c:\Backups\BackupLog.txt' #> <#Example calling from Run or task Scheduler: NOTE: may need to run "Set-ExecutionPolicy RemoteSigned" in powershell to allow unsigned local scripts to execute powershell.exe -Command "& 'c:\Backups\ps7ZiptoAmazonS3.ps1' 'c:\Backups\DataFolder' 'S3Backups/DataFolder.7z' 'c:\Backups\BackupLog.txt'" #> #Check arguments and set variables if ($args.count -ge 2) { $FolderPath = $args[0] #Local folder to encrypt and send to S3 (No trailing slash) $S3ObjectKey = $args[1] #Destination on S3 where file will be stored. Note: If object exists it will be overwritten. if ($args.count -ge 3) { #start logging transcript Start-Transcript $args[2] -Append | Out-Null } } else { #Default preset arguments used if no values are provided $FolderPath = "c:\Backups\DataFolder" $S3ObjectKey = "S3Backups/DataFolder.7z" } write-host "`r`n`r`n" #Start log file with 2 blank lines $dtStart = [System.DateTime]::Now #Save start time to calculate total time below write-host ($dtStart.ToString() + " - Script started") #Zip and Encrypt settings $EncryptionKey = "secret_password" #Data is encrypted using Strong AES-256 encryption. Recommended password size is 10-16 characters. $TemporaryPath = "c:\Temp" #No trailing slash. Used to store zip file localy before uploading. File will be deleted after upload but not if there is an exception (see below). #Amazon S3 account settings (See http://aws-portal.amazon.com/gp/aws/developer/account/index.html?action=access-key): $S3AccessKeyID = "ABCDEFGHIJKLMNOPQRST" #Your Amazon S3 Access Ky ID $S3SecretKeyID = "0123456789abcdefghijklmnopqrstuvwzyz" #Your Amazon S3 Secret Access Key $S3BucketName = "S3BackupBucket" #S3 Bucket name where the file will be stored Write-Host ($dtStart.ToString() + " - FolderPath:'$FolderPath' S3BucketName:'$S3BucketName' S3ObjectKey:'$S3ObjectKey'") #Load AWS Assembly and Initialize 7Zip path (only runs once when debugging script) if($7ZipPath -eq $null){ $7ZipPath = "C:\Program Files\7-Zip\7z.exe" #Path to 7Zip $AWSDOTNETSDKPath = "C:\Program Files (x86)\AWS SDK for .NET\bin\AWSSDK.dll" #Path to Amazon SDK for x64 system. Remove " (x86)" when running on an x86 machine if (-not [System.IO.File]::Exists($AWSDOTNETSDKPath)) { #Try path for x86 system $AWSDOTNETSDKPath = $AWSDOTNETSDKPath.Replace(" (x86)","") if (-not [System.IO.File]::Exists($AWSDOTNETSDKPath)) { #File not found, throw exception write-error "ERROR: Cannot find Amazon Web Services SDK. Please download .NET SDK from http://aws.amazon.com/sdkfornet/ and update AWSDOTNETSDKPath in script." exit 2 } } #Load AWS Assembly [Reflection.Assembly]::LoadFile($AWSDOTNETSDKPath) | Out-Null } #Function for computing Amazon S3 MD5 digest (128 bit base64 encoded) using System.Security.Cryptography.MD5 function Hash-MD5 ($file) { $cryMD5 = [System.Security.Cryptography.MD5]::Create() $fStream = New-Object System.IO.StreamReader ($file) $bytHash = $cryMD5.ComputeHash($fStream.BaseStream) $fStream.Close() return [Convert]::ToBase64String($bytHash) } #Zip and Encrypt files Write-Host ([System.DateTime]::Now.ToString() + " - Zip and Encrypt Files") $FolderName = $FolderPath.Substring($FolderPath.LastIndexOf("\")+1) #grab the name of the backup folder $TimeStamp = [datetime]::Now.ToString("yyyy-MM-dd_HHmm") #Create unique timestamp string $strFile = [String]::Format("{0}\{1}_{2}.7z", $TemporaryPath,$FolderName,$TimeStamp) #Create filename for the zip #7z options: add, type 7z, Archive filename, Files to add (with wildcard. Change \* to \prefix* or \*.txt to limit files), compression level 9, password, encrypt filenames, Send output to host so it can be logged. #See http://dotnetperls.com/7-zip-examples or the 7zip help file for more info on command line switches. #NOTE: -mx7 and -mx9 get around 10 to 1 compression for text files and take round 5 minutes for 1GB of data. Use -mx3 or -mx5 to increase speed but decrease compression level. & $7ZipPath "a" "-t7z" "$strFile" "$FolderPath\*" "-mx9" "-r" "-p$EncryptionKey" "-mhe" | out-host #Call 7z.exe to create archive file if($LASTEXITCODE -ne 0){ #Detect errors from 7Zip. Note: 7z will crash sometimes if file already exists Write-Error "ERROR: 7Zip terminated with exit code $LASTEXITCODE. Script halted." exit $LASTEXITCODE } #Create Amazon PutObjectRequest. Details at http://docs.amazonwebservices.com/sdkfornet/latest/apidocs/html/T_Amazon_S3_Model_PutObjectRequest.htm Write-Host ([System.DateTime]::Now.ToString()) " - Creating S3 PutObjectRequest" $AmazonS3 = [Amazon.AWSClientFactory]::CreateAmazonS3Client($S3AccessKeyID, $S3SecretKeyID) $S3PutRequest = New-Object Amazon.S3.Model.PutObjectRequest $S3PutRequest.BucketName = $S3BucketName $S3PutRequest.Key = $S3ObjectKey $S3PutRequest.FilePath = $strFile Write-Host ([System.DateTime]::Now.ToString()) " - Compute File MD5 Digest" $strMD5 = Hash-MD5($strFile) #NOTE: may take 5-30 seconds to compute digest for 1GB file. $S3PutRequest.MD5Digest = $strMD5 #Use MD5 digest to ensure data is not corrupted during transport. $dFileSize = (Get-Item $strFile).Length / 1MB #Get archive size in MB Write-Host ([System.DateTime]::Now.ToString()) ([String]::Format(" - Uploading File:'{0}' Size:{1:#,###.#} MB", $strFile, $dFileSize)) $S3Response = $AmazonS3.PutObject($S3PutRequest) #NOTE: upload defaults to 20 minute timeout. #If upload fails it will throw an exception and $S3Response will be $null if($S3Response -eq $null){ Write-Error "ERROR: Amazon S3 put requrest failed. Script halted." exit 1 } Write-Host ([System.DateTime]::Now.ToString()) " - Delete Local 7z File:'$strFile'" Remove-Item $strFile -Force $dtEnd = [datetime]::Now $tsTotal = $dtEnd.Subtract($dtStart) Write-Host ($dtEnd.ToString()) " - Script finished. Total time: " $tsTotal write-host "`r`n`r`n" #End log file with 2 blank lines if ($args.count -ge 3) { #stop transcript Stop-Transcript }

It seems to work pretty well so far, taking about 5-10 minutes to zip and encrypt 1GB of SQL Backups down to 100MB and then upload it to Amazon S3. From there we can use tools like CloudBerry S3 Explorer to browse or download files when needed. The monthly costs to keep data on Amazon S3 is $0.150 per GB, with $0.10 per GB transfer in and $0.150 per GB transfer out. With a 1 week backup retention and minimal data-out transfers we expect to pay around around $10 to $20 a month and should be able to access it much quicker than if we were using off-site tape storage. Cloud computing FTW!

UPDATE 6/14/2010: I just posted the script used to create the Database backup folder.

Parallel ForEach file processing in IronPython: Get some TPL love in IPY!

noreply@blogger.com (Greg Bray) — Fri, 12 Feb 2010 23:58:00 +0000

I recently have started playing around with the Task Parallel Library (TPL) that will be shipping in .NET 4, which lets you easily spread a workload across multiple cores using a simple Parallel.ForEach statement. I have a few IronPython scripts that I want to convert to start using multiple threads, so I thought I would try using the .NET 3.5 version of the TPL from inside of IronPython. It turns out that the TPL works just fine, but writing thread safe code in IronPython can be tricky. First off, the System.Threading.Interlocked class does not work on IronPython integers because python integers are immutable. That really sucks, because it means that the only way to change a global integer value in a thread safe manner is to use locks or the System.Threading.Monitor class. One interesting workaround is to use a python list instead of an integer. Appending a value to a list is an atomic operation, so instead of calling Interlocked.Add you can just append all the values to an empty list an use sum to return the aggregate value or len() to get the number of items in the list (emulating incrementing a variable). It is a bit of a hack, but sometimes easier that using locks. Also, you have to remember that the print statement is not thread safe, so instead you should use something like System.Text.StringBuilder to buffer your print statements and then print them once you are back to a single thread code section.

Below is a small sample script that I wrote to test processing a large number of text files using IronPython. It will search through all files in a given directory that match a given pattern (ie: *.txt) and return the total number of files and lines. It will also search for a given tolken inside each file and return the total number of matches. This is a task that is very easy to run on multiple cores and is a perfect fit for the Parallel.ForEach method. By default it will search your temporary files folder, which on my workstation had 69 .txt files with a total of 469,286 lines. The single threaded version took about 2.3 seconds to run and the multi-threaded version took between 0.8 and 1.0 seconds. The workload was spread across 8 cores, which caused a 2x improvement in speed, even thought this example is still primarily bound by the drive IO speed. Still it shows how using the TPL can greatly increase performance for common tasks.

"""Uses the Task Parallel Libray (TPL) to processes files using multiple threads.
Please download and install the Reactive Extensions (Rx) for .Net 3.5 from http://msdn.microsoft.com/en-us/devlabs/ee794896.aspx before running this script
Usage: import the file into an existing script or run the following from the IronPython console:
execfile("c:\PathToFile\Filename.py")
"""
import clr
from System import Environment
from System.IO import File,Directory
from System.Diagnostics import Stopwatch
from System.Text import StringBuilder
from System.Threading import Monitor
clr.AddReference('System.Windows.Forms')

strAssembly = "C:\Program Files (x86)\Microsoft Reactive Extensions\Redist\DesktopV2\System.Threading.dll";
if not File.Exists(strAssembly): # Try both 32bit and 64bit file path
    strAssembly = "C:\Program Files\Microsoft Reactive Extensions\Redist\DesktopV2\System.Threading.dll";
    if not File.Exists(strAssembly): # Library does not exist, prompt with error message
        print "TPL assembly not found. Please download and install the Reactive Extensions (Rx) for .Net 3.5 from http://msdn.microsoft.com/en-us/devlabs/ee794896.aspx"
        raise Exception("Task Parallel Libray not found. Please download and install the Reactive Extensions (Rx) for .Net 3.5");
        #Rx and TPL for .NET 3.5 can be downloaded from  http://msdn.microsoft.com/en-us/devlabs/ee794896.aspx

#Load TPL
clr.AddReferenceToFileAndPath(strAssembly)
from System.Threading.Tasks import *

#Method to process files. See test call at bottom of file
def TPLProcessFiles(strTolken, strPath=None, strFilter="*.txt"):
    """Finds all occurences of strTolken in strPath for files that match strFilter.
        strTolken = string tolken used for searching. NOTE: search is case-insensitive and includes partial match (cat will match Category).
        strPath = Path to the directory used for searching. Default is system's temp directory
        strFilter = file match filter for search. Default is *.txt"""
    print "Start TPLProcessFiles('%s', '%s', '%s')" % (strTolken, strPath, strFilter)
    
    if strPath == None:
        strPath = Environment.GetEnvironmentVariable("TEMP")
        print "Using temporary files path: %s" % strPath
    elif not Directory.Exists(strPath):
        raise Exception("Invalid path: " + strPath)
    
    #Setup
    swTime = Stopwatch()
    swTime.Start()
    LogWriter.Reset()
    Results.Reset()
    strFiles = Directory.GetFiles(strPath, strFilter)

#Single thread test code. Useful for when you get an AggregateException_ctor_DefaultMessage exception in the Parallel.ForEach version or for benchmarking
    #for aFile in strFiles:
    #    SearchFile(aFile, strTolken)
    
    #TPL version with multiple threads
    pfeResult = Parallel.ForEach(strFiles, lambda aFile: SearchFile(aFile, strTolken))
    
    print LogWriter.ToString() #Have to use LogWriter class below because print is not a threadsafe function
    strResults = Results.ToString() #Have to store results in a thread safe class
    swTime.Stop()
    print "Finished. Total Files: %d  LogCallCount: %d   Total Time: %0.3fs  Total Cores: %d" % (len(strFiles), LogWriter.GetCallCount(), float(swTime.ElapsedMilliseconds)/1000, Environment.ProcessorCount)
    print "Results - %s" % strResults

#File search method    
def SearchFile(strFilename, strTolken):
    """Method used for searching a file. This will be called from the Parallel.ForEach loop.
        Returns a tuple of (Filename, Total line count, Total tolken count).
        NOTE: Code must be thread safe, so you cannot use the print statement or increment global variables."""
    #LogWriter.WriteLine("Start file: %s searching for %s" % (strFilename, strTolken))
    f = open(strFilename, "r") #open file as readonly
    t = f.read() #get text
    f.close()
    Results.Add(1, t.count("\n"),t.count(strTolken)) #Add to global results

#Class used to store results    
class Results:
    Files = 0 #static members
    Lines = 0
    Tolkens = 0   
    
    @staticmethod
    def Reset(): #NOTE: this is not thread safe, but we will only call this method from a single thread
        Results.Files = 0
        Results.Lines = 0
        Results.Tolkens = 0

@staticmethod
    def Add(intFiles, intLines, intTolkens):
        Monitor.Enter(Results) #Use the System.Threading.Monitor class to ensure addition is thread safe
        Results.Files += intFiles
        Results.Lines += intLines
        Results.Tolkens += intTolkens
        Monitor.Exit(Results) #Finish thread safe code

@staticmethod
    def GetValues():
        return (Results.Files, Results.Lines, Results.Tolkens)
        
    @staticmethod
    def ToString():
        return "Total files: %d  Total lines: %d  Total tolkens: %d" % Results.GetValues()
        
#Log writer class to buffer print statements    
class LogWriter:
    sbLog = StringBuilder() #StringBuilder is thread safe when used as a static member
    CallCount = [] #Integer operations are not threadsafe, but appending to a list is
    
    @staticmethod
    def WriteLine(strMessage): #NOTE: must be thread safe. String 
        LogWriter.sbLog.Append(strMessage+"\r\n")
        LogWriter.CallCount.append(1)
    
    @staticmethod
    def ToString():
        return LogWriter.sbLog.ToString()
    
    @staticmethod
    def Reset():
        LogWriter.sbLog.Length = 0
        LogWriter.intCallCount = []

@staticmethod
    def GetCallCount():
        return sum(LogWriter.CallCount)

if __name__ == "__main__":
    TPLProcessFiles("Error") #Run test if this file is loaded from the IronPython Console

TPL and Parallel.ForEach in .Net 3.5 using Reactive Extensions for .NET (Rx)

noreply@blogger.com (Greg Bray) — Sat, 06 Feb 2010 23:39:00 +0000

The next version of the .Net framework and Visual Studio both have some pretty cool features to help programmers work with multiple cores, which is great but doesn’t help the majority of us that are stuck with .Net 3.5 for the foreseeable future. Luckily Erik Meijer and the Cloud Programability Team have back ported the Parallel Extensions Framework (PFX) to .Net 3.5 and Silverlight 3 as part of the Reactive Extensions for .NET (Rx). Rx adds the IObservable<T> and IObserver<T> interfaces, which are the mathematical duality of the IEnumerable<T> and IEnumerator<T> and provide tools for doing Reactive Programming. There are many different ways to use Rx, but internally they all use the Task Parallel Library (TPL) as the “special sauce” to automate processing tasks across multiple threads.

Parallel.ForEach is a part of the TPL that can be used to unroll an outer loop and have it run across multiple threads. Take the following example. This is standard single threaded code that loops through a collection of 2 card Texas Holdem starting hands and evaluates all possible 7 card hands that include those two cards:

There are over 2 million hands in the inner loop, which gets run once for each of the possible starting hands in the outer loop. There are 12 offsuit starting hands and 4 suited starting hands with an Ace and a King, which means that the outer loop would run 16 times, however those 16 executions are separate and could easily be run across multiple threads. That is where Parallel.ForEach comes in. Here is the same code, which will automatically be scheduled across multiple threads:

The bulk of the code is the same, with the only changes being that the inner block gets converted into a Lambda Expression (could have also used a delegate) and instead of incrementing lCount in the inner loop we increment a local loop variable and then atomically add it to the global value. We could have used System.Threading.Interlocked.Increment to atomically increment lCount inside the inner loop, but this adds a lot of unneeded locks that slow down all the threads. Keeping a local copy of the values and only locking once at the end provides much better performance.

And with those few small changes we are able to start using multiple threads, which on my local machine with 4 cores and 8 threads ended up decreasing the processing time from 1.55 seconds to 0.67 seconds and more than doubling the number of hands processed per second from 16,420,998 to 37,816,595.

If you want to start using the TPL you can download it here and add a reference to the System.Threading.dll located in the “C:\Program Files (x86)\Microsoft Reactive Extensions\Redist\DesktopV2” folder. And while you are at it you might as well play around with Rx too!

Enjoy!

VS2010 Parallel Computing Features Tour: Wow!

noreply@blogger.com (Greg Bray) — Wed, 20 Jan 2010 19:40:00 +0000

Microsoft Visual Studio 2010 and .NET Framework 4.0 are in beta 2 right now and should be available in a few months, but after watching videos about the features of C# 4.0 and the features of Visual Studio 2010 I don’t know how much longer I can wait! I just finished watching a video about the parallel computing features in VS2010 and I seriously am drooling over the multi-core programming tools available in the next release! C# 4.0 was introduced by Anders Hejlsberg at PDC 2008, but it is great to see it finally coming to fruition. And with .NET 4 splitting the download into Client and Full profiles it now can be downloaded and installed on most machines at about half the size (30-40MB vs 70-160MB for .NET 3.5).

Watin 2.0 and Gallio 3.1 in x64 environment: fixing System.Reflection.ReflectionTypeLoadException and System.Runtime.InteropServices.COMException "The interface is unknown" issues

noreply@blogger.com (Greg Bray) — Thu, 03 Dec 2009 23:56:00 +0000

We use Gallio/MbUnit and Watin at my work for unit testing of an ASP.NET website, as it provides excellent reports including the raw Watin events (click, type, select) and screenshots for failed cases. Today I spent a few hours trying to get the project working on my new Windows 7 x64 development machine, and things did not start out very well. When I first tried to load the project in the Gallio Icarus runner none of the unit tests showed up and it indicated that an exception was thrown while exploring tests. I tried using both the x64 and x86 versions of Gallio/MbUnit, and even tried using the TestDriven.NET loader from within Visual Studio, but couldn’t get anything to work. Here is the full error message for Google’s sake:

[error] An exception was thrown while exploring tests.
Location: D:\SVN_Development\OQA Unit Testing\OQA Testing Framework\OQA Testing Framework\bin\Debug\OQA Testing Framework.EXE
Reference: OQA Testing Framework, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null
Details: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information. 
   at System.Reflection.Module._GetTypesInternal(StackCrawlMark& stackMark) 
   at System.Reflection.Assembly.GetTypes() 
   at Gallio.Common.Reflection.Impl.NativeAssemblyWrapper.GetTypes() in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Common\Reflection\Impl\NativeAssemblyWrapper.cs:line 72 
   at Gallio.Framework.Pattern.TestAssemblyPatternAttribute.PopulateChildrenImmediately(IPatternScope assemblyScope, IAssemblyInfo assembly) in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Framework\Pattern\TestAssemblyPatternAttribute.cs:line 147 
   at Gallio.Framework.Pattern.TestAssemblyPatternAttribute.Consume(IPatternScope containingScope, ICodeElementInfo codeElement, Boolean skipChildren) in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Framework\Pattern\TestAssemblyPatternAttribute.cs:line 75 
   at Gallio.Framework.Pattern.DefaultPatternEvaluator.Consume(IPatternScope containingScope, ICodeElementInfo codeElement, Boolean skipChildren, IPattern defaultPrimaryPattern) in c:\RelEng\Projects\MbUnit v3.1\Work\src\Gallio\Gallio\Framework\Pattern\DefaultPatternEvaluator.cs:line 153

The error message indicates that it is due to an issue with missing assemblies, and suggested that you make sure the CopyLocal option is set to true for all 3rd party references, but that didn’t help either. Finally I created a test project and started ripping the original project to peaces to find out why it was failing. Turns out it worked fine if I remove a reference to an interface stored in the Watin.Core.Dll assembly, so I setup a workaround and was finally able to load the tests. Only problem was that it would fail whenever the test tried to load Watin. At this point I figured it had to be an issue with the Watin dll, and it turns out that it was simply trying to load an x64 version of the library that doesn’t actually exist (the library is x86 only). This link suggested setting Visual Studio to use x86 as the build platform type instead of “Any CPU”, and once I did that the original project started working. Yay!

It didn’t take long though for something else to fail, as it threw an exception as soon as one of the tests tried to access the Document object. This time the exception was System.Runtime.InteropServices.COMException with Message="The interface is unknown. (Exception from HRESULT: 0x800706B5)" coming from mshtml.HTMLDocumentClass.IHTMLDocument2_get_url(). Another quick search lead to another StackOverflow answer suggesting to run Visual Studio as an Administrator. That seem to do the trick!

So if you are using an x64 machine to do unit testing with Gallio, MbUnit, and Watin, make sure to set the platform to x86 and make sure to run Visual Studio and Icarus as an Administrator! I should probably get back to working on those unit tests now :-P

Enjoy!

Channel 9 Full Screen Video Player: My first Bookmarklet

noreply@blogger.com (Greg Bray) — Tue, 10 Nov 2009 20:52:00 +0000

I really enjoy watching videos from Channel 9, Microsoft’s community site that targets users and developers. It is a great way to keep updated on the latest technologies and learn more about future products before they come out. The only thing that I don’t like is that the website is not very user friendly for people that have multiple monitors. This surprises me, as most developers that I know use at least two monitors, sometimes three or four. Even after their recent layout change, which increased the size of the video player significantly, the embedded video player is still too small to be able to read code from screencasts. There is of course a full screen button, but this doesn’t really work when multitasking on multiple monitors as it will close as soon as you click on another browser or window on another screen. Both Silverlight and Flash have this issue, which they consider to be a security feature, but it renders the full screen mode worthless useless you stop using your second monitor while watching the video.

Currently there are only two workarounds to the Silverlight/Flash full screen issue, both of which YouTube has already implemented using buttons at the top of the video player. The first is enlarging the video so that it takes up more of the page, and the second is opening it in a new page that can be resized to as large or small as you want. Alternatively you can use a different video player such as the embedded Windows Media Player, but while the WMV links on Channel 9 display full screen in the browser they sometimes have buffering or quality issues and cannot take advantage of the Smooth Streaming that the Silverlight player can offer. All I really wanted was a way to enlarge the video to fit the full width of the browser window, so I figured why not just write a script to do it?

Bookmarklets are essentially small snippets of javascript code that live in a browser’s favorites and do simple things like opening a Wikipedia box for searching, Subscribe/Share/Note the current webpage in Google Reader, using Google SideWiki, or adding any item to an Amazon Wishlist. Due to cross-site scripting restrictions implemented in most browsers they are relatively safe, so they shouldn’t be able to drain your bank account or spam all your friends on Facebook. Plus unlike Greasemonkey scripts they will only run when you click on the link in your favorites, so they won’t slow down or change your normal browsing habits. This seems like a perfect fit for resizing the Silverlight video player on Channel 9.

So after dissecting the way that the Google bookmarklets worked I started coding my own. If you want to try it out you can simply drag this link to your favorites (or right click and select "Add to Favorites"): C9FullScreen and then click on the new link that it created after browsing to your favorite Channel 9 video. To see how it works, here is the code that gets embedded into the C9FullScreen link:

And here is the script that gets injected into the page to resize the video:

//C9FullScreen: Increases the size of the Silverlight player to the full width of the window
//Works with Internet Explorer, Safari, Firefox, and Chrome
//Author: Greg Bray
//Information: http://codeblog.theg2.net/2009/11/channel-9-full-screen-video-player-my.html
//Blog: http://blog.theg2.net
//License: Released to Public Domain
//Version 1.1 released 5/22/2010 to use new zap file and fix issue with http://channel9.msdn.com/learn/courses/SharePoint2010Developer/SharePoint2010DeveloperRoadmap/SharePoint2010IntegrationwithPowerShell/

//alert('Start JS Code');

//Reverse a string. Used to get PID from URL
//Code From: http://www.bytemycode.com/snippets/snippet/400/
String.prototype.reverse = function(){
    splitext = this.split("");
    revertext = splitext.reverse();
    reversed = revertext.join("");
    return reversed;
}

//Method to enlarge the Channel 9 Silverlight Player.
function C9SLFullScreen(){
    var b=document.body;
    var d=document.getElementById('ctl00_MainPlaceHolder_Starter_MediaPlayer');
    if(d){
        //Good test video is http://channel9.msdn.com/posts/Charles/Introducing-Click-to-Run-in-Office-2010/
        //alert("Player DIV found. Now check if Silverlight Player is already loaded.");
        o = d.getElementsByTagName('object');
        if(o.length==1){ //Video is already playing
            //alert('Player already loaded. Clear player and grab URL/PID');
   if(navigator.userAgent.toLowerCase().indexOf('chrome/') > -1){
    alert('Player has already loaded. Resizing the video in Chrome will cause the player to load a partial version of the clip from the cache. Please refresh the page and try going C9FullScreen before starting the video.');
   } else {
    myregexURL = /m=(.+),autostart=/ //Regex to get video URL from object
    myregex = /\/ch9((?:\/\d)+)\// //Regex to get pid from video url
    mURL=myregexURL.exec(d.getElementsByTagName('object')[0].getElementsByTagName('param')[1].value)[1]; 
    mpID= myregex.exec(mURL); //Match regex
    pID = mpID[1].replace(/\//g,"");
    pID = pID.reverse();
    //alert("pID:"+pID+"\r\n"+mURL);
    d.removeChild(o[0]);//Remove active Silverlight object. Use o[0] to reference the player object
   }
        } else { //Video player not loaded. Grab movie url and pid.
            myregex = /\/ch9((?:\/\d)+)\//
            mURL=d.getElementsByTagName('a')[0].href; 
            mpID= myregex.exec(mURL); //Match regex
            if(mpID){
                pID = mpID[1].replace(/\//g,"");
                pID = pID.reverse();
            } else { //Learn/Courses videos do not include the id in the url, so grab it from the rdf tag. UPDATE 7/23/2010: Now standard C9 shows don't include it in the URL either!
                myregex2 = /\/(\d+)\/Trackback.aspx/
                dSummary = document.getElementById("contents");
                if(dSummary){
                    strHtml = dSummary.innerHTML;
                    pID = myregex2.exec(strHtml)[1];
                } else {
                    alert("Error: Cannot find trackback element containing video ID.");
                    return;
                }
            }
            //alert("pID:"+pID+"\r\n"+mURL);
        }

//Hide bar at top
        document.getElementById("cb_main").style.display = "none";
        document.body.style.padding = "0 0 0 0";
  //Get window size (Leave a little extra as some browsers will act strange with the scroll bar.
        var h = window.innerHeight || document.documentElement.clientHeight || document.getElementsByTagName('body')[0].clientHeight;
        var w = window.innerWidth || document.documentElement.clientWidth || document.getElementsByTagName('body')[0].clientWidth;
        z=document.createElement('div');
        //Set Style (see http://www.comptechdoc.org/independent/web/cgi/javamanual/javastyle.html)
        //z.style.position = "absolute";
        //z.style.left = "20px";
        //z.style.top = "200px";
        intW = (w-35);
        intH = (h-20);
        z.style.width = intW+"px";
        z.style.height = intH+"px";
        z.style.marginLeft = "5px";
        z.style.marginTop = "5px";
        z.style.marginBottom = "10px";
        z.style.backgroundColor = "LightGrey";
        z.style.border = "medium solid black";
        //z.innerHTML = s;//'<center>'+s+'</center>'; 
        bStart ="true"; //true=Starts onload, false=waits for click. Download will start immediatly, so true is recommended. NOTE: the AutoHide feature does not work (must be set to true otherwise it is always hidden).
        z.innerHTML = '<param name="source" value="http://channel9.msdn.com/App_Themes/default/VideoPlayer10_01_18.xap" /><param name="initParams" value="m='+mURL+',autostart='+bStart+',autohide=true,showembed=true,postid='+pID+'" /><param name="background" value="#00FFFFFF" /><a href="http://go.microsoft.com/fwlink/?LinkID=124807" style="text-decoration: none;"></a></object>'       
        b.insertBefore(z, b.firstChild);
    }else{
        alert('ctl00_MainPlaceHolder_Starter_MediaPlayer not found. Make sure Channel 9 video page is loaded. Note: this does not work on the main homepage');
    }
}

C9SLFullScreen();
//alert('End JS Code');

And here is the result: A full screen video... Enjoy!

Short .Net Reflector Training Video from Pluralsight

noreply@blogger.com (Greg Bray) — Mon, 09 Nov 2009 04:51:00 +0000

Today’s Simple-Talk newsletter had a video from http://www.pluralsight.com that talked about the basic features of Red Gate’s .NET Reflector. If you are not familiar with the .NET Reflector, I highly recommend you give it a try, as it is by far one of the best .NET tools available. This short 14 minute video walks you through most of the features of the program and shows you why it can be so useful. I’ve used .NET Reflector for system testing or when a library’s documentation is incomplete or out of date. The video doesn’t cover the Analyze feature for finding all class references and dependencies, and it doesn’t talk about all the cool plug-ins you can add, but it does provide a great introduction to a very powerful tool!

Texas Holdem Hand Equity Calculation in IronPython

noreply@blogger.com (Greg Bray) — Sun, 01 Nov 2009 23:15:00 +0000

This year I have been working in my free time to create easy to use Texas Holdem poker spreadsheets based on IronPython using Resolver One. These spreadsheets can be used to calculate Win/Tie/Loss odds, but some people like to use hand equity instead since it represents a player’s overall stake in the pot. Equity of 1.0 or 100% means that they will win the entire pot, where as equity of 0.5 (50%) or 0.25 (25%) means that they will split the pot with other players. You can again use Monte Carlo analysis to run a number of trials before all of the board cards are dealt to estimate a player’s current equity in the hand. This means that if during 4 trials I would win 2 of them and split the pot with one other player on the other 2, then my equity would be (1.0+1.0+0.5+0.5)/4 = 3/4 or 75%.

Calculating Win/Tie/Loss statistics on an independent player vs player basis is very simple, as the player’s pocket cards are evaluated against each opponent one at a time to show which opponent hands are stronger or weaker than your own. This does not take into account how the pocket cards of other opponents might change the odds, since in reality you usually play verse a table of opponents not just one opponent at a time. Changing the spreadsheet to work against the whole table would require changing the Monte Carlo simulation from a “row by row” analysis to a “table by table” analysis, where by you setup a table of opponents using a random hand for each player given the information that you have about their hand (AK, KK or QQ, Group1, etc..) and then generate a random 5 card hand based on the currently dealt cards.

There are a lot of issues that you have to watch out for in this type of Multiway Isometric Ranged Equity Calculation, but the basics are simple so long as you have a method of comparing multiple player’s hands for a given board. The C# poker library that I use is mainly focused on heads-up hands, but it only took me about an hour to whip up a method that would take a board setup and a list of player pocket hands and return the scored value and equity for each hand as well as the highest ranked hand and a list of the winners. Here’s the code:

#This function can be used to calculate equity for multiple hands and a given board
def HandEquity(ulBoard, lstulHands, bPrintResults=False):
    """Used to calculate hand equity for a given 5 card poker hand and a list of players hands. If a hand wins then it's equity is 1, a loss is 0, and a tie is 1/n where n is the number of hands that tie.
        ulBoard: A 5 card hand for the board represented as a 52 bit mask in an unsigned long. Use HoldemHand.Hand.ParseHand method to parse "Ac Kd Th 3c 2d" type strings into bitmasks.
        lstulHands: a list of 2 card pocket hand masks as unsigned longs. Invalid hands that duplicate board cards are accepted but will get None for their score and equity. Use Use HoldemHand.Hand.ParseHand("Ac Kc") or HoldemHand.PocketHands.Query("AK").ToArray() to get hand masks.
        bPrintResults: True will print the results to the console. Use False for fastest results. NOTE: change 'pass #print' or '#print' lines to 'print' to enable more detailed debug messages 
    Returns a tuple of (lstintScores, lstfEquity, intHighScore, lstHighScorers):
        lstintScores = integer representing the scored value for each hand. If hand is invalid given the board cards, value will be None.
        lstfEquity = float representing the hand's equity. If hand is invalid given the board cards, value will be None. Sum of all valid hands will equal 1.
        intHighScore = integer representing the highest scored value (ie what the winner(s) had). Use HoldemHand.Hand.DescriptionFromHandValueInternal(intHighScore) to get a string description
        lstHighScorers = list of the index to each of the winners. use len(lstHighScorers) to find out how many winners there were.
    Example:
        testBoard = ((288<< (13 * 3)) + (7<< (13 * 2))) #0b100100000000000000011100000000000000000000000000 is Ts 7s 4h 3h 2h
        testHands = [3, 3 << 13, 3 << (13*2), int('1100',2)] #3 is 0b11 for 3c 2c, "3 << 13" is 3d 2d, "3 << (13*2)" is 3h 2h, and int('1100',2) is 4c 5c
        (lstintScores, lstfEquity, intHighScore, lstHighScorers) = HandEquity(testBoard, testHands)"""
    intTotalHands = len(lstulHands)     
    if intTotalHands < 1:
        raise Exception("Error: lstulHands must have at least one hand.")
    if HoldemHand.Hand.BitCount(ulBoard) != 5:
        raise Exception("Error: ulBoard must be a 5 card hand.")
    if(bPrintResults):
        print "Start HandEquity with %d hands and board cards: %s" % (intTotalHands, HoldemHand.Hand.MaskToString(ulBoard))
    lstintScores = [None]*intTotalHands #List that holds scores for all the handss
    lstfEquity = [None]*intTotalHands #List that holds equity values for all the hands
    intHighScore = -1 #The highest score currently found
    lstHighScorers = [] #list of the index for hands that currently have the highscore
    intScore = None #temp variable to hold the current score
    
    #NOTE: Hand equity will not be available until after this loop finishes!
    for intIndex,aHand in enumerate(lstulHands):
        if HoldemHand.Hand.BitCount(aHand) != 2:
            raise Exception("Error: Pocket hands must contain 2 cards. lstulHands index %03d HandMask=0x%0.13X contains %d cards." % (intIndex, aHand, HoldemHand.Hand.BitCount(aHand)))
        if(aHand & ulBoard == 0): #Make sure there are no duplicate cards on the board and hand first
            lstintScores[intIndex] = intScore = HoldemHand.Hand.Evaluate(aHand | ulBoard) #evaluate the hand and board cards to get the score
            if intScore > intHighScore:
                intHighScore = intScore #Set new high score
                lstfEquity[intIndex] = float(1)
                for aScoreIndex in lstHighScorers: #Reset old high scores equity to zero
                    lstfEquity[aScoreIndex] = 0
                lstHighScorers = [intIndex] #Reset high score list
            elif intScore == intHighScore: #There is a tied pot, so calculate partial equity
                lstHighScorers.append(intIndex) #Add this index to the current high score
                for aScoreIndex in lstHighScorers: #Reset old high scores equity to partial equity
                    lstfEquity[aScoreIndex] = 1 / float(len(lstHighScorers))
            else:
                lstfEquity[intIndex] = 0 #losing hand gets no equity
            #print "%03d: Hand %s score is %s (%d)" % (intIndex, HoldemHand.Hand.MaskToString(aHand), HoldemHand.Hand.DescriptionFromHandValueInternal(intScore),intScore)
        else:
            pass #print "%03d: Hand %s has duplicate cards. Score and Equity are None." % (intIndex, HoldemHand.Hand.MaskToString(aHand))
    
    #Hand equity is now finished.
    if(bPrintResults): #Print results for each hand
        print "Best hand is '%s' with %d winner(s) getting %0.2f equity in the hand." % (HoldemHand.Hand.DescriptionFromHandValueInternal(intHighScore), len(lstHighScorers), 1 / float(len(lstHighScorers)))
        for intIndex,aHand in enumerate(lstulHands):
            if(lstintScores[intIndex] == None):
                print "%03d: Hand %s HandScore=None      Equity=None." % (intIndex, HoldemHand.Hand.MaskToString(aHand))                  
            else:
                print "%03d: Hand %s HandScore=%09d Equity=%0.2f." % (intIndex, HoldemHand.Hand.MaskToString(aHand), lstintScores[intIndex], lstfEquity[intIndex])  
    return (lstintScores, lstfEquity, intHighScore, lstHighScorers) #Return Results

Here is a sample print-out when bPrintResults is True:

testBoard = ((288<< (13 * 3)) + (7<< (13 * 2))) #0b100100000000000000011100000000000000000000000000 is Ts 7s 4h 3h 2h
testHands = [3, 3 << 13, 3 << (13*2), int('1100',2)] #Hands: 3c 2c, 3d 2d, 3h 2h, and 4c 5c
(lstintScores, lstfEquity, intHighScore, lstHighScorers) = HandEquity(testBoard, testHands, True)

Start HandEquity with 4 hands and board cards: Ts 7s 4h 3h 2h
Best hand is 'Two pair, Three's and Two's with a Ten for a kicker' with 2 winner(s) getting 0.50 equity in the hand.
000: Hand 3c 2c HandScore=033622016 Equity=0.50
001: Hand 3d 2d HandScore=033622016 Equity=0.50
002: Hand 3h 2h HandScore=None Equity=None
003: Hand 5c 4c HandScore=016942384 Equity=0.00

This is just a first step towards a full table Monte Carlo Simulator, but it gives you all of the details that are needed to find the winner(s) for a hand and calculate hand equity. The code used to generate a random board would be the same as in the other poker spreadsheets, but the player hand selection logic would need to use some sort of Round Robin algorithm to make sure that each player gets equal access to all of their possible pocket cards. Then you would use the HandEquity method to calculate the player’s equity for each hand and aggregate the results. My next goal is to change the spreadsheet code to use multiple threads, so I probably won’t tackle multi-way analysis anytime soon.

Windows 7 Issues with JMicron JMB36X Controller in AHCI mode

noreply@blogger.com (Greg Bray) — Fri, 02 Oct 2009 04:15:00 +0000

I had a lot of issues when setting up a RAID controller when installing Windows 7 a few months ago, and today I had more problems trying to get an IDE drive to work in the same machine. The ASUS P6T motherboard has two hard drive controllers: one Intel ICH10R SATA controller for the 6 red SATA ports, and one JMicron controller for the two orange SATA ports and the single IDE connection. When I installed an old IDE hard drive to the JMicron controller, it was recognized in Windows but I was never able to format the drive. It kept saying that the window needed to be refreshed or that I needed to reboot the machine.

After trying a few times with no progress, I decided to check and see if there was any updated drivers on the ASUS website. Turns out there was a recent BIOS update for the JMB322 firmware, so I download it to a jump drive, rebooted the computer and used the EZ Flash 2 utility built into the BIOS to run the update (I love this feature!) After changing the ICH10R settings back to RAID and setting the volume to be the primary hard drive and default boot device I was able to load back into Windows, but now there was another issue: There were two JMicron JMB36X devices listed in the device manager and only one of them was working.

I tried removing both of them, thinking that one was a duplicate, but they both were detected again and one of them always listed “This device cannot start. (Code 10)” as the error message. The BIOS settings had the JMicron controller setup as AHCI mode, but no matter what I tried I could not get it to work. I even tried the latest drivers right from the JMicron website, but still nothing.

Finally I tried changing the JMicron Controller settings from AHCI to IDE, and it started working again. This time I could detect the drive and the formatting worked just fine. Seems like the P6T motherboard still has a few bugs to work out specifically with the SATA and IDE controllers when using Windows 7 64bit edition. If I had to do it again I would probably ditch the hardware RAID array and instead buy an SSD drive for much better performance and a lot less headaches trying to set things up.

Powershell V2 training videos on Channel 9

noreply@blogger.com (Greg Bray) — Tue, 15 Sep 2009 17:21:00 +0000

If you are a Windows system administrator and are not familiar with Powershell yet, you should be! Even programmers should find a .NET based command shell useful in many situations where they want to test an assembly or use it in a simple script. I just found out that Channel 9 has a PSV2 tag for all of their Powershell V2 videos, which cover different aspects of Powershell in a short video format. These videos don't appear to have a lot of views or comments, but they are definitely good learning material and could use a bit more exposure. Check out this one on Variables, Types, and Operators:

Also if you are using Powershell, I highly recommend using the now free Powershell Analyzer for creating and editing your scripts. I think that it still runs V1 of Powershell, but it is the best IDE that I have worked with so far. The Powershell Plus that replaced it just doesn't fit my needs quite as well.

Learn WPF on a Sunday Morning!

noreply@blogger.com (Greg Bray) — Sun, 13 Sep 2009 19:08:00 +0000

I have been working on a few projects that involve WPF and Silverlight these last few weeks, and I have to say that the learning curve is a bit steeper than I expected. I have worked with Windows Forms and ASP.NET Web Forms for a while and know most of the tricks regarding layout and data binding, but WPF brings a lot of new technology and features into the mix. I had tried to jump straight into Silverlight design using Expressions Blend, but I got lost and overwhelmed in the process. This weekend I took a step back and started with learning WPF, which is the way that applications are designed in Silverlight. There are a lot of great resources, but all I needed to get moving were a few videos from the www.WindowsClient.net website. Their series on WPF showed the basics for layout and design options in a simple WPF application. Here are some good videos to get you started:

1. Creating a simple WPF application (10:19)
2. Using XAML in WFP (10:34)
3. Overview of the different layout panels in WPF (14:05)
4. Simple WPF Data Grid Editing using ListView and GridView (31:03)
5. WPF Effects (11:41)
6. Host a WPF Control in a Win32 Application (19:59)

There is plenty more to learn about WPF (Animations, Styles, Video, etc), but this is a great way to get a start using WPF.

Enjoy!

Simple Talk article about TSQL Select For XML clause

noreply@blogger.com (Greg Bray) — Wed, 02 Sep 2009 22:25:00 +0000

I had to do some work today creating a C# web service that would query a MS SQL database and return an XmlDocument with the results. After writing the query I tried loading the data into a DataTable and then using the WriteXML method to turn it into XML, but the layout wasn't exactly what I wanted and the use of a TableAdapter, StringWriter, XMLTextWriter, and XMLDocument seemed a bit excessive. I then remembered that MS SQL 2000, 2005, and 2008 support the For XML clause on Select statements. I had read an article from Simple Talk a few months ago about the different options and by adding "FOR XML RAW ('LogEntry'), ROOT ('LogEntries')" to the end of the query I was able to get the SQL server to format the data for me and then use the ExecuteXmlReader() method to create an XmlReader and load the data into an XmlDocument. The article is a great read for anyone that needs to work with XML data from an MS SQL server.

http://www.simple-talk.com/sql/learn-sql-server/using-the-for-xml-clause-to-return-query-results-as-xml/

.NET Dynamic Languages Deep Dive: Bring your scuba gear!

noreply@blogger.com (Greg Bray) — Tue, 28 Jul 2009 06:32:00 +0000

So I’ve been playing around with IronPython for a little while now and have been having a lot of fun learning to combine dynamic and static programming techniques. I haven’t tried to do anything too fancy yet, but using an IronPython powered spreadsheet to create a Texas Holdem Monte Carlo Simulator did show that you can get reasonably good performance just hacking away at a problem, even when there are a few extra levels of indirection, binding, and code generation added into the mix.

Lately I’ve been keeping my Google Reader RSS feed full of cartoons, news, and blog posts, and I just barely got around to digesting Bruno Terkaly’s break down of C# 4.0 and the Dynamic Language Runtime. He uses his blog to do presentations, so there usually are a lot of diagrams and a few key paragraphs of text, but if you have been keeping up with the developments then the pictures really help to pull things into perspective.

This post in particular was expanding on Jim Hugunin’s talk about the use of Dynamic Languages in Microsoft .NET from PDC 2008 almost a year ago. I had seen Anders Hejlsberg’s presentation about The Future of C#, but I hadn’t had a chance to watch the video about the DLR until today. If anyone is wondering why there is so much hype about the DLR, the new Dynamic type in C#/VB.NET, or the emergence of IronPython and IronRuby, I suggest taking a look at both of these videos, as they show what in my mind is the future of programming. These technologies have been picking up steam over the last year or two, but when .NET 4.0 and VS 2010 ship in the near future, I expect to see a lot more applications taking advantage of this new powerful platform.

The DLR in particular is quite fascinating, in that it bridges the gap between Dynamic and Static languages, as well as creating a method for connecting .NET code to COM, Javascript, Python, Ruby, and whatever else you want to tie into it with a custom binder. Also, with the DLR being supported in Silverlight, there are a lot of new opportunities for running managed dynamic code directly from a browser. Gestalt for instance is a newly released project hosted at the Microsoft mix online community labs, which has the goal of letting web developers replace or supplement their JavaScript code with Ruby, Python & XAML directly embedded into HTML pages using the standard <Script> tags that get parsed by Javascript and Silverlight. Very interesting stuff!