Wictor Wilén

Visual guide to Azure Access Controls Services authentication with SharePoint 2010 - part 1

Wictor Wilén — Wed, 01 Feb 2012 01:38:19 PST

A year and a half ago I posted the Visual guide to Windows Live ID authentication with SharePoint 2010 series, a post that got a tremendously amount of hits (and still gets) and tons of comments (and new ones still coming in). It showed quite a cumbersome way to Live ID enable your SharePoint 2010 Web Applications using the Microsoft Service Manager, MSM, (which works some times and some times not). Although it did/do work it is not the best way to enable Live ID authentication to your SharePoint 2010 web site. The MSM required you to first test in their INT environment and get approval before putting it into production, and you had to follow a set of guidelines on how to use Live ID logos etc etc, not mentioning all the manual configuration.

Microsoft has a service in its Windows Azure offering called Access Control Services. This is essentially an Identity Federation Provider, living in Windows Azure. This IP not only allows you to federate Live ID authentication but also Google ID, Facebook ID etc. In this post, and subsequent ones, I'll do a visual guide on how to configure SharePoint 2010 to use Windows Azure Access Control Services, ACS, to handle your authentication.

Configuring Azure ACS

First of all let's get acquainted with Windows Azure Access Control Services. But before you start you need to have a Windows Azure subscription. Unfortunately this isn't for free, but if you have an MSDN Subscription you can take advantage of the MSDN Azure benefits. Once you have your subscription you head on over to http://windows.azure.com and sign in with your Live ID. Once you're signed in you are in the Azure Management Portal.

On the left hand side is the navigation and select Service Bus, Access Control & Caching (this is the part that was/is called Azure AppFabric), #1 in the image below. This will load all the AppFabric Services and you will get a new navigation tree on the left hand side and the Ribbon menu will update. To create your Service Namespace, which is like a "container" for the AppFabric services, click on the New button in the Ribbon (#2).

This will bring up a dialog where you create your new namespace. First of all select the services you want in this namespace - for this authentication sample we only need the Access Control (#1). Secondly you need to specify a unique namespace for your Service Namespace (#2). After that select an appropriate Region (#3) and optionally a Subscription if you have multiple ones. Once your satisfied click Create Namespace (#4) to start the creation.

The creation will take a couple of minutes, so now it's a good time to take that coffee you all been waiting for! As always when dealing with AuthN, coffee breaks are good, that's what my good ol' buddy Spence always nagging about. Wait until the service namespace has the status of Active until proceeding further.

Once it has the Active Status, select your newly created Service Namespace (#1) and choose Access Control Service in the Ribbon menu (#2). This will open the Access Control Services administration. Note: you will navigate away from the Azure management and reuse the same browser window.

The Access Control Service administration site contains a lot of configuration options. You will see a left hand side navigation where you can set up everything from Identity Providers, IP, and Trusted Parties to custom certificates and get details on how to integrate this with your applications. The first thing we will do here is to add a couple of identity providers, click on Identity Providers in the menu.

Identity Providers

The Identity Providers menu option shows all the Identity Providers that currently is available for this ACS Service Namespace. By default you will only see Windows Live ID. While Windows Live ID might work (if you can live with all the guid based identities) it's very convenient to add other IP's here. Click on Add to add a new IP.

I like to add the Google Identity Provider, since all the user identities from Google IP will use the Google login id (e-mail). Select Google amongst the preconfigured IP's and then click Next.

The next page gives you some customizations options for the Google IP. Change anything you want here and click Save to continue.

This should take you back to the IP page and you should now see both Windows Live ID and Google listed there.

The Relying Party

Next thing to do is to add a Relying Party Application, that is our SharePoint Web Application. Choose Relying party applications in the left hand menu. You will have no RP's configured by default so click on Add to create a new one.

You will now see a form and it is here things must be written correctly otherwise you will not get the AuthN to work with SharePoint 2010. First of all give your RP a nice and easy name (#1). Secondly is the realm (#2), if you remember from the Live ID visual guide the Realm is important. Use a URI instead of URL, it's easier to remember and always works. In this case I choose uri:visualauthn. Then we also need to fill in the Return URL (#3). The return URL must point to http://server/_trust/default.aspx when dealing with SharePoint 2010 (of course replace with your server name, localhost also works in a test environment).

The next things to configure is the tokens. First of all SAML 1.1 must be used as Token Format (#1), SAML 2.0 is default in ACS so make sure to change this. Leave Token encryption policy to None (#2). Then finally an important piece - the Token lifetime. By default this is set to 600 seconds and you need to increase this value. The reason for that is that SharePoint 2010 has the expected token lifetime configured to 600 seconds and when SharePoint validates the token, which is after it's been issued by ACS it will fall outside the lifetime. So you have two options here lower the SharePoint lifetime or increase it in ACS, in this case I've done the latter and set it to 700 seconds.

The rest of the configuration is left intact. If you like you can uncheck Windows Live ID if you do not want Live ID users to sign in with this RP. Click Save when you're done.

You should now see your newly created Relying Party.

Some rules!

We now have two IP's connected to the RP - each of the IP's has a set of outgoing claims to our RP and we need to make sure that the claims received from the IP's to the RP are passed through to SharePoint as outgoing claims from the RP. This is done through the Rule Groups. Select Rule groups in the left hand menu. You will see a Rule Group called "Default Rule Group for Visual AuthN" - this group was automatically created for us when we created the RP: Now click on the rule group to create the actual rules.

Note that there are no rules by default. To create the default rules, just click on Generate to create them.

First we need to select for which IP's to generate the rules, make sure both Live ID and Google (in this case) are selected and click the Generate button.

Now ACS will generate the default rules for all selected IP's. Click Save to complete the rules setup.

I will in follow up posts on this one, show you how to fiddle a bit with these rules. But for now we're just using all the default settings.

Certificates

Next thing to do is to create a certificate that we will be using for Token Signing. The Management Portal makes it very easy for us to make a self signed certificate for testing and demo purposes. For production scenarios either purchase an X.509 certificate or request one from your local Certification Authority (CA) (for instance AD Certificate Services). Just make sure it's a certificate for sign and encrypt the payload. Navigate to the Certificates and Keys in the ACS Management Portal. Click on Add next to the Token Signing certificates.

First of all make sure that the correct Relying Party is selected, it should be the one you just created. ACS allows you to have multiple RP's so just make an extra check.

If you want to create a self signed certificate, take a look at the middle of that page. There is a small snippet that you just can copy and paste to create your own certificate (if you have the MakeCert utility, which is a part of the Windows SDK).

Copy and paste that into a command window or a PowerShell console. This will create your signing certificate and store in the My store on the box you run the command at.

Now you need to export this certificate into two files (with and without the private key). One to upload to ACS and one to import into SharePoint 2010 later. You can export the certificates using the Certificates MMC Snap-In (like in the previous visual Live ID guide) or use PowerShell, which will impress your colleagues more.

The PowerShell I use to export the certificate to one password protected Pfx file (with key) and a Cer file (without key) are the following:

$cert = @(dir cert: -Recurse | Where-Object { $_.subject -like "CN=visualauthn*" })[0] $type = [System.Security.Cryptography.X509Certificates.X509ContentType]::Cert $bytes = $cert.Export($type) [System.IO.File]::WriteAllBytes("c:\visualauthn.cer", $bytes) $type = [System.Security.Cryptography.X509Certificates.X509ContentType]::Pfx $pass = read-host "Password" -assecurestring $bytes = $cert.Export($type, $pass) [System.IO.File]::WriteAllBytes("c:\visualauthn.pfx", $bytes)

As you can see I grab the certificate with the correct subject and then use the .NET classes to export the certificates and finally save the bytes into files. Replace the exported filenames with your own and also the subject on the first line when doing this for your service,

The Pfx file must be uploaded to ACS. You should still be on the Add Token Signing certificate page and now choose to upload the Pfx certificate and then enter the password you used when exporting it. Make sure that you choose to use this certificate as primary certificate and then click Save.

By now we're done with ACS. Let's head on over to configuring SharePoint.

Configuring SharePoint

Now it's time for the fun stuff - SharePoint. First of all you need to have a Web Application that uses Claims Authentication. If your web app is in Classic mode, either create a new one or upgrade from Classic to Claims.

Trusted Root Authority

Now we need to make sure that the SharePoint farm trusts the certificate used by ACS to sign the tokens. This is done by uploading the other certificate file into SharePoint, using PowerShell. The following code is used to import the .cer file:

asnp microsoft.sharepoint.powershell $cert = Get-PfxCertificate "C:\visualauthn.cer" New-SPTrustedRootAuthority -Certificate $cert -Name "Visual AuthN ACS"

To verify that the certificate is imported as a trusted certificate in SharePoint, go to Central Administration > Security > Manage Trust. You should see the name of the trust there:

The Trusted Identity Provider

Next up is to add the ACS RP as a Trusted Token Issuer in SharePoint. Once again we'll do this using PowerShell. Here it is really important that you specify the exact realm as entered when you created the RP in ACS (see line 1 in POSH below). Then you also need the Sign In URL for your RP. Modify line 2 below to match the URL for your Service Namespace (bold and red). Next we define a claim mapping for the identity claim that we want to use, in this case the e-mail address. Finally we just add the new trusted identity token issuer

$realm = "uri:visualauthn" $signinurl = "https://visualauthn.accesscontrol.windows.net/v2/wsfederation” $map1 = New-SPClaimTypeMapping "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress" -IncomingClaimTypeDisplayName "Email" –SameAsIncoming New-SPTrustedIdentityTokenIssuer -Name "Visual AuthN ACS" -Description "ACS rocks!" -Realm $realm -ImportTrustCertificate $cert -ClaimsMappings $map1 -SignInUrl $signinurl -IdentifierClaim $map1.InputClaimType

Now it's time to modify our Web Application to use this ACS RP. Go to Central Administration and Web Application administration. Choose the Web Application you want to enable the RP for (remember that it has to be a Claims Web App). Choose to modify Authentication Providers from the Ribbon menu and select the correct Zone (normally Default). Then scroll down to Claims Authentication Types and check the Trusted Identity Provider checkbox and then our own ACS Trusted Identity Provider. Once that's done click Save.

One final thing here before testing it all. I personally prefer to add a User Policy on the web application directly for one of the users that will log in through the ACS RP. You can of course log in using Windows AuthN and then set permissions inside the site if you prefer so. But this is how I do it. Select the web application and then click User Policy in the Ribbon. Then click Add User choose All Zones and enter the Google e-mail of the user that you will test with, give the user Full Control on the web application. Make sure that you type the e-mail correct - SharePoint will by default validate anything that you write in in Claims mode (more on this in another post).

With this policy in place all should be set to test drive it all.

Test it!

Now all we have to do is test it. Browse to the web application for which you added the Trusted Identity Provider, once it's loaded you will be presented by the default multiple login page. The drop down will show all the available AuthN providers for the web application. To use the ACS login we only need to choose that provider in the list.

Once the provider is selected you will be redirected to the ACS RP login page. In this case we will see two possible providers to use - Live ID and Google. Click on the Google button and you will be redirected, once again...

..this time to the Google log in page which will ask you for username and password. Enter the username (e-mail) that you used when creating the user policy for the web application and log in. In this case, with Google, Google will ask for confirmation that you trust the ACS RP. Choose Allow and you'll be redirected back to the RP which will seamless redirect you to SharePoint.

...and voila! You're in! Take a look at the username in the upper right corner - it should be the e-mail address of the Google ID you used.

Summary

That was it - a visual guide on how to configure federated authentication using Windows Azure Access Control Services and SharePoint 2010. It is this easy! Even though this article was quite lengthy you can do it all in a couple of minutes (compare that to the way I previously showed)!

Enhanced Search Migration Tool for SharePoint 2010

Wictor Wilén — Mon, 30 Jan 2012 13:22:54 PST

The SharePoint Enterprise Search Migration Tool (SMT), created by Microsoft, is a great little tool for moving/migrating search settings from one SharePoint Search Service Application to another, and even from a SharePoint 2007 SSP to a SharePoint 2010 SSA or FAST for SharePoint. The tool is available for download from the MSDN Archive - both as a binary and its source code. It is a console application that creates an XML when exporting the settings and uses the same XML when importing the settings, and it works great in a scripting environment. The SMT that's available from MSDN Archive allows you to migrate Best Bets, Search Scopes and Site Collection Search settings.

Introducing the Enhanced Search Migration Tool!

Consider using this tool when moving search settings from an SSA in a staging environment to an SSA in a production environment, or from production to a standby-farm. There's some stuff missing then - it's not enough just migrating best bets, search scopes etc. In my case I had to move the Managed Properties between our environments. Ok, you can script it all using PowerShell, but I'd prefer having something that automatically could move Managed Properties from dev to test to stage to production to standby.

So I took the source for the SMT and added functionality to work with the Managed Properties. It will export the Managed Properties from one SSA including its configuration and mapped crawled properties to the same XML file as the other Search Settings, and can then be used to import the Managed Properties to another SSA.

Here's how it works:

Export Managed Properties:

SearchMigrationTool.exe -export -managedproperty managedprops.xml

Import Managed Properties:

SearchMigrationTool.exe -import -managedproperty -conflictBehavior prompt managedprops.xml

As you can see it follows the same routine as the default SMT does and supports conflict resolution. And you can of course combine the export and import with the other SMT switches such as BestBet, Scope, SearchSettings or All (which exports/imports everything).

Features

The features of this Enhanced SMT are:

Import/Export Managed Properties, including all their configuration
Only works for SharePoint 2010 to SharePoint 2010 (no 2007 and no FAST)
Only maps existing crawled properties to the managed property
It will not change the type of a Managed Property
Handles the case when an SSA is present in multiple proxy groups (bug in original SMT)

Sample usage

Here's a sample on how to use it. Assume that your developers need to add a new Managed Property. Instead of them writing feature receivers or PowerShell to create the Managed Properties they just write a snippet of XML, like this:

The admins then use the Search Migration Tool to import this new Managed Property (as a part of the update POSH or similar).

Once this is done you will see the Managed Property in your SSA.

Can't be easier?

Support for this!

There is no support for this tool at all, but I would appreciate any feedback on it. I've had it running for a while amongst my friends and received positive feedback.

Hopefully my additions will be merged into the "official" SMT - but until then download the tool from here and get your SSA's in sync!

Download

Here are the downloads:

Note! If you're going to modify/build the SMT you need to grab the SharePoint assemblies. They are not included in this package due to licensing restrictions.

Do you want to know more about the Microsoft Certified Master or Architect programs?

Wictor Wilén — Thu, 26 Jan 2012 14:03:40 PST

I bet you will!

The Advanced Certification Team at Microsoft Learning will start a new Live Meeting series where you can learn more about the Microsoft Certified Master and Microsoft Certified Architect programs. It will be regularly held meetings where they will go into details about the programs. The program managers will make you understand the program mission and vision, how to prepare for a certification, how to apply for participation and the value of the programs. If you're interested in one or more of these programs I recommend you to attend one of the live meetings or watch the recordings. Of course attending the live meetings will allow you to directly ask questions to the program managers!

There will be live meetings for each product category (SharePoint Server 2010, Lync Server 2010, Exchange 2010, SQL Server 2008 and Active Directory).

To register for one of the live meetings head on over to the Microsoft World Wide Events site and get your slot. Currently there are two planned sessions:

2012-02-10: SharePoint 2010 MCM Overview
2012-02-15: Exchange 2010 MCM Overview

Encrypted e-mails causes corrupted Crawled Properties in SharePoint 2010 Search Service Applications

Wictor Wilén — Tue, 17 Jan 2012 21:45:55 PST

For a couple of weeks (ahem, months) I've been struggling with a strange Search Service Application issue. Some time back I went to check out on some Crawled Properties when making a tool to help copying settings between SSA's (more on this tool in another post). Then I noticed that there were tons of Crawled Properties with just garbled binary data(!) as the property name.

I searched like crazy for a while to find where these came from, there was nothing in the logs of any kind related to this. I could not locate any documents related to the Crawled Properties. I could not delete them, somehow they are connected to some content (at least that's what the system says), but there were no document samples. I created a new SSA and crawled the same corpus and the same (almost) corrupted junk crawled properties appeared.

A couple of days back I finally found out where they came from. With the grace from Microsoft support I did some queries on the SSA databases and found another crawled property that was inserted at the same time as these junk properties. This property had document samples! And those samples were e-mail .msg files. And specifically it was e-mails with encrypted content! I copied these files onto a brand new farm without any content and was able to reproduce the corrupted properties.

How to reproduce the corrupted Crawled Properties

To verify that this had to do with encrypted e-mail messages, and just not the ones I found; I encrypted an e-mail, sent it to myself and exported it as an .msg file. I took this .msg file and added it to a document library in a hot new VM (yea, I only got new ones since I had to rebuild all my VM's last weekend due to a corruption issue on one of my base images). Then I fired of a full crawl, with full logging enabled, and watched the Crawled Properties of the SSA. And as expected they showed up after just a minute.

So, be careful about having encrypted e-mail messages in your farm! Or prevent the issue...

How to prevent the issue

So, how do I get rid of these corrupted properties? Unfortunately there is no good (supported) way, at the time of this writing, except deleting your SSA and create a new one and before crawling the data remove the files or create a Crawl Rule.

If you already have these corrupted properties or if you want to prevent new corrupted properties you can create a Crawl Rule that excludes .msg files. That will help the situation - but you will not be able to search the .msg files (if they are encrypted you cannot search them anyways!).

Summary

These corrupted properties does not do any harm. You cannot use them and you don't notice anything else on the SSA except that they are there and annoys you. Only thing I can think of is that if you have a lots of them, you can run into trouble. There is a limit of 500.000 crawled properties per SSA! Sounds a lot, but for two .msg files I saw about 1.600 corrupted properties!

I hope this helps someone and I hope there will be a fix for this in the future - if that happens I'll update the post.

Visual Studio 11 Developer Preview with the new SharePoint Developer tools

Wictor Wilén — Mon, 09 Jan 2012 21:25:25 PST

Visual Studio 11 Developer Preview is now available for download für alles and it does not only include the Windows 8 stuff like the previous preview did - this one contains the thing we all want - the SharePoint Developer tools.

Overall the performance of Visual Studio 11 is blazingly fast! I regret I tested it - since I will go back to 2010 tomorrow (or even tonight). They team has done a great job and included a lot of the PowerTools natively; such as the new Solution Explorer, the improved search feature etc.

But, back to the SharePoint Developer tools 11! What's new for all the kool kids! Here's a few highlights.

Content Type and List Designer

We all love our CAML, FieldRefs etc. But it isn't that productive. VS11 contains the long awaited Content Type Designer and also a very similar designer for Lists.

This is how the designer looks like when you're creating a Content Type:

When creating a list you have a great view to create Views:

Sandbox and Office 365 support

New features in the Office 365 space for Visual Studio was an easy bet. They have incorporated the things from the SharePoint Power Tools, such as better compile time support for the Sandbox etc. The best feature here is the Publish feature, which allows you to publish your package to a URL or local directory. You can now publish a SharePoint WSP directly to the sandbox in Office 365 - but you can't activate the solution from within Visual Studio.

Other improvements

There are much more improvements such as profiling support, better JavaScript debugging support etc. Read all about the new stuff here: http://msdn.microsoft.com/en-us/library/ee290856(VS.110).aspx

But...you really need to test it out. I'm really looking forward to start working with projects in VS11!

Scheduled incremental crawls suddenly stopped due to a stale Timer Service in SharePoint 2010

Wictor Wilén — Mon, 02 Jan 2012 13:36:11 PST

It is always fun to get back on site after a couple of days off work. SharePoint 2010 is like an annoying little critter, if you're not there to cuddle with it it will do the most strange things.

I currently have a support case open regarding some issues with crawled properties (I hope that will be another story to tell another day) and went into the Search Service Application admin pages in Central Admin to check some things. When poking around I noticed that the incremental crawl hasn't been run for a few days - actually it stopped working on the 31st of December last year (sounds like ages ago now :-). In this farm we have three Search Service Applications and only this one hadn't been incrementally crawled - the other two worked just fine. I fired up an incremental crawl manually and that worked, waited for the next incremental crawl to start - and it didn't. Also tried a full crawl manually - which worked fine, but the scheduled crawls never started.

I searched through the trace logs and event logs and found nothing that I didn't expect to see.

I tried to Synchronize() the Search Service Instances - which I've previously done when similar stuff happened by running some PowerShell snippets on the servers in the farm. Nothing happened!

Ok, what is actually triggering the incremental crawls then? It is triggered by timer jobs called Indexing Schedule Manager on XXX, and there is one job per server. In this case we have two servers running search and I took a look at them and noticed that one of these jobs hadn't been run since the last scheduled crawl - while the other one had been running all the time (every 5 minutes).

I tried starting the timer job both through the Central Admin web UI and through PowerShell -but it just refused to start. Now this is strange! Now I took a look at the Timer Job History on this specific server and noticed that no timer jobs had been executed on that specific machine since that time when the scheduled crawl last was executed. Thankfully the trace logs were there and I could see that just after 2:00 AM on the 31st of December the Timer Service stopped working (no references to owstimer.exe at all since then). There's no warning or error or nothing that indicates that it would have stopped working (and the sptimerv4 service was still running on the machine). The only thing in there was a warning about a Forefront conflict. Nevertheless I needed to get it working and currently don't have time to spend on why this happened (if it happens again, then let's dig deeper).

I tried to do a restart-service sptimerv4 on the failing machine, without luck, it couldn't be gracefully shut down. So I actually had to kill the owstimer.exe process and then start the timer service.

Once the timer service was killed and started everything spun up and the incremental crawled kicked in after a couple of minutes and have been working perfect since then.

Interesting thing here was that I did not see any early warning signs, errors in the logs or similar, and the only way we found out that the Timer process had gone into a stale state was that the scheduled crawl for ONE of the Search Service Applications was not running.

Summing up the year of 2011 and embracing 2012

Wictor Wilén — Mon, 26 Dec 2011 19:43:10 PST

It's that time of the year, when you're thinking about what you've done and accomplished the last twelve months. I've been writing a summary for the last five years (2006, 2007, 2008, 2009 and 2010) and I always think it's fun to look back at the year gone and do some predictions for the upcoming one.

This year has been totally crazy - I've been enjoying my work and clients/projects at Connecta and I totally love that we have such a strong team and offering. I can really feel the momentum we have in our team and projects, and nothing is stopping us now...

Writing

As usual I've been jotting down a couple of blog posts, some better than others (IMO) and unfortunately there are several that I just haven't had time to publish. I have on average 1.000 subscribers to my feed and a whole lot of other users finding my writings through the search engines - thank you all readers! These are the top ones for the last 12 months.

Interesting to see jQuery and JavaScript posts topping the list - might give us a hint on what's been popular this year... The post that still is #1 is the Fix the SharePoint DCOM 10016 error on Windows Server 2008 R2 (remember where you read this first :-).

I also finally got my book SharePoint 2010 Web Parts in Action published in April. It took over a year of writing, re-writing, testing, reviewing etc etc to get it done. And when the books finally arrived in printed form it was such a great feeling. It's a tough competition on SharePoint books nowadays, and I'm glad to see quite good sales figures and especially great reviews (if you haven't already - head on to Amazon and tell me what you think).

MVP again...

In April I was re-awarded the Microsoft Most Valuable Professional - SharePoint Server award. This is an award for your on- and offline community contributions. Being a part of the SharePoint MVP community is great and gives you a lot of amazing contacts with some great people.

The Master!

The MCM - Microsoft Certified Master - program was the highlight of the year! In April/May I attended the R8 rotation of the SharePoint 2010 MCM program which was three long and tiresome weeks with best-in-breed SharePoint training by the best of the best. I passed all three written exams and the qualification lab on first attempt and was allows to call my self a SharePoint 2010 MCM in the end of May. Since that rotation I've really felt that my SharePoint skills and confidence has improved - and I do think that also my customers feel the same. I did a post about my experiences of the MCM program a couple of months back - and if you're interested go ahead and read it.

Conferences and travels

This year has also been filled with a lot of conferences and travelling. First of all in march all MVP's met up in Redmond, at the Microsoft campus, for the annual MVP Summit. This was my first visit to Redmond and the summit and I met a lot of the people I've only met through the online world previously. I had a blast with paintball, late nights and beers talking to some really great friends. A couple of months later I went back to Redmond for the MCM - the weather was the same as in March! October and November was quite hectic with another trip over the pond to the SPC 2011, a trip to the European SharePoint Conference in Berlin and finally the Southeast Asia SharePoint Conference in Singapore. I also managed to do a couple of conferences on home turf as well.

I've already planned a couple of trips for 2012 - at least two trips to US again, one is the SPC 2012 in Las Vegas where I hope to meet a lot of you. I have also planned a trip to London in late April for the International SharePoint Conference. This will be an extraordinary conference with two really cool tracks - one IT-pro and one dev, where we'll build one big solution during three days. I promise to get back to you with more information about it as soon as it's available.

SharePoint

Yea, as usual this year has been all about SharePoint for me. I've seen the product mature, I've learnt more about it (again) than I ever could expect and I see a huge and increasing demand for the platform. If you're new in IT - bet on SharePoint!
We've also seen the birth of Office 365. I'm running both my work and personal e-mail in the hosted Exchange solution and I'm very satisfied. This fall I've spent a lot of time with SharePoint Online trying to do real work and deployments. To put in in nice words I should say that I'm more of an on-premise guy!

Predictions

In these recap posts I've done some predictions for the upcoming year. I had three last year and I was right about two of them: Windows 8 is communicated and with Hyper-V support.
I've been thinking quite hard on what predictions to have this year, trying not to reveal anything or stating anything obvious. So here are my predictions for 2012.

Silverlight will die! - yeah, Microsoft haven't put it in those words, but that's what's going to happen. We'll probably see Silverlight doing it lasts breaths on the Windows Phone 7 platform though. HTML5 and JavaScript will take over.
Second Browser World War - for a couple of years browsers have tried to strive for a common standard, and Internet Explorer has finally caught up. Since the HTML5 standard is far from standardized we will see Firefox, Chrome, Safari and IE add its own proprietary "interpretations" of the standard which will diversify the browser world once again. It's already happening, unfortunately...
Windows Phone momentum - I do think that the movement of Windows Phone 7+ will finally accelerate 2012. Great hardware, Nokia, great apps, unlimited marketing money and people getting bored of icons in a row are a few of the reasons. And I never ever had my WP7 phone crash - but for the Android devices we have in the family it's a part of the daily routine...
Less cloudy - this is more of a wish than a prediction but I do hope that 2012 will be less cloudy. I'm so fed up with the term "cloud"! Of course hosted services will still be a major option but I do think that more and more customers are going to look at internal hosting and/or hybrid solutions.

What do you think?

Thank you and a happy new year

So, I do not think I will be posting anything more this year! It's been a fantastic year but I'm sure 2012 will beat 2011 big time. Thank you all readers and followers and thank you to all new friends I've met throughout this year and thank you to everyone supporting me and has to put up with late night IM's or e-mails (you know who you are!).

Now, enjoy your last days of 2011 and have really great New Years Eve - I know I will (even though enjoy incorporates some work!).

Suddenly getting Access Denied on your SharePoint 2010 User Profile Sync

Wictor Wilén — Sat, 17 Dec 2011 15:57:32 PST

The last week I stumbled upon a really interesting new and shiny User Profile Synchronization issue - one of these things that just make your day! We had to manually initialize a full synchronization, after doing some updates to one of the user profile properties, and the user profile synchronization would not just start...

Everything looked fine (on the surface) and we tried the incremental sync, which also looked like it was starting but nothing happened. The sync service was up and running and the FIM services was started, the MIISClient showed no activity. We took a look at the timer jobs, which are responsible for kicking of the synchronizations and saw that they all failed with the error message Access Denied.

No more than this simple error message. Since the timer jobs are executed using the Farm Account this sounded very peculiar. Oh, and you who still have your Farm Account in the local administrators group would probably never see this error, you'll be aware of this in a minute!

Next resort was to dig in to the trace logs (ULS), using my favorite SharePoint tool: ULSViewer. And there we had a Critical and an Unexpected entry, related to this Access Denied error message.

The accompanying stack trace showed me that it was some problems getting the instance of the Management Agents.

Time to fire up my second best SharePoint tool: Reflector! Peeking into the failing method in the User Profile assembly revealed that the Access Denied was thrown while trying to retrieve the MA's using WMI. Now, this sounds really weird. As usual nothing has been changed in the farm (and this time I knew it for a fact) - but you should always check with the admins, which I did and no new policies or similar had been applied to any machines recently.

On the machine that hosted the sync service/FIM Services I started a Management Console and added the WMI snap-in and took a look at the Security tab for the local machine. And this is what I found:

The security settings for the MicrosoftIdentityIntegrationServer looked not just right, it more or less looked liked the default WMI security settings. (And as you can see the Administrators group is there - so that's why you who still have the farm account in the administrators group, probably never see this error...). A quick comparison with the identical staging environment showed a whole lot more permissions.

To be exact the following permissions did not exist for the environment that showed the Access Denied:

WSS_ADMIN_WPG group - Execute Methods, Provider Write, Enable Account, Remote Enable
FIMSyncOperators group - Execute Methods, Provider Write, Enable Account, Remote Enable
FIMSyncBrowse group - Execute Methods, Provider Write, Enable Account, Remote Enable
FIMSyncPasswordSet group - Execute Methods, Provider Write, Enable Account, Remote Enable

I fixed the permissions using the WMI management console, went back to Central Admin and started the synchronization manually and within a minute the synchronization was running beautifully!

I do not know about any supportability on this one - and you should NOT be doing this unless you really have to. A safer way might be to unprovision and reprovision the User Profile Synchronization Service - this should correctly set this permissions.

So, what caused this then? I've not yet found the actual source of the problem (which is frustrating to me at least), but I now how to fix it if it appears. Safe sources tell me that this might happen when you're doing a backup of the UPSS (and this is done in the correct way, which stops and starts the UPSS so no ongoing syncs are interfering with the backup). Question remains open...

Happy Christmas everyone!

SharePoint Online and External Data using JSONP

Wictor Wilén — Fri, 11 Nov 2011 05:22:23 PST

It was some time since I did a real blog post and I have been fiddling with a specific topic, which I'm going to write about, for quite some time now. I've been working an Office 365 Intranet and been doing two conferences lately where I've demonstrated Office 365 and Windows Azure integration. One of the challenges (and boy, there are many) of Office 365 and SharePoint Online are access to External Data or services. In a few blog posts I will describe how you can work around these issues using some very simple techniques. All that is to it is that you have to "think outside the box" and not always go down the traditional SharePoint way of doing things.

So let's get started! How do I in my SharePoint Online solution access remote/LOB data in a SQL Server database, web service or what not? The first and obvious candidate most think of is Business Connectivity Services - didn't they announce that in Anaheim!? Yea, it can be done and it works - but not as you (and I previously) expected. Steve Fox (MSFT) did show it at the SharePoint Conference 2011 and also wrote a blog post about it. The caveat is that you have to go through the Client Object Model to access the data in the external list, not using the Sandboxed server side API - that dog won't hunt.

I'm going to do basically the same thing as Steve did but using a slightly different technique - which I find more easy and more straightforward, considering I'm not using the asynchronous Client Object Model but instead uses standard jQuery, ~~jQuery templates~~ JsRender, JSONP and WCF.

This is how we'll build this little sample. First of all we have a database stored in SQL Azure. This database is surfaced through a WCF endpoint in a Windows Azure Web Role. This WCF service will be consumed by a JavaScript using jQuery and JSONP as transport in a Web Part (I'll use a Sandboxed Visual Web Part for this) and rendered using JsRender (not jQuery templates which I recently shown in a few sessions).

The secret sauce - JSON-P

First of all before digging into how this is actually implemented we need to sort out how we're actually going to transport data from the WCF endpoint to SharePoint Online.

The best and easiest way is to use JSON notation to transport the data; jQuery has great support for that and it's very easy to program using JSON structures in JavaScript. The only problem is that SharePoint Online and the WCF service is going to be hosted on different domains - and therefore it will be a cross-domain scripting issue. To overcome this issue JSONP (JSON with Padding) can be used. JSONP is a very clever method. Instead of returning a string from the remote domain and then being parsed by the caller, a JavaScript method is returned, which returns the JSON structure when evaluated, and this returned response is dynamically appended to the calling document as a script tag. Smart huh? Only problem is that we need to fiddle somewhat with our WCF endpoints, and that's what we'll discuss next...

Setting up the WCF endpoint

In this sample we need a database with one or more tables and then generate an ADO.NET Entity Data Model from that. That data model is then exposed through a WCF Data Service. WCF Data Services uses the OData protocol and supports JSON format but not JSONP. Fortunately I was not the first one lacking this feature and there is a simple extension you can download to accomplish this. This extension is an attribute that you add to your Data Service class and makes the service support the $format=json and $callback=? query string parameters.

Once you have downloaded and included the class into your project you decorate your DataService class with the JSONPSupportBehavior attribute and it should look something like this:

[JSONPSupportBehavior()]
[System.ServiceModel.ServiceBehavior(IncludeExceptionDetailInFaults = true)]
public class Users : DataService {
    public static void InitializeService(DataServiceConfiguration config) {
        config.SetEntitySetAccessRule("Users", EntitySetRights.AllRead);
        config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
    }
}

Now we're all set to consume this from SharePoint Online.

Consuming the JSONP enabled WCF endpoint from a Web Part

Create a new Empty Sandboxed SharePoint project and then add a Sandboxed Visual Web Part (make sure that you have the Visual Studio SharePoint Powertools installed to get this Sandbox enabled Visual Web Part). We're only going to use HTML and JavaScript for this one, so you could do the same thing using a Content Editor Web Part, SharePoint Designer or whatever you prefer.

Note: I'm using the new JsRender technique here, instead of the just recently abandoned jQuery Templates. The JsRender is not even in beta yet so the syntax might/will be changed over time. Read more about the change here.

First of all in our user control we need to import the jQuery 1.7 file. Instead of uploading it to our SharePoint site we'll use a CDN for this (which will boost your overall performance and save you one file to maintain).

<script type="text/javascript" src="https://ajax.aspnetcdn.com/ajax/jQuery/jquery-1.7.min.js"></script>/<

The JsRender.js file is not yet available on any CDN's so we have to add that JavaScript file into a new Empty Module SPI. You can deploy it to any folder or library, in this case I'm deploying it to the SiteAssets library of the site. Since we're talking about SharePoint Online here and we cannot guarantee that the file will be in a specific site this JavaScript file must be included dynamically or by using server side code to get the correct Web Url. We're doing the former one, and you will see the code in just a bit.

The next thing is to define some markup where we would like to show our data from the WCF service. We'll define a table with an id equal to "list" and add a nice default row, which just says loading (makes it a bit more nice to the user).

<h1>Users</h1>
<table>
    <tbody id='list'>
        <tr><td><img src="/_layouts/images/loadingcirclests16.gif" />Loading data.../</td></tr>
    </tbody>
</table>

After that we need to define the template that JsRender will use when applying the data to the table. This is done in a script tag using the type="text/x-jquery-tmpl" like this:

<script id="UsersTemplate" type="text/x-jquery-tmpl">
    <tr>             
        <td><b>{{=Name}}</b></td>              
        <td>{{=Company}}</td>         
    </tr>     
</script>

The final thing is to add the script that will load the data from the WCF service and then when it retrieves the response it will use the JsRender template to populate the table.

<script type="text/javascript">
    SP.SOD.executeOrDelayUntilScriptLoaded(doTheRemoteStuff, 'SP.js');
    var rootWeb;

    function doTheRemoteStuff() {
        var clientContext = new SP.ClientContext();
        var siteColl = clientContext.get_site();
        rootWeb = siteColl.get_rootWeb();
        clientContext.load(rootWeb);
        clientContext.executeQueryAsync(succeeded, failed);
    }

    function succeeded() {
        var elm = document.createElement('script');
        elm.type = 'text/javascript';
        elm.src = rootWeb.get_serverRelativeUrl() + '/SiteAssets/JsRender.js';
        document.getElementsByTagName('head')[0].appendChild(elm);

        $.getJSON("http://mycloudapp.cloudapp.net/Users.svc/Users?$filter=email ne ''&$format=json&$callback=?",
            function (data) {
                $("#list").empty();
                $("#list").html($("#UsersTemplate").render(data.d));
            });    
    }
    function failed() {
        SP.UI.Notify.addNotification('Something failed...');
    }
</script>

The script starts with using the Script-On-Demand features of SharePoint 2010 to delay execute the doTheRemoteStuff() function. Once the SP.js file is loaded the function is executed. The function will use Client Object Model to retrieve the Url to the root web, to ensure that this Web Part will work on sub webs as well. This is needed so that we dynamically can add the JsRender.js file that was added to the SiteAssets library in the root web.

Once we have the root web loaded the succeded() method is going to execute. This is where we first dynamically insert the JsRender.js file into the head element, referencing our own JsRender.js file. Once that is done we will use the jQuery getJSON method to get all users from the Data Service (in this case all users that has an e-mail address). To make the response a JSONP response it is important to add $format=json&callback=? to the query string in the getJSON method.

When the response is returned the table is cleared and the rendered data is added as inner HTML of the table. The jQuery extension method $(template).render(data) is the JsRender method that generates the HTML from the template and the JSON strucutre.

Once this project is deployed and activated in the Sandbox and the Web Part is added to a page it will first show you the loading message and then after just a second a nicely table with the data from your SQL Azure backend.

Summary

This post intends to demonstrate how to solve a very common problem in SharePoint Online - fetching data from LOB systems. I used SQL Azure and a Windows Azure Web Role hosted WCF service, but it can be any kind of JSONP supported WCF service. The holy grail in this case is to JSONP enable the WCF service.

Watch this space for a continuation of similar posts...

Recap of the European SharePoint Conference 2011

Wictor Wilén — Thu, 20 Oct 2011 03:53:30 PDT

Back home after a few days in Berlin for the European SharePoint Conference 2011. It was a great conference with good speakers and really nice attendees. It was three days full of sessions, expert panels, shoot-outs and SharePoint fun! Thanks to everyone who was there (especially those who came to my sessions :-) and the team behind the conference! And as always it great to meet up with the SharePoint MVP's, MCM's and now even MCA's!

During the attendee dinner the conference team announced the dates for the next European SharePoint Conference, which is set to the first week in December 2012. This one will not be in Berlin, but more likely in Barcelona, Paris or Copenhagen (Barcelona is the one everyone cheered for, so my guess is that the other two are out of question :-)

For this conference I did two development presentations. I've done similar sessions before but this time I changed them into be even more based on best practices and experiences from the field, since we're quite a bit into the product cycle. I got good feedback on it and in some of the sessions I attended I noticed the same approach from the speakers. For you who either didn't attend my sessions or didn't have time to write down all the code samples in your notebooks - you can find the presentations on the conference web site and you can find the code samples here:

I'm planning (note planning - no promises) on doing a couple of follow up posts on specifics in the demos.

Now it's only two more conferences this year; Singapore and Stockholm...

SharePoint Conference 2011 wrap-up

Wictor Wilén — Mon, 10 Oct 2011 23:39:53 PDT

I've now been home for about 48 hours since visiting Anaheim, California, for this years edition of the Microsoft SharePoint Conference. It has been a great week in California with colleagues, friends, clients and new acquaintances.

This year, eight people from my company, Connecta, travelled over for the conference. We all had a blast with some spare time before and after the conference, which included a visit to Six Flags - Magic Mountain and a drive along the Pacific coast. We also met up with a few old friends and colleagues and had a good time with our clients, who also attended the conference.

Even though nothing new actually was shared during the conference (saving all the fireworks for next year in Las Vegas) it was a really great conference. I focused on the sessions where I knew the speakers would put on a great show, which I think was a clever choice - you can never experience that from the recordings or by downloading the decks. The keynote was pretty dull and just a big marketing show. The only highlights was the big fail-over demo and the MCA announcement. The MCA - Microsoft Certified Architect - is now available for SharePoint 2010 as well, read more on the Masters blog. Congrats to Spence and Kimmo - our two very first MCA's!

I got a lot of questions regarding the MCA program and two essential things were not mentioned in the keynote. First; MCM is a pre-requisite for the MCA and secondly; you need to be MCM on the same version as you're trying for MCA on.

On Wednesday I sat in the Perficient booth for the Meet the Masters panel together with three MCM colleagues. I had a great time talking about my MCM experience and asking questions from the audience. It's great to see the that the recognition of the MCM program is increasing.

SharePoint conferences are more than just attending sessions! Compared to a lot of other technical conferences the SharePoint conferences is very much about the social activities. There are multiple vendor activities and community activities going on 24/7, ranging from SharePints to early morning runs. The Monday night I had four parallel parties going on! These activities are in my opinion one of the most important aspects of the SharePoint conferences. You meet your old friends and make new ones and broaden your network. You'll learn who's an expert in what which makes it more easier for you to get help or reach out when in doubt. Know what you know and know what you don't know - and never confuse those two!

Now I'm looking forward to next week with a conference in Berlin, then Singapore a couple of weeks later...and of course - Las Vegas next year!

Changes in the SharePoint 2010 Cumulative Update packaging since August 2011

Wictor Wilén — Wed, 07 Sep 2011 17:34:52 PDT

A couple of days ago the SharePoint 2010 Cumulative Update for August 2011 was released. Always a good time to see some things fixed and some things break. Installing a Cumulative Update is always a risky business, and you should only install them if you any experience problems that the CU resolves and only when you thoroughly tested it.

One CU to rule them all!

Without going into details about the content and fixes in the August 2011 CU there is one other thing that is of real interest - and that is how MIcrosoft has changed the packaging process for the Cumulative Updates. Since the release of SharePoint 2010 there has been talk and discussions about über-packages. That is Server Cumulative Packages that contains both the Server hotfixes as well as the Foundation hotfixes, which makes the update process a bit easier and faster - you only need to install one package. Up until now there has been mixed messages on the über-packages and caused some confusion. The recommendation has been that you need to install both SPF and SPS CU's on a SPS installation and all three if you're using Project Server.

Since yesterday (that's when I noticed it at least) the SharePoint Updates page on MSDN has been updated with new information regarding this packaging with this:

The packaging of cumulative updates changed as of August 31, 2011. [...] As a result of the new packaging, it is no longer necessary to install the SharePoint Foundation cumulative update and then install the SharePoint Server cumulative update.

It looks like we have a new guidance in town!This has been "promised" before and I hope that it will work as planned this time, only time will tell. For more history and information read Spencer Harbars post on the SP1 and June CU discussion.

What this essentially comes down to is that you only need to download one of these patches (links to August 2011 CU within parenthesis) from now on...

SharePoint Foundation 2010, contains SPF CU (KB253050)
SharePoint Server 2010, contains SPF + SPS CU (KB253048)
SharePoint Server 2010 with Project Server 2010, contains SPF + SPS + Project Server CU (KB253049)

This will save you a couple of minutes/hours downloading and you'll save a few minutes installing multiple CU's on each server.

Note: The correct version of this CU should be 14.0.6109.5002, if you were really quick you might have downloaded the 14.0.6109.5000 build.

But...there's always a but. Please, please, please test all CU's, SP's, patches, hotfixes, shoes and whatever very thoroughly!

Conference and presentation season - fall 2011

Wictor Wilén — Wed, 31 Aug 2011 21:58:33 PDT

This fall is going to be pretty busy in terms of conferences and presentations and I'll have my fair share. Here's what I've planned for this fall, so far.

Webinar: No Farm Solution in sight!

On Tuesday the 6th of September I will do a webinar discussing SharePoint Online and Office 365 and how you can build solutions using SharePoint Online, Silverlight, Windows Azure and more without creating any farm solutions:

When and where: 6th of September at 11:00 CET. The interwebz

SharePoint in the cloud!? Does it work?

This is a breakfast seminar that my company Connecta is arranging with Cornerstone where I will present Office 365 and SharePoint Online specifically. I will show you how SharePoint Online works and what you need to do to get it to work, and I will also share our experiences moving from an on-premise solution with SharePoint and Exchange to the Office 365 platform.

When and where: 16th of September at 8:00 or 13:00. Stockholm at the Connecta office

More information at http://cornerstone.se/sv/Event/Frukostseminarium-SharePoint-i-molnet-Funkar-det/

European SharePoint Conference, Berlin

For the European SharePoint Conference I will have two sessions, which I already written about.

When and where: 17th - 20th October. Berlin, Germany

More information: http://www.sharepointeurope.com/

SharePoint Conference Southeast Asia, Singapore

For the second year I will go down to the beautiful Singapore for the SharePoint Conference in Southeast Asia. This year I will also have two sessions.

Mastering customizations of the SharePoint 2010 Ribbon Menu
Enhancing Office 365/SharePoint Online using Windows Azure

I'm sure that the conference will be as great as last year!

When and where: 8-9 November. Singapore.

More information here: http://www.sharepointconference.asia/

SharePoint and Exchange Forum 2011, Stockholm

The SharePoint and Exchange Forum is the SharePoint and Exchange conference in Scandinavia and of course if you're in the vicinity you should head on over to Stockholm. This year Göran Husman and the crew has managed to get a really good line of presenters - it will be a blast! I will present one session at this conference, and it will not be a development topic, but instead a session about Service Application Federation - the good, the bad and the ugly!

When and where: 14-15 November. Stockholm, Sweden

More information here: http://www.seforum.se/

I hope to see you on the road somewhere!

Yet another object to dispose correctly in SharePoint 2010 - SPUserSolution

Wictor Wilén — Wed, 24 Aug 2011 16:37:15 PDT

If you've been in the SharePoint business for a while (at least a couple of days) you should be aware of the SharePoint objects that needs to be properly disposed; SPSite and SPWeb in particular. Objects that need disposal inherits from the IDisposable interface and requires that you call the Dispose() method when you're done with the object - this is to ensure that the object frees up resources that the .NET managed garbage collector cannot free up automatically. This includes objects such as non-managed SQL connections, resource handles, file handles etc. Disposing objects is nothing unique for SharePoint - all (real) .NET developers know how to dispose of a SQL connection. You can read more about the best practices around disposing SharePoint objects in the MSDN Disposing Objects article. Not doing this properly will eventually lead to application crashes, high memory usage and/or bad performance.

The SPUserSolution object requires disposal

If you are working with sandboxed or user solutions in code you've probably used the SPUserSolution object. This is another of these objects that needs your disposal-attention. A solution package (farm or user code) is essentially a cabinet file (compressed file) containing all the solution artifacts and assemblies. When activating a user code solution programmatically in the solution gallery you will get this SPUserSolution object which internally will contain a number of stream objects (also inheriting from IDisposable) and a non-managed file handle. These object must be disposed manually to avoid any resource or memory leaks. Here is a proper way to add and activate a user code solution

byte[] bytes = ...;
// grab the solution gallery
SPDocumentLibrary solutions = (SPDocumentLibrary)newSite.GetCatalog(SPListTemplateType.SolutionCatalog);
// add the solution to the gallery
SPFile solutionFile = solutions.RootFolder.Files.Add("mysolution.wsp", bytes);
// activate Solution
using (SPUserSolution solution = newSite.Solutions.Add(solutionFile.Item.ID)) {
    // do your thing...
}

Summary

This was just a sample and a reminder that you need to be aware of what's happening when working with some of the SharePoint objects - make sure that you do your "Disposal dance" the correct way. The list goes long on what objects need to be disposed correctly - just check your objects if they inherit from IDisposable using Visual Studio or Reflector. Not all objects do any specific disposing though and some can be ignored - but better be safe than sorry.

Tip: products such as CodeRush from DevExpress has some great built-in tools to directly identify objects inheriting from IDisposable and helps you make sure that they are properly disposed.

I'm on the SharePoint Pod Show talking about Web Parts

Wictor Wilén — Mon, 15 Aug 2011 03:45:01 PDT

The 65th SharePoint Pod Show is out featuring...tada...me :-)

The SharePoint Pod Show is THE podcast about SharePoint and is done by Rob Foster, Nick Swan and Brett Lonsdale and has featured a lot of great SharePointers from all around the world throughout the years. If you haven't already listened to the podcasts, then you got 65 episodes to catch up on! There are some epic ones, such as my favorite one #50 - which is about performance tuning. And make sure that you subscribe - you don't want to miss their SPC11 Road-trip...

This episode was recorded at the MVP Summit earlier this year in Redmond between two sessions. It was my first time meeting most of the MVP's in real life and Rob was the one interviewing me. He's one heck of an interviewer asking questions like an ice-hockey radio commentator in some weird southern accent... It was great fun and we discussed SharePoint development and Web Parts development in particular. We talked through how to get started with SharePoint development, how to build Web Parts and why I wrote my book (SharePoint 2010 Web Parts in Action).

Until next time.