Servé's SharePoint Blog

Authentication setting: where are you?

noreply@blogger.com (Servé Hermans) — Wed, 18 Jan 2012 10:25:00 +0000

When you are planning to consume data from all kinds of data sources you need to understand why, where and what to specify for authentication settings. The authentication landscape is fairly complex even from an end user perspective. There are many places where you can specify or create connections and finding them is not always easy: rich office clients, authoring tools, browser, service applications, secure store, BCS, external content types and web applications all play a part in the outcome of the mapped set of credentials which is accessing the data source.

Why is this important?

Users do not care what you have specified on the SharePoint side. They already have this nice Excel sheet connected to a SQL database and they have heard that you can render it on a SharePoint site as well. But somehow it doesn’t seem to work. What will you tell your end users? How are you going to educate them?
To get a better understanding of authentication settings we need to make a difference between getting data from within client applications and from within a browser on a SharePoint site or through a SharePoint Service.
Let us focus on the client side.

End user tools

The end user has a number of client tools available which are capable of fetching data from data sources:

Rich office clients

Word
Outlook
SharePoint Workspace
Excel
InfoPath

BI Authoring tools

Excel with or without PowerPivot for Excel
Report Builder 3
PerformancePoint Dashboard Designer
SharePoint Designer 2010
Visio 2010
Business Intelligence Developer Studio

Data connections

Secondly, there are various ways of defining external sources:

Embedded data connections

Like in Excel you create them through the Ribbon

External data connections

.odc or .udc files you have saved to a SharePoint data connection library

External content types

Running on the client through the BDC runtime

Data sources

Finally, there are various data sources to fetch data from. To name a few:

Databases
WCF services
Web services
Custom data sources
.Net Connectivity Assemblies

External content types

Let us focus a bit on External Content types. If you need to get data from an external source you may want to use External content types in case you would like to reuse the data in various places or need basic CRUD actions on the data itself. A tool like SharePoint Designer 2010 may be used to define these External Content Types. I will not describe the various steps you need to perform but instead focus on the different authentication settings.
When configuring External content types the authentication mode is selected. Each authentication mode represents a different way of processing incoming user credentials and mapping them to a set of credentials for the external content type.
To simplify matters the following list shows you how the various settings map to what:

Mapped to Impersonated Identity

Digest credentials
External Credentials
RDBCredentials
Windows credentials

Mapped to Users Identity

Pass through

Mapped to BDC Identity

Revert to self

Impersonated Identity

The first four settings all map to the impersonated identity. It basically means that the user credentials will be replaced by another credential set. And that is where the Secure Store plays its part. The Secure Store allows you to define so called target applications.
A target application defines a mapping between a (group of) member(s) to a credential set. The credential set can be a Windows Identity or an External Identity. A member can be the user account or for instance an active directory group or in some cases the application pool account of a SharePoint service.

Pass through

Meaning that the users identity will be used to access the data source. In case the data source is more than one server hop away from the user, this will fail as the user identity cannot be delegated to the data source. Another option would be for instance to use Kerberos authentication.

Revert to self

It means that the user identity will be replaced by the BDC identity which is the identity the BDC runtime is running on the client side. In browser based scenarios this would often result in the application pool id of the BCS service application.
What we just described are the client side authentication settings for External content types. These content types may surface in for instance Outlook.
In most BI authoring scenarios however, you are not using External Content Types. Instead you are connecting your client tools to data sources like SQL Server Analysis cubes, SQL databases or Web Services.
It greatly depends which tool you are using, which service and if you are doing this client side only or are planning to use it on SharePoint sites as well.

Embedded vs. external connection files

In all those cases your data connection can be defined as embedded or as an external file. What does that mean? Embedded means that you are using the various client applications to define data sources which are saved as part of your file. The file could be Excel or perhaps Visio etc. External means that the file pulls the data connection info from another file, called the ODC or UDC file. Often there are various ways of “externalizing” existing embedded connections to external ones. Obviously, the benefit of external ones are reusability amongst others.

Classified Intel: Excel Services

Take for instance Excel. If you connect your Excel sheet to an external data source you can do so through the Ribbon -> Data -> Get External Data

If you connect to a SQL database for instance, a wizard will guide you through the configuration steps. Generally it means specifying a server name, a database name, a table and that’s it. The most common mistake: although this works for you in the client tool, it most likely won’t work in the browser.

If you render the same sheet with Excel services you would need to specific the settings through the button you can see in the Screenshot: Authentication settings. I will describe how that works later on, but keep in mind that this is the easiest example!
It already becomes different when using for instance Excel with PowerPivot. And if you would be using InfoPath than specifying these settings will keep you looking as there is no such button and you need to export the connection file, modify the XML and so forth.
For now we are talking client side only and it is important to remember that this kind of setting is called embedded.
Now, it may be that you are consolidating connection settings into a SharePoint connection library. In that case you can export the connection we just made in Excel by again using the Ribbon -> Data -> Connections -> select your connection -> Properties -> Select Definition Tab -> Choose button Export Connection File.
You can store the connection file in a SharePoint library for use. Later on you will see that you need to trust connection libraries to be used by SharePoint. But for now, we are ok with that.

Prepare for lift off: PowerPivot for Excel

If you have installed PowerPivot for Excel than you will notice that the PowerPivot window has its own Get External Data part in the Ribbon:

If you would like to pull info from Analysis Services for instance you will be prompted for authentication settings again. Keep in mind that this is authentication against the cube not the database.

Now this is information that will be used when rendering it on the client. If you would like to publish it to SharePoint and would like to use PowerPivot for SharePoint to render the pivottable, then this information may or may not be used. Instead you would need to define authentication settings through the browser:

Data refresh for PowerPivot

Credentials

Data sources

Refresh on behalf of the user

As you can see in the screenshots it basically tells SharePoint that whenever the user clicks on the refresh button in the browser, which authentication setting should be used: the account which has been specified as part of the PowerPivot for SharePoint Service Application. Or a predefined Windows user credential or the credentials specified by a target application ID. As I have explained earlier the target application ID (SSS ID) is a mapping between members and a set of credentials. If you are using an SSS ID here, PowerPivot for SharePoint will look up that application in the Secure Store and next will look for a member which matches the current user. If it finds that current user as a member it will map it to a different set of credentials.

Refresh on behalf of SharePoint

The other section is quite similar to the previous one and will be used by SharePoint to refresh data according to a schedule. The difference is that the first setting is referring to the credentials contained in the workbook. Meaning that in this case the settings you have defined as a user in the Excel workbook – PowerPivot window, will be used.

On the server side it is different

If you have made it this far you are probably interested in how things are configured on the SharePoint site. The answer is: it depends on the Service Application you are using. Still, most authentication settings and service applications share a similar model:

Connection settings

NONE: If a server side connection setting specifies none then this means that SharePoint will use the identity of the SA. In most cases this is the Application pool identity the Service App is running in. The important thing here is that you can specify which SA’s use which target application of the Secure Store in those cases where none is specified. The general idea is to map the application pool account to an unattended service account which will have access on the data source. You don’t want to add the application pool identities to the data sources. Instead you create a target application for each Service Application where the application pool identity is the member and the set credentials are the so called unattended service account. You will add the unattended service account (read account) access to your data source. This is a very useful scenario and often used for BI solutions. In most BI solutions you don’t want to query complex cubes on a per user identity. It would definitely require large amounts of memory on the SharePoint server as each query result will be cached for each user. In those cases you are using the unattended service account which will greatly reduce the cache and improve performance.
Windows credentials: again, if you have Kerberos enabled this may be useful for you but otherwise you might find yourself facing a double hop scenario and windows cannot pass your credentials a second time.
SSS ID: a scenario where you actually know who has access to the data source, for instance if you have a special BI group of readers defined. In that case you create a target application at the Secure Store and add the AD group as a member and a set of credentials as the account which actually has access to the data source. Very useful for all kinds of locked down situations.

Complex?

Do you get it? There are many places to define authentication and you have to carefully plan those when rolling out advanced features. Below you will find an example. You will be surprised!
Summarized example for Excel and PowerPivot:
The example below will show you where you can specific authentication settings. Together with the explanation I have given you in this blog article you should be able to understand which authentication setting is applicable when. The example also shows you Excel and PowerPivot only. For other client tools and service applications it is different but at the same time similar patterns are applicable.

Excel:

Data tab in the Ribbon

Client side settings

Windows credentials
Database account
Excel services settings

None: no authentication (meaning that the Excel Services unattended target application ID will be used)
Windows authentication (useful when using Kerberos)
SSS ID (use the mapping as defined by the target application within the Secure Store)

PowerPivot for Excel

Data tab in the Ribbon

Client side settings

Windows credentials
Database account

SharePoint

Excel services Service App

Unattended target application ID

PowerPivot for SharePoint Service App

Unattended target application ID (so called, as specified by administrator)

Excel sheet

PowerPivot refresh schedule

On user behalf

As specified by admin
Specific windows credentials
SSS ID

Automatic according to schedule

As specified by workbook
Specific user account
SSS ID

Mini debriefing

What about the other Service Applications?The pattern for most other Service Applications is the same:

Either you are directly connecting to the data source: for instance from the client tools like Excel
Or you are connecting through an External Content Type
Or a SharePoint service is connecting for you either by passing the credentials, using a specific target application or by falling back to a so called unattended target application. In those cases the Secure Store determines the target credentials which will be used to access the data source.

What about the other Client applications?Keep in mind that Client applications like Excel and InfoPath may offer the same functionality for specifying the authentication settings but in different places. Or sometimes completely hidden for the end user. Take for instance InfoPath. You are able to define a data connection but no SSS ID. In that case you need to export the connection to an external file, edit it with notepad, uncomment the lines about authentication and voila you are able to enter a SSS ID.
It is almost impossible to completely cover all differences at once but patterns are similar and MSDN will help as well.

Could not load file from input library: Word Automation

noreply@blogger.com (Servé Hermans) — Thu, 01 Dec 2011 17:30:00 +0000

Today I was struggling a little bit with Word Automation Services. I am trying to test Word Automation services and see how it affects memory and cpu utilization. For that purpose I have created a shared documents library on a team site and pulled a few lines of C# code to create a conversionjob from the Web like you can see below. I extended the script to output all Errorcodes and Errormessage from the so called FailedItems which is a collection you can access through the Status object.

The error
During testing I received an ErrorCode 6 and the ErrorMessage:

Could not load file from the input library.

According to this link: http://technet.microsoft.com/en-us/library/ff808318.aspx#FailedConversion there are various ways of troubleshooting issues but the one I just mentioned is not described.
After some more web searches I discovered an article which states that Word automation uses one core for each active conversion job. The default setting is 1 and you need to make sure that you have at least one free core available. Well, that didn’t seem the issue. More interesting is the fact that the article states that if Word Automation Services is not able to complete your request like in situations where your server is too busy, it will yield the same error message but with a different Error code (3).

The root cause
In the end I discovered by using the ULS viewer and by running the Word automation services on just one server that the application pool account which is running the Word Automation services, did not have access rights.

The Solution
Run SharePoint Management Shell with the following:

$w = Get-SPWebApplication –Identity http://yourwebapplicationurl
$w.GrantAccessToProcessIdentity(“domain\yourapppoolaccountrunnigWordAutomation”)

If you are not sure which account is running Word automation then I would suggest to go to:

Central Admin –> Application Management –> Manage Service Applications
Manage Service Applications –> Select Word Automation Services and click Properties in the Ribbon
Write down the Application pool
Browse to Security –> Configure Service Accounts
Select from the drop down the Service Application pool you have written down previously
Write down the Account that shows beneath the drop down

AlternateCSSUrl + Save site to template = AlternateHeader issue

noreply@blogger.com (Servé Hermans) — Fri, 25 Nov 2011 11:37:00 +0000

Last week I was experiencing broken foundation pages or malformed top navigation bars. In my case this was related to having a custom (aka Saved) site template and the AlternateCSSUrl setting.

The AlternateCSSUrl setting
Most companies have a specific corporate identity and as a SharePoint designer they ask you to change the look and feel of the teamsites to adhere to that identity. In some cases people use themes to implement the color scheme and main font family. Something which is very useful as it also creates the gradients for you. In other cases this might not be enough and additional CSS is required to change for instance the position, background images, line spacing etcetera.

There are various ways to attach your custom CSS to the page. Some people are activating the Publishing Infrastructure and Publishing features. This results in a Masterpage link on the Site settings page. On that page you are able to enter an Alternate CSS link. That link will be automatically added to every page by the CSS link control which is part of the default master page. It will be added after your (themed) Core css files.

But what if you are not using the publishing features? In those cases I would definitely not recommend editing the master page. There are two options:

- Build a FeatureActivated event receiver which will set the AlternateCSSUrl property and apply the theme. Do not forget to implement the FeatureDeactivating feature as well.

- Build a custom action which is a scriptlink and add a scriptblock which will include the CSS in the header using javascript for instance.

Personally I like the first option most and use the second one for including for instance CSS that is required by JQueryUI. The disadvantage of the second one is that it will be injected, which might be noticeable.

If you are implementing the first method then you will need an event receiver. The Event Receiver would look like:

Save Site as template
Assume that you have created a new site using an out of the box Teamsite template and the solution that applies a new theme and sets an alternateCSSUrl using a feature event receiver and feature stapling. Everything looks fine and you start configuring your site. At some point you would like to save the current site as a template in order to create subsites using that template.

You browse to site settings and select the option Save Site to template. This link will be hidden as soon as you are using the publishing features and it is not recommended to save templates of publishing sites.

The saved template will be stored in the Solutions gallery of the site.

Now create a subsite using the newly saved site template. Browse to Site Actions -> New site -> select your custom site template, enter a subsite name, URL name and press OK. The subsite will be created and once that process finishes you will be redirected to the homepage of the newly created subsite.

The issue
If you have followed the steps as described above you have a newly created subsite. Try to browse to the Site settings page. You may experience one of the following symptoms:

- the header shows but nothing beneath the top navigation

- you see a site settings page but there CSS code is displayed in the top navigation bar

- you encounter an unexpected error (most likely)

What has happened?
If you try to render for instance the Site Settings page, the page will try to render the top navigation. As the Site Settings page is a foundation page, the top navigation is different than other pages from the _layouts directory which are related to the publishing features like Masterpage or Welcome page links.

The Settings.aspx page uses the v4.master unless you have specified otherwise.

The v4.master page defines a contentplaceholder for the top navigation. Inside the placeholder you will see a ASP:Menu control for rendering the top navigation. However, the contents of the placeholder control will be overridden by the settings.aspx page.
In the settings.aspx page you will see that instead of the ASP:Menu control an user control will be used called LinkSection.ascx. This control you can find in the ControlTemplates folder on each SharePoint server.

The LinkSection control checks a web property called AlternateHeader. According to MSDN the AlternateHeader can be used to define an ASPX page to render an alternative top navigation.

When we check the AlternateHeader property of our newly, malformed subsite, we will notice that it has been provisioned with our AlternateCSSUrl value!

Well, that is certainly NOT what we want and it seems to be a bug.

The solution
Try to set the AlternateHeader back to the null value using PowerShell and you will discover that the subsite works perfectly, including the custom CSS and theme like on the root site:

$w = Get-SPWeb -Identity http://urltoyour/subsite
$w.AlternateHeader = $null

Note:
It seems that the issue has been solved by a hotfix but as most companies only apply Service Packs and no CU’s unless business critical, I would definitely recommend to clear the AlternateHeader yourself.

Slide-deck: Business Continuity for SharePoint 2010

noreply@blogger.com (Servé Hermans) — Tue, 01 Nov 2011 18:55:00 +0000

A while ago I was presenting "Business Continuity for SharePoint 2010" at the SharePoint black belts for architects, organized by Ton Stegeman.

When we talk about Business Continuity many people consider the subject boring and drop it as quickly as possible. And to be honest, at times it can be boring. It is not as fancy as some dazzling JQuery magic or Silverlight stuff. But to be honest again, architects cannot just ignore it. They have to think things through from multiple angles.

To help you with that I have created one schematic overview which outlines the various aspects of Business Continuity.

The thing that I want to emphasize is that Business Continuity is a continuous process of health management, monitoring, governance and changing business requirements. These aspects really drive you planning for high availability, disaster recovery and backup & restore strategies.

Hopefully this presentation will show you things you didn't think about before. If you think something is missing please let me know and I will add it.

Download instructions:

Click the download link: Business Continuity.pptx
When the presentation opens it Google docs, click the Download button at the top right to download the PPTX which looks a lot better than on Google Docs.

Unable to open workbook several issues

noreply@blogger.com (Servé Hermans) — Sun, 30 Oct 2011 21:04:00 +0000

Symptom:
If a user clicks on an Excel worksheet on a SharePoint 2010 site, SharePoint tries to open and render the file in the browser first (depends on settings). The user receives the message "Unable to open workbook".

Cause:
Services and Web Applications in the farm are configured upon start to use an account. For Excel Services these are linked to an application pool. If the configured account does not have the required permissions on the content databases, the service won't be able to open the workbook.

ULS log:
On the application server that is running Excel Services (if you have multiple, check them all) run ULSViewer. You might be seeing messages that contain: There is a compatibility range mismatch between the Web server and database. In the custom columsn you might be seeing: GenericFileOpenError.

Follow these steps to resolve the issue:
- Lookup your Excel Services account
- In Central Administration -> Security -> Configure Service Accounts
- At the dropdown select 'Service Application Pool - Excel Services Application'
- Write down the account shown at the bottom (Select an account for this component)
- On the application server start -> all programs -> SharePoint 2010 products -> SharePoint 2010 management shell
- (account that is performing the action has SecurityAdmin role and DBCreator role in SQL Server)
- At the command prompt of the management shell type
- $myWebApp = Get-SPWebApplication -Identity
- $myWebApp.GrantAccessToProcessIdentity("")
- Check if the security settings are applied correctly if you have access to the SQL Server
- On the SQL Server connect with SQL Server Enterprise Manager to the SharePoint instance
- Under Security -> Logins ->
- Verify the Excel Services account is listed
- Right-click the properties of the Excel Services Account
- Verify if the Excel Services account has a user mapping to the
- Content databases as DBO_OWNER
- Account is assigned to the WSS_CONTENT_APPLICATION_POOLS roles
- Restart Excel Services
- In Central Administration -> System settings -> Manage Services on server
- Select the App server(s) on which Excel Services is running
- Stop the service and start the service again

If you solve the issue, users might still report that they are unable to open the workbook. However, the message might be different this time:

- Workbook cannot be opened because it exceeds the maximum workbook size
- This is a settings of the Excel Services Application. You are able to change it through Central Admin. By default this settings is 10MB. The bad thing is that the Excel client will not be launched automatically. So the user is at a loss what to do.
- Workbook contains protected sheet or cell and cannot be opened. This is by design and the Excel client application will automatically open.

Workarounds:
- Tell the user to not click on an Excel link directly but instead use the Ribbon or contextual menu to Edit the workbook in Excel
- Change the settings of the document library so that the files will be opened in the Client by default. In the ribbon select -> Library -> Library settings -> Advanced settings -> Open files in client

Related issues:
- Similar issues may happen for other service applications like Visio services, Performance Point services etc

Related resources:
Accounts and permissions: http://technet.microsoft.com/en-us/library/cc678863.aspx

Build your own mailing engine using Powershell

noreply@blogger.com (Servé Hermans) — Sun, 16 Oct 2011 19:01:00 +0000

In this post I will show you a couple of useful Powershell tricks like how to resolve a name in Active Directory, how to send a mail using an external rich HTML file and how to store results in unique log files.

It is not too difficult to find examples on how to send a mail using Powershell. The problem that I faced however was that I wanted to use an external file for the body, use rich HTML and plumb some variables into the body of the mail which I could replace at runtime and add some people in the CC.
This is particular useful for sending e-mail in batch and in fact, I am using it by looping through a list of recipients and URLs using a SharePoint list.

In this post I will show you a couple of snippets (the rest you will have to create yourself). Let's start!

First of all you need a couple of variables like:

          $server = "[put your smtp relay server here]"
          $client = new-object System.Net.Mail.SmtpClient $server
          $subject = "Weekly news"
          $sender = "Fabrikam@mycompany.com"
          $recipient = "[List of comma separated e-mail addresses]"
          $recipientCC = "[List of comma separated e-mail addresses]"

The recipient list should be build carefully. If it contains names which cannot be resolved or has for instance two commas instead of one, or contains a ; instead of a , it will fail utterly.

Secondly, we need to load the body of the text. I have stored the body in a separate file in the same directory as the script called body.txt. This file contains HTML including a tag called [URLHERE] which I will replace dynamically with a URL I pull from a SharePoint list. Here we go:

          $body = (gc .\body.txt | out-string)
          $url = "[put your dynamically loaded URL here]"
          $body = $body -replace "[URLHERE]" , $url

If you are doing mailings, this plumbing might be useful and of course the variable $url would be different on each iteration so you have to create that yourself.

Let us create the message, add the CC people and also make sure the HTML of the body will be interpreted correctly:

          $msg = new-object System.Net.Mail.MailMessage $sender, $recipient, $subject, $body
          $msg.CC.Add($recipientCC)
          $msg.IsBodyHTML = $true

And send the message!

          $client.Send($msg)

Now, if you want to add logging consider to write message to a unique file:

          $results = "LoggingMyScript{0:yyyyMMdd-HHmm}.txt" -f (Get-Date)

By using logging messages like:

          write-output "[Debug message here]" | out-file $results -append

Finally, if you are looping through user display names like: Lastname, Initial (Firstname), those will not be resolved automatically to an e-mail address. Somewhere on a blog (sorry, can't remember) I discovered the following piece of script which does the job for you:

          $ads = New-Object System.DirectoryServices.DirectorySearcher([ADSI]'')
          $ads.filter = "(&(objectClass=Person)(displayName=$displayName))"
          $s = $ads.FindOne()
          if ($S -ne $null)
          {
                   $mail = $s.GetDirectoryEntry().mail
          }

There you go. Build your own mailing engine using Powershell! Use your imagination to connect to a SharePoint list and add all of the error handling and so forth. Have fun!

GUID should contain 32 characters

noreply@blogger.com (Servé Hermans) — Sun, 16 Oct 2011 18:49:00 +0000

One a migrated site a user may encounter an unexpected error when clicking on a Add a new document link. When viewing the ULS log on the web server you might see the following error:

System.FormatException: Guid should contain 32 digits with 4 dashes (xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)

Do you notice the difference in the ' Add document' and the 'Add new link' styles in the screenshot above?

To resolve this: delete the Listview web part from the page and add it again. You will notice that the newly added Listview web part has a different link. It now displays Add document with a plus icon in front of it. This looks different than the migrated links from other listview web parts as you can see in the screenshot.

Database too old and needs upgrade

noreply@blogger.com (Servé Hermans) — Sun, 09 Oct 2011 19:05:00 +0000

If you are migrating to SP2010 you might encounter this issue which can have different symptoms. In this article I will describe the symptoms, a possible root cause and how to resolve it.

Symptoms

You have migrated a SharePoint 2007 content database to SharePoint 2010 using the mount-spcontentdatabase powershell command. When you review the database status using Central Admin -> Upgrade and Migration -> Review database status -> the status column Shows the message Database is too old and needs upgrade

Additional symptoms

The content database appears under Application management -> Manage Contentdatabase. The number of site collections is correct (not zero). If you try to list the details of a site collection, no details appear. Try to browse Central Admin -> Application Management -> View all Site Collections. Select a site collection of which you are sure it belongs to the newly migrated content database. View the details on the right side (preview pane). If these are not showing at all AND you have the message Database is too old and needs upgrade, you are probably having the same issue.
Another one:The site collections cannot be browsed. Visiting root pages of a site collection yield a HTTP code 200 and in the browser the user will see one line of text: Server Error. If you browse the Site settings of the site collection (_/layouts/settings.aspx) you will see a HTTP Code 403: Forbidden.
Another one:
Reviewing the settings of the content database yields a "Object not set to a reference". Try to browse the details of the content database by using Central Admin -> Application management -> Manage content database. Select the newly migrated content database by clicking on it. If you see an object not set to a reference message you are probably experiencing the same issue.

Cause

You have probably run the mount-spcontentdatabase powershell command without having the appropriate permissions on SQL. This may results in a succesful attachment and upgrade but with the above symptoms. Make sure the account that is performing the upgrade has the following permissions:
- It must have domain user account permissions.
- It must be a member of the local administrators group on each server in the SharePoint Server 2010 farm, excluding SQL Server and the Simple Mail Transfer Protocol (SMTP) server.
- This account must have access to the SharePoint Server 2010 databases.
- If you use any Windows PowerShell operations that affect a database, the setup user administrator account must be a member of the db_owner role.
- This account must be assigned to the securityadmin and dbcreator SQL Server security roles during setup and configuration.

Resolution

Try to dismount the spcontentdatabase. Do not try to delete the contentdatabase without unprovisioning it first. Assign the proper permissions on SQL especially the SecurityAdmin, DBCreator and DBO owner roles. Refer to the TechNet article Accounts and Permissions. When permissions are set correctly, remount the database, preferably in an preupgraded state.
If you mount the spcontentdatabase and receive a message like "Cannot mount spdatabase because the database ID already exist, run mount spcontentdatabase and assign a new databaseid", you may have an issue with an uncorrectly dismounted database. It might be that the database is no longer listed in the manage content databases section of Central Admin, but still appears on the Review database status section under Upgrade and Migration. This may be cause by not unprovisioning the content database before deleting it. Use powershell to get the database (Get-SPContentDatabase) and run the unprovision and then the delete statement. If the unprovisioning fails, just run the delete statement. This may also be related to not having enough permissions on SQL Server level. See above.

How to add notification and moderation to your Facebook social plugin for blogspot

noreply@blogger.com (Servé Hermans) — Fri, 10 Jun 2011 19:26:00 +0000

Adding the facebook social plugin to this blog was very easy. After a while though I was wondering how to get notified and moderate comments.

Most posts out there describe how to request an app_id on Facebook Developer. That only takes a couple of minutes but still that was not enough. You need to add two meta tags in the head of the blogtemplate where you refer the app_id and admins and moderator id. Still that is not enough.

You also need to go to the facebook integration page of your app and fill out the so called canvas fields.

So let me describe this in more detail:

1. Create an app, browse to the following link: https://www.facebook.com/developers/createapp.php

2. After that you will see you app_id, grab it and keep it for later

3. Next on you App settings page select facebook integration from the left menu

4. For canvas page you need to enter a tag which is unique. I have used hermansberghem, the first part of my blogaddress

5. Next on the Canvas URL enter the full URL of your blog. In my case http://hermansberghem.blogspot.com

6. Save your changes

7. Edit you blogspot template, put the following meta statements just before the tag:

<meta content='228558950503716' property='fb:app_id'/>
<meta content='100002186857445' property='fb:admins'/>
<meta content='100002186857445' property='fb:moderator'/>

Of course put your app_id in the first tag and your user id in the second and third tag. Your user id is not your login name. If you dont know your user id then I would suggest to create a photoalbum on your facebook profile and click on the album. In the URL you will see the id. That is your user id.

8. Next you need to add the comments section to your blog template. What I did is expand all the widgets by checking the box and find the backlinks line:

<b:includable id="backlinks" var="post"></b:includable>

Before that line I posted the following:

<div id='fb-root'/><script src='http://connect.facebook.net/en_US/all.js#xfbml=1'/><fb:comments expr:href='data:post.url' num_posts='5' width='660'/>
</b:includable>

As you can see that I use the expr: syntax to get the link to the post. I wanted comments on the homepage and detailpages so this works great for me.

That is it!

Added Facebook Social Plugin

noreply@blogger.com (Servé Hermans) — Mon, 06 Jun 2011 19:46:00 +0000

Thanks to Joel Oleson I have replaced comments by the Facebook Social Plugin. This means that you won't be able to read the old comments anymore or use Google's commenting to leave a message. Instead you can now use the new Facebook plugin as I think it works nicer.

C2WTS: Accounts used by application pools or service identities are in the local machine Administrators group

noreply@blogger.com (Servé Hermans) — Fri, 08 Apr 2011 17:30:00 +0000

There is one Health rule that conflicts with a best practice when using the Claims to Windows Token Service leading to a warning that you are using an account that is part of the local administrators group. In this case we have indeed explicitly added the service account that is used by the C2WTS service to the local administrators group on each server that is running this service.

C2WTS – why do I need it?

Some server applications need C2WTS to translate claims to Windows credentials for outbound claims. That is, if your web app uses classic mode or windows claims authentication. Maybe you also remember that SharePoint uses claims within the farm, but some out of the box service applications do not allow outbound claims authentication (yet).
So what does this mean? Basically I am saying that if for instance you would like to connect your Excel sheet to a data source which is not claims aware, a possible solution would be to configure Kerberos constrained delegation and the C2WTS to be able to connect to that data source.

Configuring the C2WTS: required local server permissions

The Configuring Kerberos Authentication for Microsoft SharePoint 2010 products guide tells you how to configure the service account used by the C2WTS service. On Page 97 it explains how to configure the required local server permissions that C2WTS requires.

Step 4a. tells you explicitly to Add the service account to the Local Administrators Group

The conflict: health analyzer rule

The rule: Accounts used by application pools or service identities are in the local machine Administrators group which you can find in Central Admin under Monitoring –> Rule definitions, checks if you are using accounts in Web app pools or services that are part of the local administrators group.
This rule conflicts with the guidance around the C2WTS service account as described in the Kerberos guide.

What to do?

You might want to ignore the rule. On the other hand what I wanted to know is: is it really necessary to have the service account for C2WTS part of the local farm admin group. So what I did is I configured Kerberos constrained delegation for Excel Services and a SQL server data source and not use the local farm admin group for the service account.
Guess what? It still works! I tried it different ways, with an external content type for instance and still the same results. It works. Now, that leaves me puzzled as I don’t grasp why the they put in the recommendation in the guide. As long as it works, I will ignore the advice.

Document ID Service software boundary

noreply@blogger.com (Servé Hermans) — Wed, 06 Apr 2011 15:00:00 +0000

Today I discovered that the Document ID Service has a software boundary which prevents the feature from working as expected. That is something I can live with. The shocking reality however is that there is NO visual feedback for the end user and at the same time, the chance that users will hit the boundary is quite real.

Activating the Document ID service on an existing site

In most cases the Team site template is favorite and people don’t tend to select the Document center template when they want to collaborate on a document. No surprise as the Document Center template is quite, uh, simplistic as the only thing you will see after creation is a document library and a slightly different homepage. The good thing though is that the template has document sets and ID service already activated.

Now, this means that people who would like to work with Document IDs need to activate the feature at the Site collection Level. No problem you might think, just browse from Site Actions –> Site settings –> Site Collection Administration –> Site collection features –> Document ID Service –> Activate. Either you are the lucky one and it works or you will have to wait a bit, or it will never work at all.

No it is not because of the timer jobs

I have seen blog posts literally saying that you needed to go to Central Admin –> Monitoring –> Job definitions and have a look at two different timer jobs:

The Document ID Assignment job which runs daily
The Document ID enable/disable job which runs daily

Even if you run these jobs “now” it might happen that your Document ID Service is activated but nothing happens.

The real issue and boundary

If you have a site that has more than 20 libraries and lists in total, activating the Document ID Service does not work.

Yup, it is true! I did not invent this myself. Of course you don’t believe me so why don’t you try it and watch the ULS log! The first thing that you will notice is this message:

Document ID/IsSiteTooBig: Site 'http://...' - too many doclibs in web '...', 21 > 20

The message suggest that you have too many Document libraries. But fact is that I had only 16 libraries and 5 lists so it is the total of both libraries and lists! Also, the feature seems to be activated but there is no DocID column added to the libraries.
Unless I am utterly mistaken this means that when your site grows, it cannot have Document IDs added to all of them. Why? Performance? Possible. I don’t know but if anyone from MSFT can comment on it I would be most grateful.

Visio: the server failed to process the request

noreply@blogger.com (Servé Hermans) — Sun, 03 Apr 2011 20:00:00 +0000

I had a situation where the message the server failed to process the request appeared on a workflow status page where there is Visio web part showing you graphically the status of your workflow like the picture is showing below:

There are already some articles out there which are about the same subject and it depends on your authentication scenario if that article might be helpful for you.
Take for instance the article from Satish TK called Visio 2010 Services Error – The server failed to process the request. That articles is valid when you are using a secure store and where Visio has an Application ID.

The scenario

My scenario is much more out of the box where the Visio Service application is taking data from a SharePoint list and bind it to Visio shape data. There are no external databases involved for instance. Now, the only difference is that I am using Kerberos authentication for incoming claims. Does it matter? Probably not much as within the Farm claims is used and the Application pool identity of the Visio Service Application is used.
Now let me first show you the errors in the ULS logging that are linked to this message in the browser:

The errors

1. Failed to get Vector Diagram for Visio file (null) page (null) Exception

2. Failed to generate Vector diagram for file Http://...... Error : System.Data.SqlClient.SqlException: Cannot open database "WSS_Content_Collaboration" requested by the login. The login failed

3. SharePoint Web Services Round Robin Service Load Balancer Event: EndpointFailure Active Endpoints: 1 Failed Endpoints:2 Affected Endpoint: http://.....:32843/f456f4f6449b4a669c3c52e167c6acf9/VisioGraphicsService.svc

The solution

1. My first advice would be to run Visio Service Application using a dedicated service account, let us call it VisioSvcAcc

2. If you are using Kerberos (otherwise skip this step): Configure a service principal name: SETSPN –S SP/VisioServices YOURDOMAIN\VisioSvcAcc

3. If you are using Kerberos (otherwise skip this step): Configure constrained Kerberos delegation paths both for the service account as for the Computers running Visio Services. You configure that for each backend SQL / whatever system that has been configured for Kerberos authentication. There is no need to configure this for SharePoint databases. Have a look at the Kerberos Guide for SP2010 for detailed steps. In short: using MMC AD snap in you select the service account, delegation tab, select trust this user for delegation to specified services, use any authentication protocol, click the Add button, select user and computer and enter the name of the service account. Click OK, select the services for MSSQLSVC and repeat the steps for the other principal names.

Are you still with me? If not, forget about the Kerberos steps and read on!

4. Start SharePoint 2010 management shell

$w = Get-SPWebApplication –Identity http://yourwepappurl
$w.GrantAccessToProcessIdentity(“YOURDOMAIN\VisioSvcAcc”)

5. Central Admin –> Manage services on server –> Select App server that is running Visio –> Start Visio Graphics Service –> repeat step for other App servers running Visio

6. Central Admin –> Application Management –> Manage Service Application –> New –> Visio Services Application. Name: Visio Service Application; Create new application pool. Under configurable enter YOURDOMAIN\VisioSvcAcc; select Add this service applications proxy to the farms default proxy list.

Summarized

So, the most important steps here are: a. Grant access to the Visio service account on the process of your web application b. When creating the Visio service App, use a dedicated App pool and the dedicated Visio service account.

That is it! If you follow these steps you will notice that there is no need to add permission to content databases yourself. Also, there is no need to configure the unattended account or use the Secure Store and Application ID if you use SharePoint out of box without Visio tapping into external data source.

Other resources

Trick: use query string values to provision InfoPath form

noreply@blogger.com (Servé Hermans) — Wed, 30 Mar 2011 12:00:00 +0000

I had a business case where a user clicks on a link to open a modal dialog showing a prepopulated InfoPath form. The default values needed to be pulled in from the query string. In this post I will outline the trick that I used.
The original idea actually came from an old blog post of mine (2007):

SLP - InfoPath extension for custom lookup pages

I decided to reuse it in a different context and supported way.

An example
Imagine you have a list of links and the link is formatted like:

http://url/newitem.aspx?title=StepByStep&BookID=12&Prize=25

Now, if you would feed this to an InfoPath form you will notice that the Title field will be pulled from the query string as a default value to the InfoPath form field that is named Title. So far, so good. But what about the other two values?

The trick is to edit the page that is showing your InfoPath form, add a web part to it that contains a piece of JavaScript and let it provision your InfoPath form. There are some nasty things that we need to solve but let’s start by creating a custom InfoPath form by creating a list that will store Book Orders.
Create the custom New Item InfoPath form
The quickest way of doing this is by editing the default New Item form from within the browser.
1. Create a site using the Team Site template

2. Site Actions –> More Options –> List –> Custom List. Name: BookOrders

3. On the BookOrders list –> Ribbon –> Create Column –> BookId –> OK

4. Create Column –> Prize –> OK
5. Ribbon –> Customize list section –> Click the icon (Modify Form web parts)

6. Choose Default new form from the drop down. Your screen should look like this:

7. Leave it like this, later on we will return here to Add a web part
If you would render this InfoPath form and inspect the page elements you will discover that there are no elements that contain the IDs BookID or Prize. Instead you will see all kinds of GUID like numbers as IDs. Basically these refer to the naming containers. So how do we map query string key value combinations to these page elements on the form? That, is our neat little trick!
Add special tags to InfoPath form elements
In this paragraph we will edit our InfoPath form using … InfoPath and add some tags to the input fields. These tags will be used to map the fields to the Query string keys.

7. Stop editing the page and go back to your list BookOrders

8. Make sure you have the List selected so that the Ribbon shows again. From List Tools –> List menu select the Customize Form button (the big purple InfoPath icon in the Ribbon)

9. InfoPath will be launched and the form will open

10. Right-Click on the BookId field and choose Text Box properties

11. Select the Tab Advanced

12. Under screen tip enter a special prepared tag which consists of two parts:

=BookID:Enter your book ID here

The first part is our key value and the second part is our Title field which will eventually be used as a screen tip.

13. Repeat step 12 for the Prize field:

=Prize:Cost per book

14. Click on the Quick Publish icon in the Quick Access toolbar (the lightning bolt with the arrow icon) and close InfoPath

Add JavaScript to pull values from query string and provision Form
If we would stop here, nothing would happen. If you would click Add new item and hover with your mouse over the BookID field, you would notice that the screen tip would appear just like we entered above: =BookID:Enter your BookID here. Now, let us add a web part to our custom InfoPath New Item page which will contain our JavaScript. We will use a content editor web part for that.
15. Go back to your BookOrders list. From the ribbon select the Modify form web parts icon and from the drop down select (Item) New Form as shown below.
16. Click Add Web part in the top most zone and add a Content Editor web part to the page.

17. Select the web part. From the ribbon select the HTML icon as shown below

18. Select Edit HTML source from the drop down. A popup dialog shows where you can enter the HTML.

// The following code just contains snippets. The rest you'll have to figure out yourself
// I can only share what is common sense
// That however, should be quite easy

function PasteQS() {

// Get a collection of all INPUT elements on the page
var inputFieldCollection = document.getElementsByTagName("input");

// Loop through INPUT elements on the page
for (itemIndex = 0; itemIndex < inputFieldCollection.length; itemIndex++) {

// Some code skipped ...

// For all elements which have a TITLE attribute
if (inputItem.title) {

// Does the TITLE property contain a value with a '=' in it
var valuePos = titleValue.indexOf("=");
if (valuePos != -1) {

// Some code skipped ...
// Remember your constructued field is something like =KEY:TITLE
// keyValue = So fetch the KEY value, this you use to pull the right value from the Querystring
// titleValue = Also fetch the TITLE value, this you use to set the overwrite the TITLE field of the INPUT element
//    we don't want users to see our =KEY:TITLE value when they hover their mouse right?

// Paste the value from the Querystring into your form
inputItem.value = queryString(keyValue);
}
}
}
}

function PageQuery(q) {
if (q.length > 1) {
this.q = q.substring(1, q.length);
}
else {
this.q = null;
}
this.keyValuePairs = new Array();
if (q) {
for (var i = 0; i < this.q.split("&").length; i++) {
this.keyValuePairs[i] = this.q.split("&")[i];
}
}
this.getKeyValuePairs = function () { return this.keyValuePairs; }
this.getValue = function (s) {
for (var j = 0; j < this.keyValuePairs.length; j++) {
if (this.keyValuePairs[j].split("=")[0] == s)
return this.keyValuePairs[j].split("=")[1];
}
return false;
}
this.getParameters = function () {
var a = new Array(this.getLength());
for (var j = 0; j < this.keyValuePairs.length; j++) {
a[j] = this.keyValuePairs[j].split("=")[0];
}
return a;
}
this.getLength = function () { return this.keyValuePairs.length; }
}

function queryString(key) {
var page = new PageQuery(window.location.search);
return unescape(page.getValue(key));
}

19. Save the page and try it yourself!

20. Construct a link to the modified InfoPath form. Take into account that this is not the same URL as was before you started to edit the InfoPath New form page. What you can do is click on Add item and then open the dialog in a new TAB, copy the URL

21. To test your form add something to the copied URL like: BookID=12&Prize=15. You will notice that the info will be provisioned automatically into your Form.

Things you need to add as well

22. Push your script to execute on page load using _spBodyOnLoadFunctionNames.push

23. When you submit the form you will discover that the data is not saved. You will need to tell InfoPath that the info on the form has changed. The easiest way to do this is after you have pasted the data into the Form, call the focus and blur method for each element that you changed. Do this after you have pasted the last value on the form (so build an array of elements you have changed).

24. When testing I discovered that in a certain non IE browser, the script was already running before all page elements were rendered correctly. I was lazy and added a slight delay (window.setTimeOut) which is baaaad. However, there is more you can do (like script dependencies and so on) and if you do, please add it to this post so that I can use it. *grin*

Well, that is it, hopefully you now see that this is really powerful.

Complete list of custom action IDs

noreply@blogger.com (Servé Hermans) — Mon, 28 Mar 2011 19:00:00 +0000

The article Default Custom Action Locations and IDs describes a table which shows the locations, custom action group IDs, and custom action IDs that are used in the Custom action element.
Unfortunately it does not cover everything so I decided to build are more complete list and post it on my blog.

So you might be wondering what this list is all about. I am using it a lot to hide actions from for instance the Site Settings or menus. In other cases you are planning to add your own actions to Ribbons, Central Admin, Site settings, Context menus etc. If you build such a feature using for instance the CKSDEV Hide Custom Action feature, you need to enter three values: the location, the group and the action ID. And that is exactly where this list comes in handy. Enjoy!

Download: Complete list of all Custom Action IDs, Location and Group IDs

Hiding site templates

noreply@blogger.com (Servé Hermans) — Sun, 27 Mar 2011 21:00:00 +0000

According to the article “Supported and unsupported scenarios for working with custom site definitions and custom area definitions in SharePoint 2003, 2007 and 2010”, modifying the webtemp*.xml files is not supported by Microsoft.

That raises the question how to hide site templates that you do not want to offer to the business. A method that was often used is to edit the webtemp*.xml files and change the attribute to hidden. Well, basically, that is still the technique on how to do it but by using copies of the original files. SharePoint will merge the files and you change will override the original setting.

It is no rocket science. This is how to do it:

Create a Visual Studio Empty SharePoint project
Add a mapped folder to the SharePoint root
Add folders for 1033 and XML
Add an empty XML file the XML folder called webtempHide.xml
Copy <template> and <configuration> definitions from the existing webtemp*.xml files that you want to hide from you 14\TEMPLATE\1033\XML\ location
Make sure the attribute ‘Hidden’ is set to TRUE
Deploy solution

Example – Hide Access related site templates:

 
 
<Template Name="ACCSRV" SetupPath="SiteTemplates\AccSrv" ID="2764">
<Configuration ID="0" Title="Access Services Site" Hidden="TRUE" ImageUrl="/_layouts/images/bsprev.png" Description="Microsoft Access Server" DisplayCategory="Collaboration" >    </Configuration>
<Configuration ID="1" Title="Assets Web Database" Hidden="TRUE" ImageUrl="/_layouts/images/AccessAssets.png" Description="Create an assets database to keep track of assets, including asset details and owners." DisplayCategory="Web Databases" VisibilityFeatureDependency="1CC4B32C-299B-41AA-9770-67715EA05F25" SubWebOnly="TRUE" ProvisionAssembly="Microsoft.Office.Access.Server.Application, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" ProvisionClass="Microsoft.Office.Access.Server.Template.TemplateProvisioningProvider" ProvisionData="Assets.accdt" >    </Configuration>
<Configuration ID="3" Title="Charitable Contributions Web Database" Hidden="TRUE" ImageUrl="/_layouts/images/AccessCharitableContributions.png" Description="Create a database to track information about fundraising campaigns including donations made by contributors, campaign related events, and pending tasks." DisplayCategory="Web Databases" VisibilityFeatureDependency="1CC4B32C-299B-41AA-9770-67715EA05F25" SubWebOnly="TRUE" ProvisionAssembly="Microsoft.Office.Access.Server.Application, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" ProvisionClass="Microsoft.Office.Access.Server.Template.TemplateProvisioningProvider" ProvisionData="CharitableContributions.accdt" >    </Configuration>
<Configuration ID="4" Title="Contacts Web Database" Hidden="TRUE" ImageUrl="/_layouts/images/AccessContacts.png" Description="Create a contacts database to manage information about people that your team works with, such as customers and partners." DisplayCategory="Web Databases" VisibilityFeatureDependency="1CC4B32C-299B-41AA-9770-67715EA05F25" SubWebOnly="TRUE" ProvisionAssembly="Microsoft.Office.Access.Server.Application, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" ProvisionClass="Microsoft.Office.Access.Server.Template.TemplateProvisioningProvider" ProvisionData="Contacts.accdt" >    </Configuration>
<Configuration ID="6" Title="Issues Web Database" Hidden="TRUE" ImageUrl="/_layouts/images/AccessIssues.png" Description="Create an issues database to manage a set of issues or problems. You can assign, prioritize, and follow the progress of issues from start to finish." DisplayCategory="Web Databases" VisibilityFeatureDependency="1CC4B32C-299B-41AA-9770-67715EA05F25" SubWebOnly="TRUE" ProvisionAssembly="Microsoft.Office.Access.Server.Application, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" ProvisionClass="Microsoft.Office.Access.Server.Template.TemplateProvisioningProvider" ProvisionData="Issues.accdt" >    </Configuration>
<Configuration ID="5" Title="Projects Web Database" Hidden="TRUE" ImageUrl="/_layouts/images/AccessProjects.png" Description="Create a project tracking database to track multiple projects, and assign tasks to different people." DisplayCategory="Web Databases" VisibilityFeatureDependency="1CC4B32C-299B-41AA-9770-67715EA05F25" SubWebOnly="TRUE" ProvisionAssembly="Microsoft.Office.Access.Server.Application, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" ProvisionClass="Microsoft.Office.Access.Server.Template.TemplateProvisioningProvider" ProvisionData="Projects.accdt" >    </Configuration>
</Template>

SharePoint will still merge existing webtemp*.xml files with your additions where you addition will overwrite existing definitions. No need for modifying existing webtemp*.xml files anymore.

Excel Services could not obtains a Windows Identity

noreply@blogger.com (Servé Hermans) — Wed, 02 Mar 2011 11:06:00 +0000

The SP2010 Kerberos guide is really helpful. Sometimes though you just forget to execute a step. Which of course leads to happy accidents.
I do not necessarily say it happens to me *evil grin* but let me explain the issue and show you the step from the guide that I, uhh, someone missed....

The issue

If you have a workbook with some external data in it and you refresh the data in the sheet (Data -> Refresh all connections), you might see the following messages appearing in the ULS and Event logs:

Message in browser:
- The data connection uses Windows Authentication and user credentials could not be delegated

ULS entry start:
- Metadata: Product = Excel Services Application, Category = Excel Calculation Services, Level = Medium
- Message: starting request of type RefreshAll

ULS entry first error:
- Metadata: Product = SharePoint Foundation, Category = Claims Authentication
- Message: Could not retrieve a valid windows identity for NTName='DOMAIN\USERNAME', UPN='USERNAME@FQDN'
- Message: Token could not be zero

ULS entry second error:
- Metadata: Product = Excel Services Application, Category = Excel Calculation Services
- Message: Failed to get WindowsIdentity from IClaimsIdentity

Windows Event Viewer entry:
- Metadata: eventId = 5252, Task Category = External Data
- Message: Credential delegation failed because Excel Services Application was unable to obtain a Windows Identity

The solution

Log onto the servers which are running C2WTS and give the C2WTS the following permissions:

a. Add the service account to the local Administrators Groups.

b. In local security policy (secpol.msc) under user rights assignment give the service account the following permissions:

i. Act as part of the operating system
ii. Impersonate a client after authentication
iii. Log on as a service (if you have started it before from within Central Admin, this may already be the case)

c. Safe side? Restart IIS

SP2010 Do we need to Load balance the Topology Service Application

noreply@blogger.com (Servé Hermans) — Thu, 03 Feb 2011 12:41:00 +0000

This post is dedicated to the discussion around whether if you need to load balance the Topology service app in a multi farm scenario where you have a farm that consumes Services from another farm.

I will try to reference the various blog posts that are about the same subject and at the same time show you a detailed diagram that outlines some of it.

The discussion I would like to trigger here is two fold:

First of all do we need to load balance the topology service to achieve higher availibility?
Secondly, I have seen contradictions in the blog articles. Some people claim the Topology service is responsible for load balancing across service app instances whereas other people claim that these end point urls are stored in the cache of the service app proxies on the consumer farm.

Now let me stop throwing terms and start right there at the beginning by describing a common real world scenario.

Real World Scenario: cross farm services

Assume you want to centralize your Search, User profiles and Corporate taxonomies. What would you do? Maybe you are planning to host different farms like a dedicated farm for your intranet, a specialized farm for your Departments where they can launch apps and a commodity farm for basic, standard team sites. To allow these farms to share the services you will need to introduce another farm often referred to as the Enterprise services farm.
Imagine you have created an Enterprise services farm which has two Application servers. Take an example: the managed metadata service app. Typically you will have a service instance running on each Application server. Both URIs will be published by the Topology service app to each consuming farm. Secondly the consuming farms will push there URIs to the URI cache of each Service App proxy. This may sound complicated but if you look at the diagram below you might be better able to understand it.

Example: detailed overview Shared Services

There are some steps involved because you are able to consume services between farms. You will need to setup a trust and next publish the services.
After you have done that you are able to consume the services on the Consuming farm.

To illustrate: if you have a webpart that makes a call to the managed metadata service app, it will use the service app proxy to determine the end-point of the app. The first time the service app proxy will retrieve the URIs from the topology service app at the Publishing farm. After that it will cache the end points in its own URI cache. Its own load balancing component is using both URIs. In case one of them does not respond, the URI will be taken offline.

Secondly there is a timer job, called the Application address refresh which calls the Topology service app on the Publishing farm every 15 mins. On the Publishing farm, the Topology service app discovers which end point URLs are available and returns those URLs to the Consuming farm. At the Consuming farm these URLs are pushed to the local Service App proxies in case they are updated or deleted.

The Contradictions

If you are reading some blog posts you might already be wondering if I am telling you the truth. Fact is, I have stumbled upon some contradictions.

The post written by Russ Maxwell: http://blogs.msdn.com/b/russmax/archive/2010/05/06/sharepoint-2010-shared-service-architecture-part-2.aspx, shows you that after the proxies have cached the URIs locally, they reach out directly to the Service Apps, not requesting the URIs first through the Topology service app.

However, some posts like the ones from Steve Peschka: http://blogs.technet.com/b/speschka/archive/2011/01/04/additional-info-on-load-balancing-the-sharepoint-2010-topology-service.aspx and the one from Josh Gave: http://blogs.msdn.com/b/besidethepoint/archive/2010/12/08/load-balancing-the-sharepoint-2010-topology-service.aspx make it appear if the Topology service is called every time a Service app proxy needs to connect to the Publishing farm. If that would be the case then we might have a single point of failure!

Single point of failure

The topology service is often published using a server NETBIOS name and a port number (32844 I believe).. That would indeed mean that we have a single point of failure.

Now, what happens if this service app goes down? To my opinion it means that the Application address refresh job will not be able to update the URI endpoints anymore. Secondly, you would not be able to publish new Service Apps to consuming farms anymore. Finally, at the consuming farm, Service apps would still be running fine as the Service app endpoint URLs are fetched from the URI cache of the proxy. The internal load balance component of the Service App proxy would recognize if an endpoint is down and take it offline automaticaly.

Summarized
What does this all mean? To my opinion it would mean that if the Toplogy service app on the Publishing farm goes down you would still have some time to fix the issue and bring it back online. I do not think that the Service Apps that are consumed will be interupted. Although I must admit I haven't tried it out myself yet.

Your input here!

The question is, do we need to load balance the Topology service app on the Publishing farm? Personally, I would say that if your farm is offering services globally 24x7, then yes. You probably do not want to get out of bed in case things go wrong. On the other hand it all depends on your service level and the decision is up to you.
Also, the contradictions still remain: is the Topology service being called every time by the Service app proxies on the consuming farm, or do they rely on their URI cache.

Fact is, I may have it all wrong! ;-) To be on the safe side, I will go through the tests of bringing down the Topology service and have a close look on the ULS log, filter: category = Topology, message = WcfSendrequest..

I have written this blog post to get your input on this! Did you try this out yourself? What are your experiences? Input is more than welcome. Thanks!

New logo has arrived!

noreply@blogger.com (Servé Hermans) — Wed, 05 Jan 2011 19:25:00 +0000

Dear readers, thanks for all the votes on the logo design contest! I have selected the nr. 1 logo from the poll and the result you can see at the top.

I received more than 216 entries! In just three days.... That was awesome. So on my blog you will find the black variation whereas on my TripleNeo site you will see the main variation which is on a white background and has the word Neo colored.

Please help me selecting a logo for my new company

noreply@blogger.com (Servé Hermans) — Mon, 03 Jan 2011 11:11:00 +0000

Dear readers, as you probably know I have started as a freelance SharePoint consultant. After 8 years working as the Lead SharePoint architect within global pharmaceutical companies like Organon BioSciences, Schering-Plough and Merck (MSD in Europe).

My new company is called TripleNeo and I am looking for a new logo. I have started a contest on 99designs where designers can submit logos and people can rate and eventually the winning design wins the money.

I NEED YOUR HELP!

Please have a look below and let me know what you like most. You can go to the website, use the rating controls, or just add a comment to this blog post.

Custom Logo Design at 99designs

New look and feel

noreply@blogger.com (Servé Hermans) — Mon, 15 Nov 2010 19:58:00 +0000

Today I decided to change the look and feel of my blog. Why?

Maybe because it is so tightly integrated with Blogger.
Maybe because most of the Apps I use nowadays have these fancy Black GUI's.
Or maybe because this style gives me more horizontal space?

It also means that this blog is now more like my other blogs.... Other blogs? Yes, if you are into Game Development, which is a hobby of mine, have a look at servehermans.blogspot.com and enjoy. On that blog I am mostly babbling about iPhone and iPad game development.

Technology stack overview

noreply@blogger.com (Servé Hermans) — Thu, 16 Sep 2010 14:17:00 +0000

In this post I will try to give you a schematic overview of technologies, principles and protocols and how they relate to each other. The ones that I will describe are AJAX, JSON, REST, SOAP and JQUERY. I will also add a staccato description of these for dummies and maybe if I have some time left I will add an example. So let’s start with the nice picture! ;-)

AJAX

• PRESENTATION: HTML or XHTML and CSS
• DYNAMIC DISPLAY AND INTERACTION: Document Object Model and JavaScript
• EXCHANGE OF DATA AND MANIPULATION: XML and XSLT, also JSON possible
• ASYNCHRONOUS COMMUNICATION: XMLHttpRequest
• GLUE: JavaScript (sometimes VBScript)
• COLLECTION: of technologies

JSON

• INTERCHANGE OF DATA: lightweight text-based format, alternative to XML
• BASIC TYPES: String, Integer, Boolean, Array and Objects
• VERIFICATION: goes trough JSON Schema to verify the contents of an object
• SAME ORIGIN POLICY: the reply must come from the same DNS domain as where the request was made. JSONP / JavaScript / JSONRequest are workarounds.
• SECURITY CONCERNS: returning script instead of JSON data. Workaround: use a JSON parser. Unless browser supports native JSON like IE8, Firefox 3.5, Opera 10.5, Chrome, Safari. Else, use libraries like JQUERY
• ALTERNATIVES: XML although XML tends to be larger
• DISADVANTAGES: no binary representations possible, unless encoded as text, no standard path notation like XPath.

REST

• PRINCIPLE: an application can interact with a resource by knowing the identifier of the resource and the action required like GET, PUT or DELETE. The resource returns a representation of the requested data which can be HTML, XML, JSON etc.
• ARCHITECTURE: REST is an architecture and is not a standard.
• CLIENT-SERVER: clients are separated by the server. Clients don't care about data storage, servers don't care about presentation.
• STATELESS: any state is held in the client. Client context is not stored on the server.
• CACHEABLE: responses may be cached
• UNIFORM ARCHITECTURE: between client and servers decouples the architecture. Each one can evolve on its own.
• SECURITY RISKS: Code on Demand by returning java applets or JavaScript.
• RESTFUL WEB SERVICE: a simple web service using HTTP and using the principles or REST. The web service is built around the URI to the resource, the MIME type of the data supported by the web service and the ACTION like put, get, delete and post.
• FRAMEWORK IMPLEMENTATIONS: MS WCF Data Services, PHP Symphony and many others
• ALTERNATIVE: SOAP which is a protocol, not an architectural principle

SOAP

• PROTOCOL: to exchange data in a structured manner, providing a basic messaging framework upon web services can been built.
• ALTERNATIVE: built RESTful web service as these correctly make use of the HTTP defined methods

JQUERY

• FRAMEWORK: JavaScript framework to manipulate DOM, CSS and to communicate with SERVER (AJAX)
• CHARACTERISTICS: work with DOM cross browser with support of CSS 1.3. Use of events, effects, AJAX and is extensible.

Example

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
<head>
<title>Show content from page</title>

<script src="http://ajax.microsoft.com/ajax/jquery/jquery-1.3.2.min.js" type="text/javascript"></script>

<script type="text/javascript">
$(document).ready(function() {

$.getJSON("http://yourserver/_vti_bin/ListData.svc/Pages/",function(data) {

var count = 0;
$.each(data.d.results, function(i,result) {
var title = result.Title;
html = "<table border='0' style='float: left'> \
<tr><td style='color:blue'>" + title +"</td></tr> \
</table>";
$('#resultarea').append($(html));
});
});
});

</script>

</head>
<body>
<div id="resultarea">
</div>
</body>
</html>

In the example as shown above I have demonstrated how to call a RESTful webservice, the listdata.svc from SharePoint, get data through an asynchronous call, get back JSON data and use JQuery to display it. All of course done using AJAX technologies.

Push or pull publishing pages from SharePoint 2010

noreply@blogger.com (Servé Hermans) — Wed, 28 Jul 2010 12:00:00 +0000

Sometimes when you build a large Internet facing site you need to push or pull content to or from other systems. It could be that you are working with external content providers who will provide for instance daily news articles. Or maybe you have to send out e-mail campaigns and you would like to pull content from your site into your newsletters. In these cases you might wonder if it is possible to use for instance the new RESTful services which uses HTTP to get or send information to your pages libraries.
In this article I will perform a small POC to see if that is possible and what the alternatives are.

The ListData.svc WCF web service

The ListData.svc web service did not exist with SharePoint 2007. You can access it through the _vti_bin virtual directoy like:

http://www.company.com/en-us/news/_vti_bin/ListData.svc

So what does it provide:

It provides an ATOM feed of all the lists and links to URL’s that provide more data. As you can see below it will show you for instance a list of collections. The pages library would be one of them. Now, if you would attach the HFRD attribute to your query it would give you access to all of the content within the pages library.

Next to that you can Query the content by appending your query to the URL, including filtering, sorting and paging. An example would be something like:

http://.../_vti_bin/ListData.svc/Pages?$filter=(Country eq 'Netherlands')

But that is not all! You can actually add, delete or update content. So if your HTTP Header contains the verb DELETE you could delete the second item by calling the same URL but then with:

http://.../_vti_bin/ListData.svc/Pages(1)

And even better, caching and security has been taken care of as well.
And if you use ASP.NET Ajax 4 you can query WCF Data Services like the ListData.svc natively and it will give you the benefit of using templates.

Publishing page returned by ListData

Now, what about the information you get back from the ListData.svc when querying a pages library?
Imagine you content type is called Company News and is inheriting from the out of the box content type article page.
If you would query the page through the ListData web service it would give you either the ATOM XML feed or the JSON data feed (depends on your type of request).
Have a look at the columns below. As you can see some of the columns are “system” columns like Content Type ID, some of the are from the article page like “Page Content” and some are defined by yourself like “Publisher”.

ContentTypeID	0x01000C568D50A14D9……
Name	MyNews.aspx
Title	Company X has announced new campaign
PageLayout	http://www.company.com/_catalogs/masterpage/mynewslayout.aspx
Publisher	Reuters
~~Page Content~~	Today it was announced that Company X has …..

Not everything will be returned

You have probably already noticed that Page Content as shown in the table above, has been removed and colored red. To me it was a little surprise that it was not being returned by the ListData web service. After doing some tests I discovered that:

Site columns of type Publishing HTML are not returned by ListData
Site columns of type Publishing Image are not returned by ListData
possibly more columns are being omitted (haven’t checked all of them yet) that are publishing specific

Can I use ListData with Publishing pages

The answer is probably: Yes in a limited way. As you don’t have the ability to change for instance the Page Content, you could still update the list item fields like the Title, Publishing Date, or custom fields like the Number of Times Read and so forth. It depends if you have need for that.
To give you a real world example: if you would to add something like “Did this article answer your question” you could add a counter to your article called “Usefulness” and increase that by one if someone clicks on a button “yes”, entirely through Javascript by calling the ListData web service.

Are there alternatives?

CMIS: CMIS stands for Content Management Interoperability Specification. It is an "interoperability specification for interacting with document-centric content repositories via HTTP-based protocols.".

It is a new standard driven by Alfresco, Day, EMC, Fatwire, IBM, Microsoft and Open Text.

Now, Microsoft has released a connector for CMIS for SharePoint 2010. Is it important? Yes! As we can more easily integrate SharePoint 2010 with other systems through an official standard. This comes with the connector:

· A CMIS Producer for SharePoint that makes SharePoint lists and document libraries available to other systems using the CMIS interfaces.

· A CMIS Consumer Web Part, enabling content from other CMIS supported repositories to be displayed and edited within SharePoint. The downside is that CMIS does not support WCM in its first release.
Client side object model: this does not support Publishing webs to create a page
ListItemCollection.asmx: this would allow you to create a content type with Publishing HTML site columns but you cannot get Rich markup across the wire.
Your own custom wrapper. It would basically mean that you create your own web service that allows for pushing content into SharePoint and your code would take care of creating the publishing page.

Your own code could be something like this:

SPSite site = new SPSite(Properties.Settings.Default.rootSiteUrl + Properties.Settings.Default.targetURL);
SPWeb web = site.OpenWeb();
PublishingWeb pubWeb = PublishingWeb.GetPublishingWeb(web);
PublishingPage newPubPage = pubWeb.GetPublishingPages().Add(pageName, GetPageLayout());
SPListItem newPage = newPubPage.ListItem;
SPList PagesLibrary = newPage.ParentList;
newPage["Title"] = pageTitle;
newPage["Page Content"] = Microsoft.SharePoint.Utilities.SPEncode.HtmlDecode(pageContent);
newPage["Article Date"] = articleDate;
newPage.Update();
newPage.File.CheckIn("");
newPage.File.Publish("Published on creation");
newPage.File.Approve("Approved on creation");

Summary

Working with SharePoint 2010 Publishing Pages through the ListData.svc is very limited, hence could be useful in some cases. Most of the publishing related site columns are not being returned by the web service. Other methods fail for the same reason. It looks like a custom web service is the only solution.
Any feedback or working solution is appreciated!

SharePoint 2010 Web Content Management vs. Tridion

noreply@blogger.com (Servé Hermans) — Fri, 04 Jun 2010 10:42:00 +0000

Where SharePoint 2007 was lacking a number WCM capabilities, SP2010 WCM starts to look promising. Time get a high level overview of SP2010 WCM capabilities and compare them against the market leader Tridion (according to Gartner and Forester research).

WCM Solutions

Although this post will focus on SP2010 and Tridion, it is good to know that there is more out there. Take for instance Fatwire, Autonomy Interwoven, Opentext, Day Communique, Vignette, Oracle, IBM, EMC, Ektron, SiteCore, Drupal, EZ and so on.
I am only able to focus on Tridion and SharePoint as I have developed and designed internet facing site for both tools.
Research agencies like Gartner and Forrester not only focus on high level capabilities but take into account other aspects as well like strategy, ability to deliver, marketing, customer reviews and so on. This post will only focus on the high level capabilities.

High Level Capabilities

The following list shows you a number of capability groups. There may be more but I think that most of the detailed WCM capabilities will fit into one of those categories:

Content Management

Dynamic Contextual Delivery

Globalization and Localization

Insights

Content Deployment

Integration

Social Computing

Marketing Instruments

Accessibility

Detailed capabilities

Now let me zoom in a little bit on the ones I defined above.
►Content management
• Usability (true in line vs. in-context) •E diting (integration with Desktop tools) •Template based •Content review and approval • Change Management • Metadata • Library services • Digital asset management (audio, video, images) • Site structure management • Navigation management
►Dynamic Contextual Delivery
• Audience targeting • Algorithm based recommendations • Personalization rules • Managing segments • Content filtering (tags, categories, navigation) • Content aggregation (rollup) • Multichannel delivery (mobile, print)

►Globalization and localization
• Content inheritance (multilingual) • Layout inheritance (multi brand) • Translation services (workflow, 3rd party integration) • Language aware components • Managing translations (resources, labels, taxonomies) • Multi lingual GUI • Global Assets libraries • Scalability across the WAN
►Insights
• Native Web Analytics • 3rd party analytics integration • Track and traceability • Data warehousing

►Content deployment
• Multi site / channel • Multi lingual • Staging / Authoring environments • Content hubs • Replication of content and or features

►Integration
• Hybrid user profiles • Content feed management outside-in and inside-out • Federation model • Client side APIs • LoB connectivity
►Social computing
• User generated content (tagging, rating, comments) • Integration with Social Network Providers • Provisioning models (for Community sites, Forums) • Activity feeds
►Accessibility
• Registration •Authorization •Authentication • Validation • WCAG 2.0 AA+ • Cross browser
The list above is probably not complete but I think I have a pretty nice list to start with. So if you are comparing WCM solutions try to keep these things in mind and be honest about it.

SharePoint 2007 WCM weaknesses

So let us be honest about SharePoint 2007. Although I saw a lot of nice internet facing sites, SharePoint 2007 was lacking some capabilities:

Content targeting: although you could use the Content Query web part or audiences to target content to visitors, it was lacking real contextual, even rules based driven content delivery. Imagine someone registering on your web site, but not having the ability to instantly compile the audiences. Good luck with your targeting. Also, people quickly extended the Content Query web part so that it could use at least query string parameters and so on. But still the filtering was limited.
Web Content management and reuse (try to add a web part as part of your rich text, use folders inside page libraries, usage of taxonomies)
Multilingualism (multi lingual GUI, translations API, translatable metadata)
Content Deployment

Replication
Multi farm Synchronization
Stability (ask you editors to go wild when you content deployment is running, and add some variations on top of that)

Marketing tools (Analytics, overlays, business rules, organizing content)

SharePoint 2010 WCM strengths

►Social computing
• Rating, tagging, commenting, integration with Social Network providers
►Insights
• Web Analytics • Developers dashboard
►Content deployment
• Improvements to support massive exports using snapshots • Improved variations using timer job • Side by side page differences • Usage of content hub for pushing out changes to content types and taxonomies • Managed metadata services for defining taxonomies • Inline editing has improved a lot. Try to add multimedia like videos or images and web parts right inside your rich text field.
►Scalability
• Large page libraries now support folders • Consuming service applications
►Reliability
• Quality checks: spelling and checks for page assets that are not published
► Multi lingual features
• Multi lingual GUI • Translatable taxonomies • Language aware components (take for instance the refinement panel that shows drill down capabilities for multi lingual taxonomies) • Import and export of translations • Translatable columns • Translation services API

What about Tridion?

To my opinion, Tridion is really strong in the area of Blue printing, something I think you can mimic with SharePoint but is not quite the same. Blueprinting layout is like defining master pages and page layouts that are reused across web sites. You could build for instance the same site in different languages. Blueprinting content is like content deployment and variations where you push out content to different web sites where you can reuse them or start localizing them. Discovering localized items, content dependencies etc is more advanced.

Tridion has some integration points with SharePoint in terms of storing assets in SharePoint and publishing them to Tridion.

But the real strong point here is that Tridion is filling in the gap of marketing tools. Where SharePoint has been focusing a lot on the content management aspect. Tridion has some nice features that are really useful for the web marketeers. Take for instance the overlay feature that add color overlays to your web page so that you can instantly see the most used components on your web page.
So Tridion is better on the delivery, analysis and optimization of content consumption. The so called Marketing tools capabilities and the Blue Printing.
But on the other side you could add for instance Google Analytics or Yahoo Analytics to your master page (piece of script) and get some nice results as well. Btw. Google Urchin might be an option as well.

Summary

SharePoint 2010 fills in on a number of WCM capabilities when compared to SP2007. The fact that SP2010 has a large developers community out there and it’s extensibility makes it an interesting choice for future internet facing sites. Tridion, when compared to SP2010 is stronger in the area of marketing tools and blueprinting.

Location aware content rollup

noreply@blogger.com (Servé Hermans) — Fri, 28 May 2010 14:23:00 +0000

In this post I will show you how to roll up content from the site you are currently on using the content query web part on a page layout on SharePoint 2010.

As you can see on the right you can tell the Content Query web part to roll up content from the whole site collection, or a specific site or a specific list. The question people often ask is where can I define to rollup content from the current site.
The reason is they want to add the Content Query web part to a page layout and not as a provisioned web part to a web part zone on a page. The answer is simple. You cannot type in relative paths. If you do so SharePoint will tell you to define paths that start with a forward slash. That would mean that you have to define the exact location. Not nice, especially if you need thise to work on a multilingual site.
Let us inspect the properties using SharePoint Designer.

As you can see, the Web URL is translating the forward slash to ~SiteCollection/. So what about if we change that to ~Site/. Try it yourself and you will discover that it actually works quite well.
If you add it like that to your page layout outside your webpart zone, you can just start creating pages using that page layout and by default it will start to roll up content from the site you are on.
I have seen posts that do basically the same using custom code and overriding the web url on load but this is the same and does not require you to release a new version of the content query web part. However, if you want end users to be able to use it like this you have two options:
1. You can use Visual Studio to override the Web URL property on Load. Waldek has a nice post about that (sorry for the URL, will probably add a reference later on)
2. You can export the web part properties, change the XML, upload the web part to the site and drag it onto the page. Same effect.

Summary

By using SharePoint Designer and changing the Web URL property of the Conten Query webpart to ~Site/ you can have a default page layout that has a content query webpar that will rollup content from the current site. This is what I call location aware content rollup.