Ivan on Software

Royalty-Free PDF Conversion and Manipulation Web Service

noreply@blogger.com (Ivan Neganov) — Wed, 08 Dec 2021 03:42:00 +0000

PDF generation, anyone? There are tons of libraries to do all sorts of things with PDF. But time and again I witness the pain that PDF handling task causes organizations, even though it should long be totally mundane, one would think...

Why is this so? You have to pay for a good PDF tool. Sometimes a lot. And the many free alternatives either focus on narrow things, or do not produce the best quality, or are hard to use, or are not for the platform of choice at the organization. The biggest obstacle is of course the licensing. If you cannot get a completely free license, then there is often a lot of friction when trying to get a paid-for variant... Here is one example of how a conversation could evolve:

- "We already have a license for PDF converter X!".

- "But it does not do what we need/or it is too hard to use!" etc., etc.

And needless to say that this sort of complexity typically arises in the kind of organizations that do a lot of document management and depend on PDF generation.

Having been through this recently myself I have got inspired to try and help my customer, Transport Canada, and other teams that may find themselves in a similar situation, and share a way to reliably generate high quality PDF documents and do some basic manipulations on them, absolutely free of charge.

Indeed, there has long been several useful PDF utilities made available on Linux, and there is the prominent LibreOffice suite that can generate great quality PDF documents free of charge. So I thought why not give it a try and put these applications inside of a Docker container and write a service that would accept HTTP requests and launch them?

Such service would allow tapping into the richness of available open source tools, many of which have been out there for decades, and if you are a developer then it would let you easily adjust which specific tools would you want to run, or how would you want to scale the service. Pretty flexible.

The service is now being adopted by Marine department at Transport Canada, and I hope it will evolve and serve them well. And since Transport Canada has a great policy of sharing the source code for some of their applications, I am happy to share a link to its public Github repository, which also has a detailed description of how it works and how to handle it. I will just list its "no frills" but much sought-after basic capabilities here:

Conversion of popular office and image formats to PDF (thanks to LibreOffice!)
Merging of office documents and images into a single PDF document (thanks to GhostScript!)
Populating and "flattening" of fillable PDF forms (thanks to pdftk-java and PDF Toolkit!)

Have fun converting to PDF for free!

Migrate Data from a Cosmos DB Azure Table API

noreply@blogger.com (Ivan Neganov) — Fri, 21 May 2021 21:23:00 +0000

If you need to migrate data from or into Azure Cosmos DB you can use Microsoft’s data migration tool to do this. The tool is versatile but the documentation isn't providing all the answers. Specifically, in a scenario when you need to migrate data from Cosmos DB instance configured as an Azure Table API to a JSON file, you can use the tool, but the settings you need to provide are not obvious. Here are the settings which worked for me for its two main tabs: Source Information and Target Information (you would see them if you run the tool dtui.exe):

Source Information

Prepare data for assembling a connection string:
1. Grab the value of Azure Table Endpoint from Overview page, for example: https://name-of-your-cosmos-db-account.table.cosmos.azure.com:443/
2. Modify this URL, replacing table.cosmos.azure.com with documents.azure.com
3. Grab the value of PRIMARY KEY from Connection String page
4. Grab the name of the root node on the Data Explorer page, this will be your database name.

2. Assemble the connection string as follows:

AccountEndpoint=https://name-of-your-cosmos-db-account.documents.azure.com:443/;AccountKey=primary-key-goes-here;Database=Your-DB-Name

3. Expand the root node (Your-DB-Name) and take a note of a table you want to export, for example My-Table-Name

4. Fill in the form on the Source Information tab:

Import from: Azure Cosmos DB

Connection String: use connection string created in step 2. Click Verify button, it should work.

Collection:  My-Table-Name 

Other fields: leave them to defaults, or you can optionally specify a query to limit the export

5. Click Next button to configure Target Information

Target Information

Export To: JSON file
Choose Local File radio button option, specify path, optionally select Prettify JSON.
Click Next to complete the wizard and run through the export.

The JSON file should be saved in the directory you specified, or if you didn’t - in the folder you have started the data management tool from.

UPDATE: importing from JSON into Azure Table storage in Cosmos DB also works. Same manipulations with connection string as described above for exporting scenario are needed. In addition, depending on your situation you may want to fill out extra parameters describing whether to regenerate Ids or not, etc.  The tutorial on using the data migration tool covers these well.

Command-Line Utility to Validate LUDOWN Files

noreply@blogger.com (Ivan Neganov) — Tue, 05 Jan 2021 02:04:00 +0000

Microsoft has released Bot Framework Composer in May 2020, and since then the tool was under active development. It allows to rapidly create rich conversational bots that leverage adaptive dialogs, language generation, skills and more.

In working with the Composer I have found that one of practical challenges was the "teaching" LUIS to recognize intents and entities: the labelling was quite verbose, the number of training examples was in dozens per intent (at least), and on top of that the documentation of the new LUDOWN format could be improved, even though the format is by far more convenient than JSON.

I've done some digging thanks to Composer being released as open source, and found a library that the Composer uses for parsing and validating the .lu files: @microsoft/bf-lu.

I thought it was easier to validate the .lu files in CLI mode, so I ended up writing my own CLI to help with quick and verbose validation. As a "bonus" it can also create a temporary LUIS app from an .lu file since the @microsoft/bf-lu validation may miss certain errors, which LUIS would complain about when attempting to create an app.

Here is the utility: https://www.npmjs.com/package/@softforte/lu

While the new Bot Framework CLI offers similar and comprehensive features, I still find this utility handy for day-to-day LUIS development and hope that it will save you some troubleshooting time. Check it out!

Single Sign-On for Two Angular Apps with Local Accounts in Azure B2C Tenant

noreply@blogger.com (Ivan Neganov) — Sun, 24 Nov 2019 23:36:00 +0000

In this day and age Single Sign-On (SSO) is thought of as a commodity, a "flag" an admin turns on somewhere, which makes logging into multiple related applications automatic to the end user. Indeed, mainstream identity providers support SSO for many protocols and across them for several years now.

That's the mindset I had when I was approaching the SSO configuration in Azure B2C tenant. It ended up being a much more cumbersome task than I have expected, hence this post. While in a way it is a regurgitation of information already available on the subject on the Internet, I hope that the description of my "SSO journey" that follows will help reducing the research and experimentation time associated with SSO setup in Azure B2C that otherwise may be needed in order to get it working.

Applications and SSO objective

I have two Angular 8 SPA applications hosted independently on two different domains app1.mydomain.com and app2.mydomain.com. I needed SSO between them, so that when a user signs into one, and then browses to another either in the same browser tab or in a new tab, the user should not be prompted for credentials.

Both applications are registered in the same Azure B2C tenant, and use the same policy. Importantly, they only use local accounts for authentication, this was my constraint. I use MSAL library for authentication/authorization. The application is redirecting users to the B2C policy's sign-in page.

What I wish have worked but didn't...

So I have started with using the built-in Sign up and Sign in user flow, also tried Sign up and Sign in v2 flow with same results. If you go to properties of your flow in B2C web UI, there is a Single sign-on configuration setting under Session behavior. I've set it to Policy as I had two applications sharing the same policy, then saved the user flow.

It is when there was still no single sign-on I have realized that I was up for a longer ride here.

What worked, but was the wrong path

MSAL documentation describes the library's support for SSO. There are two ways to indicate SSO intention to MSAL library: by using login hint or session identifier (SID). Obviously the MSAL library supports this because the underlying identity provider (IdP) does, or it would be pointless.

So the idea here is to log in to the first application with user's credentials, then pass the SID or login hint to the second application, and B2C should authenticate the user to the second application without displaying prompts.

Cannot obtain SID from Azure B2C

I tried hard, but could not find a way to get SID value from the Azure B2C IdP. I would think it is a claim emitted by the IdP in response to a successful sign on, which appears to be the case for Azure AD IdP, but I had not much luck with Azure B2C IdP.

Extra call to obtain login hint value

The other option, the login hint I could work with. Just get the login claim from the identity or access JWT token returned by B2C and use it as a hint, right? Well, to my surprise the login claim was not present in JWT tokens returned by B2C IdP configured with a built-in Sign up or sign in policy.

That's OK, we can make an MS Graph profile API call and get our login that way, paying with a few hundred milliseconds of page load time for this. Hmmm.....

MSAL Hurdles

It is logical to start with MSAL-Angular if you are in an Angular application... Unfortunately the library is behind the MSAL core, and when it comes to SSO, and specifically passing on login hint, it just does not work.

While the MSAL Angular is appending the login hint as a login_hint extra query parameter to the IdP call, the core Angular library expects the hint as a property of the AuthenticationParameters object. This results in ServerRequestParameters.isSSOParam() call returning false, resulting in the core MSAL library not understanding the login hints and not attempting to establish SSO.

I had to refuse from relying on MSAL-Angular and interact directly with MSAL core library. This got it to work, but as we will see later on, MSAL-Angular "will be back" on the scene.

Sharing the Login Hint between Apps

OK, if I hardcode the login name as a login hint for the second application, then it works, I get the single sign-on as advertised, (or almost!) Now the challenge is to grab the username obtained upon successful logon to the first application through the Graph API call, and share it with the second application before the user authenticates to it.

Since the apps are on separate domains they do not see each other's state, even if it is in localStorage. Probably the simplest way around this is by using messaging API to communicate between the current window of the first app and a hidden iFrame pointing to the second app, making the latter set the username in its localStorage in response to a received message to use later on as a login hint. Here is an example of this technique.

At this point, the whole process was feeling too fragile and complex to me for what it does: too many obstacles, as if Microsoft was trying to implicitly warn me against this path "hinting" that there was a better way.

PII and Sign Out Concerns

And should I persevere and get over the MSAL-Angular incompatibility, the login hint sharing complexity, and accept the extra time that it takes to make a profile Graph call, I would still face the following issue: the login hint that I am sharing between the applications is what is classified as Personally Identifiable Information (PII). Immediately this becomes a concern from compliance perspective.

Last but not least there is a sign out complexity here: since in the above approach I store the login hint in localStorage, I need to make sure to clear it when a user signs out, or closes her browser tabs.

Under the pressure of the above considerations, which would have turned a seemingly simple identity solution to a needlessly complex subsystem with potential vulnerabilities, I had to look for an alternative.

Custom Identity Experience Framework Policies to the Rescue

Once I've understood that I've exhausted the options available in the built-in policies (or user flows as they are also referred to), I had to turn to custom Identity Experience Framework (IEF) policies.

First things first, to take advantage of custom policies, one needs to follow this Azure B2C preparation guidance word for word to get the environment ready for creation of custom policies.

Next, make sure to configure Azure Application Insights for monitoring B2C custom policies, as otherwise it will be quite hard to troubleshoot them.

Get signInName Claim in Access Token

I was looking for a way to avoid having to make the MS Graph call. I came across this great StackTrace thread, which shows how to emit the signInName claim as a part of access and id tokens for the local Azure B2C accounts.

The detailed instructions in the thread allow adding a signInName claim to the tokens, which is quite helpful. And if you like me happen to hit the following error in process of getting it to work:

Orchestration step '1' of in policy 'B2C_1A_signup_signin of tenant 'xxxxxxxxxx.onmicrosoft.com' specifies more than one enabled validation claims exchange

Then the following thread contains the remedy.

Single Sign-On "Just Works"

Yes it just works as a much welcomed side effect. it was not obvious to me, as the thread was solving a different issue, namely the lack of username in the claims. I did have to modify one line in the SelfAsserted-LocalAccountSignin-Username Technical Profile in TrustFrameworkExtensions.xml (see the highlighted line below):

This is all that I had to do. Now:

There is no need to share login hints and deal with associated compliance risks
There is no need to make MS Graph API calls and deal with latency
MSAL-Angular library "is back in the picture" and can be used again.

Life is good!

MSAL acquireTokenSilent() and Azure B2C Permission Scopes

noreply@blogger.com (Ivan Neganov) — Tue, 12 Nov 2019 18:38:00 +0000

One thing that was not obvious to me when securing an Angular app with Azure B2C tenant had to do with using permission scopes.

Let's say that you have authenticated through loginRedirect(), but need to make a call to acquireTokenSilent() MSAL API from within your SPA app. Perhaps you are writing your own route guard or something... You need to pass an array of scopes to the method call. There are two ways to get this to work:

1. When you register your app in Azure B2C, it creates a scope for it named user_impersonation. You can take its value (https://yourdomain.onmicrosoft.com/your-app-name/user_impersonation) and pass it to the acquireTokenSilent()method as a single-item array. Or you can create your own scope instead...

You may get an error back from the B2C when you call acquireTokenSilent() with this scope: AADB2C90205: This application does not have sufficient permissions against this web resource to perform the operation. To fix it you need to grant admin consent to the scope through the B2C tenant.

2. There is another way. Check out how MsalGuard class is implemented. It calls acquireTokenSilent()with a single-item array consisting of the app's clientId which we've got through the app registration. That works without any additional consents.

So both ways work, but there are important differences between them:

In the former case, we are making a call to the https://yourdomain.b2clogin.com/yourdomain.onmicrosoft.com/yourpolicy/oauth2/v2.0/authorize endpoint and pass 3 space-separated values in the scope query string argument: https://yourdomain.onmicrosoft.com/your-app-name/user_impersonation

openid profile

.

In the latter case, the call to the endpoint is not made at all in my case. MSAL "knows" it is authorized as it has got the access token from preceding call to loginRedirect(). Actually, let's take a look at what Fiddler shows when we call loginRedirect(), specifically I am interested in which scopes it passes on:

In the former case it is https://yourdomain.onmicrosoft.com/your-app-name/user_impersonation openid profile
In the latter case, it is only openid profile

Here is a good description of the meaning of these scopes.

With that, here is my takeaway: MSAL converts the clientId scope we pass in a call to its loginRedirect(), acquireTokenSilent() etc. calls to the openid and profile scopes known to Microsoft Identity Platform. It then also is smart enough to resolve calls for access token locally as long as it is valid.

We can also present our SPA app as an API to the identity platform, create a permission for it, consent it, then acquire token for accessing it. But in a basic authentication scenario such as "is user logged in or not?", there is no benefit in doing so. It may be useful if we have complex permissions in our application and want to be dynamically calling different permission scopes we define for various parts of our application.

Extract and Inspect All SharePoint Solutions with PowerShell

noreply@blogger.com (Ivan Neganov) — Wed, 26 Jun 2019 19:31:00 +0000

Migration or upgrades of SharePoint content databases commonly involve provisioning of WSP solutions. At times you may find yourself in need to search for a particular feature GUID, which is burried somewhere inside of one of the dozens solution files that you have extracted from a farm in question.

If you are on Windows Server 2012 or higher, you can leverage expand.exe command to extract CAB files (WSP files are CAB files). Here is an one-liner PowerShell command to extract contents of your WSP solutions to respective folders:

dir *.wsp | % { New-Item -Type directory -Path ".\$($_.Name.Remove($_.Name.Length - 4))"; expand.exe $_.Name -F:* $_.Name.Remove($_.Name.Length - 4)}

How to use: First place your solutions to a folder, CD to it, then run the above command, which will create a folder per solution extracted and dump its contents in there.

Now you can quickly tell whether the feature Id you are after is among the ones extracted. For example, the following one-liner command will list all feature Ids, Titles as well as paths to Feature.xml files in a table format:

dir Feature.xml -Recurse | % { $path=[system.io.path]::combine($_.Directory, $_.Name); [xml]$doc = Get-Content -Path $path; $obj = New-Object PSObject -Property @{Path=$path; Id=$doc.Feature.Id; Title=$doc.Feature.Title;}; $obj} | select Id, Title, Path

Oh, and almost forgot that this may also be handy: you can use this line to dump all farm solution files to your current directory, once you make sure you are running it inside of elevated SP PowerShell session:

(Get-SPFarm).Solutions | ForEach-Object{$var = (Get-Location).Path + "\" + $_.Name; $_.SolutionFile.SaveAs($var)}

Happy migrating!

Azure AD Authentication and Graph API Access in Angular and ASP.NET Core

noreply@blogger.com (Ivan Neganov) — Mon, 17 Jun 2019 05:27:00 +0000

Wow, it's been quiet here... Enough with the intro ;) and onto the subject, which I find interesting and worthy of writing about...

Consider this scenario, which I think makes a lot of practical sense: a web single-page application (SPA) authenticates users against Azure AD using OpenID Connect implicit grant flow. Then some of the SPA's client-side components make queries to Graph API, while others hit its own server-side Web API.

What follows is highlights from my experience implementing this scenario. These are the packages I was using:

ASP.NET Core 3.0 preview 5

Microsoft.IdentityModel.Tokens v5.3.0

Angular v7.2.5
MSAL for Angular v0.1.2
MSAL for JS v0.2.1

Client-side components obtain access tokens from Azure AD and pass them along with calls to MS Graph API, or to the ASP.NET Web API. The former case is standard and well-explained, while the latter one is less so, and therefore more interesting. ASP.NET is configured to use bearer token authentication and creates user identity, which the rest of server-side logic can then use for its reasoning.

When validating tokens coming down from client components of the application, I used code similar to the one shown below, inside of ConfigureServices method:

// Example of using Azure AD OpenID Connect bearer authentication.
services.AddAuthentication(sharedOptions =>
{
    sharedOptions.DefaultScheme = JwtBearerDefaults.AuthenticationScheme;
}).AddJwtBearer(options => 
{
    options.Authority = "https://login.microsoftonline.com/11111111-1111-1111-1111-111111111111";
    options.TokenValidationParameters = new TokenValidationParameters
    {
        ValidIssuer = "https://login.microsoftonline.com/11111111-1111-1111-1111-111111111111/v2.0",
        ValidAudiences = new string ["22222222-2222-2222-2222-222222222222"]
    };
});

where 11111111-1111-1111-1111-111111111111 is the tenant Id, and 22222222-2222-2222-2222-222222222222 is the client Id of application registration.

One of motivations for this post was the issue I kept getting with this authentication logic. I originally also had an extra property setting on TokenValidationParameters object:

IssuerSigningKey = new X509SecurityKey(cert)

The above line assigns a public key encoded as X.509 certificate chain to be used later to decode a signature applied to the token by the Azure AD. Check out the always-excellent insights from Andrew Connell, where he explains the need for the key-based signature checks when validating tokens and a mechanism to obtain the public key (the "cert" in the line of code above).

My logic was however failing with the error IDX10511 "Signature validation failed. Keys tried...", and my research into nuances of RSA algorithm implementation in ASP.NET and JSON Web Token encoding was fruitless until I have found this thread on GitHub, and this related thread.

It turned out that my signature validation was fine, although the line above was not needed, because the library I rely on for token validation, the Microsoft.IdentityModel.Tokens, takes care of it automatically by making a call to obtain the Azure JSON Web Key Set, and deserializing response to .NET public keys used for signature checking.

The actual wrong part had to do with my usage of access tokens: an access token obtained for a Microsoft Graph API resource happens to fail signature validation when used against a different resource (ASP.NET custom Web API in my case). This fact and that ASP.NET error message here could be improved is covered in detail in the above GitHub threads.

What I had originally, which I refer to as "naive" configuration, is shown on figure below.

On this image, I have an Azure app registration for my web application, requesting some graph permission scopes. Then during execution I acquire token on the client (1), use it when sending requests to Graph API (2), but fail to do the same against my ASP.NET Web API (3), which results in IDX10511 error.

What is interesting here, is that:

This setup kind of makes sense: I have an app, it is registered, and it wants to use access token it gets from Azure to let its own API "know" that a user has logged in.
The problem can be fixed by sending an ID token instead of access token in step (3). OpenID Connect protocol grants ID token upon login, which signifies authentication event, while access token signifies authorization event. ID token's signature is validated without errors, and ASP.NET creates a claims identity for the signed in user.

What is not good about this design, is that the ID token is not meant to be used in this way. While one can choose to deviate from protocol's concept, it is not wise to do so without a compelling reason, since all tooling and third party libraries won't do the same.

Specifically, here are the problems I could identify with the above design:

OpenID Connect, and OAuth 2.0 by extension use different grant flows depending on types of clients used. For a web browser, it is Implicit Grant, then for a server-side client it is one of other flows, depending on a scenario. We are in essence trying to use a token issued to one audience, when calling another audience. In my example, the Angular SPA and Web API are on the same domain. If they were hosted on different domains, this issue would have been more obvious.
Microsoft uses an OAuth 2.0 extension, and on-behalf-of flow (aka OBO flow), which will be useful in scenarios when we decide to have our ASP.NET Web API enhanced by having it also access Graph API or another Microsoft cloud API. The current setup is not going to work with the OBO flow.

The figure below shows an improved design:

This time we treat server-side Web API as a separate application as far as Azure AD is concerned. We do have to make our SPA application acquire access token twice as shown in calls (1) and (3), doing it so for each audience: once for Graph, and second time - for our own API. Then both calls to Graph (2) and to our own API (4) succeed.

Also, this design is fitting well with the OAuth paradigm. In fact, by the time we decide to augment our Web API and start making on-behalf-of calls from within it, we have already implemented its "first leg".

Lastly, a couple notes about the MSAL Angular configuration. Here is mine:

    MsalModule.forRoot({
      clientID: environment.azureRegistration.clientId,
      authority: environment.azureRegistration.authority,
      validateAuthority: true,
      redirectUri: environment.azureRegistration.redirectUrl,
      cacheLocation: 'localStorage',
      postLogoutRedirectUri: environment.azureRegistration.postLogoutRedirectUrl,
      navigateToLoginRequestUrl: true,
      popUp: false,
//      consentScopes: GRAPH_SCOPES,
      unprotectedResources: ['https://www.microsoft.com/en-us/'],
      protectedResourceMap: PROTECTED_RESOURCE_MAP,
      logger: loggerCallback,
      correlationId: '1234',
      level: LogLevel.Verbose,
      piiLoggingEnabled: true
    }),

MSAL will automatically acquire access token right after an id token is acquired after calling MsalService.loginPopup() with no scopes passed in as arguments. Commenting out or removing the consentScopes config option results in MSAL defaulting to using apps's client Id as an audience and returning a somewhat useless access token with no scopes in it.

I did this as I wanted to explicitly request separate access tokens for Graph and for my Web API. The way to do it is through passing scopes corresponding to an application to a call to MsalService.acquireTokenSilent(scopes). I am now thinking of changing it to pass the scopes of the Graph API initially, so that my first access token is useful. For the second one I have no choice but to call the MsalService.acquireTokenSilent(myWebApi_AppScopes)again.

Thoughts about Building Multilingual Publishing Site on SharePoint 2013 - Part 3 of 3

noreply@blogger.com (Ivan Neganov) — Wed, 26 Jun 2013 04:33:00 +0000

This is a third part of the series describing my experience in planning and building a multilingual WCM site on SharePoint 2013.

Part 1 discusses basic requirements and architecture of the authoring site.
Part 2 focuses on managed navigation for the authoring site.
This part discusses Cross-Site Publishing (XSP) and publishing sites.

A Quick Introduction to Cross-Site Publishing

Microsoft has released a blog series describing in detail how to set up a fictitious Contoso site leveraging XSP. Contoso is a product-centric web site using a concept of category pages and catalog item pages to illustrate the applicability of the XSP. A product sold at Contoso electronic store can be categorized by a hierarchy of categories. For example, a specific product, the "Datum SLR Camera X142", is categorized as Cameras >> Digital Cameras. There are only two kinds of pages that we need here: a product catalog item page showing product information, and a category page showing products matching the category. So, if you pick "Cameras" category from the navigation menu, you will see products qualifying as cameras; if you pick "Digital Cameras" - you will see a narrower set of products qualifying as digital cameras. Regardless of which category in the hierarchy you pick the principal is the same. So we need to figure the current category and render matching products for it. Category pages do exactly that. Next, you click on a specific product listed by the current view of the category page, and then the product details are rendered by the catalog item page, which accepts a unique product identifier. And so you can surface the entire product database on the SharePoint site by using just two pages - a category page and catalog item page.

There are two "magic ingredients" here. Firstly, there is the ability to publish and consume lists as catalogs. What this does is it creates a search result source in the consuming site, and optionally pins the terms from the term set used to categorize the catalog as navigation terms to the navigation term set of the site consuming the catalog. Also behind the scenes, the Microsoft.SharePoint.Publishing.Navigation.TaxonomyNavigation class callable from SharePoint.Publishing.HttpPublishingModule class "becomes aware" of how to detect requested URLs constructed of categories and unique identifiers aka "Friendly URLs" on this site, and route them to appropriate category or catalog item pages. Secondly, it is that the pages "can be made aware" of their current context and use this knowledge when issuing search queries for catalog items. This ability is there thanks to a set of well-known search query variables available to developers or information workers placing search web parts on pages.

Cross-Site Publishing as a Web Content Management Approach

Things started to look a bit confusing when I tried to apply the XSP concept to the site publication scenario that I have described in Part 1. Here are the hurdles I faced:

1. A web site page is hard to fit in the product-centric model described above, because a page can often be both a category and a catalog item at the same time. Commonly sites use section landing pages, the ones the top navigation menus point at, which contain content and at the same time act as logical parents to child pages down the site map hierarchy. Let's say we make a landing page to be an XSP catalog category page. Then, according to the product-centric model we access its child pages as we would access catalog items - by clicking on a list of these pages rendered by our landing page. Well, this is usually not how information architects expect users to navigate their web sites, unless they are online product stores. What we often need instead is navigation menus, the familiar top navigation and current navigation controls to let us access all pages on the site.

2. Now think for a second about the managed navigation discussed in Part 2 and the requirement we had about maintaining the fidelity between authoring and publishing sites. Every page on the authoring site has a corresponding term used for navigation and forming a friendly URL. Because we use the managed navigation on the authoring site, we get our top and current navigation menus populated by these terms. We want the same behavior on the publishing site. This brings the question: "Which pages should be the category pages, and which ones should be the catalog item pages?" If we were to follow the classical product-centric approach, and designate pages corresponding to the top-level nodes as category pages, and leaf pages as item pages, we would lose the drop-down menus on the publishing site:

When I say "designate" I mean that we "tag" the pages with the terms or, in other words, assign term values to corresponding pages by setting managed metadata fields of these pages.

3. On another extreme, if we tag each and every page on the authoring site with a term, then when we consume the pages library catalog, all our pages will logically become the category pages. How do we now render the catalog item information?

We resolved the confusion by tagging all the pages on the authoring site, effectively making all of them to become the category pages on the publishing site, and modified their page layouts to make them simultaneously act as catalog item pages. This looked like a promising strategy, because by making all of our pages into the category pages, we would get exactly the same top and current navigation elements as on the authoring site, for free. The only thing left to do was to make the category pages render catalog item information - should be doable on a search-driven site.

If you examine an automatically created content item page, you will see that it is based on a page layout, which in turn leverages Catalog Item Reuse (CIR) web parts. Each CIR web part renders contents of a specific managed search property specified in the SelectedPropertiesJSON property. Provided that the managed properties correspond to the columns on the authoring site, this results in rendering the same content on the publishing page as on the authoring page. Below is an example of a CIR web part rendering a value of a managed property corresponding to the Page Content column.

Note that the value of UseSharedDataProvider property should be set to True on all CIR web parts on the page except for one, which serves as a data provider for the rest of them. All CIR web parts on a page form what’s called a query group, with one CIR web part acting as a data provider, and the rest of them – as data consumers. The data provider CIR web part in addition to SelectedPropertiesJSON property has DataProviderJSON property set as shown in the following example.

The value of DataProviderJSON property sets properties on objects of DataProviderScriptWebPart type. The key properties here are:

QueryTemplate – defines keyword search filtering criterion using a query variable such as {URLTOKEN.1} in the above example.
SourceID – a unique ID of the search result source corresponding to the catalog being consumed. The search result source is created automatically when the catalog is connected to.
Scope – a URL of the catalog source list.

An easy way to get started with the CIR web parts is to let SharePoint auto-generate catalog Category and Item pages and page layouts when connecting to a catalog, then harvest the web part markup from the auto-generated page layouts.

So conceptually the problem is solved now: we can create our own page layout and then a category page based on it, and configure CIR web parts to select the managed properties of interest from the catalog. If the markup of the page layout and master page is the same as on the authoring site, then CIR web parts essentially replace the content fields, and the publishing page appears visually identical to its authoring counterpart, and the navigation is working, except that only a single page exists on the publishing site. Pretty cool.

Now let's consider some practical aspects of getting the XSP-based publishing site up and running.

XSP Navigation Term Set and Vanity Names

To properly publish pages library as a catalog we need to designate a field uniquely identifying each page, and a managed metadata field used for page categorization. We have come up with two fields for this purpose and defined them at the site collection level in order to make sure the required managed properties get created automatically when we do a full crawl:

Vanity Name - this is a text field where we enter a unique friendly name of each page.

XSP Category - is a managed metadata field using a new term set named XSP Navigation.

On the authoring site we are creating and managing a master term set, which is applied to the source variation, then its terms are getting re-used and translated on the target variations. We certainly want to avoid duplication of the effort required to manage the terms when we tag our pages, but we cannot reuse the existing master term set, because SharePoint complains about it being already used when we consume the catalog. We need a new term set, and so we just pin top-level terms with children from the master Site Navigation Term set. Another important thing we need to do is to create a Root term in the new XSP Navigation term set so that we could hook up to it when we consume the catalog. This is illustrated on the figure below:

The convention we have used was that the value of the Vanity Name field must match the value of the XSP Category Node for any given page on the site. This is important because it allows us to structure the query we issue from the data provider CIR web part as follows:

VanityNameOWSTEXT:{URLTOKEN.1}

This query means "select items where the value of the Vanity Name managed property contains the last segment of the current URL". So for the URL http://www.softforte.com/softforte-corporate/about-us the {URLTOKEN.1} == "about-us", and due to the convention the Vanity Name == "about-us" as well. The page is a category page in the XSP terms, and has a managed term created when we were consuming the catalog, which points at the URL /softforte-corporate/about-us. This is exactly how we have hacked the category pages, giving them the ability to act as catalog item pages at the same time.

Navigation Translation on Publishing Sites

The terms used on the publishing sites are already translated by virtue of setting their labels for each culture, and customizing term-driven page settings down the re-use chain originating from the authoring source variation's navigation term set. The challenge however exists with regards to selecting the proper translation for the corresponding publishing site.

In order to localize content and navigation of a variations-based site you do not need to install a language pack. The locale such as fr-CA is selected when a variation label is created. This is different for a publishing site, which does not rely on variations. The only way I found to get terms to translate to French was to install the French language pack, then create a site collection using the French target language template. In order to still manage the site in English, the Alternate Language can be set under Site Settings >> Language Settings. This forces the SharePoint to read the language preferences the browser is sending to it. So if my preferred language is English, I will see the the managed navigation in English, if it is French then it will automatically use the French term translation if one is available. The limitation here is the language packs - we can only publish in the languages supported by language packs, at least to the best of my knowledge.

Do I Really Need a Single Page on My Publishing Site?

Since most of the real-life sites use multiple page layouts, the same page layouts need to be mirrored over to the publishing sites and CIR web parts configured there instead of content fields. Then, if all the content on the authoring site is static, i.e. there are no web parts, just the content fields, then we can simply create catalog category pages, one page per page layout, and that would be all we need to do.

Real web sites use web parts to render dynamic content. Since web parts are not stored in index, cross-site publishing will not render them. This means that the web parts need to be re-created on the publishing site. Needless to say that since publishing and authoring sites have different security requirements, the web parts would need to be created differently on publishing and authoring sites.

Where ever possible, we should therefore utilize Content Search (CS)web parts, which would get the content from the SharePoint index just as the CIR web parts do. We were able to meet 100% of requirements to the dynamic elements on the site using CS web parts, thanks to their great flexibility. When a CS web part is used on an authoring site page, it runs in a different context than when it is
used on a publishing site. Therefore it needs to be adapted to the publishing site, and simply copying it over won't work. Most of the time the adaptation is quite simple however, as the same managed properties are available on both sites, and therefore the search queries are similar.

How big of an issue is the fact that the dynamic elements on content pages need to be recreated on a publishing site? We found it to be quite manageable in our case. The level of work duplication was minor for us, although it depends on how many dynamic web parts are you planning for. A general corollary here is you need to plan thoroughly your pages in advance, to get the most out of the XSP.

So while you could only use a single category page to render all pages based on a specific page layout, once you add dynamic web parts to the mix you need to create a new page in the publishing site for each corresponding authoring page with the web part. If the same web part is used on multiple pages, then it makes sense to embed it into publishing page layout directly in order to reduce the number of publishing pages you need to create.

Here is an example of how an authoring page corresponds with a publishing page layout and a publishing page:

On the above illustration, the legend is as follows:

Purple color - content fields;
Gray color - items inherited from a master page or page layout;
Green color - web parts;
Blue color - navigation control.

The Inconvenient Result Source ID Value

In the code snippet above the DataProviderJSON property included a SourceID parameter, which is the ID of a Search Result Source that is automatically configured when a catalog is consumed on the site. This property controls the scope of search queries issued by the web parts participating in the query group by adding filtering to select items from the list with a specific ID and source web site URL. If you let the SharePoint create catalog item and category pages automatically when connecting to a catalog, this Source ID would be embedded in the page layout. Pretty inconvenient, especially as each time you re-connect the Source ID changes, and you cannot control it by exporting and importing site search configuration.

After researching alternatives, we ended up using a provisioning script, which would first provision the publishing page layouts, then determine the result source ID, then check out the layouts, replace the ID in them, and check them back in... like I've done in the old days...

The Big Picture

Getting back to our business requirements described in Part 1, by utilizing the XSP approach we have created 3 publishing sites, one for each variation. The publishing site corresponding to the source variation is not accessible to the Internet users, but the information workers can preview English-level content on it exactly how it would appear to anonymous visitors on the live site. This architecture required us to turn Pages libraries on each variation label into catalogs, and consume the catalogs from the corresponding publishing sites, meaning that we had to have 3 different setups of the publishing page layouts, pages and catalogs. This may seem like a lot, yet since the pages set up on the publishing sites are not created by information workers, but instead are viewed as a part of the application provisioning exercise, and since they are almost identical across the three publishing sites, it was a quite reasonable decision.

Conclusion

The XSP is new, and it can be difficult to depart from the product-centric model used in Microsoft demos. Practically it all gets down to retrieving content from a search index, something we've been doing for a while with SharePoint. The XSP takes the concept to the next level of manageability and implementation convenience, and does not require us to write compiled code any more to take advantage of it. And of course SharePoint sticks to its traditions: planning is key to properly setting up the publishing sites and minimizing duplication of effort. As a result we get modern-looking dynamic publishing web sites and an enterprise-class content management process.

Thoughts about Building Multilingual Publishing Site on SharePoint 2013 - Part 2 of 3

noreply@blogger.com (Ivan Neganov) — Wed, 19 Jun 2013 21:27:00 +0000

This is a second part of the series describing my experience in planning and building a multilingual WCM site on SharePoint 2013.

Part 1 discusses basic requirements and architecture of the authoring site.
This part focuses on managed navigation for the authoring site.
Part 3 discusses cross-site publishing and publishing sites.

Term Set Organization

SharePoint 2013 Managed Navigation is at the heart of our solution. We have defined a global term set containing the navigation hierarchy. The terms at the top level correspond to the items at the top navigation bar of the site:

So there is no single root or parent term for the global navigation term set, rather - there are several top-level terms. It is important to note that every single page needs to have a corresponding term in the term set pointing at it:

Next, on the Site Settings >> Navigation configuration page we set the managed navigation to point at the global navigation term set for the source variation label. The root site of the site collection is still using structured navigation. This site is not going to be exposed publicly and is logically above the top level the Internet users will see.

Terms Sharing (Reuse)

When variations are created by clicking on Site Settings >> Variation Labels >> Create Hierarchies the background job will automatically create site collection-scoped term sets and re-use the terms from the source term set to the target term sets. Going forward, when an information worker publishes a new page to the source variation label, the Variations Propagate Page Job Definition background job would propagate the page, and if the page is pointed at by a term, it will re-use this term from the target variations site (Term reuse scenario A). If a page is published first and propagated to target variations, and a term pointing at it is created at a later time, then the term will not be reused on target variations and would need to be manually reused by selecting Reuse Terms option on the parent term of the term to be reused.

Terms can also be reused by manually pinning top-level terms instead of relying on the variations propagation job. In this case, the terms created automatically by variations can be removed. Next, terms from the source term set can be pinned along with the children using the Pin Term With Children menu option on the term set used for the target variation label's navigation top-level (Term reuse scenario B). The advantage of scenario B over scenario A is that regardless of when a page is published and propagated, the term will always get reused. The advantage of scenario A is that only terms for published pages get reused, which may better align with a publishing business process. The good thing is that you can mix both scenarios in a single term set as they are term-specific.

Terms Translation

In order to translate a term you need to change a value of current language in the Language drop-down box on a GENERAL tab, and enter a translated value for the Default Label. Regardless of which term set you do it on (global term set, the one for the en-CA label site or the one for the fr-CA label site), it updates the source term's value, i.e. the one belonging to the global term set. Below is an example of translating a term to Russian:

There is an interesting side effect of term translation. If a term is translated prior to being reused (scenario A) or pinned (scenario B) then the managed URL of the propagated page is created based on the translated label. Here is an example of how this looks for a Russian variation label:

The fix is easy: selecting the Customize check box does it. In case of the scenario B, the friendly URL to a page in a target variation will return 404, since Target Page Settings on a TERM-DRIVEN PAGES tab for the target term are not defined. The URL of the target page needs to be entered manually.

So as you might have noticed, terms connected via pinning or term reuse share certain properties, such as Default Label, which can only be changed in one place - on the source term, while you are allowed to independently customize term-driven pages settings. This opens up a possibility of having locale-specific URLs, especially in languages utilizing Latin alphabet: a page in source variation could have one term-driven URL, and a replica of the page in the target variation could have a completely different term-driven URL. This improves readability of the site and supports SEO.

On our project we went ahead with the Scenario A, assuming our information workers would get training to make the following logical workflow happen:

The user creates a page in source variation;
The user creates a term in global term set;
The user points a term at the new page;
The page undergoes approval process and gets published.
Variations Propagate Page Job Definition timer job picks up the page and creates a replica in target variations. It also reuses the term pointing at the page, and correctly points reused term(s) at the page's replica(s).
The term can be translated now without a harm to the term-driven URLs.
User can optionally change the term-driven page URLs for the target variations' pages.

Should the workflow not be followed for whatever reason, the terms can always be manually reused, and incorrectly translated URLs fixed as shown above.

Search Indexing and Managed Navigation

I have noticed that a term needs to have either the Show in Global Navigation Menu or the Show in Current Navigation Menu checkbox checked (or both of them) in order for the search to be able to index the page corresponding to the term. If the term is neither on global nor on current navigation, and Hide physical URLs from search field is set to True for the corresponding page, then the page will not get indexed. For the pages we did not want on the navigation, but still needed to be in the index we were able to workaround this behavior by selecting the Show in Current Navigation Menu checkbox, but using a page layout that does not render a current navigation control.

Global versus Local Term Sets

One last note about using the global term set. We could have used a site collection-scoped term set (aka local term set) instead. The strategy was to keep all terms used for navigation in one place and then reuse them. Even for the purpose of cross-site publishing it is not necessary to have a global term set. Still we have chosen a global term set on the basis of a "why not?" approach. It gave us greater flexibility and eliminated potential issues having to do with local scope.

Thoughts about Building Multilingual Publishing Site on SharePoint 2013 - Part 1 of 3

noreply@blogger.com (Ivan Neganov) — Mon, 17 Jun 2013 12:18:00 +0000

Publishing approach has drastically changed in SharePoint Server 2013, and the Web Content Management (WCM) solution and infrastructure architecture approaches need to be revised from the ground up as a result. The guidance suggests 2 methods of content publishing: Author-in-place and Cross-site publishing, and provides a decision flowchart for choosing one over the other. Although content deployment publishing method is still supported, it is no longer in the list of recommendations. While the reasons for this are unknown to me, at least I know that the content deployment does not work with the new and sought-after managed navigation feature. With the above in mind, how do you practically approach construction of a multilingual Internet-facing SharePoint-based WCM site utilizing clean URLs and a content publishing process suitable for an enterprise?

This blog post is the first one in a 3-part series dedicated to the aspects of a multilingual WCM site architecture. It sets the context and describes the architecture of the authoring site.
Part 2 describes managed metadata navigation on the authoring site;
Part 3 describes the architecture of publishing sites and utilizing the cross-site publishing.

Setting the Context

At Navantis, we needed to build a bi-lingual Internet presence publishing site for our Canadian customer, based on an on-premises installation of SharePoint Server 2013. The key requirements at a high level were as follows:

The content of the site should be available in French and English;
Although the authors could publish French version of a piece of content independently of its English version, most of the time it was required for the two versions to go live simultaneously, such as in cases of company news announcements;
The site had two kinds of content - "non-volatile", updated rarely, and "volatile" corporate blog content updated monthly when new postings were published;
Some pages in addition to containing content, such as text and images, needed to have dynamic elements pointing at other pages, for example products widget, blog postings widget, home page image slider widget, etc.;
English and French versions of the content would be hosted on two sites addressed by two different DNS host names;
Authors and approvers should be able to preview the content with high fidelity to its production appearance, and follow a translation and approval business process before exposing it to the anonymous Internet users.

Pretty typical, I would think...

General Architecture

So, we have started with planning an application utilizing Cross-Site Publishing (XSP) method, and studying Microsoft guidance including the case study of Mavention WCM site. I must point out that the case study and the excellent blog of Mavention's Waldek Mastykarz were very much helpful to us. Yet our case was not identical to the Mavention's site. The key differences were as follows:

We used managed navigation on the authoring site. This was done in order to provide the fidelity of the look and feel between the authoring and publishing sites to the authors and approvers. Here we have willingly deviated from the guidance recommending structured navigation for authoring sites that leverage XSP. Another reason for this choice was that it provided for an easy fallback to the author-in-place publishing method, should there be implementation issues with the XSP method.
We were hosting authoring and publishing sites in two separate web applications in order to separate them physically. The customer 's security policy allowed placing both authoring and publishing sites to the perimeter network, and so we did.

Assets Site

Some of the key architecture decisions to make have to do with where to store and how to publish binary assets created by information workers, such as images and videos. Search index does not store binary content. This means that in an authoring-publishing setup the images need to be either shared or copied between authoring and publishing sites. While the images that are a part of the branding can be copied over during site provisioning, this cannot be done with the images used by the information workers when creating content pages.

The two alternatives I see here are developing a solution for synchronizing the images between the authoring and publishing sides, and using a shared assets site. The synchronization solution could theoretically leverage content deployment between source and target site collections for the images, although the information workers would need to maintain "dummy" pages referencing all the used images so that they could be picked up and transferred over by content deployment. We have chosen the same approach as in the Mavention case study - a shared assets site. It is important to note some consequences of this decision:

Unless we keep authoring and publishing sites in the same web application and use path-based site collections, we have to use absolute URIs when referencing the assets (starting with protocol identifier, ex. http://www.softforte.com/styles/softforte.jpg). This was our case based on requirement #5 above.
Assets site would need to use author-in-place publishing method and allow information workers to have authenticated access to the assets site, while also being available publically to anonymous users.
If you use host-named site collections for your publishing sites, which are a general recommendation on the grounds of improved scalability, you further limit the available options: with host-named site collections there is only one security zone, and securing authenticated traffic with SSL becomes problematic.

So we have chosen to use the unsecured HTTP channel with NTLM authentication for the assets site.

Blogs Site
We chose to keep blog postings in a sub-site of the main authoring site because only one Pages library can be used per web site, and we wanted to have a separation between the two libraries in order to be able to manage publishing workflows for pages and blog posts independently of each other. Since we relied on search web parts to surface content from the blogs, it was not a problem to retrieve blogs content regardless of its location. The only thing to be aware of is configuration of navigation on the blogs site - it needed to be inherited from the parent site in order to maintain its navigation context.

Variations and Publishing Process

In order to meet the requirement #2 asking for pages in both languages to be published simultaneously we have taken a rather elegant approach: we have used 3 variations: en-US (source English), en-CA (target English), fr-CA (target French). The source variation is not visible to the Internet users, but still allows to approve and then publish content in English language, an action that triggers propagation of published updates across the variations. Then in case of the en-CA language variation label the content simply gets approved to become published, and in case of fr-CA language it undergoes a translation process then gets approved and published. Such approach lets us publish French and English versions of a page quasi-simultaneously, leaving it up to the approver to determine the interval between approving French and English publications.

Customizations and Search-Driven Content

We have built the site 100% without having to write custom server-side components. All customizations were client-side only, 80% of them were the design templates for working with the Content Search web parts. The latter were used universally for almost each widget from a page with a dynamic list of blogs to a widget surfacing blog postings on select pages, to a content rotator with sliding images on the home page. After solving provisioning issues thanks to Chris O'Brien's blog post, we were able to speed up deployment cycles and significantly improve the development process. After all, it wouldn't be an exaggeration to state that the site was 100% search-driven.

We have also used traditional site search box and a search results page. For a relatively static web presence site we have opted to not use a search center, and instead use a custom search results page. We wanted to be able to search for blogs separately on the blogs listing page. There we have used content search web part, and a search box web part connected to it, and made the query to filter out anything but the blog postings. The global search results page would include a wider set of results, and allow user an option to switch to blog results only. I have earlier described how to configure this functionality. Because of our use of variations, each variation label had its own search results page, with a query scoped to only content residing in that variation label site.

With all the advantages of surfacing the content in content search web parts, I feel that the main drawback has to do with the obscurity of the content selection method, being the search query issued from a content search web part. This obscurity slows down development by making troubleshooting search queries or search results issues more difficult. The benefits clearly outweigh the complexities, and there are several tools at our disposal: firstly the content search web parts themselves with the excellent ability to do query results previewing; secondly there is a Search Query Tool released by Microsoft; lastly there is a REST web interface which can be used also to troubleshoot the queries using a browser. Still we cannot fully get away from the high complexity of the search sub-system, and from this point of view it is reminiscent to me of the past experience writing search-driven web parts in previous versions of SharePoint. If you compare complexity of troubleshooting your search queries with troubleshooting of equivalent SQL queries you will know what I mean, so just allow some extra time for getting it right before you start adding a lot of content and the time of full crawl would increase dramatically. The less time it takes to do a full crawl in your development environment the sooner you will get the search-driven functionality to a solid state.

One search-related issue we have encountered is worth mentioning: the pages we were authoring had Hide physical URLs from search fields set to False by default. Logically, we were getting duplicated sets of results in return to some of our search queries. Setting this field to True followed by re-indexing the content, only shows term-driven URLs in search results - duplication problem solved!

Master Page, Page Layouts and Branding

In general we have used a pretty standard approach that is in principal not much different from the one used in SharePoint 2010. The new Design Manager helps by letting us use HTML editor of choice, and not rely on the SharePoint Designer. The cost of this convenience was that we had to provision both *.html and *.aspx versions of our page layouts using features. When only the *.html versions are provisioned by a feature their conversion to *.aspx files isn't triggered automatically. Still we needed both of the kinds of files as during development it is important to have identically configured development environments that are ready for branding work. Exporting and re-importing a design package didn't quite work for us as we noticed this process changed columns on intrinsic content types we relied upon - a topic that deserves a blog post and a research on its own.

The site had responsive web design, which we have selected over the device channels option since it allowed us to stay with the same set of master pages and page layouts, and change only the style sheets with some minimal JavaScript support for the changes in navigation at smaller view port sizes. This "modern" style not only impacts the design of the master page and page layouts, but also affects the design of the content, in general making it more complicated. In our case we have provided sample "Lorem ipsum" content for all of the default pages. The sample content was intended to demonstrate to the information workers by a way of example how to structure the real content, and which CSS classes to apply to it.

The next topic, managed metadata navigation, is another key part of a WCM site architecture. It is described in the Part 2 of the series.

Filtering Items by Date Range using SharePoint 2013 REST API

noreply@blogger.com (Ivan Neganov) — Sun, 09 Jun 2013 14:11:00 +0000

For some reason I always had extra difficulty when filtering results by dates in SharePoint, especially when I needed a "between" filter. This time I needed to render a calendar on a page accessed anonymously. I used a FullCalendar jQuery plugin for this, and the SharePoint 2013 REST API. In my implementation when the end user changes a month in the calendar, it calls a function passing it start and end dates of a date range. The function then needs to retrieve JSON from SharePoint corresponding to the date range.

I started with turning on Team Collaboration Lists feature on my publishing site and creating the Calendar list. Then I have got my REST endpoint, this time around with a "greater than or equal to" filter:

http://devserver2012/_api/web/lists/getbytitle('events')/items?$select=Id,Title,Description,EventDate,EndDate,fAllDayEvent&$filter=EventDate ge datetime'2013-06-08T00:00:00'

The reason for my blog post is that the above RESTful call doesn't work. Research has pointed out a few things:

I am not the only one. Other people have had the issue. Rob Windsor's comment was quite useful to me.
The MSDN guidance has a table of supported filtering operations. First of all, it incorrectly capitalizes the numeric comparison operations. For example, instead of "Ge" operator which does not work, I was using "ge" which does work. Second of all, the references pointing back at www.odata.org were unfortunately broken there, at the time of this writing at least.
Later I've learned that my filter actually works with a Boolean "and" operator as I will show below. Does this make it unsupported? It probably does, according to the mentioned MSDN article, although the article itself clearly needs a revision.
OData documentation is your friend. I find it hard to read, yet it is ultimately helpful. You can see query options supported by OData in section 4.5 on this page. Notice how the operators are all lowercase in the examples. The formatting of dates in filter queries can be found here. This is where the datetime'2013-06-08T00:00:00' comes from. One interesting detail, although the format spec does not indicate this, the following works just as well: datetime'2013-06-08T00:00:00Z'. Apparently the ISO-8601 is still honored.

Getting back to my original REST call - what "fixes" it? If I were to use ListData.svc instead then I'd get the following:

http://devserver2012/_vti_bin/ListData.svc/Events?$select=Id,Title,StartTime,EndTime,AllDayEvent&$filter=StartTime ge datetime'2013-06-10' and EndTime lt datetime'2013-06-12'

The above call actually works, although it ignores time portion when filtering. You can see that I get my "between" filter in there, also the column names are different - it uses display names of the columns with removed spaces. Also notice how the date values are formatted, actually conflicting with the OData format spec. In general, if you examine the Atom results returned, you will also find that date formatting is relaxed (no "Z" Zulu time indicator). So what! We can celebrate now... Well, almost - the call to ListData.svc won't work for anonymous users. Since this was a deal breaker to me, I kept on looking. Turning to the ULS log shows that the error message "The field 'EventDate' of type 'DateTime' cannot be used in the query filter expression." comes from here:

Microsoft.SharePoint.SPListItemEntityCollectionCamlQueryBuilder.CheckFieldRefUsage(SPField field, FieldRefUsage fieldRefUsage)

With help of Reflector we see that the SPField.Filterable property is logically enough the driving force here. Then using SharePoint Manager to examine the value of this property for the EventDate ("Start Time") field of the Event content type we of course find that its Filterable property is set to false, and this is what is causing the error message to be returned. I should mention that there is little value in working around this behavior by using calculated fields - they are also rejected as filters by this method. To prove that it actually works in principle I've created a DateTime column from scratch and named it "Date3". Here is how my REST call now looks like:

http://devserver2012/_api/web/lists/getbytitle('events')/items?$select=Id,Title,Description,EventDate,EndDate,fAllDayEvent,Date3&$filter=Date3 ge datetime'2013-06-13T00:00:00Z' and Date3 le datetime'2013-06-14T00:00:00Z'

This works when accessing the list anonymously. Problem solved and lesson learned: do not rely on the OOB Calendar list for anonymously accessible events, or be prepared to add your own custom DateTime columns to that list.

At the time of this writing I was running SharePoint Server 2013, Enterprise CAL, March 2013 update.

Customizing Search Navigation on a SharePoint 2013 Publishing Site

noreply@blogger.com (Ivan Neganov) — Thu, 30 May 2013 05:24:00 +0000

Managed metadata-driven navigation in SharePoint 2013 lets us use clean and implementation-independent URLs, which are often required in WCM applications. I was working on a web presence site for my customer, which used clean URLs. A simple search interface was used on it: a search box on a master page, and a search results page containing search results web part, and a search box web part. I wanted that search box web part to have a drop-down list of options so the Internet users could choose whether to search the entire site, or just the blogs.

First of all, in the past with SharePoint 2010 and 2007 we were using scopes to achieve this effect. It works differently in SharePoint 2013. Under Site Settings >> Search Settings you need to define the links to different destination pages:

Fig. 1 - Site Collection Search Settings

These links would point at two different search results pages and pass user's search keywords as a query string to these pages. In my case the first link "All Results" was pointing at a site search results page, the second one "Blog Results" - at a blogs listing page.

Second of all, we need to configure the search box web part. If you want to use the drop-down menu in the search box residing on the master page then you simply need to select the option titled "Turn on the drop-down menu inside the search box, and use the first Search Navigation node as the destination results page." on the Fig. 1 above. In my case I wanted no drop-down menu there, but needed one on the search results page. This is done on the search box web part as shown on the Fig. 2.

Fig. 2 - Search Box ToolPart showing how to configure the drop-down menu

That's pretty easy and gets you what you need. There is one problem however. Look at the Search Settings page illustrated on Fig.1 - the URL is pointing at an ASP page, which we certainly would like to change from /auth/en-ca/Pages/Search-Results.aspx to something like /auth/en-ca/results. Should be simple, especially as exactly this is done under the "Which search results page should queries be sent to?" section on the same page. Well, turns out that it does not work if you attempt doing it through the UI - it will complain about incorrect URL format unless you give it a page reference.

These links are referred to as "Search Navigation nodes" for a reason. If you open PowerShell and check out the navigation of your current web, you will see the new property of Microsoft.SharePoint.Navigation.SPNavigation type named SearchNav there:

Fig. 3 - SearchNav is a new property of the SPNavigation type in SharePoint 2013

OK great. The next obvious thing to do is change the URL of the "All Results" Search Navigation node from /auth/en-ca/Pages/Search-Results.aspx to /auth/en-ca/results using the PowerShell. Guess what - it would silently not update the URL property.

The way I have got it to work was by creating the navigation node entirely from PowerShell. I also had to use the SPNavigationNode constructor overload accepting a Boolean isExternal flag and set it to $true in order to bypass the URL check, which SharePoint does and for some reason ignores the managed metadata-based navigation URLs. So the PowerShell workaround in my case looks as follows:

$node1 = new-object `
-TypeName "Microsoft.SharePoint.Navigation.SPNavigationNode" `
-ArgumentList "All Results", "/auth/en-ca/results", $true
$node2 = new-object `
-TypeName "Microsoft.SharePoint.Navigation.SPNavigationNode" `
-ArgumentList "Blog Results", "/auth/en-ca/blogs", $true
$web = Get-SPWeb http://devserver2012/auth/en-ca
$web.Navigation.SearchNav.AddAsFirst($node1)
$web.Navigation.SearchNav.AddAsLast($node2)

This gets all my URLs to be "clean", including the ones pointed at from the Search Box drop-down menus. Interestingly, after creating the links through PowerShell you can modify their title, description, or even the URL properties through the UI without issues.

I was running SharePoint Server 2013 with March 2013 CU.

Custom Alerts in SharePoint - Templates or Code?

noreply@blogger.com (Ivan Neganov) — Sun, 28 Aug 2011 16:17:00 +0000

My client needed a customized alert created each time a new issue was added to a standard Issues list. The only difference from out-of-the-box alert functionality was that customized alert Email message needed to have an Issue ID in its subject. I haven’t customized alerts before so I did some research, which showed that:

you can customize HTML of an alert through alert templates and use placeholders to insert values in an Email message.
there is an API letting you to tap into alert message processing pipeline and customize Email message before it is sent out.

There is a lot of information online on the subject, yet I still had difficulties with this seemingly easy problem. The following resources were most useful to me: MSDN (http://msdn.microsoft.com/en-us/library/bb802949.aspx), Albert Meerscheidt’s post about alerts in WSS 3.0 and Yaroslav Pentsarskyy’s post about customizing alerts in SharePoint 2010.

For me empowered with this information the task came down to writing a correct alert template, or so I thought. Take a look at this fragment of an out-of-the-box immediate alert template defining Email message subject text (immediate alerts are sent right away while digest alerts are sent later as a summary):

<Subject>
<GetVar Name="AlertTitle" />
<HTML><![CDATA[ - ]]></HTML>
<GetVar Name="ItemName" />
</Subject>

The interesting part here is placeholders “AlertTitle” and “ItemName” and the way they are used. I have a field named “ID” (it is a part of a standard Issues list), and a naive approach of writing <GetVar Name=”ID” /> didn’t get me anywhere. Same result with adding a custom text column named “ATextColumn” and then doing <GetVar Name=”ATextColumn” />. Well, the <GetVar> element is a part of CAML Vew Schema and yields a value of a local or a global variable set in current page context, but what are these placeholders and how are they set? At this point I have realized that my effort estimates were a little too optimistic. Then I bumped into a help article about alerts in WSS 2.0. Among other things it had a list of “tags” that could be included in alert templates. Here it is:

Tag	Description
SiteUrl	The full URL to the site.
SiteName	The name of the site.
SiteLanguage	The locale ID (LCID) for the language used in the site. For example, 1033 for U.S. English.
AlertFrequency	Immediate (0), Daily (1), or Weekly (2).
ListUrl	The full URL to the list.
ListName	The name of the list.
ItemUrl	The full URL to the item.
ItemName	The name of the item.
EventType	ItemAdded (1), Item Modified (2), Item Deleted (4), DiscussionAdded (16), Discussion Modified (32), Discussion Deleted (64), Discussion Closed (128), Discussion Activated (256).
ModifiedBy	The name of the user who modified an item.
TimeLastModified	The time the item was last modified.
MySubsUrl	The full URL to the My Alerts on this Site page in Site Settings.

Some of these tags are used in the out-of-the-box templates. I tried the rest of them and all have worked. So it appears as we are limited to using these 12 tags only, and that it is an old functionality which survived WSS 2.0, WSS 3.0 and SharePoint Foundation 2010 releases. If someone knows more about it please post a comment to validate, disprove or complete this statement.

Another thing that comes out of reflecting over template XML is usage of <GetVar Name=”OldValue#{Field}” /> or <GetVar Name=”NewValue#{Field}” /> or <GetVar Name=”DisplayName#{Field}” />. These elements are descendents of <Fields> element for immediate alerts, and of <RowFields> element for digest alerts. If you inspect generated alert body HTML then you would notice that fields (or columns) are iterated over and their values are inserted in the body except when a field is listed inside of <ImmediateNotificationExcludedFields> or <DigestNotificationExcludedFields> elements. So then <Fields> element establishes a loop, and the {Field} must be a contextual variable inside this loop. With the above syntax you can get display name, old or new values for each field into the Email body and exclude fields you don’t want to be listed.

Great, but how do I get the ID into my Email’s subject? I don’t want to list a bunch of fields in my subject, just the ID, so I don’t want to use <Fields> element there. The API did the trick. I have created a class implementing IAlertNotifyHandler interface and used regular expressions to replace a placeholder with a value:

public class MessageCustomizer : IAlertNotifyHandler

{

  public bool OnNotification(SPAlertHandlerParams parameters)

  {

      string webUrl = parameters.siteUrl + parameters.webUrl;



      using (SPSite site = new SPSite(webUrl))

      using(SPWeb web = site.OpenWeb())

      {

          string to = parameters.headers["To"];

          string subjectTemplate = parameters.headers["Subject"];

          string itemId = parameters.eventData[0].itemId.ToString();



          //

          // Below we are replacing a placeholder we have 

          // created in our alert template with the actual value.

          //



          string subject = Regex.Replace(

              subjectTemplate,

              "#ID#",

              itemId,

              RegexOptions.IgnoreCase);

          bool result = SPUtility.SendEmail(

              web,

              false,

              false,

              to,

              subject,

              parameters.body);

          return result;

      }

  }

}

We still need a customized alert template – firstly to insert our own custom placeholder (in my example #ID#) and secondly to register the MessageCustomizer class so its OnNotification() method would get called. Here is updated fragment defining Email’s subject:

<Subject>
<HTML><![CDATA[Issue ID #ID#: ]]></HTML>
<GetVar Name="AlertTitle" />
<HTML><![CDATA[ - ]]></HTML>
<GetVar Name="ItemName" />
</Subject>

Registration of MessageCustomizer class and its assembly is done inside of <NotificationHandlerClassName> and <NotificationHandlerAssembly> elements of the template:

<NotificationHandlerAssembly>AlertCustomization, Version=1.0.0.0, Culture=neutral, PublicKeyToken=bda7bcef852778f0</NotificationHandlerAssembly>

<NotificationHandlerClassName>AlertCustomization.MessageCustomizer</NotificationHandlerClassName>

We can now wire up the templates and the handler. Handler’s assembly needs to go to the GAC, then you copy and rename alerttemplates.xml file sitting in 14\TEMPLATE\XML folder, add your template (or again copy an existing one and change it), then you register this file with SharePoint running stsadm –o updatealerttemplates command. I didn’t find PowerShell cmdlets equivalent to this command. Lastly you need to assign your template to a list using SPList.AlertTemplate property. You can write a PowerShell script or use a feature receiver. The latter approach is demonstrated in Yaroslav Pentsarskyy’s post mentioned earlier.

So we are arriving at a standard “It depends…” answer to the question of whether to customize alerts via templates or via code. Regardless of which approach works for you for any such customization that is not an ad hoc fix or a proof of concept you are looking into creating a package including deployment script, a SharePoint solution and possibly a feature with feature receiver. The functionality is almost identical between WSS 3.0 and SharePoint Foundation 2010 with the latter adding SMS support. Also with Visual Studio 2010 it is much easier to package things, yet you are probably still looking into a few hours of work to get it done right.

Calling WCF Web Services from a SharePoint Timer Job

noreply@blogger.com (Ivan Neganov) — Wed, 13 Jul 2011 04:34:00 +0000

Imagine that you are building an enterprise application on top of SharePoint 2010, which is installed on a multi-server farm. The application consumes a WCF web service – custom libraries use generated proxy classes and endpoint configuration is stored inside of a web application’s web.config file. Configuration settings are applied to all servers in the farm by a script when application is provisioned. And your application needs to be deployed to development, testing and production farms, all of which have differences in how WCF endpoints and bindings are configured, including variance in WCF binding types.

Now imagine that you need to call this web service from two places: from your custom web application code and from a timer job, perhaps because you need to cache results for better performance, but also be able to fall back to synchronous call when results get outdated. You will face a complication: how do you configure your WCF client when you invoke the service from a timer job?

You have a few options:

1. Implement a configuration file OWSTIMER.exe.config;

2. Construct binding and endpoint objects inside of a timer job, set their properties through code then create a channel or extended ClientBase<T> object and execute a service method call on it.

3. Load service model configuration from application’s web.config file and create and populate appropriate binding and endpoint objects. Then create and use a channel or a ClientBase<T> object to invoke the service method.

Option 1 has issues with provisioning files to locations not intended for custom application files and keeping same configuration information in two locations on all farm servers.

Option 2 hard-codes binding information, which makes it very difficult to maintain code and troubleshoot WCF issues in multiple environments.

Option 3 is apparently the best choice since configuration is stored in one place, it can easily be changed in web.config, and the changes will affect WCF service client objects in both locations. So let us look at what’s involved in implementing the option 3.

Before you can load a configuration object from web.config file you need to locate it. Here we can leverage SPIisSettings object, which is available for each security zone. Next you load web.config content by using WebConfigurationManager.OpenMappedConfiguration() method:

private System.Configuration.Configuration GetWebConfig(SPUrlZone zone)
{
    SPIisSettings iisSettings = this.WebApplication.IisSettings[zone];
    string rootPath = iisSettings.Path.ToString();
    WebConfigurationFileMap map = new WebConfigurationFileMap();
    map.VirtualDirectories.Add(
        "/",
        new VirtualDirectoryMapping(rootPath, true));
    System.Configuration.Configuration webConfig =
        WebConfigurationManager.OpenMappedWebConfiguration(
        map,
        "/web.config");

    return webConfig;
}

Once you have obtained configuration object, you need to infer type of binding from service model configuration, and apply all attributes to it, as well as create an endpoint. Given names of a binding and an endpoint you find corresponding BindingCollectionElement and ChannelEndpointElement. The actual binding object is created using .NET Reflection and value of BindingCollectionElement.BindingType property:

private MyServiceClient MakeMyServiceClient(
    System.Configuration.Configuration config)
{
    //
    // Get endpoint and binding names from settings.
    //

    string bindingName = GetValue("Key_MyBindingName");
    string endpointName = GetValue("Key_MyEndpointName");

    //
    // Determine endpoint and binding elements used.
    //

    var sectionGroup = ServiceModelSectionGroup.
        GetSectionGroup(config);
    ChannelEndpointElement endpointElement = null;

    for (int i = 0; i < sectionGroup.Client.Endpoints.Count; ++i)
    {
        if (sectionGroup.Client.Endpoints[i].Name == endpointName)
        {
            endpointElement = sectionGroup.Client.Endpoints[i];
            break;
        }
    }

    BindingCollectionElement collectionElement = sectionGroup.
        Bindings.BindingCollections.Find(
            item => item.BindingName == endpointElement.Binding);
    IBindingConfigurationElement bindingConfig = new
        List<IBindingConfigurationElement>(
            collectionElement.ConfiguredBindings).Find(item =>
                item.Name == endpointElement.BindingConfiguration);

    //
    // Create address and binding of proper type and populate them.
    //

    Binding binding = (Binding)collectionElement.BindingType.
        GetConstructor(new Type[0]).Invoke(new object[0]);
    bindingConfig.ApplyConfiguration(binding);
    EndpointAddress address = new EndpointAddress(
        endpointElement.Address);

    MyServiceClient client = new MyServiceClient(binding, address);
    return client;
}

That’s it. The credit for MakeMyServiceClient() method goes to Microsoft. When I was searching for examples of inferring binding type and properties from configuration I bumped into implementation of a read-only property named Binding on an internal type Microsoft.SharePoint.Administration.Claims.SPSecurityTokenServiceApplication inside of Microsoft.SharePoint assembly. I have reused that code with minor deviations in the example above. Open up Reflector and take a look at that property.

Property Bag in Application Setting Manager Does Not Find a Key… Easily

noreply@blogger.com (Ivan Neganov) — Thu, 14 Apr 2011 03:47:00 +0000

I am working on a custom SharePoint 2010 application now, which relies on Microsoft SharePoint 2010 Guidance for developers. Today I came across an interesting issue while utilizing Application Setting Manager component of the guidance library. I thought I’ve set up my application according to the instructions, but I could not get settings retrieved. It turned out that there is a nuance that was not obvious from documentation, which I’d like to point out.

So the setting manager stores configuration settings in a property bag of a web site, site collection, web application or a farm. It can be hierarchical, or you can specify level at which you want to store a setting. Quite powerful. In my case I used a site collection scoped feature to deploy my settings and store them in a site collection property bag. Here is a simplified fragment of my feature receiver class provisioning application settings:

var site = properties.Feature.Parent as SPSite;
var locator = SharePointServiceLocator.GetCurrent();

// Acquire instances of config manager and property bag
var configManager = locator.GetInstance<IConfigManager>();
configManager.SetWeb(site.RootWeb);
var bag = configManager.GetPropertyBag(ConfigLevel.CurrentSPSite);

// Provision application settings
configManager.SetInPropertyBag("Key1", "Value1", bag);
configManager.SetInPropertyBag("Key2", "Value2", bag);

To retrieve the settings I used the following logic:

var locator = SharePointServiceLocator.GetCurrent();
var config = locator.GetInstance<IConfigManager>();
var bag = config.GetPropertyBag(ConfigLevel.CurrentSPSite);

if (!bag.Contains(key))
{
    throw new MyKeyNotFoundException();
}

I would be not finding the key and getting an exception. My problem was in just relying on the intellisense – why not? Well, the Microsoft guidance had it right though: in the examples the check is made as follows:

if (!config.ContainsKeyInPropertyBag(key, bag))
{
    throw new MyKeyNotFoundException();
}

And this one works fine. You can check quite easily what is in your site’s property bag; this and availability of the source code of Microsoft.Practices.SharePoint.Common library help to understand what is going on. Here is how you can see your site collection’s property bag contents using PowerShell (it is stored in the root web of the site collection):

$web = Get-SPWeb http://yoursitecollectionaddress/
$web.AllProperties | fl

You will find that your keys are in the form PnP.Config.Key.Key1._Site_ where the highlighted parts are attached to your actual key. The prefix is a constant, the suffix depends on a property bag type you use. Now if you try calling original method bag.Contains(key) passing it PnP.Config.Key.Key1 – then it will find the key!

One other observation, – you are probably using SharePoint Service Locator pattern with the setting manager, and like I do have unit tests. I needed to replace implementation of IConfigManager and IPropertyBag interfaces with mock classes. Without knowing about this “feature” my mock implementation would find a value for the key as follows: bag.Contains(“Key1”) == true, which would add confusion when the same key would not work in a web site. So try to always use config.ContainsInPropertyBag(key, bag) method instead.

Extending CoreResultsWebPart to Handle Search Queries Written in FAST Query Language

noreply@blogger.com (Ivan Neganov) — Sat, 29 Jan 2011 19:46:00 +0000

This was one of unanswered items in my recent two presentations on Search at TSPUG and MSPUG, so I was driven to figure it out and eventually did get it to work although not without some controversial steps. In this post I chose to also describe other approaches I tried and things I learned along the way, which didn’t necessarily get me to the end goal but may still be useful in your specific scenario. If you are looking for just extending CoreResultsWebPart so that it can “understand” FQL then you may want to scroll down a bit. You can download the complete source code here. I was testing my code on a SharePoint Server 2010 with December 2010 CU installed.

As you may know search web parts in SharePoint 2010 are no longer sealed, which gives you lots of flexibility via extending them. CoreResultsWebPart is probably the most important of all and therefore it is a great candidate for being extended. I wanted to take a search phrase passed as a query string argument to a search results page and write my own FQL query using this phrase as a parameter. My FQL query would do something interesting with it, for example use XRANK to boost documents created by a particular author. I certainly wanted to leverage all the goodness of CoreResultsWebPart, just use my FQL query instead of a plain search phrase. Contrary to my expectations it turned out to be not trivial to accomplish. So let’s dive into the details.

The story started with it completely not working, so I was forced to write a custom web part that used query object model and in particular the KeywordQuery class to deal with queries written in FAST Query Language (FQL). This is the one I have demonstrated at MSPUG and (with little success) at TSPUG. Below is a fragment showing how the query is submitted and results are rendered (BTW here is related MSDN reference with example).

using (KeywordQuery q = new KeywordQuery(ssaProxy))
 {
     q.QueryText = fql;
     q.EnableFQL = true;
     q.ResultTypes = ResultType.RelevantResults;
     ResultTableCollection tables = q.Execute();
     
     if (tables.Count == 0)
     {
         return;
     }

     using (ResultTable table = tables[ResultType.RelevantResults])
     {
         while (table.Read())
         {
             int ordinal = table.GetOrdinal("Title");
             string title = table.GetString(ordinal);
             TitleLabel.Controls.Add(
                 new LiteralControl(
                     String.Format("<br>{0}</br>\r\n", title)));
         }

         table.Close();
     }
 }

As you can see, rendering of results is very basic. Of course it can be made whatever it needs to be but why bother if there is CoreResultsWebPart around, which does it great already and is also highly customizable? So at this point I rolled up my sleeves. I have a book titled “Professional SharePoint 2010 Development” with a red Boeing 767 on the cover, and in Chapter 6 there is a great discussion of how to extend CoreResultsWebPart. Also Todd Carter speaks about it and shows a demo in his excellent video on Search. If you haven’t seen it I highly recommend spending an hour and 16 minutes watching it (Module 7).

Empowered with all this I wrote a web part that extended the CoreResultsWebPart and used a FixedQuery property to set a custom query, being almost one-to-one copy of Todd’s demo example. Here is the listing of this web part class (note how ConfigureDataSourceProperties() method override is used to set the FixedQuery property):

public class ExtendedCoreResultsWebPartWithKeywordSyntax : CoreResultsWebPart
{
    protected override void ConfigureDataSourceProperties()
    {
        const string LongQueryFormat = "(FileExtension=\"doc\" OR FileExtension=\"docx\") AND (Author:\"{0}\")";
        const string ShortQuery = "(FileExtension=\"doc\" OR FileExtension=\"docx\")";
        string query = null;

        if (String.IsNullOrEmpty(AuthorToFilterBy))
        {
            query = ShortQuery;
        }
        else
        {
            query = String.Format(LongQueryFormat, AuthorToFilterBy);
        }

        this.FixedQuery = query;
        base.ConfigureDataSourceProperties();
    }

    [SPWebCategoryName("Custom"),
    Personalizable(PersonalizationScope.Shared),
    WebPartStorage(Storage.Shared),
    WebBrowsable(true),
    WebDisplayName("Author to filter by"),
    Description("First and last name of the author to filter results by.")]
    public string AuthorToFilterBy { get; set; }
}

The web part actually filters results by a given author. It uses keyword query syntax and not the FQL so we are far from being done yet. Remember how in the previous code fragment there was a line q.EnableFQL = true;? If just we could set it somewhere we would be essentially done! Well right but the KeywordQuery object is not directly accessible from the CoreResultsWebPart because it uses federation object model on top of the query object model (as do other search web parts). Purpose of the federation object model is sending same query to multiple search results providers and aggregating results later either in different spots on results page or in the same list. This is done by abstracting each search results provider by means of a Location class. Important classes in federation object model are shown on the diagram below.

As you can see, CoreResultsWebPart is connected to the federation OM through CoreResultsDatasource and CoreResultsDatasourceView types with the latter actually doing all the hard work interacting with the model. And also our objective, the EnableFQL property, exists in the FASTSearchRuntime class, which in turn sets this property on the KeywordQuery class, and as I showed at the beginning, this is what’s required to get FQL query syntax accepted.

As Todd Carter and the authors of Professional SharePoint 2010 Development point out, we need to extend the CoreResultsDatasourceView class in order to be able to control how our query is handled, and wire up our own class by also extending CoreResultsDatasource class. CoreResultsDatasourceView class creates a LocationList with a single Location object and correctly determines which concrete implementation type of ILocationRuntime to wire up based on search service application configuration. In other words, federation by default is not happening for the CoreResultsWebPart. There is another web part, FederatedResultsWebPart, and another view class, FederatedResultsDatasourceView, whose purpose is to do exactly that. With that, let us get back to our objective.

If we were using SharePoint Enterprise Search then we would be almost done, because public virtual method AddSortOrder(SharePointSearchRuntime runtime) defined in the SearchResultsBaseDatasourceView class would let us get our hands on the instance of ILocationRuntime. But since we deal with FASTSearchRuntime we are sort of out of luck. Yes there exists a method overload AddSortOrder(FASTSearchRuntime runtime) but it is internal! This is where the controversy that I have mentioned at the beginning comes to play: I was not able to find a better way than to invoke an internal member via Reflection. My way works for me, but keep in mind that usually methods are made private or internal for a reason. I used Reflection to access internal property LocationRuntime of the Location object. I don’t know why this property is internal. If someone knows or has a better way to get at the FASTSearchRuntime instance or the KeywordQuery instance – please leave a comment! Here is a code fragment showing extension of the CoreResultsDatasourceView and getting an instance of FASTSearchRuntime from there.

class CoreFqlResultsDataSourceView : CoreResultsDatasourceView
{
    public CoreFqlResultsDataSourceView(SearchResultsBaseDatasource dataSourceOwner, string viewName)
        : base(dataSourceOwner, viewName)
    {
        CoreFqlResultsDataSource fqlDataSourceOwner = base.DataSourceOwner as CoreFqlResultsDataSource;

        if (fqlDataSourceOwner == null)
        {
            throw new ArgumentOutOfRangeException();
        }
    }

    public override void SetPropertiesOnQdra()
    {
        base.SetPropertiesOnQdra();
        // At this point the query has not yet been dispatched to a search 
        // location and we can set properties on that location, which will 
        // let it understand the FQL syntax.
        UpdateFastSearchLocation();
    }

    private void UpdateFastSearchLocation()
    {
        if (base.LocationList == null || 0 == base.LocationList.Count)
        {
            return;
        }

        foreach (Location location in base.LocationList)
        {
            // We examine the contents of an internal 
            // location.LocationRuntime property using Reflection. This is 
            // the key step, which is also controversial since there is 
            // probably a reason for not exposing the runtime publically.
            Type locationType = location.GetType();
            PropertyInfo info = locationType.GetProperty(
                "LocationRuntime",
                BindingFlags.NonPublic | BindingFlags.Instance,
                null,
                typeof(ILocationRuntime),
                new Type[0],
                null);
            object value = info.GetValue(location, null);
            FASTSearchRuntime runtime = value as FASTSearchRuntime;

            if (null != runtime)
            {
                // This is a FAST Search runtime. We can now enable FQL.
                runtime.EnableFQL = true;
                break;
            }
        }
    }
}

By the way, another limitation of my approach is that using Reflection requires full trust CAS policy level. That said, finally we have arrived at our objective – we can set the flag on the FASTSearchRuntime, and it will understand our FQL queries. Our extended search results web part will show results as directed by the query (in the attached source code it uses XRANK) and leverage presentation richness of the CoreResultsWebPart.

Decks and Source Code from January 19th TSPUG Meeting

noreply@blogger.com (Ivan Neganov) — Fri, 21 Jan 2011 13:03:00 +0000

Thanks to all who came to Wednesday’s TSPUG meet up. I’ve uploaded a package with presentation and source code files. I was able to resolve the first of the questions unanswered during my previous talk about SharePoint search technologies. Although I meant to show it, I didn’t get to talk much about promoting user profile properties for selection in FAST user context for visual best bets. Basically as Steve Peschka has pointed out, the first thing that needs to be done is permissions granted to the profile store. This will let you see the list of properties in his Property Explorer tool. Secondly, to promote profile properties to be available for selection in FAST Search user context administration section you need to edit value of FASTSearchContextProperties property of FAST Query search service application (SSA). I wrote a command-line utility for this purpose, which can be invoked as follows:

FASTProfilePropertyUpdater.exe –ssaId <FAST query SSA Guid> -action Add|Remove -property <User Profile property name>

Code fragment below demonstrates how SSA reference is acquired and the “Add” operation is accomplished. Full source code is a part of the package. Ideally you want to use PowerShell script to manage this, I just wrote an executable as it was easier to get it working quickly for me.

SPFarm farm = SPFarm.Local;
 var settingsService = SPFarm.Local.Services.GetValue<SearchQueryAndSiteSettingsService>();
 var serviceApp = settingsService.Applications[ssaId];
 var searchApp = serviceApp as SearchServiceApplication;

 if (null == searchApp)
 {
     Console.WriteLine(
         "Cannot find search service application with the ID '{0}'.", 
         ssaId);
     return 2;
 }

 string properties = (String)searchApp.Properties[FASTSearchContextProperties];
 Console.WriteLine(
     "Updating service application properties for property key '{0}'... Value before update was: '{1}'",
     FASTSearchContextProperties,
     properties);
 List<string> propertiesList = properties.Split(',').
     Select(p => p.Trim()).ToList();
 
 if (ContextPropertyActions.Add == action &&
     properties.IndexOf(propertyName, StringComparison.Ordinal) < 0)
 {
     propertiesList.Add(propertyName);
     properties = string.Join(",", propertiesList.ToArray());
     searchApp.Properties[FASTSearchContextProperties] = properties;
     searchApp.Update(true);
     Console.WriteLine("Property added. Updated value is: '{0}'.", properties);
     return 0;
 }

Also in the package is the source code for a web part demonstrating how to dynamically boost search relevance ranking from within an FQL query by using XRANK keyword.The demo of this web part didn’t go well, but the code is actually working (I dragged the wrong web part to the page during presentation!). Also someone has asked me about proximity-based filtering of results in FAST query language. Yes this is possible - there is a number of keywords that support this.

Automate Internet Proxy Management with PowerShell

noreply@blogger.com (Ivan Neganov) — Sun, 09 Jan 2011 03:18:00 +0000

The solution is very simple, yet the issue was so annoying to me that I thought to share this. I have a laptop and do work on it on site at my client and when I am at home. When I am in the office connected to my client’s network I need to use a proxy to access the Internet, while usually I do not need a proxy when I am located elsewhere. You normally set up a proxy using browser settings. With Internet Explorer 8 you open a browser, then go to Tools >> Internet Options >> Connections >> LAN Settings. There you configure the proxy parameters. As a part of configuration you can turn the proxy on or off by setting or clearing a Use a proxy server for your LAN check box, and as you switch the proxy the rest of your proxy configuration settings such as exceptions is preserved for you. Great!

My problem was that on almost daily basis I had to open the browser, navigate to settings then check or uncheck that box. I would come to work and forget to go through the steps so my browser would get stuck, and then I would go “oh yeah, I turned the proxy off last night”. Of course the reverse would happen at home… I tolerated this too long because I didn’t expect that a simple solution exists, but it does and here it is:

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ProxyEnable key stores a DWORD value, either 1 or 0 to indicate if the proxy is enabled.

So I wrote two 1-line PowerShell scripts and added two shortcuts to them to my Windows taskbar: Enable Proxy, Disable Proxy! Here is how the script to enable proxy looks like:

Set-ItemProperty "HKCU:\Software\Microsoft\Windows\CurrentVersion\Internet Settings" -Name ProxyEnable -Value 1

Then I’ve got excited, and wrote yet another rock star PowerShell script. Here it is:

$matches = @(ipconfig /all | findstr "myclientdomain")
if($matches.Count -gt 0){
./EnableProxy.ps1
}else{
./DisableProxy.ps1
}

That’s right I am using output of the ipconfig command to determine if I am on the client’s network and if yes then I turn on my proxy. After I have added a scheduled task running at user logon to invoke this script, the quality of my professional life has improved.

Writing Trace Output to ULS Log in SharePoint 2010

noreply@blogger.com (Ivan Neganov) — Thu, 23 Dec 2010 22:45:00 +0000

SharePoint 2010 supports writing custom messages to ULS log. A blog post by Waldek Mastykarz of Mavention provides a good example of how to do this. UPDATE 02/24/2011: Andrew Connell's article on MSDN from December 2010 provides essential information on diagnostics logging.

Ability to send messages from System.Diagnostics.Trace.Write() and Trace.Fail() method overloads to ULS is yet another “logging novelty” I will be focusing on in this post.

There is a new type in Microsoft.SharePoint assembly, which was not there in version 12 – Microsoft.SharePoint.SPULSTraceListener. As its name suggests it allows sending trace messages to ULS.

If you are instrumenting your custom SharePoint code and need to write diagnostic traces to ULS then you can wire up SPULSTraceListener type in your web.config and use tracing infrastructure available in System.Diagnostics.Trace. Here are the steps:

1. Add a trace listener element to web.config. For more information see description of system.diagnostics element on MSDN.

<system.diagnostics>
    <trace>
        <listeners>
            <remove name="Default"/>
            <add 
                name="ULSTraceListener" 
                type="Microsoft.SharePoint.SPULSTraceListener, Microsoft.SharePoint, version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
        </listeners>
    </trace>
</system.diagnostics>

In this XML snippet note the <remove name=”Default”>. This is done to remove default trace listener, which is wired up by default and displays a pop-up window with the message when Trace.Fail() is called.

2. In central administration site configure diagnostic logging (Central Administration >> Diagnostic Logging). If you are using Trace.Write() method calls then they are logged as Verbose and so in order to actually see them in ULS logs you need to make sure that throttling level is set to Verbose for SharePoint Foundation > Service Connections category. If you use Trace.Fail() overloaded methods then the message will be written as a High level, which by default is on, so the message will normally end up being written to a log file. And yes, all messages will be written to the log under Service Connections category.

What About TraceContext.Write()?

UserControl, Page and HttpContext classes have a property named Trace of type System.Web.TraceContext. It allows tracing messages using ASP.NET trace infrastructure, for example: HttpContext.Current.Trace.Write(string message);. You can send ASP.NET trace messages to ULS using standard technique but with a caveat:

Add trace configuration element under system.web element in your web.config as follows:

<system.web>
  <trace enabled="true" requestLimit="1000" mostRecent="true" writeToDiagnosticsTrace="true"/>
    ...

Here the key attribute is writeToDiagnosticsTrace instructing ASP.NET trace to use tracing infrastructure from System.Diagnostics. Now trace messages written using TraceContext.Write() will be sent to ULS.

Now the caveat: adding system.web/trace element appears to destabilize SharePoint. I have tried this on SharePoint Foundation and SharePoint Server 2010 with October 2010 CU, and although tracing does work as expected I have got a consistent error when trying to create a new web part page:

Looking at the ULS log, I found these messages logged under SharePoint Foundation category:

SharePoint Foundation    General    b9y3    High    Failed to open the file 'C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\Resources\wss.resx'.

SharePoint Foundation    General    b9y4    High    #20015: Cannot open "": no such file or folder.
SharePoint Foundation    General    b9y4    High    (#2: Cannot open "": no such file or folder.)
SharePoint Foundation    General    b9y9    High    Failed to read resource file "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\Resources\wss.resx" from feature id "(null)".

As soon as I disable the system.web/tracing the error disappears. I am not sure if this is a bug or adding <trace> element under <system.web> is not supported. In any case there is enough of rich diagnostic features available, and if you are using tracing in your application then just avoid System.Web.TraceContext.Write() in favor of System.Diagnostics.Trace.Write() or at least use it temporarily if you need to. By the way System.Diagnostics.Debug.Write() also sends messages to trace log.

UPDATE (01-16-2011): There is yet another issue caused by turning on trace that I ran into today: SharePoint designer cannot open a site in a web application for which tracing is enabled, and keeps prompting for credentials. If you check out traffic with Fiddler, you will find that SPD makes requests to http://devserver2010/_vti_bin/shtml.dll/_vti_rpc, which return a 401 UNAUTHORIZED. Not even verbose logging reveals anything useful. Well, commenting out <trace> element in web.config restores normal functionality.

Configure Links Web Part to Open URLs In a New Window With SharePoint 2010

noreply@blogger.com (Ivan Neganov) — Mon, 13 Dec 2010 18:04:00 +0000

With SharePoint 2010 (or SharePoint Foundation 2010) you can customize list views using XSLT, which is very powerful and for simple changes also quite simple. Here is an example of how to make URLs in Links list web part open up pages in a new browser window.

Links web part as well as other list-based web parts uses a predefined list view to render its content. Since views are now defined using XSLT style sheets, all that is needed to control the HTML is to override appropriate XSLT template with a custom one that has HTML markup that we need, for example <a href=”….” target=”_blank”>. List web part’s edit mode panel contains a text field titled XSL Link under Miscellaneous category where you can provide a URL of an XSLT file, which will override default rendering behavior.

Let’s say we have a Links list with a few URLs in it and we have added a Links web part to a page. We want to have all links shown by the web part open up in a new window. We need to find an XSLT template which describes rendering of the field with a link. One easy way to determine this is to sort by the URL field then take a note of the value of SortField query string value, which tells us that the field name we need is URLNoMenu. URL field is not a part of default view, default view has a URL (URL with edit menu) column, so you would need to add a column titled URL. As shown in the picture below, I’ve added my own view “Only Url” to Links list with this field only and set the web part to use this view.

Now we need to find an XSLT template for the URLNoMenu field. Out-Of-The-Box templates are stored under the 14 hive in TEMPLATE\LAYOUTS\XSL on server file system. For what we need there are 2 important files there: main.xsl and fldtypes.xsl. The former file is a default file used for formatting, which imports other XSLT files including fldtypes.xsl. The latter file is the one which contains many templates for rendering different fields. Doing search inside this file for URLNoMenu yields:

<xsl:template name="FieldRef_URLNoMenu_body" ddwrt:dvt_mode="body" match ="FieldRef[@Name='URLNoMenu']" mode="Computed_body">

Next we copy this template XML element and put it in a new XSLT file. I named mine TargetBlank.xsl. Inside the new file we change HTML anchor tag to include target=”_blank” (or anything else we need) and save the file. Once we specify URL to this new file in the XSL Link field of a web part being edited, it will override all default formatting, therefore we also need to make sure that other templates still apply. Thanks to XSL import and overriding mechanism this is easy to do, we just need to import main.xsl file, and our version of the FieldRef_URLNoMenu_body template will override the one located inside fldtypes.xsl. Here is the complete listing of the TargetBlank.xsl.

<xsl:stylesheet xmlns:x="http://www.w3.org/2001/XMLSchema" 
                xmlns:d="http://schemas.microsoft.com/sharepoint/dsp" 
                version="1.0" 
                exclude-result-prefixes="xsl msxsl ddwrt" 
                xmlns:ddwrt="http://schemas.microsoft.com/WebParts/v2/DataView/runtime" 
                xmlns:asp="http://schemas.microsoft.com/ASPNET/20" 
                xmlns:__designer="http://schemas.microsoft.com/WebParts/v2/DataView/designer" 
                xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
                xmlns:msxsl="urn:schemas-microsoft-com:xslt" 
                xmlns:SharePoint="Microsoft.SharePoint.WebControls" 
                xmlns:ddwrt2="urn:frontpage:internal">
  <xsl:import href="/_layouts/xsl/main.xsl"/> 
    <xsl:output method="html" indent="no"/>
    <xsl:template name="FieldRef_URLNoMenu_body" 
                  ddwrt:dvt_mode="body" 
                  match ="FieldRef[@Name='URLNoMenu']" 
                  mode="Computed_body">
        <xsl:param name="thisNode" select="."/>
        <xsl:variable name="url" select="$thisNode/@URL" />
        <xsl:variable name="desc" select="$thisNode/@URL.desc" />
        <xsl:choose>
            <xsl:when test="$url=''">
                <xsl:value-of select="$desc" />
            </xsl:when>
            <xsl:otherwise>
                <xsl:choose>
                    <xsl:when test="@Format='Image'">
                        <img onfocus="OnLink(this)" src="{$url}" alt="{$desc}" />
                    </xsl:when>
                    <xsl:otherwise>
                        <a onfocus="OnLink(this)" href="{$url}" target="_blank">
                            <xsl:choose>
                                <xsl:when test="$desc=''">
                                    <xsl:value-of select="$url" />
                                </xsl:when>
                                <xsl:otherwise>
                                    <xsl:value-of select="$desc" />
                                </xsl:otherwise>
                            </xsl:choose>
                        </a>
                    </xsl:otherwise>
                </xsl:choose>
            </xsl:otherwise>
        </xsl:choose>
    </xsl:template>
</xsl:stylesheet>

In this file the xsl:stylesheet element is copied from the main.xsl file. Also note the xsl:import and the xsl:output statements. Lastly we need to upload the XSLT file and reference its URL in the XSL Link field of our web part. That’s all there is to it. Our links should now open in a new browser window.

Decks, Source Code and Resources from October 19 MSPUG Meeting

noreply@blogger.com (Ivan Neganov) — Wed, 20 Oct 2010 18:01:00 +0000

Thanks a lot to the folks who showed up yesterday at the Mississauga User Group meeting. Also I’d like to thank Ray Outair for his efforts in running the group in celebration of its first year anniversary. I have posted the deck for my presentation “Leveraging SharePoint 2010 Search Technologies” and the source code of the demo web part demonstrating FQL capabilities on SoftForte site here.

If you are planning search deployment I highly recommend checking out technical diagrams on TechNet: http://technet.microsoft.com/en-ca/library/cc263199.aspx. There are 4 diagrams there dedicated specifically to Search. Also check out downloadable content here http://technet.microsoft.com/en-ca/library/cc262788.aspx – it has been recently updated.

As I mentioned at the meeting, I also recommend a book titled Professional Microsoft Search: FAST Search, SharePoint Search, and Search Server by Mark Bennett, Jeff Fried, Miles Kehoe and Natalya Voskresenskaya. It is a great resource on planning and architecture of SharePoint Enterprise Search, and specifically FAST Search products.

Last but not least is a reference to Steve Peschka’s blog post where Steve introduces a Search Explorer tool that he wrote. Steve’s blog is an essential resource on developing Search and other SharePoint development topics.

So I’d like to re-cap on a few questions we left unanswered yesterday:

1. FAST visual best bets were displayed, but did not get filtered based on User Context’s managed properties. The question is how to create/promote properties into user context for such filtering to happen correctly. UPDATE: I figured this one out

2. I was not lucky extending CoreResultsWebPart and make it process custom FQL queries, and was forced instead to write a web part from scratch. The question is whether there is a supported way to extend CoreResultsWebPart for FAST search queries.
UPDATE: this one is now also addressed. Check it out here.

3. Somebody has mentioned that they could not get PowerPoint visual preview to work with FAST search results, suspecting that SSL encryption between FAST Search Server and query SSA was to blame. Looks like some research is needed here.

I plan to look into these and post my findings. Please feel free to comment on if you get answers sooner.

Enumerate SharePoint 2010 Databases

noreply@blogger.com (Ivan Neganov) — Sun, 19 Sep 2010 17:43:00 +0000

If you are working with SharePoint in a lab environment and use one shared database server to store databases from multiple farms then with SharePoint 2010 you can quickly lose control of your databases since there are much more of them now. When you set up a lab farm you typically run Products Configuration Wizard which leaves you with several databases created whose names contain GUIDs, which makes it hard to distinguish between them and relate them to corresponding farms.

Let’s say you need to get rid of a farm and want to delete corresponding databases after. What do you do? If you haven’t yet disconnected from the farm and uninstalled SharePoint then PowerShell comes at help. Log in to one of the farm servers, start up SharePoint 2010 Management Shell and enter this command:

Get-SPDatabase | % {$_.Name}

It will list names of all databases utilized in the current farm. You can now create a script to drop these databases or do whatever you needed to do with them. If however it is too late and you have uninstalled your servers, you can still get the list of database names using a SQL query:

USE SharePoint_Config
select distinct name from Objects WITH(NOLOCK) where Name in (
select Name COLLATE Latin1_General_CI_AS_KS_WS from sys.databases WITH(NOLOCK))
GO

Replace SharePoint_Config with the name of configuration database for your farm. SharePoint products use Latin1_General_CI_AS_KS_WS collation so you need to do a cast. It is of course not supported to issue queries directly against SQL Server so I would use this approach for development environment only, and only when you are really cleaning it up and PowerShell isn’t an option.

Alternatively to avoid the mess in your database names from the start you can also pre-create them. Here is the guidance. Lastly, although the guidance does not suggest sharing database role between farms in production environment, it is quite practical to share a SQL server in development environment if you are not using your farm for performance and capacity testing.

UPDATE: on technet there is a list of databases used by SharePoint (a section in storage and SQL Server capacity planning article): http://technet.microsoft.com/en-us/library/cc298801.aspx#section1a

Slipstreamed Installation of SharePoint Server 2010 with June 2010 CU

noreply@blogger.com (Ivan Neganov) — Mon, 30 Aug 2010 11:34:00 +0000

I wanted to create a slipstreamed installation point for SharePoint Server 2010, which would include June 2010 CU. I could not find guidance on how to do this for the current version. MOSS 2007 guidance recommends extracting patches into \Update folder of the installation point. This works as well for the SharePoint 2010. Would the installer also pick up and apply executable packages, or their extracts in subfolders under the \Update? I tried both variants – it does not.

The reason I went that path was because June 2010 CU consists of 6 packages, which is unusual given regular MOSS CUs single file format. Also when extracting each of the patches manually to a common folder the following two files are duplicated between KB983319 and KB983497 patches: osrchwfe-x-none.msp, osrchwfe-x-none.xml; and the following two - between KB2281364 and KB983497: pplwfe-x-none.msp, pplwfe-x-none.xml. By reviewing KB articles it appears that the files are identical so it is ok to overwrite them, which is what I’ve done using a script:

office-kb2124512-fullfile-x86-glb.exe /extract:C:\temp\sharepoint\Updates
office-kb2204024-fullfile-x64-glb.exe /extract:C:\temp\sharepoint\Updates
office-kb2281364-fullfile-x64-glb.exe /extract:C:\temp\sharepoint\Updates
office-kb983319-fullfile-x64-glb.exe /extract:C:\temp\sharepoint\Updates
office-kb983497-fullfile-x64-glb.exe /extract:C:\temp\sharepoint\Updates
spf-kb2028568-fullfile-x64-glb.exe /extract:C:\temp\sharepoint\Updates

Next I copied extracted files to the \Updates folder in my installation point.

After installing SharePoint and running configuration wizard most of the patches were applied, except for KB2124512 and KB2204024 according to DLL version check. I am running Windows 2008 R2, and SQL Server 2008 R2, and made a farm installation of SharePoint. It could be that either these patches do not apply in my environment, or something is not working in my installation. I will post an update if I find out more. For now I consider that the slipstream approach is generally the same as before even for multi-file CUs such as this one.

Modified Date Field is not Visible after Content Deployment

noreply@blogger.com (Ivan Neganov) — Fri, 28 May 2010 05:19:00 +0000

It is the simple things, or at least ones you assume must be simple that often create some extra anxiety with SharePoint – at least from what I’ve seen. Here is one I ran into a couple of days ago: my customer runs a SharePoint 2007 publishing portal with February 2010 CU, where content is created in authoring environment, then content deployment job transfers it to production farm located in perimeter zone and accessible to anonymous Internet users. The application is using a significant amount of custom code. In many cases SharePoint API is used as a data provider, and is accessed through an adapter layer, which helps decoupling custom code from the platform. Quite logical.

An example of how the adapter accesses SharePoint API is retrieving SharePoint document library information through SPList.GetItems(SPQuery).GetDataTable(). The DataTable instance is then returned back to the consuming custom code. The problem is that Modified column is missing from the returned DataTable in production farm, while it is there in authoring farm, which certainly breaks a lot of application logic.

It turns out that a Content Deployment Job Path setting (Central Administration > Operations > Manage Content Deployment Paths and Jobs > Content Deployment Path) is to blame: Security Information must be set to All in order for the issue to go away, and you would need to re-create the target site collection. Now this will not be a completely painless move – you will start getting a content deployment warning: User security information cannot be properly imported without setting UserInfoDateTime option to ImportAll. This is because you have a Deploy User Names checkbox cleared, which you should according to the guidance. Here is a related post about this warning.

What actually happens is when Security Information is not set to All, this results in hiding certain fields, including Modified date field. So the GetItems(SPQuery) will still return correct items satisfying the SPQuery parameter, but if you inspect properties of an SPField for the returned items you will see that:

 item.Fields["Modified"].CanToggleHidden == false;
 item.Fields["Modified"].Hidden == true;

and the SPListItemCollection.GetDataTable() method omits hidden fields when it constructs the DataTable. Knowing the above you can either edit content deployment path and do a clean content deployment, or if it is problematic for some reason, then you can change your custom code to manually construct a DataTable rather than using SPListItemCollection.GetDataTable().

UPDATE: The issue was fixed in April Cumulative Update (see http://support.microsoft.com/kb/981040/ for more details). Thanks to Bill Brockbank for noting this. A good moment to emphasize importance of running up-to-date software.

Records Management in SharePoint 2010

noreply@blogger.com (Ivan Neganov) — Thu, 29 Apr 2010 17:53:00 +0000

SharePoint 2010 has a lot more to offer in records management space than its predecessor. Rez will be presenting at Mississauga SharePoint User Group on May 4th. The topic is SharePoint 2010 Records Management Features. Rez has many years of experience with SharePoint and deep knowledge of the technology. I look forward to and highly recommend next week's session.