An Antic Disposition

Asking the right questions about Office 2010's OOXML support

2009-11-17T11:00:00.002-05:00

There is more OOXML controversy in the news, this time in Denmark. I don't claim to understand all the nuances of the accusations, since I don't read Danish, and Google Translates makes it sound at times like a discussion about loaves of rye bread or something, but the gist of it, as I can surmise from this account, is whether Office 2010 will "support the complete ISO-approved version of OOXML". Microsoft's spokesperson says it will. Mogens Kühn Pedersen, chair of the Danish Standards Committee, says it will not.

This is the kind of dispute where you can go around in circles with for days and not reach agreement. The problem is they are arguing over words, not facts, and they do not agree perfectly on the meaning of the words. Words like "support" and "complete" and "conform" are used in different ways, with different meanings and intents.

Let's try to escape the equivocation and instead try to establish the underling facts. I can't promise that this will clarify the situation any. In fact I suspect we'll end up even more confused about what exactly Office 2010 actually supports. But replacing a false certainty with an honest uncertainty is progress of a kind. It gives us something we can build on.

First, we need to acknowledge that OOXML entered ISO as one standard, and was transformed, via the BRM and ISO ballot, formally into 4 standards, ISO/IEC 29500 Parts 1, 2, 3 and 4. Within these parts are are several different conformance targets and conformance classes. In particular, these 4 standards encompass two different and incompatible schemas for many of its features: "Strict" and "Transitional". What was submitted in the Fast Track is essentially the "Transitional" schema. What was created by the BRM was the "Strict" schema. This is where Microsoft made most of its "concessions" in order to turn "No" votes into "Yes" votes. So things like support for spreadsheet dates before the year 1900, the elimination of VML graphics, etc., these are all in the "Strict" schema. All the legacy "DoItLikeWord95" garbage was in "Transitional" only. Several NBs voted to approve OOXML because the assertion that "Transitional" would not be written in documents produced by future versions of MS Office. The promise was that it was...well...transitional, for moving legacy binary documents into XML. Few people want to support two different document standards (both ODF and OOXML) in the first place. But to require support for two different and incompatible versions of OOXML -- that is simply intolerable.

In any case, because of these two conformance classes, anyone who claims that their product supports "OOXML" in an unqualified sense, without stating which conformance target or conformance classes they are supporting, is not stating anything of substance. It is like trying to buy an electrical plug adapter by just saying "I need electricity". Merely saying "conformance to OOXML" means nothing. You need to state the conformance targets and classes that you support. Remember, the conformance language of OOXML is so loose that even a shell statement of "cat foo.docx > /dev/null" would qualify as a conformant application. I assume that Office 2010 supports at least that.

Of course, the alleged assertion that Office 2010 supports OOXML "completely" is a bit more problematic. What exactly does this mean? Does this mean that Office 2010 supports all conformance classes and targets of all 4 parts of OOXML? Including being a Strict consumer? A Strict producer? That would be a good thing, IMHO, if it were true. But that is not what WG4 was recently told in Seattle, where they were told that Office would not write out Strict documents until Office 16. That would put it out to the middle of the next decade, assuming the typical 3-year Office release schedule.

So I'll lay out my assertions (with the caveat that Office 2010 is not complete and shipped yet) as:

Office 2010 will conform to the Transitional consumer and producer classes defined in the OOXML standards. Any bugs that are found in the shipped version of Office 2010 will be "fixed" by retroactively changing the standards to match what Office actually does, as is currently being done by Microsoft-packed SC34/WG4 committee with similar bugs found in Office 2007's OOXML support.
Office 2010 will not have conforming support for OOXML Strict producer or consumer classes.
Office 2010 will write dozens of non-interoperable, proprietary extensions into their OOXML documents, extensions which are not defined by the OOXML standards and which have not been reviewed or standardized by any standards committee and which will not be fully interoperable with other OOXML editors, or even with previous versions of MS Office.

So instead of arguing over the meaning of "support" and "complete" I suggest some alternate questions for Microsoft, to give them the opportunity to clarify exactly what kind of support for OOXML will be coming in Office 2010:

Exactly what ISO/IEC 29500:2008 conformance classes and targets will Office 2010 conform to?
Is this contingent on first changing the conformance requirements of the published ISO/IEC 29500:2008 standards to match what Office 2010 actually supports? Or is there a commitment to support the published standards as they was approved by JTC1 national bodies? In other words, is Microsoft committed to conform to the standards, or are we back to changing the standards to "conform" to Microsoft?
Will Microsoft Office 2010 write out only markup that is fully described in the OOXML standards? Or will it write out proprietary markup extensions that are not fully defined in the standards? In other words, will Office 2010 be "strictly conformant" with the ISO/IEC 29500:2008 standards?

The problem you run into here is that there are really two different OOXML standards: the new and improved OOXML Strict conformance class, the one that was "sold" to ISO NBs, the one that garnered the approval votes, and then the old ugly one, the "haunted" specification, the Transitional conformance class, supported only by Microsoft Office. Anyone considering adopting OOXML should have perfect clarity as to which one they are adopting, especially since these are two very different standards, both formally and logically. Just as it is problematic to speak about OOXML support in a product without stating which conformance classes and targets are supported, it is equally a defect of any adoption policy to be loose in what version of OOXML is being proposed for adoption.

IMHO, if you must state a requirement for OOXML (along with ODF), at least specify it clearly, and state a requirement for "strict conformance" (meaning no extensions) of the Strict conformance classes of ISO/IEC 29500:2008. To do otherwise is to essentially specify a requirement for the use of Microsoft Office and Microsoft Office alone.

ODF 1.2, Part 3 goes out for Public Review

2009-11-16T12:15:00.001-05:00

A major milestone for ODF 1.2 was reached on Friday. Part 3 of ODF 1.2, which specifies document packaging (how a document's XML, images and metadata are combined into a single file and are optionally encrypted or signed), went out for a 60-day public review period. This public review period will run through January 12th, 2010. A public review is a necessary OASIS procedure before a Committee Draft can be approved as a Committee Specification and then as an OASIS Standard.

The official announcement of the review has more information, including links to download the public review draft and information on how to submit comments on the draft.

Compared to the packaging specification used in ODF 1.0 and ODF 1.1, the main differences are:

We've split this material into its own specification, since these packaging conventions are more widely applicable, and in fact have been more widely used than just in ODF. For example, the International Digital Publishing Forum, who standardize the increasingly important ePub digital book format, use ODF's packaging as the base of their Open eBook Publication Structure Container Format (OCF) 1.0 specification.
We've added digital signature support (chapter 4) based on the W3C's XML Digital Signature Core, including the ability to use standardized extensions such as XAdES.
We now have an RDF-based metadata framework with OWL ontology for the manifest file (chapter 5).
We include a more detailed conformance definition has been added, including conformance targets for packages, producers and consumers, as well as a separate conformance class for extended packages.
Generally, a redraft of the specification to ISO style guidelines.

This specification is only 34 pages long, so if you're at all interesting please give it a look over between now and January 12th and send along any comments to via the office-comment list. Anything that improves the specification is welcome, from reports of typographical errors, to technical omissions or errors, to suggestions for future features. It is all good.

And if you want to follow along, you can track the incoming comments in several ways:

Subscribe to the office-comment list mentioned above.
View the archives of the off-comment list.
View the public review comments we're tracking in JIRA. I have a python script that scrapes the office-comment list and enters them into JIRA. This will be more complete than the office-comment list because it will includes additional comments from ODF TC.
I have another script that takes each new issue from JIRA and sends it out via Twitter. So you can follow all new ODF issues by subscribing to @ODFJIRA. Depending on your Twitter reader, you might be able to mark some issues as "favorites" and return to them later to see how they have been resolved.

Also, keep your eye open for the announcement of a public review for ODF 1.2, Part 1 (ODF Schema) and Part 2 (OpenFormula), which will be ready for review soon.

The Final OOXML Update: Part III

2009-10-27T09:35:00.005-04:00

This is Part III of an 5-part series on the state of OOXML today. Previous to starting this series, I had not posted about OOXML in over a year. Part I showed how Microsoft, despite their promises that control of OOXML would be handed over to an independent, international committee, have instead stuffed the committee that maintains OOXML (JTC1/SC34/WG4) with Microsoft employees. And in Part II I looked at how the final published text of OOXML failed to account for all BRM decisions, and described the steps that ISO was taking to remedy this obvious procedural flaw.

In this Part I'll look at how Microsoft is using their dominance in SC34 to push through hundreds of changes and additions to OOXML, in a misuse of a procedure intended for correcting drafting errors, to make OOXML "conform" to Microsoft's monopoly product.

Let's start by taking a look at the OOXML defect log [PDF] that SC34/WG4 uses to track their large list of errors and omissions discovered in the published standard. This defect report currently amounts to over 800 pages, longer than the entire ODF 1.0 standard. But it is well worth downloading and browsing through.

Some of these changes will be made in Technical Corrigenda while others are proposed for Amendments. What is the difference? SC34/WG4 itself made the distinction clear, in a presentation (N 1187 for those with access) it made to the SC34 Plenary in Prague, where it outlined its practice for deciding which changes would be made in corrigenda versus amendments:

All of the following criteria should be met for the defect to be resolved by Corrigendum:

1) WG 4 agrees that the defect is an unintentional drafting error.
2) WG 4 agrees that the defect can be resolved without the theoretical possibility of breaking existing conformant implementations of the standard.
3) WG 4 agrees that the defect can be resolved without introducing any significant new feature.

Unless all the above criteria are met, the defect should be resolved by Amendment.

These are reasonable criteria and no objections were made when these guidelines were presented to SC34.

A key procedural point is that in ISO/IEC it is the JTC1 NBs who are the consensus body that has the authority to create international standards. All ballots which create or substantial modify standards must be approved by JTC1. This includes DIS ballots, FDIS ballots, FDAM ballots and DTR ballots. So standards, technical reports and amendments are ultimately approved or disapproved by JTC1 NBs. Although subcommittees in JTC1, such as SC34, provide the technical expertise and author and review work, they are not the standardizing authority. The exception that proves the rule is with corrigenda, which are authored and approved entirely at the SC level. However, this small area of autonomy in defect correction comes with carefully delineated bounds. A SC can author, approve and publish corrigenda by itself, but only to make corrections.

So if we look at JTC1 Directives 15.4.2.2, we read (with my emphasis) "A technical corrigendum is issued to correct a technical defect.... Technical corrigenda are not issued for technical additions which shall follow the amendment procedure...". And in 15.4.1 "technical addition" is defined as: "Alteration or addition to previously agreed technical provisions in an existing IS."

So amendments, which require approval by JTC1, are used for altering or extending the provisions of a standard, while corrigenda are used to correct errors introduced in drafting or publication. This dichotomy is common in other standards organizations. For example, in OASIS, a technical committee is able to approved and publish "Approved Errata" but these are restricted to changes that do not break conformance of existing implementations. Anything beyond that is considered a substantive change to the standard and requires review approval by the OASIS membership.

Clear enough? In fact, in many cases WG4 appears to follow this important distinction. Some of the proposed changes are simple and benign. For example, some BRM issues were fixed, but in being fixed caused informative example markup in the standard to be incorrect. A quick fix of these items via corrigenda is most welcome.

However, in other cases (in fact most of the cases), the Microsoft-dominated WG4 appears to have overstepped the permissible bounds for corrigenda, and indeed gone far, far beyond what it stated it would be doing in corrigenda. Let's look at a few examples.

(Sadly, the general public is not given access to the text of the draft corrigenda (the DCOR) but those on the inside can follow along by reading N 1252 in the SC34 document repository.)

Let's start by looking at items 16, 17, 36, 52, 53 and 133 in DCOR for ISO/IEC 29500 Part 4. These make changes and additions to the WordProcessingML schema. Deletions are noted in red strikethroughs, and additions in blue:

This is not correcting a drafting error. This is not correcting a publishing error. This is a substantial addition to the schema as you can see above.

It is argued, in the defect log, that this change is needed because, without it, ISO/IEC 29500 cannot represent change tracking in mathematical equations. However, this is exactly the type of change that WG4's guidelines and JTC1 Directives exclude from corrigenda and place into amendments. The schema of OOXML is certainly an "agreed technical provision of an existing IS". So how can adding math change tracking support to the schema be anything other than an "addition to previously agreed technical provisions"? And how can anyone in WG4 believe that adding dozens of lines to the schema can be done "without the theoretical possibility of breaking existing conformant implementations of the standard"? What about, for example, applications that were programmed to use the published OOXML schema, such as any application that uses a validating parser, or an schema-directed editor, or a program that generates code stubs from the schema, or does XML-to-relational DB mapping? Not only is there a theoretical possibility of breaking such applications, there is a theoretical certainty.

(Ironically, it should be noted that Microsoft was very keen to beat up on ODF for not having change tracking for mathematical equations, all while hiding the fact that OOXML lacked complete support for this feature as well.)

Another example, #122 in the DCOR.

It changes a type in chart, from a byte to an int and in doing so extends its allowed range considerably. How did anyone think that this was a change that was "without the theoretical possibility of breaking existing conformant implementations of the standard"? Isn't there enough theoretical and practical expertise in WG4 to know that changes like this break compatibility?

For this change the rationale in the defect log explains the logic of it:

The standard states that the ST_Period simple type uses the XML Schema ST_Period data type and supports a range 2–255.

These observations are incompatible with existing documents and should be updated to reflect such prior art.

And so on and so on. If you search through the defect log, you will see the phrase "existing documents" used dozens of times. That appears to be how many discussions in WG4 end. It shuts down debate like an appeal to "national security" or "executive privilege", arguments that trumps all others. It doesn't matter what WG4 previously told SC34, or what JTC1 Directives say, if ISO/IEC 29500 does not match what Microsoft Office actually writes out, then this is by definition a drafting error, and the standard will be "corrected" to conform with MS Office. Let that sink in for a little, until you realize how backwards this is.

I invite you to go back to the defect log [PDF] and search for "BRM". You will find several oddities. For example, among these proposed changes are some that actually reverse BRM decisions. Yes, you heard me correctly. SC34/WG4, the Microsoft-dominated committee that maintains OOXML, is undoing various BRM decisions that enabled OOXML to be approved in the first place. Why? Well, of course, to make the standard conform more to Microsoft Office.

For example, take DR 09-0159 "General: Unintended incompatibilities between Transitional schema and Ecma-376" or DR 09-0275 "BRM: serial date representation" with this comment:

Although this text is in accord with the detailed amendments resolved at the BRM, it is against the spirit of the desired changes for many countries. We believe that due to time limitations at the BRM, this change was made without sufficient examination of the consequences, and was made in error by the BRM (in which error the UK played a part).

(Norbert Bollow, a member of the Swiss NB, has some good analysis of the return of the leap year bug in spreadsheets. And Jomar Silva with the Brazilian NB tracks some additional breaking changes on a wiki.)

Ah, So WG4 is now interpreting the "spirit of the BRM" through their shamanic communion with the ISO Weltgeist, and each time their oracle come back with the same response: "Change OOXML so it 'conforms' to Microsoft Office 2007". How convenient for Microsoft.

For most standards, multiple vendors work together to improve interoperability and to increase their conformance with the standard. But with OOXML a single vendor stuffs the committee and works to make the standard better conform to Microsoft's monopoly product.

So although Microsoft Office does not conform to ISO/IEC 29500 today, I have no doubt that within a few months it will fully conform. But not a single line of code will have changed in the Office product. Office 2007 will be retroactively made to conform to ISO/IEC 29500. What will happen is the standard will be modified to match that single vendor's products, by misapplication of an ISO procedure intended for fixing minor drafting errors.

So why go through all this trouble? I believe this is all about getting the OOXML standard "corrected" so Microsoft can push for it to get it officially adopted around the world. The only reason they've held back so far is because MS Office does not actually implement ISO/IEC 29500 today. So it would have been counter productive for them to push for official adoption. However, once this oversight is remedied, by changing the standard to match their product, then watch out.

The side effect, perhaps unintended, is that the OOXML standard is thus clearly marked to be unstable and unsuitable for adoption or implementation. With 800 pages of defects and more being found, and a Microsoft-dominated committee that changes the standard with no objective technical justification, the exact contents of the OOXML standard is tentative, uncertain and temporary. Four corrigenda documents and two amendment documents are currently being balloted, including many breaking changes. More corrigenda and amendments are on the way. There is no provision for a version attribute or any other indicator to declare which of the multiple incompatible versions of the standard a document conforms to. What competitor would risk implementing the standard, knowing that Microsoft dominates WG4, which has shown it is willing to change the standard at Microsoft's whim? The risk is simply too large. A competitor would simply be putting their head in the lion's mouth.

And at the same time WG4 rushes to make OOXML conform to Office 2007, Microsoft is moving on with Office 2010, now in technical preview. Office 2010 will be extending OOXML in hundreds of places. Where is SC34 in this? Where is the new work proposal for OOXML 1.1? Where the are discussions? The drafts? None of this exists. If Microsoft wanted to, they could have submitted these changes to SC34 at the recent meeting in Seattle, but they preferred to reserve discussion of the Office 2010 changes for a private meeting in Redmond the day after the SC34 Plenary ended, a snub to SC34 and their fictional control of OOXML.

So Microsoft is now off extending OOXML, and this whole ISO escapade with OOXML seems for naught. (I hear also that Microsoft is also backing off the submission of their Extensible Page Specification (XPS) to ISO as well, saying that "an Ecma Standard is good enough".) It appears that Microsoft got what they wanted from ISO and is moving on. Who said it would last more than a night? As my grandmother used to say, "Why buy the cow when you can get the milk for free?"

In any case, the future looks like something like this:

ISO/IEC 29500:2008's future is uncertain. If the whole i4i patent thing goes against Microsoft, the standard will probably need to be withdrawn.
ISO/IEC 29500 with Corrigenda and Amendments will eventually line up with Office 2007 SP2 sometime in 2010/11.
But before that happens, Office 2010 will ship with hundreds of extensions that are not described in ISO/IEC 29500 but are documented in proprietary "implementation notes" on Microsoft's web site.
"Office 15" will ship sometime around 2013. It will have further proprietary extensions to ISO/IEC 29500, also not standardized. Office 15 will still be supporting "transitional" OOXML, just like Office 2007 and Office 2010 did. "Transitional" OOXML is the variation that has all the deprecated crud, like VML and "DoItLikeWord95" in it.
"Office 16" will ship sometime around 2016. It will finally support the "strict" schema of ISO/IEC 29500, but with 3 generations of proprietary extensions layered on top of it. And that assumes ISO/IEC 29500 actually still exists. In 2015 — 5 years after its last amendment — it will be up for "periodic review" in ISO and may be withdrawn if it appears to have been abandoned by Microsoft.

The pattern is clear: OOXML will be extended by Microsoft much faster than it will be standardized and corrected by ISO. This will make the ISO version of OOXML, currently not supported by Microsoft, even more irrelevant in the future.

The Final OOXML Update: Part II

2009-10-16T08:30:00.002-04:00

In Part I of this OOXML update, my first post on the topic in over a year, I showed you how Microsoft maintains strong control over the OOXML standard. Despite their earlier promises that control of OOXML would be handed over to an independent, international committee, a look at attendance records reveals that the committee that maintains OOXML (JTC1/SC34/WG4) consists mainly of Microsoft employees, who outnumber any other company or organization on the committee 10-to-1.

In this, Part II of my OOXML update, I'll tie up another loose end from the immediate aftermath of the DIS 29500 BRM.

Let's start by casting our memories back to April, 2008. The BRM was over. NBs had reviewed thousands of pages and submitted thousands of defects. And Microsoft/Ecma made thousands of responses. And the BRM approved thousands of changes. Then, as a final step, NBs were asked if they wanted to change their vote from their original September 2007 vote, based on the changes made to the standard by the BRM.

Curiously, NBs were asked to make their final decision without actually seeing the text of the standard they were being asked to approve. ISO leadership denied requests from several NBs, a formal SC34 resolution requesting this text, as well as NB appeals, all which asked to have access to the "final DIS" text that would eventually be published. The ISO chief, in his response to the NB appeals, called the final text of OOXML "irrelevant" (prophetic words, indeed!) and would only permit NBs to have access to a list of over 1,000 resolutions from the BRM, many of which gave great editing discretion to the Microsoft consultant who would eventually produce the final text of the specification.

I discussed why the lack of a final DIS text was a problem back in May 2008:

We are currently approaching a two month period where NB's can lodge an appeal against OOXML. Ordinarily, one of the grounds for appeal would be if the Project Editor did not faithfully carry out the editing instructions approved at the BRM. For example, if he failed to make approved changes, made changes that were not authorized, or introduced new errors when applying the approved changes. But with no final DIS text, the NB's are unable to make any appeals on those grounds. By delaying the release of the final DIS text, JTC1 is preventing NB's from exercising their rights.

Would you make thousands of changes to code and then not allow anyone to test it, and then release it internationally? Of course not. Doing so would amount to professional malpractice. But that is essentially what ISO did with OOXML.

Well, guess what happened? Indeed, the published text of OOXML failed to carry out all of the editing instructions made by the BRM. Several of the BRM resolutions were ignored altogether. Several others were applied inconsistently or erroneously. Although I am aware of no systematic review of all 1,000+ BRM decisions, some NBs have gone back and reviewed the published text against BRM decisions that should have addressed their own NB's reported comments. They have found many "discrepancies" and these have now been reported as defect reports [PDF].

Whether the flaws in the published text are intentional or accidental, grave or minor, does not really matter here. Errare humanum est. The problem is that it was 100% predictable that human error would cause problems like this when dealing with text changes of this volume. The issue is not whether there will be errors introduced. The presence of many errors was guaranteed. The question was whether NBs are entitled to base their vote on all relevant information, including the final text of the standard, or whether relevant information, indeed the most relevant information -- the text of the specification they are voting on -- may be withheld from inspection. For ISO leadership to deny NBs the ability to review the very text they were voting was irresponsible.

The good news is that ISO leadership has changed since this debacle, and JTC1 is currently revising Fast Track procedures, in part to deal with the abuses of DIS 29500. One of the changes they are making is to the Fast Track procedure will require that the final DIS text be distributed to NBs before the final vote. This is progress and it is good to see these changes made, though it is unfortunate that it required a failure before such obvious and prudent precautions were instituted. Leadership entails foreseeing and preventing problems, not simply reacting to them. In any case, the NBs that appealed to ISO on the basis of not being allowed to see the final text should feel vindicated now. The NBs of India, Brazil and South Africa were right.

In Part III of this Update, I'll bring the story up to the present day, and in Part IV I'll update the story through the year 2016.

Protocols, Formats and the Limits of Disclosure

2009-10-12T21:39:00.001-04:00

A few words today on an important distinction that deserves greater appreciation, since it lies at the heart of several current interoperability debates. What I relate here will be well-known to any engineer, though I think almost anyone can understand the gist of this.

First, let's review the basics.

Formats define how information is encoded. For example, HTML is the standard format for describing web pages.

Protocols define how encoded information is transmitted from one endpoint to another. For example, HTTP is the standard protocol for downloading web pages from web servers to web browsers.

There are other such format/protocol pairs, such as MIME and SMTP for emails. When we talk about "web standards" we talk about formats (often described by W3C Recommendations) and protocols (often described in IETF RFCs).

An instance of data that conforms to a given format standard might be given any number of terms: a web page, a document, an image, a video, etc., according to the underlying standard. The instance of a format is a data, bits and bytes that you can save to your hard drive, burn to a CD, email, etc. Data in a format is persistent and has a static representation.

But what is an instance of a protocol? It is a transaction. It is ephemeral. You can't easily save an instance of HTTP or SMTP on your hard drive, or email it to someone else. A protocol is complex dance, a set of queries and responses, often a negotiation of capabilities that preface the data transmission.

There is a key distinction between formats and protocols when it comes to interoperability. The key is that a protocol typically involves the negotiation of communication details between two identifiable parties, each of whom can state their capabilities and preferences, as well as conform to the capabilities of the datalink itself. Software running on each endpoint of the transaction can adapt as part of this negotiation.

You may be familiar with this from the modem days, where this "handshaking" procedure was audibly manifest to you whenever you connected to a remote host. But although you don't hear or see it, this negotiation still occurs with protocols today, behind the scenes.

For example, when you request a web page, your client negotiates all sorts of parameters with the web server, including packet size and timings (at the TCP/IP level) to authentication, language, character set and cache preferences (at the HTTP level). This negotiation of capabilities is essential for handling the diversity of difference web servers and web clients in existence today.

With a protocol, you have two technical endpoints communicating and negotiating the parameters of the data exchange. In other words, you have software on both ends of the communication able to execute logic to adapt to the needs of the other endpoint and the capabilities of the underlying datalink.

However, when it comes to formats, things are different.

Let's use an word processor document as an example of a format instance. I author a document, and then I send it out, via email, as an attachment on my blog, burned on a conference CD-ROM, posted to a document server or whatever. I have no idea who the party on the receiving end will be, nor what software they will be using. They could be running Microsoft Office, but they could also be using OpenOffice, Google Docs, Lotus Symphony, WordPerfect, AbiWord, KOffice, etc. I, as the document author, have no ability to target my document to the quirks of the receiving party, since their identity and capabilities are unknown and in general unknowable.

Since a document is not executable logic, it cannot adapt to the quirks of various endpoints. A document is static. When it comes time to interpret the document, you don't see two vendor endpoints adapting and negotiating. You see only one piece of software, the receiving party's application, and they need to interpret a static data instance in a given format.

In other words, with document formats, there is no dynamic negotiation, because at the time when your write a document out, you have no idea what the reading application will be. And although the application that reads the document may know the identity of the writing application (via metadata stored in the document for example), it has no ability to negotiate with the writing application, since that application is not present when the document is being loaded.

OK. Simple enough. However, a confused understanding this distinction will lead you to muddled reasoning about interoperability and how it is achieved.

Although it is not ideal, having Microsoft disclosure the details of exactly how they implement various proprietary protocols and even their quirky implementation of standard protocols, this may enable 3rd parties to code to these details. If the disclosure is timely, complete and accurate, this information may be useful. I think of the SAMBA work, for example.

However, no amount of disclosure from Microsoft on how they interpret the ODF standard will help. We see that today, with Office 2007 SP2, where it strips out ODF spreadsheet formulas. Having official documentation of this fact from Microsoft, in the form of "Implementation Notes" does not help interoperability. Why? Because when I create an ODF document, I do not know who the reader will be. It may be a Microsoft Office user. But maybe it won't. It very well could be read by many different users, using many different programs. I cannot adapt my document to the quirks of all the various ODF implementations.

When you deal with formats, interoperability is achieved by converging on a common interpretation of the format. Having well-documented, but divergent interpretations does not improve interoperability. Disclosure of quirks is insufficient. Disclosure presumes a document exchange universe where the writing application knows that the reader will be Microsoft Office and only Microsoft Office and therefor the writer can adapt to Microsoft's quirks. That is monopolist's logic. Interoperability with competition only comes when all implementors converge in their interpretation of the format. When that happens we don't need disclosures. We just follow the standard.

The Final OOXML Update: Part I

2009-10-01T09:30:00.008-04:00

I have not written a blog post on OOXML for well over a year now. My last post on this topic was on August 17th, 2008 and covered the contentious appeals process which followed the DIS 29500 Fast Track ballot. So I hope that one more post, 14 months later, will not seem excessive to my critics. There is too much good stuff going on with ODF these days, with ODF 1.2 coming soon, inter-vendor work at plugfests, the ODF Toolkit, and continual national adoption, for me to waste much time on OOXML. "Let the dead bury their own dead" is my attitude here. That said, I have received several requests for an update on OOXML, so I will oblige with some quick observations in what (I hope) is my final update on this sad chapter in standardization.

I'll structure this update over a handful of posts, each one looking at a single topic. In this post I'll cover the tight control Microsoft maintains over the OOXML standard, despite their earlier assertions to the contrary.

From the beginning of the Fast Track procedure Microsoft encouraged NBs to approve OOXML with the promise that their approval of the specification would guarantee that it would be handed over to the "global community" for maintenance. Vote against the standard — because it was obviously flawed — and you would lose this unique opportunity to transfer control from proprietary interests at Microsoft to the benevolent and international meritocracy of ISO. This was one of the main "selling points" for OOXML and what Microsoft repeatedly sold.

For example, here was Jerry Fishenden, Microsoft's National Technology Office in the UK:

There's an easy question to consider here: would you prefer the Microsoft file formats to continue to be proprietary and under Microsoft's exclusive control? Or would you prefer them to be under the control and maintenance of an independent, open standards organisation? I think for most users, customers and partners that's a pretty easy question to answer: they'd prefer control and maintenance to be independent of Microsoft. And the good news is that the Open XML file formats are already precisely that: currently under the control of Ecma International (as Ecma-376) and, if the current voting process is positive, eventually under the control of ISO/IEC

Or Microsoft's Jason Matusow:

I still hear patently untrue claims that MS controls Open XML - this wasn't true following the adoption of Ecma 376, and is now permanently a moot argument.

Microsoft Australia said:

This is encouraging and should be a reminder to all that the Open XML standard will be controlled by the international community not by any commercial business or other organisation - including Microsoft.

Chris Capossela , Microsoft Senior VP said it thus:

If Open XML is approved as an ISO/IEC standard, the story would not end there – like any other standard, maintenance affords the opportunity for continually updating and improving the standard. In this case, the global community would be in control of the evolution of this standard going forward – a fitting result given that this format will be widely used around the world for years to come.
.
.
.
Now, the global community has the opportunity to take control of the future of the specification by ratifying Ecma 376 as an ISO/IEC standard. We know that it will be in good hands when this happens based on the tremendous work and improvements that have been made to the specification during the ISO/IEC process over the past 14 months. We are committed to the healthy maintenance of the standard once ratification takes place so that it will continue to be useful and relevant to the rapidly growing number of implementers and users around the world.

(If you watched the video linked to from the letter, you will hear Chris say that Microsoft "has transferred stewardship of the file formats to the global community".)

Well, that was what was promised. But how did it turn out in reality?

Let's take a look at who actually attends meetings of SC34/WG4, the technical committee that should have made the question of OOXML control "moot" and puts it "under the control of ISO/IEC".

If you look at the attendance records, summarized in the following table, you will find that the committee regulars consist primarily of Microsoft employees. In many of the meetings, Microsoft employees outnumber all other attendees combined. And then there is the "Microsoft Co-Prosperity Sphere", the Microsoft consultants, Microsoft business partners and Microsoft-funded research institution, which further contribute to Microsoft's effective domination of the meetings.

Person	Employer	NB	4/16/2009	4/30/2009	5/14/2009	5/28/2009	6/11/2009	6/22/2009	7/16/2009	7/30/2009
Makoto Murata	Consultant	Japan	1	1	1	1	1	1	1	1
Doug Mahugh	Microsoft	Ecma	1	1	1	1	1	1	1	1
Shawn Villaron	Microsoft	Ecma	1	1	1	1	1	1	1	1
Dave Welsh	Microsoft	US	1	1	1	1	1	1	1	1
Jirka Kosek	Consultant	Czech Rep	1	1	1	1	1		1	1
Rex Jaeshcke	Microsoft Consultant	Ecma	1	1		1	1	1	1	1
Gareth Horton	Data Watch	UK		1	1	1	1	1	1	1
Jesper Lund Stocholm	Ciber	Denmark	1	1	1		1	1	1
Isabelle Valet-Harper	Microsoft	Ecma	1	1	1	1	1		1
Mohamed Zergaoui	Innovimax	France	1	1	1	1	1		1
Mario Wendt	Microsoft	Germany	1	1		1	1	1	1
Alex Brown	Griffin Brown Digital Publishing	UK	1	1		1	1	1	1
Florian Reuter	Novell	Ecma	1	1		1	1		1
Jaeho Lee	University of Seoul	Korea		1	1		1		1
Caroline Arms	Library of Congress	Ecma	1	1	1
Francis Cave	Francis Cave Digital Publishing	UK	1	1	1
Rick Jelliffe	Consultant	Unauthorized				1	1			1
Nasser Kettani	Microsoft	Côte d'Ivoire					1		1
Pia Lange	Dansk Standard	Denmark					1	1
Kimmo Bergius	Microsoft	Finland				1		1
Juha Vartiainen	Finnish Standards	Finland	1
Jean Paoli	Microsoft	Ecma			1
Sam Oh	Sungkyunkwan University	Korea						1
Klaus Peter Eckert	Fraunhofer Fokus	Germany						1
Jung-Jin Yang	Catholic University of Korea	Korea						1
Keld Simonsen	RAP	Norway						1
Amruta Gulinakar	Microsoft	Ecma							1

So is this really handing over control? Is it really independent? And is it really global?

Let's look at it in graphical form. In this chart I tally up the participation from each entity (company, organization or unaffiliated individual) attending WG4 meetings. This takes account of all 8 published meeting minutes for WG4. It shows the total participation over those meetings. So if a company sent 8 people to one meeting, this is scored the same as if they sent 1 person to each of 8 meetings. It is the overall participation for an entity that is measured relative to the total participation of all entities at the meetings. Note also that the "Microsoft" tally is of Microsoft employees only. The rest of the Microsoft Co-Prosperity, for purposes of this chart I am all counting as "independent" entities. So this picture is the most complimentary view possible of the degree of concentration in WG4. Obviously, Microsoft's control is much higher if we take account of these other inter-entity obligations.

I suppose this is "global" in a sense, in the same way one could stage an "International Food Festival" and then have McDonalds show up and contribute a Big Mac from the U.S., a Big Mac from Germany, a Big Mac from the Ivory Coast, a Big Mac from Finland and another Big Mac from Brazil and so on. Certainly, you could claim this was "international", but you would be laughed right out of the festival if you did.

By way of comparison, here it the same analysis, plotted on the same scale, for the most recent 8 meetings of the OASIS ODF TC. As you can see it is much flatter. No company has more than 20% or so of the participation, and no two companies combined have control of the TC.

Now don't get me wrong. There are certainly some independent people in WG4 and I would not want anyone to denigrate their efforts. They are not all Microsoft employees, consultants, business partners and research institutions that Microsoft is funding. But they are mostly so. Attend any OOXML meeting and look to your right, look to your left, and most likely one is a Microsoft employee and another is economically tightly tied to Microsoft.

Of course, I would not expect that Microsoft would be absent from this work either. After all, they authored the specification and have most of the relevant technical experts. But a glance over the attendance records shows that they are not gracing the committee with their file format gurus. Instead they are stuffing it with "program managers" and "technology directors" and other assorted non-experts. The problem appears to be that their file format experts are all cursed with American residency and so have little value in stuffing a committee that has one-country/one-vote rules. Thus the spectacle of a room filled with Microsoft employees wrapped in different colored flags.

So I don't think one can truthfully say (in Jason's words) that it is "patently untrue" that Microsoft controls OOXML. Or that OOXML "control and maintenance" is "independent of Microsoft" as Jerry promised it would be, or that the "global community would be in control" as Chris said. I don't think those are accurate statements, given the evidence. I think the results fall far short of what was promised back when Microsoft were trying to secure a positive vote in ISO.

And this is not just me complaining. At the recent SC34 Plenary meeting in Seattle, delegates from several NBs approached me, voicing concerns at the domination Microsoft was asserting over the committee. (Perhaps this explains the substantial number of people who attended only one WG4 meeting and then never returned?) There is no easy solution here. Remember, we are dealing with a company that has demonstrated that it is willing to spend millions of dollars to protect its Office monopoly franchise from any pro-competition standards initiative.

The Former ISO Secretary-General, when interviewed about the OOXML farce, was asked about claims of Microsoft domination and admitted that he was powerless to stop this:

Companies have no direct vote on the International Standards, which are adopted according to voting by national member bodies, on the basis of one vote per country... As a stakeholder in the process, Microsoft and other interests certainly participated in the process to establish national positions. ISO and IEC national members are fully responsible for the way national votes are formed and relevant national interests consulted.

Evidently there is no one capable of fixing this. ISO says that domination by a single corporation is not their responsibility, because only NBs vote and each NB determines its own participation rules. But individual NBs also don't see a problem, because any single one of them only has one Microsoft employee at the meeting. So the NB itself is not necessary stuffed (although that does happens occasionally as well). So by placing Microsoft employees in many NB delegations and putting the overflow into the Ecma delegation, Microsoft can still dominate the ISO committee and not trigger a rule violation in ISO or in any NB.

This is essentially how Microsoft hacked ISO. Now that the flaw has been demonstrated, any large international corporation with sufficient funds and interest can exploit it as well. So long as the rules remain as they are, ISO is vulnerable. ISO defends this criticism by pointing out what good work they've done in the past, and how they rarely have problems of this kind before. But this shows little appreciation for the nature of the problem which have been demonstrated. It is like arguing that a newly discovered (though long latent) security flaw in an operating system is insignificant because you've never had an attack before now. Of course, this misses the point entirely. Once the vulnerability is known and publicly exploited, you're living on borrowed time until you can secure the system. Today ISO is living on borrowed time and is very close to becoming a Microsoft-infested zombie committee.

That is all for Part I of this update. In the next Part I'll look at the maintenance of OOXML, and the most peculiar way in which the Microsoft-dominated committee is putting aside BRM decisions and making other breaking changes to the specification, in an bizarre attempt to make ISO OOXML conform to to the Microsoft Office standard.

What's New in ODF Maintenance?

2009-09-17T22:55:00.006-04:00

Some Q&A, in the form of a self-interview. As with anything on this blog, these are my opinions.

Question: How are you doing today, Rob?

Rob: Very well thank you. I just finished attending a good set of working group (WG) meetings, and the Plenary meeting of ISO/IEC JTC1/SC34 in Seattle, Washington.

Question: Anything newsworthy to report?

Rob: Most of the meeting was routine, moving various documents forward, inch by inch, at the glacial pace we expect from ISO. But one noteworthy event this week was the first meeting of SC34/Ad Hoc Group 3.

Question: What is Ad Hoc Group 3?

Rob: Ad Hoc Group 3, or "AHG3," is an temporary working group that was charged with determining how SC34 would fulfill its maintenance obligations relating to ISO/IEC 26300, the ISO/IEC transposition of the ODF standard. Francis Cave, of the UK, presided over the AHG3 meeting, which was well attended by 20 or so delegates.

Question: What do you mean by "maintenance"?

Rob: In this context maintenance refers to the activities that occur between major revisions of a standard, primarily the collection of defects, and the approval and publication of corrections.

Question: So what did Ad Hoc Group 3 do?

Rob: AGH3 met for around 8 hours over two days. Its most notable act was to orchestrate its own dissolution, by proposing that a new, permanent working group, WG 6, be created by SC34 to carry out pretty much the same charge given to AHG3. WG6 will likely also have the same Convenor and participants as AHG3.

Other than that, the primary benefit of the AHG3 meeting was to bring everyone up to speed on the current state of various ODF defect reports and errata documents, and to discuss ways in which we can better align the ISO/IEC version of ODF and keep it in alignment, with the OASIS version.

Question: So who owns ODF maintenance?

Rob: The OASIS ODF TC owns the maintenance of the OASIS ODF standard, and WG6 will own this activity for the equivalent ISO/IEC text. However, neither committee has absolute freedom of action, both being governed by applicable procedural rules of their parent organizations, as well as various joint agreements between OASIS and JTC1.

Question: Won't this lead to a divergence of the OASIS and ISO/IEC versions?

Rob: It is a stated goal of JTC1 that ISO/IEC 26300, as the PAS transposed version of ODF, remain in sync with the OASIS version of ODF. This goal is shared with OASIS. OASIS and JTC1 have jointly approved a set of maintenance principles which call for the OASIS ODF TC to take the lead on maintenance.

Although we have not worked out all the details, a natural way to avoid divergence is to treat this as a two-phase process:

Defect reports from SC34 are submitted to the OASIS ODF TC
Corrections originate in the OASIS ODF TC, and are then transmitted to SC34

Question: But that doesn't guarantee that these texts will not diverge, right?

Rob: At the very least there will be a delay between the correction of the OASIS version of the text and the publication of the ISO/IEC version of the correction. This delay will typically be around 5 months. If both OASIS and SC34 adhere the agreed maintenance principles, any divergence should amount to no more than such a delay.

Note that we already have some degree of divergence. For example, OASIS published ODF 1.0 Approved Errata 1.0 in November 2008, and submitted these changes to SC34 where they have not been acted on. And OASIS published ODF 1.1 back in early 2007 and has failed to submit these changes to SC34. Although there appears to be no adverse market effects from this degree of divergence, both sides agree that this is not an ideal situation and have agreed in principle to take steps to remedy this situation.

Also, we shouldn't worry about the possibility of intentional or malicious divergence, since careful consideration of the ownership of the copyright and trademark for the ODF standard, as well as terms governing any associated patent claims, suggests that any rogue attempt to fork the ODF standard would be fraught with peril. ISO/IEC 26300 may lag OASIS ODF, but it cannot go beyond the corresponding OASIS text in any substantive way.

Question: So what role does WG6 have then?

Rob: SC34 has several WGs, each dealing with a different standard or group of standards. This WG structure is important for National Bodies, which often allocate their resources (personnel, travel budgets, etc.) at the level of a WG. It is also a logistic convenience for scheduling meetings.

WG6's charter sounds impressive, "All SC 34 projects and activities relating to the maintenance of ISO/IEC 26300 OpenDocument Format". However, since JTC1 has already assigned coordination of ODF maintenance to the OASIS ODF TC, it is not clear what, if any, residual tasks remain for WG6 to perform. But even if its official responsibilities are minor, WG6 does serve a useful purpose as a focal point for SC34 technical experts having an interest in ODF. Under the aegis of WG6, these experts might, for example, create defect reports for submission to OASIS and give feedback on OASIS draft corrigenda.

Question: But what about contributions in the other direction? How do SC34 technical experts contribute to ODF?

Rob: As stated in the joint memorandum of principles between OASIS and JTC1: "National Body input, including but not limited to the submission of defects and amendments, can best be achieved by the participation of JTC 1 experts in the ODF TC of OASIS".

The exact answer will depend on your specific goal. If you participate via SC34, your opinion will count as one vote on your NB delegation, and in turn your delegation will have one vote among many NBs in SC34, which amounts to one source of feedback, among many others, to the OASIS ODF TC. So essentially you will have a much diluted and rather indirect way of expressing your thoughts on the technical work of the OASIS ODF TC.

Or you could join OASIS and and the ODF TC directly, as an individual member, in which case you can contribute directly in our work and your vote will be equal to mine or any other member of the ODF TC.

So if your priority is to present a national view via national body participation, then by all means participate via SC34 and WG6. This is also good if your priority is racking up frequent flier miles via attendance at international meetings. But if you are looking for the most effective way to contribute directly to the standard, then you cannot do better than joining the OASIS ODF TC. This is probably why nine members of JTC1/SC34 already are OASIS ODF TC members.

Question: So we've talked so far about maintenance. But what about major revisions, such as ODF 1.2, how are they handled?

Rob: Major revisions developed in OASIS may be submitted to JTC1 via the PAS transposition process. This results in a JTC1 ballot and does not directly involve SC34 technical experts, although in some cases the SC34 national "mirror committee" may be consulted by their NBs in the preparation of a national position on the PAS ballot. In any case, WG6 would have no formal role in this process, nor can I imagine an informal role for them.

A Standard I Would Use: Auto Unsubscribe

2009-08-19T08:50:00.000-04:00

I don't get a lot of spam, at least not in the traditional sense of "unsolicited commercial email". But I do get a lot of solicitations from online retailers with whom I have done business. As we all know, even a single order can trigger weekly emails. Multiply that by all places I do business with, and I end up with a lot of unwanted emails.

Certainly, the vast majority of these companies offer an unsubscribe option, with instructions clearly marked at the bottom of the email. These instructions tend to have a URL which I click and one of three things happen:

Link automatically unsubscribes me
Link takes me to a web page that asks for confirmation and maybe a little survey, or a list of mailings which I can opt-in or out of.
Link takes me to a login page, where I need to remember my login id, navigate to a profile and perform several other steps before I can unsubscribe. I have some mailing lists that I have never been able to unsubscribe to at all. I end up defining inbox rules to delete the mailings altogether.

The idea for a standard is this: Can we encode these unsubscribe mechanisms, or at least the first two mechanisms, in a standard way in the mail message itself, so that an email client can allow the user to simply push a button and activate the unsubscribe procedure? If done right, I could even be in my inbox view and select several emails and unsubscribe to those lists all at once. Ideally no further user interaction would be required. And certainly I want to avoid the requirement to hunt through an email for the unsubscribe link.

Since emails can come in a variety of formats, from text, to HTML to RTF, it might make sense to handle this in the mail headers rather than the email body itself.

Of course, we want this to be simple and declarative and not require general-purpose scripting support, for simplicity and security reasons.

I think this would be a relatively simple standard to create. We just need some conventions for an email to declare a RESTful API for unsubscribing to a list.

Of course, maybe there is something like this already out there?

How Not to Read a Patent

2009-08-13T21:05:00.006-04:00

There is perhaps no occasion where one can observe such profound ignorance, coupled with reckless profligacy, as when a software patent is discussed on the web. Note the recurring pattern, which is repeated every two weeks or so. A patent issues, or a patent application is published or patent infringement suit is brought, and within minutes the web is full of instant pundits, telling us what the patent covers, how it should not have been granted, how it is entirely obvious, or how it applies to everything in the world, and how it presages a self-induced mutually assured destruction that now leads us on to the plains of Armageddon. If I had a nickel for every time this happens...

By way of disclaimer, I am not a lawyer, but I am blessed that my self-avowed ignorance in this area is coupled with a certain knowledge of the limits of my understanding, a handicap seemingly not shared by many other commentators. I know what I do not know, and know when to seek an expert.

In the past few days we have had a bumper crop of pontification on the significance of two XML-related patents, one newly issued to Microsoft (7,571,169), and another older one (5,787,449) owned by i4i, whose infringement has resulted in a large judgment and injunction against Microsoft. I've found the web coverage of both patents to be an unmitigated muddle.

I'm not going to comment on the merits of either one of these patents, but I'd like to make a few basic observations that may be of some assistance to those who comment on future patent issues.

A patent has a description known as the "specification". And it has a list of numbered "claims". Although the specification can define terms that are then referred to in the claims, it is the granted claims that define the scope of the patent, not the specification.
If all you do is read the abstract and the first few paragraphs of a patent, then you may know the general topic of the patent, but you do not really know its scope. If you then go off and cry, "Oi vey, this patent covers XHTML, SVG, RDF, Pringles and Obama Healthcare Plan" then you do your readers a disservice. You must parse the very specific and often obtuse language of the claims in order to understand exactly what a patent covers. There is no short cut. This is not like a book where you can understand the plot by reading the back cover. But over and over again, I see people who just read the abstract, maybe glanced at a diagram, and then feel equipped to hold forth at length on the substance of the patent.
When you try to understand patent claims, you will encounter a dense form of legal English. Claims are not written for a layperson and do not presume that you will understand it easily. The drafting of patent claims is a black art, like writing device drivers, and if you are not versed in its intricacies, then your statements on any given patent are apt to be wide of the mark. Claims are full of magic words. Know what you do not know. If you do not recognize at sight and know the interpretation requirements of a means-plus-function claim (which is key in the '449 patent), or you are not crystal clear on the distinction between the verbs "consist" and "comprise", then you probably should not be the first (or loudest) person to speak on what a patent claims.
If you are reading an application, know that during the "prosecution" of that patent, when it is reviewed by the USPTO, some of the claims of the patent may be thrown out, for any of several reasons, including prior art identified by the examiners. However, the specification of the patent is unlikely to change much. So an issued patent often has a very broadly-written specification, that covers the entirety of the originally claimed invention, though the the issued patent might have only a subset of the original claims allowed. So if you are have an issued patent and you look at only the specification, you can easily be fooled into thinking it covers far more than it does. For example, the '169 patent from Microsoft had half the original claims thrown out in the prosecution. If you don't know that and are reading only the specification, not the granted claims, then you will incorrectly think the patent is far broader than it actually is.
Know what a priority date is, and how that is affected by a continuation. I've read all sorts of nonsense based on not appreciating that. Take a look at the '169 patent, for example. It says it was filed in 2004. But if you look closely you see it was a continuation of a 2002 application. You can moan and groan all you want about prior art, but if you don't get your dates right you're off to a bad start in your analysis.
In an infringement suit, like with the '449 patent, be sure to look at the actual court record. Typically there is a Markman (claim construction) hearing, where the court will determine the meaning of terms used in the patent claims. If you have not read the court's claims construction opinion in the i4i versus Microsoft case, then your commentary is uninformed by probably the most important document in this case (well, next to the patent itself).

Well, that's enough for tonight. Repent. Sleep on it. And realize that making sense of a complex patent takes time, if you're going to do it right. Ergo, the first impressions you read from the instant pundits on the web will tend to be shallow, imperfectly informed and often wrong. Heck, even everything I said in this post may be wrong.

ODFDOM 0.7 Released

2009-07-21T14:05:00.002-04:00

I'm pleased to report that the 0.7 release of the ODF Toolkit Union's ODFDOM library has just been released. This is an open source (Apache 2.0 license) Java toolkit for programmatically reading, writing and manipulating ODF documents. The code is 100% Java and does not require that you have OpenOffice or any other ODF editor installed. It operates directly on the document itself.

You can download the JAR and JavaDoc from the distribution here. You will want to go through J. David Eisenberg's tutorials on ODFDOM as well.

If you sign up for a Toolkit Union account, you'll be able to participate in the user's mailing list (users@odfdom.odftoolkit.org) where we welcome your ideas, bug reports and patches. Or better yet, move over to the dev list (where the real fun is) and contribute actively!

The ODF Toolkit also has an active "Conformance Tools" project, including the ODF Validator, and our most-recent project, AODL, which is a .NET/C# module.

At the Toolkit Union we're able to host more such ODF-related toolkit projects, in other programming languages. I don't see any good reason to have two Java API's for ODF, but I'd love to see broader coverage, especially in some of the more-widely used scripting languages like Python or Perl. I have even seen some interest in a PHP/ODF module. I think there is some advantage to getting a "critical mass" of programmers interested in working with ODF together in one place to bounce ideas off of each other. So if you are interested in joining this effort, sign up at the Toolkit Union's web site, or send me a note if you want to talk first.

[22 July 2009 -- more on ODFDOM 0.7 in this post by project lead Svante Schubert]

July 20, 1969

2009-07-19T10:00:00.004-04:00

Paul Calle (1928-) drew science fiction illustrations, including for now-legendary magazines like Galaxy Science Fiction and Amazing Stories. Then a funny thing happened. The science fiction became real. Sputnik launched in October, 1957. NASA was created in 1958. Glenn orbited the earth in 1962. The space race was on.

In the Fall of 1962 NASA chief James Webb created the NASA Art Program. Under the direction of James D. Dean, artists were invited to Cape Canaveral, to observe and interact with the astronauts and staff and to document history in the making.

The invitation letter stated:

When a major launch takes place, more than two hundred cameras record every split second of the activity. Every nut, bolt and miniaturized electronic device is photographed from every angle...but as Daumier pointed out a century ago, "The camera sees everything but understands nothing." It is the emotional impact, interpretation and hidden significance of the events that lie within the artist's vision.

Paul Calle was one of the artists engaged, during the Mercury, Gemini and Apollo missions. Calle's illustration of the Apollo 11 moon landing is one of the iconic images of landing, as reproduced in the 1969 air mail stamp, Scott # C76, in 6-color Giori process:

We'll be remembering a lot tomorrow — the astronauts, the engineers, the journalists — that all contributed to what was a defining moment of the modern era. Let's also remember the artists and the part they played.

(And why don't we have any good moon landing songs from the time?)

How I think Wikipedia works

2009-06-30T15:40:00.001-04:00

I have a mental model of how Wikipedia works and behaves. This may not reflect reality, but it is how I, as an end-user, expect Wikipedia to behave. I think these are reasonable expectations regarding things like standards of proof and balance and that if the real Wikipedia differs substantially from these expectations, then we have a problem.

Please let me know if my mental model differs from reality.

First, I assume that we deal with facts, not opinions. So an editor cannot state a personal opinion such as, "Citizen Kane is the greatest movie ever made", since there is no objective, recognized scale for cinematic greatness.

However, saying, "Citizen Kane topped the list of 'Greatest Films' according to a 2002 poll of directors and film critics by Sight & Sound magazine" would be fine. It is a factual statement, albeit a statement about an opinion, but the factual portion of it is verifiable. It is a fact about an opinion and that is OK.

But if I made the statement, "Citizen Kane is the greatest movie ever made" and cited the Sight & Sound article, this would not be proper, since that article does not establish the fact of the greatest movie, but only the fact of a poll that collected opinions on the greatest movie. A fact about the existence of an opinion (or even a polled opinion) does not assert the truth of the opinion.

Similarly, a statement, "Gone with the Wind has been criticized for its long running time" would not be properly cited by merely referencing a source that states its length as 238 minutes. That citation would merely be evidence of its length, not that its length was inordinate. You need a citation for the length being criticized.

Similarly, if another recognized expert stated, "Gone with the Wind was too short and failed to cover the entire Mitchell novel", then I would expect both opinions to be mentioned, not merely selecting an arbitrary opinion.

I also expect that cited sources have recognized (not merely self-declared) expertise in the area. So, I would find it idiosyncratic if an article on cinema said, "Citizen Kane is the greatest film ever made, according to a fan blog post by Joe Blow, a ophthalmologist in Podunk, Michigan", since he would be a source cited outside any area of recognized expertise.

I also, as a user, expect Wikipedia to give a balanced view of issues. This does not mean equal time to all fringe opinions. Although I expect there to be multiple views presented on the propriety of the Iraq War, I would not expect that someone who believes that Abraham Lincoln was an alien from the planet Quthbral to have a section in the Lincoln article, even if he could cite a blog post or a photocopied article, or self-published book on the subject. Ditto for Flat Earth Society members, holocaust deniers and those who think the Apollo moon landing was filmed on a Hollywood sound stage.

On the other hand, I don't expect that every fact requires a citation. For example as a user, I don't expect to see citations whenever someone says "Mercury is the closest planet to the Sun". Similarly, I would find it odd if someone removed that assertion for lack of a citation.

However, I would be suspicious if someone writes something in the form, "Mercury is the hottest planet because it is closest to the Sun". Although the it is well known that Mercury is the closest planet, it does not follow that it is the hottest. In fact, Venus is the hottest planet. It is a subtle form of editorializing, where an editor can inadvertently introduce personal assumptions into an article. I'm assuming Wikipedia editors are on the watch for this kind of thing.

On the other hand, some things clearly logically follow from known facts. If we know that John Brown was buried on January 23rd, 1582, then we should, absent contrary evidence, safely be able to state that his date of death was on or before January 23rd, 1582. I would not expect someone to revert such a statement as being unfounded, speculation, original research, etc. It logically follows based on our knowledge of how the world works.

Does anyone know whether the above statements have any basis in the aspirations or actual practice of Wikipedia editors and admins? Sadly, my recent reading of some articles suggests that these reasonable expectations are routinely flouted and bear little resemblance to reality.

ODF Plugfest

2009-06-26T17:30:00.003-04:00

Although the term may be alien to some, "plugfests" have been around for around 20 years. A plugfest is when implementors of the same interface get together and test the interoperability of their products. In the beginning this was done with wired standards, USB, etc. (thus 'plug'). Over the years the term was applied to networking at all higher levels of the protocol stack. The concept is also applicable to document exchange formats like ODF.

We had an ODF Plugfest last week in the Hague. Although we've had interoperability workshops and camps before that attracted a handful of vendors, this was the first one that had nearly universal participation from ODF vendors. I'm not going to recap the details of the plugfest. Others have done that already. But I will share with you some of my conclusions, based on long discussions with other participants, from whose insights I have greatly benefited.

In an ideal world, specifications would be perfect and software applications would be bug-free and users would read the manuals and we would achieve perfect interoperability instantly by anointment of the salubrious unction of standardization. But to the extent this planet's population obdurately persists in imperfection, we are resigned to make additional efforts in pursuit of interoperability. We are not alone in this regard. The only standards that don't need to work on interoperability are those standards that no one implements.

We should use every licit technique at our disposal to give the user the best experience with ODF we can. In a competitive market you can not get away with telling your customer, "Sorry, your spreadsheet doesn't work because page 652, clause 23 says 'should' rather than 'shall'". If you did that you would not have that customer for long. (Unless, of course, you have a monopoly, in which case many seemingly irrational, anti-consumer actions can occur, seemingly without consequences.)

Further, I assert:

Users want real-world interoperability, and not excuses
Real-world interoperability is what users see and achieve in practice
Where vendors have the will to interoperate, achieving interoperability is a known technical problem, with known engineering solutions, but where the will to interoperate is lacking, there are no technical means of compelling interoperability
Interoperability lies at the intersection of technology, engineering standards, competition law, intellectual property and economics. There are no silver bullets, although there are a arsenal of proven techniques that can help to improve interoperability
Achieving interoperability is facilitated by a variety of cooperative activities, including standardization, test case creation, implementation testing, online validators, plugfests, defect collection and reporting

Going forward there is a promising constellation of efforts converging around ODF interoperability:

The OASIS ODF Interoperability and Conformance TC, charged with creating an ODF test suite
The OASIS ODF TC, finishing up work on ODF 1.2
OfficeShots.org, providing a way to test the interoperability of a document in many ODF editors
The ODF Toolkit Union, especially their open source ODF Validator
The Plugfest participants, who continue to add information and test scenarios to the plugfest's wiki.
Groups such as OpenDoc Society and OpenForum Europe which lend their organizational skills and enthusiasm to the effort, and often much more.

So, we're moving in the right direction. The key thing will be to sustain the momentum from the Plugfest and transition it into an ongoing effort, a Perpetual and Virtual Plugfest where the effort and the progress is continuous.

[6/29/09: I've received some emails on the photo, so here are the details:

The picture was taken at 3:30PM on the 2nd day of the workshop.

The lens was a Pentax DA 10-17mm "fisheye" zoom at 10mm. So that explains the projection distortion. The graininess and B&W was from post-processing using Nik Software's Silver Efex Pro and Sharpener Pro.]

ODF TC timeline

2009-06-23T11:50:00.002-04:00

I used a variation of this chart at the recent ODF Plugfest in the Netherlands. But the aspect ratio of a presentation slide doesn't suit this type of chart well, so here is a fuller version of what I showed there.

Those who are not familiar with standards development are sometimes amazed at how long it takes to develop a good standard. Perhaps the single-vendor, 6,000 page, 12-month escapade of OOXML in Ecma has skewed expectations. Fortunately, OOXML is the exception, not the rule. Achieving a multi-vendor consensus around a substantial technical standard will always be time-consuming, but it is time that is well spent.

OASIS standards go through several stages of development:

Working Draft (WD)
Committee Draft (CD)
Public Review Draft
Committee Specification
OASIS Standard

Progressing from one step to another is by ballot. The first 4 stages are advanced by vote of the Technical Committee (TC), while the last stage (OASIS Standard) is by a ballot of all OASIS members. As a draft advances through stages 1-4, an increasing degree of consensus is required. So, a CD requires only simple majority, whereas a Committee Specification requires 2/3 approval, with no more than 1/4 disapproval. Some of these stages allow iteration. So we can, and typically do, have several WD's and several CD's.

If you want more detail on the nitty-gritty details, here is a flow chart of the OASIS standards approval process.

I occasionally get a question along the lines of: "What has the ODF TC been doing for the past couple of years?" The following timeline should give you an idea. I've indicated the time spent developing ODF 1.0 and ODF 1.1, along with some other milestone activities, such as the PAS transposition of ISO/IEC 26300, the publication of ODF 1.0 Approved Errata 01 and the creation of the various ODF subcommittees. I've also indicated the dates of each of the ODF 1.2 WD's and CD's.

As you can see, we've been quite busy. After iterating on WD's during 2007 and 2008, we've now moved on to CD's. This is not a drawn out process, but simply the ODF TC working with full transparency, making all of the intermediate drafts available for public inspection.

All of the planned feature work for ODF 1.2 is now completed. The remaining work is to address the various editorial and technical comments that have been submitted to our comment list, as well comments from TC members and JTC1/SC34. The goal is to have no known defects in ODF 1.2 before we send it out for a Public Review. Of course, previously-unknown defects will likely be identified during the Public Review, and we have a process for handling these. I'll comment more on that process, and Public Reviews in general, when we get closer to that stage.

ODF Lies and Whispers

2009-06-09T17:30:00.004-04:00

There is an interesting disinformation campaign being waged against ODF. You won't see this FUD splattered across the front pages of blogs or press releases. It is the kind of stuff that is spread by email and whispers, and you or I rarely will see it in the light of day. But occasionally some of it does cross my desk, and I'd like to share with you some recent examples.

First up is this instance, from a small Baltic republic, where a rather large US-based software company was recently arguing to the national standards committee for the adoption of OOXML instead of ODF. Here are some of the points made by this large company in a letter:

There is no software that currently implements ODF as approved by the ISO

(They then link to Alex Brown's comment from Wikipedia). I think this demonstrates the triangle-trade relationship among Microsoft, Alex Brown (and other bloggers) and Wikipedia, by which Microsoft FUD is laundered via intermediaries to Wikipedia for later reference as newly minted "facts". No wonder one of Microsoft's first actions during their OOXML push was to seize control of the Wikipedia articles on ODF and OOXML via paid consultants. In any case, Alex's claims were rebutted long ago.

ODF has a number (more than a hundred) of technical flaws which haven't been addressed for 3 years despite change requests addressed to OASIS by countries such as Japan and United Kingdom. There are discussions between OASIS and ISO/IEC JTC 1 SC 34 regarding true ownership of ISO ODF, which is a reason why the flaws in ISO ODF aren't being addressed. In a recent SC 34 meeting in Prague a new ISO ODF maintenance committee has been formed because ISO / IEC 26300: 2006 is not being presently maintained.

This is not true. First, the ODF TC has received zero defect reports from any ISO/IEC national body other than Japan. Second, we responded to the Japanese defect report last November. Amazingly, Alex Brown is implicated in this FUD one as well. It was false then and it is false now. At the exact time Alex was quoted in the press as saying the the ODF TC was not acting on defect reports (October 8th, 2008), we had in fact already sent our response to the defect report out to public review (August 7th, 2008) and then completed that reivew (August 22nd), after quite a bit of active technical discussion with the submitter of the original defect report (Murata Mokoto). How Alex translated that into "Their defect reports are being shelved" and "Oasis has not been acting on reports of defects" is beyond me. It must be particularly embarrassing that Murata-san wrote to the OASIS list, within days of Alex's FUD, "I am happy with the way that the errata has been prepared." How could Alex be ignorant of these facts? Why was he lying to the press? How is this conformant with his leadership role in JTC1/SC34 and his participation in BSI? Also observe the triangle-trade route of FUD in this case from Alex to Doug Mahugh to Wikipedia, this time for negative edits in the OASIS article.

IBM currently recommends not using OASIS ODF 1.1 and to instead use OASIS ODF 1.2 which is currently not complete and will not be complete and ISO certified before 2010/2011. OASIS on the other hand have started work on ODF 2.0 which will not be backwards compatible.

This is an odd one, demonstrably false. IBM Lotus Symphony supports ODF 1.1. We have no ODF 1.2 support at present. I wonder where they came up with this one? It is totally bizarre. Although we have started to gather requirements for "ODF-Next", the contents of that version, and to what degree it will be backwards compatible, has not even been discussed by the TC, let alone determined. So this is pure FUD, trying to make ODF sound risky to adopt, and then lying about IBM's support for it, and our position on ODF 1.2.

The list goes on, including claims that no one supports ODF 1.0 or ODF 1.1, etc., but you get the gist of it. The particulars are interesting, of course, but more so the reckless disregard for the truth, and the triangle-trade relationship between notable bloggers, Wikipedia, and Microsoft's whisper campaign.

Another current example is part of Microsoft's attempt to duck and cover from criticism over their interoperability-busting ODF support in Office 2007 SP2. I've heard variations on the following from three different people in three different countries, including from government officials. So it is getting around. It goes something like this:

We (Microsoft) wanted to be more interoperable with ODF. In fact we submitted 15 proposals to the ODF TC to improve interoperability, but IBM and Sun voted them down.

Nice story, but not true. Certainly Microsoft submitted 15 proposals. But they were never voted on by the TC, because Microsoft chose not to advance them for a vote. They opted not to have these proposals considered for ODF 1.2. It was their choice alone and their decision alone not to put these items up for a vote. I would have been fine with whatever decision Microsoft wanted to make in this situation. I'm not criticizing their decision. I'm just saying we need to be clear that the outcome was entirely due to their decision, and not to blame IBM or Sun for Microsoft's choice in this matter.

I think I can trace this FUD back to a May 13th blog post from Doug Mahugh where he wrote:

We then continued submitting proposed solutions to specific interoperability issues, and by the time proposals for ODF 1.2 were cut off in December, we had submitted 15 proposals for consideration. The TC voted on what to include in version 1.2, and none of the proposals we had submitted made it into ODF 1.2.

This certainly is an interesting statement. There is nothing I can point to that is false here. Everything here is 100% accurate. However, it seems to be reckless in how it neglects the most relevant facts, namely that the proposals did not make it into ODF 1.2 at Microsoft's sole election. It is as if Lee Harvey Oswald had written a note: "Went to Dallas and saw a parade today. Tried to see a movie, but had to leave early. Heard later on the radio that the President was shot". This would have been 100% accurate as well, but not the "whole truth". In any case, the rundown of the facts in this question are on the TC's mailing list.

So what is one to do? You obviously can't trust Wikipedia whatsoever in this area. This is unfortunate, since I am a big fan of Wikipedia. I want it to succeed. But since the day when Microsoft decided they needed to pay people to "improve" the ODF and OOXML articles, these articles have been a cesspool of FUD, spin and outright lies, seemingly manufactured for Microsoft's re-use in their whisper campaign. My advice would be to seek out official information on the standards, from the relevant organizations, like OASIS, the chairs of the relevant committees, etc. Ask the questions in public places and seek a public, on-the-record response. More people are willing to lie than face of consequences of being caught lying. That is the ultimate weakness of lies. They cannot stand the light of public exposure. Sunlight is the best antiseptic.

The Battle for ODF Interoperability

2009-05-17T20:30:00.001-04:00

Last year, when I was socializing the idea of creating the OASIS ODF Interoperability and Conformance TC, I gave a presentation I called "ODF Interoperability: The Price of Success". The observation was that standards that fail never need to deal with interoperability. The creation of test suites, convening of multi-vendor interoperability workshops and plugfests is a sign of a successful standard, one which is implemented by many vendors, one which is adopted by many users, one which has vendor-neutral venues for testing implementations and iteratively refining the standard itself.

Failed standards don't need to work on interoperability because failed standards are not implemented. Look around you. Where are the OOXML test suites? Where are the OOXML plugfests? Indeed, where are the OOXML implementations and adoptions? Microsoft Office has not implemented ISO/IEC 29500 "Office Open XML", and neither has anyone else. In one of the great ironies, Microsoft's escapades in ISO have left them clutching a handful of dust, while they scramble now to implement ODF correctly. This is reminiscent of their expensive and failed gamble on HD DVD on the XBox, followed eventually by a quick adoption of Blue-ray once it was clear which direction the market was going. That's the way standards wars typically end in markets with strong network effects. They tend to end very quickly, with a single standard winning. Of course, the user wins in that situation as well. This isn't Highlander. This is economic reality. This is how the world works.

Although this may appear messy to an outside observer, our current conversation on ODF interoperability is a good thing, and further proof, to use the words Microsoft's National Technology Director, Stuart McKee, that "ODF has clearly won".

Fixing interoperability defects is the price of success, and we're paying that price now. The rewards will be well worth the cost.

We've come very far in only a few years. First we had to fight for even the idea and acceptance of open standards, in a world dominated by a RAND view of exclusionary standards created in smoke filled rooms, where vendors bargained about how many patents they could load up a standard with. We won that battle. Then we had to fight for ODF, a particular open standard, against a monopolist clinging to its vendor lock-in and control over the world's documents. We won that battle. But our work doesn't end here. We need to continue the fight, to ensure that users of document editors, you and I, get the full interoperability benefits of ODF. Other standards, like HTML, CSS, EcmaScript, etc., all went through this phase. Now it is our turn.

With an open standard, like ODF, I own my document. I choose what application I use to author that document. But when I send that document to you, or post it on my web site, I do so knowing that you have the same right to choose as I had, and you may choose to use a different application and a different platform than I used. That is the power of ODF.

Of course, the standard itself, the ink on the pages, does not accomplish this by itself. A standard is not a holy relic. I cannot take the ODF standard and touch it to your forehead say "Be thou now interoperable!" and have it happen. If a vendor wants to achieve interoperability, they need to read (and interpret) the standard with an eye to interoperability. They need to engage in testing with other implementations. And they need to talk to their users about their interoperability expectations. This is not just engineering. Interoperability is a way of doing business. If you are trying to achieve interoperability by locking yourself in a room with a standard, then you'll have as much luck as trying to procreate while locked in a room with a book on human reproduction. Interoperability, like sex, is a social activity. If you're doing it alone then you're doing it wrong.

Standards are written documents -- text -- and as such they require interpretation. There are many schools of textual interpretation: legal, literary, historic, linguistic, etc. The most relevant one, from the perspective of a standard, is what is called "purposive" or "commercial" interpretation, commonly applied by judges to contracts. When interpreting a document using an purposive view, you look at the purpose, or intent, of a document in its full context, and interpret the text harmoniously with that intent. Since the purpose of a standard is to foster interoperability, any interpretation of the text of a standard which is used to argue in favor of, or in defense of, a non-interoperable implementation, has missed the mark. Not all interpretations are equal. Interpretations which are incongruous with the intent of standardization can easily be rejected.

Standards can not force a vendor to be interoperable. If a vendor wishes deliberately to withhold interoperability from the market, then they will always be able to do so, and, in most cases, devise an excuse using the text of the standard as a scapegoat.

Let's work through a quick example, to show how this can happen.

OpenFormula is the part of ODF 1.2 that defines spreadsheet formulas. The current draft defines the addition operator as:

6.3.1 Infix Operator "+"

Summary: Add two numbers.
Syntax: Number Left + Number Right
Returns: Number
Constraints: None
Semantics: Adds numbers together.

I think most vendors would manage to make an interoperable implementation of this. But if you wanted to be incompatible, there are certainly ways to do so. For example, given the expression "1+1" I could return "42" and still claim to be interoperable. Why? Because the text says "adds numbers together", but doesn't explicitly say which numbers to add together. If you decided to add 1 and 41 together, you could claim to be conformant. OK, so let's correct the text so it now reads:

6.3.1 Infix Operator "+"

Summary: Add two numbers.
Syntax: Number Left + Number Right
Returns: Number
Constraints: None
Semantics: Adds Left to Right.

So, this is bullet-proof now, right? Not really. If I want to, I can say that 1+1 =10, if I want to claim that my implementation works in base 2. We can fix that in the standard, giving us:

6.3.1 Infix Operator "+"

Summary: Add two numbers.
Syntax: Number Left + Number Right, both in base 10 representations
Returns: Number, in base 10
Constraints: None
Semantics: Adds Left to Right.

Better, perhaps. But if I want I can still break compatibility. For example, I could say 1+1=0, and claim that my implementation rounds off to the nearest multiple of 5. Or I could say that 1+1 = 1, claiming that the '+' sign was taken as representing the logical disjunction operator rather than arithmetic addition. Or I could do addition modulo 7, and say that the text did not explicitly forbid that. Or I could return the correct answer some times, but not other times, claiming that the standard did not say "always". Or I could just insert a sleep(5000) statement in my code, and pause 5 seconds every time the an addition operation is performed, making a useless, but conformant implementation And so on, and so on.

The old adage holds, "It is impossible to make anything fool- proof because fools are so ingenious." A standard cannot compel interoperability from those who want resist it. A standard is merely one tool, which when combined with others, like test suites and plugfests, facilitates groups of cooperating parties to achieve interoperability.

Now is the time to achieve interoperability among ODF implementations. We're beyond kind words and empty promises. When Microsoft first announced, last May, that it would add ODF support to Office 2007 SP2, they did so with many fine words:

"Microsoft Corp. is offering customers greater choice and more flexibility among document formats"
Microsoft is "committed to work with others toward robust, consistent and interoperable implementations"
Chris Capossela, senior vice president for the Microsoft Business Division: "We are committed to providing Office users with greater choice among document formats and enhanced interoperability between those formats and the applications that implement them"
"Microsoft recognizes that customers care most about real-world interoperability in the marketplace, so the company is committed to continuing to engage the IT community to achieve that goal when it comes to document format standards"
Microsoft will "work with the Interoperability Executive Customer Council and other customers to identify the areas where document format interoperability matters most, and then collaborate with other vendors to achieve interoperability between their implementations of the formats that customers are using today. This work will continue to be carried out in the Interop Vendor Alliance, the Document Interoperability Initiative, and a range of other interoperability labs and collaborative venues."
"This work on document formats is only one aspect of how Microsoft is delivering choice, interoperability and innovative solutions to the marketplace."

So the words are there, certainly. But what was delivered fell far, far short of what they promised. Excel 2007 SP2 strips out spreadsheet formulas when it reads ODF spreadsheets from every other vendor's spreadsheets, and even from spreadsheets created by Microsoft's own ODF Add-in for Excel. No other vendor does this. Spreadsheet formulas are the very essence of a spreadsheet. To fail to achieve this level of interoperability calls into question the value and relevance of what was touted as an impressive array of interoperability initiatives. What value is an Interoperability Executive Council, an Interop Vendor Alliance, a Document Interoperability Initiative, etc., if they were not able to motivate the most simple act: taking spreadsheet formula translation code that Microsoft already has (from the ODF Add-in for Office) and using it in SP2?

The pretty words have been shown to be hollow words. Microsoft has not enabled choice. Their implementation is not robust. They have, in effect, taken your ODF document, written by you by your choice in an interoperable format, with demonstrated interoperability among several implementations, and corrupted it, without your knowledge or consent.

There are no shortage of excuses from Redmond. If customers wanted excuses more than interoperability they would be quite pleased by Microsoft's prolix effusions on this topic. The volume of text used to excuse their interoperability failure, exceeds, by an order of magnitude, the amount of code that would be required to fix the problem. The latest excuse is the paternalistic concern expressed by Doug Mahugh, saying that they are corrupting spreadsheets in order to protect the user. Using a contrived example, of a customer who tries to add cells containing text to those containing numbers, Doug observes that OpenOffice and Excel give different answers to the formula = 1+ "2". Because all implementations do not give the same answer, Microsoft strips out formulas. Better to be the broken clock that reads the correct time twice a day, than to be unpredictable, or as Doug puts it:

If I move my spreadsheet from one application to another, and then discover I can’t recalculate it any longer, that is certainly disappointing. But the behavior is predictable: nothing recalculates, and no erroneous results are created.

But what if I move my spreadsheet and everything looks fine at first, and I can recalculate my totals, but only much later do I discover that the results are completely different than the results I got in the first application?

That will most definitely not be a predictable experience. And in actual fact, the unpredictable consequences of that sort of variation in spreadsheet behavior can be very consequential for some users. Our customers expect and require accurate, predictable results, and so do we. That’s why we put so much time, money and effort into working through these difficult issues.

This bears a close resemblance to what is sometimes called "Ben Tre Logic", after the Vietnamese town whose demise was excused by a US General with the argument, "It became necessary to destroy the village in order to save it."

Doug's argument may sound plausible at first glance. There is that scary "unpredictable consequences". We can't have any of that, can we? Civilization would fall, right? But what if I told you that the same error with the same spreadsheet formula occurs when you exchange spreadsheets in OOXML format between Excel and OpenOffice? Ditto for exchanging them in the binary XLS format. In reality, this difference in behavior has nothing to do with the format, ODF or OOXML or XLS. It is a property of the application. So, why is Microsoft not stripping out formulas when reading OOXML spreadsheet files? After all, they have exactly the same bug that Doug uses as the centerpiece of his argument for why formulas are stripped from ODF documents. Why is Microsoft not concerned with "unpredictable consequences" when using OOXML? Why do users seem not to require "accurate, predictable results" when using OOXML? Or to be blunt, why is Microsoft discriminating against their own paying customers who have chosen to use ODF rather than OOXML? How is this reconciled with Microsoft's claim that they are delivering "choice, interoperability and innovative solutions to the marketplace"?

A follow-up on Excel 2007 SP2's ODF support

2009-05-07T14:30:00.003-04:00

Wow. My previous post seems to have attracted some attention. When I woke up on Monday morning, made my coffee and logged into to my email, I found out that my geeky little analysis of Office 2007 SP2's ODF support had sparked some interest. I did not intend it to be more than an update for the handful of the "usual suspects" who regularly follow ODF issues via various blogs, many of which you see listed to your right. If I had any foreknowledge or expectation that this post would end up being on SlashDot, GrokLaw, ZDnet, IDG, Reuters, CNet, etc., I would have done a better job spell checking, and maybe toned down the rhetoric a little (just a little).

But this widespread interest in the topic tells me one thing: ODF is important. People care about it. People want it to succeed, and when this success is threatened, whether for deliberate or accidental reasons, they are upset. Although Office 2007 SP2 also added PDF and XPS support, you don't see many stories on that at all.

I've been trying to respond to the many comments by anonymous FUDsters and Fanboys on various web sites where my post is being discussed. However, it is getting rather laborious swatting all the gnats. They obviously breed in stagnant waters, and there is an awful lot of that on the web. Since all links lead back here anyways, it will be much simpler to do a recap here and address some of the more widespread errors.

The talking points from Redmond seem to be consistent, along the lines of:

We did a 100% perfect and conforming implementation of ODF 1.1 to the letter of the standard. If it is not interoperable, then it is the fault of the standard or the other applications or some guy we saw sneaking around back on the night of the fire. In any case, it is not our fault. We just design, write, test and sell software to users, businesses, governments and educational institutions. We have no influence over whether our products are interoperable or not. What effect SP2 has on users or the market -- that's not our concern. Come back in 50 years when you have a 100% perfect standard and maybe we'll talk.

In other words, all of those Interoperability Directors and Interoperability Architects at Microsoft seem to have (hopefully temporarily) switched into Minimal Conformance Directors and Minimal Conformance Architects, and are gazing at their navels. I hope they did not suffer a reduction in salary commensurate with the reduction in their claimed responsibilities.

In any case, their argument might be challenged on several grounds. First up is the question of whether the ODF documents written by Excel 2007 SP2 indeed conform to the ODF 1.1 standard. This is not a hard question to answer, but please excuse this short technical diversion.

Let's see what the ODF 1.1 standard says in section 8.1.3 (Table Cell):

Addresses of cells that contain numbers. The addresses can be relative or absolute, see section 8.3.1. Addresses in formulas start with a “[“ and end with a “]”. See sections 8.3.1 and 8.3.1 for information about how to address a cell or cell range.

And the referenced section 8.3.1 further says:

To reference table cells so called cell addresses are used. The structure of a cell address is as follows:

The name of the table.

A dot (.)

An alphabetic value representing the column. The letter A represents column 1, B represents column 2, and so on. AA represents column 27, AB represents column 28, and so on.

A numeric value representing the row. The number 1 represents the first row, the number 2 represents the second row, and so on.

This means that A1 represents the cell in column 1 and row 1. B1 represents the cell in column 2 and row 1. A2 represents the cell in column 1 and row 2.

For example, in a table with the name SampleTable the cell in column 34 and row 16 is referenced by the cell address SampleTable.AH16. In some cases it is not necessary to provide the name of the table. However, the dot must be present. When the table name is not required, the address in the previous example is .AH16

So, going back to my test spreadsheets from all of the various ODF applications, how do these applications encode formulas with cell addresses:

Symphony 1.3: =[.E12]+[.C13]-[.D13]
Microsoft/CleverAge 3.0: =[.E12]+[.C13]-[.D13]
KSpread 1.6.3: =[.E12]+[.C13]-[.D13]
Google Spreadsheets: =[.E12]+[.C13]-[.D13]
OpenOffice 3.01: =[.E12]+[.C13]-[.D13]
Sun Plugin 3.0: [.E12]+[.C13]-[.D13]
Excel 2007 SP2: =E12+C13-D13

I'll leave it as an exercise to the reader to determine which one of these seven is wrong and does not conform to the ODF 1.1 standard.

Next is the question of the relationship between interoperability and conformance. So we are not building skyscrapers in the air, let's start with a working definition of interoperability, say that given by ISO/IEC 2382-01, "Information Technology Vocabulary, Fundamental Terms":

The capability to communicate, execute programs, or transfer data among various functional units in a manner that requires the user to have little or no knowledge of the unique characteristics of those units

I think we probably have a better sense of what conformance is. Something conforms when it meets the requirements defined by a standard.

So let's explore explore the relationship between conformance to a standard and interoperability.

First, does interoperability require a standard? No. There have been interoperable systems without formal standards. For example, there is a degree of interoperability among spreadsheet vendors on the basis of the legacy Excel binary file format (XLS), even though the binary format was never standardized and never defines spreadsheet formulas. Another example is the SAX XML parsing API. Widely implemented, but never standardized. We may call them informal or de facto standards.

Additionally, many standards start out as informal technical agreements and specifications that achieve interoperability among a small group of users, who then move it forward to standardization so that a broader audience can benefit. But the interoperability came first and the formal standard came second. See the history of the Atom syndication format for a good example.

Second, Is interoperability possible in the presence of non-conformance? Yes. For example, it is well known that the vast majority of web pages (93% by one estimate) on the web today do not conform to the HTML standard. But there is a not unsubstantial degree of interoperability on the web today in spite of this lack of conformance. Generally, interoperability does not require perfection. It requires good faith and hard work. If perfection were required, nothing would work in this world, would it?

Third, if a standard does not define something (like spreadsheet formulas) then I am allowed to do whatever I want, right? This is true. But further, even if ODF 1.1 did define spreadsheet formulas you would still be allowed to do whatever you want. Remember, these are voluntary standards. We can't force you to do anything, whether we define it or not.

So what then is the precise relationship between conformance and interoperability? I'd state it as:

In general, conformance is neither necessary nor sufficient for to achieve interoperability.
But interoperability is most efficiently achieved by conformance to an open standard where the standard clearly states those requirements which must be met to achieve interoperability.

In other words, the relationship is due to the efficiency of this configuration to those who wish to interoperate. Conformance is neither necessary nor sufficient to achieve interoperability in general, but interoperability is most efficiently achieved when conformance guarantees interoperability. When I talk about "standards-based interoperability" I'm talking about the situation when you are in the neighborhood of that optimal point.

The inefficiency of other orientations is seen with HTML and Web browsers. Because of the historically low level of HTML conformance by authoring tools and users who hand-edit HTML, browsers today are much more complex then they would otherwise need to be. They need to handle all sorts of mal-formed HTML documents. This complexity extends to any tool that needs to process HTML. Sure, we have a pretty good grip on this now, with tools like HTML Tidy and other robust parsers, but this has come at a cost. Complexity eats up resources, both to coders and testers, but also runtime resources, memory and processing cycles. More complex code is harder to maintain and secure and tends to have more bugs. Greater conformance would have lead to a more efficient relationship between conformance and interoperability.

Similarly, the many years of non-conformance in browsers, most notably Internet Explorer, to the CSS2 standard has resulted in an inefficiency there. From the perspective of web designers, tool authors and competing browser vendors, the lack of conformance to the standards has increased the cost needed to achieve interoperability, a cost transferred from a dominate vendor who chose not to conform to the standards, to other vendors who did conform.

The efficiency of conformance to open standards in particular is the clarity and freedom it provides around access to the standard and the contingent IP rights needed to implement the standard.

So back to ODF 1.1. What is the relationship between conformance and interoperability there? Clearly, it is not yet at that optimal point (which few standards ever achieve) where interoperability is most-efficiently achieved. We're working on it. ODF 1.2 will be better in that regard than ODF 1.1, and the next version will improve on that, and so on.

Does this mean that you cannot create interoperable solutions with ODF? No, it just means that, like most standards in IT today, you need to do some interoperability testing with other vendor's products to make sure your product interoperates, and make conformant adjustments to your product in order to achieve real-world nteroperability. Most vendors who don't have a monopoly would do this naturally and in fact have done this, as my chart indicated. Complaining about this is like complaining about gravity or friction or entropy. Sure, it sucks. Deal with it. Although it may not pay as much as being a professional mourner, work as a programmer is more regular. And giving value to customers will always bring more satisfaction than than standing there weeping about how code is hard.

In any case, this comes down to why do you implement a standard. What are your goals? If your goal is be interoperable, then you perform interoperability testing and make those adjustments to your product necessary to make it be both conformant and interoperable. But if your goal is to simply fulfill a checkbox requirement without actually providing any tangible customer benefit, then you will do as little as needed. However, if your goal is to destroy a standard, then you will create a non-conformant, non-interoperable implementation, automatically download it to millions of users and sow confusion in the marketplace by flooding it with millions of incompatible documents. It all depends on your goals. Voluntary standards do not force, or prevent, one approach or another.

To wrap this up, I stand on the table of interoperability results in the previous post. SP2 has reduced the level of interoperability among ODF spreadsheets, by failing to produce conforming ODF documents, and failing to take note of the spreadsheet formula conventions that had been adopted by all of the other vendors and which are working their way through OASIS as a standard.

If we note the arguments used by Microsoft in the recent past, they have argued that OOXML must be exactly what it is -- flaws and all -- in order to be compatible with legacy binary Office documents. Then they argued that OOXML can not be changed in ISO, because that would create incompatibility with the "new legacy" documents in Office 2007 XML format. But when it comes to ODF, they have disregarded all legacy ODF documents created by all other ODF vendors and take an aloof stance that looks with disdain on interoperability with other vendor's documents, or even documents produced by their own ODF Add-in. The sacrosanctness of legacy compatibility appears to be reserved, for strategic reasons, for some formats but not others. We'll redefine the Gregorian calender in ISO to be interoperable with one format if we need to, but we won't deign, won't stoop, won't dirty ourselves to use the code we already have from the ODF Add-in for Microsoft Office, to make SP2 formulas interoperable with the other vendors' products, to benefit our own users who are asking for ODF support in Office. As I said before, this ain't right.

OpenDocument Format: The Standard for Office Documents

2009-05-05T21:28:00.003-04:00

A belated note that an article of mine on ODF was recently published in IEEE Internet Computing, called "OpenDocument Format: The Standard for Office Documents". I think it is a good introduction to ODF, what it is, where it came from and why it is important. They allow authors to post a copy on their websites. So feel free to link to it, but any redistribution will need to be negotiated with the publisher.

At the same time I've taken the opportunity to put together a new web page of some of my other publications, workshop and conference presentations. I have few others that I want add, once I find them. But this is a start.

Update on ODF Spreadsheet Interoperability

2009-05-03T17:45:00.007-04:00

[2009/05/07 -- I've posted a follow up article on this topic which you may want to read]

A couple of months ago I did some experiments on the interoperability of ODF spreadsheets, the theory and practice. In that earlier post I looked at the then current ODF implementations, including:

OpenOffice.org 2.4
Google Spreadsheets
KOffice KSpread 1.6.3
IBM Lotus Symphony 1.1
Microsoft Office 2003 with the Microsoft-sponsored CleverAge Add-in version 2.5
Microsoft Office 2003 with Sun's ODF Plugin

I created a test document in each of those editors and then loaded each test document in each of the other editors. I showed what worked, what didn't, and made some suggestions on how interoperability could be improved. I found only two notable failures, when the Microsoft/CleverAge Add-in for Excel loaded KSpread and Symphony documents. The other scenarios I tested were OK:

		CleverAge	Google	KSpread	Symphony	OpenOffice	Sun Plugin
		Created In
Read In	CleverAge	OK	OK	Fail	Fail	OK	OK
	Google	OK	OK	OK	OK	OK	OK
	KSpread	OK	OK	OK	OK	OK	OK
	Symphony	OK	OK	OK	OK	OK	OK
	OpenOffice	OK	OK	OK	OK	OK	OK
	Sun Plugin	OK	OK	OK	OK	OK	OK

I lot has happened in the two months since I did that analysis. Several of the applications I tested have been updated:

CleverAge has released version 3.0 of their Add-in.
OpenOffice 3.01 is now out and in wide use.
Symphony 1.3 is now in beta.
The Sun ODF Plugin is now at version 3.0.
Microsoft Office 2007 SP2 has been released, with integrated ODF support.
KOffice 2.0 RC 1 is now available.

I haven't been able to get the release candidate of KOffice installed, so I'm still including KSpread 1.6.3 in my tests, but for the rest I have created new test files in each editing environment, saved them to ODF format and then loaded the resulting documents into each of the other editors. From these test documents I was able to perform 42 different test combinations.

I'll explain a bit more how I tested, then give you the table of results, and finally make some observations and recommendations.

The test scenario I used was a simple wedding planner for a fictional user, Maya, who is getting married on August 15th. She wants to track how many days are left until her wedding, as well as track a simple ledger of wedding-related expenses. Nothing complicated here. I created this spreadsheet from scratch in each of the editors, by performing the following steps:

Enter the title in A1 "May's Wedding Planner" and increased font size to 14 point.
Enter formula = TODAY() in B3 and set US style MM/DD/YY date format/
Enter the date of the wedding as a constant in cell B4, also setting date format.
Added simple calculations on cells B6-B8, to calculate days, weeks and months until the wedding.
A11 through E16 is a simple ledger of the kind that is done thousands of times a day by spreadsheet users everywhere. Once you have the formula set up in column E (Balance = previous balance + credits - debits) then you can simply copy down the formula to the new row for each new entry.

The resulting spreadsheet looks something like this:

Feel free to download a zip of all of the test spreadsheet files. The file names should be self-explanatory.

Here is what I found when I tested the various scenarios:

		Google	KSpread	Symphony	OpenOffice	Sun Plugin	CleverAge	MS Office 2007 SP2
		Created In
Read In	Google	OK	OK	OK	OK	Fail	OK	Fail
	KSpread	OK	OK	OK	Fail	Fail	OK	Fail
	Symphony	OK	OK	OK	OK	OK	Fail	Fail
	OpenOffice	OK	OK	OK	OK	OK	OK	Fail
	Sun Plugin	OK	OK	OK	OK	OK	OK	Fail
	CleverAge Plugin	OK	OK	OK	OK	Fail	OK	OK
	MS Office 2007 SP2	Fail	Fail	Fail	Fail	Fail	Fail	OK

So what is happening here?

CleverAge appears to have heeded the advice from my earlier blog post and now correctly processes KSpread and Symphony spreadsheets. This is great news and they deserve credit for that work. But this is a small bit of good news in a table that now shows awful lot of red. Let's see if we can figure this out.

First, some combinations that worked previously, when I tested two months ago, are now not working:

Symphony 1.3 beta hangs when attempting to load the spreadsheet created with the CleverAge 3.0 ODF Add-in. Symphony 1.1 also hangs when trying to load that same spreadsheet. However both versions of Symphony work fine when loading the CleverAge 2.5 spreadsheet from two months ago. The CleverAge document appears to be valid, so my guess is that this is a bug in the Symphony 1.3 beta. I'll pass this document on to the Symphony development team to see what they say.
KSpread 1.6.3 does not read formulas from OpenOffice 3.01 documents. KSpread had no problems with OO 2.4 documents. The problem appears to be that OpenOffice 3.01, by default, writes out documents according to the ODF 1.2 draft which puts formulas in the OpenFormula namespace. But KSpread is expecting them in the legacy namespace. The result is that spreadsheet formulas are dropped when the document is loaded in KSpread.
In a similar way, Sun's new ODF Plugin writes out documents according to the ODF 1.2 draft. KOffice is unable to handle these files. This also causes problems for Google Spreadsheets and the Microsoft/CleverAge Plugin for Excel, which report errors "We were unable to upload this document" and "The converter failed to open this file".

The new entry to the mix is Microsoft Office 2007 SP2, which has added integrated ODF support. Unfortunately this support did not fare well in my tests. The problem appears to be how it treats spreadsheet formulas in ODF documents. When reading an ODF document, Excel SP2 silently strips out formulas. What is left is the last value that cell had, when previously saved.

This can cause subtle and not so subtle errors and data loss. For example, in the test document I presented above, the current date is encoded using the TODAY() spreadsheet function. If the formulas are stripped, then this cell no longer updates, and will return the wrong value. Similarly, if Maya tries to continue her ledger of expenses by copying the formula cells from column E down a row, this will cause incorrect calculations, since there is no longer a formula to copy, so she would just be copying the prior balance. In general, SP2 converts an ODF spreadsheet into a mere "table of numbers" and any calculation logic is lost.

In the other direction, when writing out spreadsheets in ODF format, Excel 2007 SP2 does include spreadsheet formulas but places them into an Excel namespace. This namespace is not what OpenOffice and other ODF applications use. It is not the ODF 1.2 namespace. It isn't even the OOXML namespace. I have no idea what it is or what it means. Not every ODF application checks the namespace of formulas when loading documents, but the ones that do reject the SP2 documents altogether. And the ones that do not check the namespace try and fail to load a formula since it is syntactically different than what they expected. The applications essentially display a corrupted document that is shows neither the formula nor the value correctly. For example, a SP2 document, loaded in MS Office using the Sun ODF Plugin looks like this:

Similar corruption occurs when loading the Excel 2007 SP2 spreadsheet into KSpread, Symphony and OpenOffice. Google doesn't import the document at all.

I must admit that I'm disappointed by these results. This is not a step forward compared to where we were two months ago. This is a big step backwards. Spreadsheet interoperability is not hard. This is not rocket science. Everyone knows what TODAY() means. Everyone knows what =A1+A2 means. To get this wrong requires more effort than getting it right. It is especially frustrating when we know that the underlying applications support the same fundamental formula language, or something very close to it, and are tripped up by lack of namespace coordination. Whether it is accidental or intentional I don't know or care. But I cannot fail to notice that the same application -- Microsoft Excel 2007 -- will process ODF spreadsheet documents without problems when loaded via the Sun or CleverAge plugins, but will miserably fail when using the "improved" integrated code in Office 2007 SP2. This ain't right.

I have some suggestions for how to move things forward again. There will be a lot less red on the above table if two simple changes are made:

Sun should write out formulas in ODF 1.1 format, using the legacy "oooc" namespace prefix that the other vendors are using. Remember, the other vendors are using that namespace specifically for compatibility with OO's ODF documents. This is the current convention. To unilaterally switch, without notice or coordination, to a new namespace, is not cool. When ODF 1.2 is an approved standard, then we all can move there in a coordinated fashion, to cause users minimal inconvenience. But the above table clearly shows the confusion that results if this move is not coordinated. I know OO 3.01 has an option to save in ODF 1.0/1.1 format. IMHO, this setting should be the default. I'm not sure if the Sun Plugin has a similar configuration option, but I hope it does.
In addition to writing out compatible formulas as per the above comments on the Sub Plugin, Microsoft should remove the code in SP2 that causes it to reject every other vendor's spreadsheet documents. Give the user a warning if you need to, but let them have the choice.

Finally, let me try to anticipate and debunk some of the counter-arguments which might be raised to argue against interoperability.

First, we might hear that ODF 1.1 does not define spreadsheet formulas and therefore it is not necessary for one vendor to use the same formula language that other vendors use. This is certainly is true if your sole goal is to claim conformance. If your business model requires only conformance and not actually achieving interoperability, then I wish you well. But remember that conformance and interoperability are not mutually exclusive options. An application can be conformant to a standard and also be interoperable, if you use the legacy formula namespace and syntax. So the desire to be conformant is not an excuse for not also being interoperable, or at least not a valid excuse. One might also wryly note that Microsoft has several Directors of Interoperability, not Directors of Minimal Conformance, and they workshops are called Document Interoperability Initiatives, not Minimal Conformance Initiatives. The difference between minimal conformance and interoperability is well illustrated in these tests.

Remember, it is not particularly difficult or clever to to take an adverse reading of a standard to make an incompatible, non-interoperable product. Take HTML, for example. It does not define the attributes of unstyled (default) text. So I could create a perfectly conformant browser implementation that makes all default text be 4-point Zapf Dingbats, white text on a white background. It would conform with the standard, but it would be perfectly unusable by anyone. If you try hard enough you can create 100% conformant, but non-interoperable, implementations of almost most standards. Standards are voluntary, written to help coordinate multiple parties in their desires for interoperability. Standards are not written to compel interoperability by parties who do not wish to be interoperable.

(A side point is that SP2's implementation of ODF spreadsheets does not, in fact, conform to the requirements of the ODF standard, but that is another story, for another blog post.)

We might also hear concerns that supporting other vendors' ODF spreadsheet formulas cannot be done because this formula language is undocumented. The irony here is that the formula language used by OpenOffice (and by other vendors) is based on that used by Excel, which itself was not fully documented when OpenOffice implemented it. So an argument, by Microsoft, not to support that language because it is not documented is rather hypocritical. Excel supports 1-2-3 files and formulas and legacy Excel versions (back to Excel 4.0) neither of which have standardized formula languages. Why are these supported? Also, the fact that the Microsoft/CleverAge add-in correctly reads and writes the legacy ODF formula syntax shows not only that it can be done, but that Microsoft already has the code to do it. The inexplicable thing is why that code never made it into Excel 2007 SP2.

We'll probably also hear that 100% compatibility with legacy documents is critical to Microsoft users and that it is dangerous to try to save Excel formulas into interoperable ODF formulas because there is no guarantees that OpenOffice or any other ODF application will interpret them the same as Excel does. So one might try to claim that Microsoft is protecting their customers by preventing them from saving interoperable spreadsheet formulas. But we should note that fully-licensed Microsoft Office users have already been creating legacy documents in ODF format, using the Microsoft/CleverAge ODF Add-in. These paying Microsoft Office customers will now see their existing investment in ODF documents, created using Microsoft-sanctioned code, get corrupted when loaded in Excel 2007 SP2. Why are paying Microsoft customers who used ODF less important than Microsoft customers who used OOXML? That is the shocking thing here, the way in which users of the ODF Add-in are being sacrificed.

If you are cynical, you might observe that if Excel 2007 SP2 allowed Microsoft/CleverAge ODF Add-in formulas to work correctly, then SP2 would need to allow all vendors' formulas to work, since the other vendors are using the same legacy namespace. The only way for Microsoft to make their legacy ODF documents work and to exclude other vendors would be (hypothetically) to specifically look in the document for the name of the application that created the document, and allow their ODF Add-in but reject OpenOffice, etc. IANAL, but I think something like that would look very, very bad to competition authorities. So the only way out, if your goal (hypothetically) is to avoid interoperability, is to sacrifice your existing Office customers who are using the Microsoft/CleverAge ODF Add-in. It serves them right for not sticking to the party line in the first place. This'll teach 'em good.

Of course, I am not that cynical. I was taught to never assume malice where incompetence would be the simpler explanation. But the degree of incompetence needed to explain SP2's poor ODF support boggles the mind and leads me to further uncharitable thoughts. So I must stop here.

As I mentioned before, this is a step backwards. But it is just one step on the journey. Let's look forward (and move forward). This is just code. Code can be fixed. We know exactly what is needed to have good interoperability of spreadsheet formulas. In fact most of the code already exists for this. The only thing we need now is to actually go do it and not get too far ahead, or lag too far behind from the other implementations. This is more a question of timing and coordination than hard technical problems.

[2009/05/07 -- For more on this topic, see my "A follow-up on Excel 2007 SP2's ODF Support"]

Shooting Daffodils

2009-04-21T10:45:00.001-04:00

I like daffodils. I've been planting a couple hundred additional bulbs each fall, so that now I have a lovely spring-time display, right around this time.

In past years I would walk through the garden and take a photo here and there, mainly while standing, shooting straight down, not paying particular attention to the lighting or the composition. Flower "mug shots" I'd call them. Then last year, I started doing macro (close-up) photography. Although the results were technically adequate -- sharp, detailed closeups -- they were...well... rather dull, symmetrical and artless.

This year I've decided to try something different. I realized that a flower can be posed like a person. I guess that is obvious in retrospect, but it never occurred to me before that the poses of classical portraiture, like the 2/3 view, over-the-shoulder, profile view, etc., apply to flowers as well as people. And you don't need to show all of the flower. A close up of part of it can also be interesting.

I've also worked to improve my technique, shooting with a tripod and remote trigger, using the McClamp to steady and isolate the blossoms in the field, using small erapertures to get greater depth of field, locking the mirror up before shooting to reduce any residual camera shake, shooting on days and at times where harsh shadows can be avoided, etc.

Here are three examples, intimate portraits, all shot on location in my garden. You can view more on my Flickr page.

A time for decision

2009-04-13T22:54:00.003-04:00

April 15th is Tax Day in the United States, the day by which we must file our income tax returns for 2008 and pay any balance due.

The day before, April 14th, is also a day of reckoning, with another outstretched hand asking for money. This is the day which marks the end of "mainstream support" for Microsoft Windows XP and Office 2003. After this date, licensed owners of these products will no longer receive free support and updates.

Depending on how consumers respond, one of three things will result from this end of life.

Users migrate to Vista/Office 2007
Users stay on unsupported Microsoft products for the near term and wait for Windows 7/Office 14 to come out in 2010.
Users take the opportunity to evaluate the available alternatives, including open source.

Since Windows XP is the most widely-deployed version of Windows, and Windows is the most-widely deployed operating system in the world, a lot of licenses will be up for grabs as IT shops decide what to do next. Especially in these tough economic times, upgrading to Vista just to see Vista become obsolete in less than a year doesn't make sense. But neither does remaining on an unsupported version of Windows.

This is a significant opportunity for alternatives, such as Linux and other open source applications, to increase their representation on the desktop. We should spend the next nine months making it especially easy for Microsoft's seemingly unwanted and expendable Windows XP and Office 2003 customers to migrate to better alternatives. The Windows/Office release calendar and economic conditions have combined to make this a huge upgrade cycle. In 2010 almost everyone will be looking to upgrade. An opportunity like this does not come every year. Let's make the most of it!

Project BudBurst

2009-03-29T17:55:00.002-04:00

The crocuses have bloomed here in Westford, one of first four flowers of my spring garden, the others being Galanthus nivalis (Snowdrops), Iris reticulata (dwarf Iris) and Eranthis hyemalis (Winter Aconite). But the crocuses are the most noticeable, since I have naturalized them in small drifts over the lawn.

For a few years I've been keeping a garden journal and have recorded the dates of first bloom for various flowers. So I see that for the snowdrops, the first bloom was March 14th this year, March 26th in 2008 and March 24th in 2007. Is this global warming? From just three observations, there is no way of telling.

But what if we had thousands of people record such information all over the country and pool their observations? Then we might be able to observe some interesting patterns. That is the idea of Project BudBurst, a distributed public field study run by a group of researches from UCAR, the Chicago Botanic Garden and the College of Forestry and Conservation of University of Montana. You sign up for a free account, state your location (US only, sorry) and pick from a list of local plant species that you can observe.

The emphasis is on widespread, native species, so the exotic bulbs I have in my garden won't be of use. But I can report observations on things like dandelions or white pines. Depending on the type of plant, you report the date it reaches each of various "phenophases" such as first flower, fully flowering, pollen release, first ripe fruit, etc. Different types of plants will have different phenophases. Volunteers enter their observations which are then plotted, along with all the other data on Google Maps.

Aside from climate change research, I wonder if this might also be useful for predicting the onset of spring allergies?

Taking Control of Your Documents

2009-03-24T19:08:00.008-04:00

How to free yourself from Microsoft Office dependency in three easy steps

The Objective

When you save a document in your word processor, your work is encoded in a particular file format. You often have a choice of formats that you can use, with names like DOC, DOCX, RTF, WPD or ODT. Your choice of format will influence whether others can easily read your document today, whether you yourself will be able to read your document ten years from now, and whether you will be able to migrate painlessly to another word processor or operating system if and when you choose to do so.

Although many users simply click “Save” and give no thought to which format is being used under the covers, this unthinking use of the word processor's default settings is a recipe for vendor lock-in. In fact, several vendors intentionally set their default format to be ones which will only work well with their own software, fostering dependency on that vendor's software and lessening the user's ability to take advantage of other options in the market. The more documents you save and accumulate in a vendor's proprietary format, the harder it will be for you to consider any other choices.

The objective of this paper is to show you, the user, how to extricate yourself from this cycle of dependency and take control of your documents. Specifically, we show how you can, in three easy steps, free yourself from a Microsoft Office dependency. In the end you may, of course, choose to remain on Microsoft Office. You may decide to migrate to an alternative word processor. That, in the end, is your choice. But by following the three steps outlined below, your freedom of action will be preserved, and your choice of word processor will be based on your priorities and your needs, and not forced on you by your current application vendor.

Step 1: Take control of the default format

The older versions of Microsoft Office, Office 97-Office 2003), by default save documents in a family of binary formats with the extensions DOC (Word), XLS (Excel) and PPT (PowerPoint). Although these formats are proprietary Microsoft formats, over the past decade 3rd party applications have developed the capability to read and write these formats.

However, starting in Office 2007 Microsoft suddenly switched the default format to something called Office Open XML (OOXML). This format is not widely supported outside of Office 2007. So if you save a document in the OOXML format you make it harder for anyone else to read your document unless they are also using Microsoft Office 2007. In almost all cases, the same document, if saved in the legacy DOC format will be more interoperable. Staying with the default choice, OOXML, only restricts your choices and make you more dependent on Microsoft Office. Of course, that is why Microsoft made OOXML the default format.

The first step to liberate yourself from Microsoft Office dependency is to change the default format in Microsoft Office 2007 away from OOXML and back to the early binary formats supported by Office 97-2003, which are widely supported by 3^rd party applications. This is a neutral step that preserves the status quo. By making these changes you will still be able to read and edit any OOXML documents that are sent to you, but all new documents you create will be saved in the more widely supported DOC/XLS/PPT formats.

If you are using Microsoft Office 2003 or earlier, then you should skip this Step and move on to Step 2, since OOXML is not the default format in those earlier Office versions.

To change the defaults, you will need to load Word 2007, Excel 2007 and PowerPoint 2007 and follow the following steps.

Word 2007

Click the Office Button (the unlabeled logo button in the upper left of the program).
Click “Word Options” at the bottom of the dialog.
Go to the “Save” section.
For the “Save files in this format” setting, choose “Word 97-2003 Document(*.doc)”.
Click OK.

Excel 2007

Click the Office Button (the unlabeled logo button in the upper left of the program).
Click “Excel Options” at the bottom of the dialog.
Go to the “Save” section.
For the “Save files in this format” setting choose “Excel 97-2003 Workbook (*.xls)”.
Click OK.

PowerPoint 2007

Click the Office Button (the unlabeled logo button in the upper left of the program).
Click “PowerPoint Options” at the bottom of the dialog.
Go to the “Save” section
For the “Save files in this format” setting, choose “PowerPoint Presentation 97-2003”.
Click OK.

Administrators should also note that these settings may be made directly in the Windows Registry, and automatically pushed out to a work group via a login script or group policy. The registry settings corresponding to the above changes are:

HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\Word\Options
Add String DefaultFormat=Doc

HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\Excel\Options
Add DWORD DefaultFormat=38 (Hexadecimal)


HKEY_CURRENT_USER\Software\Microsoft\Office\12.0\PowerPoint\Options
Add DWORD DefaultFormat=0 (Hexadecimal)

Step 2: Enable OpenDocument Format Support

Now that you've made the first steps towards taking control of your documents by preventing the lock-in effects of the OOXML default, it is time to take further control. You'll now want to enable OpenDocument Format (ODF=ISO/IEC 26300) support in Microsoft Office, so you can save and exchange documents using the free and open International Standard while remaining in the familiar Microsoft Office interface.

ODF is an XML-based, open document format standard, designed to be platform- and application-neutral and support interoperable use across applications, eliminating vendor lock-in. ODF is supported by many applications, including office suites from Sun, IBM, Novell and Google, as well as open source projects like OpenOffice, KOffice and AbiWord. Additional applications supporting ODF are listed on Wikipedia.

Microsoft Office does not currently support ODF “out of the box”, but you can enable ODF support in Office by installing a “plugin”, sometimes called an “add-in”. A plugin will add additional options or menu items to the Microsoft Office UI, allowing you to open and save documents in ODF format. In some cases you can even set ODF as the default format for new documents.

There are three main choices for adding ODF support to Microsoft Office:

Sun Microsystems has published an “ODF Plugin for Microsoft Office” which supports Office 2000, XP, 2003 and 2007 SP1.
Microsoft has sponsored an open source project on SourceForge for an “ODF Add-in for Microsoft Office”, which supports Office 2007, and also Office 2003 and Office XP if the Microsoft Office Compatibility Pack is also installed
Microsoft has announced that Office 2007 Service Pack 2 (SP2) will enable ODF support in Office 2007, but this code is not yet available.

Step 2 is to evaluate and adopt a plugin to add ODF support to Microsoft Office. Start using ODF now, saving your documents in the open standard document format. This allows you to remain in Office, for now, while building your familiarity and comfort level with ODF.

Step 3: Exercise your Right to Choose a Native ODF Editor

The plug-in approach is a transitional approach. It allows you to continue working in Microsoft Office while you enable ODF support side-by-side. But at some point you will want to consider your options. Maybe you find that converting back and forth to ODF format in MS Office is slow. Maybe you are using Office 2003 currently, but want to avoid paying for an Office 2007 upgrade when mainstream support for Office 2003 comes to an end on April 14th, 2009. At some point you will want to move to an application that supports ODF natively. You are free at this point and have a wide variety of choices.

You can stay on Windows or consider moving to Linux or the Mac.
You can stay with a traditional client editor, or move to a web based editor.
You can use commercial software, or use open source software.

The important thing is that you have taken control of your documents. You are no longer dependent on Microsoft Office and its file format. You have broken free of the vendor lock-in. You are free to choose an alternative word processor when you want to and if you want to. Until then, be comfortable in knowing that you are keeping your options open while remaining in control of your documents.

Taking Control of Your Documents by Rob Weir is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License.

This paper is also available in ODF and PDF formats.

Introducing Planet ODF

2009-03-23T20:50:00.001-04:00

I have an early Document Freedom Day present for you. Planet ODF is a feed aggregator based on Sam Ruby's Planet Venus, which itself is a refactoring of Planet 2.0.

Planet ODF aggregates several blogs, news sources, discussion forums and other online services related to ODF. I've tried to be semi-intelligent so you don't get random stories about the Oregon Department of Foresty or non-ODF blog posts by me. I'll tune the feeds over time, but the hope is to make it 100% ODF relevant content.

If you have a blog, discussion forum or any other ODF-related content with an Atom or RSS feed and want it included, then please let me know. It doesn't need to be 100% ODF. You can discuss your cats 90% of the time and ODF 10% of the time and I can set up a filter to bring in the relevant content.

Also, I've set up an OpenDocument Format group on the social bookmarking site Diigo. (I abandoned del.icio.us when the Microsoft/Yahoo takeover rumors started.) Even if you don't have a blog or a web site with a feed, you can use Diigo to bookmark any articles you think are relavent to ODF. If you send those links to the OpenDocument Format group, then they will automatically be included in the Planet ODF feed.

Enjoy, and pass on the good news.

What should I blog about? (Trying out Skribit)

2009-03-18T10:40:00.000-04:00

I've been tinkering with my blog, prepping it for an upcoming host move. The feeds are being redirected and I've cleaned up a few other things. There may be temporary periods of non-availability (unavailability?) but hopefully that will be rare.

I've also come across an interesting service called "Skribit" that allows readers of An Antic Disposition to suggest and vote on topics for me to write on. At any one time I have a dozen or more blog posts in draft form, awaiting my attention to complete. With Skribit I can give you a preview of what I'm working on and solicit your votes on what you want to read most. You are also free to suggest additional topics as well.

So if you look over on the upper right of the page you'll see a tab called "Suggestions?". Click on that and you can see the topics I'm currently working on. You can suggest and vote on topics there. And if you are reading this blog via a feed....uh...then you probably don't see that tab. But you can go directly to my skribit page and do the same thing. If using Skribit proves useful or even mildly amusing I'll add a link to it in the feed items as well.

So give Skribit a try. I'm enabling anonymous suggestions for now, though that obviously will only remain until someone figures out how to use it for spam.