Jamie's shared items in Google Reader

The Observer Design Pattern in Delphi – Pull

2009-11-08T21:14:00Z

As mentioned in the previous article The Observer Design Pattern in Delphi – Push I intended to do a “Pull” article to show, what I think is, a significant “feature” of this pattern and one that I thought was underplayed in the Head First Design Patterns book that I’m learning from. I wrote code for this article a couple of times and in the end simply refactored the code from the previous

Dependency Injection For Dummies

2009-11-09T08:26:44Z

The goal of this series is to introduce programming patterns and practices to developers who have little to no familiarity with them. This series does not intend to dive into the intricacies of each pattern / practice, but to give a brief overview that will (hopefully) inspire developers to learn more about them.

Share Your Unicode Migration Story

2009-10-28T17:26:00Z

Delphi developers everywhere (and I am including C++Builder developers here), I am asking for your help in preparing a white paper that I hope will serve the greater community for a long time to come. I have been asked to assemble a white paper on Unicode migration for Delphi and C++ developers, and I want to hear your story.

When Senior Director of Delphi Solutions Mike Rozlog originally approached me about putting together this paper, I was enthusiastic. However, there was a slight problem.

While I can easily write about issues concerning the size of PChars or the ins-and-outs of the new UnicodeString type, I felt that this white paper was too important to simply repeat the obvious. Instead, I really wanted to go for an “in the real world this is what you have to do” approach. And while it is a noble goal, it is simply too much for one developer. We each have our specializations, and each of us, if we do run into migration problems, is going to do so in our particular domain.

For example, I am primarily a database developer, and I often work at a pretty high level; querying databases, building user interfaces, generating reports, you know, the bread-and-butter kind of stuff. And as far as Unicode migration goes, I’ve got it easy. Several of my larger applications have converted with few or no problems.

But some of you work much closer to the metal, making extensive use of calls to the operating system API, using sophisticated third-party libraries, manipulating data at the bit and byte level, and generally working in a world where the size of characters and strings matters. If so, your Unicode migration has likely encountered challenges, some incidental and some significant, that you’ve had to solve along the way. It’s these real world stories that I want to hear about.

Now, you might be wondering, why have I been asked to write this white paper, given that I don’t normally have to deal with some of the more serious Unicode migration issues? It’s a valid question, and there is an equally valid answer. Another of my skills, beyond software development, is the ability to communicate clearly and simply about complex topics. With your input, I intend to produce a white paper that is organized, clear, and helpful to those whose Unicode migration is not yet started or complete. And, I want the paper to reflect the wealth of experience of the collective Delphi community on the subject, not just my own experience.

Here is what I am looking for. If you have solved a Unicode-related challenge in converting existing code or techniques to Delphi 2009 or later, I want your input. The basic input that I need is your name and email address (so I can contact you if I have questions), the company you work for (if you are willing to share that), a description of the problem, and a description of the solution. Code samples that demonstrate the problem and solution are preferred (though we do not want anything that is proprietary or non-disclosable).

If your solution is included in the white paper, you will be recognized for your contribution by name, unless you specifically request to remain anonymous.

You can email your contribution to Tim Del Chiaro at Tim.DelChiaro@embarcadero.com. Alternatively, you can submit your contribution using the Get Published interface at the Embarcadero Developer Network (EDN). A detailed description of how to submit your contribution can be found in the EDN article located http://edn.embarcadero.com/article/40018.

Of course, you can also send questions or contributions to me at cjensen@jensendatasystems.com. Please include the words “Unicode migration” in the subject line. Please also feel free to contact me if you need help putting together your story.

We are looking to release this white paper as soon as possible, so we necessarily have to set a deadline for contributions. Please submit your Unicode migration story by 5:00pm (GMT-8), November 27^th, 2009.

Now, finally, a few words to please the Lawyers. Any Unicode migration contribution you send to me, Tim Del Chiaro, or upload to the Get Published interface, will be interpreted as explicit permission from you for Embarcadero Technologies to include your contribution, or descriptions of it, in the Unicode migration white paper, as well as in any other form, and that you have the right to grant this permission.

Thank you for your consideration. I sincerely hope to hear from many of you with your stories, and look forward to producing a paper that will help all Delphi developers with their Unicode migration challenges.

TDM Rerun #14: A Portable XML

2009-10-25T21:59:00Z

The code unit, OmniXML.pas, which contains the XML representation interfaces, parser and writer, was written by a single programmer, Miha Remec (he is also the guy behind the www.omnixml.com website). He started writing it in 2000, because he was missing a native Delphi DOM parser, one that would represent the DOM the same way as it was designed. The best Delphi parser around at that time was OpenXML, but it used classes to represent XML elements, not interfaces. OmniXML uses interfaces, derived from the IXMLNode (as specified by the DOM). That also makes it almost completely compatible with the MSXML parser, which uses the same approach.

- A Portable XML, The Delphi Magazine 105, May 2004

In the 2004 May issue I wrote about OmniXML, a native Delphi XML parser. I described the OmniXML approach and wrote few short pieces of code that demonstrated its use.

Today, five years later, OmniXML is still strong and I’m still using it, as you can see in my Fluent XML series.

Links: article (PDF, 45 KB), source code (ZIP, 795 KB).

---
Published under the Creative Commons Attribution 3.0 license

Delphi Legends 2009 Community Award

2009-10-24T16:58:48Z

We reached at the final of our poll and we are ready to present the one who will receive the “Delphi Legends Community Award” this year. First of all, a big thank you for all which voted (you are approx. 500 coders who voted). We should also be grateful to the Team members who passed by [...]

Pascal on the iPhone

2009-10-13T18:03:23Z

Olaf Monien, Holger Flick and I (Daniel Magin) are playing in our free time with a beta version from RemObjects. marc hoffman and his team did a great job to integrate the Oxygene stuff now also to the apple mac side. I have developed a lot of stuff on Xcode with Objective C and (my personal opinion) this shows me why i love pascal.

it was only some simple steps to produce a native iPhone Application in Delphi like style. Olaf and I have developed a test application to show the current location of our iPhone. Yeaaahhhh and all written in pascal (!).

for Design we use the original Apple Interface Builder. to develop the code in pascal remobjects is producing a pascal plugin in monotouch. also i use my original apple certificate in the IDE so i have deployed the application to my real iPhone. The iPhone App is full native compile. So it can be pushed later also to the iTunes appstore.

so it is only some time to wait to have a public Pascal compiler for the iPhone.

Here the Application on the iPhone Simulator

Here some Pascal Code

An here on my physical iPhone

regards

daniel

btw: think delphi!

Working with Delphi’s new Exception.StackTrace

2009-10-12T16:30:16Z

One feature I often miss when using Delphi is the support for proper exception stack traces at run-time. You know, those useful stack traces that show you exactly where an exception occurred, ideally with the method name and line number of where the exception was raised. Both .NET and Java have excellent stack trace support built right into the framework and the Exception classes. You just call Exception.StackTrace (.NET) or Exception.getStackTrace (Java) and get a detailed analysis of where the exception was thrown and how it got passed to your exception handler.

Unfortunately, Delphi never had good (built-in) run-time support for stack traces. The features for stack traces during debugging in the IDE are/were okay, but there were nothing in the language or framework which helped you to find out programmatically where an exception occurred at run-time and, more importantly, how it got passed to your exception handler (besides the original exception address, maybe). So, I was happy to see that Delphi 2009 finally introduced a new StackTrace property which, I hoped, would return a full-blown stack trace when you caught an exception.

The initial happiness soon wore off when I realized that the StackTrace property was really just a placeholder to return a stack trace from a possible stack trace provider rather than a real stack trace implementation. So, without such a provider (and there’s none that comes directly with Delphi), there is still no way to get a stack trace for your exceptions. Though a bit disappointing, the good thing is that there’s now finally a standardized way to get a stack trace, even if it’s not implemented by default.

Exception reporting tools such as Eurekalog or madExcept or debug helpers such as the JclDebug unit can register themselves as providers and use their engines to return a stack trace when an exception is raised. I’ve built a small unit which demonstrates this with the Jcl in combination with our logging tool SmartInspect and I’ve heard Fabio of Eurekalog is working on a similar feature for his component:

unit StackTrace;

interface

uses
  SysUtils, Classes, JclDebug;

implementation

function GetExceptionStackInfoProc(P: PExceptionRecord): Pointer;
var
  LLines: TStringList;
  LText: String;
  LResult: PChar;
begin
  LLines := TStringList.Create;
  try
    JclLastExceptStackListToStrings(LLines, True, True, True, True);
    LText := LLines.Text;
    LResult := StrAlloc(Length(LText));
    StrCopy(LResult, PChar(LText));
    Result := LResult;
  finally
    LLines.Free;
  end;
end;

function GetStackInfoStringProc(Info: Pointer): string;
begin
  Result := string(PChar(Info));
end;

procedure CleanUpStackInfoProc(Info: Pointer);
begin
  StrDispose(PChar(Info));
end;

initialization
  // Start the Jcl exception tracking and register our Exception
  // stack trace provider.
  if JclStartExceptionTracking then
  begin
    Exception.GetExceptionStackInfoProc := GetExceptionStackInfoProc;
    Exception.GetStackInfoStringProc := GetStackInfoStringProc;
    Exception.CleanUpStackInfoProc := CleanUpStackInfoProc;
  end;

finalization
  // Stop Jcl exception tracking and unregister our provider.
  if JclExceptionTrackingActive then
  begin
    Exception.GetExceptionStackInfoProc := nil;
    Exception.GetStackInfoStringProc := nil;
    Exception.CleanUpStackInfoProc := nil;
    JclStopExceptionTracking;
  end;
end.

The unit merely starts and stops the exception tracking of the Jcl, implements a minimal stack trace provider and registers for the Exception provider events. The GetExceptionStackInfoProc and CleanUpStackInfoProc functions are automatically called by the RTL to give the provider the opportunity to initialize and cleanup the stack trace after an exception occurred. GetStackInfoStringProc is called indirectly when you access the StackTrace property and is responsible for returning the actual stack trace.

So, how do you use this unit? Let’s have a look at the following example:

...

uses
  ..., StackTrace;

type
  TForm1 = class(TForm)
    Button1: TButton;
    procedure Button1Click(Sender: TObject);
  private
    procedure SomeMethod;
  end;

implementation

procedure TForm1.Button1Click(Sender: TObject);
begin
  try
    SomeMethod;
  except
    // Log the exception: We use SmartInspect here because it has
    // built-in support for Exception.StackTrace but you could also
    // access the StackTrace property here directly.
    SiMain.LogException;
  end;
end;

procedure TForm1.SomeMethod;
begin
  raise Exception.Create('A test exception');
end;

...

As you can see, using this unit is just a matter of adding it to our uses clause. It won’t get any simpler than that. The unit will take care of registering/unregistering itself as a stack trace provider and when you now access the StackTrace property of an Exception object, you will get a detailed stack trace. To include the method names and line numbers in the stack trace, make sure to let the linker include debug symbols into your application and to enable the ‘Use debug .dcus’ option in case you also want line numbers from the VCL and RTL methods.

The SmartInspect Console showing the stack trace of an exception

Now, when you use SmartInspect for logging and have a stack trace provider registered, all your logged exceptions automatically include the exception’s call stack. Pretty useful, isn’t it?

The Observer Design Pattern in Delphi – Push

2009-10-11T22:57:00Z

A summary and [my rendition of] the Delphi code taken from the second chapter of the book, Head First Design Patterns from O’Reilly. As I mentioned in the first post in this series, I’m studying Design Patterns using the Head First Design Patterns book from O’Reilly. As part of this learning process I’m working through the existing examples written in Java and recreating them in Delphi. This

Xml Serialization - Basic Usage

2009-10-08T13:56:00Z

This post relates to XmlSerial.pas which provides XML serialization and de-serialization using the new RTTI in Delphi 2010.

This class has been built to serialize any public and published property or Field.
This was done to mimic the behavior found in the .NET Xml Serialization with the goal of having a set of objects that can serialize in .NET using Delphi Prism and Win32 using Delphi 2010. The complete goals of what I want to accomplish with XmlSerial.pas look at the source code, I detailed out what still needs to be done.

There are two ways to work with the Xml Serialization, one depends on a Pointer to the Type Information, the other uses Generics to get it form the type specified as a parameter.


// Method 1:
var
 o : TypeIWantToSerailze;
 s : TXmlTypeSerializer;
 x : TXmlDocument;
 v : TValue;
begin
  s := TXmlTypeSerializer.create(TypeInfo(o));
  x := TXmlDocument.Create(Self); // NEVER PASS NIL!!!
  s.Serialize(x,o);
  x.SaveToFile('FileName.txt');
  v := s.Deserialize(x);
  o := v.AsType<TypeIWantToSerailze>;
  x.free;
  s.free;
end;

// Method 2:
var
 o : TypeIWantToSerailze;
 s : TXmlSerializer<TypeIWantToSerailze>;
 x : TXmlDocument;
begin
  s := TXmlTypeSerializer<TypeIWantToSerailze>.create;
  x := TXmlDocument.Create(Self); // NEVER PASS NIL!!!
  s.Serialize(x,o);
  x.SaveToFile('FileName.txt');
  o := s.Deserialize(x);
  x.free;
  s.free;
end;

And here is the full code showing how to do this using the generic version.


unit uPerson;
interface

type
  TPerson = class(TObject)
  private
    FLastName: String;
    FBirthday: TDateTime;
    FMiddleName: String;
    FFirstName: String;
    function GetFullName: String;
   published
   public
     property FirstName : String read FFirstName write FFirstName;
     property LastName  : String read FLastName  write FLastName;
     property MiddleName : String read FMiddleName write FMiddleName;
     property FullName : String read GetFullName;
     property Birthday : TDateTime read FBirthday write FBirthday;
     procedure Save(FileName : String);
     class function Load(FileName : String) : TPerson;
   end;


implementation
uses
 XmlDoc,
 Classes,
 XmlSerial;

{ TPerson }

function TPerson.GetFullName: String;
begin
 result := FFirstName + ' ' + FMiddleName + ' ' + FLastName;
end;

class function TPerson.Load(FileName: String): TPerson;
var
  lSerialize : TXmlSerializer<TPerson>;
  lOwner : TComponent;
  lDoc   : TxmlDocument;
begin
  lOwner := TComponent.Create(nil);  // Required to make TXmlDocument work!
  try
    lDoc := TXmlDocument.Create(lOwner);  // will be freed with lOwner.Free
    lDoc.LoadFromFile(FileName);
    lSerialize := TXmlSerializer<TPerson>.Create;
    try
      result := lSerialize.Deserialize(lDoc);
    finally
      lSerialize.Free;
    end;
  finally
    lOwner.Free;
  end;
end;

procedure TPerson.Save(FileName: String);
var
  lSerialize : TXmlSerializer<TPerson>;
  lOwner : TComponent;
  lDoc   : TxmlDocument;
begin
  lOwner := TComponent.Create(nil);  // Required to make TXmlDocument work!
  try
    lDoc := TXmlDocument.Create(lOwner);  // will be freed with lOwner.Free
    lSerialize := TXmlSerializer<TPerson>.Create;
    try
      lSerialize.Serialize(lDoc,Self);
      lDoc.SaveToFile(FileName);
    finally
      lSerialize.Free;
    end;
  finally
    lOwner.Free;
  end;
end;

end.

The project that shows how to use this object.


program Project12;

{$APPTYPE CONSOLE}

uses
  SysUtils,
  Windows,
  XmlSerial,
  Forms,
  uPerson in 'uPerson.pas';

var
  P : TPerson;

begin
  Application.Initialize;
  P := TPerson.Create;
  try
   P.FirstName := 'John';
   P.MiddleName := 'C';
   P.LastName := 'Doe';
   P.Birthday := EncodeDate(1995,2,14);
   P.Save('C:\test.xml');
  finally
    P.Free;
  end;

  p := TPerson.Load('C:\test.xml');
  try
    writeln(P.FullName);
    Writeln(DateToStr(P.Birthday));
    Readln;
  finally
    p.Free;
  end;
end.

Output:


John C Doe
2/14/1995

Current output of the XML file, notice the Date, it's something I want to change, if you check the roadmap in the xmlserial.pas source.


<TPerson>
   <FirstName>John</FirstName>
   <LastName>Doe</LastName>
   <MiddleName>C</MiddleName>
   <Birthday>34744</Birthday>
</TPerson>

Hopefully this covers enough of how to use this to get you started.
I will show how to customize the behavior using attributes in a future post.

RTTI Article List

INI persistence the RTTI way

2009-10-08T12:05:00Z

This post is based around the IniPersit.pas code that I just released.

I commonly create configuration classes to create a common and easy way to access information stored in INI, Registry, or XML. In these examples I will show how I used the new RTTI and Attributes in Delphi 2010 to provide a new way of creating a configuration class that access information stored in an INI file.

Lets first start off showing how to use the New Unit, then we can pull back the covers and show how it works.


unit ConfigSettings;
interface
uses
  IniPersist;

type
  TConfigSettings = class (TObject)
  private
    FConnectString: String;
    FLogLevel: Integer;
    FLogDirectory: String;
    FSettingsFile: String;
  public
    constructor create;
    // Use the IniValue attribute on any property or field
    // you want to show up in the INI File.
    [IniValue('Database','ConnectString','')]
    property ConnectString : String read FConnectString write FConnectString;
    [IniValue('Logging','Level','0')]
    property LogLevel : Integer read FLogLevel write FLogLevel;
    [IniValue('Logging','Directory','')]
    property LogDirectory : String read FLogDirectory write FLogDirectory;

    property SettingsFile : String read FSettingsFile write FSettingsFile;
    procedure Save;
    procedure Load;

  end;

implementation
uses SysUtils;

{ TApplicationSettings }

constructor TConfigSettings.create;
begin
  FSettingsFile := ExtractFilePath(ParamStr(0)) +  'settings.ini';
end;

procedure TConfigSettings.Load;
begin
// This loads the INI File Values into the properties.
   TIniPersist.Load(FSettingsFile,Self);
end;

procedure TConfigSettings.Save;
begin
// This saves the properties to the INI
   TIniPersist.Save(FSettingsFile,Self);
end;

end.


program Project13;

{$APPTYPE CONSOLE}

uses
  SysUtils,
  IniPersist,
  ConfigSettings;

var
  Settings : TConfigSettings;

begin
    Settings := TConfigSettings.Create;
    try
      Settings.ConnectString := '\\127.0.0.1\DB:2032';
      Settings.LogLevel := 3;
      Settings.LogDirectory := 'C:\Log';
      Settings.Save;
    finally
      Settings.Free;
    end;

    Settings := TConfigSettings.Create;
    try
    Settings.Load;
    WriteLn(Settings.ConnectString);
    Writeln(Settings.LogLevel);
    Writeln(Settings.LogDirectory);
    finally
      Settings.Free;
    end;
    Readln;
end.

Output:


\\127.0.0.1\DB:2032
3
C:\Log

Resulting INI File:


[Database]
ConnectString=\\127.0.0.1\DB:2032
[Logging]
Level=3
Directory=C:\Log

As you can see by the above code there really is not much too it, if you want a field or a property to be stored in the INI File, you just need to add the IniValue Attribute.


  TExampleClass = class (TObject)
  private
    FConnectString: String;
  public
    [IniValue('Database','ConnectString')]
    property ConnectString : String read FConnectString write FConnectString;
  end;

The constructor of the INIValue allows you to specify the Section, Name you the field or Property stored in. It also allows you to specify a default value if the name & section did not exist in the INI File.


  IniValueAttribute = class(TCustomAttribute)
  private
    FName: string;
    FDefaultValue: string;
    FSection: string;
  published
     constructor Create(const aSection : String;const aName : string;const aDefaultValue : String = '');
     property Section : string read FSection write FSection;
     property Name : string read FName write FName;
     property DefaultValue : string read FDefaultValue write FDefaultValue;
  end;

...

constructor IniValueAttribute.Create(const aSection, aName, aDefaultValue: String);
begin
  FSection := aSection;
  FName := aName;
  FDefaultValue := aDefaultValue;
end;

So the magic is really contained in TIniPersist


  TIniPersist = class (TObject)
  private
    class procedure SetValue(aData : String;var aValue : TValue);
    class function GetValue(var aValue : TValue) : String;
    class function GetIniAttribute(Obj : TRttiObject) : IniValueAttribute;
  public
    class procedure Load(FileName : String;obj : TObject);
    class procedure Save(FileName : String;obj : TObject);
  end;

The load and save methods are nearly identical, so lets take a look at load.


class procedure TIniPersist.Load(FileName: String; obj: TObject);
var
 ctx : TRttiContext;
 objType : TRttiType;
 Field : TRttiField;
 Prop  : TRttiProperty;
 Value : TValue;
 IniValue : IniValueAttribute;
 Ini : TIniFile;
 Data : String;
begin
 ctx := TRttiContext.Create;
 try
   Ini := TIniFile.Create(FileName);
   try
     objType := ctx.GetType(Obj.ClassInfo); 
     for Prop in objType.GetProperties do
     begin
       IniValue := GetIniAttribute(Prop);
       if Assigned(IniValue) then
       begin
          Data := Ini.ReadString(IniValue.Section,IniValue.Name,IniValue.DefaultValue);
          Value := Prop.GetValue(Obj);
          SetValue(Data,Value);
          Prop.SetValue(Obj,Value);
       end;
     end;
     for Field in objType.GetFields do
     begin
       IniValue := GetIniAttribute(Field);
       if Assigned(IniValue) then
       begin
          Data := Ini.ReadString(IniValue.Section,IniValue.Name,IniValue.DefaultValue);
          Value := Field.GetValue(Obj);
          SetValue(Data,Value);
          Field.SetValue(Obj,Value);
       end;
     end;
   finally
     Ini.Free;
   end;
 finally
   ctx.Free;
 end;
end;

So you can see we basically loop through all the properties and field check for an Attribute, if it exists we get the current value which sets the TypeInfo in the TValue object. Then we assign the string returned from the INI file into the TValue and call SetValue()

Lets look at the two methods called.


  class procedure SetValue(aData : String;var aValue : TValue);
  class function GetIniAttribute(Obj : TRttiObject) : IniValueAttribute;

Lets look first at SetValue(). You will see that it depends on the TypeInfo being present in the TValue being passed in. We check the TValue and perform the conversions required to convert the String to the Correct Type before storing it
into the TValue.


class procedure TIniPersist.SetValue(aData: String;var aValue: TValue);
var
 I : Integer;
begin
 case aValue.Kind of
   tkWChar,
   tkLString,
   tkWString,
   tkString,
   tkChar,
   tkUString : aValue := aData;
   tkInteger,
   tkInt64  : aValue := StrToInt(aData);
   tkFloat  : aValue := StrToFloat(aData);
   tkEnumeration:  aValue := TValue.FromOrdinal(aValue.TypeInfo,GetEnumValue(aValue.TypeInfo,aData));
   tkSet: begin
             i :=  StringToSet(aValue.TypeInfo,aData);
             TValue.Make(@i, aValue.TypeInfo, aValue);
          end;
   else raise EIniPersist.Create('Type not Supported');
 end;
end;

Now lets take a look at GetIniAttribute() the goal of this method is to check to see if a given TRttimember (Field or Property) has the IniValue attribute, and if it does return it, otherwise return NIL.


class function TIniPersist.GetIniAttribute(Obj: TRttiObject): IniValueAttribute;
var
 Attr: TCustomAttribute;
begin
 for Attr in Obj.GetAttributes do
 begin
    if Attr is IniValueAttribute then
    begin
      exit(IniValueAttribute(Attr));  // Exit with a parameter new in Delphi 2010
    end;
 end;
 result := nil;
end;

So all in all, its really not much code, and it make usage simple. Now granted TIniValue is not all that complex, but this situation could be applied to a variety of other applications.

RTTI Article List

RTTI - Practical Examples

2009-10-08T11:51:00Z

Well it took a lot longer than I wanted to get this code out.

There were many more rough edges than I wanted, but after some gentle and firm requests I realized I needed to get this out before the rough edges are finished.

Specifically the XmlSerial.pas has a road map in source code detailing what I still need to get done.

What is released:

IniPersist.pas Allows easy mapping of properties and fields to an INI File.

XmlSerial.pas Object and Record Serialization and De-serialization to XML

ObjDs.pas Read-only mapping of Objects to TClientDataSets.

RttiUtils.pas Things to help with common RTTI needs.

How to get the code:
Follow the above links to each unit, or just use SVN.

I am now working on some blog posts to show the how to use this code.

In-Memory DataSets: ClientDataSet and .NET DataTable Compared: Part 1 Overview

2009-09-30T08:04:00Z

As some of you know, I have been a big fan of Delphi's ClientDataSets since they were first introduced in Delphi 3 (that's way back in 1997). When .NET shipped, its data access framework, ADO.NET, also included an in-memory dataset, named the DataTable. (.NET also includes the DataSet class, but in most cases, the DataTable class bears the strongest resemblance to the ClientDataSet.)

Both ClientDataSets and DataTables are in-memory datasets, and as such, share a lot of features in common. On the other hand, they are radically different in a number of interesting ways. In this series of articles, which begins with this one, I will examine the general features of in-memory datasets, and provide a direct comparison between ClientDataSets and .NET DataTables.

This article begins with a introduction to in-memory datasets in general. In future posts I will provide explicit code examples of how to perform various tasks with these two datasets, including how to create them in code, reading and writing data, sorting, filtering, persisting, navigating, and so on.

I hope you enjoy.

Developing with Disconnected Datasets

Disconnected datasets are database table-like structures that are stored in memory. These types of datasets are sometimes referred to as cached datasets or in-memory datasets. In this series they will be referred to as in-memory datasets.

In-memory datasets are structured, high-performance, self-describing data structures temporarily stored in memory. A significant feature of in-memory datasets is that they maintain, and can persist, a change log. The change log permits you to programmatically determine what changes have been made to the data since some point in time, often when the data was originally loaded into the dataset. This information is essential if you need to persist these changes back to an original source, such as a Web service, underlying database, or other persistence mechanism.

Data persistance in the .NET framework is based on in-memory datasets, and Delphi has included this capability since Delphi 3 in the form of the ClientDataSet.

This series begins with an overview of the features that make in-memory datasets so useful, including their self-descriptive nature, ability to hold sophisticated relational data structures, their close association with XML, and their persistence and management of change information. How these features are surfaced in both ClientDataSets and .NET datasets is discussed in this section.

In-memory datasets are the cornerstone of modern software development. Nothing confirms this statement as much as Microsoft's commitment to in-memory datasets as a central aspect of the database framework in .NET, ADO.NET.

While most developers consider in-memory datasets for the presentation layer in applications, this use represents only a fraction of the possibilities for these powerful data structures. As this series will demonstrate, the characteristics of in-memory datasets make them a valuable tool for many different aspects of application development. The following are the essential feature of in-memory datasets:

High performance

Self describing

Flexible

Change log managing

Persistable

Individually, these characteristics provide a compelling argument for using in-memory datasets in your applications. But it is the combination of these features in a single, easy to use class that makes them so valuable for a wide range of software features. The following sections look at each of these features in greater depth.

High Performance

In-memory datasets reside entirely in RAM (random access memory). Consequently, operations on the data they contain, including searches, filters, and sorts, are very fast. This is particularly true with respect to ClientDataSets, since these can have indexes on this data as well. But even for .NET datasets, which currently support a single index at any given moment (the primary index), data-related operations are many times faster than those that require disk reads (as is the case with a physical database).

Self Describing

In-memory datasets are formally designed around the concept of a database table. Unlike an array or sequence, whose data elements have a data type, and that's about it, the fields of a data table each have a name, a data type, and sometimes a data size (for example, the size of a text field or precision of a floating point number).

In addition, the fields of a data table may have constraints, such as a required field constraint, or referential integrity constraints when two or more in-memory tables are related. This information is typically referred to as metadata, which is data about data.

In ClientDataSets, you access the metadata of a dataset using the Fields property of the dataset, which contains a collection of TField instances. In .NET data tables, you access this information using the Columns property, which contains a collection of DataColumn instances.

Flexible

In-memory datasets are designed to hold nearly any kind of data that might be stored in a physical database. This includes primitive data values, such as integers, strings, real numbers, and date/time values. But it also includes variable length objects, such as memos and Blobs (binary large objects). As a result, an in-memory dataset can hold the pages of a Web site, PDF files, and even executables (.EXEs and .DLLs). If it can be stored in a file on disk, it can be stored in an in-memory dataset (obviously, subject to the limits imposed by your available RAM).

Change Log Managing

Both ClientDataSets and .NET DataTables have a change log. The change log permits you to manage the unresolved changes that have been posted to the dataset's data since you loaded it into memory. This management includes the ability to determined precisely what changes have occurred (which records were inserted, deleted, and field-level modifications), revert changes to their prior state, cancel all changes, or commit those changes permanently, thereby erasing the change log. With ClientDataSets, this change log if held in the Delta property. For .NET DataTables, you use the RowStateFilter of a DataView to access the change log.

To manage the change log for a ClientDataSet, you use its methods, such as RevertRecord, UndoLastChange, CancelChanges, and ApplyUpdates. In addition, you can use the RecordStatus, StatusFilter, and Fields properties to examine the change log contents.

With .NET DataTables, you use the methods of the DataTable and DataView classes to control the change log, including NewRow, DeleteRow, AcceptChanges, and RejectChanges. To examine the change log, you use the RowStateFilter and Rows properties.

Persistable

Of all the features supported by in-memory datasets, the ability to persist state is arguably the most powerful. Not only can you save an in-memory dataset's data, but you can save its change log as well. Specifically, it is possible to save the current state of an in-memory dataset to a file, Web service, or memo field of a database, and then to restore that dataset at a later time to its exact prior state. In short, there is absolutely no difference between the in-memory dataset prior to, and following, its persistence.

Consider the following scenario: After loading data into memory, and making several edits to an in-memory dataset, that dataset can be written to a file. At a future time, that dataset can be restored from the file, and the edits that were previously performed can be examined and rejected or accepted.

Furthermore, since the change log is restored to its exact prior state, that information can be used to resolve those edits to the underlying database from which the data was originally loaded. No information is lost during the time that the dataset is in storage, no matter how long its state was persisted.

Copyright (c) 2009 Cary Jensen. All Rights Reserved

Unhibernating – with a T-shirt

2009-09-29T18:35:00Z

Dear reader,

I’m fully aware that I was silent for a long long time.

That’s how life goes. Sometimes you have time to think and write and sometimes software has bugs, children are growing, house is telling you that it shouldn’t be in the “fixer-upper” mode anymore and the body starts warning that the guaranty has expired. Plus the days suddenly have only 24 hours and not 28 as it was customary for the last ten years.

In short – I had no time. Life was going on. And around. And over me. Especially over me.

Luckily, everything is turning out fine and I will again write about my Delphi adventures.

Today I visited the RAD Studio 2010 presentation organized by Embarcadero and Slovenian Delphi dealer, Marand. Mark Barrington and Pawel Glowacki were showing RAD Studio 2010, Embarcadero’s database tools and All-Access. Sadly, I only had time to attend the Delphi session which was very good and informative. (Thumbs up, Pawel!)

There were also T-shirts and I was lucky to get one.

Nice T-shirt but it’s even better if you look closer at the code.

Parallel Erathostenes sieve! That’s definitely the code I can appreciate!

[For the OTL fans – OTL is alive and well and I will publish next version sometime in October. And then I’ll write the documentation. Promise.]

---
Published under the Creative Commons Attribution 3.0 license

Class constructors and Generics

2009-09-13T10:39:32Z

The new great addition to the Delphi language in Delphi 2010 is the possibility to specify a class constructor and a class destructor to your class/record. I will not describe this feature in this post since you can see the online documentation for it on the Embarcadero Doc Wiki. The part I am interested in is the combination of class constructors and generics. As you might already know generic types aren’t really “run-time entities” but rather “compile-time” ones. This makes the initialization of “static” members of the type a bit more complicated. See for example this record:

type
  TMyType<T> = record
  private class var
    FSomeList: TList<T>;

  public
    ...
    ...
  end;

In this case the static FSomeList variable needs to be initialized to make sense. Normally, in non-generic classes, you would use the “initialization” and “finalization” sections to create and then destroy that variable. In generic classes this is impossible though. You have to get some workarounds (like initializing the list lazily), but you still cannot destroy the list on finalization.

Well, not anymore! Using class constructors and destructors allows you to easily initialize any static member of a type:

type
  TMyType<T> = record
  private class var
    FSomeList: TList<T>;

    class constructor Create;
    class destructor Destroy;
  public
    ...
    ...
  end;

class constructor TMyType<T>.Create;
begin
  // Executed on application initialization.
  FSomeList := TList<T>.Create();
end;

class destructor TMyType<T>.Destroy;
begin
  // Freed on application termination.
  FSomeList.Free;
end;

There is a catch tough. Since generic types are compile-time entities, any unit that uses a specialized generic type basically defines that type in itself. For example if you use TList<String> in four of your units, all four units will internally declare the TList<String> class. This results in the TList<T>’s class constructor to be executed four times – once per unit. This is expected behavior since each type specialization has different static members. For example:

type
  TDistinctType<T> = record
  private class var
    FMarker: Integer;

    class constructor Create;
  end;

If TDistinctType<String> is used in multiple units, each unit’s version has it own FMarker, which means it needs to be initialized for each unit.

The conclusion — be aware that class constructors and destructors for generic types may execute multiple times.

What's New In RAD Studio 2010 With Mike Rozlog

2009-09-11T23:46:27Z

Aspect Oriented Programming with Delphi Prism

2009-09-10T08:00:06Z

I’ve created a short, eleven minute video showing the basics of Aspect Oriented Programming with Delphi Prism. AOP is a really cool, powerful, and easy to use language feature of Delphi Prism. You can read more about AOP and how it works on Wikipedia. You can read more ...

A visit with Delphi 1 (redux)

2009-09-10T08:00:06Z

When last we met[1], I was reminiscing about my reunion with the very first release of Delphi. I still get a thrill running "delphi.exe" and watching it just instantly appear. Of course time marches on, and today I'm sad to report that my old pal no longer runs on my current ...

Custom Marshalling/UnMarshalling in Delphi 2010

2009-09-01T20:43:26Z

Introduction
Some days ago, Embarcadero has presented the new version of RAD Studio, 2010.
The are many new features, but you can find in a lot places around the web, so
I won’t repeat them here.

One of the things widely requested from all Delphi programmers all over the world over the past few years, including myself, is
certainly a new and more powerful RTTI.

The new system of RTTI has finally arrived, and pave the way for a large number of applications.
One area that has benefited from the new RTTI is for sure the marshaled objects.

Marshaling is defined as follows:

“In computer science, marshalling (similar to serialization) is the process of
transforming the memory representation of an object to a data format suitable for
storage or transmission. It is typically used when data must be moved between
different parts of a computer program or from one program to another.
The opposite, or reverse, of marshalling is called unmarshalling (demarshalling) (similar to deserialization).”
–WikiPedia

In Delphi 2010 the process of serialization and deserialization is handled respectively by a Marshaller and an Unmarshaller.

The built-in format for the serialization of any Delphi object is JSON.
There are 2 main classes responsible for serializing objects into JSON, both present in the unit DBXJSONReflect:
- TJSONMarshal
- TJSONUnMarshal

Let’s say you have an object defined as follow:

type

TKid = class
FirstName: String;
LastName: String;
Age: Integer;
end;

To serialize and deserialize an instance of TKid it requires the following steps:

var

Mar: TJSONMarshal; //Serializer
UnMar: TJSONUnMarshal; //UnSerializer
Kid: TKid; //The Object to serialize
SerializedKid: TJSONObject; //Serialized for of object
begin
Mar := TJSONMarshal.Create(TJSONConverter.Create);
try
Kid := TKid.Create;
try
Kid.FirstName := 'Daniele';
Kid.LastName := 'Teti';
Kid.Age := 29;
SerializedKid := Mar.Marshal(Kid) as TJSONObject;
finally
FreeAndNil(Kid);
end;
finally
Mar.Free;
end;
//Output the JSON version of the Kid object
WriteLn(SerializedKid.ToString);
// UnMarshalling Kid
UnMar := TJSONUnMarshal.Create;
try
Kid := UnMar.UnMarshal(SerializedKid) as TKid;
try
//now kid is the same as before marshalling
Assert(Kid.FirstName = 'Daniele');
Assert(Kid.LastName = 'Teti');
Assert(Kid.Age = 29);
finally
Kid.Free;
end;
finally
UnMar.Free;
end;
end;

Simple, isn’t it?
To access the JSON string that is our object, we must call the method ToString.
The JSON representation of this object SerializedKid can be saved to file,
sent to a remote server, used by a Web page from a web service, stored on a database or sent into space (!!!).
The Delphi application re-read the JSON string, you can recreate the object as it was at the time of serialization.
But anyone with a JSON parser can still read the data in our object, even non Delphi client.
These are the advantages of having used an open format and standard.

So far the simple part …
How serialize a field differently from the default?

Suppose we add the date of birth to our TKid:

type

TKid = class
FirstName: String;
LastName: String;
Age: Integer;
BornDate: TDateTime;
end;

Serialize a TDateTime, localized and that I have in JSON string is a float, because for Delphi TDateTime is a decimal number.
If I read the data from another program Delphi, no problem, but if I wanted to read a script in JavaScript? or. NET? or Ruby?
Then I use a format “DATA” to understand, even for these languages.
The new engine provides the serialization too.
Is needed, however, to tell the Marshaller and UnMarsheller how to represent and reconstruct a particular
object field by two statements like the following:

//marshaller

Marshaller.RegisterConverter(TKid, 'BornDate',
function(Data: TObject; Field: string): string
var
ctx: TRttiContext; date : TDateTime;
begin
date := ctx.GetType(Data.ClassType).GetField(Field).GetValue(Data).AsType<TDateTime>;
Result := FormatDateTime('yyyy-mm-dd hh:nn:ss', date);
end);
//UnMarshaller
UnMarshaller.RegisterReverter(TKid, 'BornDate',
procedure(Data: TObject; Field: string; Arg: string)
var
ctx: TRttiContext;
datetime:TDateTime;
begin
datetime := EncodeDateTime(StrToInt(Copy(Arg, 1, 4)),
StrToInt(Copy(Arg, 6, 2)),
StrToInt(Copy(Arg, 9, 2)),
StrToInt(Copy(Arg, 12, 2)),
StrToInt(Copy(Arg, 15, 2)),
StrToInt(Copy(Arg, 18, 2)), 0);
ctx.GetType(Data.ClassType).GetField(Field).SetValue(Data, datetime);
end);

The anonymous method is called when the marshaller serializes the field ‘BornDate’ is called “Converter” while Unmarshaller anonymous method that calls when he has to reconstruct the object from the JSON string is the “Reverter”.
Thus serializing a TKid assure you that my object is readable both by Delphi from another language without loss of information.

But what happens when I have to serialize a complex type?

Suppose we extend TKid this:

type

TTeenager = class(TKid)
Phones: TStringList;
constructor Create; virtual;
destructor Destroy; virtual;
end;

We must define a Converter and a Reverter for the TStringList class.
We can do it this way:

var

Marshaller: TJSONMarshal;
UnMarshaller: TJSONUnMarshal;
Teenager: TTeenager;
Value, JSONTeenager: TJSONObject;
begin
Marshaller := TJSONMarshal.Create(TJSONConverter.Create);
try
Marshaller.RegisterConverter(TTeenager, 'BornDate',
function(Data: TObject; Field: string): string
var
ctx: TRttiContext; date : TDateTime;
begin
date := ctx.GetType(Data.ClassType).GetField(Field).GetValue(Data).AsType<TDateTime>;
Result := FormatDateTime('yyyy-mm-dd hh:nn:ss', date);
end);
Marshaller.RegisterConverter(TStringList, function(Data: TObject): TListOfStrings
var
i, count: integer;
begin
count := TStringList(Data).count;
SetLength(Result, count);
for i := 0 to count - 1 do
Result[i] := TStringList(Data)[i];
end); //TStringList Converter
Teenager := TTeenager.CreateAndInitialize;
try
Value := Marshaller.Marshal(Teenager) as TJSONObject;
finally
Teenager.Free;
end;
finally
Marshaller.Free;
end;
// UnMarshalling Teenager
UnMarshaller := TJSONUnMarshal.Create;
try
UnMarshaller.RegisterReverter(TTeenager, 'BornDate',
procedure(Data: TObject; Field: string; Arg: string)
var
ctx: TRttiContext;
datetime: TDateTime;
begin
datetime := EncodeDateTime(StrToInt(Copy(Arg, 1, 4)),
StrToInt(Copy(Arg, 6, 2)),
StrToInt(Copy(Arg, 9, 2)),
StrToInt(Copy(Arg, 12, 2)),
StrToInt(Copy(Arg, 15, 2)),
StrToInt(Copy(Arg, 18, 2)), 0);
ctx.GetType(Data.ClassType).GetField(Field).SetValue(Data, datetime);
end);
UnMarshaller.RegisterReverter(TStringList, function(Data: TListOfStrings): TObject
var
StrList: TStringList;
Str: string;
begin
StrList := TStringList.Create;
for Str in Data do
StrList.Add(Str);
Result := StrList;
end); //TStringList Reverter
Teenager := UnMarshaller.Unmarshal(Value) as TTeenager;
try
Assert('Daniele' = Teenager.FirstName);
Assert('Teti' = Teenager.LastName);
Assert(29 = Teenager.Age);
Assert(EncodeDate(1979, 11, 4) = Teenager.BornDate);
Assert(3 = Teenager.Phones.Count);
Assert('NUMBER01'=Teenager.Phones[0]);
Assert('NUMBER02'=Teenager.Phones[1]);
Assert('NUMBER03'=Teenager.Phones[2]);
finally
Teenager.Free;
end;
finally
UnMarshaller.Free;
end;
end;

There are different types of Converter and Reverter.
In the the DBXJSONReflect there are 8 types of converters:

//Convert a field in an object array
TObjectsConverter = reference to function(Data: TObject; Field: String): TListOfObjects;
//Convert a field in a strings array
TStringsConverter = reference to function(Data: TObject; Field: string): TListOfStrings;
//Convert a type in an objects array
TTypeObjectsConverter = reference to function(Data: TObject): TListOfObjects;
//Convert a type in a strings array
TTypeStringsConverter = reference to function(Data: TObject): TListOfStrings;
//Convert a field in an object
TObjectConverter = reference to function(Data: TObject; Field: String): TObject;
//Convert a field in a string
TStringConverter = reference to function(Data: TObject; Field: string): string;
//Convert specified type in an object
TTypeObjectConverter = reference to function(Data: TObject): TObject;
//Convert specified type in a string
TTypeStringConverter = reference to function(Data: TObject): string;

Each of them deals with a particular conversion object representation in the final serialization, in our case we will use them to convert to JSON.

Also in the DBXJSONReflect unit are defined many “Reverter” dealing with retrieving
the serialized version of the data and use it to reconstruct the object previously serialized.
Because they are complementary to the Converter, I will not copy them here.

As a final example, we derive from TProgrammer by TTeenager adding a list of Laptops in the properties.

Is therefore necessary to introduce a new pair of Converter / Reverter.
In this example I have defined all the converter and reverter in another unit in
order to have more readable code:

type

TLaptop = class
Model: String;
Price: Currency;
constructor Create(AModel: String; APrice: Currency);
end;
TLaptops = TObjectList<TLaptop>;
TProgrammer = class(TTeenager)
Laptops: TLaptops;
constructor Create; override;
destructor Destroy; override;
class function CreateAndInitialize: TProgrammer;
end;
// Implementation code…
var
Marshaller: TJSONMarshal;
UnMarshaller: TJSONUnMarshal;
Programmer: TProgrammer;
Value, JSONProgrammer: TJSONObject;
begin
Marshaller := TJSONMarshal.Create(TJSONConverter.Create);
try
Marshaller.RegisterConverter(TProgrammer, 'BornDate', ISODateTimeConverter);
Marshaller.RegisterConverter(TStringList, StringListConverter);
Marshaller.RegisterConverter(TProgrammer, 'Laptops', LaptopListConverter);
Programmer := TProgrammer.CreateAndInitialize;
try
Value := Marshaller.Marshal(Programmer) as TJSONObject;
finally
Programmer.Free;
end;
// UnMarshalling Programmer
UnMarshaller := TJSONUnMarshal.Create;
try
UnMarshaller.RegisterReverter(TProgrammer, 'BornDate', ISODateTimeReverter);
UnMarshaller.RegisterReverter(TStringList, StringListReverter);
UnMarshaller.RegisterReverter(TProgrammer, 'Laptops', LaptopListReverter);
Programmer := UnMarshaller.Unmarshal(Value) as TProgrammer;
try
Assert('Daniele' = Programmer.FirstName);
Assert('Teti' = Programmer.LastName);
Assert(29 = Programmer.Age);
Assert(EncodeDate(1979, 11, 4) = Programmer.BornDate);
Assert(3 = Programmer.Phones.Count);
Assert('NUMBER01' = Programmer.Phones[0]);
Assert('NUMBER02' = Programmer.Phones[1]);
Assert('NUMBER03' = Programmer.Phones[2]);
Assert('HP Presario C700' = Programmer.Laptops[0].Model);
Assert(1000 = Programmer.Laptops[0].Price);
Assert('Toshiba Satellite Pro' = Programmer.Laptops[1].Model);
Assert(800 = Programmer.Laptops[1].Price);
Assert('IBM Travelmate 500' = Programmer.Laptops[2].Model);
Assert(1300 = Programmer.Laptops[2].Price);
finally
Programmer.Free;
end;
finally
UnMarshaller.Free;
end;
finally
Marshaller.Free;
end;
end;

Unit CustomConverter.pas contains all needed Converters/Reverts as anon methods.

unit CustomConverter;

interface
uses
DBXJSONReflect,
MyObjects; //Needed by converter and reverter for TLaptops
var
ISODateTimeConverter: TStringConverter;
ISODateTimeReverter: TStringReverter;
StringListConverter: TTypeStringsConverter;
StringListReverter: TTypeStringsReverter;
LaptopListConverter: TObjectsConverter;
LaptopListReverter: TObjectsReverter;
implementation
uses
SysUtils, RTTI, DateUtils, Classes;
initialization
LaptopListConverter := function(Data: TObject; Field: String): TListOfObjects
var
Laptops: TLaptops;
i: integer;
begin
Laptops := TProgrammer(Data).Laptops;
SetLength(Result, Laptops.Count);
if Laptops.Count > 0 then
for I := 0 to Laptops.Count - 1 do
Result[I] := Laptops[i];
end;
LaptopListReverter := procedure(Data: TObject; Field: String; Args: TListOfObjects)
var
obj: TObject;
Laptops: TLaptops;
Laptop: TLaptop;
i: integer;
begin
Laptops := TProgrammer(Data).Laptops;
Laptops.Clear;
for obj in Args do
begin
laptop := obj as TLaptop;
Laptops.Add(TLaptop.Create(laptop.Model, laptop.Price));
end;
end;
StringListConverter := function(Data: TObject): TListOfStrings
var
i, count: integer;
begin
count := TStringList(Data).count;
SetLength(Result, count);
for i := 0 to count - 1 do
Result[i] := TStringList(Data)[i];
end;
StringListReverter := function(Data: TListOfStrings): TObject
var
StrList: TStringList;
Str: string;
begin
StrList := TStringList.Create;
for Str in Data do
StrList.Add(Str);
Result := StrList;
end;
ISODateTimeConverter := function(Data: TObject; Field: string): string
var
ctx: TRttiContext; date : TDateTime;
begin
date := ctx.GetType(Data.ClassType).GetField(Field).GetValue(Data).AsType<TDateTime>;
Result := FormatDateTime('yyyy-mm-dd hh:nn:ss', date);
end;
ISODateTimeReverter := procedure(Data: TObject; Field: string; Arg: string)
var
ctx: TRttiContext;
datetime :
TDateTime;
begin
datetime := EncodeDateTime(StrToInt(Copy(Arg, 1, 4)), StrToInt(Copy(Arg, 6, 2)), StrToInt(Copy(Arg, 9, 2)), StrToInt
(Copy(Arg, 12, 2)), StrToInt(Copy(Arg, 15, 2)), StrToInt(Copy(Arg, 18, 2)), 0);
ctx.GetType(Data.ClassType).GetField(Field).SetValue(Data, datetime);
end;
end.

Last hint…
Every serialization/unserialization process can create “warnings”.
Those warnings are collected into the “Warnings” property of the Ser/UnSer Object.

Conclusions
In this post I tried to introduce the basics of the new serialization engine in Delphi 2010.
During the next ITDevCon to be held in Italy next November 11.12, I’ll have a talk in which I will extensively talk about serialization and RTTI.
All interested smart developers are invited

ITALIAN P.S.
Se qualche programmatore italiano volesse avere la versione in italiano di questo post può lasciare un commento e vedrò di accontentarlo

You can find the DUnit project Source Code

Introduction to Language Integrated Query with Delphi Prism, Part 2

2009-07-30T21:45:00Z

In part 1 of this 2 part series I provided you with a basic introduction to Language Integrated Query, or LINQ. In this second part I take a look at several specific implementations of LINQ, including LINQ to Objects, LINQ to DataSets, and LINQ to XML. I also have an opportunity to introduce iterator methods, special methods that you implement to return a sequence for use by LINQ queries.

LINQ to Objects

LINQ to Objects is any version of LINQ that does not employ a LINQ provider. Consequently, all LINQ queries shown in part 1 of this series are examples of LINQ to Objects.

Here is another example.

var Customers := GetCustomers;
var query := from c in Customers
  where (c.Age > 10)
  and (c.Age < 40)
  and (c.Active = true<) select c;
for each m in query do
  ResultsList.Items.Add(
    String.Format("Name: {0}, Age={1}",
                  [m.Name, m.Age]));

Note: If you had wanted to create a query that produced a sequence of objects that only included the Name and Age members, you could have used the technique shown in part 1 where the select clause returned a newly created object (which was an anonymous object in that example, but could have been any compatible type).

In the preceding query, a sequence of Customer objects is returned by the GetCustomers function. This particular function is an example of an iterator method. Iterator methods are discussed in the following section.

Granted, this is a simple example. However, there are many different methods in the .NET framework class library that return arrays, collections, and other queryable data sources. As a result, it is possible to use LINQ to perform a wide variety of useful tasks.

Examples of some of these uses can be found in the Windows SDK for .NET 3.5. For example, there are a number of examples of how to use LINQ to explore files and directories on the local file system. This is possible because the System.IO.Directory.GetFiles method returns an array of strings containing the names of the files in a given directory. Arrays, as you’ve seen, can be used in LINQ queries.

Interator Methods and Yield Statements

Iterators are special methods that return a sequence. You declare an iterator by including the iterator keyword in the method signature. Each element in the sequence is specifically returned by the iterator through a call to yield.

This is demonstrated in the following code sample. First, the iterator is declared using the following statement.

method GetLowDigits: sequence of Integer; iterator;

Next, the implementation of the iterator includes one or more calls to yield, which, like exit, can be used to return a value (though yield does not cause a return from the method in the same way that exit does).

method MainForm.GetLowDigits: sequenbce of Integer;
begin
for i: Integer := 1 to 9 do
  yield i;
end;

The following code demonstrates the use of this iterator.

var numbers := GetLowDigits;
for each n in numbers do
  MessageBox.Show(n.ToString);

Iterator methods are not executed when the sequence is assigned to a sequence reference. Specifically, the var declaration and initialization in the preceding code segment did not cause the GetLowDigits iterator to execute. Instead, it was the specific iteration over the sequence in the for each loop that caused the iterator to be executed. Furthermore, the iterator was not executed all at once. Instead, the iterator is executed up until it returns the first element of the sequence, that element is then used by the for each loop, which will then call back to the iterator to get the next value in the sequence, and so on.

The yield keyword can be used to return either a single element of the sequence or a reference to a sequence (which could be an array, a method that returns an array, or other similar reference).

Here is another example of an iterator and a sequence. This code requires a type declaration, which looks like the following.

Customer = class
private
  FName: String;
  FAge: Integer;
public
  property Name: String read FName write FName;
  property Age: Integer read fAge write FAge;
end;

Here is the declaration of the iterator.

method GetCustomers: sequence of Customer; iterator;

And here is the implementation of the iterator.

method MainForm.GetCustomers: sequence of Customer;
begin
  yield new Customer(Name:= 'Allan', Age := 10);
  yield new Customer(Name := 'Bob', Age := 25);
  yield new Customer(Name := 'Craig', Age := 18);
end;

Now that you understand the iterator, it is easy to see that the following LINQ query, which was introduced in the preceding section, populates the listbox named ResultList with the values Name: Bob, Age=25 and Name: Craig, Age=18.

var Customers := GetCustomers;
var query := from c in Customers
where (c.Age > 10) and (c.Age < 40) and (c.Active = true<) select c;
for each m in query do
ResultsList.Items.Add(
String.Format("Name: {0}, Age={1}", [m.Name, m.Age]));

The implementation of the iterator in this case used the named parameters feature of Delphi Prism’s nameless constructors to return each of the customers (which in reality would have probably been populated with data from an external source, such as a database). Also, the Customers type declaration used type inference to determine that Customers variable was a sequence of Customer.

What is LINQ to DataSets

LINQ to DataSets is a LINQ provider that permits you to use LINQ queries against DataTables, DataRows, and other classes in the ADO.NET framework. In order to use LINQ to DataSet, you must add the System.Data.DataSetExtensions assembly to the references section of your Delphi Prism project.

This section is designed to provide you with a brief introduction to LINQ to DataSet. As such, it does not go into ADO.NET, which is the data connectivity framework in .NET.

Note: LINQ to DataSet should really probably be called LINQ to DataTable or LINQ to DataRows, or even LINQ to ADO.NET. None of the operations in LINQ to DataSet are performed on instances of the DataSet class. Instead, they are performed on DataTables and DataRows.

As you learned earlier in this article, LINQ queries can be performed on any object that implements the IEnumerable<T> interface. DataTables (and DataRows) do not implement IEnumerable. However, they both have extension methods that support IEnumerable. These are defined in the DataTableExtensions class, which is located in the System.Data.DataSetExtensions namespace (which is why you must add this assembly to your project references folder).

There are only three extension methods in the DataSetExtensions class. These are AsDataView, AsEnumerable, and CopyToDataTable. For DataTables, the key method is AsEnumerable, which returns an enumerable collection of DataRows (records).

This is demonstrated in the following segment code.

method MainForm.LINQToDataSet_Click(sender: System.Object;
  e: System.EventArgs);
var
  Connection: DataStoreConnection;
  Adapter: DataStoreDataAdapter;
  DataTable1: DataTable;
begin
  Connection := new DataStoreConnection(conStr);
  Connection.Open;
  try
    Adapter := new DataStoreDataAdapter(
      'SELECT * FROM   CUSTOMER', connection);
    DataTable1 := new DataTable;
    DataTable1.TableName := 'Customer';
    Adapter.Fill(DataTable1);
    var query :=
      from cust in DataTable1.AsEnumerable
      where cust['ON_HOLD'].Equals(DbNull.Value)
      and (cust.Field<string>('STATE_PROVINCE') = 'CA');
    for each c in query do
    begin
      ResultsList.Items.Add(
        String.Format('Customer Number: {0}' +
        ' Company Name: {1}', c['Cust_No'].ToString,
        c['Customer'].ToString));
    end;
  finally
    Connection.Close;
  end;
end;

Once the DataTable has been populated with data from the SQL query, the AsEnumerable method of the DataTable is called, and this object is used in the LINQ query. Interestingly enough, the object returned from the AsEnumerable method is a generic IEnumerable<T>, where the type is a collection of DataRows.

As you consider this code segment, you might conclude that you would be better off using a SQL statement that performs the filtering of the results, rather than use a LINQ query. While that may be true in some instances, if you want to load a DataTable once, and perform numerous queries on the returned value, the LINQ query offers performance benefits. (Note that another alternative is to use a DataView to filter and sort the DataTable results.)

Note that in addition to LINQ to DataSet, the .NET framework provides the LINQ to SQL provider. Unlike LINQ to DataSet, which can be used against any database for which there is a .NET data provider, LINQ to SQL is specifically designed to work only with Microsoft SQL Server. LINQ to Entities is another LINQ provider that can be used against supported databases.

What is LINQ to XML

LINQ to XML provides a programming model, similar to XPath, with which you can create, edit, query, and save XML documents. LINQ to XML provides you an alternative to using the XML DOM (document object model), which is what is used by the XMLDocument class in the .NET framework. Like the DOM, LINQ to XML works with the XML document in memory. From there you can read, query, and write data to the XML document, after which you can stream it to a service or write it to a file, if you desire.

As is the case with the LINQ to DataSet topic that precedes this section, this discussion is designed to provide you with a basic introduction to LINQ to XML. It does not, however, cover general XML issues.

In fact, LINQ to XML is far more involved than LINQ to DataSet. This is because LINQ to XML not only supports features necessary for querying, but as mentioned earlier, an entire programming model for working with XML. As a result, this section will cover just enough information to get you started.

Unlike LINQ to DataSet, where most of the functionality is found in extension methods added to DataTable and DataRow, LINQ to XML, found in the System.Xml.Linq namespace, includes a large number of concrete classes that you can use to work with XML documents.

Of these classes, there are three that you are likely to use most often. These are XDocument and XElement, which descend from XNode, and XAttribute, which descends from XObject. (XNode, by the way, also descends from XObject.)

You can create a valid XML document using either XDocument or XElement, though XDocument provides a little more support for this. Specifically, there are certain advanced features of XML documents that you can access through the XDocument class which are unavailable through the XElement class. XAttribute, by comparison, is used to define attributes of XML elements.

But before you get started, at a minimum you will need to add the System.Xml.Linq namespace to your uses clause. In addition, you may have to add one or more of the following additional namespaces, depending on what features your code will use: System.Xml, System.Xml.Schema, System.Xml.XPath, and System.Xml.Xsl.

Creating XML Documents using LINQ to XML

As mentioned previously, you can create XML documents using either the XDocument or the XElement class. The following example demonstrates how to create an XML document using the XElement class.

element := new XElement('customers',
               new XElement('customer',
                 new XAttribute('custno', 1001),
                 new XElement('name', 'John Doe'),
                 new XElement('address',
                   '101 Broadway Avenue'),
                 new XElement('city', 'New York'),
                 new XElement('state', 'NY'),
                 new XElement('zip',
                   '00123')                                            ),
              new XElement('customer',
                  new XAttribute('custno', 1002),
                  new XElement('name', 'John Doe'),
                  new XElement('address',
                    '1001 Main Street'),
                  new XElement('city', 'Los Angeles'),
                  new XElement('state', 'CA'),
                  new XElement('zip', '90123')
              )
          );
element.Save(XmlFileName);

This code creates an XML file with a root element named customers. The root element has two child elements named customer. Each child element has one attribute and four child elements. The following is how the XML file created by this code looks.

<?xml version="1.0" encoding="utf-8"?>
<customers>
 <customer custno="1001">
  <name>John Doe</name>
  <address>101 Broadway Avenue</address>
  <city>New York</city>
  <state>NY</state>
  <zip>00123</zip>
 </customer>
 <customer custno="1002">
  <name>John Doe</name>
  <address>1001 Main Street</address>
  <city>Los Angeles</city>
  <state>CA</state>
  <zip>90123</zip>
 </customer>
</customers>

Creating this same file using an XDocument looks similar, though there are differences.

The documentation for LINQ with XML includes a large number of demonstrations of XML definition using declarations like the one provided previously. These declarations, however, are anything but flexible. In other words, they always create the same XML file, which is rarely useful.

The following example, which obtains its data from an ADO.NET data reader, produces an XML file similar in structure to the preceding one, except that its data is entirely based on the results of a SQL query.

Connection := new DataStoreConnection(conStr);
Connection.Open;
try
  Command := Connection.CreateCommand;
  Command.CommandText := 'select cust_no, customer, ' +
    'address_line1, city, state_province, ' +
    'postal_code from customer';
  DataReader := Command.ExecuteReader;
  try  
    document := new XDocument(
      new XElement('customers'));
    while DataReader.Read do
    begin
      attribute := new XAttribute('custno',
        DataReader.GetString(0));
      element :=
        new XElement('customer',
          new XElement('name', DataReader.GetString(1)),
          new XElement('address',
            DataReader.GetString(2)),
          new XElement('city', DataReader.GetString(3)),
          new XElement('state',
            DataReader.GetString(4)),
          new XElement('zip',
            DataReader.GetString(5))
        );
      element.Add(attribute);
      document.Root.Add(element);                
    end;
  finally
    DataReader.Close;
  end;
  document.Save(XmlFileName);
finally
  connection.Close;
end;

Querying with LINQ to XML

Querying using LINQ to XML is similar to other LINQ queries, though there are classes and methods that you use in LINQ to XML that are not found in other LINQ technologies. In short, once you have a reference to a queryable object (an IEnumerable, a sequence, or other similar construct), you use query statements to retrieve the data you are interested in. The following example demonstrates a LINQ to XML query.

element := XElement.Load(XmlFileName);
var childList :=
  from el in element.Elements
  where String(el.Element('state')) = 'CA'
  order by String(el.Attribute('custno')) desc
  select new XElement('document',
    new XElement('CustomerName', el.Element('name')) ,
      new XElement('Address', el.Element('address')),
      new XElement('CityStateZip',
        el.Element('city').Value + ', ' +
        el.Element('state').Value + ' ' +
        el.Element('zip').Value));
for each e in childList do
begin
  ResultsList.Items.Add(
   e.Element("CustomerName").Value);
  ResultsList.Items.Add(e.Element("Address").Value);
  ResultsList.Items.Add(
    e.Element('CityStateZip').Value);
  ResultsList.Items.Add(String.Empty);
end;

This query returns a sequence of XElement references, which have a structure similar to, though different than, the original XElements. For example, the new XElements are not named customer, they are named document. Furthermore, there are no attributes in the new XElements, and there are only three child nodes.

The preceding example demonstrated a query using the XElement class. The following code shows the same basic query, however an XDocument is used in this example.

document := XDocument.Load(XmlFileName);
var childList :=
  from el in document.Descendants
  where String(el.Element('state')) = 'CA'
  order by String(el.Attribute('custno')) desc
  select new XElement('document',
    new XElement('CustomerName',
      el.Element('name')),
    new XElement('Address', el.Element('address')),
    new XElement('CityStateZip',
      el.Element('city').Value + ', ' +
      el.Element('state').Value + ', ' +
      el.Element('zip').Value));
for each e in childList do
begin
  ResultsList.Items.Add(
    e.Element("CustomerName").Value);
  ResultsList.Items.Add(e.Element("Address").Value);
  ResultsList.Items.Add(
    e.Element('CityStateZip').Value);
  ResultsList.Items.Add(String.Empty);
end;

Modifying Exiting XML using LINQ to XML

Once XML is in memory, you can add elements or attributes, remove elements or attributes, as well as modify the data for elements or attributes. If you load the XML from an existing source, such as a file, make modifications, and then save the XML back to the original source, you have effectively changed the XML.

This is demonstrated in the following code.

document := XDocument.Load(XmlFileName);
var childList :=
  from el in document.Descendants
  where String(el.Element('state')) = 'CA'
  order by String(el.Attribute('custno')) desc
  select el;
//make changes
for each e in childList do
begin
  e.Element('state').SetValue('California');
end;
//save the changes
document.Save(XmlFileName);

There is an important characteristic of the preceding code that you should note. Specifically, the select clause of the LINQ to XML query returned a sequence of XElements from the original XDocument. While these represent a possible subset of all XElements returned by the Descendants method of the XDocument, they are still child elements of the original XDocument.

When changes are made to the items in the sequence returned by the query, those changes are actually being made to the corresponding XElements in the XDocument. This is why the changes are preserved when the subsequent call to XDocument.Save is made.

By comparison, if the select clause had created new XElements, similar to how the last query in the preceding section demonstrated, the subsequent changes would have been performed on the newly created objects, which are not child elements of the XDocument.

Summary

This two part series has provided you with a brief introduction to LINQ with Delphi Prism. Here you have learned how to construct LINQ Queries, the nature of LINQ to Objects, define and implement iterators, and have been given a preview of LINQ to DataSets and LINQ to XML.

Introduction to Language Integrated Query with Delphi Prism: Part 1

2009-09-03T11:26:20Z

Language Integrated Query, or LINQ (pronounced link), is a declarative programming language developed by Microsoft for the .NET framework. In a declarative programming language you specify what result you want to achieve without needing to specify how those results are produced.

SQL SELECT statements are an example of declarative programming. With a given SQL statement you specify what data you want returned. It is up to the SQL engine to determine how that data is derived.

Traditional Delphi, on the other hand, is categorized as imperative programming. When writing Delphi code, you generally describe in detailed terms how you want the result obtained. You do this using control structures, expressions, and explicit calls to functions and procedures.

This article is designed to provide you with a general overview of using LINQ with Delphi Prism, the latest .NET development tool from Embarcadero Technologies and RemObjects. For a more detailed discussion of LINQ and its related topics, refer to the latest version of the .NET framework SDK.

Overview of LINQ

Language Integrated Query comes in a variety of different flavors, depending on the version of the .NET framework you are compiling against. For example, there is LINQ to Objects, LINQ to DataSets, LINQ to SQL, and LINQ to XML. Each of these technologies, which were introduced in .NET 3.0, provides classes you can use in conjunction with LINQ to work with objects in various domains of the .NET framework.

In addition, some of the major new features that are emerging in .NET also are LINQ related. One example is LINQ to Entities, which is part of the Entity Framework. The Entity Framework is one of the latest database technologies being promoted by Microsoft.

Turning our attention to the general case of LINQ, LINQ queries can be divided into three basic parts. These are

A queryable source of data
A query definition
Query execution

The following sections describe each of these parts in greater detail.

Queryable Sources of Data

In strictly .NET terms, a queryable source of data is any object that implements the generic IEnumerable<T> interface, which includes List<T> and Dictionary<TKey, TValue>. In Delphi Prism, this also includes any collection that is declared as a sequence, as well as arrays (both fixed and dynamic).

Some of the LINQ technologies, such as LINQ to DataSet and LINQ to XML, are implemented through LINQ providers, which are support classes that enable LINQ operations. These providers usually include special methods that provide access to an IEnumerable<T> reference based on an object that otherwise does not support this interface. For example, LINQ to DataSet provides the AsEnumerable extension method to the DataTable class.

The following variable declaration defines a simple queryable data source. In this case, the data source is a sequence.

var seq: sequence of Integer := [1,2,3,4,5,6,7,8,9];

Query Definitions using Query Syntax

A query definition consists of two parts, a query variable and a query expression. Furthermore, query expressions are defined using one of two techniques. The first is called query syntax, and the second is called method syntax. This section describes query syntax. Method syntax is described later in this article.

Query expressions are defined using a combination of one or more LINQ statements, comparison operators, extension methods, and value references. At a minimum, a LINQ query includes a from clause, which defines an alias and a data source. Consider the following code segment:

var numbers: sequence of Integer := [1,2,3,4,5,6,7,8,9];

var myquery := from c in numbers select c;

The first line of code defines the data source, and the second line defines the query. The query in this case selects all items from the sequence numbers, as can be seen in the following figure (the code that populates the list box has not been shown yet).

Figure 1. The LINQExamples project

The c in the preceding query is an alias, and it is used to reference items in the numbers sequence. The select part of the query, which contrary to SQL standards, appears at the end of the statement, defines what is returned by the query. In this case, the query returns each of the items in the sequence.

The preceding query really just creates another sequence that is no different from the one that it queried. Most LINQ queries, by comparison, either select subsets of the data, performs data transformations, or other similar operations. The following query includes a where clause that limits the myquery sequence to those numbers between 4 and 8, inclusively. As you can see, the alias is essential for performing this operation.

var numbers: sequence of Integer := [1,2,3,4,5,6,7,8,9];
var myquery := from c in numbers where (c >= 4) and (c <= 8) select c;

In both queries shown so far in this section, the select clause could have been omitted. Specifically, if what we are returning is exactly the same type of item as in the sequence, we can omit the select clause altogether. In other words, omitting the select clause is somewhat similar to a SELECT * clause in a SQL query. The following query, which does not have a select clause, produces the same result as the preceding one.

var numbers: sequence of Integer := [1,2,3,4,5,6,7,8,9];
var myquery := from c in numbers where (c >= 4) and (c <= 8);

The select clause in LINQ is required when you are querying objects with one or more members, and want to only return a subset of its members, or want to return transformed data. For example, the following query returns a sequence of objects whose values are equal to the square of the original values. The objects returned by the select clause, in this case, are anonymous types. (An anonymous type of an object whose type is never explicitly declared, one of many advanced language features supported by Delphi Prism.)

var numbers: sequence of Integer := [1,2,3,4,5,6,7,8,9];
var myquery := from c in numbers select new class(Value := c * c);

LINQ queries can be relatively complicated. In addition to the from, where, and select keywords shown here, Delphi Prism also supports the following LINQ operations: order by, group by, join, with, take, skip, reverse, and distinct. In addition, LINQ queries can make use of lambda expressions, both in the where and select parts (Lambda expressions are discussed briefly later in this article). Finally, similar to SQL, LINQ queries can include subqueries.

Executing LINQ Queries

Defining a query and executing a query are two distinct steps (though as you will learn shortly, Delphi Prism provides a mechanism for combining these operations). Specifically, the query defined in the preceding section identifies what the query will do, but does not execute the query. In LINQ this is referred to as deferred execution.

You cause the query to be executed by iterating over it using a for each loop. Each execution of the loop brings back another instance of whatever the query returns. For example, if you used a for each loop to iterate over the query shown in the preceding example, each iteration of the loop would return a different instance of the anonymous type.

The following is the entire code sequence (short as it is), which shows how the Value member of the returned objects are assigned to the list box (which is name ResultList in this example).

ResultsList.Items.Clear;
var numbers: sequence of Integer := [1,2,3,4,5,6,7,8,9];
var myquery := from c in numbers select new class(Value := c * c);
for each m in myquery do
ResultsList.Items.Add(m.Value.ToString);

When written to the list box, this query result looks like the following.

Figure 2. Data from an anonymous type returns from a LINQ query

Similar to the alias in the query definition, the for each loop defines a local variable that holds the value of the item that is returned in each iteration. In the preceding code, this variable is named m. In most cases, Delphi Prism uses type inference to determine the type of this variable (which is fortunate in this case, since we used an anonymous type).

Not all LINQ queries use deferred execution. To cause the results of the query to retrieve immediately, you can use either the IEnumerable methods ToList<TSource> or ToArray<TSource>. An example of this is shown in the following code.

var numbers: sequence of Integer := [1,2,3,4,5,6,7,8,9];
var MyResults: Array of Integer :=
(from c in numbers where (c >= 4) and (c <= 8)).ToArray();
for each val in MyResults do
ResultsList.Items.Add(val.ToString);

Similarly, if you use any of the grouping operations, such Average, Count, First, and so forth, the query is executed immediately. This is demonstrated in the following code.

var numbers: sequence of Integer := [1,2,3,4,5,6,7,8,9];
var MyResults: Double := (from c in numbers).Average();
MessageBox.Show(MyResults.ToString);

LINQ Query Method Syntax

As mentioned earlier, LINQ query expressions can be defined using both query syntax and method syntax. Method syntax is implemented through extension methods on the System.Linq.Enumerable class, and corresponds roughly to the LINQ query syntax operators.

The truth is that when you use query syntax in your query expressions, the compiler converts these into method syntax. Another way to put this is that any query that you can write using query syntax can also be defined using method syntax. Interestingly, the opposite is not true. Specifically, there are some queries that you must define using method syntax, as there is no equivalent in query syntax.

Unlike in query syntax, there are no aliases in method syntax. Instead, the methods are called on your IEnumerable object directly, using dot notation. These operations are typically performed in a single expression. In other words, using dot notation, method syntax sometimes includes a number of method calls, implemented through a chain of calls.

This is demonstrated in the following code segment, which produces a query identical to the first one listed in the preceding section “Query Definitions using Query Syntax.”

var myquery := numbers.where(c -> c >= 4).
where(c -> c <= 8).orderby(c -> c).select(c -> c);

The preceding statement is a single statement, even though it has been wrapped onto two lines due to the limits of the column space in this article. Fortunately, this wrapped form can still be compiled by Delphi since its compiler is designed to ignore white space. Also, the arguments of the where, orderby, and select clauses are lambda expressions. Lambda expressions are discussed in the following section.

As with the query expressions, the select method call here is not necessary, since there is no transformation being performed on the returned items.

The syntax is a little different when your sequence contains more complex data. This can be seen from the following code, which is the method syntax version of the code segment described in the following section “LINQ to Objects.”

var Customers := GetCustomers;
var query := Customers.Where(Customer -> Customer.Age > 10).
Where(Customer -> Customer.Age < 40).
Where(Customer -> Customer.Active);

Lambda Expressions

Lambda expressions are anonymous functions that can accept zero or more parameters, and typically return a value. The parameters appear on the left-hand side of the lambda operator, and are enclosed in parentheses (unless there are zero or one parameters, in which case the parentheses are optional).

The parameters, if present, are following by the lambda operator, which is the -> character sequence in Delphi Prism (C# uses => as the lambda operator). When reading a lambda expression aloud, the lambda operator is spoken “goes to.” As a result, the first expression in the preceding query expression is read “customer goes to customer dot age greater than 10.”

The expression on the right-hand side of the lambda operator includes either an expression or a statement. This is the value that is returned by the lambda expression (if the statement is an expression lambda). It is possible, however, to include an expression block that does not resolve to an expression. These lambdas are called statement lambdas. Statement lambdas should not be used in method calls when using method syntax.

In most cases, lambda expressions take advantage of the compiler’s ability to infer the type of the input parameters.

Summary

This article has provided you with a brief introduction to the composition of language integrated queries. In part 2 of this article, which will appear within a month of this article, numerous examples of LINQ queries are shown, including technologies such as LINQ to Objects, LINQ to DataSets, and LINQ to XML.