Midnight Researcher Notes

JSON–yet another tutorial :)

noreply@blogger.com (Ebeid Soliman) — Thu, 01 Dec 2011 23:43:00 +0000

What is JSON?

JSON (or JavaScript Object Notation) is a highly portable data interchange format. While its structure is recognized natively by Javascript (as it is Javascript), its formatting conventions are easily recognized by other C-like languages.

JSON is built on two structures:

A collection of name/value pairs. In various languages, this is realized as an object, record, struct, dictionary, hash table, keyed list, or associative array. Example:

var jason = {
 "age" : "28",
 "hometown" : "Cairo, Egypt",
 "gender" : "male"
 };

The object created in the example could be accessed like any other object.

An ordered list of values. In most languages, this is realized as an array, vector, list, or sequence. Example,

var family = [{
 "name" : "Jason",
 "age" : "24",
 "gender" : "male"
 },
 {
 "name" : "Kyle",
 "age" : "21",
 "gender" : "male"
 }];

This array could be accessed like any array of objects.

Since JSON and XML are both used for data exchange, you may find yourself in the middle of deciding which format to use.

Reasons to choose JSON over XML

JSON is easier to read than XML (especially of large amounts of data)
JSON requires less tags than XML – XML items must be wrapped in open and close tags whereas JSON you just name the tag once
Because JSON is transportation-independent, you can just bypass the XMLHttpRequest object for getting your data.
JavaScript is not just data – you can also put methods and all sorts of goodies in JSON format.
JSON is better at helping procedural decisions in your JavaScript based on objects and their values (or methods).
You can get JSON data from anywhere, not just your own domain.

Reasons to choose XML over JSON

Easy to take XML and apply XSLT to make XHTML.
XML is supported by many more desktop applications than JSON.
XML can put JSON in it on the way back to the client.
AJAX includes XML in it and not JSON.

.Net Support for JSON

.Net framework provides System.Web.Script.Serialization.JavaScriptSerializer to serialize and deserialize objects. Here is a simple example:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
namespace JSON_Demo1
{
  public partial class Page1 : System.Web.UI.Page
  {
      protected void Page_Load(object sender, EventArgs e)
      {
          Employee oEmployee1 = new Employee { Name = "Emp1", ID = "11", Age = "30" };
          Employee oEmployee2 = new Employee { Name = "Emp2", ID = "22", Age = "31" };
          Employee oEmployee3 = new Employee { Name = "Emp3", ID = "33", Age = "20" };
          List oList = new List() { oEmployee1, oEmployee2, oEmployee3 };
          System.Web.Script.Serialization.JavaScriptSerializer oSerializer = new System.Web.Script.Serialization.JavaScriptSerializer();
          string sJSON = oSerializer.Serialize(oList);
          Response.Write(sJSON);
      }
  }
  public class Employee
  {
      public string Name { get; set; }
      public string Age { get; set; }
      public string ID { get; set; }
  }
}

When you run the above application, you will see the JSON array on the response like so:

Now let’s change this example a little bit so we can make Ajax call to get the JSON data and parse it using javaScript. Change the code-behind to look like:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Web.Services;

namespace JSON_Demo1
{
  public partial class Page1 : System.Web.UI.Page
  {
      protected void Page_Load(object sender, EventArgs e)
      {
          Response.Write(GetJSONData());
      }

      [WebMethod()]
      public static string GetJSONData()
      {
          Employee oEmployee1 = new Employee { Name = "Emp1", ID = "11", Age = "30" };
          Employee oEmployee2 = new Employee { Name = "Emp2", ID = "22", Age = "31" };
          Employee oEmployee3 = new Employee { Name = "Emp3", ID = "33", Age = "20" };
          List oList = new List() { oEmployee1, oEmployee2, oEmployee3 };
          System.Web.Script.Serialization.JavaScriptSerializer oSerializer = new System.Web.Script.Serialization.JavaScriptSerializer();
          return oSerializer.Serialize(oList);
      }
  }
  public class Employee
  {
      public string Name { get; set; }
      public string Age { get; set; }
      public string ID { get; set; }
  }

}

now change the .aspx page to look like:

<%@ Page Language="C#" AutoEventWireup="true" CodeBehind="Page1.aspx.cs" Inherits="JSON_Demo1.Page1" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
  <title></title>
  <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js" type="text/javascript"></script>
  <script type="text/javascript">
      $(document).ready(function () {
          $.ajax({
              type: "POST",
              url: "Page1.aspx/GetJSONData",
              contentType: "application/json; charset=utf-8",
              dataType: "json",
              success: function (msg) {
                  BuildTable(msg.d);
                  }
          });
      });
    
      function BuildTable(msg) {
          var table = '<table> <thead> <tr> <th>Name</th> <th>ID</th> <th>Age</th> </thead> <tbody>';
          var myJSONObject = eval( msg );
          for (var i = 0; i < 3; i++) {              
              var row = '<tr>';
              row += '<td>' + myJSONObject[i].Name + '</td>';
              row += '<td>' + myJSONObject[i].ID + '</td>';
              row += '<td>' + myJSONObject[i].Age + '</td>';
              row += '</tr>';
              table += row;
          }
          table += '</tbody></table>';
          $('#Container').html(table);
      }
      </script>
</head>
<body>
  <div id="Container">
  </div>
</body>
</html>

Now let’s run the application and see the output:

In the previous example, we created a WebMethod, then call it from jQuery and pass the output of it to JavaScript method BuildTable(msg). This method convert the string msg into a JSON array of objects and then loop on that array to build the shown simple data table. We also rendered the JSON data to the browser directly from server-side code, just to show the original data and how it transformed using JSON.

In this post we gave a brief background about JSON, when use it and when use XML to send data, how .Net provides native serializer for JSON, and how to deserialize it using JavaScript. We may look into more advanced topics regarding JSON in future posts.

Good Luck

An introduction to Microsoft Message Queuing–Part 2

noreply@blogger.com (Ebeid Soliman) — Tue, 22 Nov 2011 17:52:00 +0000

In the previous post, we explored very basic concepts of MSMQ. We used non-transactional queue to send and receive a message. In this post we will try to dive more into Message Queuing Queues.

Simply, queues are logical containers that Message Queuing uses to store and later forward messages, providing the bases for the loosely coupled aspects of Message Queuing. Queues can be created by applications and by Message Queuing. Queues created by applications or by users in an MMC snap-in are referred to as application queues. Queues created by Message Queuing are referred to as system-generated queues (we will talk about system- generated queues in another post).

Depending on the service provided by the queue, application queues can be public or private, and they can be transactional or nontransactional.

Application queues can play any of the following roles:

Destination queues : any queue used to send/receive messages between applications.
Administration queues : used for acknowledgment messages returned by Message Queuing or connector applications.
Response queues : used by receiving applications to return response messages to the sending application.
Report queues : used to store report messages returned by Message Queuing.

Destination Queues

Destination queues are any queue used to send/receive messages between applications. The sending application on computer 1 sends the messages to the queue and the receiving application on computer 2 reads the messages from the queue. Typically, destination queues are located on the same computer as the receiving application in order to minimize network traffic.

Public Versus Private Queues The decision to use public or private destination queues depends primarily on whether you want other applications to be able to locate the queues or not. Message Queuing registers private queues locally by storing a description of each queue in a separate file in the local queue storage (LQS) folder on the local computer (the default Lqs folder is %windir%\System32\MSMQ\Storage\Lqs). Also a description of each public queue created on the local computer is also stored locally in a separate file in the Lqs folder.

Adv. of Public Queues:

Registered in the directory service, so it can be located by other Message Queuing applications.
Persistent and its registration information can be backed up.

Adv. of Private Queues:

Requires minimal directory service overhead(faster to create, no latency, and no replication)
Can be created and deleted when the directory service is not working
Can be accessed directly by name without query the directory service.

Div. of Private Queues:

It is registered on local computer, so it is properties cannot be obtained by Message Queuing applications running on remote computers. Private queues can be exposed to other applications by making the location of the private queue known to the applications. To distribute the location of a private queue, an application can send the format name of the private queue as the response queue property of a message.

Transactional Versus Nontransactional Queues The decision to use transactional or nontransactional queues is based on whether the applications accessing the queue will be sending and receiving messages within the context of a transaction. So, what is a transaction ? a transaction is to execute a multiple-steps process such that either all or none of the steps will complete. In reality, transactions are handled by rolling back any steps that have already occurred if the entire transaction is not completed successfully.

When sending messages, only transactional queues can accept messages sent in the context of a transaction. These messages are also referred to as transactional messages. In a similar way, nontransactional queues can only accept messages sent outside the context of a transaction. Note that only transactional messages are delivered with exactly-once-delivery (EOD) guarantees.

When receiving messages, only local transactional queues can be accessed within the context of a transaction. The transactional queue must be local to the receiving application. On the other hand, nontransactional queues can be accessed within or outside of a transaction. Also, transactional queues, local or remote, can be accessed outside of a transaction (because we ask it to do less than what it can do).

If you just want to be able to recover lost messages, don’t use transactional queues. You can set the Recoverable property of a every message you sent. Or you can sent the queue property DefaultPropertiesToSend.Recoverable to true.

Creating a Transactional Queue

Through the snap in : just check the check-box Transactional below the queue name textbox.
Programmatically : just like privateQueue = MessageQueue.Create(".\\Private$\\privateQueue", true);

MessageQueue have a property called Transactional that you can check to ensure that the queue is transactional

MSMQ supports two types of transactions: Internal and External.

Internal Transactions

Class MessageQueueTransaction can be used to Begin(), Commit(), Abort() the transaction. It also can be passed through Send() and Receive() methods to that operation falls under a transaction. The class also exposes a Status property to give the transaction status. Transaction status can be one of:

Initialized : The transaction has been initialized but not yet started.
Pending : The transaction has been began but not committed or aborted yet.
Committed : The transaction has been committed.
Aborted : The transaction has been aborted.

Transaction Types when sending or receiving using MessageQueue class through transactional queues, you could pass one of the values in MessageQueueTransactionType enumeration. This specifies how you would like to interact with the queue. These values are:

Single : each queue operation will be doine in a separate internal transaction.
None : enable you to receive a message from a transactional queue, but outside a transaction. It also enables us to send a transactional message to a non-transactional queue.
Automatic : used with external transactions to direct the send or receive operations to use an already existing transaction context.

Now lets compile these pieces into one example to get sense of what internal transactional queues means.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Messaging;

namespace MSMQ_Demo3
{
 class Program
 {
     static void Main(string[] args)
     {
         MessageQueueTransaction mqTran = new MessageQueueTransaction();
         MessageQueue queueA = MessageQueue.Create(".\\Private$\\TranQueueA", true);
         MessageQueue queueB = MessageQueue.Create(".\\Private$\\TranQueueB", true);
         mqTran.Begin();
         try
         {
             queueA.Send("Message A", "Label A", mqTran );
             queueB.Send("Message B", "Label B", mqTran );
             mqTran.Commit();
         }
         catch(Exception ex)
         {
             mqTran.Abort();
             Console.WriteLine(ex.Message);
         }             

         MessageQueue queueC = new MessageQueue(".\\Private$\\TranQueueA");
         MessageQueue queueD = new MessageQueue(".\\Private$\\TranQueueB");
         string strMsg = "";
         mqTran.Begin();
         try
         {
             Message msg = queueC.Receive(mqTran);
             msg.Formatter = new XmlMessageFormatter(new Type[1] { typeof(string) });
             strMsg = msg.Body.ToString();
             msg = queueD.Receive(mqTran);
             msg.Formatter = new XmlMessageFormatter(new Type[1] { typeof(string) });
             strMsg += " \n ";
             strMsg += msg.Body.ToString();
             Console.WriteLine(strMsg);
             mqTran.Commit();
         }
         catch (Exception ex)
         {
             mqTran.Abort();
             Console.WriteLine(ex.Message);
         }
         Console.WriteLine();
     }
  
 }
}

In this example we created two transactional queues, send two messages to them. Then we received these messages. Both the sending and receiving done in a transaction that commits only after the success of all operations. If any errors happened, we abort the whole transaction and rollback all operations done in it.

In future posts we will talk more about MSQM details.

An introduction to Microsoft Message Queuing

noreply@blogger.com (Ebeid Soliman) — Thu, 17 Nov 2011 12:49:00 +0000

What is MSMQ ? Message Queuing (also known as MSMQ) is a messaging infrastructure and a development tool for creating distributed messaging applications for Microsoft Windows operating systems. Applications developed for Message Queuing send messages to queues, which are temporary storage locations, from which messages can proceed to their final destination as conditions permit.

Installing MSMQ The procedures to install MSMQ on Windows Server 2008, Winows Server 2008 R2, Windows 7, Windows Vista, Windows XP, and Windows Server 2003 are available here

Basic Messaging

The image shows the basic messaging scenario. You have two applications: a sending application and a receiving application. A sending application prepares a message and puts it into a queue. A message can be any piece of information that the receiving application understands. The receiving application receives the message from the queue and processes it. One thing to note here is that the sender and the receiver are not tightly coupled and they can work without the other application being online.

Messaging in .Net To build a messaging application we need to create at least three components: queue, application to send messages, and another application to receive these messages.

Types of queues The basic types of MSMQ queues are: private and public. Public queues are those that are published in the active directory. So, applications running on different servers throughout the network can find and use public queues through the active directory. Private queues on the other hand are available locally on a particular machine and are not published in the active directory.

Creating a Queue Queues can be created either programmatically or through the computer management snap-in (if you installed MSMQ on your machine).

Programmatically The System.Messaging.MessageQueue class provides all the necessary functionality to work with and manipulate MSMQ queues. It is like a wrapper around message queuing. MessageQueue.Create(path) creates a non-transactional message queue at the specified path (we will talk about transactional and non-transactional queues later). For public queues, path is MachineName\QueueName. For private queues, MachineName\Private$\QueueName. You can use "." for the local computer. Code should look like:

MessageQueue privateQueue = MessageQueue.Create(".\\Private$\\privateQueue");

Through the computer management snap-in Open the computer management snap-in (right click computer, then manage). Navigate to Services and Applications, then Message Queuing. Right click on Private Queues –> New –> Private Queue. Or right click on public Queues –> New –> Public Queue.

Sending a message Use Send() method of your previously created MessageQueue object like so

privateQueue.Send("Message Body (could be any managed object)", "Message Label");

Receiving a message There are two types of operations with respect to reading a message fom the Queue: Peeking and Receiving. When we receive a message from the queue, the message is removed from the queue. Peeking is similar to Receiving but here the message is not removed from the queue. Code could look like

MessageQueue anotherPrivateQueue = new MessageQueue(".\\Private$\\privateQueue");

System.Messaging.Message msg = anotherPrivateQueue.Receive();

Additional Considerations before writing the first complete messaging application

You have to check if there is already a Message Queuing queue with the same name before creating your new queue. Use MessageQueue.Exists(path) to check for queue existence.

When receiving a message and before accessing its body, you have to set the Formatter property of the Message object. There is a lot of built-in formatters. For now, we can use XMLFormatter.

Now, let’s get things together into a “Hello, World” example. Open Visual Studio, New project, Windows forms. All reference to System.Messaging namespace. Then edit your code to look like so:




using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Messaging;



namespace MSMQ_Demo

{

    class Program

    {

        static void Main(string[] args)

        {

            try

            {

                MessageQueue privateQueue;

                if (!MessageQueue.Exists(".\\Private$\\privateQueue"))

                    privateQueue = MessageQueue.Create(".\\Private$\\privateQueue");

                else

                    privateQueue = new MessageQueue(".\\Private$\\privateQueue");



                privateQueue.Send("Hello, World !", "Message Label");

                MessageQueue anotherPrivateQueue = new MessageQueue(".\\Private$\\privateQueue");

                System.Messaging.Message msg = anotherPrivateQueue.Receive();

                msg.Formatter = new System.Messaging.XmlMessageFormatter(new Type[1] { typeof(string) });

                Console.WriteLine(msg.Body.ToString());

            }

            catch (Exception ex)

            {

                Console.WriteLine(ex.Message);

            }

        }

    }

}

If you run the project, you will see a message-box displaying “Hello, World !”. This message content created by had been sent from MSMQ queue, received by another MSMQ, and processed. In our example both the sending and the receiving queues exist in the same application for the sake of simplicity. In real world applications they reside in very different areas.

This is a very basic introduction to MSMQ and non-transactional queues. In future posts, we will explore more advanced topics in MSMQ.

Asserts in NUnit

noreply@blogger.com (Ebeid Soliman) — Fri, 21 Oct 2011 17:19:00 +0000

Assertions (or asserts for short) are helper methods that can assist us in determining whether a method under test is performing correctly or not. They let us assert conditions, data, and so on. All assertion methods will report failures (that’s when the assertion is false) or errors (that’s when we get an unexpected exception) and report these through the NUnit test runner. when a failure or error occurs, execution of the current test method is aborted. Other Test within the same test fixture will still be run.

Classic Asserts

Assert.AreEqual (expected, actual, string message)
- For floating-point numbers we need to specify the tolerance of the equality.
- Assert.AreEqual(3.33, 10.0/3.0, 0.01) // Checks the result but looks only for the first two decimal places.
Assert.Less(x, y) // asserts that x < y
Assert.Greater(x, y) // asserts that x > y
Assert.LessOrEqual(x, y) // asserts that x <= y
Assert.GreaterOrEqual(x, y) // asserts that x >= y
Assert.IsNull(object, string message)
Assert.IsNotNull(object, string message)
Assert.AreSame(expected, actual, string message)
Assert.IsTrue(bool condition, string message)
Assert.IsFalse(bool condition, string message)
Assert.Fail(string message)
- Fails the test immediately, can be used to mark sections of code that should not be reached.

*message in all previous statements is optional.

Constrain-Based Asserts

These are new assertion style introduced in NUnit 2.4. That new style enable many expectations to be evaluated together in a single assertion:

Assert.That(4, Is.LessThan(5) & Is.GreatThan(0));

Assert.That(actual, Is.EqualTo(expected))
Assert.That(actual, Is.Not.EqualTo(expected));
Assert.That(actual, Is.AtMost(expected)); //equivalent to Assert.LessOrEqual()
Assert.That(actual, Is.Atleast(expected)); //equivalent to Assert.GreaterOrEqual()
Assert.That(expected, Is.Null);
Assert.That(expected, Is.Not.Null); //equivalent to Assert.That(expected, !Is.Null);
Assert.That(expected, Is.Empty); //for collections and strings
Assert.That(actual, Is.InstanceOfType(expected));
Assert.That(actual, Has.Length(expected));
Assert.That(actualCollection, List.Contains(expectedValue));
- Assert.That({5, 3, 2}, List.contains(2)) //will pass
Assert.That(actualCollection, Is.SubsetOf(expectedCollection));
- Assert.That({5, 3, 2}, Is.SubsetOf({5, 4, 3, 2, 1})
Assert.That(actual, Text.StartsWith(expected)); //asserts that the expected string is at the beginning of actual
Assert.That(actual, Text.Matches(expected)); //asserts that the expected regular expression string matches actual
FileAssert.AreEqual(FileInfo expected, FileInfo actual);
- FileAssert.AreEqual(String pathToExpected, String PathToActual);

NUnit and Exceptions

Part of your code behavior may be throwing a specific exception in certain cases. If you want to assert that your code throws the exceptions that it designed to throw, you need to tag your test method with attribute: [ExpectedException(typeof(ExpectedException))].

   1:  [TestFixture]

   2:  public class ImportListTest

   3:  {

   4:      [Test]

   5:      [ExpectedException(typeof(ArgumentNullException))]

   6:      public void NullList()

   7:      {

   8:          Class1.ImportList(null);

   9:      }

  10:  }

This test method in now expected to throw an exception [from the call to ImportList(null)]. If it doesn’t, the test will fail. If a different exception is thrown (even a superclass of the one specified), the test fails. The test passes only if the exact exception specified is thrown.

Temporarily ignoring tests

If you wrote the test code first and will start writing code incrementally (something like TDD), you may temporarily ignore tests that you still implementing the code required to pass them. You could just tag these tests with attribute [Ignore(“Optional Message”)] . NUnit will report that these tests were skipped and show a yellow bar in the GUI, so you won’t forget about it later.

Structuring unit tests in NUnit

noreply@blogger.com (Ebeid Soliman) — Fri, 21 Oct 2011 17:19:00 +0000

Our goal in the previous post Introduction to NUnit, was just to introduce the unit testing using NUnit as simple as possible. Now it is the time to elaborate more on NUnit framework, structuring and organizing your test cases. If you examined the test code again:

   1:  using System;

   2:  using NUnit.Framework;

   3:  namespace ClassLibrary1

   4:  {

   5:      [TestFixture]

   6:      public class CmpTest

   7:      {

   8:          [Test]

   9:          public void LargestOf3()

  10:          {

  11:              int[] numbers = new int[] { 8, 9, 7 };

  12:              Assert.That(Cmp.Largest(numbers), Is.EqualTo(9));

  13:          }

  14:      }

  15:  }

The code is pretty straightforward. The using statement on line 2 brings in the necessary NUnit classes.

Next, we have the class definition on line 6: each class that contains tests must be annotated with a [TestFixture] attribute. The class have to be public, so the test runners can run it. The class must have a public, no-parameter constructor.

Finally, the test class contains individual methods annotated with [Test] attributes. Any public, parameterless method specified with a [Test] attribute will be run automatically by NUnit.

Structuring Unit Tests

As a good object-oriented design concept, a class should be focused on one responsibility and only one. This applies to test fixtures as well. Test fixtures should be focused on verifying behavior in a specific scenario. Its name should reflect this scenario. Tests inside that fixture should test different behaviors within this scenario. Tests should be organized around behaviors, not necessarily individual methods. To keep things readable and clear in the test runner output, favor putting fixture classes under a namespace that includes the name of the class that the fixture are testing.

   1:  namespace solutionName.Test.ShoppingCartTest

   2:  {

   3:      [TestFixture]

   4:      public class EmptyCartFixture

   5:      {

   6:          [Test]

   7:          public void OverallRateIsZero() { }

   8:      }

   9:  }

Categories

NUnit provides an easy way to mark and run individual test and fixtures by using categories. A category is just a name that we define. We can associate different test methods with one or more categories, that enable us to select which categories we want to exclude (or include) when running the tests. A category is specified as an attribute.

   1:  using System;

   2:  using NUnit.Framework;

   3:  namespace ClassLibrary1

   4:  {

   5:      [TestFixture]

   6:      public class CmpTest

   7:      {

   8:          [Test]

   9:          [Category("Short")]

  10:          public void LargestOf3()

  11:          {

  12:              int[] numbers = new int[] { 8, 9, 7 };

  13:              Assert.That(Cmp.Largest(numbers), Is.EqualTo(9));

  14:          }

15:

  16:          [Test]

  17:          [Category("Long")]

  18:          [Category("Fred")]

  19:          public void LargestOf3Alt()

  20:          {

  21:              int[] numbers = new int[3];

  22:              numbers[0] = 1;

  23:              numbers[1] = 1;

  24:              numbers[2] = 1;

  25:              Assert.That(Cmp.Largest(numbers), Is.EqualTo(1));

  26:          }

27:

  28:          [SetUp]

  29:          public void LargestOf3Alt2()

  30:          {

  31:              int[] numbers = new int[1];

  32:              numbers[0] = 0;

  33:              Assert.That(Cmp.Largest(numbers), Is.EqualTo(0));

  34:          }

  35:      }

  36:  }

In the GUI, if you excluded “Short” methods from run, only LargestOf3() will be selected and executed.

When categories are used, only the tests in the selected categories will be run. Those tests in categories that are not selected are not reported at all. If no category selected in the GUI, then all methods will be executed unless it is tagged with Explicit attribute. The Explicit attribute causes a test or test fixture to be ignored unless it is explicitly selected for running.

Per-Method Setup and Teardown We should write our test in order that each test can run independently of other tests; this allows us to run any test at any time in any order. In order to accomplish that we may need to reset things between tests or clean up after a test has run. [Setup] and [TearDown] attributes are used for these tasks.

Per-Fixture Setup and Teardown If we need to set something up or clean up after the entire test class has run, use [TestFixtureSetup] and [TestFixtureTearDown] for that.

Although setup and teardown methods generally come in pair, they don’t have to do so.

Suppose we need some sort of database connection for each test. Rather than duplicating code in each test method that connects to and disconnects from the database, we could use SetUp and TearDown methods. Since creating the initial connection to the database can be slow, we may want to do that only once before all the tests run by using TestFixtureSetUp.

   1:  [TestFixture]

   2:  public class DBTest

   3:  {

   4:      private SqlConnection dbConn;

5:

   6:      [TestFixtureSetUp]

   7:      public void PerFixtureSetup()

   8:      {

   9:          dbConn = new SqlConnection("ConnectionString");

  10:          dbConn.Open();

  11:      }

12:

  13:      [SetUp]

  14:      public void PerTestSetup()

  15:      {

  16:          //populate database with test data

  17:      }

18:

  19:      [TearDown]

  20:      public void PerTestTearDown()

  21:      {

  22:          // clean up database

  23:      }

24:

  25:      [TestFixtureTearDown]

  26:      public void PerFixtureTearDown()

  27:      {

  28:          dbConn.Close();

  29:          dbConn.Dispose();

  30:      }

31:

  32:      [Test]

  33:      public void AccountAccess()

  34:      {

  35:          // Uses dbConn

  36:      }

37:

  38:      [Test]

  39:      public void EmployeeAccess()

  40:      {

  41:          // Uses dbConn

  42:      }

  43:  }

In this example, here is the methods run sequence: PerFixtureSetup >> PerTestSetup >> AccountAccess >> PerTestTearDown >> PerFixtureTearDown >> PerFixtureSetup >> PerTestSetup >> EmployeeAccess >> PertestTearDown >> PerFixtureTearDown.

In this post we tried to elaborate more on the NUnit framework and how we structure and organize test cases in NUnit.

Introduction to NUnit

noreply@blogger.com (Ebeid Soliman) — Fri, 21 Oct 2011 17:17:00 +0000

What is unit testing?

According to Jeff Canna, unit testing ensures that a particular method of a class successfully performs a set of specific tasks. Each test confirms that a method produces the expected output when given a known input.

What is NUnit? NUnit is an open source framework that facilitates unit testing for all .NET languages. It allows you to build your test cases directly into the code of the project.

How do I get NUnit? Download the appropriate file from here and install it. That’s it !!

Write your code

We will write our code now and let it drive our testing cases and exploration of NUnit. Make a new project, class library that similar to the code below. Below an implementation of a “largest” method that returns the largest number in a list.

   1:  using System;

   2:  namespace ClassLibrary1

   3:  {

   4:      public class Cmp

   5:      {

   6:          public static int Largest(int[] list)

   7:          {

   8:              int max = Int32.MaxValue;

   9:              for (int index = 0; index < list.Length - 1; index++)

  10:              {

  11:                  if (list[index] > max)

  12:                      max = list[index];

  13:              }

  14:              return max;

  15:          }

  16:      }

  17:  }

Test your code

Now it comes to the testing step. Writing test cases is a big topic to be discussed here. We will try to make our first test as simple as possible because there is much to be tested the first time besides the code itself. First, add reference to NUnit and then write our test cases.

Right click on the references in “Solution Explorer” and click add a reference.

Click on tab “Browse”

Navigate to your NUnit directory and go into the bin subdirectory. For me this directory is in “C:\Program Files (x86)\NUnit 2.5.10\bin\net-2.0\framework”

Select nunit.framework.dll and click OK.

Then add a new class library to the project and add the code below to it. This code uses the NUnit namespace, attributes, Assert, and Is. We will explain all of these in detail in further posts. For now, just build the project.

   1:  using System;

   2:  using NUnit.Framework;

   3:  namespace ClassLibrary1

   4:  {

   5:      [TestFixture]

   6:      public class CmpTest

   7:      {

   8:          [Test]

   9:          public void LargestOf3()

  10:          {

  11:              int[] numbers = new int[] { 8, 9, 7 };

  12:              Assert.That(Cmp.Largest(numbers), Is.EqualTo(9));

  13:          }

  14:      }

  15:  }

Run your tests

Now we have got our code, and our test cases. The next step is to run that test cases to test our code and that is the role of test runners. A test runner knows to look for the [TestFixture] attribute of a class and for the [Test] methods within it. The runner will run the tests, accumulate some statistics on which tests passed and failed, and report the results to you. There are many test runners, you can use the NUnit GUI which is installed by default with NUnit (will be located in the programs menu of the start menu).

Using NUnit GUI

In the NUnit GUI, select File > Open or press Ctrl + O. This would show the Open Project dialog. Browse to the folder which contains the assembly containing your tests and open it.

Now you can test your code by clicking the Run button.

When we run a selected test, the GUI will display a large, colored, status bar. If all the tests pass, the bar is bright green. If any test fails, the bar is red. If any test was skipped, the bar is yellow.

It is obvious that our test case here have been failed. Try to fix the code and run the test again until it succeed

Log4net Extended Tutorial

noreply@blogger.com (Ebeid Soliman) — Mon, 17 Oct 2011 02:48:00 +0000

Logging is one of the most effective debugging and troubleshooting techniques although some engineers claim that it is not that helpful or it will slow the application. My experience with it is always positive.

I use log4net from the Apache Software Foundation as my logging layer for .Net applications. log4net is available here as a zip archive.

Hello World Example [uses BasicConfigurator]

Download log4net from here.
Open Visual Studio and create a new Console application project.
Add to the project a reference to the \bin\net\2.0\release\log4net.dll assembly in the log4net distribution.
Modify your Main() method like so:

   1:  using System;

   2:  namespace ConsoleApplication1

   3:  {

   4:      class Program

   5:      {

   6:          static void Main(string[] args)

   7:          {

   8:              log4net.Config.BasicConfigurator.Configure();

   9:              log4net.ILog log = log4net.LogManager.GetLogger(typeof(Program));

10:

  11:              log.Debug("Hello World!");

  12:              log.Info("Just info.");

  13:              log.Warn("careful this is a warning.");

  14:              log.Error("Bad news: Error occurred.");

  15:              log.Fatal("Here is the end.");

  16:          }

  17:      }

  18:  }

Compile and run the application, and you'll see output to the console like so:

In the Hello World example we configured log4net in the most basic way possible just to get started. Using the BasicConfigurator (line 8) will cause log4net to output log entries to the console using the default layout. Line 9 requests a logger from the LogManager object. Logger objects implement the ILog interface, which is what your application will use to instrument itself and percolate log entries.

Lines 11-15 use the logger to log a few statements with various severity levels. log4net defines 5 such levels:

Debug: fine-grained statements concerning program state, typically used for debugging;

Info: informational statements concerning program state, representing program events or behavior tracking;

Warn: statements that describe potentially harmful events or states in the program;

Error: statements that describe non-fatal errors in the application; this level is used quite often for logging handled exceptions;

Fatal: statements representing the most severe of error conditions, assumedly resulting in program termination.

XML Configurator

In the previous example we used the BasicConfigurator. If we just changed the configurator to XmlConfigurator and run the application you will get the following output error:

log4net:ERROR XmlConfigurator: Failed to find configuration section 'log4net' in

the application's .config file. Check your .config file for the <log4net> and <

configSections> elements. The configuration section should look like: <section n

ame="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler,log4net" /

>

This error resulted because XMLConfigurator relies on an XML document to supply configuration for log4net. The static Configure method have 10 overloads to accept the XML configuration from a file, stream, URI, or an XmlElement object. The parameterless overload of the method we used instructs log4net to load the XML from the application’s configuration file. Since our application doesn’t contain any configuration files (till now), the Configure method returned the above error.

To overcome this error, add Application Configuration File to the project and add the following XML configuration:

   1:  <configuration>

   2:    <configSections>

   3:      <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net"/>

   4:    </configSections>

5:

   6:    <log4net>

   7:      <appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender">

   8:        <layout type="log4net.Layout.SimpleLayout" />

   9:      </appender>

10:

  11:      <root>

  12:        <level value="ALL" />

  13:        <appender-ref ref="ConsoleAppender" />

  14:      </root>

  15:    </log4net>

  16:  </configuration>

Compile and run, you will see output changed to be like so:

In addition to the Configure method, XmlConfigurator offers a static ConfigureAndWatch() method that accepts a FileInfo reference. ConfigureAndWatch() will monitor the XML configuration file and reconfigure log4net when a change is detected in this file. This allows you to alter logging behavior while the application is running to fit your current troubleshooting requirements.

If we look at the application configurations file, the declaration of the log4net configuration section on line 3 is mandatory. The log4net configurations section [lines 6-15] consists of two sub-sections: appender and root. log4net appender is the place where your application log entries end up (Console in our example). We will talk about appenders later. The root logger controls the general behavior of log4net. In this example, the root logger is told to send everything to the console appender.

Taming log4net Output

In the previous examples we were logging all types of severity levels. If you want to filter the logging output to fit your application’s life current phase (development, production) or troubleshooting needs, you can just change the value of <level /> XML element in your app.config. The level value instructs the logger of the minimum level at which log entries should be processed. It can be: ALL (everything is enabled), DEBUG (identical to ALL), INFO (everything except DEBUG), WARN (warn, Error, and Fatal), ERROR (Error and Fatal), FATAL (only Fatal), OFF (logging is disabled).

The Structure of log4net

Before going further with log4net, let’s talk about its structure. log4net built using the layered approach, with four main components inside of the framework. These are Logger, Repository, Appender, and Layout.

The Logger is the main component with which your application interacts. It is also the component that generates the log messages.

The Repository is responsible for maintaining the logical organization of loggers inside the framework. Currently log4net only supported the hierarchical organization through log4net.Repository.Hierarchy namespace. If you want to extend log4net to implement new organization, you have to implement the log4net.Repository.ILoggerRepository interface or inherit class log4net.Repository.LoggerRepositorySkeleton. Almost all developers do not use any of these Repository classes; they only use the LogManager class to automatically manage the repositories and the loggers.

The Appender is used to define the output medium that log statements will go for it at the end of the day. As the name suggests, these components append themselves to the Logger component and relay the output to an output stream. You can append multiple appenders to a single logger. There are several appenders provided by the log4net framework. If you want to write your own appender, you can start by inheriting thelog4net.Appender.AppenderSkeleton class, which works as an adapter between your class and the IAppender interface. You can use Appender Filters to filter logging events sent from appenders before processing it.

The Layout component is used to display the final formatted output to the user. The output can be shown in multiple formats, depending upon the layout we are using. It can be linear or an XML file. The layout component works with an appender. There is a list of different layouts in the API documentation. You cannot use multiple layouts with an appender. To create your own layout, you need to inherit thelog4net.Layout.LayoutSkeleton class, which implements the ILayout interface.

Appenders

An appender is an object that persists your log messages someplace. In the table below a list of the appenders provided by log4net.

Appender	Description
AdoNetAppender	Appender that logs to a database.
AnsiColorTerminalAppender	Appends logging events to the terminal using ANSI color escape sequences.
AspNetTraceAppender	Appends log events to the ASP.NET TraceContext system.
BufferingForwardingAppender	Buffers events and then forwards them to attached appenders.
ColoredConsoleAppender	Appends logging events to the console colored.
ConsoleAppender	Appends logging events to the console.
DebugAppender	Appends log events to the Debug system.
EventLogAppender	Writes events to the system event log.
ForwardingAppender	This appender forwards logging events to attached appenders.
FileAppender	Appends logging events to a file.
LocalSyslogAppender	Logs events to a local syslog service.
MemoryAppender	Stores logging events in an array.
NetSendAppender	Logs entries by sending network messages using the NetMessageBufferSend native function.
OutputDebugStringAppender	Appends log events to the OutputDebugString system.
RemoteSyslogAppender	Logs events to a remote syslog daemon.
RemotingAppender	Delivers logging events to a remote logging sink.
RollingFileAppender	Appender that rolls log files based on size or date or both.
SmtpAppender	Send an e-mail when a specific logging event occurs, typically on errors or fatal errors.
SmtpPickupDirAppender	Send an email when a specific logging event occurs, typically on errors or fatal errors. Rather than sending via smtp it writes a file that another service, such as the IIS SMTP agent, can use to manage sending the messages.
TelnetAppender	Appender that allows clients to connect via Telnet to receive log messages.
TraceAppender	Appends log events to the Trace system.
UdpAppender	Sends logging events as connectionless UDP datagrams to a remote host or multicast using the UdpClient class.

FileAppender and RollingFileAppender

These appenders write log messages to files. A typical configuration for the FileAppender might look like this:

   1:  <appender name="FileAppender" type="log4net.Appender.FileAppender">

   2:      <file value="log.txt" />

   3:      <appendToFile value="true" />

   4:      <encoding value="utf-8" />

   5:      <layout type="log4net.Layout.SimpleLayout" />

   6:  </appender>

Configuration attributes and its description are:

file: the full or relative path to the log file;

appendToFile: boolean indicating whether the log file should be appended (true) or overwritten (false). If false, the file overwrite occurs during log4net initialization. If unspecified, the log file is appended;

immediateFlush: boolean indicating whether to flush the log file TextWriter after each log message is written. The default is true (flush each message after its written);

lockingModel: allows control over the log file locking strategy. This can be either "log4net.Appender.FileAppender+MinimalLock" to allow for loose file locking or "log4net.Appender.FileAppender+ExclusiveLock" to lock the file during program execution.

Keep in mind that the log file managed by FileAppender will be allowed to grow without bounds. The RollingFileAppender provides basic log file management, configurable size- or date-boxing of the log file, and limited rolling backups of the log file.

   1:  <appender name="RollingFileAppender" type="log4net.Appender.RollingFileAppender">

   2:      <file value="log-file.txt" />

   3:      <appendToFile value="true" />

   4:      <rollingStyle value="Size" />

   5:      <maxSizeRollBackups value="10" />

   6:      <maximumFileSize value="1MB" />

   7:      <staticLogFileName value="true" />

   8:      <layout type="log4net.Layout.SimpleLayout" />

   9:  </appender>

Configuration attributes and its description are:

rollingStyle: this controls how log files are "rolled," and can be one of the following values:
- Once: the log file is rolled every time log4net is initialized (typically at application startup);
- Size: the log file is rolled once it breaches a certain size;
- Date: the log file is rolled based on the current date;
- Composite: the log file is rolled based on size constraints and the current date;

maximumFileSize: the size cap on the log file. This is an expression of size in the form of "#(KB|MB|GB)". For instance, "100KB" or "10MB";

maxSizeRollBackups: the maximum number of rolled log file backups to maintain when rollingStyle is SIZE; when rollingStyle is COMPOSITE, this indicates the maximum number of roll-offs maintained per day; this property has no effect when rollingStyle is ONCE or DATE;

datePattern: the date pattern used to roll files based on date. The value of this parameter needs to adhere to the format used by the SimpleDateFormatter class;

staticLogFileName: a bit of a misnomer - when true this setting indicates whether log4net should actively write logs to the configured file (log-file.txt in our example configuration) and maintain rolling backups by copy. When false, this setting indicates that log4net will actively log to the latest roll-off file (e.g., log-file1.txt, log-file2.txt, log-file3.txt, etc);

countDirection: indicates how roll-off file numbering is managed. When this parameter is >= 0, the newest log file will have the largest number; e.g., log-file.txt.5 will be newer than log-file.txt.4. When countDirection < 0, the newest log file will have the lowest number; e.g., log-file.txt.1 will be newer than log-file.txt.2. If unspecified, countDirection defaults to (-1);

Keep in mind that when using a file appender, the user running the logging process must have rights to create and/or modify the log file in order for log messages to be written properly. In addition, log4net will create the log file if it does not exist, but it will not create directories in the log file path that do not already exist. If log4net encounters a problem initializing the file appender (e.g., it cannot create the log file for security reasons), the log file will not be written but your application will continue to execute normally.

Using multiple appenders

You can use multiple appenders by specifying each appender you need under the root logger.

Layouts and patterns

A layout is just a template for your log messages. Layouts are specified per-appender, and you can specify only one layout for an appender:

   1:  <appender name="ColoredConsoleAppender" type="log4net.Appender.ColoredConsoleAppender">

   2:      <layout type="log4net.Layout.SimpleLayout" />

   3:  </appender>

The PatternLayout allows you to specify a printf-style template for your log entries using a "conversion pattern," and gives you the opportunity to decorate each entry with some valuable instance data. For instance, this configuration:

   1:  <appender name="ColoredConsoleAppender" type="log4net.Appender.ColoredConsoleAppender">

   2:      <layout type="log4net.Layout.PatternLayout">

   3:          <conversionPattern value="%date [%thread] %-5level %logger - %message%newline" />

   4:      </layout>

   5:  </appender>

produces a log that appears like so:

The conversion pattern string can include literal text and the following format expressions:

expression	value
%appdomain	the friendly name of the appdomain from which the log entry was made
%date	the local datetime when the log entry was made
%exception	a formatted form of the exception object in the log entry, if the entry contains an exception; otherwise, this format expression adds nothing to the log entry
%file	the file name from which the log entry was made; note that using %file has a significant performance impact and I don't recommend using it
%identity	the user name of the active user logging the entry; this one is less reliable than %username; note that using %identity has a significant performance impact and I don't recommend using it
%level	the severity level of the log entry (DEBUG,INFO, etc)
%line	the source code line number from which the log entry was made; slow
%location	some rudimentary call stack information, including file name and line number at which the log entry was made; using
%logger	the name of the logger making the entry; more on this in a bit
%method	the name of the method in which the log entry was made; also slow
%message	the log message itself (don't forget this part!)
%newline	the value of Environment.NewLine
%timestamp	the milliseconds between the start of the application and the time the log entry was made
%type	the full typename of the object from which the log entry was made
%username	the Windows identity of user making the log entry; slow
%utcdate	the UTC datetime when the log entry was made
%%	a percent sign (%)

A common usage of the %logger expression is to identify the source class in the log entries. Give the class name to the logger like:

private static log4net.ILog Log = log4net.LogManager.GetLogger( System.Reflection.MethodBase.GetCurrentMethod().DeclaringType );

and the output will be something like:

ConsoleApplication1.Program [INFO]- this is an info message

Using this technique can help you control logging concerns of your objects. For example, add a new class “MyClass” to your ConsoleApplication and use the above approach in declaring your loggers. Add the following after the <root> section in your app.config :

   1:  <logger name="ConsoleApplication1">

   2:        <level value="ALL" />

   3:        <appender-ref ref="FileAppender" />

   4:  </logger>

   5:  <logger name="ConsoleApplication1.Program">

   6:        <level value="ALL" />

   7:        <appender-ref ref="ConsoleAppender" />

   8:  </logger>

   9:  <logger name="ConsoleApplication1.MyClass">

  10:        <level value="ALL" />

  11:  </logger>

This way, the console output will display only the log statements from the Program class, the log file will contain the log statements from both Program class and MyClass class. Although MyClass doesn’t have an appender associated with it, the container namespace does and that appender who processes MyClass log statements. If you changed the level value to “OFF” for ConsoleApplication1.MyClass, the log file will not contain log statements from MyClass. When applying the previous technique keep into consideration the following:

Appenders accumulate through the hierarchy: if both the Program class and ConsoleApplication1 namespace are each configured to append to the console, any log from the Program class will show up twice in the console (once for the class logger and once for the namespace logger).

Specific logger levels deeper in the hierarchy overcome the general levels : if the Program class logging level is set to ALL and the ConsoleApplication1 namespace logging level is set to OFF, logs from the program class are still written to the appenders configured for ConsoleApplication1.

Log event context

Beside the expressions mentioned above, sometimes you need to log context information. Modify your console application to look so:

   1:  using System;

   2:  namespace ConsoleApplication1

   3:  {

   4:      class Program

   5:      {

   6:          private static log4net.ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

   7:          static void Main(string[] args)

   8:          {

   9:              log4net.Config.XmlConfigurator.Configure();

  10:              log4net.ThreadContext.Properties["myContext"] = "Logging from Main";

  11:              log.Info("Just info.");

  12:              Console.ReadLine();

  13:          }

  14:      }

  15:  }

Now update the app.config to look like:

   1:  <?xml version="1.0" encoding="utf-8" ?>

   2:  <configuration>

   3:    <configSections>

   4:      <section name="log4net"

   5:        type="log4net.Config.Log4NetConfigurationSectionHandler, log4net"/>

   6:    </configSections>

7:

   8:    <log4net>

9:

  10:      <appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender">

  11:        <layout type="log4net.Layout.PatternLayout">

  12:          <conversionPattern value="%logger (%property{myContext}) [%level]- %message%newline" />

  13:        </layout>

  14:      </appender>

15:

  16:      <root>

  17:        <level value="ALL" />

  18:        <appender-ref ref="ConsoleAppender" />

  19:      </root>

  20:    </log4net>

  21:  </configuration>

Compile and run, you will got a log statement similar to:

ConsoleApplication1.Program (Logging from Main) [INFO]- Just info.

What we did here is that we added a property “myContext” to the ThreadContext static class and assigned a simple string value. This property processed by the appender through the %property{myContext} expression in the layout conversion pattern.

There are three hierarchy logging contexts available in log4net, where properties in the more granular contexts override property values in less granular contexts.

Context	Description
log4net.GlobalContext	A global context shared across all application threads and app domains. If two threads set the same property on the GlobalContext, one value will overwrite the other.
log4net.ThreadContext	Any properties set in this context are scoped to the calling thread. In other words, in this context two threads can set the same property to different values without stomping on each other.
log4net.ThreadLogicalContext	This context behaves similarly to the ThreadContext, except that the scope is defined by logical thread boundaries. I'll be honest and say that I've never used the ThreadLogicalContext in my career, but if you're working with a custom thread pool algorithm or hosting the CLR, you may find some use for this one.

Calculated context values

Context property values don't have to be strings. You can set the value of a context property to any object reference; then the object's ToString can be used to obtain the needed context property when a logging event occurs. This technique could be used to log whatever state you want at the time of each logging event.

ThreadContext Stacks

ThreadContext and ThreadLogicalContext can store property values in stacks available via the Stacks static property of each class. Pushing a property value onto a stack returns an IDisposable that, when disposed, pops the property value off of the stack. The logging output reflects the state of the context stack at each logging event; the stack is represented as a space-delimited list, with newer items appearing later in the list. Its like printing the current call stack in every log statement. check this example:

   1:  using System;

   2:  namespace ConsoleApplication1

   3:  {

   4:      class Program

   5:      {

   6:          private static log4net.ILog log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

   7:          static void Main(string[] args)

   8:          {

   9:              log4net.Config.XmlConfigurator.Configure();

  10:              using (log4net.ThreadContext.Stacks["myContext"].Push("Main"))

  11:              {

  12:                  FirstAction();

  13:                  SecondAction();

  14:              }

  15:              Console.ReadLine();

  16:          }

17:

  18:          static void FirstAction()

  19:          {

  20:              using (log4net.ThreadContext.Stacks["myContext"].Push("FirstAction"))

  21:              {

  22:                  UtilityRoutine();

  23:              }

  24:          }

25:

  26:          static void SecondAction()

  27:          {

  28:              using (log4net.ThreadContext.Stacks["myContext"].Push("SecondAction"))

  29:              {

  30:                  UtilityRoutine();

  31:                  FirstAction();

  32:              }

  33:          }

34:

  35:          static void UtilityRoutine()

  36:          {

  37:              using (log4net.ThreadContext.Stacks["myContext"].Push("UtilityRoutine"))

  38:              {

  39:                  log.Info("this is an info message");

  40:              }

  41:          }

  42:      }

  43:  }

Compile and run, you will get these log messages:

ConsoleApplication1.Program (Main FirstAction UtilityRoutine) [INFO]- this is an info message

ConsoleApplication1.Program (Main SecondAction UtilityRoutine) [INFO]- this is an info message

ConsoleApplication1.Program (Main SecondAction FirstAction UtilityRoutine) [INFO]- this is an info message

Filters

Filters are applied to individual appenders via the log4net configuration, and they help the appender determine whether a log event should be processed by the appender. The following configuration defines two file appenders, each with a unique filter applied:

   1:  <?xml version="1.0" encoding="utf-8" ?>

   2:  <configuration>

   3:    <configSections>

   4:      <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net"/>

   5:    </configSections>

   6:    <log4net>

   7:      <appender name="LogFileAppender" type="log4net.Appender.FileAppender">

   8:        <file value="log.txt" />

   9:        <filter type="log4net.Filter.LevelRangeFilter">

  10:          <levelMin value="DEBUG" />

  11:          <levelMax value="WARN" />

  12:        </filter>

  13:        <layout type="log4net.Layout.SimpleLayout" />

  14:      </appender>

  15:      <appender name="ErrorFileAppender" type="log4net.Appender.FileAppender">

  16:        <file value="errors.txt" />

  17:        <filter type="log4net.Filter.LevelRangeFilter">

  18:          <levelMin value="ERROR" />

  19:          <levelMax value="FATAL" />

  20:        </filter>

  21:        <layout type="log4net.Layout.SimpleLayout" />

  22:      </appender>

  23:      <root>

  24:        <level value="DEBUG" />

  25:        <appender-ref ref="LogFileAppender" />

  26:        <appender-ref ref="ErrorFileAppender" />

  27:      </root>

  28:    </log4net>

  29:  </configuration>

The filter on the first appender will log statements with levels DEBUG, INFO, and WARN; the filter on the second appender will log statements with levels ERROR and FATAL. There are many filters available in log4net, we will review them quickly.

log4net.Filter.LoggerMatchFilter Filters log events based on the name of the logger object from which they are emitted. This filter can be configured using the following properties:

loggerToMatch: a string value to match against the message's logger name. The match is made using the String.StartsWith method;

acceptOnMatch: a boolean value indicating whether a matching logger name results in accepting the message (true) or rejecting it (false). This defaults to true, meaning that only matching logger names will be allowed into the appender.

log4net.Filter.LevelMatchFilter Filters log statements that match (or don’t match) a specific logging level. This filter can be configured using the following properties:

levelToMatch: the log level to match - either DEBUG, INFO, WARN, ERROR, or FATAL;

acceptOnMatch: a boolean value indicating whether to accept log levels matching the levelToMatch property (true), or reject log levels matching the levelToMatch property (false). Defaults to true.

log4net.Filter.LevelRangeFilter Similar to the LevelMatchFilter, except that instead of filtering a single log level, this filters on an inclusive range of contiguous levels. This filter can be configured using the following properties:

levelMin: the minimum log level to match - either DEBUG, INFO, WARN, ERROR, or FATAL;

levelMax: the minimum log level to match - either DEBUG, INFO, WARN, ERROR, or FATAL;

acceptOnMatch: a boolean value indicating whether to accept log levels matching the levelToMatch property (true), or to punt filtering to the next filter in the chain (false). Defaults to true. Note that any log level outside of the [levelMin, levelMax] range is denied by this filter.

log4net.Filter.StringMatchFilter Filters log events based on a string or regular expression match against the log message. This filter can be configured using the following properties:

regexToMatch: a regular expression to match against the log message. Note this regex is created with the Compiled option enabled for performance;

stringToMatch: a static string to match against the log message. The match is made using the String.IndexOf method to see if the static string exists in the log message, which is a case-sensitive search.

acceptOnMatch: a boolean value indicating whether to accept log messages matching the string or regex (true), or to deny log messages matching the string or regex (false). Defaults to true.

log4net.Filter.PropertyFilter Filters log events based on a value or regular expression match against a specific context property.

key: the name of the property value to match;

regexToMatch: a regular expression to match against the specified property value. Note this regex is created with the Compiled option enabled for performance;

stringToMatch: a static string to match against the specified property value. The match is made using the String.IndexOf method to see if the static string exists in the property value, which is a case-sensitive search.

acceptOnMatch: a boolean value indicating whether to accept messages with a property value matching the string or regex (true), or to deny messages with a property value matching the string or regex (false). Defaults to true.

log4net.Filter.DenyAllFilter This filter simply denies all filtering. When this is used, it's always at the end of a filter chain to block unwanted log messages from the appender.

Lossy logging

If you want comprehensive log to be there in case of errors but you worry about the resources consumed by logging, you might use lossy logging. Lossy logging gives the best of both worlds, under normal operation your application will not log any messages; however, if your application logs an error, a small batch of messages leading up to the error is placed into the log, giving you the error and a snapshot of system activity just before it happened.

Lets modify our code to be like:

   1:  using System;

   2:  namespace ConsoleApplication1

   3:  {

   4:      class Program

   5:      {

   6:          static log4net.ILog Log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

   7:          static void Main( string[] args )

   8:          {

   9:              log4net.Config.XmlConfigurator.Configure();

  10:              for( int i = 0; i < 100; ++i )

  11:              {

  12:                  Log.DebugFormat( "this is debug msg #{0}", i );

  13:              }

  14:              Log.Error( "error: an error occurred!" );

  15:              Log.Warn( "warning: you've been warned" );

16:

  17:              Console.ReadLine();

  18:          }

  19:      }

  20:  }

Let's modify our app.config to look like:

   1:  <?xml version="1.0" encoding="utf-8" ?>

   2:  <configuration>

   3:    <configSections>

   4:      <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net"/>

   5:    </configSections>

6:

   7:    <log4net>

   8:      <appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender">

   9:        <layout type="log4net.Layout.SimpleLayout" />

  10:      </appender>

11:

  12:      <appender name="LossyConsoleAppender" type="log4net.Appender.BufferingForwardingAppender">

  13:        <bufferSize value="20" />

  14:        <lossy value="true"/>

  15:        <evaluator type="log4net.Core.LevelEvaluator">

  16:          <threshold value="ERROR" />

  17:        </evaluator>

  18:        <appender-ref ref="ConsoleAppender" />

  19:      </appender>

20:

  21:      <root>

  22:        <level value="DEBUG" />

  23:        <appender-ref ref="LossyConsoleAppender" />

  24:      </root>

  25:    </log4net>

  26:  </configuration>

Compile and run, you will see output like:

The logging configuration defines two appenders, one very generic Console appender and a BufferingForwardingAppender. As the name implies, the latter appender buffers log messages and forwards them in batches to one or more other appenders. You can probably tell from the configuration XML that I've set this appender up with a 20-message buffer. The lossy and evaluator parameters work together to define when the log message buffer is forwarded to the "base" appender.

The program issues a total of 102 log messages (100 DEBUG, one ERROR, and one WARN), but the console only contains 20 messages (19 DEBUG and 1 ERROR). So what happened to the other 82 DEBUG and WARN messages?

When the BufferingForwardingAppender's lossy property is enabled, the appender will buffer log messages without forwarding them. If the buffer fills up, the oldest messages are dropped from the buffer to make room for the new messages.

The evaluator property determines when the appender forwards the messages from the buffer to its base appenders. There is only one evaluator defined in log4net - the LevelEvaluator. The LevelEvaluator triggers the forward when a log message is received that meets or exceeds the configured threshold. The example above is configured so that an ERROR message triggers the appender to forward its buffer to the ConsoleAppender.

Lossy Appender Types log4net provides the following appenders that can operate “lossy” :

log4net.Appender.AdoNetAppender

log4net.Appender.RemotingAppender

log4net.Appender.SmtpAppender

log4net.Appender.SmtpPickupDirAppender

log4net.Appender.BufferingForwardingAppender

Configuring log4net Programmatically

Sometimes we are in the mood to code as quickly as possible without getting into configuration files. Normally, that happens when we are trying to test something. In that case, you have another way to do the configuration. All of the long configuration files that we saw in the previous section can be defined programmatically using a few lines of code. See the following code:

   1:  // using a FileAppender with a PatternLayout

   2:  log4net.Config.BasicConfigurator.Configure( new log4net.Appender.FileAppender(new log4net.Layout.PatternLayout("%d [%t]%-5p %c [%x] &lt;%X{auth}&gt; - %m%n"),"testfile.log"));

3:

   4:  // using a FileAppender with an XMLLayout

   5:  log4net.Config.BasicConfigurator.Configure( new log4net.Appender.FileAppender(new log4net.Layout.XMLLayout(),"testfile.xml"));

6:

   7:  // using a ConsoleAppender with a PatternLayout

   8:  log4net.Config.BasicConfigurator.Configure( new log4net.Appender.ConsoleAppender(new log4net.Layout.PatternLayout("%d [%t] %-5p %c [%x] &lt;%X{abc}&gt; - %m%n")));

9:

  10:  // using a ConsoleAppender with a SimpleLayout

  11:  log4net.Config.BasicConfigurator.Configure( new log4net.Appender.ConsoleAppender(new log4net.Layout.SimpleLayout()));

You can see that while it is easy to code here, you can't configure settings for individual loggers. All of the settings that are defined here are applied to the root logger.

Log4net Best Practices

1- Use a unique logger object for each type in your application and give it the class name. Like so:

static log4net.ILog Log = log4net.LogManager.GetLogger(System.Reflection.MethodBase.GetCurrentMethod().DeclaringType);

2- Whenever you catch an exception, log it. Even if you just plan to throw it again, log it. In addition, log4net knows how to format Exception objects, so don't try and build your own string from the exception data, just pass the exception to the logging statement. Like So:

Log.Error("An exception raised due to XYZ", e);

3- Log to mark the activity and steps of your code.

4- Check for null references before logging any property. Logging code is code, and it should be treated the same way.

5- If an argument to the log method is expensive to obtain, be sure to guard your log statement with a check of the appropriate Is*Enabled property

if( Log.IsDebugEnabled ){ /* log statement */ }

6- Before an object calls on a shared dependency, consider pushing a tag onto the log context stack. This will allow you to determine the caller of the shared code that logged a particular message.

7- Whenever you use a formatted log statement, surround the format argument placeholders (the {}'s) with brackets or parentheses. This will mark the areas in each log message. This make the log scan a bit easier and makes empty or null formatting arguments more obvious.

Suggested logging levels

INFO Level

The start and end of the method

The start and end of any major loops

The start of any major case/switch statements

DEBUG Level

Any parameters passed into the method

Any row counts from result sets I retrieve

Any datarows that may contain suspicious data when being passed down to the method

Any "generated" file paths, connection strings, or other values that could get mungled up when being "pieced together" by the environment.

ERROR Level

Handled exceptions

Invalid login attempts (if security is an issue)

Bad data that I have intercepted forreporting

FATAL Level

Unhandled exceptions.

In this post I tried to cover as much as possible about logging using log4net. Although logging is extremely useful, keep in mind that it is not free, it consumes resources and the more you log the less you can find. So be careful, and Good Luck.

How I prepared for CSDP ?

noreply@blogger.com (Ebeid Soliman) — Tue, 28 Dec 2010 00:54:00 +0000

After I became an IEEE Certified Software Development Professional, many of my friends asked me about it and how to prepare for it. As a CSDP holder, let me introduce CSDP for you and how I prepared for it.

The CSDP credential is intended for mid-career software development professionals that want to confirm their proficiency of standard software development practices and advance in their careers.

Who is eligible to take the CSDP?

Licensed SW Engineers and IEEE Senior Members are already qualified for the CSDP. All others must meet the following:

Education: (need at least one)

Bachelor's degree
CSDA certificate holder
Educator at the post-baccalaureate level
Full member of the IEEE

Experience: (need at least one)

Advanced degree in software engineering and at least two years (about 3,500 hrs) of experience in software engineering/development
At least four years (about 7,000 hrs) experience in software engineering/development

Preparing for CSDP

Unfortunately, IEEE Computer Society didn’t provide or recommend a specific training material for CSDP. All what they provided is:

CSDP Exam specification: lists the knowledge area, its weight in the exam, and the competency level you should possess.
Recommend list of references.

Of course preparation depends on your knowledge level and experience you have. Below the materials that I used:

Guide to the Software Engineering Body of Knowledge | CS Press
Software Engineering, 8th Edition | Ian Sommerville | Addison Wesley
Code Complete, 2nd Edition | Steve McConnell | Microsoft Press
Software Engineering | IIT Bombay NPTEL Online Course , Prof. Rushikesh K Joshi

40 Lecture, almost 1 hour each
Almost covers all topics in the Ian Sommerville’s book: Software Engineering
http://freevideolectures.com/Course/2318/Software-Engineering

Software Engineering | UC Berkeley Online Course, Fall 2010 , Prof. Armando Fox

17 Lectures
http://freevideolectures.com/Course/2729/Software-Engineering

Principles of Software Engineering | University of Washington Online Course, Autumn 2007 , Prof. David Notkin

For software process only, the podcast series “IEEE Talks Software Process”

9 Podcasts, Avg 20 min each
http://www.computer.org/portal/web/ieee-talks-software-process/home

For software architecture, the podcast series “IEEE Software on architecture with Grady Booch”

26 Podcasts, Avg 15 min each
http://www.computer.org/portal/web/computingnow/onarchitecture

Although these resources looks comprehensive, questions in the exam may not be completely covered by these resource. So, try to enrich your knowledge and experience in software engineering as much as you can and Good Luck.

Business analysis & Requirements Engineering Certifications

noreply@blogger.com (Ebeid Soliman) — Fri, 17 Dec 2010 14:02:00 +0000

I was searching for certifications on requirements engineering and I found that there is a lot of mixing between business analysis and requirements engineering. This mix exists both on the daily use, professional certifications and organizations.

Starting by the definitions:

Business analysis is the discipline of identifying business needs and determining solutions to business problems. Solutions often include a systems development component, but may also consist of process improvement, organizational change or strategic planning and policy development. (Kathleen B Hass, Richard Vander Horst, Kimi Ziemski (2008). From Analyst to Leader: Elevating the Role of the Business Analyst Management Concepts, 2008. ISBN 1567262139. p94: "As the discipline of business analysis becomes professionalized")
Requirement engineering according to Laplante is "a sub-discipline of systems engineering and software engineering that is concerned with determining the goals, functions, and constraints of hardware and software systems.". (Phillip A. Laplante (2007) What Every Engineer Should Know about Software Engineering. ISBN 0849372283 p44)

This may lead us to conclude that business analysis as a discipline has a heavy overlap with requirements engineering, but focuses on identifying the changes to an organization that are required for it to achieve strategic goals. These changes include changes to strategies, structures, policies, processes, and information systems.

Regarding organizations: IIBA^® International Institute of Business Analysis is the independent non-profit professional association serving the growing field of Business Analysis, IREB International Requirements Engineering Board is one serving Requirements Engineering

Both IIBA and IREB offers professional certifications:

IIBA : Certified Business Analyst Professional CBAP
IREB : Certified Professional for Requirements Engineering CPRE

based on many forum posts and discussions I found that:

CBAP is targeted to highly educated BA professionals only (you need a minimum of 5 years of experience in business analysis / RE)
CPRE is aimed to everyone working in the field of requirements engineering
CPRE is a lot cheaper and can be achieved at an independent certification body

What you think ?

Could you please share your opinion and ideas with me ?

How to educate the next generation of software engineers ?

noreply@blogger.com (Ebeid Soliman) — Wed, 01 Sep 2010 10:51:00 +0000

After my completion of my master in software engineering at Nile University which we faced many problems at it (I hope it be better in the future), I tried to search for what globally constitutes a master of software engineering and I found the following:

The last effort to create a reference curriculum for graduate software engineering education was by the SEI in the early 1990s.
There were, in effect, no current community-endorsed recommendations on what to teach software engineers.
Response: create a project to create a new reference curriculum in software engineering (iSSec Project)

The Integrated Software and
Systems Engineering Curriculum iSSec Project

iSSec Project begun in May 2007 at Stevens Institute of Technology and sponsored by DoD Director of Systems and Software Engineering Kristen Baldwin. Three products planned for iSSec:

A modern reference curriculum for a master’s degree in software engineering that integrates an appropriate amount of systems engineering
A modern reference curriculum for a master’s degree in systems engineering that integrates an appropriate amount of software engineering
A truly interdisciplinary degree that is neither systems nor software engineering – it is both

What interests me the most is the first product, and it is the only completed product. A project of seven steps had been set to produce that reference curriculum:

Understand the current state of SwE graduate education (November 2007)
Create GSwE2009 0.25 (formerly GSwERC) with a small team, suitable for limited review (February 2008)
Publicize effort through conferences, papers, website, etc (continuous)
Create GSwE2009 0.50 (formerly GSwERC) suitable for broad community review and early adoption (October 2008)
Create GSwE2009 1.0 suitable for broad adoption (September 2009)
Transition stewardship to professional societies (Late 2009-Early 2010)
Foster adoption world-wide (2009 and beyond)

Steps 6 and 7 are undertaking now.

Graduate Software Engineering 2009 (GSwE2009): Curriculum Guidelines for Graduate Degree Programs in Software Engineering is the first product of the iSSEc project.

This document is a reference for anyone currently in a master program in software engineering or will enroll in such a program. GSwE2009 is a REFERENCE curriculum, not an absolute statement of requirements. It is a set of recommendations to universities on the best guidance that the authors could generate, incorporating extensive review comments from the broad community. The authors believe that GSwE2009 sets the “gold standard” for graduate software engineering education. Here is the summary of the most important parts of GSwE2009, outcomes and requirements.

Summary of Outcomes
Graduates of a master‘s program that satisfies GSwE2009 recommendations will:

CBOK : Master the CBOK (Core Body Of Knowledge).
DOMAIN : Master software engineering in at least one application domain, such as finance, medical, transportation, or telecommunications, and one application type, such as real-time, embedded, safety-critical, or highly distributed systems. That mastery includes understanding how differences in domain and type manifest themselves in both the software itself and in its engineering, and includes understanding how to learn a new application domain or type.
DEPTH : Master at least one KA (Knowledge Area) or sub-area from the CBOK to at least the Bloom Synthesis level.
ETHICS : Be able to make ethical professional decisions and practice ethical professional behavior.
SYSTEMS ENGINEERING : Understand the relationship between SwE and SE and be able to apply SE principles and practices in the engineering of software.
TEAM : Be an effective member of a team, including teams that are international and geographically distributed, effectively communicate both orally and in writing, and lead in one area of project development, such as project management, requirements analysis, architecture, construction, or quality assurance.
RECONCILIATION : Be able to reconcile conflicting project objectives, finding acceptable compromises within
limitations of cost, time, knowledge, existing systems, and organizations.
PERSPECTIVE : Understand and appreciate feasibility analysis, negotiation, and good communications with stakeholders in a typical software development environment, and be able to perform those tasks well; have effective work habits and be a leader.
LEARNING : Be able to learn new models, techniques, and technologies as they emerge, and appreciate
the necessity of such continuing professional development.
TECHNOLOGY : Be able to analyze a current significant software technology, articulate its strengths and weaknesses, compare it to alternative technologies, and specify and promote improvements or extensions to that technology.

Summary of Expected Background
GSwE2009 presumes that an entering student has:

DEGREE : The equivalent of an undergraduate degree in computing or an undergraduate degree in an engineering or scientific field and a minor in computing,
SWE COURSE : The equivalent of an introductory course in software engineering, and
EXPERIENCE :At least two years of practical experience in some aspect of software engineering or software development.

As Nile University is the only institute in Egypt that give a master degree in software engineering, I hope they follow this reference curriculum.

Individuals can also use it to guide their self-study plans in software engineering.

SQL Server Metadata

noreply@blogger.com (Ebeid Soliman) — Sun, 11 Apr 2010 16:44:00 +0000

Many times when you need to troubleshoot an SQL Server issue, you will need to collect metadata about the server, databases, and server resources in general. Here we will briefly review the mechanisms to collect these metadata.

System Base Tables

SQL server maintains a set of tables that store information about all the objects, data types, constraints, configuration options, and resources available to SQL Server. These tables are called the system base tables.

Some of these tables exist in master database –> contain system-wide information.
Some exist in every database –> contain database specific information.

You can access these tables names only if you are logged in as a system administrator. You can access through:

Run sp_help

Run

use master; 
select name from sys.objects 
where type_desc = 'SYSTEM_TABLE';

If you tried to select data from any of these system tables, you will got 208 error indicating that the object name is invalid. The only way to access these data is through dedicated administrator connection (DAC).

Keep in mind that these system base tables are used for internal purposes only within the database engine and are not intended for general use. They are subject to change and compatibility is not guaranteed.

Compatibility Views

Although it is possible to see data in the system tables in versions of SQL Server before 2005, it wasn’t recommended. For compatibility, SQL server 2005 and 2008 provided a set of compatibility views that allow access to a subset of the SQL server 2000 system tables. These views should be used for backward compatibility only; going forward, you should use catalog views.

Catalog Views

SQL Server 2005 introduced a set of catalog views as a general interface to the persisted system metadata. All the catalog views are in the sys schema, and you must reference the schema name when access the objects like:

select name from sys.databases;

For a complete list of catalog views categories, please consult http://msdn.microsoft.com/en-us/library/ms174365.aspx.

Information Schema Views

The information schema views comply with SQL-92 standard and all of it are in a schema called INFORMATION_SCHEMA. If you need to write a portable application that access the metadata you should use these views. This compliance come with the price of limited provided information (it provide the standard defined information only). If you need metadata about non-standard features, use catalog views.

For a complete list of Information schema views and its closest map to catalog views, please consult http://msdn.microsoft.com/en-us/library/ms186778.aspx

System Functions

Give us individual property values for many SQL Server instance, objects, databases. The values returned by system functions are scalar, so they can be used as values returned by SELECT statements like:

select DATABASEPROPERTYEX('msdb','Recovery');

For a complete list of system functions and its types, review http://msdn.microsoft.com/en-us/library/ms187786.aspx

System Stored Procedures

System Stored Procedures are the original metadata access tool but it had a drawback, basically you have to accept the data that it returns. Some of the procedures allow parameters but they are very limited. Catalog views are more enhanced and flexible in controlling what data appears.

I hope this brief post gives you an overall image about SQL Server metadata access mechanisms and which one to use according to your situation.

SQL Server Dedicated Administrator Connection

noreply@blogger.com (Ebeid Soliman) — Sun, 11 Apr 2010 10:08:00 +0000

Microsoft SQL Server provides a dedicated administrator connection (DAC) which allows an administrator to access a running instance of SQL Server Database Engine to troubleshoot problems on the server—even when the server is unresponsive to other client connections. The DAC is available through the sqlcmd utility and SQL Server Management Studio. The connection is only allowed from a client running on the server. No network connections are permitted.

To use SQL Server Management Studio with the DAC, connect to an instance of the SQL Server Database Engine with Query Editor by typing ADMIN: before the server name. Object Explorer cannot connect using the DAC.

To connect to a server using the DAC

In SQL Server Management Studio, with no other DACs open, on the toolbar, click Database Engine Query.
In the Connect to Database Engine dialog box, in the Server name box, type ADMIN: followed by the name of the server instance. For example, to connect to a server instance named ACCT\PAYABLE, type ADMIN:ACCT\PAYABLE.
Complete the Authentication section, providing credentials for a member of the sysadmin group, and then click Connect.

The connection is made.

If the DAC is already in use, the connection will fail with an error indicating it cannot connect.

If you are trying to that and got the following error:

Dedicated administrator connections are not supported. (ObjectExplorer)

It means that you are trying to connect Object Explorer using the DAC. Object Explorer cannot connect using the DAC; only Query Window can. That means that you cannot press the New Query button; you have to use the Database Engine Query button.

What is the SWEBOK Guide?

noreply@blogger.com (Ebeid Soliman) — Sun, 11 Apr 2010 01:41:00 +0000

SWEBOK or Software Engineering Body Of Knowledge is an all-inclusive term that describes the sum of knowledge within the profession of software engineering. Since it is usually not possible to put the full body of knowledge of even an emerging discipline, such as software engineering, into a single document, there is a need for a Guide to the Software Engineering Body of Knowledge. This Guide will seek to identify and describe that subset of the body of knowledge that is generally accepted.

SWEBOK Guide is developed by the IEEE Computer Society with leaders from industry and academia. It is ratified by ISO-IEC 24773 standard as the BOK for software engineering certifications internationally. The CSDA and CSDP are the first 2 certifications to conform to the ISO-IEC standard.

Here are the SWEBOK knowledge areas, its sub areas, and it’s references and suggested books.

1. Business Practices and Engineering Economics (3-4% questions in CSDP)

Engineering Economics
Ethics
Professional Practice
Standards

No book-length references are recommended for this topic.

The following books are suggested as supplemental reading in this area:

Morality and Machines: Perspectives on Computer Ethics, Edgar, S. L., Sudbury, Massachusetts: Jones and Bartlett, 1997. ISBN 0763717673
Computer Ethics, 2d Ed., Johnson, Deborah G, Upper Saddle River, NJ: Prentice-Hall, 1994. ISBN 0130836990
Smith and Roberson's Business Law, 11th ed., Mann, Richard A., & Barry S. Roberts, Cincinnati, OH: West Thomson Learning, 2000. ISBN 0324121849
Engineering Economy, Thusen, G.J., Prentice-Hall, 2000. ISBN 013028128X

2. Software Requirements (13-15% questions in CSDP)

Requirements Engineering Process
Requirements Elicitation
Requirements Analysis
Software Requirements Specification
Requirements Validation
Requirements Management

Any one of the following books is recommended in this area:

Software Requirements: Objects, Functions, & States, Davis, Alan. Upper Saddle River, New Jersey: Prentice Hall, 1993. ISBN 013805763X
Practical Software Requirements: A Manual of Content and Style, Kovitz, Benjamin L., Manning Publications Company, 1998. ISBN 1884777597
Mastering the Requirements Process, Robertson, James and Suzanne Robertson, New York: Dorset House, 2000. ISBN 0201360462
Requirements Engineering: A Good Practice Guide, Sommerville, Ian, New York: John Wiley & Sons, 1997. ISBN 0471974447
Software Requirements, Wiegers, Karl. Microsoft Press, 1999. ISBN 0735606315

3. Software Design (22-24% questions in CSDP)

Software Design Concepts
Software Architecture
Software Design Quality Analysis and Evaluation
Software Design Notations and Documentation
Software Design Strategies and Methods
Human Factors in Software Design
Software and System Safety

Any one of the following books is recommended in this area:

Software Architecture in Practice, Bass, Len, Paul Clements and Rick Kazman. Reading, Massachusetts: Addison-Wesley, 1998. ISBN 0201199300
Pattern Oriented Software Architecture: A System of Patterns, Buschmann, Frank, et al, John Wiley & Sons, 1996. ISBN 0471958697
Fundamentals of Object-Oriented Design in UML, Page-Jones, Meilir, Addison-Wesley, 1999. ISBN 0201699946X

The following books are suggested as supplemental reading in this area:

Fundamentals of Database Systems, Elmasri, Ramez and Shamkant Navathe, 3d Ed., Reading, Mass.: Addison Wesley, 2000. ISBN 0805317554
Software Reuse: Architecture, Process and Optimization for Business Success, Jacobsen, Ivar, Martin Griss, and Patrik Jonson, Reading, Massachusetts: Addison-Wesley, 1997. ISBN 0201924765

4. Software Construction (10-12% questions in CSDP)

Construction planning
Code design
Data design and management
Error processing
Source code organization
Code documentation
Construction QA
System integration and deployment
Code tuning
Construction tools

The following book is recommended on this topic:

Code Complete, McConnell, Steve, Microsoft Press, 1993. ISBN 1-5561-5484-4

The following books are suggested as supplemental reading in this area:

The Practice of Programming, Kernighan, Brian W. and Rob Pike, Reading, Mass.: Addison Wesley, 1999. ISBN 020161586X
The Pragmatic Programmer: From Journeyman to Master, Hunt, Andrew, David Thomas, and Ward Cunningham, Reading, Mass.: Addison Wesley, 1999. ISBN020161622X

5. Software Testing (15-17% questions in CSDP)

Types of Tests
Test Levels
Testing Strategies
Test Design
Test Coverage of Code
Test Coverage of Specifications
Test Execution
Test Documentation
Test Management

Any one of the following books is recommended in this area:

Testing Object-Oriented Systems, Binder, Robert V. Reading, Massachusetts: Addison-Wesley, 2000. ISBN 0201809389
Complete Guide to Software Testing, 2nd Ed., Hetzel, Bill, New York, New York: John Wiley & Son, 1993. ISBN 0471565679
Software Testing : A Craftsman's Approach, Jorgensen, Paul C. CRC Press, 1995. ISBN 0849308097
Testing Computer Software, 2nd Ed., Kaner, Clem, Jack Falk, and Hung Quoc Nguyen, New York, New York: John Wiley and Sons, 1999. ISBN 0471358460
Software Testing and Continuous Quality Improvement, Lewis, William. CRC Press 2000. ISBN 0849398339
The Craft of Software Testing: Subsystems Testing Including Object-Based and Object-Oriented Testing, Marick, Brian. Prentice Hall, 1997. ISBN 0131774115

6. Software Maintenance (3-5% questions in CSDP)

Software Maintainability
Software Maintenance Process
Software Maintenance Measurement
Software Maintenance Planning
Software Maintenance Management
Software Maintenance Documentation

The following book is recommended on this topic:

Practical Software Maintenance, Pigoski, Thomas M. New York, Wiley Computer Publishing, 1997. ISBN 0471170011

7. Software Configuration Management (3-4% questions in CSDP)

Management of SCM Process
Software Configuration Identification
Software Configuration Control
Software Configuration Status Accounting
Software Configuration Auditing
Software Release Management and Delivery

IEEE Standards 828-1990 and 1042-1987

8. Software Engineering Management (10-12% questions in CSDP)

Measurement
Organizational Management and Coordination
Initiation and Scope Definition
Planning
Software Acquisition
Enactment
Risk Management
Review and Evaluation
Project Close Out
Post-closure Activities

Any one of the following books is recommended in this area:

Principles of Software Engineering Management, Gilb, Tom, Reading, Mass.: Addison Wesley, 1988. ISBN 0-2011-9246-2
Rapid Development, McConnell, Steve, Microsoft Press, 1996. ISBN 1556159005
Software Engineering Project Management, 2d ed, Thayer, Richard H., IEEE Computer Society Press, Los Alamitos, CA 1997. ISBN 0-8186-8000-8
Quality Software Management, Vol. 1, Systems Thinking, Weinberg, Gerald M.. New York: Dorset House, 1992. ISBN 0932633226

The following book is suggested as supplemental reading in this area:

Developing Managerial Skills in Engineers and Scientists, Badawy, Michael K. Van Hostrand, NY, 1995. ISBN 0471286346

9. Software Engineering Process (2-4% questions in CSDP)

Process Infrastructure
Process Measurement
Process Definition
Qualitative Process Analysis
Process Implementation and Change

Either of the following books is recommended in this area:

The Capability Maturity Model: Guidelines for Improving the Software Process, Paulk, Mark, et al (Carnegie Mellon University / Software Engineering Institute). Reading, Mass.: Addison Wesley, 1995. ISBN 0201546647
Managing the Software Process, Humphrey, Watts S. Reading, Massachusetts, Addison-Wesley, 1989. ISBN 0201180952

10. Software Engineering Tools and Methods (2-4% questions in CSDP)

Management Tools and Methods
Development Tools and Methods
Maintenance Tools and Methods
Support Tools and Methods

11. Software Quality (6-8% questions in CSDP)

Software Quality Concepts
Planning for SQA and V&V
Methods for SQA and V&V
Measurement Applied to SQA and V&V

No book-length references are recommended for this topic:

The following books are suggested as supplemental reading in this area:

Software Engineering, Dorfman, M. & Thayer, R., eds. Los Alamitos, California: IEEE Computer Society Press, 1997. ISBN 0-8186-7609-4
Software Inspection, Gilb, Tom., and Dorothy Graham, Reading, MA, Addison-Wesley, 1994. ISBN 0-2016-3181-4
Practical Guide to Software Quality Management, Horch, John, Artech House, 1996. ISBN 0890068658
A Discipline for Software Engineering, Humphrey, Watts S. Reading, Massachusetts: Addison-Wesley, 1995. ISBN 0201546108
Metrics and Models in Software Quality Engineering, Kan, Stephen H. Addison Wesley, 1995. ISBN 0201729156

I’m Certified Software Development Professional:CSDP

noreply@blogger.com (Ebeid Soliman) — Sat, 10 Apr 2010 22:48:00 +0000

It is my pleasure to share with you my recent achievement. Now I’m IEEE Computer Society Certified Software Development Professional CSDP.

The CSDP credential is intended for mid-career software development professionals that want to confirm their proficiency of standard software development practices and advance in their careers.

Certified Software Development Professional should:

Possesses fundamental knowledge and understanding of computing principles and concepts and their application to the definition, design, construction, testing of software.
Be able to apply design principles with technical and economic tradeoffs to modules, subsystems, and systems in accordance with standards of practice, specifications, and principles of behavior of software as required to perform the functions as stated in the software requirements.
Has met the IEEE CS CSDP education, experience, and examination requirements.

How to become an CSDP:

Application Steps
1. Obtain CSDP candidate brochure from IEEE Computer Society http://www.computer.org/certification/
2. Review requirements
3. Complete application
4. Send completed application to IEEE CS by deadline with fee
5. Acknowledgement of payment
6. Review of application
7. If accepted, authorization to test sent to candidate
Requirements for Certification
1. Education: (need at least one)
  - You have a bachelor's degree
  - You are a CSDA certificate holder
  - You are an educator at the post-baccalaureate level
  - You are a full member of the IEEE
2. Experience: (need at least one)
  - You have an advanced degree in software engineering and at least two years (about 3,500 hrs) of experience in software engineering/development
  - You have at least four years (about 7,000 hrs) experience in software engineering/development
Preparation for Examination
1. Depends on level of expertise in areas covered by Body of Knowledge
2. Focus on areas needing the most review
3. Study from the suggested reference material
4. Can take a refresher course (see CSDP website)
5. Unless your background covers most of the Body of Knowledge, allow three months of 2-4 hours a week for study
6. Last week: assemble test access materials
Examination
1. Length: 4 hours
2. Closed book, computer based
3. At a Prometric training center (in Egypt only AMIDEAST Cairo branch)
4. Format
  • 180 multiple-choice questions that are based on concepts and experiences that should be familiar to a proficient software engineering professional with 7,000 or more hours of experience. The topic areas are grouped by question development into domains. The percentage range of questions in each area is as follows:
  - Business Practices and Engineering Economics (3–4%)
  - Software Requirements (13–15%)
  - Software Design (22–24%)
  - Software Construction (10–12%)
  - Software Testing (15–17%)
  - Software Maintenance (3–5%)
  - Software Configuration Management (3–4%)
  - Software Engineering Management (10–12%)
  - Software Engineering Process (2–4%)
  - Software Engineering Tools and Methods (2–4%)
  - Software Quality (6–8%)

General References :

Software Engineering: A Practitioner’s Approach, Latest ed.,
Pressman, Roger S., New York, New York: McGraw-Hill.
Software Engineering, Sommerville, I., Latest ed.
Reading, Massachusetts: Addison-Wesley.
SE Body of knowledge http://www.swebok.org
Online review course http://www.computer.org/certification/DistanceLearning/index.htm
Yahoo study group http://groups.yahoo.com/group/ieee_csdp/
Software Engineering Vol. 1: The Development Process, 2nd Edition.
Dorfman, M. & Thayer, R., editors,
Los Alamitos, California: IEEE Computer Society Press, 2002.
Software Engineering Vol. 2: The Supporting Processes, 2nd Edition.
Thayer, R. and M. Christensen, editors,
Los Alamitos,California: IEEE Computer Society Press, 2002.
IEEE Software Engineering Collection, 1999, vols. 1-4.
Piscataway, New Jersey: IEEE.

Comprehensive Reading List :

"Goal-Question-Metric (GQM) Approach" | Software Tech News, vol. 11, no. 4, Dec. 2008
A Specifier's Introduction to Formal Methods | Jeannette M. Wing | Computer, vol. 23, no. 9, Sept. 1990, pp.10-23
Algebraic Structures | Alex Lopez-Ortiz | Alex-Lopez Ortiz web page
Applied Statistics and Probability for Engineers, 4th Edition | Douglas C. Montgomery and George C. Runger | Wiley
Applying Formal Specifications to Real-World Software Development | Girish Keshav Palshikar | IEEE Software, vol. 18, no. 6, Nov. 2001, pp. 89-9.7
Balancing Agility and Discipline: A Guide for the Perplexed | Barry Boehm and Richard Turner | Addison Wesley
Code Complete, 2nd Edition | Steve McConnell | Microsoft Press
Computer Algorithms, 2nd Edition | Ellis Horowitz, Sartaj Sahni, and Sanguthevar Rajasekaran | Silicon Press
Computer Science: An Overview, 10th Edition | J. Glenn Brookshear | Addison Wesley
Computer Security: Art and Science | Matt Bishop | Addison Wesley
Configuration Management Principles and Practices | Anne Mette Jonassen Hass | Addison Wesley
Design Patterns: Elements of Reusable Object-Oriented Software | Erich Gamma et al | Addison Wesley
Discrete Mathematics and its Applications, 6th Edition | Kenneth Rosen | McGraw-Hill
Documenting Software Architectures: Views and Beyond | Paul Clements et al | Addison Wesley
Engineering by Design, 2nd Edition | Gerard Voland | Prentice Hall
Executable UML: A Foundation for Model-Driven Architecture | Stephen J. Mellor and Marc J. Balcer | Addison Wesley
Fundamental of Object-Oriented Design in UML, 2nd Edition | Meilir Page-Jones | Addison Wesley
Guide to the Software Engineering Body of Knowledge | CS Press
IEEE-CS/ACM Joint Task Force on Software Engineering Ethics and Professional Practices | IEEE/ACM
Managing and Leading Software Projects | Richard E. Fairley | CS Press
Metrics and Models in Software Quality Engineering, 2nd Edition | Stephen H. Kan | Addison Wesley
Numerical Mathematics and Computing, 6th Edition | E. Ward Cheney and David R. Kincaid | Addison Wesley
Operating System Concepts, 8th Edition | Abraham Silberschatz, Peter Baer Galvin, and Greg Gagne | Wiley
Professional Issues in Software Engineering | Frank Bott, Allison Coleman, Jack Eaton, and Diane Rowland | Taylor and Francis
Return on Software: Maximizing the Return on Your Software Investment | Steve Tockey | Addison Wesley
Security-Oriented Pointwise Changes to SWEBOK Guide | Samuel Redwine
Software Design, 2nd Edition | David Budgen | Addison Wesley
Software Engineering, 8th Edition | Ian Sommerville | Addison Wesley
Software Inspection | Tom Gilb and D. Graham | Addison Wesley
Software Maintenance: Concepts and Practice, 2nd Edition | Penny Grubb and Armstrong A. Takang | World Scientific Publishing
Software Security Engineering: A Guide for Project Managers | Julia H. Allen, Sean Barnum, Robert J. Ellison, Gary McGraw, and Nancy R. Mead | Addison Wesley
The Essentials of Computer Organization and Architecture, 2^nd Edition | Linda Null and Julia Lobur | Jones & Bartlett
The Road Map to Software Engineering: A Standards-Based Guide | James W. Moore | CS Press
Usability Engineering | Jakob Nielsen | Morgan Kaufmann

Hope you all the best,

Bugs in the generated code of Repository Factory for Visual Studio 2008

noreply@blogger.com (Ebeid Soliman) — Fri, 26 Mar 2010 14:54:00 +0000

If you are considering using repository factory for visual studio 2008 in your next project, please be aware that it had 2 bugs in code generation. These are in the generated Remove method in the repository classes of the DAL. This happens only if the corresponding table is a bridge table that links two tables together.

If you had simple three tables in your database like the following:

The generated Remove method in User_RoleRepository class will be like the following:

public void Remove(System.Int32 role_IDSystem.Int32 user_ID)
{ 
   IDeleteFactory<DeleteUser_RoleIdentity> deleteFactory = new 
   User_RoleDeleteFactory(); 
   try 
     { 
      DeleteUser_RoleIdentity deleteUser_RoleIdentity = new 
         DeleteUser_RoleIdentity(role_IDField, user_IDField); 
      base.Remove(deleteFactory, deleteUser_RoleIdentity); 
     } 
   catch (SqlException ex) 
     { 
      HandleSqlException(ex, deleteFactory); 
     }            
    }

The parameter separator had been omitted

The parameters passed to deleteUser_RoleIdentity had been suffixed by “Field” although it should be the same name as the passed parameters.

Take care of these 2 bugs in the generated code, you have to correct them in every class.

Wisdom of Experts: Programming in vs. Programming into a language

noreply@blogger.com (Ebeid Soliman) — Wed, 26 Aug 2009 05:32:00 +0000

David Gries (1981) pointed out that “your programming tools don’t have to determine how you think about programming”. He also made a distinction between programming in a language vs. programming into a language.

Programmers who program “in” a language limit their thoughts to constructs that the language directly supports. If the language tools are primitive, the programmers thoughts will be also be primitive.

Programmers who program “into” a language, first decide what thoughts they want to express, and then they determine how to express thoughts using the tools provided by their specific language.

So, try as much as you can to program into the language you use. If your language lacks constructs that you want to to use or is prone to other kinds of problems, try to compensate for them. Invent your own coding conventions, standards, class libraries, and other augmentations.

Using Python Scripts with IIS 7

noreply@blogger.com (Ebeid Soliman) — Sat, 15 Aug 2009 05:31:00 +0000

Here are the steps you should make if you want to use python scripts on IIS 7:

Please make sure Python is installed properly or refer to Python Notes – 1 : Setup for installation steps.
Make sure CGI module is installed in IIS 7.

Control Panel -> Programs -> Program and Features -> Turn Windows features on and off -> Internet Information Services -> World Wide Web Services -> Application Development Features -> CGI module (Ensure that it is selected).

Add web application for Python, In IIS Manager, right click Sites –> Add Web Site..

In the Add Web Site dialog box, set Site name e.g.: PythonTest, and make it pointing to some folder like C:\PythonTest, then click OK

In Features View, open Handler Mappings

On the left pane click Add Script Map ...

In Request path, put "*.py" as the script files extension, In Executable select "C:\Python25\Python.exe %s %s", or change it according to your Python installation path. (The two "%s" after the executable are required for console-based script interpreters but would not be required for an Internet Server API [ISAPI]-based script interpreter). Then give the script mapping an appropriate Name, like Python. Click OK.

Create a test.py into the virtual directory (C:\PythonTest), and copy the following script into it.

print

print 'Status: 200 OK'

print 'Content-type: text/html'

print

print '<HTML><HEAD><TITLE>Python Sample CGI</TITLE></HEAD>' print '<BODY>'

print '<H1>This is a header</H1>'

print '<p>' #this is a comment

print 'See this is just like most other HTML'

print '<br>'

print '</BODY>'

Then type http://localhost/pythontest/test.py into your IE URL bar to verify it is working.

Introduction to Peer-to-Peer Computing

noreply@blogger.com (Ebeid Soliman) — Mon, 25 May 2009 04:10:00 +0000

Peer-to-peer (P2P) has become a buzzword that subsumes concepts used in communication technologies, distributed system models, applications, platforms, etc. It is not a particular initiative, nor architecture or specific technology; it rather describes a set of concepts and mechanisms for decentralized distributed computing and direct peer-to-peer information and data interchange.

So, What is Peer-to-Peer Computing ?

There is no one definition of P2P.

P2P is a class of applications that takes advantages of resources available at the edges of the internet.
The sharing of computer resources and services by direct exchange between systems.
In Peer-to-Peer computing, each computer can serve the function of both client and server. Any computer in the network can initiate questions (as a client), receive questions and transmit data (as a server).

We could conclude these definitions by proposing the following definition.

P2P computing is a distributed computing architecture with the following properties:

Resource sharing
Dual client/server role of network nodes.
Decentralization/Autonomy
Scalability
Robustness/Self-Organization

Once you understood that P2P is a distributed computing architecture, you might be started asking: Why P2P architecture since we have Client/Server architecture ?

Client/Server architecture is a well known, powerful architecture. In it, the server is the data and functionality source, and clients requests data and processing from it. Although Client/Server architecture is very successful (www [HTTP], FTP, Web services, etc…), it have some issues; which are:

Scalability is hard to achieve.
Presents a single point of failure.
Requires administration.
Didn’t utilize the resources at the network edges (clients).

Although many of these issues had been solved with many solutions, P2P tried to address these issues in a new and very different way. The following figure emphasizes the general difference between P2P and client/server architecture models.

Why P2P Now ?

Since the concepts that underlie the P2P computing model is not a new computing concept, it is reasonable to ask why the P2P model has burst on the scene at this time. This happened due to many recent changes in the computing world:

The ubiquity of connected computers has come close to enabling anywhere, anytime access to the Net and its resources.
The critical mass of computer users.
Improvements in communications bandwidth, still on a fast-track growth curve, make it possible to move large amounts of data and rich media content from one location to another.
Today’s PCs are sufficiently robust, in terms of processing power and storage capacity, to handle the extra services required in a P2P environment.
the emergence of complementary technologies, including recent advances in wireless and software agents that provide more avenues for interesting P2P applications.

While all of these are necessary conditions for P2P computing, something more is required. History shows that a trigger is needed for a new technology to really take off. In the computer industry this is often referred to as a “killer app” The electronic spreadsheet triggered the proliferation of the PC, and the Mosaic browser triggered the transformation of the Internet into the World Wide Web.

The P2P computing model found its trigger with Napster, followed by Gnutella. Their huge popularity got everyone talking about P2P and helped further stimulate other P2P applications such as Freenet and SETI@home.

P2P Benefits

Efficient use of resources.
- Unused bandwidth, storage, processing power at the edge of the network.
Scalability
- Consumers of resources also donate resources.
- Aggregate resources grow naturally with utilization.
Reliability
- Replicas of data.
- Geographic distribution
- No single point of failure.
Ease of administration
- Nodes are self organize.
- No need to deploy servers to satisfy demand.
- Built-in fault tolerance, replication, and load balancing.

Since P2P is not well-defined and it is only defined by set of characteristics and properties that are attributed to P2P systems, we will discuss some of common characteristics of P2P systems. This does not imply that every P2P system has to comply with all of these characteristics or even with a fixed number of them. These are general features that can be used to identify P2P systems.

Structural Characteristics
1. Decentralization
  1. This includes distributed storage, processing, information sharing, etc..
  2. Advantage
    1. increased extensibility.
    2. Higher system availability and resilience.
  3. Disadvantage
    1. Difficult to get or maintain a global view of the system state.
    2. System behavior is not deterministic.
    3. Interoperability is a big issue.
2. Self-organizing
  1. The different system components work together without any central management instance assigning roles and tasks.
  2. Disadvantage
    1. It is difficult to determine the system structure or predict the system behavior as long as no system-wide, governing policies apply.
3. Fault-tolerance
  1. Since there is no central point of failure.
    1. The system performance variance due to nodes leaves and joins might imply that there is a lack of consistency.
Operational Characteristics
1. Transparency
  1. This means transparency to the application or user in forms of communications, location, access, replication transparency for data and information. Also the scale of system should be kept transparent to the user.
  2. This requires a middleware layer.

P2P Applications

P2P is good for:

File sharing (Napster, Gnutella, Kazza)
Multiplayer games (Unreal Tournament, DOOM)
Collaborative applications (ICQ, share whiteboard)
Distributed computation (Seti@home)
Ad-hoc networks

P2P can be applied in many areas, and there are massive work in many areas to utilize the P2P concepts and techniques.

Python Notes 14: Advanced Network Operations

noreply@blogger.com (Ebeid Soliman) — Sat, 16 May 2009 02:07:00 +0000

We have explored the usual issues in network programming, both on client side and server side. In this post we will discuss some advanced topics in network programming.

Half-Open Sockets
Normally, sockets are bidirectional—data can be sent across them in both directions. Sometimes, you may want to make a socket be unidirectional so data can only be sent in one direction. A socket that's unidirectional is said to be a half-open socket. A socket is made half-open by calling shutdown(), and that procedure is irreversible for that socket. Half-open sockets are useful when

You want to ensure that all data written has been transmitted. When shutdown() is called to close the output channel of a socket, it will not return until all buffered data has been successfully transmitted.
You want to have a way to catch potential programming errors that may cause the program to write to a socket that shouldn't be written to, or read from a socket that shouldn't be read from.
Your program uses fork() or multiple threads, and you want to prevent other processes or threads from doing certain operations, or you want to force a socket to be closed immediately.

The socket. shutdown() call is used to accomplish all of these tasks.

The call to shutdown() requires a single argument that indicates how you want to shut down the socket. Its possible values are as follows:

0 to prevent future reads
1 to prevent future writes
2 to prevent future reads and writes

Once shut down in a given direction, the socket can never be reopened in that direction. Calls to shutdown() are cumulative; calling shutdown(0) followed by shutdown(1) will achieve the same effect as calling shutdown(2).

Timeouts

TCP connections can be held open indefinitely, even if there's no traffic flowing across them. Timeouts are useful
for discovering error conditions or communication problems in some instances.

To enable timeout detection on a Python socket, you call settimeout() on the socket, passing it the number of seconds until a timeout is reached. Later, when you make a socket call and nothing has happened for that amount of time, a socket.timeout exception is raised.

Transmitting Strings
One common problem that arises when sending data across the network is that of transmitting variable-length strings. When you read information from a TCP stream you don't know when the sender has finished giving you a piece of data unless you build some sort of indication into your protocol. There are two common approaches to solving this problem:

End-of-string identifier

Terminate the string with ‘\n’ or NULL
Problem: Terminator might occur in the data if we transmit binary data.
Solutions:

Escape the identifier.
Encode data in base64
use different if found in data and send the new identifier before the data.

Leading fixed-length size indicator

Send a constant number of bytes containing the size of the string.
The “size” itself could be sent as characters or as binary data, characters are simpler, however you have to pad them to get a constant length.

Using Broadcast Data

When you broadcast a UDP packet, it's sent to all machines
connected to your LAN. The underlying transport, such as Ethernet, will have a special mode that lets you do this without having to repeat the packet for each computer.
On the receiver's side, when a broadcast packet is received, the kernel looks at the destination port number. If it has a process listening to that port, the packet is sent to that process. Otherwise, it's silently discarded. Therefore, simply sending out a broadcast packet will not harm or impact machines that don't have a server listening for it.
Broadcast packets are often used for the following types of activities:

Automatic service discovery: For instance, a computer might send out a broadcast packet looking for all print servers of a particular type.
Automatic service announcements: A server providing a service for a LAN might periodically broadcast the availability of that service. Clients would listen for those broadcasts.
Searching for LAN computers that implement a specific protocol. For instance, a chat program might send out a broadcast packet looking for other people on the LAN with the same chat program. It might then compile a list and present it to the user.

To be able to broadcast data, you need to set the socket option on client and server as follows:

s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)

On the sender, instead of sending to a particular IP, send to ‘<broadcast>’

s.sendto(‘<broadcast>’,123)

In this post we dealt with a few advanced issues in network programming.

Python Notes 13: Network servers

noreply@blogger.com (Ebeid Soliman) — Fri, 15 May 2009 23:38:00 +0000

For a client, the process of establishing a TCP connection is a two-step process that includes the creation of the socket object and a call to connect() to establish a connection to the server. For a server, the process requires the following four steps:

Create the socket object.
Set the socket options (optional).
Bind to a port (and, optionally, a specific network card).
Listen for connections.

Example of these steps:

host = '' # Bind to all interfaces
port = 51423
# Step 1 (Create the socket object)
s = socket. socket(socket.AF_INET, socket.SOCK_STREAM)
# Step 2 (Set the socket options)
s.setsockopt(socket.SOL_SOCKET, socket.SOREUSEADDR, l)
# Step 3 (Bind to a port and interface)
s.bind((host, port))
# Step 4 (Listen for connections)
s.listen(5)

Setting and Getting Socket Options
There are many different options that can be set for a socket. For general-purpose servers, the socket option of greatest interest is called SOREUSEADDR. Normally, after a server process terminates, the operating system reserves its port for a few minutes, thereby preventing any other process (even another instance of your server itself) from opening it until the timeout expires. If you set the SOREUSEADDR flag to true, the operating system releases the server port as soon as the server socket is closed or the server process terminates.This is done through:

s.setsockopt(socket.SOL_SOCKET, socket.SOJEUSEADDR, l)

Binding the Socket

The next step is to claim a port number for the server. This process is called binding. To bind to a port, you call:
s.bind((‘’, 111))

The first argument to bind() specifies the IP address to bind to it. It's generally left blank, which means "bind to all interfaces and addresses."

Listening for Connections
The last step before actually accepting client connections is to call listen(). This call tells the operating system to prepare to receive connections. It takes a single parameter, which indicates how many pending connections the operating system should allow to remain in queue before the server actually gets around to processing them.

Accepting Connections

Most servers designed to run indefinitely and service multiple connections, this is usually done with a carefully designed infinite loop. Example:

import socket
host = '' # Bind to all interfaces
port = 51423
s = socket.socket(socket.AF_INET, socket.SOCKJTREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, l)
s.bind((host, port))
print "Waiting for connections..."
s.listen(l)
while l:
    clientsock, clientaddr = s.acceptQ
    print "Got connection from", clientsock.getpeername()
    clientsock.close()

Using User Datagram Protocol

To use UDP on the server, you create a socket, set the options, and bind () just like with TCP However, there's no need for listen () or accept ()—just use recvf rom().
This function actually returns two pieces of information: the received data, and the address and port number of the program that sent the data. Because UDP is connectionless, this is all you need to be able to send back a reply. Example, echo server:

import socket, traceback
host = '' # Bind to all interfaces
port = 51423
s = socket.socket(socket.AF_INET, socket.SOCK_DCRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, l)
s.bind((host, port))
while l:
    try:
        message, address = s.recvfrom(8l92)
        print "Cot data from", address
        s.sendto(message, address)   #Echo it back
    except (Keyboardlnterrupt, SystemExit):
        raise
    except:
        traceback.print_exc()

In this post we have clarified some points in network servers.

Python Notes 12 : Network clients

noreply@blogger.com (Ebeid Soliman) — Fri, 15 May 2009 23:17:00 +0000

After we have explored the basics of network programming in brief in the previous post, we will discuss network clients in more details in this post.

Understanding Sockets
Sockets are an extension to the operating system's I/O system that enable communication between processes and machines. It can be treated the same as standard files, with the same interface methods so in many cases, a program need not know whether it's writing data to a file, the terminal, or a TCP connection. While many files are opened with the open () call, sockets are created with the socket () call and additional calls are needed to connect and activate them.

Creating Sockets

For a client program, creating a socket is generally a two-step process.

Create the actual socket object.
Connect the socket to the remote server.

When you create a socket object, you need to tell the system two things:

The communication type: the underlying protocol used to transmit data. Examples of protocols include IPv4 (current Internet standard), IPv6 (future Internet standard), IPX/ SPX (NetWare), and AFP (Apple file sharing). By far the most common is IPv4.
The protocol family: defines how data is transmitted.
For Internet communications, which make up the bulk of this book, the communication type is almost always AF_INET (corresponding to IPv4). The protocol family is typically either:
- SOCK_STREAM for TCP communications or
- SOCK_DGRAM for UDP communications

For a TCP connection, creating a socket generally uses
code like this:

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) To connect the socket, you'll generally need to provide a tuple containing the remote hostname or IP address and the remote port. Connecting a socket typically looks like this:
s.connect(("www.example.com", 80))

Finding the port number

Most operating systems ship with a list of well-known server port numbers which you can query. On windows systems, you can find this file at C:\Windows\System32\drivers\etc\services. To query this list, you need two parameters:

A protocol name
A port name.

This query is like:

>>>print socket.getservbyname(‘ftp’,’tcp’)

21

You didn't have to know in advance that FTP uses port 80.

Getting Information from a Socket
Once you've established a socket connection, you can find out some useful information from it.

s.getsockname() #Get your IP address and port number

s.getpeername() #Get the remote machine IP address and port number

Socket Exceptions

Different network calls can raise different exceptions when network errors occur. Python's socket module actually defines four possible exceptions:

socket.error for general I/O and communication problems.
socket.gaierror for errors looking up address information
socket.herror for other addressing errors.
socket.timeout for handling timeouts that occur after settimeout() has been called on a socket.

Complete Example

The example program takes three command-line arguments: a host to which it will connect, a port number or name on the server, and a file to request from the server. The program will connect to the server, send a simple HTTP
request for the given filename, and display the result. Along the way, it exercises care to handle various types of potential errors.

import socket, sys
host = sys.argv[l]
textport = sys.argv[2]
filename = sys.argv[3]
try:
    s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
except socket.error, e:
    print "Strange error creating socket: %s" % e
    sys.exit(l)
    # Try parsing it as a numeric port number.
try:
    port = int(textport)
except ValueError:
    # That didn't work, so it's probably a protocol name.
    # Look it up instead,
try:
    port = socket.getservbyname(textport, 'tcp')
    except socket.error, e:
    print "Couldn't find your port: %s" % e
    sys.exit(i)

try:
    s.connect((host, port))
except socket.gaierror, e:
    print "Address-related error connecting to server: %s" % e
    sys.exit(i)
except socket.error, e:
    print "Connection error: %s" % e
    sys.exit(l)
try:
    s.sendall("CET %s HTTP/1.0\r\n\r\n" % filename)
except socket.error, e:
    print "Error sending data: %s" % e
    sys.exit(i)
while 1:
    try:
        buf = s.recvB048)
    except socket.error, e:
        print "Error receiving data: %s" % e
        sys.exit(l)
    if not len(buf):
        break
    sys.stdout.write(buf)

Using User Datagram Protocol

In UDP there is no sufficient control over how data is sent and received. Working with UDP clients differs than TCP clients in the following:

When create the socket ask for SOCKDGRAM
instead of SOCKSTREAM; this indicates to the operating system that the socket will
be used for UDP instead of TCP communications.
When call socket.getservbyname(), pass ‘udp’ instead of ‘tcp’.

In this post we discussed network clients in a little bit depth. In the next post we will discuss network servers.

Python Notes 11 : Introduction to Network Programming

noreply@blogger.com (Ebeid Soliman) — Fri, 15 May 2009 08:00:00 +0000

Network Overview

Python provides a wide assortment of network support.

Low-level programming with sockets (if you want to create a protocol).
Support for existing network protocols (HTTP, FTP, SMTP, etc...).
Web programming (CGI scripting and HTTP servers).
Data encoding

Network Basics: TCP/IP

Python’s networking modules primarily support TCP/IP.

TCP - A reliable connection-oriented protocol (streams).
UDP - An unreliable packet-oriented protocol (datagrams).

TCP is the most common (HTTP, FTP, SMTP, etc...). Both protocols are supported using "sockets".

A socket is a file-like object. Allows data to be sent and received across the network like a file. But it also includes functions to accept and establish connections. Before two machines can establish a connection, both must create a socket

Network Basics: Ports

In order to receive a connection, a socket must be bound to a port (by the server). A port is a number in the range 0-65535 that’s managed by the OS. Used to identify a particular network service (or listener). Ports 0-1023 are reserved by the system and used for common protocols:

FTP Port 20
Telnet Port 23
SMTP (Mail) Port 25
HTTP (WWW) Port 80

Ports above 1024 are reserved for user processes.

Socket programming in a nutshell

Server creates a socket, binds it to some well-known port number, and starts listening.
Client creates a socket and tries to connect it to the server (through the above port).
Server-client exchange some data.
Close the connection (of course the server continues to listen for more clients).

Socket Programming Example

The socket module

Provides access to low-level network programming functions. The following example is a simple server that returns the current time

import time, socket

s = socket(AF_INET, SOCK_STREAM)#Create TCP socket

s.bind(("",8888)) #Bind to port 8888

s.listen(5) #Start listening

while 1:

client,addr = s.accept() #Wait for a connection

print "Got a connection from ", addr

client.send(time.ctime(time.time())) #Send time back

client.close()

Notes: The socket first opened by server is not the same one used to exchange data.Instead, the accept() function returns a new socket for this (’client’ above).listen() specifies max number of pending connections

The following example is the client program for the above time server which connect to time server and get current time.

from socket import *

s = socket(AF_INET,SOCK_STREAM) #Create TCP socket

s.connect(("google.com",8888)) #Connect to server

tm = s.recv(1024) #Receive up to 1024 bytes

s.close() # Close connection

print "The time is", tm

Notes: Once connection is established, server/client communicate using send() and recv(). Aside from connection process, it’s relatively straightforward. Of course, the devil is in the details. And are there ever a LOT of details.

The Socket Module

The socket module used for all low-level networking, creation and manipulation of sockets, and general purpose network functions (hostnames, data conversion, etc...). It’s a direct translation of the BSD socket interface.

Utility Functions

socket.gethostbyname(hostname) # Get IP address for a host
socket.gethostname() # Name of local machine
socket.ntohl(x) # Convert 32-bit integer to host order
socket.ntohs(x) # Convert 16-bit integer to host order
socket.htonl(x) # Convert 32-bit integer to network order
socket.htons(x) # Convert 16-bit integer to network order

Comments: Network order for integers is big-endian. Host order may be little-endian or big-endian (depends on the machine).

The socket(family, type, proto) function creates a new socket object. Family is usually set to AF_INET. Type is one of:

SOCK_STREAM Stream socket (TCP)
SOCK_DGRAM Datagram socket (UDP)
SOCK_RAW Raw socket

Proto is usually only used with raw sockets:

IPPROTO_ICMP
IPPROTO_IP
IPPROTO_RAW
IPPROTO_TCP
IPPROTO_UDP

Socket methods

s.accept() # Accept a new connection
s.bind(address) # Bind to an address and port
s.close() # Close the socket
s.connect(address) # Connect to remote socket
s.fileno() # Return integer file descriptor
s.getpeername() # Get name of remote machine
s.getsockname() #Get socket address as (ipaddr,port)
s.getsockopt(...) # Get socket options
s.listen(backlog) # Start listening for connections
s.makefile(mode) # Turn socket into a file like object
s.recv(bufsize) # Receive data
s.recvfrom(bufsize) # Receive data (UDP)
s.send(string) # Send data
s.sendto(string, address) # Send packet (UDP)
s.setblocking(flag) #Set blocking or nonblocking mode
s.setsockopt(...) #Set socket options
s.shutdown(how) #Shutdown one or both halves of connection

There are a huge variety of configuration/connection options. You’ll definitely want a good reference at your side

The SocketServer Module

Provides a high-level class-based interface to sockets. Each protocol is encapsulated in a class (TCPServer, UDPServer, etc.). It also provides a series of handler classes that specify additional server behavior.

To create a network service, need to inherit from both a protocol and handler class. Example, the same time server we done before:

import SocketServer

import time

# This class actually implements the server functionality

class TimeHandler(SocketServer.BaseRequestHandler):

def handle(self):

self.request.send(time.ctime(time.time()))

# Create the server

server = SocketServer.TCPServer("",8888),TimeHandler)

server.serve_forever()

Notes: The module provides a number of specialized server and handler types. Ex: ForkingTCPServer, ThreadingTCPServer, StreamRequestHandler, etc.

Common Network Protocols

Modules are available for a variety of network protocols:

ftplib FTP protocol
smtplib SMTP (mail) protocol
nntplib News
gopherlib Gopher
poplib POP3 mail server
imaplib IMAP4 mail server
telnetlib Telnet protocol
httplib HTTP protocol

These modules are built using sockets, but operate on a very low-level. Working with them requires a good understand of the underlying protocol. But can be quite powerful if you know exactly what you are doing

The httplib Module

Implements the HTTP 1.0 protocol and can use to talk to a web server.

HTTP in two bullets:

Client (e.g., a browser) sends a request to the server

GET /index.html HTTP/1.0

Connection: Keep-Alive

Host: www.python.org

User-Agent: Mozilla/4.61 [en] (X11; U; SunOS 5.6 sun4u)

[blank line]

Server responds with something like this:

HTTP/1.0 200 OK

Content-type: text/html

Content-length: 72883

Headers: blah

[blank line]

Data

...

Making an HTTP connection

import httplib

h = httplib.HTTP("www.python.org")

h.putrequest(’GET’,’/index.html’)

h.putheader(’User-Agent’,’Lame Tutorial Code’)

h.putheader(’Accept’,’text/html’)

h.endheaders()

errcode,errmsg, headers = h.getreply()

f = h.getfile() # Get file object for reading data

data = f.read()

f.close()

You should understand some HTTP to work with httplib.

The urllib Module

A high-level interface to HTTP and FTP which provides a file-like object that can be used to connect to remote servers

import urllib

f = urllib.urlopen("http://www.python.org/index.html")

data = f.read()

f.close()

Utility functions

urllib.quote(str) # Quotes a string for use in a URL
urllib.quote_plus(str) # Also replaces spaces with ’+’
urllib.unquote(str) # Opposite of quote()
urllib.unquote_plus(str) # Opposite of quote_plus()
urllib.urlencode(dict)
# Turns a dictionary of key=value pairs into a HTTP query-string

Examples

urllib.quote("ebeid@ieee") #Produces "ebeid%40ieee"

urllib.unquote("%23%21/bin/sh") #Produces "/bin/sh"

The urlparse Module

Contains functions for manipulating URLs

URL’s have the following general format

scheme:/netloc/path;parameters?query#fragment

urlparse(urlstring) - Parses a URL into components

import urlparse

t = urlparse.urlparse("http://www.python.org/index.html")

#Produces (’http’,’www.python.org’,’/index.html’,’’,’’,’’)

urlunparse(tuple) - Turns tuple of components back into a URL string

url = urlparse.urlunparse((’http’,’www.python.org’,’foo.html’, ’bar=spam’,’’))

# Produces "http://www.python.org/foo.html?bar=spam"

urljoin(base, url) - Combines a base and relative URL

urlparse.urljoin("http://www.python.org/index.html","help.html")

# Produces "http://www.python.org/help.html"

In this note we explored horizontally the network programming capabilities of Python. Every single module and topic mentioned here, needs multiple posts to cover it. In the upcoming posts, we will dig into network programming in detail.

Python Notes – 10 : Threading

noreply@blogger.com (Ebeid Soliman) — Sat, 28 Mar 2009 12:55:00 +0000

Welcome to our tenth note in the Python learning process. In this note we will talk about threading, threads communication and synchronization.

Threads basics

A running program is called a "process". Each process has memory, list of open files, stack, program counter, etc…. Normally, a process executes statements in a single sequence of control-flow.

The following commands create an entirely new process: fork(),system(), popen(), etc… This child process runs independently of the parent. Has own set of resources. There is minimal sharing of information between parent and child.

On the other side, a thread is kind of like a process (it’s a sequence of control-flow). Except that it exists entirely inside a process and shares resources. A single process may have multiple threads of execution. This is extremely useful when an application wants to perform many concurrent tasks on shared data.

Problems with Threads

Scheduling : To execute a threaded program, must rapidly switch between threads. This can be done by the user process (user-level threads) or Can be done by the kernel (kernel-level threads).
Resource Sharing : Since threads share memory and other resources, you must be very careful. Operation performed in one thread could cause problems in another.
Synchronization : Threads often need to coordinate actions because they can get "race conditions" (outcome dependent on order of thread execution). You will often need to use locking primitives (mutual exclusion locks, semaphores, etc...)

Python Threads

Python supports threads on the following platforms : Solaris, Windows, systems that support the POSIX threads library (pthreads).

Thread scheduling is tightly controlled by a global interpreter lock and scheduler. Only a single thread is allowed to be executing in the Python interpreter at once. Thread switching only occurs between the execution of individual byte-codes. Long-running calculations in C/C++ can block execution of all other threads. However, most I/O operations do not block.

Python threads are somewhat more restrictive than in C. Effectiveness may be limited on multiple CPUs (due to interpreter lock). Threads can interact strangely with other Python modules (especially signal handling). Not all extension modules are thread-safe.

The thread module

The thread module provides low-level access to threads, thread creation, and Simple mutex locks.

Creating a new thread

Thread.start_new_thread(func,[args [,kwargs]]) Executes a function in a new thread. Syntax like:

import thread

import time

def print_time(delay):

while 1:

time.sleep(delay)

print time.ctime(time.time())

thread.start_new_thread(print_time,(5,)) # Start the thread

# Go do something else

statements

…….

The function print_time will execute in a separate thread and will continue printing the time every 5 seconds. Python will continue executing our statements also.

Thread termination

Thread silently exits when the function returns. Thread can explicitly exit by calling thread.exit() or sys.exit(). Also uncaught exception causes thread termination (and prints error message). Other threads continue to run even if one had an error.

Simple locks

allocate_lock(). Creates a lock object, initially unlocked. Only one thread can acquire the lock at once. Threads block indefinitely until lock becomes available.

import thread

lk = thread.allocate_lock()

def foo():

lk.acquire() # Acquire the lock

… #critical section

lk.release() # Release the lock

You might use this if two or more threads were allowed to update a shared data structure.

The main thread

When Python starts, it runs as a single thread of execution. This is called the "main thread." which is on its own and it’s not a big deal. However, if you launch other threads it has some special properties.

Termination of the main thread

If the main thread exits and other threads are active, the behavior is system dependent. Usually, this immediately terminates the execution of all other threads without cleanup. Cleanup actions of the main thread may be limited as well.

Signal handling

Signals can only be caught and handled by the main thread of execution. Otherwise you will get an error (in the signal module). The keyboard-interrupt can be caught by any thread (non-deterministically).

The threading module

The threading module is a high-level threads module that implements threads as classes (similar to Java). It provides an assortment of synchronization and locking primitives. It is built using the low-level thread module.

Creating a new thread (as a class)

When defining threads as classes all you need to supply is the following:

A constructor that calls threading.Thread.__init__(self)
A run() method that performs the actual work of the thread.

A few additional methods are also available

t.join([timeout]) # Wait for thread t to terminate
t.getName() # Get the name of the thread
t.setName(name) # Set the name of the thread
t.isAlive() # Return 1 if thread is alive.
t.isDaemon() # Return daemonic flag
t.setDaemon(val) # Set daemonic flag

Example: Inherit from the "Thread" class, provide required methods, and utilize the available methods.

import threading, time

class PrintTime(threading.Thread):

def __init__(self,interval):

threading.Thread.__init__(self) # Required

self.interval = interval

def run(self):

while 1:

time.sleep(self.interval)

print time.ctime(time.time())

t = PrintTime(5) # Create a thread object

t.start() # Start it

Daemon threads

Normally, interpreter exits only when all threads have terminated. However, a thread can be flagged as a daemon thread (runs in background). Interpreter really only exits when all non-daemonic threads exit. You can use this to launch threads that run forever, but which can be safely killed.

Threads synchronization

The threading module provides the following synchronization primitives:

Mutual exclusion locks
Reentrant locks
Conditional variables
Semaphores
Events

When would you need these threads synchronization mechanisms ?

When threads are updating shared data structures.
When threads need to coordinate their actions in some manner (events).

The Lock object

Provides a simple mutual exclusion lock. Only one thread is allowed to acquire the lock at once. Most useful for coordinating access to shared data.

import threading

data = [ ] # Some data

lck = threading.Lock() # Create a lock

def put_obj(obj):

lck.acquire()

data.append(obj)

lck.release()

def get_obj():

lck.acquire()

r = data.pop()

lck.release()

return r

The RLock object

A mutual-exclusion lock that allows repeated acquisition by the same thread. Allows nested acquire(), release() operations in the thread that owns the lock. Only the outermost release() operation actually releases the lock.

import threading

data = [ ] # Some data

lck = threading.Lock() # Create a lock

def put_obj(obj):

lck.acquire()

data.append(obj)

...

put_obj(otherobj) # Some kind of recursion

...

lck.release()

def get_obj():

lck.acquire()

r = data.pop()

lck.release()

return r

The Condition object

Creates a condition variable. Synchronization primitive typically used when a thread is interested in an event or state change. Could help in the producer-consumer classic problem.

data = [] # Create data queue and a condition variable

cv = threading.Condition()

# Consumer thread

def consume_item():

cv.acquire() # Acquire the lock

while not len(data):

cv.wait() # Wait for data to show up

r = data.pop()

cv.release() # Release the lock

return r

# Producer thread

def produce_item(obj):

cv.acquire() # Acquire the lock

data.append(obj)

cv.notify() # Notify a consumer

cv.release() # Release the lock

Semaphores

A locking primitive based on a counter. Each acquire() method decrements the counter. Each release() method increments the counter. If the counter reaches zero, future acquire() methods block. Common use: limiting the number of threads allowed to execute code

sem = threading.Semaphore(5) # No more than 5 threads allowed

def fetch_file(host,filename):

sem.acquire() # Decrements count or blocks if zero

...

sem.release() # Increment count

Events

A communication primitive for coordinating threads. One thread signals an "event" while other threads wait for it to happen.

e = Event() # Create an event object

def signal_event(): # Signal the event

e.set()

def wait_for_event(): # Wait for event

e.wait()

def clear_event(): # Clear event

e.clear()

Event is similar to a condition variable, but all threads waiting for event are awakened.

Locks and Blocking

By default, all locking primitives block until lock is acquired. In general, this is uninterruptible. Fortunately, most primitives provide a non-blocking option

if not lck.acquire(0):

# lock couldn’t be acquired!

This works for Lock, RLock, and Semaphore objects. On the other hand condition variables and events provide a timeout option

cv = Condition()

...

cv.wait(60.0) # Wait 60 seconds for notification

On timeout, the function simply returns. Up to caller to detect errors.

The Queue Module

Provides a multi-producer, multi-consumer FIFO queue object. It can be used to safely exchange data between multiple threads.

q = Queue(maxsize) # Create a queue.
q.qsize() # Return current size.
q.empty() # Test if empty.
q.full() # Test if full.
q.put(item) # Put an item on the queue.
q.get() # Get item from queue

The Queue object also supports non-blocking put/get.

q.put_nowait(item).
q.get_nowait()

These raise the Queue.Full or Queue.Empty exceptions if an error occurs. Return values for qsize(), empty(), and full() are approximate.

Things to consider when using threads

Global interpreter lock makes it difficult to fully utilize multiple CPUs.
You don’t get the degree of parallelism you might expect.
Not all modules are thread-friendly. Example: gethostbyname() blocks all threads if nameserver down.

In this note we will talked about threading, threads communication and synchronization. In the upcoming notes, we will talk about more advanced topics in Python programming.

Python Notes – 9 : Serialization

noreply@blogger.com (Ebeid Soliman) — Fri, 27 Mar 2009 06:53:00 +0000

Welcome to our ninth note in our Python learning process. We talked previously about files and how to handle it but we talked about writing and reading only the primitive data types as integers and strings. We also talked about objects and classes. Now, what if we want to write a compound data type or a complex object to a file. This note will talk about writing objects to files, which is called object serialization.

pickle

The pickle module is a Python built-in module that object serialization and de-serialization. To store a data structure, use the dump method and then close the file in the usual way:

>>> pickle.dump(12.3, f)

>>> pickle.dump([1,2,3], f)

>>> f.close()

Then we can open the file for reading and load the data structures we dumped:

>>> f = open("test.pck","r")

>>> x = pickle.load(f)

>>> x

12.3

>>> type(x)

<type 'float'>

>>> y = pickle.load(f)

>>> y

[1, 2, 3]

>>> type(y)

<type 'list'>

Each time we invoke load, we get a single value from the file, complete with its original type.

What can be serialized and de-serialized

The following types can be serialized and de-serialized using pickle:

None, True, and False
integers, long integers, floating point numbers, complex numbers
normal and Unicode strings
tuples, lists, sets, and dictionaries containing only picklable objects
functions defined at the top level of a module
built-in functions defined at the top level of a module
classes that are defined at the top level of a module
instances of such classes whose__dict__ or __setstate__() is picklable

Things to consider when using pickle

Attempts to pickle unpicklable objects will raise the picklingError exception; when this happens, an unspecified number of bytes may have already been written to the underlying file.
Trying to pickle a highly recursive data structure may exceed the maximum recursion depth, a RuntimeError will be raised in this case. You can carefully raise this limit with sys.setrecursionlimit().

cPickle

The cPickle is an optimized version of pickle written in C, so it can be up to 1000 faster than pickle.

marshal

The marshal module can also be used for serialization. Marshal is similar to pickle, but is intended only for simple objects. Can’t handle recursion or class instances. On the plus side, it’s pretty fast if you just want to save simple objects to a file. Data is stored in a binary architecture independent format.To serialize:

import marshal

marshal.dump(obj,file) # Write obj to file

To unserialize:

obj = marshal.load(file)

shelve

The shelve module provides a persistent dictionary. It is works like a dictionary, but data is stored on disk.

Keys must be strings. Data can be any object serializable with pickle.

import shelve

d = shelve.open("data") # Open a ’shelf’

d[’foo’] = 42 # Save data

x = d[’bar’] # Retrieve data

Shelf operations

d[key] = obj # Store an object

obj = d[key] # Retrieve an object

del d[key] # Delete an object

d.has_key(key) # Test for existence of key

d.keys() # Return a list of all keys

d.close() # Close the shelf

In this note will talked about writing objects to files, which is called object serialization. Object serialization is very useful in persisting your application logic to resume its execution later, transfer of execution to remote machine, and many other applications scenarios.

Python Notes – 8 : Object-Oriented Basics

noreply@blogger.com (Ebeid Soliman) — Fri, 27 Mar 2009 06:51:00 +0000

Welcome to our eighth note in our Python learning process. This note will talk about object oriented features in Python.

Classes and Objects

A class definition looks like this:

class Point:

pass

Class definitions can appear anywhere in a program, but they are usually near the beginning (after the import statements). By creating the Point class, we created a new type, also called Point. The members of this type are called instances of the type or objects. Creating a new instance is called instantiation. To instantiate a Point object, we call a function named Point:

>>> blank = Point()

The variable blank is assigned a reference to a new Point object. A function like Point that creates new objects is called a constructor. If you tried to get the type of blank, you got instance:

>>> type(blank)

<type 'instance'>

If you tried to print blank:

>>> print blank

<__main__.point instance at 0x01922AF8>

The result indicates that blank is an instance of the Point class and it was defined in __main__ . 0x01922AF8 is the unique identifier for this object, written in hexadecimal (base 16).

Attributes

We can add new data to an instance using dot notation:

>>> blank.x = 3.0

>>> blank.y = 4.0

These new data items called attributes.

>>> print blank.y

4.0

>>> x = blank.x

>>> print x

3.0

Sameness

To find out if two references refer to the same object, use the == operator. For example:

>>> p1 = Point()

>>> p1.x = 3

>>> p1.y = 4

>>> p2 = Point()

>>> p2.x = 3

>>> p2.y = 4

>>> p1 == p2

0

Even though p1 and p2 contain the same coordinates, they are not the same object. If we assign p1 to p2, then the two variables are aliases of the same object:

>>> p2 = p1

>>> p1 == p2

1

This type of equality is called shallow equality because it compares only the references, not the contents of the objects. To compare the contents of the objects - deep equality - we can write our own function to do that, like that:

def samePoint(p1, p2) :

return (p1.x == p2.x) and (p1.y == p2.y)

Now if we create two different objects that contain the same data, we can use samePoint to find out if they represent the same point.

>>> p1 = Point()

>>> p1.x = 3

>>> p1.y = 4

>>> p2 = Point()

>>> p2.x = 3

>>> p2.y = 4

>>> samePoint(p1, p2)

1

Copying

Aliasing can make a program difficult to read because changes made in one place might have unexpected effects in another place. Copying an object is often an alternative to aliasing. The copy module contains a function called copy that can duplicate any object:

>>> import copy

>>> p1 = Point()

>>> p1.x = 3

>>> p1.y = 4

>>> p2 = copy.copy(p1)

>>> p1 == p2

0

>>> samePoint(p1, p2)

1

Copy works fine for objects that doesn't contain any embedded objects. If the object contains references to other objects, Copy will copy the embedded references to the destination. This ends up that the both copies reference the same internal objects.

You can use deepcopy which copies not only the object but also any embedded objects.

>>> b2 = copy.deepcopy(b1)

Now b1 and b2 are completely separate objects.

The initialization method

The initialization method is a special method that is invoked when an object is created. The name of this method is __init__.

class point:

def __init__(self, x = 0, y = 0):

Self.x = x

Slef.y = y

When we invoke the point constructor, the arguments we provide are passed along to init:

>>> first = point(5,7)

>>> first.x

5

>>> first.y

7

Because the parameters are optional, we can omit them:

>>> second = point()

>>> second.x

0

>>> second.y

0

We can also provide a subset of the parameters by naming them explicitly:

>>> third = point(y=10)

>>> third.x

0

>>> third.y

10

The str method

The __str__ method of any class is called by the Python in any operation that requires the class instance to be converted to string. Operations like that are print. Syntax like that:

class xyz:

def __str__(self):

return "Our class xyz"

>>> a = xyz()

>>> a

<__main__.xyz instance at 0x02627300>

>>> print y

Our class xyz

Instances as parameters

You can pass an instance as a parameter in the usual way. For example:

def printPoint(p):

print '(' + str(p.x) + ', ' + str(p.y) + ')'

Instances as return values

Functions can return instances. For example:

def sumPoints(A,B)

Z = Point ()

Z.x = A.x + B.x

Z.y = A.y + B.y

return Z

Operator overloading

Operator overloading means changing the definition and behavior of the built-in operators when they are applied to user-defined types. For example, to override the addition operator + , we provide a method named __add__ in our point class :

class Point:

def __add__(self, other):

return Point(self.x + other.x, self.y + other.y)

the first parameter is the object on which the method is invoked. The second parameter is conveniently named other to distinguish it from self. Now, when we apply the + operator to Point objects, Python invokes add :

>>> p1 = Point(3, 4)

>>> p2 = Point(5, 7)

>>> p3 = p1 + p2

>>> print p3

(8, 11)

The expression p1 + p2 is equivalent to p1. add (p2), but obviously more elegant. You can change the behavior of many operators through overloading their respective functions, which are available at http://www.python.org/doc/2.2/ref/numeric-types.html

Inheritance

Inheritance is the ability to define a new class that is a modified version of an existing class. The new class inherits all of the methods of the existing class. The new class may be called child class or subclass. The syntax is like:

class class1(object):

K = 7

def __init__(self, color='green'):

Self.color = color

def Hello1(self):

Print "Hello from class1"

def printColor(self):

print "preferred ", self.color

class class2(class1):

def Hello2(self):

print "Hello from class2"

print self.k

Here class2 is the child of class1.

>>> c1 = class1('blue')

>>> c2 = class2('red')

>>> c1.Hello1()

Hello from class1

>>> c2.Hello2()

Hello from class2

7

Child class can access parent class methods

>>> c2.Hello1()

Hello from class1

The parent constructor called automatically for Childs, as following:

>>> c1.printColor()

preferred blue

>>> c2.printColor()

preferred red

You can check for class methods, attributes using hasattr method:

if hasattr(class1, "Hello2"):

print c1.Hello2()

else:

print "Class1 does not contain method Hello2()"

Class1 does not contain method Hello2()

To check the inheritance relation between two class :

if issubclass(class2, class1)

Print "Class2 is a subclass of Class1”

In this note we tried to cover as much as we can of the Python object oriented features. We give it a more advanced note in the future.