Midnight Researcher Notes

Syntax Modification using Microsoft Roslyn

noreply@blogger.com (Ebeid Soliman ElSayed) — Fri, 23 Aug 2013 22:56:00 +0000

In the previous two posts we just used the analysis capabilities of Microsoft Roslyn. In other words, we only get information about the code under inspection, we never changed it. In this post we going to change the code through Roslyn.

Before we start we need to highlight that data structures that represent code under inspection in Roslyn APIs are immutable. This means that it cannot be changed after they are created (in order to safely share them between tools). This applies to syntax trees, compilations, symbols, semantic models, and every other data structure in the Roslyn API. Instead of modification, new objects are created based on specified differences to the old ones.

Creating and Modifying Trees

We used SyntaxTree before to create syntax trees from a code snippet. For creating a syntax tree bit-by-bit, you have to compose syntax nodes hierarchically in a bottom-up fashion. To create SyntaxNodes you must use the Syntax class factory methods. For each kind of node, token, or trivia there is a factory method which can be used to create an instance of that type.

In the following example we going to replace a using directive by a new one we create.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Roslyn.Compilers;
using Roslyn.Compilers.CSharp;
using Roslyn.Services;
using Roslyn.Services.CSharp;

namespace RoslynDemo4
{
    class Program
    {
        static void Main(string[] args)
        {
            SyntaxTree tree = SyntaxTree.ParseText(
@"using System;
using System.Collections;
using System.Linq;
using System.Text;
 
namespace HelloWorld
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine(""Hello, World!"");
        }
    }
}");

            var root = (CompilationUnitSyntax)tree.GetRoot();
            // Get the old Using directive [using System.Collections;]
            var oldUsing = root.Usings[1];

            // Construct the new Using directive
            NameSyntax name = Syntax.IdentifierName("System");
            name = Syntax.QualifiedName(name, Syntax.IdentifierName("Collections"));
            name = Syntax.QualifiedName(name, Syntax.IdentifierName("Generic"));
            var newUsing = oldUsing.WithName(name);

            // Replace the old Using directive by the new Using directive
            root = root.ReplaceNode(oldUsing, newUsing);

            SyntaxTree newTree = SyntaxTree.Create(root);
            Console.WriteLine(newTree.ToString());
            Console.ReadLine();
        }
    }
}

NameSyntax is the base class for the following names that can appear in C# code:

IdentifierNameSyntax which represents simple single identifier names like System and Roslyn
GenericNameSyntax which represents a generic type or method name such as List<int>
QualifiedNameSyntax which represents a qualified name of the form <left-name>.<right-identifier-or-generic-name> such as System.IO

In our example we constructed a NameSyntax to represent System.Collections.Generic incrementally. Then we used the ReplaceNode method to replace the using System.Collections; directive with the using System.Collections.Generic; directive. We then constructed a new SyntaxTree using the modified root and print it. You should see the modified HelloWorld program:

Modifying Syntax Trees using SyntaxRewriters

In a previous post we implemented our SyntaxWalker. Now we going to use the same technique and implement our SyntaxRewriter. Both SyntaxWalker and SyntaxRewriter are subclasses of SyntaxVisitor which recursively visit every node in the syntax tree starting from the passed node. The SyntaxRewriter class can be used to apply a transformation to a specific type of SyntaxNode. It is also possible to apply a set of transformations to multiple types of SyntaxNode wherever they appear in a syntax tree. It uses depth-first search algorithm for SyntaxTree traversal.

In the following example we will implement our own SyntaxRewriter which will replace all variable declaration types with the var type. Create a new Console application and add a new class myRewriter to it. Edit the myRewriter.cs to look like:

using System;
using Roslyn.Compilers;
using Roslyn.Compilers.CSharp;

namespace RoslynDemo5
{
    class myRewriter : SyntaxRewriter
    {
        private readonly SemanticModel SemanticModel;

        public myRewriter(SemanticModel semanticModel)
        {
            this.SemanticModel = semanticModel;
        }

        public override SyntaxNode VisitLocalDeclarationStatement(LocalDeclarationStatementSyntax node)
        {
            // To avoid the case of multiple variables in a single declaration like the following one
            // Type variable1 = expression1,
            //      variable2 = expression2;
            if (node.Declaration.Variables.Count > 1)
            {
                return node;
            }

            // To avoid the case of case of a declation without an initializer like the following
            // Type variable;
            if (node.Declaration.Variables[0].Initializer == null)
            {
                return node;
            }
            
            // Get the first varaible in the declation
            VariableDeclaratorSyntax declarator = node.Declaration.Variables.First(); 
            // Get the Type used in the declaration
            TypeSyntax variableTypeName = node.Declaration.Type;
            // Bind the Type used in the declaration to a Symbol
            TypeSymbol variableType = (TypeSymbol)SemanticModel.GetSymbolInfo(variableTypeName).Symbol;
            // Get the whole intialization expression
            TypeInfo initializerInfo = SemanticModel.GetTypeInfo(declarator.Initializer.Value);
            if (variableType == initializerInfo.Type)
            {
                // Constructing a new Type with the same leading/trailing in the old type to maintain vertical whitespace and indentation
                TypeSyntax varTypeName = Syntax.IdentifierName("var").
                    WithLeadingTrivia(
                    variableTypeName.GetLeadingTrivia()
                    ).
                    WithTrailingTrivia(
                    variableTypeName.GetTrailingTrivia()
                    );
                // Do the replacement and return the new node
                return node.ReplaceNode(variableTypeName, varTypeName);
            }
            else
            {
                return node;
            }
        }
    }
}

myRewriter class is simply inheriting from SyntaxRewriter class and implments the VisitLocalDeclarationStatment() method. This method is called by the Visit method on SyntaxRewriter, which recursively visits a node and each of its children. The VisitLocalDeclarationStatment method is called on local declaration statements only. It tokes the starting SyntaxNode, if you passed the tree root (as we will do) it will visit the the local declaration statements that exists in the whole program.

Our rewriter takes the semantic model under inspection in the constructor.

In the VisitLocalDeclarationStatment method we check the declaration statement to filter out the cases that are not within our interest. For these cases we return the node unchanged.

For the cases we interested in, we retrieve the variable and the type used for its declaration. Get a symbol for the type and the whole initialization expression.

Then we check that the type used in the declaration is the type used in the initialization to avoid the situations where the declaration may be casting the initializer expression to a base class or interface. Removing the explicit type in these cases would change the semantics of a program.

Then we construct a new TypeSyntax representing the var type with any leading or trailing tivia (white spaces and tabs) to conserve the indentation. Then we replace the node and return the new node.

Let’s go back to our Main method in the Program.cs file. In order to use the myRewriter class, edit the Main method to look like:

 static void Main(string[] args)
        {
            // Create a test Compilation of our current project
            Compilation test = CreateTestCompilation();

            // Loop on each SyntaxTree in the Compilation
            foreach (SyntaxTree sourceTree in test.SyntaxTrees)
            {
                // Get the tree SemanticModel
                SemanticModel model = test.GetSemanticModel(sourceTree);
                // intialize myRewriter using the tree SemanticModel
                myRewriter rewriter = new myRewriter(model);
                // put our rewriter into action
                SyntaxNode newSource = rewriter.Visit(sourceTree.GetRoot());

                if (newSource != sourceTree.GetRoot())
                    Console.WriteLine(newSource.GetText().ToString());
                Console.ReadLine();
            }
        }

Here we create a test Compilation using CreateTestCompilation method, which we will add soon. We then loop for each SyntaxTree in this test Compilation. For each SyntaxTree we get the SemanticModel and pass it to our myRewriter constructor. Then we call the Visit() method on the SyntaxTree root. This will cause our VisitLocalDeclarationStatement method to be called and do its syntax transformation task. If the new SyntaxTree is different than the old SyntaxTree, this means that we did syntax modifications and we print the new code.

Now it add the following method to your code inside Program.cs.

private static Compilation CreateTestCompilation()
        {
            SyntaxTree programTree = SyntaxTree.ParseFile(@"..\..\Program.cs"); 
            SyntaxTree rewriterTree = SyntaxTree.ParseFile(@"..\..\myRewriter.cs");

            List sourceTrees = new List();
            sourceTrees.Add(programTree);
            sourceTrees.Add(rewriterTree);

            MetadataReference mscorlib = MetadataReference.CreateAssemblyReference("mscorlib");
            MetadataReference roslynCompilers = MetadataReference.CreateAssemblyReference("Roslyn.Compilers");
            MetadataReference csCompiler = MetadataReference.CreateAssemblyReference("Roslyn.Compilers.CSharp");

            List references = new List();
            references.Add(mscorlib);
            references.Add(roslynCompilers);
            references.Add(csCompiler);

            return Compilation.Create("TransformationCS", null, sourceTrees, references);

        }

This method basically creates two SyntaxTrees of our project code. Then make a list of the required references for these source code files. Then it sum it all in the factory method Compilation.Create() and return the resulted Compilation object.

When you run the project, you will see a modified version of the Program.cs file. Notice that our myRewriter class replaced the type used in all variables declarations to be of type var.

pressing Enter will bring the modified version of myRewriter.cs file, which have the same kind of change.

We have used the Roslyn Compiler APIs to write our own SyntaxRewriter that searches all files in a C# project for certain syntactic pattern, analyzes the semantics of source code that matches that pattern, and transforms it.

Now you can grab a paper and pencil, and start designing the next big thing in refactoring tools industry :)

Semantic Analysis using Microsoft Roslyn

noreply@blogger.com (Ebeid Soliman ElSayed) — Thu, 22 Aug 2013 18:11:00 +0000

In a previous post we talked about using Microsoft Rosyln Syntax API to deal with syntax text in terms of SyntaxTrees and SyntaxNodes. But as we we all know, a single source code or code snippet can’t make a useful program. 99% of the time we end up with many source code files that depend on so many externals: assembly references, namespace imports, or other code files. The meaning (semantic) of SyntaxNodes depend heavily on these externals and may change due changes in these externals even if its enclosing source code file have not been changed.

The Compilation class help us deal with source text in the context of its dependents and externals. A Compilation is analogous to a single project as seen by the compiler and represents everything needed to compile a Visual Basic or C# program such as assembly references, compiler options, and the set of source files to be compiled. With this context you can reason about the meaning of code. Compilations allow you to find Symbols – entities such as types, namespaces, members, and variables which names and other expressions refer to. The process of associating names and expressions with Symbols is called Binding.

Creating a Compilation

In this following example we will create a SyntaxTree for our traditional HelloWorld program. Then we created a Compilation out of this SyntaxTree and added a reference to MS Core Library.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Roslyn.Compilers;
using Roslyn.Compilers.CSharp;
using Roslyn.Services;
using Roslyn.Services.CSharp;

namespace RoslynDemo3
{
    class Program
    {
        static void Main(string[] args)
        {
            SyntaxTree tree = SyntaxTree.ParseText(
@"using System;
using System.Collections.Generic;
using System.Text;
 
namespace HelloWorld
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine(""Hello, World!"");
        }
    }
}");

            var root = (CompilationUnitSyntax)tree.GetRoot();

            Compilation compilation = Compilation.Create("HelloWorld")
                             .AddReferences(MetadataReference.CreateAssemblyReference("mscorlib"))
                             .AddSyntaxTrees(tree);

            foreach (MetadataReference reference in compilation.ExternalReferences)
                Console.WriteLine(reference.Display);

            Console.ReadLine();
        }
    }
}

You can supply all the syntax trees, assembly references, and options in one call or you can spread them out over multiple calls. To add the reference we just used the metadata name, which is the same name you find when you add a reference through Visual Studio’s Reference Manager.

Now when you run your program, you should get an output similar to the following:

The SemanticModel

Once we have a Compilation we can get a SemanticModel for any SyntaxTree in that Compilation. SemanticModels can be queried to answer questions like “What names are in scope at this location?” “What members are accessible from this method?” “What variables are used in this block of text?” and “What does this name/expression refer to?”

In the following example we going to modify our example a little bit. We will get the semantic model of our HelloWorld program. Then use this model to get semantic Symbol that represents the first using statement (of type NameSpaceSymbol). Then we use that symbol to get a list of namespaces inside the System namespace.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Roslyn.Compilers;
using Roslyn.Compilers.CSharp;
using Roslyn.Services;
using Roslyn.Services.CSharp;

namespace RoslynDemo3
{
    class Program
    {
        static void Main(string[] args)
        {
            SyntaxTree tree = SyntaxTree.ParseText(
@"using System;
using System.Collections.Generic;
using System.Text;
 
namespace HelloWorld
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine(""Hello, World!"");
        }
    }
}");

            var root = (CompilationUnitSyntax)tree.GetRoot();

            Compilation compilation = Compilation.Create("HelloWorld")
                             .AddReferences(MetadataReference.CreateAssemblyReference("mscorlib"))
                             .AddSyntaxTrees(tree);

            var model = compilation.GetSemanticModel(tree);
            var nameInfo = model.GetSymbolInfo(root.Usings[0].Name);
            var systemSymbol = (NamespaceSymbol)nameInfo.Symbol;

            foreach (var ns in systemSymbol.GetNamespaceMembers())
                Console.WriteLine(ns.Name);
            Console.ReadLine();
        }
    }
}

When you run the above code you should get an output like the following:

You can also use the query methods we showed in the previous post to retrieve a certain node and then use the semantic model to get more information about that node. You could use the code below to get the node that represent the “Hello, World!” string in our code snippet, get information about its symbol type (System.String) , and even get information about the assembly that contains this type.

            var helloWorldString = root.DescendantNodes()
                                       .OfType()
                                       .First();
            var literalInfo = model.GetTypeInfo(helloWorldString);
            Console.WriteLine(literalInfo.Type.ContainingAssembly.BaseName);

or enumerate the public methods of the System.String class

            var stringTypeSymbol = (NamedTypeSymbol)literalInfo.Type;
 
            Console.Clear();
            foreach (var name in (from method in stringTypeSymbol.GetMembers()
                                                              .OfType()
                               where method.ReturnType == stringTypeSymbol &&
                                     method.DeclaredAccessibility == 
                                                Accessibility.Public
                               select method.Name).Distinct())
            {
                Console.WriteLine(name);
            }

Somehow, we just scratched the surface of sematic analysis in this post. using both syntax and semantic analysis we can do more advanced and meaningful code focused tasks (which we will do in the next post).

Syntax Analysis using Microsoft Roslyn

noreply@blogger.com (Ebeid Soliman ElSayed) — Tue, 20 Aug 2013 20:49:00 +0000

In a previous post we talked about Microsoft Roslyn. In this post will get our hands dirty with Roslyn syntax analysis in order to develop our first code-focused program. We will see how the SyntaxTree of HelloWorld program look like and how we can traverse and query it.

The Syntax API exposes the syntax trees the compilers use to understand Visual Basic and C# programs. They are produced by the same parser that runs when a project is built or a developer hits F5. The four primary building blocks of syntax trees are:

The SyntaxTree class, an instance of which represents an entire parse tree.
The SyntaxNode class, instances of which represent syntactic constructs such as declarations, statements, clauses, and expressions.
The SyntaxToken structure, which represents an individual keyword, identifier, operator, or punctuation.
And lastly the SyntaxTrivia structure, which represents syntactically insignificant bits of information such as the whitespace between tokens, preprocessor directives, and comments.

Traversing Syntax Trees

First, download Microsoft Roslyn CTP from here. and Install it.You will need Visual Studio 2012 and Visual Studio 2012 SDK in order to install Roslyn.

Now open your Visual Studio 2012 and create a new Console Application project. Add the following to your project references

Now edit your code to look like:

using System;
using System;
using Roslyn.Compilers;
using Roslyn.Compilers.CSharp;

namespace RoslynDemo1
{
    class Program
    {
        static void Main(string[] args)
        {
            SyntaxTree tree = SyntaxTree.ParseText(
                @"using System;
 
                namespace HelloWorld
                {
                    class Program
                    {
                        static void Main(string[] args)
                        {
                            Console.WriteLine(""Hello, World!"");
                        }
                    }
                }");

            foreach (SyntaxNode node in tree.GetRoot().ChildNodesAndTokens())
                PrintSyntaxTree(node);

            Console.ReadLine();
        }
        static void PrintSyntaxTree(SyntaxNode node)
        {
            if (node != null)
            {
                Console.WriteLine(node.Kind + " | " + node.ToString());
                foreach (SyntaxNode child in node.ChildNodes())
                    PrintSyntaxTree(child);
            }
        }
    }
}

Now, run our program. You should see an output similar to the following:

Now let’s explain what we already did. First, we included the needed namespaces. The common Syntax APIs are found in the Roslyn.Compilers and the Roslyn.Compilers.Common namespace, while the language specific Syntax APIs are found in Roslyn.Compilers.CSharp and Roslyn.Compilers.VisualBasic.

Second, we tried to get a syntax tree corresponding to our HelloWorld program using the SyntaxTree.ParseText function, which produces a syntax tree by parsing the passed source code.

Third, in order to traverse the syntax tree we get its root, then get its child nodes. Then loop on these child nodes and their children recursively to print its kind and the string representation. C# node kind is on of the kinds in enum Roslyn.Compilers.CSharp.SymbolKind.

Inside the PrintSyntaxTree method, by playing in the order of node printing and the recursive call, you can apply your tree traversal logic. You can also inspect specific elements of the tree:

var root = (CompilationUnitSyntax)tree.GetRoot(); 
var helloWorldDeclaration = (NamespaceDeclarationSyntax)root.Members[0];
var programDeclaration = (TypeDeclarationSyntax)helloWorldDeclaration.Members[0];
var mainDeclaration = (MethodDeclarationSyntax)programDeclaration.Members[0];
var argsParameter = mainDeclaration.ParameterList.Parameters[0];
Console.WriteLine(argsParameter.ToString() );

Query Methods

In addition to traversing syntax trees you can also explore the syntax tree using the query methods defined on SyntaxNode. These methods are similar to XPath. You can also use these methods with LINQ to quickly find things in a tree. For example, the following statements uses a LINQ expression and the DescendantNodes method to locate the same parameter as in the previous example:

            var firstParameters = from methodDeclaration in root.DescendantNodes()
                                                    .OfType()
                                  where methodDeclaration.Identifier.ValueText == "Main"
                                  select methodDeclaration.ParameterList.Parameters.First();

            var argsParameter2 = firstParameters.Single();
            Console.WriteLine(argsParameter2.ToString());

SyntaxWalkers

If you want to find all nodes of a specific type in a syntax tree, for example, every property declaration in a file, you can do that by extending the SyntaxWalker class and overriding the VisitPropertyDeclaration method, you can process every property declaration in a syntax tree without knowing its structure beforehand.

Now let’s implement, as an example, a SyntaxWalker that examines the entire syntax tree and collects any using directives it find which aren’t importing a System namespace.

Create a new C# Console application and edit it to look like:

using System;
using Roslyn.Compilers;
using Roslyn.Compilers.CSharp;

namespace RoslynDemo2
{
    class Program
    {
        static void Main(string[] args)
        {
            SyntaxTree tree = SyntaxTree.ParseText(
@"using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Roslyn.Compilers;
using Roslyn.Compilers.CSharp;
 
namespace TopLevel
{
    using Microsoft;
    using System.ComponentModel;
 
    namespace Child1
    {
        using Microsoft.Win32;
        using System.Runtime.InteropServices;
 
        class Foo { }
    }
 
    namespace Child2
    {
        using System.CodeDom;
        using Microsoft.CSharp;
 
        class Bar { }
    }
}");

            var root = (CompilationUnitSyntax)tree.GetRoot();
        }
    }
}

As you noticed, this source text contains using directives scattered across four different locations: the file-level, in the top-level namespace, and in the two nested namespaces.

Now, add a new class to your project and edit it to look like the next code. Basically we inherit from the SyntaxWalker class, and over the VisitUsingDirective method. This method is called by Visit method, which recursively visits a node and each of its children. VisitUsingDirective method is called for using directives only, inside it we collect the namespaces that starts with “System.” but not equal to “System”.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Roslyn.Compilers;
using Roslyn.Compilers.CSharp;

namespace RoslynDemo2
{
    class myWalker : SyntaxWalker   
    {
        public readonly List Usings = new List();

        public override void VisitUsingDirective(UsingDirectiveSyntax node)
        {
            if (node.Name.GetText().ToString() != "System" &&
                !node.Name.GetText().ToString().StartsWith("System."))
            {
                this.Usings.Add(node);
            }
        }
    }
}

Now let’s go back to the Main method and use our walker by adding the following code snippet to the end of Main method. Basically it passes the syntax tree root to myWalker.Visit method, which internally call the VisitUsingDirective method. Then we loop and print the collected usings.

When you run your program, you should see an output similar to the following:

In this post we gave a quick example about obtaining a SyntaxTree of HelloWorld program and explored its nodes. We also built a SyntaxWalker that walk the syntax tree and collect the nodes of our interest only. Later we will see how to fully manipulate SyntaxNodes and SyntaxTrees.

Overview of Microsoft Roslyn – The first compiler-as-service product.

noreply@blogger.com (Ebeid Soliman ElSayed) — Tue, 20 Aug 2013 02:00:00 +0000

Introduction

From developer perspective, compilers are black boxes -- source code goes in one end, magic happens in the middle, and object files or assemblies come out the other end. During their job, compilers build up a deep understanding of our code. For decades this valuable information was unreachable.

The Roslyn project aims to open compilers for developers and offer it as services. Through Roslyn, compilers become services with APIs that can be used for code related tasks in your tools and applications.

In order to understand the APIs that Roslyn offers, we need to understand the general compiler phases:

The parse phase where source is tokenized and parsed into syntax that follows the language grammar.
The declaration phase where declarations from source and imported metadata are analyzed to form named symbols.
The binding phase where identifiers in the code are matched to symbols.
The emitting phase where all the information built up by the compiler is emitted as an assembly.

For each phase of these phases, Roslyn provides a set of APIs to access its services and an object model that allow access to the information at that phase.

The parsing phase is exposed as a syntax tree
The declaration phase as a hierarchical symbol table
The binding phase as a model that exposes the result of the compiler’s semantic analysis
The emit phase as an API that produces IL byte codes.

Rosyln API Layers

Compiler APIs contains a representation of a single invocation of a compiler (including assembly references, compiler options, and source code files) which provides the object models that corresponds to the compiler pahases.
1. Scripting APIs represents a runtime execution context for C# or VB piece of code. It contains a scripting engine that allows evaluation of expressions and statements as top-level constructs in programs.
Services APIs represents the starting point for doing code analysis and refactoring over entire solutions by organizing all the information about the projects in a solution into single object model, offering you direct access to the compiler layer object models without needing to parse files, configure options or manage project to project dependencies. In addition, the services layer surfaces a set of commonly used APIs used when implementing code analysis and refactoring tools that function within a host environment like the Visual Studio IDE.
Editor Services APIs allows a user to easily extend Visual Studio IDE features (such as IntelliSense, refactorings, and code formatting). This layer has a dependency on the Visual Studio text editor and the Visual Studio software development kit (SDK).

Working with Syntax

The fundamental data structure exposed by the Compiler APIs is the syntax tree. It represents the lexical and syntactic structure of source code, No part of the source code is understood without it first being identified. These trees enable tools to manipulate source code without editing it as text.

Syntax Trees Attributes

Syntax trees have three key attributes:

Full fidelity which means that the syntax tree contains every piece of information found in the source text, every grammatical construct, every lexical token, and everything else in between including whitespace, comments, and preprocessor directives.
Round-trippable which means that it can be traced back to the text it was parsed from and get the text representation of the sub-tree rooted at that node. This means that syntax trees can be used as a way to construct and edit source text. By creating a tree you have by implication created the equivalent source code, and by editing a syntax tree, making a new tree out of changes to an existing tree, you have effectively edited the source code.
Immutable and thread-safe which means that after a tree is obtained, it is a snapshot of the current state of the code, and never changes. This allows multiple users to interact with the same syntax tree at the same time in different threads without locking or duplication.

Syntax Tree Elements

Each syntax tree is made up of nodes, tokens, and trivia.

Syntax Nodes: the primary elements of syntax trees. It represent syntactic constructs such as declarations, statements, clauses, and expressions. They are non-terminal nodes in the syntax tree, which means they always have other nodes and tokens as children. Each category of syntax nodes is represented by a separate class derived from SyntaxNode.

Parent property returns the node parent.
ChildNodes() returns a list of child nodes in sequential order based on its position in the source text. This list does not contain tokens.
DescendantNodes(), DescendantTokens(), and DescendantTrivia() returns a list of child nodes in sequential order based on its position in the source text.
Each syntax node subclass exposes all the same children through strongly typed properties.

BinaryExpressionSyntax node class

Property Left/Right of type ExpressionSyntax
Property OperatorToken of type SyntaxToken

IfStatementSyntax node class

Optional property ElseClauseSyntax which returns null if the child is not present.

Syntax Tokens are the terminals of the language grammar, representing the smallest syntactic fragments of the code. They are never parents of other nodes or tokens. Syntax tokens consist of keywords, identifiers, literals, and punctuation.
Syntax Trivia represent the parts of the source text that are largely insignificant for normal understanding of the code, such as whitespace, comments, and preprocessor directives. Because trivia are not part of the normal language syntax and can appear anywhere between any two tokens, they are not included in the syntax tree as a child of a node. you can access them through token’s LeadingTrivia or TrailingTrivia collections.
Spans In order to make each node knows its position within the source text, each node has two properties of type TextSpan. A TextSpan object is the beginning position and a count of characters, both represented as integers. Each node has two TextSpan properties:

The Span property is the text span from the start of the first token in the node’s sub-tree to the end of the last token. This span does not include any leading or trailing trivia.
The FullSpan property is the text span that includes the node’s normal span, plus the span of any leading or trailing trivia.

Kinds Each node, token, or trivia has a Kind property, of type SyntaxKind, that identifies the exact syntax element represented. The Kind property allows for easy disambiguation of syntax node types that share the same node class.
Errors When the parser encounters code that does not conform to the defined syntax of the language, it uses one of two techniques to create a syntax tree.

If the parser expects a particular kind of token, but does not find it, it may insert a missing token into the syntax tree in the location that the token was expected. A missing token represents the actual token that was expected, but it has an empty span, and its IsMissing property returns true.
The parser may skip tokens until it finds one where it can continue parsing. In this case, the skipped tokens that were skipped are attached as a trivia node with the kind SkippedTokens.

In this post we explored Microsoft Roslyn, an interesting product that offer the compiler as a service and opens a lot of doors in front of developers to develop code-focused tools and applications. In later posts we will gradually show Roslyn in action.

How to debug your Pex parameterized unit tests

noreply@blogger.com (Ebeid Soliman ElSayed) — Tue, 13 Aug 2013 18:43:00 +0000

We talked before about Microsoft Pex, and how it choose the shown inputs. We then talked more about building blocks of parameterized unit tests and its patterns.

What we missed till now is how to debug your parameterized unit tests. In order to debug your parameterized unit tests, do the following:

Go to Tools >> Options >> Pex >> Diagnostic.
Change the BreakOnStart option to True.

Now set a break point at any line of code inside your parameterized unit test.
Right click inside your parameterized unit test and select Run Pex Explorations.
You will see a dialog popping asking to attach a debugger.
Select your current Visual Studio instance.

Once attached, press F5.
Now Visual Studio debugger will jump to your break point in the parameterized unit test and stop there. You can now step through you parameterized unit test code.

You will notice that parameter’s values are those selected by Pex. The debugger will run the parameterized unit test for each input value(s) chosen by Pex, So you can inspect the PUT behavior against a specific input value.

Convert Microsoft Pex path conditions from Z3 native format to SMT-LIB

noreply@blogger.com (Ebeid Soliman ElSayed) — Thu, 01 Aug 2013 21:24:00 +0000

We talked before about getting the code path conditions from Microsoft Pex as in Z3 native format(.z3 file).

Sometimes you may need to convert from Z3 native format to SMT-LIB standard. You can do that using the Z3 C# APIs. The only trick here is: you have to use the Microsoft.Z3.dll that come with your current Microsoft Pex installation (C:\Program Files\Microsoft Pex\bin\Microsoft.Z3.dll).

To show how it works, let’s create a new C# console application. Add refeerence to Microsoft.Z3.dll that come with your Pex installation.

Edit your code to look like:

using System;
using System.IO;
using Microsoft.Z3;

namespace Z3ToSMTLIB
{
    class Program
    {
        static void Main(string[] args)
        {
            if(args.Length == 0)
                Console.WriteLine("Please provide input file name.");
            if (!File.Exists(args[0]))
                Console.WriteLine("The specified file doesn't exist.");
            using (Context ctx = new Context(new Config()))
            {
                string SMTLIB= ctx.ParseZ3File(args[0]).ToString();
                StreamWriter writer = new StreamWriter(args[0].TrimEnd("z3".ToCharArray()) + "SMTLIB");
                writer.Write(SMTLIB);
                writer.Close();
            }
        }
    }
}

Right click your project in the solution explorer and select Add >> New Item >> select Application Configuration File >> click Add. Edit the newly created App.config file to look like: Now you can run your code and pass a file in Z3 native format and get a file in SMT-LIB format. This SMT-LIB format opens a lot of possibilities that we will explore later.

The Z3 Constraint Solver, a developer perspective

noreply@blogger.com (Ebeid Soliman ElSayed) — Tue, 30 Jul 2013 11:30:00 +0000

Z3 is a high-performance SMT solver developed at Microsoft Research. It have been integrated with many many tools that came out from Microsoft for program analysis, testing, and verification.

What SAT means ?

SAT refers to Boolean satisfiability problem where we want to determine if there exists an interpretation that satisfies a given Boolean formula. In other words, it establishes if the variables of a given Boolean formula can be assigned in such a way as to make the formula evaluate to true.

What SMT means ?

SMT stands for Satiability Modulo Theories. SMT instance is a formula in first-order logic, where some functions and predicates have additional interpretations. SMT problem is a decision problem of determining whether such a formula is satisfiable or not.

An SMT instance is a generalization of a Boolean SAT instance in which various sets of variables are replaced by predicates over a suitable set of non-binary variables.

These predicates are classified according to the theory they belong to. For instance, linear inequalities over real variables are evaluated using the rules of the theory of linear real arithmetic.

What SMT solver means ?

The goal of an SMT solver is to determine whether an SMT instance can evaluate to true or not. The same analog applies for SAT solvers.

SMT solvers

There is a lot of SMT solvers available but there is only one SMT solver with C# APIs, it is Z3. For a list of available SMT solvers, refer to this page.

Download Z3

You can download Z3 from http://z3.codeplex.com/ . I downloaded Z3 4.3.0 and extracted it to C:\z3.

C# Example

In the following example we going to let Z3 solve the following equation system:

x > 0
y = x + 1
y < 3

Solving the equations means finding values for x and y that make the whole formula evaluates to true.

Let’s create a new console application project in Visual Studio.
Add reference to Microsoft.Z3.dll which is located in the bin directory of the Z3 installation directory.
Copy the file libz3.dll from the bin directory of the Z3 installation directory to your project build directory.
Now edit your code to look like the following:

using System;
using Microsoft.Z3;

namespace Z3Demo1
{
    class Program
    {
        static void Main(string[] args)
        {
            using (Context ctx = new Context())
            {
                Expr x = ctx.MkConst("x", ctx.MkIntSort());
                Expr y = ctx.MkConst("y", ctx.MkIntSort());
                Expr zero = ctx.MkNumeral(0, ctx.MkIntSort());
                Expr one = ctx.MkNumeral(1, ctx.MkIntSort());
                Expr three = ctx.MkNumeral(3, ctx.MkIntSort());
                
                Solver s = ctx.MkSolver();
                s.Assert(ctx.MkAnd(ctx.MkGt((ArithExpr)x, (ArithExpr)zero), ctx.MkEq((ArithExpr)y, 
                    ctx.MkAdd((ArithExpr)x, (ArithExpr)one)), ctx.MkLt((ArithExpr)y, (ArithExpr)three)));
                Console.WriteLine(s.Check());

                Model m = s.Model;
                foreach (FuncDecl d in m.Decls)
                        Console.WriteLine(d.Name + " -> " + m.ConstInterp(d));
                    
                Console.ReadLine();
            }
        }
    }
}

Now let’s run the code above and see the output. The solver says the equation system is satisfiable and then gives us the x and y values that satisfy.

How it works ?

To interact with Z3 through C#, you need a Context object. Variables and numerals in your equations are modeled as Expr objects. You get these objects using member functions in the Context object (MkConts(), MkNumeral(),….). You construct your operand using member functions in the Context object (MkGt(), MkAdd(), MkLt(),…). To solve all the equations together you need to hock them up using AND operator, which is implemented using Context.MkAnd(). After hocking everything in one AND, you pass that the solver through Solver.Assert(). And as you may guessed, you obtain this Solver using Context.MkSolver().

Solver.Check() will tell you whether this equation system can be solved or not. To get the variables’ assignments that the come up with, get a Model object Solver.Model. Then use the Delcs collection to get all symbols that have an interpretation in the model. Model.ConstInterp() will get the symbol assigned value.

In this post we briefly introduced SAT, SMT, and their solvers. Then we explored the only SMT solver that written in C# and had a C# API. Now you can play with as many equations as you want and check them for satisfiability and even got the solution values.

Get path conditions from Microsoft Pex

noreply@blogger.com (Ebeid Soliman ElSayed) — Wed, 24 Jul 2013 22:26:00 +0000

We talked in a previous post about Microsoft Pex and how it choose values to explore your code through symbolically executing it and getting all its paths. A code path is the represented by all the conditions that have to be satisfied in order to make the code execution go to this path. Your code conditional statements will lead to different paths according to different input values. Test cases generated by Pex represents the code behavior for specific input values that lead to specific code path.

Within the parameterized unit test You can use the method GetPathConditionString of the PexSymbolicValue class to obtain a textual representation of the current path condition, a predicate (condition) that characterizes an execution path (below, column with title Condition) .

Methods ToRawString and GetRawPathConditionString of the PexSymbolicValue class return expressions representing symbolic values and the path condition, formatted as S-expressions.

As we know, Pex uses the Z3 SMT solver behind the scenes. If you want to get the path conditions before being passed to Z3, do the following:

Create an environment variable with name z3_logdir and value c:\z3_logdir (or any other directory you want)
Create an environment variable with name z3_loglevel and value constraintsandmodel .

Now, Pex will create *.z3 files in subdirectories in the z3_logdir you specified in step 1. These files are in Z3 native format and can be later passed to Z3. For more information about this format, refer to Z3 online documentation.

In future posts we will talk more about the Z3 native format and what we can do with it.

Bring “Sign in as Different User” back to SharePoint 2013

noreply@blogger.com (Ebeid Soliman ElSayed) — Tue, 23 Jul 2013 22:36:00 +0000

If you new to SharePoint 2013 like me and get confused looking for “Sign in as a Different User” menu command, don’t worry it is not there by design but we can easily bring it back.

Do the following steps to get the link back:

Locate the file \15\TEMPLATE\CONTROLTEMPLATES\Welcome.ascx and open it for editing.
Add the following snippet before the existing element that have id of “ID_RequestAccess”.


 Text="<%$Resources:wss,personalactions_loginasdifferentuser%>"
 Description="<%$Resources:wss,personalactions_loginasdifferentuserdescription%>"
 MenuGroupId="100"
 Sequence="100"
 UseShortId="true"
 />

Save and close the file. Now you should get the link “Sign in as Different User” back.

Create a new Web Part Page with Quick Launch menu

noreply@blogger.com (Ebeid Soliman ElSayed) — Wed, 17 Jul 2013 15:20:00 +0000

This post is about a simple task we do daily in SharePoint 2010, creating a new Web Part page. Every time we need a new Web Part page, we go to All Site Content > Create > Web Part Page > Create. Now we have a Web Part page but no quick launch menu on the left.

To show the quick launch menu on the new created page:

Open the page for editing in SharePoint 2010.
Switch to the Code view.
Click on the Advanced Mode button on the ribbon.
Now look for the following code snippet and delete it.

<SharePoint:UIVersionedContent ID="WebPartPageHideQLStyles" UIVersion="4"runat="server">
<ContentTemplate>
body # {s4-leftpanel
display: none;
}
. s4 {ca-
margin-left: 0px;
}
</ style>
</ ContentTemplate>
</ SharePoint: UIVersionedContent>

Now look for the following snippet and delete it.

<asp:Content ContentPlaceHolderId="PlaceHolderLeftNavBar" runat="server">
</ asp: Content>

Save and Close

Now when you open the page, the quick launch navigation will be there.

Getting Started with MongoDB – Part 2

noreply@blogger.com (Ebeid Soliman ElSayed) — Fri, 12 Jul 2013 22:15:00 +0000

In the previous post we explored the basics of MongoDB. In this post we going to dig deeper in MongoDB.

Indexing

Whenever a new collection is created, MongoDB automatically creates an index by the _id field. These indexes can be found in the system.indexes collection. You can show all indexes in the database using db.system.indexes.find() . Most queries will include more fields than just the _id, so we need to make indexes on those fields.

Before creating more indexes, let’s see what is the performance of a sample query without creating any indexes other than the automatically created one for _id. Create the following function to generate random phone numbers.

function (area,start,stop) {
for(var i=start; i < stop; i++) {
var country = 1 + ((Math.random() * 8) << 0);
var num = (country * 1e10) + (area * 1e7) + i;
db.phones.insert({
_id: num,
components: {
country: country,
area: area,
prefix: (i * 1e-4) << 0,
number: i
},
display: "+" + country + " " + area + "-" + i
});

Run the function with a three-digit area code (like 800) and a range of seven digit numbers (5,550,000 to 5,650,000)

populatePhones( 800, 5550000, 5650000 )

Now we expecting to see a new index created for our new collection.

> db.system.indexes.find()
{ "v" : 1, "key" : { "_id" : 1 }, "ns" : "newdb.towns", "name" : "_id_" }
{ "v" : 1, "key" : { "_id" : 1 }, "ns" : "newdb.countries", "name" : "_id_" }
{ "v" : 1, "key" : { "_id" : 1 }, "ns" : "newdb.phones", "name" : "_id_" }

Now let’s check the query without an index. The explain() method is used to output details of a given operation and can help us here.

> db.phones.find( { display : "+1 800-5650001" } ).explain()
{
        "cursor" : "BasicCursor",
        "isMultiKey" : false,
        "n" : 0,
        "nscannedObjects" : 100000,
        "nscanned" : 100000,
        "nscannedObjectsAllPlans" : 100000,
        "nscannedAllPlans" : 100000,
        "scanAndOrder" : false,
        "indexOnly" : false,
        "nYields" : 0,
        "nChunkSkips" : 0,
        "millis" : 134,
        "indexBounds" : {

},
"server" : "ESOLIMAN:27017"
}

Just to make things simple, we will look at the millis field only which gives the milliseconds needed to complete the query. Now it is 134.

Now we going to create an index and see how it improves our query execution time. We create an index by calling ensureIndex(fields,options) on the collection. The fields parameter is an object containing the fields to be indexed against. The options parameter describes the type of index to make. On production environments, creating an index on a large collection can be slow and resource-intensive, you should create them in off-peak times. In our case we going to build a unique index on the display field and we will drop duplicate entries.

> db.phones.ensureIndex(
... { display : 1 },
... { unique : true, dropDups : true }
... )

lets try explain() of find() and see the new value for millis field. Query execution time improved, from 134 down to 16.

> db.phones.find( { display : "+1 800-5650001" } ).explain()
{
        "cursor" : "BtreeCursor display_1",
        "isMultiKey" : false,
        "n" : 0,
        "nscannedObjects" : 0,
        "nscanned" : 0,
        "nscannedObjectsAllPlans" : 0,
        "nscannedAllPlans" : 0,
        "scanAndOrder" : false,
        "indexOnly" : false,
        "nYields" : 0,
        "nChunkSkips" : 0,
        "millis" : 16,
        "indexBounds" : {
                "display" : [
                        [
                                "+1 800-5650001",
                                "+1 800-5650001"
                        ]
                ]
        },
        "server" : "ESOLIMAN:27017"
}

Notice the cursor changed from a Basic to a B-tree cursor. MongoDB is no longer doing
a full collection scan but instead walking the tree to retrieve the value.

Mongo can build your index on nested values: db.phones.ensureIndex({ "components.area": 1 }, { background : 1 })

Aggregations

count() counts the number of matching documents. It takes a query and returns a number.

> db.phones.count({'components.number': { $gt : 5599999 } })
100000

distinct() returns each matching value where one or more exists.

> db.phones.distinct('components.number', {'components.number': { $lt : 5550005 } })
[ 5550000, 5550001, 5550002, 5550003, 5550004 ]

group() groups documents in a collection by the specified keys and performs simple aggregation functions such as computing counts and sums. It is similar to GROUP BY in SQL. It accepts the following parameters

key – Specifies one or more document fields to group by.
reduce – Specifies a function for the group operation perform on the documents during the grouping operation, such as compute a sum or a count. The aggregation function takes two arguments: the current document and the aggregate result for the previous documents in the group.
initial – Initializes the aggregation result document.
keyf – Optional. Alternative to the key field. Specifies a function that creates a “key object” for use as the grouping key. Use the keyf instead of key to group by calculated fields rather than existing document fields. Like HAVING in SQL.
cond – Optional. Specifies the selection criteria to determine which documents in the collection to process. If you omit the cond field, db.collection.group() processes all the documents in the collection for the group operation.
finalize – Optional. Specifies a function that runs each item in the result set before db.collection.group() returns the final value. This function can either modify the result document or replace the result document as a whole.

> db.phones.group({
... initial : { count : 0 },
... reduce : function(phone, output) { output.count++; },
... cond : { 'components.number' : { $gt : 5599999 } },
... key : { 'components.area' : true }
... })
[
        {
                "components.area" : 800,
                "count" : 50000
        },
        {
                "components.area" : 855,
                "count" : 50000
        }
]

The first thing we did here was set an initial object with a field named count set to 0—fields created here will appear in the output. Next we describe what to do with this field by declaring a reduce function that adds one for every document we encounter. Finally, we gave group a condition restricting which documents to reduce over.

Server-Side Commands

All queries and operations we did till now, execute on the client side. The db object provides a command named eval(), which passes the given function to the server. This dramatically reduces the communication between client and server. It is similar to stored procedures in SQL.

There is a also a set of prebuilt commands that can be executed on the server. Use db.listCommands() to get a list of these commands. To run any command on the server use db.runCommand() like db.runCommand({ "count" : "phones" })

Although it is not recommended, you can store a JavaScript function on the server for later reuse.

MapReduce

MapReduce is a framework for parallelizing problems. Generally speaking the parallelization happens on two steps:

"Map" step: The master node takes the input, divides it into smaller sub-problems, and distributes them to worker nodes. A worker node may do this again in turn, leading to a multi-level tree structure. The worker node processes the smaller problem, and passes the answer back to its master node.
"Reduce" step: The master node then collects the answers to all the sub-problems and combines them in some way to form the output – the answer to the problem it was originally trying to solve.

To show the MapReduce framework in action, let’s build on the phones collections that we created previously. Let’s generate a report that counts all phone numbers that contain the same digits for each country.

First we create a helper function that extracts an array of all distinct numbers (this step is not a MapReduce step).

> distinctDigits = function(phone) {
... var
... number = phone.components.number + '',
... seen = [],
... result = [],
... i = number.length;
... while(i--) {
... seen[+number[i]] = 1;
... }
... for (i=0; i<10; i++) {
... if (seen[i]) {
... result[result.length] = i;
... }
... }
... return result;
... }

> db.eval("distinctDigits(db.phones.findOne({ 'components.number' : 5551213 }))")
[ 1, 2, 3, 5 ]

Now let’s find find distinct numbers of each country. Since we need to query by country later, we will add the distinct digits array and country as compound key. For each distinct digits array in each country, we will add a count field that hold the value 1.

> map = function() {
... var digits = distinctDigits(this);
... emit( { digits : digits, country : this.components.country } , { count : 1 } );
... }

The reducer function will all these 1s that have been emitted from the map function.

>reduce = function(key, values) {
... var total = 0;
... for(var i=0; i<values.length; i++) {
... total += values[i].count;
... }
... return { count : total };
... }

Now it is time to put all pieces together and start the whole thing (the input collection, map function, reduce function, output collection).

> results = db.runCommand({
... mapReduce : 'phones',
... map : map,
... reduce : reduce,
... out : 'phones.report'
... })
{
        "result" : "phones.report",
        "timeMillis" : 21084,
        "counts" : {
                "input" : 200000,
                "emit" : 200000,
                "reduce" : 48469,
                "output" : 3489
        },
        "ok" : 1
}

Now you can query the output collection like any other collection

> db.phones.report.find()
{ "_id" : { "digits" : [ 0, 1, 2, 3, 4, 5, 6 ], "country" : 1 }, "value" : { "count" : 37 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 4, 5, 6 ], "country" : 2 }, "value" : { "count" : 23 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 4, 5, 6 ], "country" : 3 }, "value" : { "count" : 17 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 4, 5, 6 ], "country" : 4 }, "value" : { "count" : 29 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 4, 5, 6 ], "country" : 5 }, "value" : { "count" : 34 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 4, 5, 6 ], "country" : 6 }, "value" : { "count" : 35 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 4, 5, 6 ], "country" : 7 }, "value" : { "count" : 33 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 4, 5, 6 ], "country" : 8 }, "value" : { "count" : 32 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5 ], "country" : 1 }, "value" : { "count" : 5 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5 ], "country" : 2 }, "value" : { "count" : 7 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5 ], "country" : 3 }, "value" : { "count" : 3 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5 ], "country" : 4 }, "value" : { "count" : 6 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5 ], "country" : 5 }, "value" : { "count" : 5 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5 ], "country" : 6 }, "value" : { "count" : 10 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5 ], "country" : 7 }, "value" : { "count" : 5 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5 ], "country" : 8 }, "value" : { "count" : 7 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5, 6 ], "country" : 1 }, "value" : { "count" : 95 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5, 6 ], "country" : 2 }, "value" : { "count" : 104 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5, 6 ], "country" : 3 }, "value" : { "count" : 108 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5, 6 ], "country" : 4 }, "value" : { "count" : 113 } }
Type "it" for more

> db.phones.report.find({'_id.country' : 8})
{ "_id" : { "digits" : [ 0, 1, 2, 3, 4, 5, 6 ], "country" : 8 }, "value" : { "count" : 32 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5 ], "country" : 8 }, "value" : { "count" : 7 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5, 6 ], "country" : 8 }, "value" : { "count" : 127 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5, 6, 7 ], "country" : 8 }, "value" : { "count" : 28 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5, 6, 8 ], "country" : 8 }, "value" : { "count" : 27 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5, 6, 9 ], "country" : 8 }, "value" : { "count" : 29 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5, 7 ], "country" : 8 }, "value" : { "count" : 10 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5, 8 ], "country" : 8 }, "value" : { "count" : 7 } }
{ "_id" : { "digits" : [ 0, 1, 2, 3, 5, 9 ], "country" : 8 }, "value" : { "count" : 8 } }
{ "_id" : { "digits" : [ 0, 1, 2, 4, 5 ], "country" : 8 }, "value" : { "count" : 3 } }
{ "_id" : { "digits" : [ 0, 1, 2, 4, 5, 6 ], "country" : 8 }, "value" : { "count" : 121 } }
{ "_id" : { "digits" : [ 0, 1, 2, 4, 5, 6, 7 ], "country" : 8 }, "value" : { "count" : 25 } }
{ "_id" : { "digits" : [ 0, 1, 2, 4, 5, 6, 8 ], "country" : 8 }, "value" : { "count" : 27 } }
{ "_id" : { "digits" : [ 0, 1, 2, 4, 5, 6, 9 ], "country" : 8 }, "value" : { "count" : 17 } }
{ "_id" : { "digits" : [ 0, 1, 2, 4, 5, 7 ], "country" : 8 }, "value" : { "count" : 4 } }
{ "_id" : { "digits" : [ 0, 1, 2, 4, 5, 8 ], "country" : 8 }, "value" : { "count" : 4 } }
{ "_id" : { "digits" : [ 0, 1, 2, 4, 5, 9 ], "country" : 8 }, "value" : { "count" : 7 } }
{ "_id" : { "digits" : [ 0, 1, 2, 5 ], "country" : 8 }, "value" : { "count" : 14 } }
{ "_id" : { "digits" : [ 0, 1, 2, 5, 6 ], "country" : 8 }, "value" : { "count" : 162 } }
{ "_id" : { "digits" : [ 0, 1, 2, 5, 6, 7 ], "country" : 8 }, "value" : { "count" : 95 } }
Type "it" for more

The unique emitted keys are under the field _id, and all of the data returned from the reducers are
under the field value. If you prefer that the mapreducer just output the results, rather than outputting to a collection, you can set the out value to { inline : 1 }, but bear in mind there is a limit to the size of a result you can output (16 MB).

In some situations you may need to feed the reducer function’s output into another reducer function. In these situations we need to carefully handle both cases: either map’s output or another reduce’s output.

MongoDB have so many features that we didn’t even mentioned here. In later posts will continue working on them.

Getting Started with MongoDB – Part 1

noreply@blogger.com (Ebeid Soliman ElSayed) — Fri, 12 Jul 2013 18:46:00 +0000

MongoDB (from "humongous") is an open source document-oriented database system developed and supported by 10gen (founded by Dwight Merriman). First publicly released in 2009, and since then it have been a rising star in the NoSQL world. MongoDB stores structured data as JSON-like documents with dynamic schemas (technically data is stored in a binary form of JSON known as BSON), making the integration of data in certain types of applications easier and faster.

Installation

Download the latest mongoDB version from here.
Extract the archive to your preferred location (in my case C:\mongodb). MongoDB is self-contained and does not have any other system dependencies. You can run MongoDB from any folder you choose. You may install MongoDB in any directory.
MongoDB requires a data folder to store its files (default is C:\data\db ). You may specify a different path with the dbpath setting when lunching mongod.exe.

Starting the Server

To start MongoDB, open the command prompt window, and run mongod.exe from the bin directory (specify the data path if needed)
The waiting for connections message in the console output indicates that the mongod.exe process is running successfully and waiting for connections on port 27017

Connecting to the Server

To connect to the server, open another command prompt window and run mongo.exe from the bin directory.

Run MongoDB as a Windows Service

Create a log file for MongoDB (in my case c:\mongodb\log\mongo.log ).
Create a data directory for MongoDB (in my case c:\mongodb\data ).
Open the command prompt window as an administrator.
Run the following command C:\mongodb\bin>mongod.exe –install –rest –master --logpath "c:\mongodb\log\mongo.log"
Run regedit from start menu.
Go to HKEY_LOCAL_MACHINE >> SYSTEM >> CurrentControlSet >> services
Find the MongoDB directory & edit the ImagePath key.
Set value as c:\mongodb\bin\mongod --service --rest --master --logpath=C:\mongodb\log\mongo.log --dbpath=C:\mongodb\data
Save and exit registry editor
Open ComponentServices from Start menu >> Run
Locate the Mongo DB service, and reigh click >> Properties.
Set the Startup Type to Automatic. Then start the service.
1. To run the MongoDB service from command window, use net start MongoDB
Check at http://localhost:28017/ to see , MongoDB should return stats.

In case you want to remove MongoDB service C:\mongodb\bin\mongod.exe –remove

Data Model

Data in MongoDB has a flexible schema.

Database consists on a set of collections.
Collections consists of a set of documents.
- Documents in the same collection do not need to have the same set of fields or structure.
- Common fields in a collection’s documents can hold different types of data.

Based on what we mentioned, you could say that MongoDB is schema-less. But you may refer to this data modeling article before doing real work on MongoDB.

CRUD operations

When you start the mongo shell, it connect to test database by default. To create a new database or switch to another database, use use newdb
To show a list of databases, use show dbs (databases are created when you insert the first values in it. This means that any database have been created but no values inserted in it, it really doesn’t exist).
To confirm the current session database, use db
To create a collection, just insert an initial record to it. Since Mongo is schema-less, there is no need to define anything up front. The following code creates/inserts a towns collection:

db.towns.insert({
name: "New York",
population: 22200000,
last_census: ISODate("2009-07-31"),
famous_for: [ "statue of liberty", "food" ],
mayor : {
name : "Michael Bloomberg",
party : "I"
}
})

brackets like {...} denote an object with key-value pairs.
brackets like [...] denote an array.
You can nest these values to any depth.

To show a list of collections, use show collections.
To list the contents of a collection, use db.towns.find()
- You will see a system generated field _id ( composed of a timestamp, client machine ID, client process ID, and a 3-byte incremented counter) (you can override this system generated
MongoDB commands are JavaScript functions.
- db is a JavaScript object that contains information about the current database. Try typeof db
- db.x is a JavaScript object that represent a collection named x within the current database. Try typeof db.towns
- db.x.help() will list available functions related to the given object. Try typeof db.towns.insert
- If you want to inspect the source code of a function, call it without parameter or parentheses.
Functions You can create JavaScript functions and call them on the mongo shell like:

function insertCity( name, population, last_census, famous_for, mayor_info) {
db.towns.insert({
name:name,
population:population,
last_census: ISODate(last_census),
famous_for:famous_for,
mayor : mayor_info
});
}
insertCity("Punxsutawney", 6200, '2008-31-01', ["phil the groundhog"], { name : "Jim Wehrle" } )
insertCity("Portland", 582000, '2007-20-09', ["beer", "food"], { name : "Sam Adams", party : "D" } )
Now have three towns in our collection

To get a specific document, we only need _id passed to find() function in type ObjectId(findOne() retrieves only one matching document). String can be converted to ObjectId using ObjectId(str) function.

db.towns.find({ "_id" : ObjectId("51def56c1cf66f4c40bb7f4a") })

The find() function also accepts an optional second parameter: a fields object we can use to filter which fields are retrieved. If we want only the town name (along with _id), pass in name with a value resolving to 1 (or true).

db.towns.find({ "_id" : ObjectId("51def56c1cf66f4c40bb7f4a") }, { name : 1})

To retrieve all fields except name, set name to 0 (or false or null).

db.towns.find({ "_id" : ObjectId("51def56c1cf66f4c40bb7f4a") }, { name : 0})

You can retrieve documents based on criteria other than _id. You can use regular expressions or any operator.

db.towns.find( { name : /^P/, population : { $lt : 10000 } }, { name : 1, population : 1 } )
We said before that the query language is JavaScript, which means we can construct operations as we would construct objects. In the following query, we build a criteria where the population must be between 10.000 and 1 million. Ranges work also on dates.
> var population_range = {}
> population_range['$lt'] = 100000
100000
> population_range['$gt'] = 10000
10000
> population_range['$lt'] = 1000000
1000000
> population_range['$gt'] = 10000
10000
> db.towns.find( {name : /^P/, population : population_range }, {name: 1})
{ "_id" : ObjectId("51df08e72476b99608460870"), "name" : "Portland" }

You can also query based on values in nested arrays, either matching exact values or matching partial values or all matching values or the lack of matching values.

> db.towns.find( { famous_for : 'food' }, { _id : 0, name : 1, famous_for : 1 } )
{ "name" : "New York", "famous_for" : [ "statue of liberty", "food" ] }
{ "name" : "Portland", "famous_for" : [ "beer", "food" ] }
> db.towns.find( { famous_for : /statue/ }, { _id : 0, name : 1, famous_for : 1 } )
{ "name" : "New York", "famous_for" : [ "statue of liberty", "food" ] }
> db.towns.find( { famous_for : { $all : ['food', 'beer'] } }, { _id : 0, name:1, famous_for:1 } )
{ "name" : "Portland", "famous_for" : [ "beer", "food" ] }
> db.towns.find( { famous_for : { $nin : ['food', 'beer'] } }, { _id : 0, name : 1, famous_for : 1 } )
{ "name" : "Punxsutawney", "famous_for" : [ "phil the groundhog" ] }

You can query a sub-document by giving the field name as a string separating nested layers with a dot.

> db.towns.find( { 'mayor.party' : 'I' }, { _id : 0, name : 1, mayor : 1 } )
{ "name" : "New York", "mayor" : { "name" : "Michael Bloomberg", "party" : "I" } }

To query the nonexistence of a field value

> db.towns.find( { 'mayor.party' : { $exists : false } }, { _id : 0, name : 1, mayor : 1 } )
{ "name" : "Punxsutawney", "mayor" : { "name" : "jim Wehrle" } }

elemMatch

$elemMatch helps us specify if a document or a nested document matches all of our criteria, the document counts as a match and returned. We can use any advanced operators within this criteria. To show it in action, let’s insert some data into a new collection countries
> db.countries.insert({ _id : "us",
... name : "United States",
... exports : {
... foods : [
...   { name : "bacon", tasty : true },
...   { name : "burgers" }
...          ]
...            }
... })
> db.countries.insert({ _id : "ca",
... name : "Canada",
... exports : {
... foods : [
...   { name : "bacon", tasty : false },
...   { name : "syrup", tasty : true }
...          ]
...             }
... })
> db.countries.insert({ _id : "mx",
... name : "Mexico",
... exports : {
... foods : [
...   { name : "salsa", tasty : true, condiment : true }
...          ]
...           }
... })
> print( db.countries.count() )
3
Now if we need to select countries that export tasty bacon, we should $elemMatch in a query like:
> db.countries.find(
... {
...   'exports.foods' : {
...    $elemMatch : {
...      name : "bacon",
...      tasty : true
...     }
...   }
... },
... { _id : 0, name : 1 }
... )
{ "name" : "United States" }
If we didn’t used $elemMatch and wrote a query like the following, it will return countries that export bacon or tasty food, not tasty bacon
> db.countries.find(
... { 'exports.foods.name' : 'bacon' , 'exports.foods.tasty' : true },
... { _id : 0, name : 1 }
... )
{ "name" : "United States" }
{ "name" : "Canada" }

Boolean Operators

$or a prefix for criteria to return document that match either condition1 or condition2. There is a a lot of operators you can use.
> db.countries.find({
... $or : [ { _id : "mx" } , { name : "United States" } ] },
... {_id : 1} )
{ "_id" : "us" }
{ "_id" : "mx" }

Update

The find() function toke two parameters, criteria and list of fields to return. The update() function works the same way. The first parameter is a criteria (the same way you will use to retreive the document through find()). The second parameter is an object whose fields will replace the matched document(s) or a modifier operation ($set to set a field value, $unset to delete a field, $inc to increment a field value by a number).
The following query will set the field state with the string OR for the matching document.
db.towns.update( { _id : ObjectId("4d0ada87bb30773266f39fe5") }, { $set : { "state" : "OR" } } )
but the following query will replace the matching document with a new document { state : “OR”}
db.towns.update( { _id : ObjectId("4d0ada87bb30773266f39fe5") }, { state : "OR" } )

References

Although MongoDB is schema-less but you can make one document reference another document using a construct like { $ref : “collection_name”, $id : “reference_id” }. In the following query we linking New York town with the country US. Notice the display of the new country field in the New York town document.
> db.towns.update(
... { _id : ObjectId("51def56c1cf66f4c40bb7f4a") },
... { $set : { country : { $ref : "countries", $id : "us" } } }
... )
> db.towns.find( { _id : ObjectId("51def56c1cf66f4c40bb7f4a") } )
{ "_id" : ObjectId("51def56c1cf66f4c40bb7f4a"), "country" : DBRef("countries", "us"), "famous_for" : [ "statue of liberty", "food" ], "last_census" : ISODate("2009-07-31T00:00:00Z"), "mayor" : { "na
me" : "Michael Bloomberg", "party" : "I" }, "name" : "New York", "population" : 22200000 }Now we can retrieve New york from towns collection, then use it to retrieve its country
> var NY = db.towns.findOne( { _id : ObjectId("51def56c1cf66f4c40bb7f4a") } )
> db.countries.findOne( { _id : NY.country.$id })
{
        "_id" : "us",
        "name" : "United States",
        "exports" : {
                "foods" : [
                        {
                                "name" : "bacon",
                                "tasty" : true
                        },
                        {
                                "name" : "burgers"
                        }
                ]
        }
}
or in a different way > db[ NY.country.$ref ].findOne( { _id : NY.country.$id} )

Delete

Removing documents from a collections is simple, use your criteria with a call to remove() function and all matching documents will be removed. It’s a recommended practice to build your criteria in an object, use that criteria to ensure that matching documents are the expected ones, pass this criteria to remove() function.
> var bad_bacon = { 'exports.foods' : {
... $elemMatch : { name : 'bacon', tasty : false }
... } }
> db.countries.find ( bad_bacon)
{ "_id" : "ca", "name" : "Canada", "exports" : { "foods" : [ { "name" : "bacon", "tasty" : false }, { "name" : "syrup", "tasty" : true } ] } }
> db.countries.remove( bad_bacon)
> db.countries.count()
2

Reading with Code

You could ask MongoDb to run a decision function across documents.

db.towns.find( function() {
return this.population > 6000 && this.population < 600000;
} )

or in a short-hand format db.towns.find("this.population > 6000 && this.population < 600000")
or combine custom code with other criteria using $where

db.towns.find( {
$where : "this.population > 6000 && this.population < 600000",
famous_for : /groundhog/
} )

Custom code should be your last resort due to the following:

Custom code queries run slower than regular queries.
Custom code queries can’t be indexed.
MongoDb can’t optimize custom code queries.
If your custom code assume the existence of a particular field and this field is missing in a single, the entire query will fail.

In this post we explored the basics of MongoDB, a rising star in the NoSQL family and the most common document database. We saw how to install and configure it. We saw how we can store nested structured data as JSON objects and query that data at any depth. We How we can update and delete data. In the next post we going to dig deep into MongoDB.

Getting Started with Apache Cassandra

noreply@blogger.com (Ebeid Soliman ElSayed) — Tue, 09 Jul 2013 22:27:00 +0000

Apache Cassandra is “an open source, distributed, decentralized, elastically scalable, highly available, fault-tolerant, tuneably consistent, column-oriented database that bases its distribution design on Amazon’s Dynamo and its data model on Google’s Bigtable” (source: “Cassandra: The Definitive Guide,” O’Reilly Media, 2010, p. 14).

Cassandra is built to store lots of data across a variety of machines arranged in a ring, in other words scaling horizontally, rather than vertically.

Data Model

Cassandra is based on a key-value model and it is organized according to the following concepts:

Column is a key-value pair.

Column Family is a set of key-value pairs (columns in Cassandra’s terminology). They are sorted by their keys. Families are referenced and sorted by row keys.

Super Column the value of a key-value pair can be a sequence of key-value pairs as well. In this case, the outer column would be called super column.

Columns and Super Columns can equally be used within Column Families

Columns or Super Columns are stored ordered by names within their Column Families

For a better understanding of Cassandra’s data model, refer to this article by Maxim Grinev.

Installation

Download the latest Cassandra version from here (I got version 1.2.6).
Extract the archive (I extracted it to C:\apache-cassandra-1.2.6 )
If you don’t have Java installed on your machine, go and get it installed.
Add environment variables
1. Reight-click the my Computer icon on your desktop or start menu.
2. Click the Advanced tab (or the Advanced System Settings)
3. Under System Variables, click New (adjust for your own directories)
  1. Variable Name : CASSANDRA_HOME
  2. Variable Value : C:\apache-cassandra-1.2.6
  3. click OK
4. Under System Variables, click New (adjust for your own directories)
  1. Variable Name : JAVA_HOME
  2. Variable Value : C:\Program Files\Java\jre7
  3. click OK
Now, open command window and navigate to your bin directory inside Cassandra directory (C:\apache-cassandra-1.2.6\bin in my case).
Launch Cassandra by executing the comand cassandra –f (the “-f” causes it to run in the foreground). You will see lots of messages coming out. If everything goes fine, it will end up with something like this:

Now we have a running Cassandra server is expecting incoming connections on port 9160.

Once Cassandra is up and running on your machine, we can connect to the running instance using the Cassandra command-line interface, launched by running “cassandra-cli.bat”, from the Cassandra “bin” directory.

Commads

show api version; to show the current api version.
describe cluster; to show a description of the current cluster.

create keyspace TestKS; to create a keyspace and it have to be a unique name.
use TestKS; to switch to keyspace TestKS.
create column family TestCF; to create a column family TestCF within the current keyspace.
- No other schema definition is required, the column family is a collection of name/value pairs.
set TestCF[ascii('TestKey')][ascii('column1')]=ascii('TestValue'); to insert the TestKey/TestValue key/value pair into the column named column1 within the column family TestCF. You can use the with ttl = x setting at the end of the set command to make the column self-delete aft x seconds of the insertion time.
- by default Cassandra treats data as byte arrays but you can convert to other types such as Long, int, integer,…. Also timeuuid() generates new UUID. For full information about set command: type help set;
get TestCF[ascii("TestKey")]; to retrieve the value stored in the key TestKey within the column family TestCF

del TestCF[ascii(‘TestKey’)][ascii(‘column2’)]; rows and columns can be deleted by specifying the row key and/or the column name
with the del (delete) command.

list TestCF; list the data inside a column family

drop column family TestCF; removes a column family.
drop keyspace TestKS; removes a key space.

==> You can insert to super columns much like inserting to normal columns. They can be read with get, written with set, and deleted with del. The super column version of these commands uses an extra ['xxx'] to represent the extra sub-index level.

assume TestCF comparator as ascii; it decodes and helps display results of get and list requests inside the command-line interface. It can be used in the same way to set the validator and keys. By default, columns with no metadata are displayed in a hex format. This is done because row keys, column names, and column values are byte arrays. After using assume the column and value will be displayed rather than the hex code.

Type Enforcement : Cassandra is designed to store and retrieve simple byte arrays but it also have support for built-in types. When creating or updating a column family, the user can supply column metadata that instructs the CLI Cassandra client on how to display data and help the server enforce types during insertion operations.
create column family User with comparator = UTF8Type;
update column family User with
        column_metadata =
        [
        {column_name: first, validation_class: AsciiType},
        {column_name: last, validation_class: AsciiType},
        {column_name: age, validation_class: IntegerType, index_type: KEYS}
        ];
Querying data :

set User[ascii('jsmith')][ascii('first')] = ascii('John');
set User[ascii('jsmith')][ascii('last')] = ascii('Smith');
set User[ascii('jsmith')][ascii('age')] = '38';

get User where age = '38';

Update is the same as set.

set User[ascii('jsmith')][ascii('first')] = ascii('Jack');

Quite the client;

quit;

In this post we just touched the Cassandra’s iceberg. In later posts we will dig more in it and how to write .net programs against it.

جريدة الشروق : تقرير شهادات سكان عمارات العبور

noreply@blogger.com (Ebeid Soliman ElSayed) — Mon, 08 Jul 2013 16:53:00 +0000

سكان «عمارات العبور»: ضرب النار والغاز بدأ من الحرس الجمهوري بدون مقدمات

تعليقات: 117شارك بتعليقك

نشر فى : الإثنين 8 يوليو 2013 - 4:13 م
آخر تحديث : الإثنين 8 يوليو 2013 – 4:13 م

اشتباكات الحرس الجمهوري – أرشيفية

كتب- مصطفى هاشم:

رصدت "الشروق" أحداث الحرس الجمهوري التي أودت بحياة أكثر من 50 من مؤيدي الرئيس المعزول محمد مرسي و2 من الشرطة، من خلال سكان العمارات المقابلة للحرس الجمهوري، والتي يطلق عليها عمارات"العبور".

قال الدكتور محمود سليمان أحد السكان بالعمارة المقابلة للحرس الجمهوري "استيقظت بعد الفجر مباشرة على صوت الرصاص والهليكوبتر واختناق بسبب قنابل الغاز المسيل للدموع رغم أن كل زجاج شقتنا كان مغلقا.. بدأت ارى ما يحدث من خلف الزجاج ووجدت دبابات الجيش والمدرعات ناحية المتظاهرين.. ووجدتهم يحرقون خيام المعتصمين، وكان معهم بعض الشباب الذين يرتدون زيا مدنيا".

وأضاف "بعد 5 دقائق فقط من سماعي لصوت الرصاص دق باب شقتنا ووجدت 3 من الشباب يطلبون مني مساعدتهم لأن الجيش والشرطة تلاحقهم.. وبالفعل أدخلتهم في إحدى الغرف وأغلقتها عليهم.. بعدها بدقائق بسيطة كانت الشرطة العسكرية والشرطة قد ملأت عمارتنا وبدأت في ضرب النار داخل كل أدوار العمارة وبدأوا يهددون السكان، حتى أن الآثار موجودة.. ويدقون الأبواب بقوة وقاموا بكسر باب إحدى الشقق لأن صاحبها لم يفتح لهم لأنه لم يكن فيها أحد من الأساس".

وتابع: "أعتقد أن الشرطة العسكرية ضربت النار على بعض المتظاهرين الذين كانوا يحتمون بعمارتنا.. ضرب النار من قبل الجيش والشرطة استمر لنحو ساعة كاملة وطائرة الهليكوبتر كانت تحلق في الجو لكشف ما يحدث".

وأكد سليمان أن سيارات الإسعاف لم تكن موجودة طوال الساعة، وجاءت بعدما انتهى ضرب النار لحمل الجثث من الشوارع ومن العمارات".

وحول المعتصمين الذين اختبأوا في شقته قال "لم أخرجهم إلا الساعة العاشرة، واحدا بعد الآخر لأن معظم الشقق كان يختبئ فيها معتصمون وبدأنا في إخراجهم بالتتابع".

أما محمد ثابت عامل في جراج إحدى العمارات المقابلة للحرس الجمهوري فقال "الشرطة والجيش فجأة وبدون مقدمات بدأت في ضرب النار والغاز اختنق أطفالي الصغار واستيقظوا مرعوبين.. رأيت بعض المعتصمين قد دخلوا في الجراج للاحتماء به وبعضهم اختبأ تحت بعض السيارات لكن الشرطة العسكرية والشرطة دخلوا وراءهم وبدؤوا في إخراجي أيضا معهم إلى أن خرجنا للشارع واعتبروني ضمن المعتصمين وبدؤوا في سبنا وإهانتنا وضربنا، إلى أن تعرف علي بعض الضباط والمجندين الذين يخدمون في الحرس الجمهوري لأنهم يعرفونني وأخرجوني من بينهم".

أما أحمد همام عامل أمن في إحدى مكاتب المحاسبة في إحدى العمارات المقابلة للحرس الجمهوري كنت أصلي الفجر مع المعتصمين أمام الحرس الجمهوري والإمام انتهى من الصلاة سريعا وبعدها بدأ بعض المشايخ ناشد الجيش بأن الاعتصام سلميا وأنهم إخوتهم بكن فجأة وبدون مقدمات وبدون أي استفزازات لأنني كنت معهم وهم أمام العمارة التي أعمل بها.. بدأ الجيش والشرطة في ضرب النار الحي والغاز المسيل للدموع.

وأضاف "المعتصمون بدأوا في الجري ودخلوا بعض العمارات ومنهم العمارة التي أعمل بها، ولكن الشرطة العسكرية والشرطة ومباحث أمن الدولة لاحقوهم.. وأدخلت بعض النساء والأطفال والشباب داخل الشقة ووصل عددهم إلى 40 شخصا، وكانت منهم عضوة بمجلس الشعب المنحل عن حزب الحرية والعدالة..الشرطة العسكرية هددونا وهددوا السكان بضرب النار الحي إن لخرجوا من عندهم.. البعض استجاب وضربوا النار داخل العمارة التي أعمل بها".

وأضاف همام "استمر المعتصمون في الاختباء عندي إلى الساعة 12 ظهرا وبدأت بعدذلك في إخراجهم واحدا تلو الآخر لأن الشرطة العسكرية ومباحث أمن الدولة يملئون الشوارع المحيطة بالحرس.

أما أحد المعتصمين الذين اختبأوا لدى إحدى الشقق السكنية أحمد الديب فقال "لم نهاجم الحرس الجمهوري كما يقول بعض وسائل الإعلام ولو كنا نريد ذلك كنا هاجمناهم ونحن عددنا كبير بعد صلاة العشاء ولكن نحن لا نريد أن نفعل ذلك من الأساس"

وحول بداية الأحداث قال "أيقظني احدهم من النوم لصلاه الفجر استيقظت و ذهبت للوضوء ثم ذهبت للصلاة أمام الحرس الجمهوري بدا الإمام يدعي في الركعة الثانية وبدون سابق إنذار تلعثم وأنهى الدعاء و في نهايه الصلاة بدأ الضرب من 3 محاور هي صلاح سالم، ويوسف عباس، ونادي الحرس الجمهوري نفسه".

وأضاف الديب "بدأت قنابل الغاز.. ولم نكن ندري أين نذهب فنحن محاصرون بضرب الخرطوش والآلي والرصاص الحي من كل اتجاه بدون حرمه للنساء ولا الأطفال، وسقط عدد كبير من القتلى من الأطفال والنساء وكبار السن، وسقط عدد ضخم من الإصابات والإغماءات.. اختبأت في أحد العمارات وهرع عدد من المتظاهرين معي و اختبأنا وعدد كبير من النساء دخلوا احد الشقق و أغلقنا عليهم و لم اعلم مصيرهم بعد وهجم الأمن والجيش على العمارة و اعتقلوا كل من صعد اليها و لكن حماني الله عندما فتح احد السكان شقته لي وانقطعت علاقتي بالشارع حتي الظهر".

------------------------------------------------------------

The above article is retrieved from the Google cashed version because the original have been deleted.

http://webcache.googleusercontent.com/search?q=cache%3Ahttp%3A%2F%2Fshorouknews.com%2Fnews%2Fview.aspx%3Fcdate%3D08072013%26id%3D28a98832-a914-4633-92d9-16fab99a42a1&rlz=1C1CHKZ_enEG430EG430&oq=cache%3Ahttp%3A%2F%2Fshorouknews.com%2Fnews%2Fview.aspx%3Fcdate%3D08072013%26id%3D28a98832-a914-4633-92d9-16fab99a42a1&aqs=chrome.0.57j58.1469j0&sourceid=chrome&ie=UTF-8

Now the article have been removed

Redis 101– Part 2

noreply@blogger.com (Ebeid Soliman ElSayed) — Wed, 03 Jul 2013 17:47:00 +0000

We introduced many of the Redis’s fundamental concepts and commands in the last post. In this post we going to introduce some advanced features.

Publish-Subscribe

Previously we queued data that could be read by a blocking pop command. Using that queue, we made a very basic publish-subscribe model. Any number of messages could be pushed to this queue, and a single queue reader would pop messages as they were available. This is powerful but limited. Redis provides some specialized publish-subscribe (or pub-sub) commands.

SUBSCRIBE subscribe a client to a key, known as a channel in pub-sub terminology; client will block until messages are available.

PUBLISH will push message into the channel returning how many subscribers have received it

Publisher window
Subscriber window receives : the string “message”, the channel name, the published message value

UNSUBSCRIBE unsubscribe or disconnect from the specified channel. If no channel name provided, it will disconnect from all channels.

Server Information

INFO returns a list of server data, including version, process ID, memory used, and uptime.

In order to change any of the Redis’s default configurations, you need to edit redis.config file at C:\Program Files\Redis\conf . It is fairly self-explanatory.

Durability

Redis has a few persistence options:

No persistence at all, which simply keeps all values in main memory.
Forced save

Use command SAVE to force server to save database to disk.
Use command BGSAVE to force server to save database to disk asynchronously in the background.
Use command LASTSAVE to get a timestamp of the last time a Redis disk write succeeded (also provided through the last_save_time field in the server INFO output).

Snapshotting

By default Redis saves snapshots of the dataset on disk, in a binary file called dump.rdb. You can configure Redis to have it save the dataset every N seconds if there are at least M changes in the dataset, or you can manually force it by calling the SAVE or BGSAVE commands as we said before. For example, the following configuration will make Redis automatically dump the dataset to disk every 60 seconds if at least 1000 keys changed (This strategy is known as snapshotting.):

save 60 1000

Append-only file

Snapshotting is not very durable. If your computer running Redis stops, the latest data written on Redis will get lost. While this may not be a big deal for some applications, there are use cases for full durability, and in these cases Redis was not a viable option.

The append-only file is an alternative, fully-durable strategy for Redis. It became available in version 1.1. You can turn on the AOF in your configuration file:

appendonly yes

Then we must decide how often a command is appended to the file. Setting always is the more durable, since every command is saved. By default everysec is enabled, which saves up and writes commands only once a second. This is a decent trade-off, since it’s fast enough, and worst case you’ll lose only the last one second of data. Finally, no is an option, which just lets the OS handle flushing. It can be fairly infrequent, and not recommended.

appendfsync always

# appendfsync everysec

# appendfsync no

From now on, every time Redis receives a command that changes the dataset (e.g. SET) it will append it to the AOF. When you restart Redis it will re-play the AOF to rebuild the state. Append-only has more detailed parameters, you could refer to the online documentation for more details.

Security

Redis provides command-level security through obscurity, by allowing you to hide or suppress commands. This will rename the FLUSHALL command (remove all keys from the system) into some hard-to-guess value like c283d93ac9528f986023793b411e4ba2:

rename-command FLUSHALL c283d93ac9528f986023793b411e4ba2

If we attempt to execute FLUSHALL against this server, we’ll be hit with an error. We can also disable the command entirely by setting it to a blank string.

rename-command FLUSHALL ""

You can set any number of commands to a blank string to allow only a customized subset of the Redis’s commands.

Benchmarking

Redis provides an excellent benchmarking tool. It connects locally to port 6379 by default and issues 10,000 requests using 50 parallel clients. Tool test many commands and have a long output report. Tool is in C:\Program Files\Redis\

Replication

Redis supports master-slave replication. One server is the master by default if you don’t set it as a slave of anything. Data will be replicated to any number of slave servers. Configuring master-slave setting is the same as running another instance with the slaveof option in the salve’s conf file set to the master IP and port. Refer to our previous post for more details.

This concludes out Redis 101 tutorial, we tried to touch all basic and moderate features in it. Later we going to show how to write C# programs against Redis.

Running multiple Redis instances on the same server

noreply@blogger.com (Ebeid Soliman ElSayed) — Tue, 02 Jul 2013 16:42:00 +0000

Redis runs as a background process that listens to a specific port (6379 by default) for incoming requests from clients. Running multiple instances requires a separate conf file and a new init script. The conf file specifies details for the new instance, and the init script handles starting/stopping of the instance background process.

Make a copy of the redis.conf file at C:\Program Files\Redis\Conf and give it a new name redis-s1.conf. Leave both files in the same directory C:\Program Files\Redis\Conf
Open the redis-s1.conf with your favorite text editor (e.g. notepad ) and change the following:

PID File

pidfile /var/run/redis-s1.pid

Port

port 6380

Log File

logfile "C:/Program Files/Redis/logs/redis-s1.log

Data Directory (don’t forget to create that directory)

dir "C:/Program Files/Redis/data2"

For replication only : If you planning to use this instance as a slave in master-slave replication setting (where 127.0.0.1 is the master instance IP, 6379 in the master instance port)

slaveof 127.0.0.1 6379

To start the new instance

Open a new command window
Navigate to C:\Program Files\Redis
Run redis-server conf\redis-s1.conf

To connect to the new instance

Open a new command window
Navigate to C:\Program Files\Redis
Run redis-cli –h 127.0.0.1 –p 6380

Now you can have multiple Redis instances on the same server like any other DBMS.

Redis 101

noreply@blogger.com (Ebeid Soliman ElSayed) — Wed, 26 Jun 2013 20:29:00 +0000

Redis (REmote DIctionary Service) is an open-source networked in-memory key-value data store. First released in 2009, currently sponsored by VMware, and since then it have been ranked the most popular key-value store by DB-Engine. Redis creator Salvatore Sanfilippo refers to his project as a “data structure server” to capture its unique handling of complex data types and other features. Interested ? Enough talking, let’s get down to business.

Download Redis installer from here.

Installation is straight forward:

Start the Server go to Control Panel >> Administrative Tools >> Services , right click Redis Server and select Start.

Start the Client look for Redis Client (with red cube icon) in your start menu and open it. As you see it listens to port 6379 by default. You could ping your server by writing PING, if everything is good you will get PONG in return.

GET & SET

As we said before, Redis is a key-value store. We store value inside a key and retrieve it later using that key. We can use the command SET to store the value "fido" at key "server:name" (server replies with Ok if everything is good) (Note that we used colons [:] within our key. This is a valid character that often logically separates a key into segments. It’s merely a matter of convention, with no deeper meaning in Redis.)

 SET server:name "fido"

To retrieve our value, we use GET command (server replies with “fido”)

 GET server:name

You could also delete a given key and its associated value using DEL command and you can check for key existence using EXISTS (1 exists, 0 otherwise). We get and set multiple values in one command using MSET & MGET

 MSET 1 valueA 2 valueB

 MGET 1 2

Although Redis stores strings, it recognizes integers and provides some simple operations for them. If we want to keep a running total of something, we can create a count and then increment it with the INCR command (gives error for noninteger values).

 SET count 2

 INCR count

 GET count

You can also increment by any integer (INCRBY) or decrement (DECR, DECRBY).

It is good to know that these simple operations are atomic in Radis.

Expire

You can tell Radis to hold a key only for a certain period of time (in seconds) using EXPIRE command. You can test how long a key will exist for with the TTL command. It returns the number of seconds until it will be deleted (-1 means the key will never expire). All keys are permanent by default.

Setting and expiring keys is so common that Redis provides a shortcut command called SETEX. Also at any moment before the key expires, you can remove the timeout by running PERSIST KeyName.

A common trick for keeping only recently used keys is to update the expire time whenever you retrieve a value. This will ensure that your most recently used keys will remain in Redis, while the least recently used keys will just expire as normal.

Transactions

Redis like most DBMSs supports transactions. We begin the transaction with the MULTI command and execute it with EXEC command. So, wrapping two operations like SET and INCR in a single block will complete either successfully or not at all.

As you may guess from the server response inside the transaction, commands are not executed instantly. Instead, they are queued and then executed in sequence.

Similar to ROLLBACK in SQL, you can stop a transaction with the DISCARD command, which will clear the transaction queue. Unlike ROLLBACK, it won’t revert the database; it will simply not run the transaction at all. The effect is identical, although the underlying concept is a different mechanism (transaction rollback vs. operation cancellation).

Complex Data Types

Radis can store lists, hashes, sets, and sorted sets natively. These collection data types can contain a huge number of values (up to 2^32 elements or more than 4 billion) per key. That’s more than enough for all Facebook accounts to live as a list under a single key.

List

A list is a series of ordered values that can act both as queues (first value in, first value out) and as stacks (last value in, first value out). Some of the important commands for interacting with lists are RPUSH, LPUSH, LLEN, LRANGE, LPOP, and RPOP. You can immediately begin working with a key as a list, as long as it doesn't already exist as a different type.

RPUSH puts the new value at the end (right) of the list.

LPUSH puts the new value at the start (left) of the list.

LRANGE gives a subset of the list. It takes the index of the first element you want to retrieve as its first parameter and the index of the last element you want to retrieve as its second parameter. A value of -1 for the second parameter means to retrieve all elements in the list.

LLEN returns the current length of the list.

LPOP removes the first (left) element from the list and returns it.

RPOP removes the last (right) element from the list and returns it.

Use LPUSH/RPOP to make List act like a queue.

Use LPUSH/LPOP to make List act like a stack.

LREM removes (some or all) matching elements from the list. It also requires a number to know how many matches to remove. Setting the count to 0 as we do here just removes them all. Setting the count greater than 0 will remove only that number of matches, and setting the count to a negative number will remove that number of
matches but scan the list from the end (right side).

RPOPLPUSH a single command for popping values from the tail of one
list (right pop) and pushing to the head of another (left push). Surprisingly, there is no commands for RPOPRPUSH, LPOPLPUSH, or LPOPRPUSH.

Blocking Lists

It really interesting to know that Redis provides message queuing APIs natively. So, assume you want to write a simple messaging system where multiple clients can push data to one side of the queue and one or more listeners pop data from the other side of the queue. Listeners should just listen for new data and pop them as they arrive.

BRPOP the blocking version of RPOP, it blocks the connection when there are no elements to pop from any of the given lists.

now open another Redis client window and push a message into messages list

if you switched back to the blocking BRPOP command window, you will find it return the key, the popped value, and the time spent blocking

BLPOP the blocking version of left pop LPOP

BRPOPLPUSH the blocking version of right pop left push BRPOPLPUSH

Set

Sets are unordered collections with no duplicate values and are an excellent choice for performing complex operations between two or more key values, such as unions or intersections.

SADD adds the given value to the set (duplication is not allowed).

SREM removes the given value from the set.

SISMEMBER tests if the given value is in the set.

SMEMBERS returns a list of all the members of this set.

SUNION combines two or more sets and returns the list of all elements.

SINTER returns the intersection of two or more lists.

SDIFF returns the elements that exist in the first list only but not in the second list.

SUNIONSTORE / SIMTERUNION / SDIFF stores the result of operation done on the second and third parameters into the first parameter.

SMOVE moves one element from one set to another.

SCARD counts the set elements (set cardinality).

SPOP returns a random element from a set and removes it.

SRANDMEMBER returns a random element from a set but does not remove it

Sorted Sets

Sorted set is similar to a regular set, but now each value has an associated score. This score is used to sort the elements in the set. Sorted sets take something from each of the other collections data types. They are ordered like lists and are unique like sets. They have field-value pairs like hashes, but rather than string fields, they are instead numeric scores that denote the order of the values. Internally, sorted sets keep values in order, so inserts can take log(N) time to insert (where N is the size of the set), rather than the constant time complexity of hashes or lists.

ZADD add a value with a specified score.

ZRANGE returns the elements between start index (zero-based) and end index (-1 means to the end of the sorted set. Put WITHSCORES at the end of command to get elements and scores, not only the elements. To get them in reverse, insert the word REV, as in ZREVRANGE.

ZRANGEBYSCORE since we are using a sorted set, it’s more likely to get elements by score ranges instead of index ranges (x <= score <= y). We can make a score number exclusive by prefixing it with an opening parenthesis ( x < score <= y).

-inf / inf negative / positive infinities. can be used in retrieval (by index or by score). We can also range by both positive and negative values.

ZREMRANGEBYRANK / ZREMRANGEBYSCORE remove values by rank or score.

ZINCRBY Increments the score of member in the sorted set stored at key by increment. If member does not exist in the sorted set, it is added with increment as its score (as if its previous score was 0.0). If key does not exist, a new sorted set with the specified member as its sole member is created. It is possible to provide a negative value to decrement the score.

ZUNIONSTORE Computes the union of numkeys sorted sets given by the specified keys, and stores the result in destination set. It is mandatory to provide the number of input keys (numkeys) before passing the input keys and the other (optional) arguments. Parameters needed:

destination ==> a set to hold the union operation result

numkeys ==> number of keys you are going to join.

weight ==> optional number(s) to multiply each score of the relative key by (if you have two keys, you can have two weights and son on)

aggregate ==> optional rule to specify how the results of the union are aggregated. This option defaults to SUM, where the score of an element is summed across the inputs where it exists. When this option is set to either MIN or MAX, the resulting set will contain the minimum or maximum score of an element across the inputs where it exists.

In the above example we prepared two sorted sets, sset1 with elements (1 “one” 2 “two”) , sset2 with elements (1 “one” 2 “two” 3 “three”). Then tell Redis to merge these two sets with the following parameters:

store the operation result in sorted set named out

we going to merge 2 sorted sets, then provide them sset1 sset2.

we going to use WEIGHTS, then provide these weights 2 3.

Hash

Hash is a collections object that can hold any number of key-value pairs.

HMSET store the key-value pairs into the hash (first parameter.

HVALS returns the values of a hash.

HKEYS returns the keys of a hash.

HGET returns the value of of a single key in a hash.

HDEL deletes a key-value pair from a hash.

HSET updates or adds a value at a specified key in a hash.

HINCRBY increments an integer field value by some count in a hash. It is possible to provide a negative value to decrement.

HLEN returns the number of fields in a hash.

Database Namespaces

You may have been wondering, which database we working on. So far, we have interacted only with a single namespace. In Redis’s terminology, a namespace is called a database and is keyed by a number. So far, we’ve always interacted with the default namespace 0 (also known as database 0).

Use SELECT command to switch to different database and provide it a DB number. Setting a value to a key in one database will not affect any other key in other databases.

Since all databases are running in the same server instance, Redis lets us move keys from one database to another with the MOVE command.

This can be useful for running different applications against a single Redis server but still allow these multiple applications to trade data between each other.

More commands

RENAME rename a key

TYPE return the type of a key’s value

DEL deletes a key-value pair

FLUSHDB removes all keys in database

FLUSHALL removes all keys from all databases on this server instance.

Redis’s data types and the complex queries it can perform make it much more than a standard key-value store. It can act as a stack, queue, or priority queue; can be an object store (via hashes); and even can perform complex set operations such as unions, intersections, and subtractions (diff). It provides many atomic commands, and for those multistep commands, it provides a transaction mechanism. It has a built-in ability to expire keys, which is useful as a cache.

In this post we just introduced Redis as a data structure server. In later post we going to cover the advanced features of Redis.

PostgreSQL 101

noreply@blogger.com (Ebeid Soliman ElSayed) — Mon, 24 Jun 2013 21:59:00 +0000

PostgreSQL is the finest open source example of the relational database management system (RDBMS) tradition. This means that PostGreSQL is a design-first data store. First you design the schema, and then you enter data that conforms to the definition of that schema.

History of PostgreSQL

PostgreSQL has existed in the current project incarnation since 1995, but its roots
are considerably older. The original project was written at Berkeley in the early 1970s
and called the Interactive Graphics and Retrieval System, or “Ingres” for short. In the
1980s, an improved version was launched post-Ingres—shortened to Postgres. The
project ended at Berkeley proper in 1993 but was picked up again by the open source
community as Postgres95. It was later renamed to PostgreSQL in 1996 to denote its
rather new SQL support and has remained so ever since.

Installation

You can download the latest PostgreSQL installer from http://www.postgresql.org/download (In this tutorial I’m using the Windows 64bit installer). Setup is straight-forward. Below are screen-shots of installation steps:

remember the superuser password, you will need it later

when the PostgreSQL installation finish, you could choose to run Stack Builder to install additional tools. In my case I chose to run Stack builder

choose your recently installed instance

and I choose to install Npgsql database driver (for future .Net development) and phpPgAdmin for administration tasks.

Stack Builder will lunch installation wizards for your chosen tools in sequence

Starting with PostgreSQL

To get started, open pgAdmin III from your start menu. On the left hand side right-click on your server instance and click Connect

enter the password you used in the setup

Once you connected to your server, you can right click on the databases node and click New Database

we going to create a simple database called book with the default settings

to start writing queries, you could open Tools menu, then click Query Tool (or press CTRL + E).

Working with Tables

Creating a table consists of giving it a name and a list of columns with types
and (optional) constraint information. Each table should also nominate a
unique identifier column to pinpoint specific rows. That identifier is called a
PRIMARY KEY. The SQL to create a countries table looks like this:

CREATE TABLE countries (country_code char(2) PRIMARY KEY,country_name text UNIQUE);

copy and paste this code in the Query Tool and click on the toolbar to run it.

To insert values in our countries table:

INSERT INTO countries (country_code, country_name)
VALUES ('us','United States'), ('mx','Mexico'), ('au','Australia'),
('uk','United Kingdom'), ('de','Germany'), ('ll','Loompaland');

To select from our table:

SELECT * FROM countries;

To delete :

DELETE FROM countries
WHERE country_code = 'll';

Now, let’s add a cities table. To ensure any inserted country_code also exists in our countries table, we add the REFERENCES keyword. Since the country_code column references another table’s key, it’s known as the foreign key constraint.

CREATE TABLE cities (
name text NOT NULL,
postal_code varchar(9) CHECK (postal_code <> ''),
country_code char(2) REFERENCES countries,
PRIMARY KEY (country_code, postal_code)
);

we constrained the name in cities by disallowing NULL values. We constrained postal_code by checking that no values are empty strings (<> means not equal). Furthermore, since a PRIMARY KEY uniquely identifies a row, we created a compound key: country_code + postal_code. Together, they uniquely define a row. now if you try to run this code you will get an error because we violating referential integrity:

INSERT INTO cities
VALUES ('Toronto','M4C1B5','ca');

To update records:

INSERT INTO cities
VALUES ('Portland','87200','us');
UPDATE cities
SET postal_code = '97205'
WHERE name = 'Portland';

Inner Join

Being a relational database gives PostgreSQL the ability to join tables together when reading them. Joining, in essence, is an operation taking two separate tables and combining them in some way to return a single table. It’s somewhat like shuffling up Scrabble pieces from existing words to make new words. The basic form of a join is the inner join. In the simplest form, you specify two columns (one from each table) to match by, using the ON keyword.

SELECT cities.*, country_name
FROM cities INNER JOIN countries
ON cities.country_code = countries.country_code;

The join returns a single table, sharing all columns’ values of the cities table plus the matching country_name value from the countries table. We can also join a table like cities that has a compound primary key. To test a compound join, let’s create a new table that stores a list of venues. A venue exists in both a postal code and a specific country. The foreign key must be two columns that reference both cities primary key columns. (MATCH FULL is a constraint that ensures either both values exist or both are NULL.)

CREATE TABLE venues (
venue_id SERIAL PRIMARY KEY,
name varchar(255),
street_address text,
type char(7) CHECK ( type in ('public','private') ) DEFAULT 'public',
postal_code varchar(9),
country_code char(2),
FOREIGN KEY (country_code, postal_code)
REFERENCES cities (country_code, postal_code) MATCH FULL
);

This venue_id column is a common primary key setup: automatically incremented integers (1, 2, 3, 4, and so on…). Creating new row will populate this column automatically. We make this identifier using the SERIAL keyword. CHECK keyword check that a column supplied value against a set of predefined values. DEFAULT provide a default value to be inserted in case user supplied nothing.

INSERT INTO venues (name, postal_code, country_code)
VALUES ('Crystal Ballroom', '97205', 'us');

Joining the venues table with the cities table requires both foreign key columns.

SELECT venues.venue_id, venues.name, cities.name
FROM venues INNER JOIN cities 
ON venues.postal_code = cities.postal_code AND venues.country_code = cities.country_code;

Outer Join

Outer joins are a way of merging two tables when the results of one table must always be returned, whether or not any matching column values exist on the other table. Let's create a table and populate it for testing this:

CREATE TABLE events (
event_id SERIAL PRIMARY KEY,
title varchar(255),
starts timestamp,
ends timestamp,
venue_id integer,
FOREIGN KEY (venue_id)
REFERENCES venues (venue_id) MATCH FULL
);

INSERT INTO events (title, starts, ends, venue_id)
VALUES ('LARP Club', '2012-02-15 17:30:00', '2012-02-15 19:30:00', 1);
INSERT INTO events (title, starts, ends)
VALUES ('April Fools Day', '2012-04-01 00:00:00', '22012-04-01 23:59:00');
INSERT INTO events (title, starts, ends)
VALUES ('Christmas Day', '2012-12-25 00:00:00', '2012-04-01 23:59:00');

Now the results of the following two queries showing you the difference between INNER JOIN and OUTER JOIN.

SELECT events.title, venues.name
FROM events JOIN venues
ON events.venue_id = venues.venue_id;

SELECT events.title, venues.name
FROM events LEFT JOIN venues
ON events.venue_id = venues.venue_id;

Finally, there’s the FULL JOIN, which is the union of LEFT and RIGHT; you’re guaranteed all values from each table, joined wherever columns match.

Indexes

RDBMSs uses indexes to reducing disk reads and query optimization (among other things). If we select the title of Christmas Day from the events table, the algorithm must scan every row for a match to return. Without an index, each row must be read from disk to know whether a query should return it. An index is a special data structure built to avoid a full table scan when performing a query. PostgreSQL (like any other RDBMS) automatically creates an index on the primary key, where the key is the primary key value and where the value points to a row on disk. Using the UNIQUE keyword is another way to force an index on a table column. PostgreSQL also creates indexes for columns targeted by FOREIGN KEY constraint.

You can explicitly add a hash index using the CREATE INDEX command, where each value must be unique. btree is the suitable data structure for ranges. For more information, refer to online documentation.

CREATE INDEX events_starts
ON events USING btree (starts);

Aggregate Functions

An aggregate query groups results from several rows by some common criteria. It can be as simple as counting the number of rows in a table or calculating the average of some numerical column. Examples:

SELECT count(title)
FROM events

SELECT min(starts), max(ends)
FROM events

Aggregate functions are useful but limited on their own. If we wanted to count all events at each venue, we could write the following for each venue ID (which is tedious):

SELECT count(*) FROM events WHERE venue_id = 1;
SELECT count(*) FROM events WHERE venue_id = 2;
SELECT count(*) FROM events WHERE venue_id = 3;
SELECT count(*) FROM events WHERE venue_id IS NULL;

Grouping

With GROUP BY, you tell Postgres to place the rows into groups and then perform some aggregate function (such as count()) on those groups.

SELECT venue_id, count(*)
FROM events
GROUP BY venue_id;

The GROUP BY condition has its own filter keyword: HAVING. HAVING is like the WHERE clause, except it can filter by aggregate functions (whereas WHERE cannot).

SELECT venue_id
FROM events
GROUP BY venue_id
HAVING count(*) = 1;

You can use GROUP BY without any aggregate functions. If you SELECT one column, you get all unique values.
SELECT venue_id FROM events GROUP BY venue_id;
This kind of grouping is so common that SQL has a shortcut in the DISTINCT keyword.
SELECT DISTINCT venue_id FROM events;
The results of both queries will be identical.

Window Functions

Window functions are similar to GROUP BY queries in that they allow you to run aggregate functions across multiple rows. The difference is that they allow you to use built-in aggregate functions without requiring every single field to be grouped to a single row. If we attempt to select the title column without grouping by it, we can expect an error.

We are counting up the rows by venue_id, and in the case of LARP Club and Wedding, we have two titles for a single venue_id. Postgres doesn’t know which title to display. Whereas a GROUP BY clause will return one record per matching group value, a window function can return a separate record for each row. Window functions return all matches and replicate the results of any aggregate function. It returns the results of an aggregate function OVER a PARTI TI ON of the result set.

Transactions

Transactions ensure that every command of a set is executed. If anything fails along the way, all of the commands are rolled back like they never happened. It’s the all or nothing motto that gives relational databases its consistency capability.

PostgreSQL transactions follow ACID compliance, which stands for Atomic (all ops succeed or none do), Consistent (the data will always be in a good state—no inconsistent states), Isolated (transactions don’t interfere), and Durable (a committed transaction is safe, even after a server crash). Transactions are useful when you’re modifying two tables that you don’t want out of sync.

We can wrap any transaction within a BEGIN TRANSACTION block. To verify atomicity, we’ll kill the transaction with the ROLLBACK command.

Stored Procedures

Every command we’ve seen until now has been declarative (executed on the client side), but you can execute code on the database side also. Example of a stored procedure and how to run it:

CREATE OR REPLACE FUNCTION add_event( title text, starts timestamp,
ends timestamp, venue text, postal varchar(9), country char(2) )
RETURNS boolean AS $$
DECLARE
did_insert boolean := false;
found_count integer;
the_venue_id integer;
BEGIN
SELECT venue_id INTO the_venue_id
FROM venues v
WHERE v.postal_code=postal AND v.country_code=country AND v.name ILIKE venue
LIMIT 1;
IF the_venue_id IS NULL THEN
INSERT INTO venues (name, postal_code, country_code)
VALUES (venue, postal, country)
RETURNING venue_id INTO the_venue_id;
did_insert := true;
END IF;
-- Note: not an “error”, as in some programming languages
RAISE NOTICE 'Venue found %', the_venue_id;
INSERT INTO events (title, starts, ends, venue_id)
VALUES (title, starts, ends, the_venue_id);
RETURN did_insert;
END;
$$ LANGUAGE plpgsql;

SELECT add_event('House Party', '2012-05-03 23:00',
'2012-05-04 02:00', 'Run''s House', '97205', 'us');

Triggers

Triggers automatically fire stored procedures when some event happens, like an insert or update. They allow the database to enforce some required behavior in response to changing data.

Let's create a function that logs any event changes into logs table. First we create the table

CREATE TABLE logs (
event_id integer,
old_title varchar(255),
old_starts timestamp,
old_ends timestamp,
logged_at timestamp DEFAULT current_timestamp
);

Next, we build a function to insert old data into the log. The OLD variable represents the row about to be changed (NEW represents an incoming row, which we’ll see in action soon enough). Output a notice to the console with the event_id before returning. Then we create our trigger to log changes after any row is updated through this function.

CREATE OR REPLACE FUNCTION log_event() RETURNS trigger AS $$
DECLARE
BEGIN
INSERT INTO logs (event_id, old_title, old_starts, old_ends)
VALUES (OLD.event_id, OLD.title, OLD.starts, OLD.ends);
RAISE NOTICE 'Someone just changed event #%', OLD.event_id;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER log_events
AFTER UPDATE ON events
FOR EACH ROW EXECUTE PROCEDURE log_event();

To test our trigger let's update an event and see the trigger output:

and the old event was logged

Views

Views are aliased queries that can be queried by its alias like any other table. Creating a view is as simple as writing a query and prefixing it with CREATE VIEW some_view_name AS. Views are good for opening up complex queried data in a simple way. If you want to add a new column to the view, it will have to come from the underlying table.

CREATE VIEW holidays AS
SELECT event_id AS holiday_id, title AS name, starts AS date
FROM events
WHERE title LIKE '%Day%' AND venue_id IS NULL;

SELECT name, to_char(date, 'Month DD, YYYY') AS date
FROM holidays
WHERE date <= '2012-04-01';

The RULE system

The rule system (more precisely speaking, the query rewrite rule system) is totally different from stored procedures and triggers. It modifies queries to take rules into consideration, and then passes the modified query to the query planner for planning and execution. More information can be found here. Example of changing query to allow updating our holidays view:

CREATE RULE update_holidays AS ON UPDATE TO holidays DO INSTEAD
UPDATE events
SET title = NEW.name,
starts = NEW.date
WHERE title = OLD.name;

UPDATE holidays SET date = '6/25/2013' where name = 'Christmas Day';

Crosstab

crosstab(text source_sql, text category_sql) takes two queries to pivot results on one of them on the results of the other.

source_sql is a SQL statement that produces the source set of data. This statement must return one row_name column, one category column, and one value column. It may also have one or more "extra" columns. The row_name column must be first. The category and value columns must be the last two columns, in that order.
category_sql is a SQL statement that produces the set of categories. This statement must return only one column. It must produce at least one row, or an error will be generated. Also, it must not produce duplicate values, or an error will be generated.

If this is the first time you use something from the tablefunc module, you may need to install it to your database by running CREATE EXTENSION tablefunc;

create table sales(year int, month int, qty int);
insert into sales values(2007, 1, 1000);
insert into sales values(2007, 2, 1500);
insert into sales values(2007, 7, 500);
insert into sales values(2007, 11, 1500);
insert into sales values(2007, 12, 2000);
insert into sales values(2008, 1, 1000);

select * from crosstab(
  'select year, month, qty from sales order by 1',
  'select m from generate_series(1,12) m'
) as (
  year int,
  "Jan" int,
  "Feb" int,
  "Mar" int,
  "Apr" int,
  "May" int,
  "Jun" int,
  "Jul" int,
  "Aug" int,
  "Sep" int,
  "Oct" int,
  "Nov" int,
  "Dec" int
);

Full-Text Search

Full-text search enables the user to search text columns with parts of the column value, not the exact value like in regular queries.

LIKE and ILIKE : LIKE and ILIKE (case-insensitive LIKE) are the simplest forms of text search. LIKE compares column values against a given pattern string. The % and _ characters are wildcards. % matches any number of any characters, and _ matches exactly one character.
- SELECT title FROM movies WHERE title ILIKE 'stardust%';
  - will return movies with titles like : “Stardust” or “Stardust Memories”
Regex : You could write the WHERE part of your queries against text columns in regular expressions. A regular expression match is led by the ~ operator, with the
optional ! (meaning, not matching) and * (meaning case insensitive). So, to count all movies that do not begin with the, the following case-insensitive query will work. The characters inside the string are the regular expression.
- SELECT COUNT(*) FROM movies WHERE title !~* '^the.*';
Bride of Levenshtein : Levenshtein is a string comparison algorithm that compares how similar two strings are by how many steps are required to change one string into another. Each replaced, missing, or added character counts as a step (Changes in case cost a point too). The distance is the total number of steps away. levenshtein() function is provided by the fuzzystrmatch contrib package.The following query return 3 because we need to replace 2 characters and and add 1 character to make bat into fads.
- SELECT levenshtein('bat', 'fads');
Trigram : A trigram is a group of three consecutive characters taken from a string. The pg_trgm contrib module breaks a string into as many trigrams as it can. Finding a matching string is as simple as counting the number of matching trigrams. The strings with the most matches are the most similar. It’s useful for doing a search where you’re OK with either slight misspellings or even minor words missing. The longer the string, the more trigrams and the more likely a match.
- SELECT show_trgm('Avatar'); -- returns {" a"," av","ar ",ata,ava,tar,vat}
PostgreSQL have more natural language processing capabilities (TSVector and TSQuery) and even can search by word phonetics (metaphone)
You could combine text search function in many interesting ways like the following query that “Get me names that sound the most like Robi n Williams, in order.”
- SELECT * FROM actors
  WHERE metaphone(name,8) % metaphone('Robin Williams',8)
  ORDER BY levenshtein(lower('Robin Williams'), lower(name));

In this post we introduced PostgreSQL, a popular open source RDBMS. In future posts we will introduce more DBMSs.

Parameterized Test Patterns using Microsoft Pex

noreply@blogger.com (Ebeid Soliman ElSayed) — Thu, 02 May 2013 21:08:00 +0000

We talked before about the difference between unit tests and parameterized unit tests. In this post we will talk about common patterns for writing good parameterized unit tests. Keep in mind that we will use these tests with Microsoft Pex (as an automatic test input generation tool) to get test inputs that trigger all the possible scenarios of the code-under-test.

Before anything, let’s clarify what are the questions we want to answer using the parameterized unit tests. There are two core questions:

What are good scenarios (sequences of method calls) to exercise the code-under-test? (Coverage)
What are good assertions that can be stated easily without re-implementing the algorithm? (Verification)

A parameterized unit test is only useful if it provides answers for both questions:

Without sufficient coverage, i.e. if the scenario is too narrow to reach all the code-under-test, the extent of the test is limited.
Without sufficient verification of the computed results, i.e. if the test does not contain enough assertions, the test does not check that the code is doing the right thing. All it would check for is that the code-under-test does not crash.

Test Patterns

1- Arrange, Act, Assert

The ’AAA’ (Triple-A) is a well-known pattern for writing unit tests. It applies to parameterized unit tests as well. A parameterized unit test using this patter is organized in three sections:

Arrange: set up the unit under test
Act: exercise the unit under test, capturing any resulting state
Assert: verify the behavior through assertions

An example of this pattern in a traditional unit test:


[Test]
pubic void AddItem()
{
      // arrange
      var list = new ArrayList();
      var item = new object();
      // act
      list.Add(item);
      // assert
      Assert.IsTrue(list.Count == 1);
}

An example of this pattern in a parameterized unit test:


[PexMethod]
pubic void AddItem(object item)
{
      // arrange
      var list = new ArrayList();
      // act
      list.Add(item);
      // assert
      Assert.IsTrue(list.Count == 1);
}

2- Assume, Arrange, Act, Assert

This pattern is an extension of the first pattern where an Assumption section is added at the beginning. An assumption restricts possible test inputs, acting as a filter. A parameterized unit test using this pattern is organized in four sections:

Assume: assume preconditions over the test inputs

Arrange: set up the unit under test

Act: exercise the unit under test, capturing any resulting state

Assert: verify the behavior through assertions

The following example tests that adding an element to any list increments the Count property. We use an assumption to filter out the case where list is a null reference.


[PexMethod]
void AssumeActAssert(ArrayList list, object item)
{
     // assume
     PexAssume.IsNotNull(list);
     // arrange
     var count = list.Count;
     // act
     list.Add(item);
     // assert
     Assert.IsTrue(list.Count == count + 1);
}

3- Parameterized Stubs

If the code-under-test already contains many assertion statements that verify its behavior, an effective parameterized unit test might be quite simple in itself, because it can leverage the assertions in the code.

[PexMethod] 
public void Add( [PexAssumeUnderTest]ArrayList list, object item) 
{ 
 list.Add(item);

The attribute PexAssumeUnderTest is a short-hand notation to make sure that the parameter is not null, and has exactly the type indicated by its declaration (not a subtype).

4- Observed Values

At any point in the parameterized unit test, if you want to log the value of any variable or parameter, just use

PexObserve.ValueForViewing("Variable Name", var_name);

5- Allowed Exceptions

Traditional unit test frameworks support the concept of expected exception, where a test case or API call is expected to throw an exception. If the test does not throw the exception or throws an exception that does not match the criteria, the execution fails. The same concept applied to parameterized unit tests.

[PexMethod][PexAllowedException(typeof(ArgumentNullException))] 
void Constructor(string value) 
{ 
 // throws ArgumentNullException if value is null 
 var myClass = new MyClass(value); 
}

6- State Relation

This pattern applies when an API call causes an internal state change that can be (partially) observed through other API calls. A classic example of such a pattern is the combination of Insert and Contains operation on any collection type:


[PexMethod] 
void InsertContains(stringvalue) 
{ 
 var list = new List(); 
 list.Add(value); 
 Assert.IsTrue(list.Contains(value)); 
}

7- Roundtrip

This pattern applies to an API that transforms its inputs in a reversible way: When the API has a function f and an inverse function f_1, then it should hold that f_1(f(x))=x for all x. A classic example of such pattern is property setters and getters, where the test fails when the setter rejects a particular argument value.


[PexMethod] 
void PropertyRoundtrip(string value) 
{ 
 // arrange 
 var target = new MyClass(); 
 
 // two-way roundtrip 
 target.Name = value;   // calls setter 
 var roundtripped = target.Name; // calls getter 

 // assert 
 Assert.AreEqual(value, roundtripped); 
}

Another example is serialization and deserialization of values.


[PexMethod] 
void ToStringParseRoundtrip(int value) 
{ 
 // two-way roundtrip 
 string s = value.ToString(); 
 int parsed = int.Parse(s); 

 // assert Assert.AreEqual(value, parsed); 
}

8- Normalized Roundtrip

The Pattern Roundtrip showed how to test a method for which in inverse operation exists. For example, int.Parse is the inverse of int.ToString. This is not the case in the other direction: int.ToString is not exactly the inverse of int.Parse, because the parsing ignores whitespace (some kind of data normalization):


int.Parse(" 5").ToString() == "5";

This pattern is applied when the API has a function f and an inverse function f_1, then it should hold that f_1(f(f_1(x)))=f_1(x) for all x where f_1(x) is defined.


[PexMethod] 
void ThreeWayRoundtrip(string value) 
{ 
 // ’hello%20world’ <= ’hello world’ 
 var normalized = Uri.EscapeDataString(value); 
 
 // ’helloworld’ <= ’hello%20world’ 
 var intermediate = Uri.UnescapeDataString(normalized); 

 // ’hello%20world’ <= ’hello world’ 
 var roundtripped = Uri.EscapeDataString(intermediate); 

 // assert 
 Assert.AreEqual(normalized, roundtripped); 
}

9- Reachability

If your test assumptions are so complicated and it is not clear whether there is any test input that fulfills them. You could use PexAssert.ReachEventually("GoalName") wherever in your test to make sure that there is at least one input that reach this point in your test. The goal name have to be passed to the PexAssertReachEventuallyAttribute constructor. A parameterized unit test fails when Pex does not find a way to reach a goal, indicated by calling the PexAssert.Reached method in a parameterized unit test annotated with the PexAssertReachEventuallyAttribute.


[PexMethod] 
[PexAssertReachEventually("passed", StopWhenAllReached = true)] 
public void ParsingSuccesful(string input) 
{ 
 // complicated parsing code 
 DateTime date; 
 if (DateTime.TryParse(input, out date)) 
 { 
  // and we want to see at least one case where parsing is successful. 
  PexAssert.ReachEventually("passed"); 
 } 
}

Multiple goals can be combined in a single parameterized unit test for more advanced scenarios by passing a list of goal identifiers in the constructor of the PexAssertReachEventuallyAttribute. Each goal identifier must be reached and notified in order for the parameterized unit test to succeed.


[PexMethod] 
[PexAssertReachEventually("parsed", "y2k", StopWhenAllReached = true)] 
public void ParsingSuccesfulWithMoreGoals(string input) 
{ 
 // complicated parsing code 
 DateTime date;
 if (DateTime.TryParse(input, out date)) 
 { 
  // and we want to see at least one case where parsing is successful. 
  PexAssert.ReachEventually("parsed"); 
 } 
 if (date.Year == 2000) 
 {
  // we want to see at least one test with the year 2000 
  PexAssert.ReachEventually("y2k"); 
 }
}

10- Reachable Implication

Implications assert a property when a predicate (condition) is true. Use PexAssert.ImpliesEventually to make an implication that should be true and that should be executed at least once. The test have to be annotated with the PexAssertReachEventuallyAttribute. A parameterized unit test fails when Pex does not find a way to make the predicate of PexAssert.ImpliesEventually evaluates true at least once.


[PexMethod] 
[PexAssertReachEventually] 
public void ParseImpliesParse(string input) 
{ 
 bool value; 
 // if TryParse succeeds, Parse should succeed too. Also make sure TryParse succeeds at least once 
 PexAssert.ImpliesEventually( bool.TryParse(input, out value), () => bool.Parse(input) );
}

11- Seed Values to Help Pex

A parameterized unit test needs concrete input data to be executed. Pex's role is to automatically generate relevant input data via code analysis. Sometimes it might be desirable or even necessary to provide manually chosen seed values to Pex to guide the automated code exploration. In effect, Pex will fuzz the provided values in ways that cause alternative execution paths to be taken. The PexArgumentsAttribute can be used to provide primitive data. Each instance of this attribute gives a list of values that must match the parameter types of the parameterized unit test (and in the same parameters order).


[PexMethod]
[PexArguments("var i = 0;", 0)]
[PexArguments("class Foo {}", 12)]
public void ParseTest(string text, int line)
{
 var parser = new Parser();
 parser.SetLine(line);
 var node = parser.Parse(text);

}

Before analyzing the branch conditions in the code, Pex will first execute the parameterized unit test with the provided values. This way, Pex acquires knowledge about the code reachable from a test, and during the subsequence code exploration Pex will try to further increase code coverage by slightly modifying the values to trigger different execution paths. In effect, Pex will fuzz the provided values.

12- Regression Tests

In these tests some we could persist a computed value in the generated test so when the generated test is executed in the future, it verifies that the (possible changed) code-under-test still computes the same value. There are several ways how outputs can be logged. For a single output value, one can use the return value of the parameterized unit test:


[PexMethod]
public int Add(int a, int b)
{
 return a + b;
}

Pex will recursively traverse the observable properties and fields of the value and add assertions in the generated test for each one of them:


[Test]
[PexGeneratedBy(typeof(Program))]
public void Add866()
{
    int i;
    i = this.Add(0, 0);
    PexAssert.AreEqual(0, i);
}

For multiple values, use out parameters in the parameterized unit tests


[PexMethod]
public void Add(int a, int b, out int result)
{
 result = a + b;
}

Which Pex will use to generate the following


[Test]
[PexGeneratedBy(typeof(Program))]
public void Add13()
{
    int i = 0;
    this.Add(0, 0, out i);
    PexAssert.AreEqual(0, i);
}

If the number of values might be dynamic, you could log these values using PexObserve.Value and observe it later using the PexObserve.ValueAtEndOfTest method:


[PexMethod]
void Add(int a, int b)
{
 int result = a * b;
 PexObserve.Value("result", result);
}

Microsoft Pex: Understanding Assumptions, Assertions, and Test-Case Failures

noreply@blogger.com (Ebeid Soliman ElSayed) — Mon, 29 Apr 2013 20:00:00 +0000

In a previous post we started using Microsoft Pex and showed how it helped exploring all possible code paths, and how that helped discovering a defect in our program logic. In that example, even our program logic is defective, all test cases succeeded. A test case fails if there is an un-caught exception or a failed assertion.

To see an example of a failed test case and how Pex could help in fixing it, let’s add the following basic function to our code:

public void Add(ArrayList a, object o)
        {
            a.Add(o);
        }

when you run Pex for that simple method, you get the following results:

As you might expected, the ArrayList object might be NULL. In situations similar to this one, when a higher-level component (the code that called Add() method) passes malformed data to a lower-level component(that Add() method), which the lower-level component rejects, then the higher-level component should be prevented from doing so in the first place.

One way to achieve this is to promote the failure condition of the lower-level component to the abstraction level of the higher-level component. This is what the Add Precondition feature of Pex does: It expresses a low-level failure condition at the abstraction level of the code under test. When added to the code-under-test as argument validation code—also called a precondition—then this effectively prevents the code under test from causing a failure somewhere in its execution. In other words, the Add Precondition feature doesn’t completely fix an error, but it promotes the failure condition to a higher, more appropriate abstraction level.

To add a precondition to your code, click the failed test case row, then on the details section (right) click Add Precondition, as in picture below

Pex will open Preview and Apply updates dialog box to show you the proposed code modifications that Pex will do.

Review and Click Apply. The modified method will look like:

        
public void Add(ArrayList a, object o)
        {
            // 
            Debug.Assert(a != (ArrayList)null, "a");
            // 
            a.Add(o);
        }

When you run Pex for our method now, we will not find a failed test case.

The concept of assertions is well known in unit test frameworks. Pex understands the built-in Assert classes provided by each supported test framework. However, most frameworks do not provide a corresponding Assume class as Pex did. PexAssume is a class to express assumptions, i.e. a precondition. The methods of this class can be used to filter out undesirable test inputs. If you do not want to use an existing test framework, Pex also has the PexAssert class. Some functionalities can be achieved using attributes instead of PexAssume methods, like PexAssumeNotNullAttribute and PexAssumeUnderTestAttribute.

 
[PexMethod]
public void Test1(object o)  //precondition: o should not be null
{
 PexAssume.IsNotNull(o);
 ...
}

[PexMethod]
public void Test2([PexAssumeNotNull]object o) //precondition: o should not be null
{ 
 ...
}

When you write an assertion, Pex will not only passively detect assertion violations, but Pex will in fact actively try to compute test inputs that will cause the assertion to fail (just as an assertion might throw a PexAssertFailedException). Pes uses these exceptions internally to stop a test case when an assumption fails.

Expected Exceptions

You can annotate the test—or the test class, or the test assembly—with one of the following attributes to indicate which exception types can or must be thrown by the test in order to be considered successful:

PexAllowedExceptionAttribute indicates that a test method, or any other method that it calls directly or indirectly, can throw a particular type of exception for some test inputs.

using System;
using Microsoft.Pex.Framework;
using Microsoft.Pex.Framework.Validation;

namespace ConsoleApplication3
{
    class Stack
    {
        int[] _elements;
        int _count;
        public Stack(int capacity)
        {
            if (capacity < 0) throw new ArgumentOutOfRangeException();
            _elements = new int[capacity];
            _count = 0;
        }
    }

    [PexClass]
    public partial class StackTest
    {
        [PexMethod]
        [PexAllowedException(typeof(ArgumentOutOfRangeException))]
        public void CtorTest(int capacity) // will not fail
        {
            Stack s = new Stack(capacity); // may throw ArgumentOutOfRangeException
        }
    }
}

PexAllowedExceptionFromTypeAttribute indicates that any method of a specified type can throw a particular type of exception for some test inputs.

using System;
using System.Collections;
using Microsoft.Pex.Framework;
using Microsoft.Pex.Framework.Validation;

namespace ConsoleApplication3
{
    [PexClass]
    public partial class ArrayListTest
    {
        [PexMethod]
        [PexAllowedExceptionFromType(typeof(NullReferenceException), typeof(ArrayListTest))]
        public void Add1(ArrayList a, object o) //will not fail
        {
            a.Add(o);
        }
    }
}

PexAllowedExceptionFromTypeUnderTestAttribute indicates that any method of the designated type under test can throw a particular type of exception for some test inputs.
PexAllowedExceptionFromAssemblyAttribute indicates that any method located in a specified assembly can throw a particular type of exception for some test inputs.

When Does Pex Emit a Test Case?

Pex supports different filters that decide when generated test inputs will be emitted as a test case. You can configure these filters with the TestEmissionFilter property that you can set for example in the PexMethod attribute. Possible values are the following:

All Emit every generated test input as a test case, including those that cause assumption violations.
FailuresAndIncreasedBranchHits (default) Emit tests for all unique failures, and whenever a test case increases coverage, as controlled by the TestEmissionBranchHits property (take values 1 or 2).
FailuresAndUniquePaths Emit tests for all failures Pex finds, and also for each test input that causes a unique execution path.
Failures Emit tests for failures only.

Regarding increased branch coverage, the TestEmissionBranchHits property controls how a branch is covered. For example:

TestEmissionBranchHits=1: Whether Pex should just consider whether a branch was covered at all. This gives a very small test suite that covers all branches Pex could reach. In particular, this test suite also covers all reached basic blocks and statements.
TestEmissionBranchHits=2: Whether a test covered it either once or twice.

The default is TestEmissionBranchHits=2, which generates a more expressive test suite that is also better suited to detect future regression errors.

using System;
using Microsoft.Pex.Framework;
using Microsoft.Pex.Framework.Settings;

namespace ConsoleApplication3
{
    [PexClass]
    partial class TestEmission
    {
        int max(int x, int y)
        {
            if (x > y)
                return x;
            else
                return y;
        }

        [PexMethod(TestEmissionFilter=Microsoft.Pex.Framework.Settings.PexTestEmissionFilter.All)]
        public void MaxTest1(int a, int b, int c, int d)    // 1 test case generated
        {
            int e = max(a, b);
            PexObserve.ValueForViewing("max", e);
        }

        [PexMethod(TestEmissionFilter = Microsoft.Pex.Framework.Settings.PexTestEmissionFilter.Failures)]
        public void MaxTest2(int a, int b, int c, int d)    // No test cases generated 
        {
            int e = max(a, b);
            PexObserve.ValueForViewing("max", e);
        }
    }
}

When Does Pex Stop ?

If the code under test does not contain loops or unbounded recursion, Pex will typically stop quickly because there is only a (small) finite number of execution paths to analyze. However, most interesting programs contain loops and/or unbounded recursion. In such cases the number of execution paths is (practically) infinite, and it is in general undecidable whether a statement is reachable. In other words, Pex would take forever to analyze all execution paths of the program.

In order to make sure that Pex terminates after a reasonable amount of time, there are several exploration bounds. All bounds have predefined default values, which you can override to let Pex analyze more and longer execution paths. The bounds are parameters of the PexMethod, PexClass, and PexAssemblySettings attributes. There are different kinds of bounds:

Constraint Solving Bounds: apply to each attempt of Pex to determine whether an execution path is feasible or not. Pex might need several constraint solving attempts to compute the next test inputs.
- ConstraintSolverTimeOut Seconds the constraint solver has to figure out inputs that will cause a different execution path to be taken.
- ConstraintSolverMemoryLimit Megabytes the constraint solver can use to figure out inputs.
Exploration Path Bounds: apply to each execution path that Pex executes and monitors. These bounds make sure that the program does not get stuck in an infinite loop, or recursive method.
- MaxBranches Maximum number of branches that can be taken along a single execution path.
- MaxCalls Maximum number of calls that can be taken during a single execution path.
- MaxStack Maximum size of the stack at any time during a single execution path, measured in number of active call frames.
- MaxConditions Maximum number of conditions over the inputs that can be checked during a single execution path.
Exploration Bounds apply to the exploration of each parameterized unit test.
- MaxRuns Maximum number of runs that will be tried during an exploration (each run uses different test inputs; not every run will result in the emission of a new test case).
- MaxRunsWithoutNewTests Maximum number of consecutive runs without a new test being emitted.
- MaxRunsWithUniquePaths Maximum number of runs with unique execution paths that will be tried during an exploration.
- MaxExceptions Maximum number of exceptions that can be found over all discovered execution paths combined.
- MaxExecutionTreeNodes Maximum number of conditions over the inputs that can be checked during all discovered execution paths combined.
- MaxWorkingSet Maximum size of working set in megabytes.
- TimeOut Seconds after which exploration stops.

The following example shows a parameterized test that involves a loop. The loop bound depends on the test inputs, and the exception can only be triggered if the loop is executed a certain number of times. Here, we used an explicit bound of 10 runs MaxRuns=10 to let Pex finish quickly. However, with this bound, Pex will most likely not be able to trigger the exception, as Pex will not unroll the loop sufficiently many times:

 [PexMethod(MaxRuns = 10)]
        public void TestWithLoop(int n)
        {
            var sum = 0;
            for (int i = 0; i < n; i++)
                sum++;
            if (sum > 20) throw new Exception();
        }

In the Pex Exploration Results you may see the Exception statement reached, but you will see on the Pex tool bar that there is 1 boundry reached. If you click on it, you will see more details:

on the right, you could click Set MaxRuns=20 to increase the MaxRuns boundry. You could do the same from the Pex tool bar and also setting boundary to infinity. Both actions will open a dialog to review and approve the code changes.

The following example shows another parameterized test that involves a loop, but this loop does not depend on the test inputs. Here, we used an explicit bound of 10 branches MaxBranches=10 to let Pex finish quickly. However, with this bound, Pex cannot even once execute the code from beginning to end, as executing the embedded loop will cause more than 10 branches to be executed.

 [PexMethod(MaxBranches = 10)]
        public void TestWithFixedLoop(int j)
        {
            var sum = 0;
            for (int i = 0; i < 15; i++)
                sum++;
            if (j == 10) throw new Exception();
        }

In those cases, where a particular run exceeded some path-specific bounds, we get a special row in the results. The Set MaxBranches= button on the results tool can be used to increase the bounds. If you increased the boundary to 20, you will get all you branched executed and the exception statement reached.

In this post we talked about how and when a test case fail, how to use precondition and assumptions to differentiate between failures from your unit code and failures from malformed input, how to allow specific exception to be raised from your tests without causing it to fail, how to control which test inputs should Pex use as a test case, and finally how to control the Pex stop criteria whether through bounding the constraint solver, or bounding the exploration paths, or bounding the exploration runs.

Getting started with Microsoft Code Digger

noreply@blogger.com (Ebeid Soliman ElSayed) — Fri, 26 Apr 2013 08:00:00 +0000

Microsoft Code Digger is Visual Studio 2012 extension that have been released few days ago by RiSE team at Microsoft Research (the same team who developed Pex). You can download the it from the Visual Studio Gallery here.

Microsoft Code Digger uses the same engine that Pex uses, and the same techniques under the hood (dynamic symbolic execution and constraint solvers). The only constrain that Code Digger have is that it only works on public .NET code in Portable Class Libraries.

Let’s try it

After installing the Code Digger extension for Visual Studio 2012, create a Portable Class Library. Let’s use the Triang() method we used in previous posts as an example here. Your code should look like:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace PortableClassLibrary1
{
    public class Class1
    {
        public static int Triang(int Side1, int Side2, int Side3)
        {
            int triOut;

            // triOut is output from the routine:
            //   Triang = 1 if triangle is scalene
            //   Triang = 2 if triangle is isosceles
            //   Triang = 3 if triangle is equilateral
            //   Triang = 4 if not a triangle

            // A quick confirmation that it's a valid triangle
            if (Side1 <= 0 || Side2 <= 0 || Side3 <= 0)
            {
                triOut = 4;
                return (triOut);
            }

            // Detect any sides of equal sides
            triOut = 0;
            if (Side1 == Side2)
                triOut = triOut + 1;
            if (Side1 == Side3)
                triOut = triOut + 2;
            if (Side2 == Side3)
                triOut = triOut + 3;
            if (triOut == 0)
            {
                if ((Side1 + Side2 <= Side3) || (Side2 + Side3 <= Side1) || (Side1 + Side3 <= Side2)) // confirm that it is a valid triangle 
                    triOut = 4;
                else
                    triOut = 1;
                return (triOut);
            }

            if (triOut > 3)
                triOut = 3;
            else if ((triOut == 1) && (Side1 + Side2 > Side3))
                triOut = 2;
            else if ((triOut == 2) && (Side1 + Side3 > Side2))
                triOut = 2;
            else if ((triOut == 3) && (Side2 + Side3 > Side1))
                triOut = 2;
            else
                triOut = 4;
            return (triOut);
        }
    }
}

Right click inside Triang() method and wait for few seconds. You will see the below Inputs / Outputs pane.

It’s the same set we got before from Pex and for the same reasons. No surprises, both tools use the same engine under hood.

Why Pex Choose These Inputs

noreply@blogger.com (Ebeid Soliman ElSayed) — Thu, 25 Apr 2013 08:00:00 +0000

In the example we gave in the previous post, it may seem that Pex chose random numbers as inputs for the Triang() method but it is not. But also its not all possible values for the inputs.

Actually, Pex generates test inputs by analyzing your program code, so it is called whitebox test generation (as opposed to blackbox test generation). For every statement in the code, Pex will eventually try to create a test input that will reach that statement. Pex will do a case analysis for every conditional branch in the code—for example, if statements, assertions, and all operations that can throw exceptions.

In other words, the number of test inputs that Pex generates depends on the number and possible combinations of conditional branches in the code (if interested to know more about that, search for symbolic execution). Pex operates in a feedback loop: it executes the code multiple times and learns about the program behavior by monitoring the control and data flow.

After each run, Pex does the following:

Chooses a branch that was not covered previously.
Builds a constraint system that describes how to reach that branch.
Uses a constraint solver to determine new test inputs that fulfill the constraints, if any exist.

The test is executed again with the new inputs, and the process repeats. On each run, Pex might discover new code and dig deeper into the implementation. In this way, Pex explores the behavior of the code.

Because our code doesn’t have any conditions that test zero length sides, Pex generated zero an input and also shows that our program is defective (because it considers a triangle with (0,0,0) as equilateral and (1,0,1) as isosceles). If we added the following lines of code after declaring triOut and before doing anything.

            
// A quick confirmation that it's a valid triangle            
if (Side1 <= 0 || Side2 <= 0 || Side3 <= 0)      
{
 triOut = 4;
 return (triOut);
}

You will get a different set of test inputs from Pex that reveal more code path combinations.

We mentioned before that pex generates test input by performing a symoblic analysis of the code under test. You can use the method GetPathConditionString of the PexSymbolicValue class to obtain a textual representation of the current path condition, a predicate (condition) that characterizes an execution path. The ToString method of PexSymbolicValue class gives you a textual representation of how a value was derived from the test input provided. To do so, add reference to Pex Framework dll (located at "C:\Program Files\Microsoft Moles\PublicAssemblies\Microsoft.Pex.Framework.dll"). Add using Microsoft.Pex.Framework; to your code then add the following code anywhere in your code to get details about the code path at that point.


 PexObserve.ValueForViewing("Condition", PexSymbolicValue.GetPathConditionString());
 PexObserve.ValueForViewing("Return Value", PexSymbolicValue.ToString(Triang(1, 1, 1)) );

Here I added it into the Triang() method we used before. Run Pex and you will see the new columns added to the results populated with conditions that led to .

ToRawString method and GetRawPathConditionString method return expressions representing symbolic values and the path condition, formatted as S-expressions.

Method PexObserve.ValueForViewing can also be used to display the value picked by Pex for any variable at any point in your code.

The same engine Pex uses is now available as part of Code Digger, a Visual Studio 2012 extension. We will talk about it in a future post.

Getting started with Microsoft Pex

noreply@blogger.com (Ebeid Soliman ElSayed) — Wed, 24 Apr 2013 08:06:00 +0000

Microsoft Pex is a white box test generation for .NET that came out of Microsoft Research and have been successfully integrated into Visual Studio 2010. It have been a result of collaborative work between Microsoft Research and the Automated Software Engineering Research Group at North Carolina State University led by Dr. Tao Xie.

You can download and install Microsoft Pex for Visual Studio 2010 from here. We have talked in a previous post about parameterized unit tests and the possibilities it brings. In this post and the following we will explore Microsoft Pex and how it can help you in understanding the input/output behavior of your code, finding inputs that cause the code-under-test to crash, and exploring parameterized unit tests to check whether your code implements the desired functionality for all inputs.

Running Pex for the First Time

In Visual Studio click File > New > Project. In the left pane of the New Project dialog box, click Visual C#. In the center pane, click Console Application, pick a Name for it and click OK. Replace your class Program with the following class. It just the classic triangle classification program :

class Program
    {
        public static String[] triTypes = { "", // Ignore 0.
                                               "scalene", "isoscelese", "equilateral", "not a valid triangle"};
        public static String instructions = "This is the ancient TriType program.\nEnter three integers that represent the lengths of the sides of a triangle.\nThe triangle will be categorized as either scalene, isosceles, equilateral\n or invalid.\n";

        public static void Main()
        {
            int A, B, C;
            int T;

            Console.WriteLine(instructions);
            Console.WriteLine("Enter side 1: ");
            A = getN();
            Console.WriteLine("Enter side 2: ");
            B = getN();
            Console.WriteLine("Enter side 3: ");
            C = getN();
            T = Triang(A, B, C);

            Console.WriteLine("Result is: " + triTypes[T]);
            Console.ReadLine();
        }

        public static int Triang(int Side1, int Side2, int Side3)
        {
            int triOut;

            // triOut is output from the routine:
            //   Triang = 1 if triangle is scalene
            //   Triang = 2 if triangle is isosceles
            //   Triang = 3 if triangle is equilateral
            //   Triang = 4 if not a triangle

            // A quick confirmation that it's a valid triangle
            if (Side1 <= 0 || Side2 <= 0 || Side3 <= 0)
            {
                triOut = 4;
                return (triOut);
            }

            // Detect any sides of equal sides
            triOut = 0;
            if (Side1 == Side2)
                triOut = triOut + 1;
            if (Side1 == Side3)
                triOut = triOut + 2;
            if (Side2 == Side3)
                triOut = triOut + 3;
            if (triOut == 0)
            {
                if ((Side1 + Side2 <= Side3) || (Side2 + Side3 <= Side1) || (Side1 + Side3 <= Side2)) // confirm that it is a valid triangle 
                    triOut = 4;
                else
                    triOut = 1;
                return (triOut);
            }

            if (triOut > 3)
                triOut = 3;
            else if ((triOut == 1) && (Side1 + Side2 > Side3))
                triOut = 2;
            else if ((triOut == 2) && (Side1 + Side3 > Side2))
                triOut = 2;
            else if ((triOut == 3) && (Side2 + Side3 > Side1))
                triOut = 2;
            else
                triOut = 4;
            return (triOut);
        }

In the Build menu, click Build Solution.

To run Pex on your code, right-click in the body of the Triang method, and click Run Pex. If this is your first time running Pex, the Pex: Select a Test Framework dialog box appears. You could select your preferred test frame (Visual Studio Unit test, or NUnit) and provide the installation path for its, then click OK. This dialog box will not appear again after you select the test framework. After a brief pause, Pex shows the results of its analysis in the Pex Exploration Results window. When you run Microsoft Pex on your .NET code, Pex generates test cases by analyzing the code-under-test. For every statement in the code, Pex will eventually try to create a test input that will reach that statement. Pex will do a case analysis for every conditional branch in the code—for example, if statements, assertions, and all operations that can throw exceptions. Each row in the table contains input/output values for the method under consideration(Traing). In the Pex Exploration Results window, click one row in the table on the left to see details in the right pane. You could select these rows and save them as unit tests. These details also displayed on the right in the Pex Explorer pane as test cases.

On the next post we will explain why Pex chose these values, and other Pex stuff :)

Unit Tests vs Parametrized Unit Tests

noreply@blogger.com (Ebeid Soliman ElSayed) — Mon, 22 Apr 2013 22:17:00 +0000

Unit Tests

Using the conventions of NUnit unit tests as test methods contained in test classes. A parameterless method decorated with a custom attribute like [TestMethod] is a test method. Usually, each unit test explores a particular aspect of the behavior of the class-under-test.

Here is a unit test written in C# that adds an element to a .NET ArrayList instance. The test first creates a new array list, where the parameter to the constructor is the initial capacity, then adds a new object to the array list, and finally checks that the addition was correctly performed by verifying that a subsequent index lookup operation returns the new object.

[TestMethod] 
void TestAdd() 
{ 
 ArrayList a = new ArrayList(0); 
 object o = new object(); 
 a.Add(o); 
 Assert.IsTrue(a[0] == o); 
}

It is important to note that unit tests include a test oracle that compares observed behavior with expected results. By convention, the test oracle of a unit test is encoded using assertions. The test fails if any assertion fails or an exception is thrown. Unit test frame- works can also deal with expected exceptions. For a quick introduction about NUnit refer previous posts.

A test suite produced by unit testing with high code coverage gives confidence in the correctness of the tested code. However, writing unit tests to achieve high coverage can be time-consuming and tedious given that test execution frameworks only automate the test executions. You still have to write your test cases !

To address this problem, several automatic unit test generation tools such as Parasoft JTest or jCUTE can automatically generate conventional unit tests. These tools, nevertheless, cannot guarantee high code coverage, unless testers manually write some tests.

Parametrized Unit Tests

The unit test above specifies the behavior of the array list by example. Strictly speaking, this unit test only says that by adding a new object to an empty array list, this object becomes the first element of the list. What about other array lists and other objects?

[TestAxiom]
void TestAdd(ArrayList a, object o) 
{
 Assume.IsTrue(a!=null);
 int i = a.Count;
 a.Add(o);
 Assert.IsTrue(a[i] == o);
}

By adding parameters we can turn a closed unit test into a universally quantified conditional axiom that must hold for all inputs under specified assumptions.

Adding parameters to a unit test improves its expressiveness as a specification of intended behavior, but we lose concrete test cases. We can no longer execute this test axiom by itself. We need actual parameters. But which values must be provided to ensure sufficient and comprehensive testing? Which values can be chosen at all?

In the ArrayList example, if we study the internal structure of the .NET Framework implementation, we observe that there are two cases of interest. One occurs when adding an element to an array list that already has enough room for the new element (i.e. the array list’s capacity is greater than the current number of elements in the array list). The other occurs when the internal capacity of the array list must be increased before adding the element.

If we assume that library methods invoked by the ArrayList implementation are themselves correctly implemented, we can deduce that running exactly two test cases is sufficient to guarantee that the parametrized TestAdd(...) succeeds for all array lists and all objects.

[TestMethod]
void TestAddNoOverflow() 
{
 TestAdd(new ArrayList(1), new object());
}

[TestMethod]
void TestAddWithOverflow() 
{
 TestAdd(new ArrayList(0), new object());
}

Splitting axioms and test cases in this way is a separation of concerns:

First, we describe expected behavior as parametrized unit tests.
Then we study the case distinctions made by the code paths of the program under test to determine which inputs make sense for testing.

So, parametrized unit tests are more general specifications than conventional unit tests because it specifies the behavior for the whole input classes other than for a single concrete value of the input. But it needs concrete parameter values to be executed.

The good news is, Given a parametrized unit test, a test-generation tool, such as Microsoft Pex, can automatically generate tests with concrete inputs for the parameters to achieve high coverage. Pex explores the behaviors of a parametrized unit test using a technique called dynamic symbolic execution. Dynamic Symbolic Execution (DSE) is a variation of symbolic execution, which systematically explores feasible paths of the program under test by running the program with different test inputs to achieve high structural coverage. It collects the symbolic constraints on inputs obtained from predicates (condition statements like if else, switch,…) in branch statements along the execution and relies on a constraint solver [like Zap or Simplify] to solve the constraints and generate new test input for exploring new path. For each set of concrete test input that leads to a new path that achieves new coverage, Pex generates a corresponding conventional unit test. So, all what you have to do is write your parametrized unit test.

In this post we gave a brief overview about parametrized unit tests and its related research areas. It’s a very important concept to understand before start learning Microsoft Pex, which we will start . In later posts, we will explore Microsoft Pex in detail.

Getting Started with Java PathFinder

noreply@blogger.com (Ebeid Soliman ElSayed) — Sun, 31 Mar 2013 00:06:00 +0000

Java™ Pathfinder, or "JPF" as we call it from here, is a highly customizable execution environment for verification of Java™ bytecode programs. JPF was developed at the NASA Ames Research Center, open sourced in 2005, and is freely available here under the NOSA 1.3 license. JPF started as a software model checker but it integrates model checking, program analysis and testing.

In this post we will go in a step by step process to setup the environment where Java Path Finder can be used within Eclipse IDE environment. We will show a simple example of using JPF.

Step 1. Install Eclipse
- Step 1.1 : Download and install the latest JDK fro your platform from here.
- Step 1.2 : Download the latest Eclipse from here. Once you have unzipped the previously downloaded file, you will see a folder named eclipse. In that folder you will find the eclipse application. We recommend you create a shortcut on the desktop to simplify the launching of eclipse. Notice that unlike Java, Eclipse does not have an installation process. Once you have unzipped the file you are done.
Step 2. Download JPF

Step 2.1 : Download Mercurial plug-in for Eclipse
- Launch Eclipse. Go to Help >> Install New Software
- Click Add
- Insert “Mercurial” in the Name box. Insert “http://mercurialeclipse.eclipselabs.org.codespot.com/hg.wiki/update_site/stable/” in the Address box. Click Ok.
- In the install dialog select the newly added site from Work with drop down box. The list below it will show the available software at this site, select the item named MercurialEclipse and click Next, then finish..
Step 2.2 : Download the JPF source code
- In Eclipse menu : File >> Import >> Mercurial >> Clone Existing Mercurial Repository >> Next.
- In the repository location URL, insert “http://babelfish.arc.nasa.gov/hg/jpf/jpf-core”. Click Next, then Finish. The JPF project files will be downloaded to your local computer.
Step 2.3 : Build the JPF

In the package explorer, select the build.xml, right click on it, select “run as” >> “Run as Ant build “.
Eclipse will build JPF and a new folder named “build” will appear in the package explorer (it may not appear even the build was successful, double check the windows explorer for it.)

If the build failed due to error missing JDK:

On Eclipse menu : Project >> Properties. Select Builders from the lift hand side list
Select Ant Builder and click New.
Select Ant Builder and click New.
In the Edit Configuration dialog, select the “JRE” tab, then select “Separate JRE” from the “Runtime JRE section”. In the drop down list, select JDK:
If you couldn’t find the JDK in the drop down list, click “Installed JRE …”
In the “Preferences” dialog, click “Add”
Select “Standard VM” and click “Next”
Click “Directory” and navigate to your JDK installation directory. Then click “Finish” to return to the previous dialog.
Select the JDK and click Ok.
Select JDK in the drop down box and click OK

Step 3 : Install and configure JPF
- Step 3.1 : Install JPF files : Copy the whole jpf-core project folder into <user.home>/projects/jpf
- Step 3.2 : Create site.properties
  - Create site.properties under <user.home>/.jpf folder.
  - To create the folder on windows, use the following command on the command window after navigating to the user home: mkdir .jpf .
  - Paste the following content into the site.properties file.
```
# JPF site configuration

jpf.home = ${user.home}/projects/jpf

# can only expand system properties
jpf-core = ${user.home}/projects/jpf/jpf-core

# annotation properties extension
jpf-aprop = ${jpf.home}/jpf-aprop
extensions+=,${jpf-aprop}
# numeric extension
jpf-numeric = ${jpf.home}/jpf-numeric
extensions+=,${jpf-numeric}
# concurrent extension
#jpf-concurrent = ${jpf.home}/jpf-concurrent
#extensions+=,${jpf-concurrent}
jpf-shell = ${jpf.home}/jpf-shell
extensions+=,${jpf-shell}
jpf-awt = ${jpf.home}/jpf-awt
extensions+=,${jpf-awt}

jpf-awt-shell = ${jpf.home}/jpf-awt-shell
extensions+=,${jpf-awt-shell}
```
- Step 3.3 : Install Eclipse plug-in
  - The JPF Eclipse plug-in allows you to invoke JPF directly from Eclipse.
  - In Eclipse go to Help >> Install New Software.
  - In the new window click Add. In the name insert JPF. In the location insert “http://babelfish.arc.nasa.gov/trac/jpf/raw-attachment/wiki/install/eclipse-plugin/update/” then click Ok.
  - From the “Work with:” drop down menu select the newly added site.
  - In the list, check “Eclipse-JPF”, click Next, then Finish.
Step 4 : Start the demo project
- Step 4.1 : Create and configure project
  - In Eclipse menu : File >> New >> Java project : project name = jpf_junit_demo
  - In the package explorer, right click on the project name. Choose “Build Path” >> “Add libraries”
  - Select User Library and click Next.
  - Click User libraries. Then New. You will see a window for creating a library. Put “jpf” as the library name and click Ok to return to the previous dialog.
  - Select the newly added jpf library and click “Add External JARs …”.
  - Select all the jar files under <user.home>/jpf/jpf-core/build and click “Open” to add them under library jpf
  - Click New to create a new user library. Name it “jpf-support”.
  - Select the newly added jpf-support library and click “Add External JARs …”.
  - Select all the jar files under <user.home>/jpf/jpf-core/lib and click “Open” to add them under library jpf-support
  - Now the User Libraries should look like the following:
  - Right click on the project name and select : New >> Folder, name the folder as test. Right click on the newly created test folder and select : Build Path >> Use as Source Folder. This folder will be used in the future to hold the JUnit test classes.
  - Now the project folder should look like the following:
- Step 4.2 : Create a class (program under test)
  - Right click on the “src” folder and choose: New >> Class. Name it Calculator and click Finish.
  - This is the content of the very simple Calculator.java which prints the sum of two integers. The Verify statements are the only JPF statements, the rest is pure java. We will explain the JPF statements through its results in the next step.
```
import gov.nasa.jpf.jvm.Verify;

public class Calculator {
    int add (int a, int b)
    {
        return a+b;
    }
    public static void main(String[] args)
    {
        int a = Verify.getInt(1,3);
        int b = Verify.getInt(5,8);
        Calculator cal = new Calculator();
        System.out.println(a + "+" + b + "=" + cal.add(a, b));
    }
}
```
  - Step 4.3 : Run JPF from Eclipse
    - Right click on the project name and select: New >> File. Name the file “jpf_junit_demo.jpf”.
    - This is the content of the newly added file
      +classpath=${config_path}/bin
      target=TestGenerator
    - In the package explorer, right click on the “jpf_junit_demo.jpf” and select “Verify …”.
    - If you couldn’t see the “Verify…” option, this means that you didn’t installed the JPF plug-in properly. See step 3.3.
    - JPF will get started and you will see its output in the console pane. It will be like the following:
```
Executing command: java -jar C:\Users\Ebeid\projects\jpf\jpf-core\build\RunJPF.jar +shell.port=4242 C:\Users\Ebeid\projects\jpf_junit_demo\jpf_junit.demo.jpf
JavaPathfinder v6.0 (rev 960+) - (C) RIACS/NASA Ames Research Center

====================================================== system under test
application: Calculator.java

====================================================== search started: 3/28/13 4:41 PM
1+5=6
1+6=7
1+7=8
1+8=9
2+5=7
2+6=8
2+7=9
2+8=10
3+5=8
3+6=9
3+7=10
3+8=11

====================================================== results
no errors detected

====================================================== statistics
elapsed time:       00:00:02
states:             new=5, visited=11, backtracked=16, end=12
search:             maxDepth=3, constraints hit=0
choice generators:  thread=1 (signal=0, lock=1, shared ref=0), data=4
heap:               new=545, released=197, max live=353, gc-cycles=13
instructions:       5098
max memory:         55MB
loaded code:        classes=82, methods=1362

====================================================== search finished: 3/28/13 4:41 PM
```
  According to our program code and the JPF results, you may already guessed what the verify statements are doing in the following lines of code:
  
  int a = Verify.getInt(1,3);
  int b = Verify.getInt(5,8);
  - Simply, Verify.getInt(x, y) get a new integer within the range [x,y].
  - That is it ?
  - Partially yes.
  - But how JPF get 12 print outputs ?
  - Ok Actually JPF starts a new clone of the program for each value resulted from a Verify.getInt() statement.
  - This means that JPF will create 3 versions of the program, each one of these versions runs with a different value of a (1, 2, 3).
  - Each version of these three versions will be used as a base to create another 4 versions of the program for variable b, each one of these versions runs with a different value of b (5, 6, 7, 8).
  So, the total versions of the program running by JPF is 3 * 4 = 12. Its like a tree where the leafs are the print statements :
  - a = 1
    - a = 1, b = 5 | System.out.println(a + "+" + b + "=" + cal.add(a, b)); | 1+5=6
    - a = 1, b = 6 | System.out.println(a + "+" + b + "=" + cal.add(a, b)); | 1+6=7
    - a = 1, b = 7 | System.out.println(a + "+" + b + "=" + cal.add(a, b)); | 1+7=8
    - a = 1, b = 8 | System.out.println(a + "+" + b + "=" + cal.add(a, b)); | 1+8=9
  - a = 2
    - a = 1, b = 5 | System.out.println(a + "+" + b + "=" + cal.add(a, b)); | 2+5=7
    - a = 1, b = 6 | System.out.println(a + "+" + b + "=" + cal.add(a, b)); | 2+6=8
    - a = 1, b = 7 | System.out.println(a + "+" + b + "=" + cal.add(a, b)); | 2+7=9
    - a = 1, b = 8 | System.out.println(a + "+" + b + "=" + cal.add(a, b)); | 2+8=10
  - a = 3
    - a = 1, b = 5 | System.out.println(a + "+" + b + "=" + cal.add(a, b)); | 3+5=8
    - a = 1, b = 6 | System.out.println(a + "+" + b + "=" + cal.add(a, b)); | 3+6=9
    - a = 1, b = 7 | System.out.println(a + "+" + b + "=" + cal.add(a, b)); | 3+7=10
    - a = 1, b = 8 | System.out.println(a + "+" + b + "=" + cal.add(a, b)); | 3+8=11
  you can use JPF for your JUnit tests to generate explore your code states as we did; just keep in mind that a small increase in the number of states of variables could result in huge number of total program states.
  
  I hope you enjoyed it, and see you later in more posts about JPF.

Midnight Researcher Notes

Syntax Modification using Microsoft Roslyn

Creating and Modifying Trees

Modifying Syntax Trees using SyntaxRewriters

Semantic Analysis using Microsoft Roslyn

Creating a Compilation

The SemanticModel

Syntax Analysis using Microsoft Roslyn

Traversing Syntax Trees

Query Methods

SyntaxWalkers

Overview of Microsoft Roslyn – The first compiler-as-service product.

Introduction

Rosyln API Layers

Working with Syntax

Syntax Trees Attributes

Syntax Tree Elements

How to debug your Pex parameterized unit tests

Convert Microsoft Pex path conditions from Z3 native format to SMT-LIB

The Z3 Constraint Solver, a developer perspective

What SAT means ?

What SMT means ?

What SMT solver means ?

SMT solvers

Download Z3

C# Example

How it works ?

Get path conditions from Microsoft Pex

Bring “Sign in as Different User” back to SharePoint 2013

Create a new Web Part Page with Quick Launch menu

Getting Started with MongoDB – Part 2

Indexing

Aggregations

Server-Side Commands

MapReduce

Getting Started with MongoDB – Part 1

Installation

Starting the Server

Connecting to the Server

Run MongoDB as a Windows Service

Data Model

CRUD operations

elemMatch

Boolean Operators

Update

References

Delete

Reading with Code

Getting Started with Apache Cassandra

Data Model

Installation

Commads

جريدة الشروق : تقرير شهادات سكان عمارات العبور

سكان «عمارات العبور»: ضرب النار والغاز بدأ من الحرس الجمهوري بدون مقدمات

نشر فى : الإثنين 8 يوليو 2013 - 4:13 مآخر تحديث : الإثنين 8 يوليو 2013 – 4:13 م

Redis 101– Part 2

Publish-Subscribe

Server Information

Durability

Snapshotting

Append-only file

Security

Benchmarking

Replication

Running multiple Redis instances on the same server

Redis 101

GET & SET

Expire

Transactions

Complex Data Types

List

Blocking Lists

Set

Sorted Sets

Hash

Database Namespaces

More commands

PostgreSQL 101

History of PostgreSQL

Installation

نشر فى : الإثنين 8 يوليو 2013 - 4:13 م
آخر تحديث : الإثنين 8 يوليو 2013 – 4:13 م