~/cesarsouza/blog

OxyPlot.Axes.LinearAxis is obsolete

2015-01-09T22:10:00.000-02:00

Warning 461 'OxyPlot.Axes.LinearAxis.LinearAxis(OxyPlot.Axes.AxisPosition, double, double, string)' is obsolete

If you try to use OxyPlot.Axes.LinearAxis' constructor with parameters, the compiler will complain telling you the method has been deprecated and shouldn't be used. I couldn't find on the web which was the alternative solution to resolve this issue, but then it occurred to me that, what really is being deprecated, is the passage of arguments through the constructor's parameters.

As such, the solution is simply to rewrite your code and call the axis' default constructor instead, using C# object initialization syntax to configure your object:

  var dateAxis = new OxyPlot.Axes.LinearAxis()
  {
      Position = AxisPosition.Bottom,
      Minimum = range.Min,
      Maximum = range.Max,
      Key = "xAxis",
      MajorGridlineStyle = LineStyle.Solid,
      MinorGridlineStyle = LineStyle.Dot,
      IntervalLength = 80
  };

Hope it can be of help if you were getting those warnings like me.

Liblinear algorithms in C#

2014-12-23T18:04:00.001-02:00

The Accord.NET Framework is not only an image processing and computer vision framework, but also a machine learning framework for .NET. One of its features is to encompass the exact same algorithms that can be found in other libraries, such as LIBLINEAR, but offer them in .NET withing a common interface ready to be incorporated in your application.

Download the machine learning framework.
Browse the source code online.

What is LIBLINEAR?

As its authors put, LIBLINEAR is a library for large linear classification. It is intended to be used to tackle classification and regression problems with millions of instances and features, although it can only produce linear classifiers, i.e. linear support vector machines.

The framework now offers almost all liblinear algorithms in C#, except for one. Those include:

As it can be seen, the command line option 4 is missing. Mode #4 refers to the Crammer and Singer's formulation for multi-class classification. However, the framework already provides different ways to obtain both multi-class as well as multi-label classifiers through both Voting and Directed Acyclic Graphs (DDAG) mechanisms.

Additionally, the framework also offers:

The framework can equally load data and load and save support vector machines using the LIBSVM format. This means it should be straighforward to create or learn your models using one tool and run it on the other, if that would be necessary. For example, given that Accord.NET can run on mobile applications, it is possible to create and learn your models in a computing grid using liblinear and then integrate it in your Windows Phone application by loading it in Accord.NET.

The advantages are that:

Learning algorithms implement one common interface, rather than several functions splitted through the code;
Algorithms are available as a concise library, ready to be integrated in your existing or new applications, instead of being part of a black-box command line tool (but it can also be used as a command line tool, see sample applications);
The algorithms can run in Windows, Windows RT, ASP.NET, Windows Phone, Android, iOS, Linux and MacOS (through Mono/Xamarin);
They can be combined with other meta-algorithms available in the framework to create multi-class and multi-label support vector machines, as well as be part of cross-validation, bootstrapping and split-set validation techniques.

When studying and porting the algorithms, I have also set up a liblinear GitHub repository page to track changes between versions. I hope this repository can also be helpful for other people willing to track modifications done the the liblinear project.

Example

Learning linearly separable problems with a linear machine

In the following example we will create a linear machine to learn a simple linearly separable binary AND problem.

// Create a simple binary AND
// classification problem:

double[][] problem =
{
    //             a    b    a + b
    new double[] { 0,   0,     0    },
    new double[] { 0,   1,     0    },
    new double[] { 1,   0,     0    },
    new double[] { 1,   1,     1    },
};

// Get the two first columns as the problem
// inputs and the last column as the output
            
// input columns
double[][] inputs = problem.GetColumns(0, 1);

// output column
int[] outputs = problem.GetColumn(2).ToInt32();

// Plot the problem on screen
ScatterplotBox.Show("AND", inputs, outputs).Hold();

The binary AND classification problem. As it can be seen, the AND classification problem is a linearly separable problem, as it is possible to draw a line separating the blue points from the red points with ease.

// However, SVMs expect the output value to be
// either -1 or +1. As such, we have to convert
// it so the vector contains { -1, -1, -1, +1 }:
//
outputs = outputs.Apply(x => x == 0 ? -1 : 1);


// Create a new linear-SVM for two inputs (a and b)
SupportVectorMachine svm = new SupportVectorMachine(inputs: 2);

// Create a L2-regularized L2-loss support vector classification
var teacher = new LinearDualCoordinateDescent(svm, inputs, outputs)
{
    Loss = Loss.L2,
    Complexity = 1000,
    Tolerance = 1e-5
};

// Learn the machine
double error = teacher.Run(computeError: true);

// Compute the machine's answers for the learned inputs
int[] answers = inputs.Apply(x => Math.Sign(svm.Compute(x)));

// Plot the results
ScatterplotBox.Show("SVM's answer", inputs, answers).Hold();

The linear SVM's answer to the linearly separable AND problem. As it can be seen, a linear SVM can correctly predict the colors for each of the points in the original problem. This happens because the SVM learning algorithm is able to find the line that separates the blue points from the red points.

Learning non-linearly separable problems with a linear machine through kernel expansions

Now, we will move a bit further. We will use an explicit kernel expansion to learn the non-linearly separable exclusive or (XOR) problem.

// Create a simple binary XOR
// classification problem:

double[][] problem =
{
    //             a    b    a XOR b
    new double[] { 0,   0,      0    },
    new double[] { 0,   1,      1    },
    new double[] { 1,   0,      1    },
    new double[] { 1,   1,      0    },
};

// Get the two first columns as the problem
// inputs and the last column as the output

// input columns
double[][] inputs = problem.GetColumns(0, 1);

// output column
int[] outputs = problem.GetColumn(2).ToInt32();

// Plot the problem on screen
ScatterplotBox.Show("XOR", inputs, outputs).Hold();

The binary XOR problem. The XOR problem is not a linerly separable problem, because it is not possible to draw a line separating the blue points from the red points. In this setting, we should expect a linear SVM learning algorithm to fail, because it will not be able to find this line that doesn't exist.

// However, SVMs expect the output value to be
// either -1 or +1. As such, we have to convert
// it so the vector contains { -1, -1, -1, +1 }:
//
outputs = outputs.Apply(x => x == 0 ? -1 : 1);


// Create a new linear-SVM for two inputs (a and b)
SupportVectorMachine svm = new SupportVectorMachine(inputs: 2);

// Create a L2-regularized L2-loss support vector classification
var teacher = new LinearDualCoordinateDescent(svm, inputs, outputs)
{
    Loss = Loss.L2,
    Complexity = 1000,
    Tolerance = 1e-5
};

// Learn the machine
double error = teacher.Run(computeError: true);

// Compute the machine's answers for the learned inputs
int[] answers = inputs.Apply(x => Math.Sign(svm.Compute(x)));

// Plot the results
ScatterplotBox.Show("SVM's answer", inputs, answers).Hold();

As we can see, the linear SVM failed to predict the correct colors for each of the points. The problem is that the answers from a linear SVM are constrained to be hyperplanes (in this 2D case, lines) that separate the points. Because there is no line that separates the blue points from the red points in the XOR problem, the linear SVM learning algorithm tries its best, finding an approximate solution, but not the XOR solution we were looking for.

// Use an explicit kernel expansion to transform the 
// non-linear classification problem into a linear one
//
// Create a quadratic kernel
Quadratic quadratic = new Quadratic(constant: 1);
            
// Project the inptus into a higher dimensionality space
double[][] expansion = inputs.Apply(quadratic.Transform);


            
// Create a new linear-SVM for the transformed input space
svm = new SupportVectorMachine(inputs: expansion[0].Length);

// Create the same learning algorithm in the expanded input space
teacher = new LinearDualCoordinateDescent(svm, expansion, outputs)
{
    Loss = Loss.L2,
    Complexity = 1000,
    Tolerance = 1e-5
};

// Learn the machine
error = teacher.Run(computeError: true); 

// Compute the machine's answers for the learned inputs
answers = expansion.Apply(x => Math.Sign(svm.Compute(x)));

// Plot the results
ScatterplotBox.Show("SVM's answer", inputs, answers).Hold();

By using an explicit kernel expansion, we can use a linear SVM to learn a non-linearly separable problem. This is possible because the kernel transformation projects the data into a higher dimensionality space where the data is indeed linearly separable. For an intuition on how this could be possible, please check the blog post kernel functions for machine learning applications.

Learning liblinear problems from libsvm format

Now, we move even further, and use a linear machine to load one of the toy LIBLINEAR problems available in LibSVM format using the framework's SparseReader class.

// Create a new LibSVM sparse format data reader
// to read the Wisconsin's Breast Cancer dataset
//
var reader = new SparseReader("examples-sparse.txt");

int[] outputs; // Read the classification problem into dense memory
double[][] inputs = reader.ReadToEnd(sparse: false, labels: out outputs);

// The dataset has output labels as 4 and 2. We have to convert them
// into negative and positive labels so they can be properly processed.
//
outputs = outputs.Apply(x => x == 2 ? -1 : +1);

// Create a new linear-SVM for the problem dimensions
var svm = new SupportVectorMachine(inputs: reader.Dimensions);

// Create a learning algorithm for the problem's dimensions
var teacher = new LinearDualCoordinateDescent(svm, inputs, outputs)
{
    Loss = Loss.L2,
    Complexity = 1000,
    Tolerance = 1e-5
};

// Learn the classification
double error = teacher.Run();

// Compute the machine's answers for the learned inputs
int[] answers = inputs.Apply(x => Math.Sign(svm.Compute(x)));

// Create a confusion matrix to show the machine's performance
var m = new ConfusionMatrix(predicted: answers, expected: outputs);

// Show it onscreen
DataGridBox.Show(new ConfusionMatrixView(m));

The confusion matrix for the binary classification problem. As it can be seen, the higher values concentrate in the diagonal. Those values indicate how many hits (correct guesses) the machine was able to make. The other values that don't lie in the diagonal indicate how many errors (and of what kind) the machine made in this classification problem.

The last version of this tutorial can also be seen on the project's wiki pages for linear machines.

References

R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin. LIBLINEAR: A Library for Large Linear Classification, Journal of Machine Learning Research 9(2008), 1871-1874. Software available at http://www.csie.ntu.edu.tw/~cjlin/liblinear

A Tutorial On Principal Component Analysis with the Accord.NET Framework

2013-06-18T11:26:00.000-03:00

Principal Component Analysis (PCA) is a technique for exploratory data analysis with many success applications in several research fields. It is often used in image processing, data analysis, data pre-processing, visualization and is often used as one of the most basic building steps in many complex algorithms.

One of the most popular resources for learning about PCA is the excellent tutorial due to Lindsay I Smith. On her tutorial, Lindsay gives an example application for PCA, presenting and discussing the steps involved in the analysis.

Souza, C. R. "A Tutorial on Principal Component Analysis with the Accord.NET Framework".
Department of Computing, Federal University of São Carlos, Technical Report, 2012.

This said, the above technical report aims to show, discuss and otherwise present the reader to the Principal Component Analysis while also reproducing all Lindsay's example calculations using the Accord.NET Framework. The report comes with complete source C# code listings and also has a companion Visual Studio solution file containing all sample source codes ready to be tinkered inside a debug application.

While the text leaves out the more detailed discussions about the exposed concepts to Lindsay and does not addresses them in full details, it presents some practical examples to reproduce most of the calculations given by Lindsay on her tutorial using solely Accord.NET.

If you like a practical example on how to perform matrix operations in C#, this tutorial may help getting you started.

Sequence Classifiers in C#: Hidden Conditional Random Fields

2013-06-03T09:20:00.000-03:00

After a preliminary article on hidden Markov models, some months ago I had finally posted the article on Hidden Conditional Random Fields (HCRF) on CodeProject. The HCRF is a discriminative model, forming the generative-discriminative pair with the hidden Markov model classifers.

This CodeProject article is a second on a series of articles about sequence classification, the first being about Hidden Markov Models. I've used this opportunity to write a little about generative versus discriminative models, and also provide a brief discussion on how Vapnik's ideas apply to these learning paradigms.

All the code available on those articles are also available within the Accord.NET Framework. Those articles provide good examples on how to use the framework and can be regarded as a practical implementation on how to use those models with the framework.

Complete framework documentation can be found live at Google Code. The framework has now been referred on 30+ publications over the years, and several more are already in the works, by me and users around the world.

Academical publications

Talking about publications, the framework has been used within my own research on Computer Vision. If you need help in understanding the inner workings of the HCRF, a more visual explanation on the HCRF derivation can also be found at the presentation I gave on Iberamia 2012 about Fingerspelling Recognition with Support Vector Machines and Hidden Conditional Random Fields [pdf].

An application to a more interesting problem, namely natural words drawn from Sign Languages using a Microsoft Kinect, has also been accepted for publication at the 9th International Conference on Machine Learning and Data Mining, MLDM 2013, and will be available publicly shortly.

As usual, hope you find it interesting!

Point and Call and the Windows Phone Ecosystem

2013-05-06T10:21:00.000-03:00

So a few days ago I bought a Windows Phone device. So far, I am impressed with the Windows Phone ecosystem.

The nicest thing is that I was finally able to test an app I've been eager to try for months: Point and Call.

Point-and-call in everyday life.

Now, the most interesting part, is that this app uses part of the Accord.NET Framework to do its magic :-)

The app author, Antti Savolainen, was kind enough to share some details about his app. It uses part of the SVMs framelet from Accord.NET to do the digit recognition, mostly based on one of the earlier CodeProject articles I've posted in the past. Needless to say, Antti did an awesome job, as the SVM part was surely just a tiny fraction of all the work in preprocessing, adjusting, locating, and doing the right things at the right times that I would never be able to figure out alone :-)

Surely, he and his company, Sadiga, deserves all the credits for this neat app!

If you would like find more interesting uses of the Accord.NET Framework, don't forget to check the framework's Showcase page for details!

Sequence Classifiers in C#: Hidden Markov Models

2013-02-14T10:12:00.000-02:00

Few days ago I published a new article on CodeProject, about classifiers based on banks of hidden Markov models to accomplish sequence classification.

While I have written about this subject in the past on this very blog, this time the article deals with Gaussian, continuous-density hidden Markov models rather than discrete ones. Plus, at this time the Accord.NET Framework has evolved much since that 2010 post, and the new article reflects most of the improvements and additions in those last two years.

In the meantime, this article is also serving as a hook to a future article, an article about Hidden Conditional Random Fields (HCRFs). The HCRF models can serve the same purpose as the HMMs but can be generalized to arbitrary graph structures and be trained discriminatively, which could be an advantage on classification tasks.

As always, I hope readers can find it a good read :-)