The thing about triangulation is that you need to know the extrinsic parameters of your cameras – the difference in location and rotation between them.
To get the camera matrices… that’s a different story that I’m going to cover shortly in a tutorial (already in writing) about Structure from Motion.

But let’s assume that we already have the extrinsic matrices. In most cases, where you know what the motion is (you took the pictures , you can just write the matrices explicitly.

Linear Triangulation

/**
 From "Triangulation", Hartley, R.I. and Sturm, P., Computer vision and image understanding, 1997
 */
Mat_ LinearLSTriangulation(Point3d u,		//homogenous image point (u,v,1)
				   Matx34d P,		//camera 1 matrix
				   Point3d u1,		//homogenous image point in 2nd camera
				   Matx34d P1		//camera 2 matrix
								   )
{
	//build matrix A for homogenous equation system Ax = 0
	//assume X = (x,y,z,1), for Linear-LS method
	//which turns it into a AX = B system, where A is 4x3, X is 3x1 and B is 4x1
	Matx43d A(u.x*P(2,0)-P(0,0),	u.x*P(2,1)-P(0,1),		u.x*P(2,2)-P(0,2),
		  u.y*P(2,0)-P(1,0),	u.y*P(2,1)-P(1,1),		u.y*P(2,2)-P(1,2),
		  u1.x*P1(2,0)-P1(0,0), u1.x*P1(2,1)-P1(0,1),	u1.x*P1(2,2)-P1(0,2),
		  u1.y*P1(2,0)-P1(1,0), u1.y*P1(2,1)-P1(1,1),	u1.y*P1(2,2)-P1(1,2)
			  );
	Mat_ B = (Mat_(4,1) <<	-(u.x*P(2,3)	-P(0,3)),
					  -(u.y*P(2,3)	-P(1,3)),
					  -(u1.x*P1(2,3)	-P1(0,3)),
					  -(u1.y*P1(2,3)	-P1(1,3)));

	Mat_ X;
	solve(A,B,X,DECOMP_SVD);

	return X;
}

This method relies very simply on the principle that every 2D point in image plane coordinates is a projection of the real 3D point. So if you have two views, you can set up an overdetermined linear equation system to solve for the 3D position.

See how simple defining a Matx43d struct from scratch and using it in solve(..) is?
I tried doing some more fancy stuff with Mat.row(i) and Mat.col(i), trying to stick to Hartley’s description of the A matrix, but it just didn’t work.

Using it

Using this method is easy:

//Triagulate points
void TriangulatePoints(const vector& pt_set1,
					   const vector& pt_set2,
					   const Mat& Kinv,
					   const Matx34d& P,
					   const Matx34d& P1,
					   vector& pointcloud,
					   vector& correspImg1Pt)
{
#ifdef __SFM__DEBUG__
	vector depths;
#endif

	pointcloud.clear();
	correspImg1Pt.clear();

	cout << "Triangulating...";
	double t = getTickCount();
	unsigned int pts_size = pt_set1.size();
#pragma omp parallel for
	for (unsigned int i=0; i		Point2f kp = pt_set1[i];
		Point3d u(kp.x,kp.y,1.0);
		Mat_ um = Kinv * Mat_(u);
		u = um.at(0);
		Point2f kp1 = pt_set2[i];
		Point3d u1(kp1.x,kp1.y,1.0);
		Mat_ um1 = Kinv * Mat_(u1);
		u1 = um1.at(0);

		Mat_ X = IterativeLinearLSTriangulation(u,P,u1,P1);

//		if(X(2) > 6 || X(2) < 0) continue;

#pragma omp critical
		{
			pointcloud.push_back(Point3d(X(0),X(1),X(2)));
			correspImg1Pt.push_back(pt_set1[i]);
#ifdef __SFM__DEBUG__
			depths.push_back(X(2));
#endif
		}
	}
	t = ((double)getTickCount() - t)/getTickFrequency();
	cout << "Done. ("<
	//show "range image"
#ifdef __SFM__DEBUG__
	{
		double minVal,maxVal;
		minMaxLoc(depths, &minVal, &maxVal);
		Mat tmp(240,320,CV_8UC3); //cvtColor(img_1_orig, tmp, CV_BGR2HSV);
		for (unsigned int i=0; i			double _d = MAX(MIN((pointcloud[i].z-minVal)/(maxVal-minVal),1.0),0.0);
			circle(tmp, correspImg1Pt[i], 1, Scalar(255 * (1.0-(_d)),255,255), CV_FILLED);
		}
		cvtColor(tmp, tmp, CV_HSV2BGR);
		imshow("out", tmp);
		waitKey(0);
		destroyWindow("out");
	}
#endif
}

Note that you must have the camera matrix K (a 3×3 matrix of the intrinsic parameters), or rather it’s inverse, noted here as Kinv.

Results and some discussion

3D view of the triangulated point cloud

Notice how stuff is distorted in the 3D view… but this is not due projective ambiguity! as I am using the Essential Matrix to obtain the camera P matrices (cameras are calibrated). Hartley and Zisserman explain this in their book on page 258, and the reasons for projective ambiguity (and how to resolve it) on page 265. The distortion must be due to inaccurate point correspondence…

The cool visualization is done using the excellent PCL library.

Iterative Linear Triangulation

Hartley, in his article “Triangulation” describes another triangulation algorithm, an iterative one, which he reports to “perform substantially better than the [...] non-iterative linear methods”. It is, again, very easy to implement, and here it is:

/**
 From "Triangulation", Hartley, R.I. and Sturm, P., Computer vision and image understanding, 1997
 */
Mat_<double> IterativeLinearLSTriangulation(Point3d u,	//homogenous image point (u,v,1)
											Matx34d P,			//camera 1 matrix
											Point3d u1,			//homogenous image point in 2nd camera
											Matx34d P1			//camera 2 matrix
											) {
	double wi = 1, wi1 = 1;
	Mat_<double> X(4,1);
	for (int i=0; i<10; i++) { //Hartley suggests 10 iterations at most
		Mat_<double> X_ = LinearLSTriangulation(u,P,u1,P1);
		X(0) = X_(0); X(1) = X_(1); X(2) = X_(2); X_(3) = 1.0;

		//recalculate weights
		double p2x = Mat_<double>(Mat_<double>(P).row(2)*X)(0);
		double p2x1 = Mat_<double>(Mat_<double>(P1).row(2)*X)(0);

		//breaking point
		if(fabsf(wi - p2x) <= EPSILON && fabsf(wi1 - p2x1) <= EPSILON) break;

		wi = p2x;
		wi1 = p2x1;

		//reweight equations and solve
		Matx43d A((u.x*P(2,0)-P(0,0))/wi,		(u.x*P(2,1)-P(0,1))/wi,			(u.x*P(2,2)-P(0,2))/wi,
				  (u.y*P(2,0)-P(1,0))/wi,		(u.y*P(2,1)-P(1,1))/wi,			(u.y*P(2,2)-P(1,2))/wi,
				  (u1.x*P1(2,0)-P1(0,0))/wi1,	(u1.x*P1(2,1)-P1(0,1))/wi1,		(u1.x*P1(2,2)-P1(0,2))/wi1,
				  (u1.y*P1(2,0)-P1(1,0))/wi1,	(u1.y*P1(2,1)-P1(1,1))/wi1,		(u1.y*P1(2,2)-P1(1,2))/wi1
				  );
		Mat_<double> B = (Mat_<double>(4,1) <<	-(u.x*P(2,3)	-P(0,3))/wi,
						  -(u.y*P(2,3)	-P(1,3))/wi,
						  -(u1.x*P1(2,3)	-P1(0,3))/wi1,
						  -(u1.y*P1(2,3)	-P1(1,3))/wi1
						  );

		solve(A,B,X_,DECOMP_SVD);
		X(0) = X_(0); X(1) = X_(1); X(2) = X_(2); X_(3) = 1.0;
	}
	return X;
}

(remember to define your EPSILON)
This time he works iteratively in order to minimize the reprojection error of the reconstructed point to the original image coordinate, by weighting the linear equation system.

Recap

So we’ve seen how easy it is to implement these triangulation methods using OpenCV’s nice Matx### and Mat_ structs.
Also solve(…,DECOMP_SVD) is very handy for overdetermined non-homogeneous linear equation systems.
Watch out for my Structure from Motion tutorial coming up, which will be all about using OpenCV to get point correspondence from pairs of images, obtaining camera matrices and recovering dense depth.

If you are looking for more robust solutions for SfM and 3D reconstructions, see:
http://phototour.cs.washington.edu/bundler/
http://code.google.com/p/libmv/
http://www.cs.washington.edu/homes/ccwu/vsfm/
Enjoy,
Roy.

Let’s start!

Some mathematical background

Don’t worry, it wont hurt. much.

So Spherical Harmonics, were invented to numerically express a whole bunch of things in physics like gravity and magnetic fields. But they also became very useful for computer graphics as they are perfect for modelling light falling on a spherical body.

But what ARE those mysterious spherical harmonics?

The way I see it, they are a series of “modes” or “eigenvectors” or “orthogonal components” of a base that spans the surface of a sphere.
To put it simple, they describe the surface of a sphere in increasing finer grained portions. Much like a Fourier decomposition does to a function, there is the base and there are coefficients that when multiplied with the base they recover the function.

How is that good for graphics?

People have used spherical harmonics mostly to model lighting of spherical objects. When you know the coefficients that describe the lighting, you can change them to Re-light an object, or De-light, or transfer the lighting conditions of one scene to another. Very useful!

Some good researchers, Basri and Jacobs, back in 2001 have formulated the first 9 harmonics as a function of the surface normal. On this page Basri references all his work on the subject: http://www.wisdom.weizmann.ac.il/~ronen/index_files/harmonic.html

But I like to reference a work that’s easier to process than Basri’s, that is the work of Wang et al from 2007. These guys made the steps to use spherical harmonics easier to follow: http://research.microsoft.com/en-us/um/people/zliu/cvpr2007.pdf.
But their algorithm is quite advanced, as it solves not only for the harmonics’ coefficients but also for the normals of the object in the image. They use some fancy optimization of an energy function over a graph, that I’m not going to discuss.
But they did make the process of finding the spherical harmonics’ coefficient very clear.

The bottom line

We should solve for a vector of 9 coefficients that describes the “lighting of the object” (a face in our case).
Each coefficient will tell us how much that specific harmonic is strong or weak, or in other words how lit is that certain area of the object.

Wang and Basri show a very simple method of using simultaneous linear equations to solve for the lighting coefficients, it depends only on knowing the normal of the object’s surface at each pixel in the image.

Getting the normals of a canonical face

So to get the normals, I thought the best way is to use a canonical model of a face (some king of an average face), instead of trying to recover the normals from the image pixels.
For that end, I used Rhino3D to model (very roughly) a shape that resembles a human face, starting from an elongated sphere.
Now all that’s left is to align the model with the face to relight, and that will supply the normals.

Cool. Then I built a small app that allows the user to move the model around until it’s aligned with the face image. I used FLTK 3.0 to do it since they have a simple interface with OpenGL, they are cross platform, and lightweight.
So I set up a scene where I have the image as the background, and the model is floating above it, half transparent so the user can find the right spot. I added functions for rotating the model, and extra stuff like turning the model opaque.

To get the normal map I used a very simple GLSL shader, that simply colors the pixel with the value of the normal nX,nY,nZ -> R,G,B.
This way the result image OpenGL renders is simply the normal map of the face model. I just grab it using glReadPixels.

Estimating spherical harmonics

So, after the model is aligned, we can assume we have the normals ready for us for each pixel in the image, and the intensity in each pixel is also known.
The first step that Wang suggests, without knowledge of the real face albedo (the real color of every pixel without any lighting effect), is to get an approximation of the 9-vector of lighting coefficients by setting a constant albedo. Easy enough, we can set the albedo to the average color in the face.
Then we can simply build a huge set of linear equations (huge as the number of pixels in the image), and solve an overdetermined system to get the 9 coefficients.

		Scalar albedo_constant = mean(face_img_hsv, smallFaceMask);

		//setup linear equation system, lighting coefficients (l) is unknown
		//I = p00 * Ht * l
		float p00 = (float)albedo_constant[2] / 255.0f;

		cout << "Build Ht("<<n<<",9)...";
		cout << "Build I("<<n<<",1)...";
		//build Ht and I
		Mat_<float> Ht(n,9);
		Mat_<float> I(n,1);
		int pos = 0;
		vector<Mat_<uchar> > face_img_chnls; split(face_img_hsv, face_img_chnls);
		for (int i=0; i<normalMapFlat.rows; i++) {
			if (smallFaceMask(i) == 0) { //is this pixel on the face?
				continue;
			}
			Ht.row(pos) = p00 * calculateSphericalHarmonicsForNormal(normalMapFlat(i));
			I(pos,0) = face_img_chnls[2](i) / 255.0f; //get V from HSV of pixel [0,1]
			pos ++;
		}
		cout << "DONE"  << endl;

		cout << "Solve" <<endl;
		solve(Ht, I, l, DECOMP_SVD);

		cout << "initial lighting coeffs: ";
		for (int i=0; i<l.rows; i++) {
			cout<<l.at<float>(i)<<",";
		}

Booyah! lighting coefficients.

But this is only the first step. Now we can get an approximation of the albedo as well, using the coefficients:

		Mat_<Vec3b> face_img_v3b = face_img;

		#pragma omp parallel for schedule(dynamic)
		for (int y=0; y<face_img.rows; y++) {
			for (int x=0; x<face_img.cols; x++) {
				if (face_mask(y,x) == 0) {
					albedo(y,x) = 0;
					continue;
				}
				Mat sph = calculateSphericalHarmonicsForNormal(normalMap(y,x));
				Mat_<float> sph_l = sph * l;
				float fsph_l = sph_l(0);

				for (int cn = 0; cn<3; cn++) {
					float fimg = face_img_v3b(y,x)[cn] / 255.0f;
					albedo(y,x)[cn] = (fimg / fsph_l);
				}
			}
		}

Done.
Now that we have an initial albedo, Wang suggests we compute the coefficients again to get a better approximation, and then the albedo again.
I however ran into some problems trying to do the second iteration, and the results always came out too dark… But even with the first iteration you can see a very nice change.
Look at the video from before, you can see the right side of the face, which is over-lit, was darkened and the left side was lit up.

Code

The code for spherical harmonics analysis of images is part of a bigger project I have been working on for some time. I also spoke of it in a previous post.
Anyway it’s up in GitHub: https://github.com/royshil/HeadReplacement/tree/master/HeadReplacement
You’re looking for 4 files:

• SpharmonicsUI.cpp
• SpharmonicsUI.h
• spherical_harmonics_analysis.cpp
• spherical_harmonics_analysis.h

You can use the CMakeLists.txt to compile, but here’s a CMakeLists.txt that should take you there in one piece (fingers crossed):

find_package(OpenCV REQUIRED)
find_package(OpenGL REQUIRED)
find_package(OpenMP REQUIRED)

######## Find and add GLEE ########
file(GLOB_RECURSE GLEE_PATH "${CMAKE_SOURCE_DIR}/GLee.c")
if(GLEE_PATH STREQUAL GLEE_PATH-NOTFOUND)
	message(STATUS "GLEE was not found")
else()
	list(LENGTH GLEE_PATH GLEE_PATH_LEN)
	if(GLEE_PATH_LEN GREATER 1)
		list(GET GLEE_PATH 1 GLEE_PATH)
	endif()
	file(RELATIVE_PATH GLEE_PATH ${CMAKE_SOURCE_DIR} ${GLEE_PATH})
	get_filename_component(GLEE_PATH ${GLEE_PATH} REALPATH)
	get_filename_component(GLEE_PATH ${GLEE_PATH} PATH)
	message(STATUS "Found GLEE at ${GLEE_PATH}")
	add_library(GLEE ${GLEE_PATH}/GLee.c)
endif()

############ Find FLTK ############
if(NOT DEFINED FLTK_PATH)
	file(GLOB_RECURSE FLTK_PATH "${CMAKE_SOURCE_DIR}/Widget.h")
	if(FLTK_PATH STREQUAL FLTK_PATH-NOTFOUND   OR   FLTK_PATH STREQUAL "")
		message(STATUS "FLTK was not found !!!!!")
	else()
		list(LENGTH FLTK_PATH FLTK_PATH_LEN)
		if(FLTK_PATH_LEN GREATER 1)
			list(GET FLTK_PATH 1 FLTK_PATH)
		endif()
		file(RELATIVE_PATH FLTK_PATH ${CMAKE_SOURCE_DIR} ${FLTK_PATH})
		get_filename_component(FLTK_PATH ${FLTK_PATH} REALPATH)
		get_filename_component(FLTK_PATH ${FLTK_PATH} PATH)
		message(STATUS "Found FLTK at ${FLTK_PATH}")
	endif()
else()
	get_filename_component(FLTK_PATH ${FLTK_PATH} REALPATH)
	message(STATUS "FLTK path set to ${FLTK_PATH}")
endif()
set(FLTK_INCLUDE_DIR ${FLTK_PATH}/include)
set(FLTK_LIB_DIR ${FLTK_PATH}/lib)

######## Relighting #######
include_directories(${FLTK_INCLUDE_DIR})
include_directories(${OpenGL_INCLUDE_DIRS})
include_directories(${GLEE_PATH})
add_library(VirtualSurgeon_Relighting
	../HeadReplacement/glm.cpp
	../HeadReplacement/spherical_harmonics_analysis.cpp
	../HeadReplacement/LaplacianBlending.cpp
	../HeadReplacement/SpharmonicsUI.cpp
	../HeadReplacement/OGL_OCV_common.cpp
	)

Note that I had to resort to some very dark magic to recover the location of FLTK and GLEE… But it’s a jungle out there.

The source of the photograph is: http://www.flickr.com/photos/roel1943/309048020/
It is released under Creative Commons 2.0 ShareAlike-Attribution. So all the results here are also CC-2.0-SA-A…

Enjoy,
Roy.

From a template FireBreath plugin to an OpenGL plugin

FireBreath is kind of an amazing project. They aim to be able to write a single source that will create plugins for all browsers and all operating systems. A daunting feat by my book. But building a MacOS Safari/Firefox plugin using their framework proved very simple…
So I started here: http://www.firebreath.org/display/documentation/Mac+Video+Tutorial
It’s a video tutorial of how to create a plugin from template, build it, install it and run it. Follow their instructions and you’ll have your plugin ready in 10 minutes.
The next step will be to make our plugin display an OpenGL scene, which is what OpenNI/NITE use to display their depth map. This was also easy, borrowing code from the FireBreath OpenGL example.
However I ended up with a smaller source since I threw away most of the stuff…

class tutorialpluginMac : public tutorialplugin {
public:
    tutorialpluginMac();
	~tutorialpluginMac();

    BEGIN_PLUGIN_EVENT_MAP()
	EVENTTYPE_CASE(FB::AttachedEvent, onWindowAttached, FB::PluginWindowMac)
	EVENTTYPE_CASE(FB::DetachedEvent, onWindowDetached, FB::PluginWindowMac)
	PLUGIN_EVENT_MAP_CASCADE(tutorialplugin)
    END_PLUGIN_EVENT_MAP()

    virtual bool onWindowAttached(FB::AttachedEvent *evt, FB::PluginWindowMac*);
    virtual bool onWindowDetached(FB::DetachedEvent *evt, FB::PluginWindowMac*);
protected:

private:
    void* m_layer;

};

void glutDisplay (void); //this is implemented in the NITE code

@interface MyCAOpenGLLayer : CAOpenGLLayer {
    GLfloat m_angle;
}
@end

@implementation MyCAOpenGLLayer

- (id) init {
    if ([super init]) {
        m_angle = 0;
    }
    return self;
}

- (void)drawInCGLContext:(CGLContextObj)ctx pixelFormat:(CGLPixelFormatObj)pf forLayerTime:(CFTimeInterval)t displayTime:(const CVTimeStamp *)ts {
    //m_angle += 1;
    GLsizei width = CGRectGetWidth([self bounds]), height = CGRectGetHeight([self bounds]);
    GLfloat halfWidth = width / 2, halfHeight = height / 2;

    glViewport(0, 0, width, height);

	glutDisplay(); //let NITE draw it's stuff

    [super drawInCGLContext:ctx pixelFormat:pf forLayerTime:t displayTime:ts];
}

@end

tutorialpluginMac::tutorialpluginMac() : m_layer(NULL) {}

tutorialpluginMac::~tutorialpluginMac()
{
    if (m_layer) {
        [(CALayer*)m_layer removeFromSuperlayer];
        [(CALayer*)m_layer release];
        m_layer = NULL;
    }
}

bool tutorialpluginMac::onWindowAttached(FB::AttachedEvent* evt, FB::PluginWindowMac* wnd)
{
	cout << "tutorialpluginMac::onWindowAttached" << endl;
    if (FB::PluginWindowMac::DrawingModelCoreAnimation == wnd->getDrawingModel() ||
		FB::PluginWindowMac::DrawingModelInvalidatingCoreAnimation == wnd->getDrawingModel())
	{
        cout << " Setup CAOpenGL drawing. "<<endl;
        MyCAOpenGLLayer* layer = [MyCAOpenGLLayer new];
        layer.asynchronous = (FB::PluginWindowMac::DrawingModelInvalidatingCoreAnimation == wnd->getDrawingModel()) ? NO : YES;
        layer.autoresizingMask = kCALayerWidthSizable | kCALayerHeightSizable;
        layer.needsDisplayOnBoundsChange = YES;
        m_layer = layer;
        if (FB::PluginWindowMac::DrawingModelInvalidatingCoreAnimation == wnd->getDrawingModel())
            wnd->StartAutoInvalidate(1.0/30.0);
        [(CALayer*) wnd->getDrawingPrimitive() addSublayer:layer];
    }
    return tutorialplugin::onWindowAttached(evt,wnd);
}

bool tutorialpluginMac::onWindowDetached(FB::DetachedEvent* evt, FB::PluginWindowMac* wnd)
{
    return tutorialplugin::onWindowDetached(evt,wnd);
}

(You guys will have to fill in the gaps… includes, etc.)

This goes in a new file, a new subclass of the generic plugin, only for Mac. For windows, you should subclass again and create the OpenGL context using WIN32 API or equivalent.

CMakeLists.txt files are also affected. Check out the repo.

NITE OpenGL rendering

Now that the plugin will just draw whatever NITE is drawing, half the battle is done. So for the drawing code I took the simple NiPointViewer example from the NITE library (get it here).
But, since we have need no windows management in the OpenNI, again we can make everything more simple. I took the code exactly as it is, and changed really just a small bit.
I added

#undef USE_GLUT
#undef USE_GLES
, which pretty much makes that code compile to a very lean code (without window management etc.).
And I rescued the glOrtho call in glutDisplay()

//#ifdef USE_GLUT
glOrtho(0, mode.nXRes, mode.nYRes, 0, -1.0, 1.0);
#if defined(USE_GLES)
glOrthof(0, mode.nXRes, mode.nYRes, 0, -1.0, 1.0);
#endif


But the rest is pretty much identical.

One more thing, we should start the Kinect driver and OpenNI stack from somewhere in the plugin loading steps. In the main.cpp file from NITE I changed the main() function to kinect_main().
I did that by adding it here, in the generic plugin (not the Mac subclass because it should be called from all OSs):

bool tutorialplugin::onWindowAttached(FB::AttachedEvent *evt, FB::PluginWindow *)
{
    // The window is attached; act appropriately
	kinect_main(0, 0);
	cout << "tutorialplugin::onWindowAttached" << endl;
    return true;
}

It now will fire when a window is attached to the plugin. The OpenGL calls will start running after the OGL context is up and starts rendering in a loop.

Source and stuff

Get the source for the Kinect-FireBreath plugin at GitHub: https://github.com/royshil/KinectPlugin

This is how it looks:

Cool.
Roy

I would like to present something I have been working on recently, a work that immensely affect what I wrote in the blog in the past two years…

To use it:
Go on this page,
Watch the short instruction video,
download the application (MacOSX-Intel-x64 Win32)
and make yourself a model!
It takes just a couple of minutes and it’s very simple…

This work is an academic research project, Please please, take the time to fill out the survey! It is very short..
The results of the survey (the survey alone, no photos of your work) will possibly be published in an academic paper.

Note: No information is sent anywhere in any way outside of your machine (you may even unplug the network). All results are saved locally on your computer, and no inputs are recorded or transmitted. The application contains no malware. The source is available here.

Note II: All stock photos of models used in the application are released under Creative Commons By-NC-SA 2.0 license. Creator: http://www.flickr.com/photos/kk/. If you wish to distribute your results, they should also be released under a CC-By-NC-SA 2.0 license.

Thank you!
Roy.

#include "opencv2/opencv.hpp"

using namespace cv;

class LaplacianBlending {
private:
	Mat_<Vec3f> left;
	Mat_<Vec3f> right;
	Mat_<float> blendMask;

	vector<Mat_<Vec3f> > leftLapPyr,rightLapPyr,resultLapPyr;
	Mat leftSmallestLevel, rightSmallestLevel, resultSmallestLevel;
	vector<Mat_<Vec3f> > maskGaussianPyramid; //masks are 3-channels for easier multiplication with RGB

	int levels;

	void buildPyramids() {
		buildLaplacianPyramid(left,leftLapPyr,leftSmallestLevel);
		buildLaplacianPyramid(right,rightLapPyr,rightSmallestLevel);
		buildGaussianPyramid();
	}

	void buildGaussianPyramid() {
		assert(leftLapPyr.size()>0);

		maskGaussianPyramid.clear();
		Mat currentImg;
		cvtColor(blendMask, currentImg, CV_GRAY2BGR);
		maskGaussianPyramid.push_back(currentImg); //highest level

		currentImg = blendMask;
		for (int l=1; l<levels+1; l++) {
			Mat _down;
			if (leftLapPyr.size() > l) {
				pyrDown(currentImg, _down, leftLapPyr[l].size());
			} else {
				pyrDown(currentImg, _down, leftSmallestLevel.size()); //smallest level
			}

			Mat down;
			cvtColor(_down, down, CV_GRAY2BGR);
			maskGaussianPyramid.push_back(down);
			currentImg = _down;
		}
	}

	void buildLaplacianPyramid(const Mat& img, vector<Mat_<Vec3f> >& lapPyr, Mat& smallestLevel) {
		lapPyr.clear();
		Mat currentImg = img;
		for (int l=0; l<levels; l++) {
			Mat down,up;
			pyrDown(currentImg, down);
			pyrUp(down, up, currentImg.size());
			Mat lap = currentImg - up;
			lapPyr.push_back(lap);
			currentImg = down;
		}
		currentImg.copyTo(smallestLevel);
	}

	Mat_<Vec3f> reconstructImgFromLapPyramid() {
		Mat currentImg = resultSmallestLevel;
		for (int l=levels-1; l>=0; l--) {
			Mat up;

			pyrUp(currentImg, up, resultLapPyr[l].size());
			currentImg = up + resultLapPyr[l];
		}
		return currentImg;
	}

	void blendLapPyrs() {
		resultSmallestLevel = leftSmallestLevel.mul(maskGaussianPyramid.back()) +
									rightSmallestLevel.mul(Scalar(1.0,1.0,1.0) - maskGaussianPyramid.back());
		for (int l=0; l<levels; l++) {
			Mat A = leftLapPyr[l].mul(maskGaussianPyramid[l]);
			Mat antiMask = Scalar(1.0,1.0,1.0) - maskGaussianPyramid[l];
			Mat B = rightLapPyr[l].mul(antiMask);
			Mat_<Vec3f> blendedLevel = A + B;

			resultLapPyr.push_back(blendedLevel);
		}
	}

public:
	LaplacianBlending(const Mat_<Vec3f>& _left, const Mat_<Vec3f>& _right, const Mat_<float>& _blendMask, int _levels):
	left(_left),right(_right),blendMask(_blendMask),levels(_levels)
	{
		assert(_left.size() == _right.size());
		assert(_left.size() == _blendMask.size());
		buildPyramids();
		blendLapPyrs();
	};

	Mat_<Vec3f> blend() {
		return reconstructImgFromLapPyramid();
	}
};

Mat_<Vec3f> LaplacianBlend(const Mat_<Vec3f>& l, const Mat_<Vec3f>& r, const Mat_<float>& m) {
	LaplacianBlending lb(l,r,m,4);
	return lb.blend();
}

To use, simply call the function LaplacianBlend with your two images and your mask, and the result will be returned.
Here’s something I did with it:

Enjoy
Roy.

WOW That was annoying! From the moment I started using a custom ROM on my Samsung Galaxy S2 (Cognition S2, to be exact), I started having issues with some apps.

Sometimes they won’t appear in search results, other times it will just say it’s not compatible. It drove me nuts!

Naturally I started with the forums – All the solutions there were about LCD DPI change. Funny, I never thought of that, but it did make sense. The only problem was that I never touched my DPI settings (Heck, I didn’t know I could).

(On a side note – tampering with the density is pretty cool… if you want to try it, you can download apps from the Market like LCD Density)

Back to my problem…

At that point, I knew for sure it was a ROM issue. My friend, who bought the same phone in Thailand, was able to install all the ‘problematic’ apps, up until the point that I installed the same ROM on his machine. So, when in doubt – Check what the guys at the awesome CyanogenMod ROM did!

My experience showed me that a lot can happen by changing build.prop file (Guys.. Do it only if you have to! I am not responsible for any damage and ALWAYS BACKUP!).

Ok now that I got that out of my system, I pulled CyanogenMod’s build.prop file, and compared it to my device’s build.prop.

My hunch told me that it was the fingerprint section. It just made sense that if the device is sending an incorrect fingerprint to the Market, it will think some apps are not incompatible.

So… My hunch was right!

The original build.prop had those lines in:

 ro.build.description=GT-I9100-user 2.3.4 GINGERBREAD XXKH3 release-keys
ro.build.fingerprint=samsung/GT-I9100/GT-I9100:2.3.4/GINGERBREAD/XXKH3:user/release-keys

The changed buildprop had this data instead:

ro.build.description=GT-I9100-user 2.3.4 GINGERBREAD XXKG2 release-keys
ro.build.fingerprint=samsung/GT-I9100/GT-I9100:2.3.4/GINGERBREAD/XXKG2:user/release-keys

I really don’t know what causes the original data to be invalid to some apps, but it did! It appears XXKH3 is for some reason incompatible… It might be a bug on Google’s side, or there is some logic I don’t understand (feel free to enlighten me in the comments)

Keep in mind, that you need to compare the lines between two ROMs which are made for YOUR device. In this case it is the GT-I9100. Just download the right ROM from CyanogenMod, and see inside the zip, under “system” folder you can get the build.prop file. Then, compare it to the file you got from your own ROM

So, to make a long story short… If you want to do like me you should do the following:

1. Pull out /system/build.prop file from your device AND BACK IT UP
   
2. Change the fingerprint attributes
   
3. Remount your system as writable
   
4. Push the modified build.prop back in your device
   
5. Reboot
   

I hope this post is helpful to some of you!

Vertex shader

varying vec3 normal;

void main()
{
    gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
    normal = gl_NormalMatrix * gl_Normal;
}

Fragment shader

varying vec3 normal;

void main()
{
    vec3 normal_normal = normalize(normal);
	gl_FragColor = vec4(normal_normal, 1.0);
}

A technique to load the shaders that will save you a lot of headaches

GLvoid* my_program;

//Error-checking function
void checkARBError(GLvoid* obj) {
	char infolog[1024] = {0}; int _written = 0;
	glGetInfoLogARB(obj, 1024, &_written, infolog);
	if(_written>0) {
		cerr << infolog << endl;
	}
}	

bool notIsAscii(int i) { return !isascii(i); }

void init_shaders() {
	const GLubyte* lang_ver = glGetString(GL_SHADING_LANGUAGE_VERSION);
	cout <<"shading language version: "<<(uchar*)lang_ver<<endl;

	const char * my_fragment_shader_source;
	const char * my_vertex_shader_source;

        //Reading shaders from files
	ifstream ifs("vshader.txt");
	ostringstream ss; ss << ifs.rdbuf();
	ifstream ifs1("fshader.txt");
	ostringstream ss1; ss1 << ifs1.rdbuf();
	ifs.close(); ifs1.close();

        //Cleaning up the strings...
	string _vertex = ss.str(); _vertex.erase(remove_if(_vertex.begin(), _vertex.end(), notIsAscii), _vertex.end());
	string _frag = ss1.str(); _frag.erase(remove_if(_frag.begin(), _frag.end(), notIsAscii), _frag.end());

	// Get Vertex And Fragment Shader Sources
	my_fragment_shader_source = _frag.c_str();
	my_vertex_shader_source = _vertex.c_str();

        //DEBUG - can remove
	cout << "vertex shader:"<<endl<<my_vertex_shader_source<<endl;
	cout << "fragment shader:"<<endl<<my_fragment_shader_source<<endl;

	GLvoid* my_vertex_shader;
	GLvoid* my_fragment_shader;

	// Create Shader And Program Objects
	my_program = glCreateProgramObjectARB();
	my_vertex_shader = glCreateShaderObjectARB(GL_VERTEX_SHADER_ARB);
	my_fragment_shader = glCreateShaderObjectARB(GL_FRAGMENT_SHADER_ARB);

	// Load Shader Sources
	glShaderSourceARB(my_vertex_shader, 1, &my_vertex_shader_source, NULL);
	checkARBError(my_vertex_shader);
	glShaderSourceARB(my_fragment_shader, 1, &my_fragment_shader_source, NULL);
	checkARBError(my_fragment_shader);

	// Compile The Shaders
	glCompileShaderARB(my_vertex_shader);
	checkARBError(my_vertex_shader);
	glCompileShaderARB(my_fragment_shader);
	checkARBError(my_fragment_shader);

	// Attach The Shader Objects To The Program Object
	glAttachObjectARB(my_program, my_vertex_shader);
	glAttachObjectARB(my_program, my_fragment_shader);
	checkARBError(my_program);

	// Link The Program Object
	glLinkProgramARB(my_program);
	checkARBError(my_program);

}

I based it on this example from NeHe.

It does periodical error checking so you can see if something is wrong, plus it will make sure the vertex shader and fragmetn shader are stripped of all non-ASCII characters.
This way the compilation will not give you cryptic errors such as “ERROR: 0:1: ‘<‘ : syntax error syntax error”…

public class MyCanvasView extends View {
...
private Bitmap bmp;
private Lock bmpLock;
private Animator a;
private float scale = 1.0f;
private Rect clip;

@Override
protected void onDraw(Canvas canvas) {
   super.onDraw(canvas);
   ...
   clip = canvas.getClipBounds();
   w = clip.width();
   w2 = w/2.0f;
   h = clip.height();
   h2 = h/2.0f;

   bmpLock.lock();
   float left = scale*(w2 - (float)bmp.getWidth()/2.0f);
   float top = scale*(h2 - (float)bmp.getHeight()/2.0f);

   Bitmap _bmp = bmp;
   canvas.drawBitmap(_bmp, left, top, paint);
   bmpLock.unlock();
   ...
}
}

To drive the animation I privately subclass a Thread, that will push out new Bitmap objects (or at least write into the data of the private Bitmap object).

private class Animator extends Thread {
   private final MyCanvasView myCanvasView;
   private AssetManager assets;
   private int start = 0;
   private int end = 10;
   private boolean loop = false;

   public Animator(MyCanvasView myCanvasView)
   {
      this.myCanvasView = myCanvasView;
      assets = myCanvasView.getContext().getAssets();
   }

   @Override
   public void run() {
   super.run();

   try {
      do {
         for (int i = start; i <= end; i++) {
            boolean bitmapLoaded;
            bitmapLoaded = tryLoadBitmap("anim" + new DecimalFormat("0000").format(i) + ".png");
            if(!bitmapLoaded) { break; }
         }
      } while(loop);
   } catch (InterruptedException e) {
      e.printStackTrace();
   }
   }

   private boolean tryLoadBitmap(String bmpFilename) throws InterruptedException {
      try {
         bmpLock.lock();
         bmp = BitmapFactory.decodeStream(assets.open(bmpFilename));
         bmpLock.unlock();
         if(bmp == null) {
            AlertDialog a = new AlertDialog.Builder(getContext()).create();
            a.setMessage("Cannot load image");
            a.show();
            return false;
         }
         myCanvasView.postInvalidate();
         sleep(25);
      } catch (IOException e) {
         e.printStackTrace();
      }
      return true;
   }
}

Notice the usage of the Lock bmpLock object to prevent racing conditions on the drawing and loading of new Bitmaps.
Also, you should probably want a mechanism to determine start, end and loop parameters. In the complete source you can see what I have done to address that.

Finally, a way to fire an animation on MyCanvasView:

public void fireAnimation() {
   if(a != null && a.isAlive()) {
      a.interrupt();
      a = null;
   }
   a = new Animator(this);
   a.start(); //TODO: reuse the object
}

This runs in excellent frame rates, and it’s not even fully optimized. So I guess the approach is correct for the Android environment. I did not, however, test this with many animators on the same layout.

Grab the complete source (including some stuff I didn’t discuss here like rotating and scaling the “Sprite”):

This is part of the PoCoMo project I have been working on in our lab.

Enjoy!
Roy.

OK let’s get down to business.

I followed a very simple technique described in this paper. I know, you say, “A Paper? Really? I’m not gonna read that technical boring stuff, give the bottom line! man.. geez.” Well, you are right, except that this paper IS the bottom line, it’s dead simple. It’s almost a tutorial. It is also referenced by the OpenCV documentation.

The method is simple:
- Extract features of choice from training set that contains all classes.
- Create a vocabulary of features by clustering the features (kNN, etc). Let’s say 1000 features long.
- Train your classifiers (SVMs, Naive-Bayes, boosting, etc) on training set again (preferably a different one), this time check the features in the image for their closest clusters in the vocabulary. Create a histogram of responses for each image to words in the vocabulary, it will be a 1000-entries long vector. Create a sample-label dataset for the training.
- When you get an image you havn’t seen – run the classifier and it should, god willing, give you the right class.

Turns out, those crafty guys in WillowGarage have done pretty much all the heavy lifting, so it’s up for us to pick the fruit of their hard work. OpenCV 2.3 comes packed with a set of classes, whose names start with BOW for Bag Of Words, that help a lot with implementing this method.

Starting with the first step:

Mat training_descriptors(1,extractor->descriptorSize(),extractor->descriptorType());

SurfFeatureDetector detector(400);
vector keypoints;

// computing descriptors
Ptr extractor(
   new OpponentColorDescriptorExtractor(
      Ptr(new SurfDescriptorExtractor())
   )
);

while(..loop a directory? a file?..) {
   Mat img = imread(filepath);
   detector.detect(img, keypoints);
   extractor->compute(img, keypoints, descriptors);
   training_descriptors.push_back(descriptors);
}

Simple!
Let’s go create a vocabulary then. Luckily, OpenCV has taken care of that, and provide a simple API:

BOWKMeansTrainer bowtrainer(1000); //num clusters
bowtrainer.add(training_descriptors);
Mat vocabulary = bowtrainer.cluster();

Boom. Vocabulary.
Now, let’s train us some SVM classifiers!
We’re gonna train a 2-class SVM, in a 1-vs-all kind of way. Meaning we train an SVM that can say “yes” or “no” when choosing between one class and the rest of the classes, hence 1-vs-all.
But first, we need to scour the training set for our histograms (the responses to the vocabulary, remember?):

vector<KeyPoint> keypoints;
Mat response_hist;
Mat img;
string filepath;
map<string,Mat> classes_training_data;

Ptr<FeatureDetector > detector(new SurfFeatureDetector());
Ptr<DescriptorMatcher > matcher(new BruteForceMatcher<L2<float> >());
Ptr<DescriptorExtractor > extractor(new OpponentColorDescriptorExtractor(Ptr<DescriptorExtractor>(new SurfDescriptorExtractor())));
Ptr<BOWImgDescriptorExtractor> bowide(new BOWImgDescriptorExtractor(extractor,matcher));
bowide->setVocabulary(vocabulary);

#pragma omp parallel for schedule(dynamic,3)
for(..loop a directory?..) {
   img = imread(filepath);
   detector->detect(img,keypoints);
   bowide.compute(img, keypoints, response_hist);

   #pragma omp critical
   {
      if(classes_training_data.count(class_) == 0) { //not yet created...
         classes_training_data[class_].create(0,response_hist.cols,response_hist.type());
         classes_names.push_back(class_);
      }
      classes_training_data[class_].push_back(response_hist);
   }
   total_samples++;
}

Now, two things:
First notice I’m keeping the training data for each class separately, this is because we will need this for later creating the 1-vs-all samples-labels matrices.
Second, I use OpenMP multi(-threading)processing to make the calculation parallel, and hence faster, on multi-core machines (like the one I used). Time is sliced by a whole lot. OpenMP is a gem, use it more. Just a couple of #pragma directives and you’re multi-threading.

Alright, data gotten, let’s get training:

#pragma omp parallel for schedule(dynamic)
for (int i=0;i<classes_names.size();i++) {
   string class_ = classes_names[i];
   cout << omp_get_thread_num() << " training class: " << class_ << ".." << endl;

   Mat samples(0,response_cols,response_type);
   Mat labels(0,1,CV_32FC1);

   //copy class samples and label
   cout << "adding " << classes_training_data[class_].rows << " positive" << endl;
   samples.push_back(classes_training_data[class_]);
   Mat class_label = Mat::ones(classes_training_data[class_].rows, 1, CV_32FC1);
   labels.push_back(class_label);

   //copy rest samples and label
   for (map<string,Mat>::iterator it1 = classes_training_data.begin(); it1 != classes_training_data.end(); ++it1) {
      string not_class_ = (*it1).first;
      if(not_class_.compare(class_)==0) continue; //skip class itself
      samples.push_back(classes_training_data[not_class_]);
      class_label = Mat::zeros(classes_training_data[not_class_].rows, 1, CV_32FC1);
      labels.push_back(class_label);
   }

   cout << "Train.." << endl;
   Mat samples_32f; samples.convertTo(samples_32f, CV_32F);
   if(samples.rows == 0) continue; //phantom class?!
   CvSVM classifier;
   classifier.train(samples_32f,labels);

   //do something with the classifier, like saving it to file
}

Again, I parallelize, although the process is not too slow.
Note how I build the samples and the labels, where each time I put in the positive samples and mark the labels ’1′, and then I put the rest of the samples and label them ’0′.

Moving on to …. testing the classifiers!
Nothing seems to me like more fun than creating a confusion matrix! Not really, but let’s see how it’s done:

map<string,map<string,int> > confusion_matrix; // confusionMatrix[classA][classB] = number_of_times_A_voted_for_B;
map<string,CvSVM> classes_classifiers; //This we created earlier

vector<string> files; //load up with images
vector<string> classes; //load up with the respective classes

for(..loop over a directory?..) {
   Mat img = imread(files[i]),resposne_hist;

   vector<KeyPoint> keypoints;
   detector->detect(img,keypoints);
   bowide->compute(img, keypoints, response_hist);

   float minf = FLT_MAX; string minclass;
   for (map<string,CvSVM>::iterator it = classes_classifiers.begin(); it != classes_classifiers.end(); ++it) {
      float res = (*it).second.predict(response_hist,true);
      if (res < minf) {
         minf = res;
         minclass = (*it).first;
      }
   }
   confusion_matrix[minclass][classes[i]]++;
}

When you take a look in my files, you will find a much complicated way of doing this. But this is the core idea – look in the image for the response histogram to the vocabulary of features (rather, feature-cluster-ceneters), run it by all the classifiers and take the one with the best score. Simple.
Consider making this parallel as well. No reason for it to be serial.

That’s about covers it.

Code

Lately I’m pushing stuff in Github.com using git rather than SVN on googlecode. Donno why, it’s just like that.
Get the whole thing at:
https://github.com/royshil/FoodcamClassifier

Follow the build instructions, they’re a breeze, and then follow the runnning instructions. It’s basically a series of command-line programs you run to get through each step, and in the end you have like a “predictor” service that takes an image and produces a prediction.

OK guys, have fun classifying stuff!
Roy.