Been working hard at a project for school the past month, implementing one of the more interesting works I’ve seen in the AR arena: Parallel Tracking and Mapping (PTAM) [PDF]. This is a work by George Klein [homepage] and David Murray from Oxford university, presented in ISMAR 2007.

When I first saw it on youtube [link] I immediately saw the immense potential – mobile markerless augmented reality. I thought I should get to know this work a bit more closely, so I chose to implement it as a part of advanced computer vision course, given by Dr. Lior Wolf [link] at TAU.

The work is very extensive, and clearly is a result of deep research in the field, so I set to achieve a few selected features: Stereo initialization, Tracking, and small map upkeeping. I chose not to implement relocalization and full map handling.

This post is kind of a tutorial for 3D reconstruction with OpenCV 2.0. I will show practical use of the functions in cvtriangulation.cpp, which are not documented and in fact incomplete. Furthermore I’ll show how to easily combine OpenCV and OpenGL for 3D augmentations, a thing which is only briefly described in the docs or online.

Here are the step I took and things I learned in the process of implementing the work.

Preparations…

Before going straight to coding, I had to prepare a few things.

• A working compilation of OpenCV – not trivial with the new version 2.0.
• A calibrated camera.
• Test data

Compiling OpenCV 2.0 proved to be a bit tricky. Even though the sourceforge project offers binary release for Win32, I compiled the whole thing from source. It turned out the binary release doesn’t contain .lib files, and anyway has compatibility issues between MS VS 2005 and 2008 – something about the embedded manifest [google]. I downloaded the freshest source from SVN, and compiled it, but it didn’t solve the debug-release problem, so I was left with using the release dlls even for debug evironment.

Initially I thought I’ll try an uncalibrated camera approach, but soon abandoned it. I had to calibrate my cameras, which I did  very easily using OpenCV’s “calibration.cpp”, which strangely is not built when building all examples – it has to be built manually. But everything went smoothly, and I soon got a calibration matrix (focal length, center of projection) and radial distortion coefficients.

Getting Test Data

For the test data I wanted to get a few views of a planar scene, where the first two views are separated only by a translation of ~5cm, as K&M do in the PTAM article. This known translation is helpful when trying to triangulate the initial features in the scene. When you have prior knowledge of where the cameras are, you can simply intersect the epipolar lines between the two views and recover the 3D position of the points – up to a scale. Keep in mind you must also have feature correspondence: a point on image A must be correlated to a point in image B.

To achieve this I set up a small program that uses Optical Flow to track some 2D features in the scene, and grab a few screens + feature vectors. See ‘capture_data.cpp’.

Stereo Initialization

Now that I have 2 views with feature correspodence:

I would like to triangulate the features. This is possible, as I discussed earlier, since I know the rotation (none), translation (5cm on -x axis) and camera calibration parameters (focal length, center of projection).

Triangulation

For triangulation, OpenCV has only recently added a couple of functions that implement triangulation [link] as shown by Hartly & Zisserman [PDF, page 12]. However, these functions are not formally documented, and in fact they are missing some important parts. This is how I used cvTriangulation(), which is the key function:

//this function will initialize the 3D features from two views
void stereoInit() {

//first load camera intrinsic parameters
FileStorage fs("cam_work.out",CV_STORAGE_READ);
FileNode fn = fs["camera_matrix"];
camera_matrix = Mat((CvMat*)fn.readObj(),true);

fn = fs["distortion_coefficients"];
distortion_coefficients = Mat((CvMat*)fn.readObj(),true);

//vector<Point2d> points[2]; //these Point2d vectors hold the 2D features, double precision, from the 2 views

//get copy of points
_points[0] = points[0];
_points[1] = points[1];
Mat pts1M(_points[0]), pts2M(_points[1]); //very easy in OpenCV 2.0 to convert vector<> to Mat.

//Undistort points
Mat tmp,tmpOut;
pts1M.convertTo(tmp,CV_32FC2);  //undistort takes only floats not doubles, so convert to Point2f
undistortPoints(tmp,tmpOut,camera_matrix,distortion_coefficients);
tmpOut.convertTo(pts1M,CV_64FC2);  //go back to double precision

pts2M.convertTo(tmp,CV_32FC2);
undistortPoints(tmp,tmpOut,camera_matrix,distortion_coefficients);
tmpOut.convertTo(pts2M,CV_64FC2);

vector<uchar> tri_status; //this will hold the status for each point, a good point will have 1, bad - 0

//now triangulate
triangulate(_points[0],_points[1],tri_status);

}

void triangulate(vector<Point2d>& points1, vector<Point2d>& points2, vector<uchar>& status) {

	//Convert points to 1-channel, 2-rows, double precision - This is important - see the code
...

	Mat ___tmp(2,pts1Mt.cols,CV_64FC1,__d);
...
	Mat ___tmp1(2,pts2Mt.cols,CV_64FC1,__d1);
...

	CvMat __points1 = ___tmp, __points2 = ___tmp1;

	//projection matrices
	double P1d[12] = {	-1,0,0,0,
						0,1,0,0,
						0,0,1,0 };	//Identity, but looking into -z axis
	Mat P1m(3,4,CV_64FC1,P1d);
	CvMat* P1 = &(CvMat)P1m;
	double P2d[12] = {	-1,0,0,-5,
						0,1,0,0,
						0,0,1,0 };  //Identity rotation, 5cm -x translation, looking into -z axis
	Mat P2m(3,4,CV_64FC1,P2d);
	CvMat* P2 = &(CvMat)P2m;

	float _d[1000] = {0.0f};
	Mat outTM(4,points1.size(),CV_32FC1,_d);
	CvMat* out = &(CvMat)outTM;

//using cvTriangulate with the created structures
	cvTriangulatePoints(P1,P2,&__points1,&__points2,out);

//we should check the triangulation result by reprojecting 3D->2D and checking distance
	vector<Point2d> projPoints[2] = {points1,points2};

	double point2D_dat[3] = {0};
	double point3D_dat[4] = {0};
	Mat twoD(3,1,CV_64FC1,point2D_dat);
	Mat threeD(4,1,CV_64FC1,point3D_dat);

	Mat P[2] = {Mat(P1),Mat(P2)};

	int oc = out->cols, oc2 = out->cols*2, oc3 = out->cols*3;

	status = vector<uchar>(oc);

	//scan all points, reproject 3D->2D, and keep only good ones
	for(int i=0;i<oc;i++) {
		double W = out->data.fl[i+oc3];
        point3D_dat[0] = out->data.fl[i] / W;
        point3D_dat[1] = out->data.fl[i+oc] / W;
        point3D_dat[2] = out->data.fl[i+oc2] / W;
        point3D_dat[3] = 1;

        bool push = true;
        /* !!! Project this point for each camera */
        for( int currCamera = 0; currCamera < 2; currCamera++ )
        {
            //reproject! using the P matrix of the current camera
			twoD = P[currCamera] * threeD;

            float x,y;
            float xr,yr,wr;
 	x = (float)projPoints[currCamera][i].x;
	y = (float)projPoints[currCamera][i].y;

            wr = (float)point2D_dat[2];
            xr = (float)(point2D_dat[0]/wr);
            yr = (float)(point2D_dat[1]/wr);

            float deltaX,deltaY;
            deltaX = (float)fabs(x-xr);
            deltaY = (float)fabs(y-yr);

			//printf("error from cam %d (%.2f,%.2f): %.6f %.6f\n",currCamera,x,y,deltaX,deltaY);

			if(deltaX > 0.01 || deltaY > 0.01) {
				push = false;
			}
        }
		if(push) {
			// A good 3D reconstructed point, add to known world points

			double s = 7;
			Point3d p3d(point3D_dat[0]/s,point3D_dat[1]/s,point3D_dat[2]/s);
			//printf("%.3f %.3f %.3f\n",p3d.x,p3d.y,p3d.z);
			points1Proj.push_back(p3d);
			status[i] = 1;
		} else {
			status[i] = 0;
		}

	}
}

OK, now that I have (hopefully) triangulated 3D features from the initial state: 2 views of a planar scene with 5cm translation on the X axis – I can move on the pose estimation.

Pose Estimation

Theoretically, if I know the 3D position of features in the world and their respective 2D position in the image, it should be easy to recover the position of the camera, because there are a rotation matrix and translation vector that define this transformation. Practically in OpenCV, finding the position of an object using 3D-2D correlation is done by using the solvePnP() [link] function.

Since I have an initial guess of the rotation and translation – from the first 2 frames – I can “help” the function estimate the new ones.

void findExtrinsics(vector<Point2d>& points, vector<double>& rv, vector<double>& tv) {
	//estimate extrinsics for these points

	Mat rvec(rv),tvec(tv);

//initial "guess", in case it wasn't already supplied
	if(rv.size()!=3) {
		rv = vector<double>(3);
		rvec = Mat(rv);
		double _d[9] = {1,0,0,
						0,-1,0,
						0,0,-1};
		Rodrigues(Mat(3,3,CV_64FC1,_d),rvec);
	}
	if(tv.size()!=3) {
		tv = vector<double>(3);
		tv[0]=0;tv[1]=0;tv[2]=0;
		tvec = Mat(tv);
	}

	//create a float rep  of points
	vector<Point2f> v2(points.size());
	Mat tmpOut(v2);
	Mat _tmpOut(points);
	_tmpOut.convertTo(tmpOut,CV_32FC2);

	solvePnP(points1projMF,tmpOut,camera_matrix,distortion_coefficients,rvec,tvec,true);

	printf("frame extrinsic:\nrvec: %.3f %.3f %.3f\ntvec: %.3f %.3f %.3f\n",rv[0],rv[1],rv[2],tv[0],tv[1],tv[2]);

//the output of the function is a Rodrigues form of rotation, so convert to regular rot-matrix
	Mat rotM(3,3,CV_64FC1); ///,_r);
	Rodrigues(rvec,rotM);
	double* _r = rotM.ptr<double>();
	printf("rotation mat: \n %.3f %.3f %.3f\n%.3f %.3f %.3f\n%.3f %.3f %.3f\n",
		_r[0],_r[1],_r[2],_r[3],_r[4],_r[5],_r[6],_r[7],_r[8]);
}

After getting the extrinsic parameters of the camera, the next step is plugging in the visualization!

Integrating OpenGL

Generally, it should be possible to create a 3D scene that matches exactly the true world scene, where the triangulated features appear in the scene aligned exactly with the world. I was not able to achieve that, but I got pretty close:

It’s basically what you do in augmented reality, you align the virtual camera’s position and rotation with the results you get from the vision part of the system. In the pose estimation we ended with a 3D rotation vector (Rodrigues form) and 3D translation vector which is used as-is, so only the rotation vector should be converted to 3×3 matrix using the Rodrigues() function.

This is the OpenGL glut display() function that draws the scene:

void display(void)
{
	glClearColor(1.0f, 1.0f, 1.0f, 0.5f);
	glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);	// Clear Screen And Depth Buffer

	//draw the background - the frame from the camers
	glMatrixMode(GL_PROJECTION);
	glPushMatrix();
	gluOrtho2D(0.0,352.0,288.0,0.0);
	glMatrixMode(GL_MODELVIEW);
	glPushMatrix();
	glDisable(GL_DEPTH_TEST);
	glDrawPixels(352,288,GL_RGB,GL_UNSIGNED_BYTE,backPxls.data);
	glEnable(GL_DEPTH_TEST);
	glPopMatrix();
	glMatrixMode(GL_PROJECTION);
	glPopMatrix();

    const double t = glutGet(GLUT_ELAPSED_TIME) / 1000.0;
	a = t*20.0;

	glMatrixMode(GL_MODELVIEW);
	glLoadIdentity();

//use the camera position 3D vector
	curCam[0] = cam[0]; curCam[1] = cam[1]; curCam[2] = cam[2];
//there seems to be some kind of offset...
	glTranslated(-curCam[0]+0.5,-curCam[1]+0.7,-curCam[2]);

//and the 3x3 rotation matrix
	double _d[16] = {	rot[0],rot[1],rot[2],0,
						rot[3],rot[4],rot[5],0,
						rot[6],rot[7],rot[8],0,
						0,	   0,	  0		,1};
	glMultMatrixd(_d);

//flip the rotation on the x-axis
	glRotated(180,1,0,0);

	//draw the 3D feature points
	glPushMatrix();
	glColor4d(1.0,0.0,0.0,1.0);
	for(unsigned int i=0;i<points1Proj.size();i++) {
		glPushMatrix();
glTranslated(points1Proj[i].x,points1Proj[i].y,points1Proj[i].z);
		glutSolidSphere(0.03,15,15);
		glPopMatrix();
	}
	glPopMatrix();

	glutSwapBuffers();

	if(!running) {
		glutLeaveMainLoop();
	}

	Sleep(25);
}

This pretty much coveres my work, in a very concise way. The complete source code will reveal all I have done, and will provide a better copy-and-paste ground for your own projects.

Things not covered in this work

Initially I tried to implement a very crucial part of the PTAM work – pairing the 3D map with 2D features in the image. This allows them to re-align the map in every frame (when the tracking is bad) so the pose estimation does not “loose grip”. In essence, they keep a visual identity for each map feature, very similar to a descriptor like SURF or SIFT, so at any point they can find where in the new image are the features and recover the camera pose from the 2D-3D correspondence. I ran into a problem utilizing OpenCV’s SURF functionality, it seems to have a bug when trying to compute the descriptor for user-given feature points.

Another thing I chose not to implement is creating a full map of the surroundings. I wanted to achieve a simple working solution for a small map (essentially a single frame), and see how it works. In the original work by K&M they constantly add more and more features to the map untill it has covered the whole surrounding room.

Code

As usual my code is available for checkout from the blog’s SNV repo:

svn checkout http://morethantechnical.googlecode.com/svn/trunk/ptam ptam

Thanks Lior for your help getting the hang of these subjects, and the opportunity to meddle with a subject I long gone wanted to explore.

I hope everyone will enjoy and learn from my enjoyment and learning.

Bye!

Roy.

It looks like it’s finally here – a way to grab the raw data of the camera frames on the iPhone OS 3.x.

A gifted hacker named John DeWeese was nice enough to comment on a post from May 09′ with his method of hacking the APIs to get the frames. Though cumbersome, it looks like it should work, but I haven’t tried it yet. I promise to try it soon and share my results.

Way to go John!
Some code would be awesome…

Roy.

Here are the steps we took to bring this project to life.

The Idea

Our vision is to create a home monitoring and controlling system, that will enable tracking different sensors around the house and also control switches.The system should be centralized by a master controller, and also wireless so it will not need cumbersome wiring throughout the house. We would also like the system to be easily controlled by a PC, so visual information could be displayed to the user, as well as allow manual control of the electronic switches.

Such a system will be able to automatically:

• Turn off the garden lighting when the light outside is bright enough,
• Control the lawn watering system,
• Control air conditioning in the house according to the temperature,
• etc.

The Hardware

As I mentioned, we were given an LPC2148 education board [link], implemented by Embedded Artists, that boasts a 12Mhz ARM CPU, and many peripheral subsystems. Among the systems are: LCD screen, numerous LEDs, a LED matrix, a fan, analog dials, USB and UART I/O and more. The boards also has a port for connecting a ZigBee module to enable RF communication, but it doesn’t contain the actual module. Since we needed remote communication between our stations, we bought 3 XBee modules from Maxstream [link].

The Software

The LPC boards can be programmed very easily using tools provided by Embedded Artists, such as GCC with Newlib for compiling (both Win and Linux), and Flashmagic (for windows) or lpc21isp for loading the compiled program. We used these tools for programing the “embedded” part of the application, for the PC client we used Java with SWT and serial port connectivity (RxTx for Windows and Linux).

Embedded program

The embedded program on the LPC board has two parts: A master and A client. The system has only one master, and up to 3 clients. The master gathers information from the clients, and controls their behavior according to the user requests. For communication between the master and clients we created a communication protocol that has only a few simple messages:

• INIT – The master sends this request to initialize the connection between itself and a client.
• ACK – This is the response for every request.
• POLL – The clients answers this request with the data from all it’s sensors.
• OPERATE – The master commands the client to switch something on or off.
• ERROR – A general error response.

All commands / responses have a unified structure described here:

General struct:
 ___________________________________________ _____
|    OP   | To | From | <----- DATA ------> | EOM
|_________|____|______|_____________________|_____

Data:

POLL (response)
_ ____________ ______ ______ ________
 | Temperature| ADC0 | ADC1 | Button |
_|____________|______|______|________|

OPERATE
_ _________
 | BITMASK |
_|_________|

This way we had a very easy implementation of the communication module, since we always had to look for only 14 bytes on the wire.

Communication with ZigBee module

To jumpstart our implementation we used the examples provided by Embedded Artists for operating the ZigBee module [link]. The communication with the ZigBee module is on the UART1 port of the MCU. The example code takes care of opening the correct GPIO pins, setting the IRQ masks to enable interrupts and provides a very simple API for transmitting and receiving characters with the XBee over UART  (described in uart.h and uart.c files).

We took the XBee example code and expanded it to be able to recieve and send data between two stations. That means setting up each station’s module with it’s ID, address, channel and target address by AT commands [spec, see page. 28]: ATID, ATCH, ATMY and ATDL. Two other key features are putting the module into command mode (rather than transmit mode) to set the mentioned parameters, this is done by ‘+++’ to enter command mode and ATCN to exit. Once we had a decent framework to communicate between the stations, we started to build the logic.

The master station logic is like so:

1. Initialize XBee module.
2. INIT all client stations, and see which station answers – these will be our “up” stations.
3. Loop:
   1. POLL each “up” station in a loop.
   2. Take care of any OPERATE requests from the PC client.

This way, the client’s logic boils down to just looping, testing for any request and taking care of it. Both client and master share most of the code, so only the main process code is essentially different. The example code uses a framework called “Preemtive OS” to allow multitasking / processes (code was bundled in the examples).

Communication with PC

Communication between the master and PC client also required some lightweight “protocol”. The communication is again over UART (0 this time, 1 is used by XBee), only now the PC is doing a UART-over-USB with the board. The protocol we ended up with supports these features:

• Commands the PC wants the master to perform look like “m=<command>=<parameters>”, such commands can be:
  • “poll=<i>”, send a POLL to the station indexed i
  • “test=<i>”, send an INIT to the station indexed i
  • “toggle=<i>_<sw>_<onoff>”, set the switch sw on the station indexed i to onoff status
• Notifications the master would like to share with the user (PC) look like “pc=<notification>”, such notifications can be:
  • “up=<i>”, the station indexed i is up
  • “master_online”,
  • “temp=<i> <temp>”, the station indexed i has a temperature reading of temp
  • “e=<error message>”, error message from the master

This is an incomplete set of features, but it’s representative of the idea we want to present.

PC client program

The PC client we built using Java, so it would (and should) be portable between OSs. The GUI was built with the SWT, using the eclipse Visual Editor plug-in which simplified the process. Communication with the LPC board is done over the serial port of the board, as I mentioned the PC creates a virtual COM port using a USB-to-UART driver (FTDI, bundled with windows) [link].

To test the GUI we created a “simulator”, that mimics the operation of a master station, that is shown in the video above.

The high-level design of the program is described in this inheritance UML diagram:

Our PC client uses serial-port connectivity based on the old javax.serial APIs. Though these APIs have been abandoned by Sun, and there is no official Win32 implementation bundled with the JDK (only for *NIXs/Solaris), a project named RxTx is upkeeping an implementation of this API for windows [link].

The Code

We are releasing the code under the BSD license, for everyone to use, enjoy, learn and expand.

It is available via the blog’s SVN repo:

PC Client (requires RxTx serial port impl and SWT)

svn checkout http://morethantechnical.googlecode.com/svn/trunk/SmartHomePCClient

Embedded program (includes all dependencies)

svn checkout http://morethantechnical.googlecode.com/svn/trunk/smarthome_embedded/final project

To compile the master program “make” in the master directory, and you’ll get an “xbee_master.hex” file, same goes for the client in the client directory (these directories don’t contain code, only a makefile). Then you have to upload the hex into the board, this is done either by “make deploy” (if you have lpc21isp), or FlashMagic on Win.

Java is compiled as usual… just remember the dependencies.

We would like to thank Sivan Toledo for the guidance, loaned hardware and inspiration.

And thanks all for listening!
Roy Shilkrot & Gil Ramon

It can also work for you, in case your IT does not block by protocol, only by address.

So here’s how to do it:

STEP 1 – Retrieve the actual link for your radio station

Go to the station’s website and try to get a link that you can play in Windows Media Center.
Some stations will let you find it, others won’t.
If this is the case, here’s what you need to do (At home!)

1. Download wireshark and install it
2. Start capturing using this filter: rtsp.request
3. Try to listen to the station
4. You should see such an output in wireshark:

   

   As you can see, there is a url here that looks like this:
   rtsp://someurl/somefile RTSP/1.0

5. Take this URL, and replace RTSP with MMS and omit the RTSP/1.0 ending.
   Your URL should look like this: mms://someurl.somefile
6. Try to play this URK in windows media player. If it works – you’re half way through

If you can get the link in any other way, it’s fine… This is only a suggestion

STEP 2 – Set up VLC as a stream server at your home

I will start with an apology. I didn’t find a proper way to do it with the GUI of version 1.0+.
On the past versions it was relatively clear, so that’s where I managed to find the command line you need to run in order for it to work
So, at this point, you need to run VLC on your PC with these parameters:

vlc mms://yourserver/yourlink :sout=#transcode{acodec=mp3,ab=64,channels=2}:duplicate{dst=std{access=mmsh,mux=asfh,dst=:8080}}

This will stream the web-radio station to port 8080.
You can test this using Windows Media Player, just open it up, and open the url mms://localhost:8080

Another thing you need to keep in mind, is that you should establish port forwarding if you are behind NAT.

 That’s the whole deal… Of course you can improve it by creating a script that will run it, or add the http interface of VLC to remote control it on another port, but I don’t want to overload this guide…

keep it simple!

I wanted to do the simplest recoloring/color-transfer I could find – and the internet is just a bust. Nothing free, good and usable available online… So I implemented the simplest color transfer algorithm in the wolrd – Histogram Matching.

Here’s the implementation with OpenCV

// Compute histogram and CDF for an image with mask
void do1ChnHist(const Mat& _i, const Mat& mask, double* h, double* cdf) {
Mat _t = _i.reshape(1,1);
Mat _tm;
mask.copyTo(_tm);
_tm = _tm.reshape(1,1);
for(int p=0;p<_t.cols;p++) {
if(_tm.at(0,p) > 0) {
uchar c = _t.at(0,p);
h[c] += 1.0;
}
}

//normalize hist
Mat _tmp(1,256,CV_64FC1,h);
double minVal,maxVal;
minMaxLoc(_tmp,&minVal,&maxVal);
_tmp = _tmp / maxVal;

cdf[0] = h[0];
for(int j=1;j<256;j++) {
cdf[j] = cdf[j-1]+h[j];
}

//normalize CDF
_tmp.data = (uchar*)cdf;
minMaxLoc(_tmp,&minVal,&maxVal);
_tmp = _tmp / maxVal;
}

// match histograms of 'src' to that of 'dst', according to both masks
void histMatchRGB(Mat& src, const Mat& src_mask, const Mat& dst, const Mat& dst_mask) {
#ifdef BTM_DEBUG
	namedWindow("original source",CV_WINDOW_AUTOSIZE);
	imshow("original source",src);
	namedWindow("original query",CV_WINDOW_AUTOSIZE);
	imshow("original query",dst);
#endif

	vector<Mat> chns;
	split(src,chns);
	vector<Mat> chns1;
	split(dst,chns1);
	Mat src_hist = Mat::zeros(1,256,CV_64FC1);
	Mat dst_hist = Mat::zeros(1,256,CV_64FC1);
	Mat src_cdf = Mat::zeros(1,256,CV_64FC1);
	Mat dst_cdf = Mat::zeros(1,256,CV_64FC1);
	Mat Mv(1,256,CV_8UC1);
	uchar* M = Mv.ptr<uchar>();

	for(int i=0;i<3;i++) {
		src_hist.setTo(cvScalar(0));
		dst_hist.setTo(cvScalar(0));
		src_cdf.setTo(cvScalar(0));
		src_cdf.setTo(cvScalar(0));

		do1ChnHist(chns[i],src_mask,src_hist,src_cdf);
		do1ChnHist(chns1[i],dst_mask,dst_hist,dst_cdf);

		uchar last = 0;
		double* _src_cdf = src_cdf.ptr<double>();
		double* _dst_cdf = dst_cdf.ptr<double>();

		for(int j=0;j<src_cdf.cols;j++) {
			double F1j = _src_cdf[j];

			for(uchar k = last; k<dst_cdf.cols; k++) {
				double F2k = _dst_cdf[k];
				if(abs(F2k - F1j) < HISTMATCH_EPSILON || F2k > F1j) {
					M[j] = k;
					last = k;
					break;
				}
			}
		}

		Mat lut(1,256,CV_8UC1,M);
		LUT(chns[i],lut,chns[i]);
	}

	Mat res;
	merge(chns,res);

#ifdef BTM_DEBUG
	namedWindow("matched",CV_WINDOW_AUTOSIZE);
	imshow("matched",res);

	waitKey(BTM_WAIT_TIME);
#endif

	res.copyTo(src);
}

Edit 31/1/10: nicer code in histMatch, better memory consumption.

The code is fairly simple.. I started using the OpenCV 2.0 C++ API and I must say it’s great, not having to worry about silly memory managment, everything is instanced on the stack and get’s released automatically.

Just a quick overview:

• Compute histogram and comulative distibution function (CDF) for each image.
• Create the lookup table (LUT) from one CDF to the other
• Apply the LUT to the image

External usage is simple

Mat src=cvLoadImage("image008.jpg");
Mat dst=cvLoadImage("image003.jpg");
Mat src_mask = cvLoadImage("image008_mask.bmp",0);
Mat dst_mask = cvLoadImage("image003_mask.bmp",0);

histMatchRGB(dst,dst_mask,src,src_mask);

Enjoy!
Roy.

http://www.engadget.com/2010/01/18/misa-digital-guitar-cuts-the-strings-brings-the-noise/
Very nice! A digital guitar…

http://www.newscientist.com/article/dn18036
An interesting concept – see-through walls w/ augmentd reality

http://gizmodo.com/5452140/one-third-of-us-11+year+olds-have-cellphones
The “Youth market”’s little brother – the “Toddler market” – is booming

http://gizmodo.com/5451876/rumor-apple-iphone-os-40-features-detailed
Some goodies from iPhone OS 4 – where is video-pixel-bytes access already?!

http://lifehacker.com/5452786/memorize-now-helps-you-commit-long-passages-to-memory
I like! A helper webapp to memorize text

http://gizmodo.com/5452684/voice-band-iphone-app-converts-bah-ba-ba-bah-into—
This is awesome.

http://gizmodo.com/5453436/googles-html5-youtube-videos-dont-need-flash
YouTube without flash: I tried it on Chrome, the video was choppy, volume control didn’t work proerly and the progressing download & play made the position marker bounce around. But in the end, anything that replaces Flash, and Adobe’s reign over internet interactive animation,  is good..

C ya’ll next week!

Roy.

Stuff I picked up on the web the last week:
http://www.billshrink.com/blog/nexus-one-vs-iphone-droid-palm-pre-total-cost-of-ownership/
Compare the leading smartphones on the market

http://www.techcrunch.com/2010/01/05/quantcast-mobile-web-apple-android/
Mobile web usage stats: iPhone 65%, Android 12%, RIM 9%

http://gizmodo.com/5442217/the-invisible-oled-laptop-to-end-all-laptops
A transparent screen – Cool? yes. Practical? Not so much.

http://www.techcrunch.com/2010/01/06/augmented-reality-vs-virtual-reality/
Augmented reality is officially more popular than virtual reality.

http://gizmodo.com/5439721/new-touchless-mobile-interface-could-eliminate-fingerprint-smudging-forever
You don’t need a mouse anymore (if you have a 154 frames-per-second camera, and very steady hands)

http://gizmodo.com/5442385/samsung-projector-phone-in-action
Samsung’s projector mobile phone in action in CES

http://weblogs.baltimoresun.com/news/technology/2010/01/apple_tablet_3d.html
Apple is putting proximity sensors on new device to allow for 3D desktop manipulation.

http://gizmodo.com/5441682/att-sdk-for-dumbphones-announced
AT&T goes app-store on dumbphones, releases SDK for BREW

C y’all next week!

Roy

Stuff I picked up on the web the last week:

http://www.techcrunch.com/2009/12/21/world-map-social-networks/
Never tired of infographics: World map of social networks

http://gizmodo.com/5429631/implausible-digital-forensics-in-tv-and-film-a-medley
Awe-some and then some. Enhance.

http://lifehacker.com/5431998/ribbit-app-delivers-voicemail-transcripts-to-your-iphone
Give Ribbit credit for the bold stand in front of G-Voice.

http://gizmodo.com/5428610/rumor-google-working-on-chrome-os+branded-netbook-with-one-secret-manufacturer
A G netbook! That’s what we’ve been missing! Not.

http://gizmodo.com/5428642/apple-patent-sees-you-computing-hands+free-in-3d
3D interface by Apple, based on position of user.

http://gizmodo.com/5433074/open-apps-on-a-virtual-iphone-thanks-to-augmented-reality
Orange Israel promoting iPhones in a cute way: iPhone inside iPhone with AR.

http://www.techcrunch.com/2009/12/23/confirmed-jajah-sold-207-million
JaJah sold to Telefonica (O2) – for 145 million euros!

See ya’ll next week!

Roy.

Forth are listed the hyperlinks thy humble servant hath collected in past days:

http://gizmodo.com/5423006/i-cant-stop-smiling-over-google-chromes-new-ad
Nice ad by G for Chrome!

http://www.techcrunch.com/2009/12/09/geoapi-creation/
Very interesting: huge database of geo-tagged information with API for developers.

http://gizmodo.com/5424468/mits-bidirectional-display-lets-you-control-objects-with-a-wave-of-your-hand
I saw this contraption in the lab and the guy demoed it for me. It’s strange looking, but an interesting concept.

http://gizmodo.com/5425146/the-real-google-phone-everything-is-different-now
A G phone!
This is only a fraction of the online buzz about…

http://gizmodo.com/5425012/the-pen-de-touch-for-driving-light-cycles
Air-Pen. Nice implementation. But, is this is how we’re going to interface with computers in the future? Don’t think so.

http://www.techcrunch.com/2009/12/14/4g-mobile-network-sweden-teliasonera/
Välkommen till Sverige – LTE! (“Welcome to Sweden – LTE!” in Swedish)
TeliaSonera launching LTE

http://www.techcrunch.com/2009/12/14/the-unofficial-google-text-to-speech-api/
Free Text-To-Speech from G! Hurrah!

http://gizmodo.com/5425874/fuse-what-your-next-touch-phone-is-going-to-feel-like
TAT are as usual a good indicator of future UI (Social AR…). This time: 3D UI, haptic interface.

http://gizmodo.com/5426963/the-android-market-is-getting-ready-to-explode
Some stats on the Android market, looks good.

http://www.techcrunch.com/2009/12/16/google-browser-size/
G seem very committed to making the web better. First tools for webmasters to make their sites faster, and making sure the resolution fits

http://gizmodo.com/5428174/shooting-challenge-anthropomorphism
Anthropomorphism: Look at the video, (though it’s an AD) it will put a smile on your face!
and then go shoot some anthropomorphous objects.

http://gizmodo.com/5428233/microsoft-and-palm-treading-water-while-other-mobile-platforms-grow
Mobile OS stats (Feb-Oct): Apple and RIM skyrocket! WinMo, Symb, Palm, Android – stagnate.

Enjoy thy weekend!

Roy.

Image<Color>* load( std::string file_name )
{
 if( file_name.find( ".pgm" ) != std::string::npos )
 {
 return loadFromPGM( file_name );
 }

 else if( file_name.find( ".ppm" ) != std::string::npos )
 {
 return loadFromPPM( file_name );
 }

 else
 {
 return loadOpenCV(file_name);
 }
}

void fromImageMaskToIplImage(const Image<Real>* image, IplImage* ipli) {
 for(int x=0;x<image->width();x++) {
 for(int y=0;y<image->height();y++) {
 //Color c = (*image)(x,y);
 Real r = (*image)(x,y);
 CvScalar s = cvScalarAll(0);
 if(r == 0.0) {
 s.val[0] = 255.0;
 }
 cvSet2D(ipli,ipli->height - y - 1,x,s);
 }
 }
}

Image<Color>* loadIplImage(IplImage* im) {
 Image<Color>* image = new Image<Color>(im->width, im->height);
 for(int x=0;x<im->width;x++) {
 for(int y=0;y<im->height;y++) {
 CvScalar v = cvGet2D(im,im->height-y-1,x);
 Real R, G, B;
 R = (Real)((unsigned char)v.val[2])/255.0f;
 G = (Real)((unsigned char)v.val[1])/255.0f;
 B = (Real)((unsigned char)v.val[0])/255.0f;
 (*image)(x,y) = Color(R,G,B);
 }
 }
 return image;
}

Image<Color>* loadOpenCV(std::string file_name) {
 IplImage* im = cvLoadImage(file_name.c_str(),1);
 Image<Color>* i = loadIplImage(im);
 cvReleaseImage(&im);
 return i;
}

Well, there’s nothing fancy here, but it does give you a fully working GrabCut implementation on top of OpenCV… so there’s the contribution.

GrabCutNS::Image<GrabCutNS::Color>* imageGC = GrabCutNS::loadIplImage(orig);
 GrabCutNS::Image<GrabCutNS::Color>* maskGC = GrabCutNS::loadIplImage(mask);

 GrabCutNS::GrabCut *grabCut = new GrabCutNS::GrabCut( imageGC );
 grabCut->initializeWithMask(maskGC);
 grabCut->fitGMMs();
 //grabCut->refineOnce();
 grabCut->refine();

 IplImage* __GCtmp = cvCreateImage(cvSize(orig->width,orig->height),8,1);
 GrabCutNS::fromImageMaskToIplImage(grabCut->getAlphaImage(),__GCtmp);
 //cvShowImage("result",image);
 cvShowImage("tmp",__GCtmp);
 cvWaitKey(30);

I also added the GrabCutNS namespace, to differentiate the Image class from the rest of the code (that probably has an Image already).

Code is as usual available online in the SVN repo.

Enjoy!

Roy.