Neuro Productions

Making a 3D Game with OpenGL: Deferred shading and stuff

2012-07-20T07:11:33Z

Lately OpenGL and C++ is pretty much the only thing I do for clients, mostly for installations and iPad apps. And that’s gowning pretty well. But as a creative code guy, I pretty much stayed in our own tech subculture. But when it comes to cutting edge graphic techniques, we can learn a lot from those other guys, the game developers.

So one sunny afternoon, I decided to do just that and I started to make my own 3D game from scratch.

I wanted to write a blog-post when it was finished. But I hugely underestimated the work (Now I understand why indie gamedevelopers take +3 years to finish a game). And I’ve already done so much it wouldn’t fit in one blog post. So I’m going to try to make multiple blog-posts, to light out different technical things I’ve used for my game, and keep you updated about the progress of the game itself.

The game:

Or more game engine, since I don’t really have any game play at the moment:

As you can see, there are still a lot of glitches and things to be done, but I’m happy where it’s going.

Some of the things I’ve done so far (not always complete, final or clean).

-The terrain generation, it’s an infinite/live generated terrain.
-Seamless day-night transitions.
-Cascade shadow maps, SSAO (screen space ambient occlusion).
-Normal maps, alpha textures, alpha shadows.
-Some simple LOD (level of detail) setup for the terrain mesh and the 3D objects.
-Bone animations in the vertex shader + interpolations between the animations. (models are loaded with Assimp)
-The main part of the rendering pipeline.
-Learned the basics of modeling and animating in Blender
-and a lot of other stuff.

The current Render pipeline

It’s very WIP at this moment, but here is a diagram of how the current rendering is done. More about this below.
(click for the full size image, open in new tab if you get the popup)

Deferred Shading:

One of the things I wanted to try was deferred shading, just for the heck of it. But I’m really glad I did. Since it seems much faster, and has some additional advantages (there are also some disadvantages)

How does it work:

You just render all the main geometry once, In 3 textures. a depth texture, a color texture with just the diffuse. and a color texture with the normals in worldspace.
you can do that in OpenGL all at once, with an FBO (frame buffer object) with multiple color attachments. And since you don’t do any lightning, its pretty fast. After that, you use those textures to calculate the lighting etc.

The big advantage is that you only have to calculate the lightning on the pixels that are actually visible on the screen. So you have no overhead of overlapping objects. And if you want to do some post processing stuff, like ambient occlusion or depth of field, you already have the depth texture and normal texture in place.

To get back from the depth to the worldspace, you just multiply the screen space with the inverse of the perspective matrix:

float depth = texture2D(depthTexture, uv_var).x*2.0-1.0;
vec3 viewSpace ;
viewSpace .xy = uv *2.0 -1.0;
viewSpace.z=depth;
vec4 worldSpace = perspectiveInvMatrix* vec4(viewSpace,1.0);
worldSpace.xyz/=worldSpace.w;

But what I really liked about it, is that you can have a lot of dynamic lights in your scene. When you use forward shading, your happy if you can use a couple of lights. But but with deferred shading you can have hundreds. You just render the lights as volumes in a separate pass (sphere for a point light, a cone for a spotlight etc). So you only have to calculate the light on the position where it really affects things, and since you use the normal and depth texture as input, only on the geometry you really see.

The light volumes rendered as normal geometrie:

The fragment shader calculation I use for the light volumes:

vec3 dir = worldPos.xyz-lightCenter; // worldPos from the depth
float l = length( dir);
float dist =1.0-pow((clamp (l ,0.0,lightSize_var)/lightSize_var),2.0); // quick light falloff
float lambert  =clamp(dot(normalize(dir) , -normal ),0.0,1.0); //shading
gl_FragColor = vec4(lightColor *dist * lambert   ,1.0);

The result of the Light pass:

I just use a blendfunct GL_ONE, GL_ONE (add) for all the lights, although technically not correct. But it looks good enough.
The lights in the final scene:

Lambert texutre lookup

An other neat trick I discovered while doing research for this, is using a Texture to lookup your shading color (found it in the Valve Team Fortress 2 Paper).
The great thing about this is that you can edit your shading in photoshop. In my case I used a 2D texture, so I can have al the subtle light differences between day and night.

This is the texture I use now, but it still needs some work:

I just use the shading color for the u and the time for the v lookup.

Here is the plain old shading:

And the shading after the look-up:
Night:

Day:

That’s it for now, hopefully some more blog-posts to follow this up

Source code and assets at Github ( at your own risk )

Hairy Kinect Pictures

2012-04-23T16:26:35Z

You may remember the Furry Santa. That gave me the idea to use the Kinect 3D data to add hair to everything!

Yes, I know, pretty useless . But I like the results :

How Does it work?

In short, make an isosurface of the depth output from the Kinect and calculate the normals.

Then calculate points on the triangles where a hair starts. Now, start growing the hair in the direction of the normal in that point. Add some random and gravity to the growing direction of the hair. And to finish it of, calculate the lightning on every hair segment for a more realistic effect.

Of course, you can also use the pixel data of the Kinect webcam to make hairs in the original object colours. But I liked the black images more.

It’s uses openFrameworks + it needs the ofxKinect addon.

[update] I now got it now running real-time with a geometry shader:

Source code is a real mess, so at your own risk : Kinect Hair.

3D Pixel Editor for iPad with source code

2012-03-29T14:08:01Z

Yeah, blogpost.

After a year doing mostly C++/OpenGL for client (installations and iPad games) I decided it was time to release my own app. Its a 3D pixel editor for the iPad!

Check out the free 3D pixel art iPad app

Watch the video to see it in action:

How did I make it…

From scratch, no Flash, Unity, Cocos2D or some other framework (beside some matrix classes I borrowed form Open Frameworks). Although there is some Objective C in the mix,the main part is done in C++/ OpenGL ES2. Does it make sense to do it like that? Probably not, but hey, you learn a lot by doing it that way and it is pretty fun to do, and in the end I hopefully have some performance advantages.

For the main part of the 2D interface I use a Flash like displaylist I made (OpenGL ES2/ bitmap based) . Which give me the freedom to create cool interactive interfaces like in Flash and I still have the performance of C++.

Of course you have UIkit and cocos2D etc for i-devices. But the main reason I made it is that it also runs on native mac and windows and can be implemented in Open Frameworks and Cinder etc . Open Frameworks en Cinder are both nice for quick prototyping, they really lack the ability to create pretty interfaces without to much work.

And because you eat your own dog-food, I also use it on I-devices. On the plus side, its OpenglES2, so it can be nicely integrated with high performance 3D content in IOS + if I ever want to port it to a non IOS OS, it shouldn’t be to much of a problem.

If you have done ActionScript, you probably recognize the classes, like npDisplayObject, npBitmapSprite, npBitmapMovieClip, npEventdispatcher, npEvent, npTouchEvent, npTweener etc. Its far from complete/finished. But I already used it in several commercial projects. And I implement and optimise things as I need them.

Both the interface and the 3D are rendered first in there own frame buffer object, This gives the main advantage that in most cases, you only have to render one of the two at the same time, + the cubes are rendered in a multi sample buffer (on iPad 2), with isn’t necessary for the interface . Sadly Al those frame buffers slows it a little bit down on the Ipad 1 if you have a few cubes. But if you you use a certain amount of cubes, it also becomes an advantage there.

Yes it’s a paid app, but…

~~It was a tuff decision for me to make, I always loved to give things away for free. But hey, maybe some of the time I put in it can be paid back this way. So I can make more apps in the future.~~

Screw it, its free !

But no worries, even if its a paid app in the app store , I still decided to open source it. And if you read my blog and have an iPad, you probably can figure out how to build it (it’s a complete xcode project)
You get the source code here (Of course, buying a copy would be appreciated )

WebGL Tadpoles

2011-08-14T10:37:07Z

Its been a while since my latest blog post, I spend this year renovating my new house and office. I also became more and more frustrated by the slow speed of Flash for my experiments, so I started learning C++ and OpenGL. Which is coming along fine now. I made a C++-OpenGL Kinect installation for a client, made an IPad app in C++ and OpenGL ES2 etc.

And although I still do a lot of Flash development for clients, I kind of lost track on what’s happening on the web. So I was surprised to see what has become of WebGL. And with OpenGL more and more becoming my new love, I thought I give it a try and remake a C++ experiment I did in JavaScript.

It uses derivatives in the shader, which is an optional extension in the WebGL specifications, and as far is I can see, its only supported by Chrome at the moment. If you have the HTML5 audio API enabled, it reacts to the music (very basic, just some speed adjustments).

» Visit the experiment (needs Google Chrome)

The 30 tadpoles are Inverse Kinematic chains moving in a 3D Perlin Noise vector field. And since they aren’t static objects, I have to project the vertics and calculate the normals every frame in JavaScript. In that sense, the JavaScript speed is reasonable. But as always, I was hoping for more. In C++ I can easily render +500 tadpoles with 4 times as many triangles per tadpole . It should be interesting to see how fast Flash can do this… maybe I try that later.

To get some anti-alias, I render it first in a frame buffer object 9 times the window size (width*3, height*3). So, no complaints about the WebGL speed… for now…

A take on Generative Art

2011-08-14T10:36:05Z

Ok, I did some generative art-like things before. But it was never my goal, they where always somehow the side effect of experimenting with code.

But recently I did a nice project for Little Miss Robot. They asked me to make an Air application that generates images with code. At that point the code didn’t really mater. The only thing that counted was the result.

Some of the type of pictures the application creates (click to enlarge):

(no source code for now, but think Perlin Noise and Delaneay triangulation ) Ill update this post when the app is available. In the meantime, there is a video.

This whas a big difference in mindset for me. The code was just the medium, not the goal, and I actually liked the approach of making interesting images instead of making interesting code.

I looked around on the internet to see what was actually going on in the generative art scene. And one thing I noticed is that a lot of people use photos to extract colours and/or shapes for there art (I also did it). Which makes a lot of sense, photos always have a nice natural colour scheme. But with everything I saw that uses this technique, the source pictures got somehow lost. So this gave me the idea to extend the original pictures instead of just using them.

Some results of exploring that concept (click to enlarge):

Original (public domain) pictures:
http://www.flickr.com/photos/library_of_congress/2179909780/
http://www.flickr.com/photos/library_of_congress/2179146618/

Source code: generative_art_source

An other thing I saw where beautiful 3d structures (mostly made with structuresynth and rendered with sunflow) So I thought it would be nice to make stuff like that in Flash.

Creating 3D structures ain’t a problem, but two things I really wanted for my renders where depth of field and ambient occlusion. So I messed a round with depth-maps to create screen-space ambient occlusion and custom blur filters for the depth of field. But the result wasn’t really satisfying. To get a good result I had to ray trace my structures. Creating a ray tracer in ActionScript seemed like an interesting challenge, so that’s what I did…

Some of the ray traced images (click to enlarge) :

Making a ray tracer is quit addictive, and soon I wasn’t satisfied with my limited feature set and at this point I don’t feel I have explored those 3D structures enough. So in the next post (coming very soon…) some more renders and more about the ray tracer + the ray tracer source code. So, stay tuned

And finally, a little preview of things to come ( I admit, the shading is a little bit overdone :p ):

Blowing Up Images

2010-06-08T15:41:34Z

After my previous experiment I thought those shapes that were created looked kinda cool. But those were optimised for speed. So this time I tried the same effect but now I aimed for a better quality.

Source code: blowing_up_src

The result of a Sunday afternoon rendering:

Update:

Ronny Welter used my source code to make a cool video of the rendering process. Now you really see how its created and why I called this post “Blowing Up Images” . I also love how he added the shadow so it look like its 3D. Great job Ronny!

Particle Transition

2010-12-08T13:00:13Z

It’s been quite silent on this blog lately. The main reason is that I have a lot of work for clients (Some great stuff, maybe i’ll post some of that later). That’s good for me, but not for my site. To break the silence, a very small test I made. It’s not that special, but it looks pretty cool, I think…

It’s dynamic, but it needs some pre-rendering. so the sliders update in the next animation sequence.

In short, I create particles of every pixel. I pre-render the out animation of the in and out bitmap with the same Perlin Noise. Then I animate them over each other.

Check the source code for the details

Source code: ParticleTrans_src

Edit: sometimes the example doesn’t work in the browser (blame twitter, and there ever changing crossdomain policy )

Web crawler – Link Visualisation

2010-06-06T18:49:54Z

Yet another small weekend project (as in “Omg , I have 2 days to make something for myself before I have to go back working on commercial projects!” )

This time I made a simple web crawler that visualises html pages and the links between them

In short: Every circle is a page, they try to group per domain, the biggest circle is the start page, the smaller the circle, the more clicks your away from it (the smallest ones are 3 clicks), if there is a connection there is a line between them (the deepest items aren’t checked for connections between each other)

An example, click to view in full size (starts from neuroproductions.be) :

It only works local due cross domain issues and spring-graphing almost 2000 nodes is not what you call fast, so no live example.
You can always download the source code and try it yourself. (Warning: the source code is a crappy mess )
Source code: WebCrawler_src

I should have put more time in it, but time flies, and I already have some new exiting ideas for next weekend…

Outlines and other Dynamic Text pimping

2010-06-06T19:00:42Z

Trying to parse binary formats was on my to do list for quite a while. And recently I saw a cool little tool made in Haxe by Adnez that parses True Type Fonts, and exports the raw data to different formats. And you since you don’t have text outlines in Flash, I wanted to give it a shot.

So I looked up the TTF specifications and started reading those bytes. Due to time shortage it wasn’t a complete success. But I got some results and the cool thing about having the raw font data is that you can manipulate it:

But as you can see, it’s not really that fast or spectacular and my parser is not what you’d call complete or bug free (A lot of Fonts don’t work or are mapped wrong). I guess the important thing is that I learned that you don’t have to be afraid of parsing binary files. It’s not so hard as I thought.

But it took some time and effort, so I made an other example to fill this Blog post (inspired by Japanese toy typography)

I added the source code, but I don’t recommend using it in a real project. There are more optimal ways to get those results. And there are definitely better TTF parsers out there like hxswfml (there is a swc you can use in Flash) or FontReader.

Source code with the two examples (without the fonts ): ttf_source

Trying HTML5 as a Flash developer

2010-02-23T10:31:04Z

We have all seen (and participated) in the recent Flash vs HTML5 discussions. But to get a good view on HTML5, I needed to try it out. So I remade an old Flash project of mine (The Twitter Browser) in HTML5.
I chose that one because I knew it would be possible, its not to complicated and it has a little bit of everything: Loading external data, scripted animation, custom interactions and it needs a decent amount of computing power.

I never did any real JavaScript, so I went to w3schools.com and browsed thru the JavaScript tutorials and started coding. It took me +- 8 hours to to finish the job: the HTML5 version.

My opinion

But first a disclaimer: I only have one day experience with JavaScript. Correct me if I say something wrong.

Possibilities

Its cool how can manipulate the DOM elements, I really loved that. And yes, It can and will replace some of the Flash out there. But there are sill a lot of limitations.

The canvas element is not an alternative for Flash. There is not something like a display-list and no mouse-events for separate elements. But I guess you can emulate that with a lot of layered canvases and other DOM elements, combined with mouse position calculations etc, but that’s going to be a mess and I don’t know if my cpu will like that… I’m also missing webcam access, decent sound handling, 3D (webGL will do that, but it won’t be in IE9), real video integration etc, the list goes on. You can say that they are details, but I used those things in almost every recent commercial project. At this point HTML5 would be a step backwards for me.

Performance

Comparing the Flash and the HTML version wouldn’t be fair. I probably did some stupid things in the HTML version, and I know I did some stupid things in the Flash version (which is a couple years old). In many ways the JavaScript version handles things smarter than the Flash version. The least thing you can say about the Flash version is that it wasn’t optimized for speed.

But I was quit surprised about the JavaScript speed. It’s decent in a modern browser. I was expecting that I had to make a striped down version (with less nodes) But it seems to work fine. But HTML5/JavaScript is no miracle, it uses a lot of CPU for this and it will crash if you have an older browser or slower computer.

JavaScript

This is my main concern with the hole HTML5 thing, sorry guys, but JavaScript sucks. People who say that JavaScript is the future of the web obviously haven’t done some real programming in it. The dynamic objects, the loose typing, the prototype model, the silent failing, they are all a real pain in the ass. For those old enough to remember, JavaScript feels a lot like/is ActionScript 1…

It’s doable if you make a 500 lines project, but if you need to do a 10,000-100,000 lines app/website (which I do daily in AS3) it’s not going to be funny any more.

Mobile Safari

Mobile Safari can’t handle it on an iphone. It goes really slow. But you can’t compare a normal cpu with a phone cpu. And I didn’t want to write about this, till I noticed something very strange and disturbing when I was testing it on the iphone.

Take a look at example below, and try to drag both images:

The left image just reacts as you would expect from an image in a browser. With the right image, I stopped the default browser behaviour, and made my own drag.

This is something very useful. Besides for dragging, you need this functionality if you want to make a html5 drawing app or, and this is the interesting part, if you want to detect mouse/touch gestures

But what seems to be the case : This doesn’t work on Mobile Safari! ~~Does this mean that Apple is blocking html touch functionality in Mobile Safari? It seems so.~~ This wasn’t right. Thanks Ryan for pointing that out! There seems to be good touch support in JavaScript/on the iphone, even multi touch. Which is very cool! -> http://www.quirksmode.org/m/tests/drag2.html

They took our jobs!

You can’t ignore that some Flash developers out there are starting to worry about there future. But I don’t think you have to. If there comes a day when Flash will be obsolete (and I don’t see that happen soon), we’re gonna be excellent JavaScript developers.

I never did any JavaScript before, and I sill managed to make that example in one day. I’m sure it’s not perfect, and I probably made a lot of beginners mistakes. But I know it won’t take me long to make the switch.

As a Flash developer you can script advanced animations and interaction and UIs from scratch. You know your math. You can create large and complex client-side apps. You know how to be creative and create great user experiences with a limited tool-set. And you really love to do all those things.

It’s that combination of talents that made us leave HTML for Flash in the first place. And I think that those talents make that Flash developers can and will kick most HTML’ers asses when it comes to JavaScript development.

So, if that day comes, there not gonna take our jobs, but were gonna take theirs.

Will I do more JavaScript/html5

WebGL is definitely still on my to-do list, but apart from that, I think I will stick with Flash for now. I have some pretty cool commercial projects coming up and Flash is still is the tool of my choice for experimenting.