Strings Attached is a Sixaxis motion-controlled puppeteering game prototype funded by Sony Computer Entertainment Europe and developed in late 2006 - early 2007 by Fresh3d, our company.
{youtube}lpTMc6PgsRU&vq=hd720|960|540|1|{/youtube}
How it all started
In summer 2006, as programmers from our team were putting the last touches on Mountain Bike Adrenaline, our latest Playstation 2 game, Yann and I were already thinking about our next project.
We had been following the PS3 buzz for quite some time (the console was scheduled to come out in November in Japan and US, and march 2007 in Europe) and were interested in working with their upcoming Sixaxis motion controller.
With only public info about the hardware available, we worked on a puppeteering game concept and submit it to SCEE (Sony Computer Entertainment Europe) for consideration. A few weeks later we were at their office in Liverpool to discuss the pre-production agreement. Although they liked the overall pupetteering idea, much of the design we had worked on was scraped and we were asked to work on something much simpler, a party game for kids. We came back with smiles on our faces and were sent a couple devkits the next week.
That was really awesome for us as it was the first time ever our small company had acces to a console devkit before launch.
Broken controller, and broken physics... again
Although it was to be a small party game, we had ambitious goals for this product. First, the player would control the character the way a puppeteer does, simply by holding the controler with one hand and moving it around. Second we were planning to use physics to drive all the joints of the character's skeleton and have animation drive physics forces and not directly joint rotation.
The idea was this would allow to unify all forces and provide smooth, seamless animation blending and interract realisticlly with environment objects. Except that we were yet again with the same broken ODE physics engine we used in our Mountain Bike Adrenaline PS2 game, and it just didn't work as can be seen in the video above. We ran into the same sticky/unstable collisions problems. As we were developping the prototype, we planned to switch later to another physics engine that was becoming available for free, but it was too late for our prototype.
The other major problem came when we discovered the controller was not capable of registering absolute space position (remember this was sold by Sony's marketing as a six axis capable controler, which it clearly was not), and that was critical to our design. We needed to be able to move the character around, and without position, we were back with mapping regular locomotion to the tilt of the pad, much like you would do with a stick but with the entire pad instead, and with less precision with that. This was useless, totally missing the point, and the entire concept sudenly became irrelevant.
When the Sony comitee evaluated our prototype, they were obviously not convinced, we didn't get the greenlight for production and the development stoped right there.
Original idea
I mentionned earlier the first game idea we came up with was changed when we met with the Sony team. The original concept was to play puppets of famous figures and trying to reproduce their best performances on top of audio excerpts from film, video clips, or shows of notorious fame. We would have a carreer mode and be able to show progress in a virtual small town on the US west coast. Christophe Bauvir who was working for us as a free-lance designer and I had written the original concept.
To illustrate the concept, I had made a few drawings of famous people in a very caricatural way. This was my first forray into caricatures and it was a lot of fun to draw:
New design, new characters
After we were asked to make a simpler game targetted at kids, and remove the famous figures (that would most probably have got us dragged into some messy legal problems), we had to replace them with two kids, a boy and a girl.
As I would move towards actual modeling and texturing for the prototype implementation, I asked JYL ( http://www.jyl.be/ ) to produce the new character concepts of the boy, the girl, and a little dog and he came up with some cool designs. Eventually, we concentrated on the girl only for the prototype.
When I started working on the first version of Sally (the name of the girl) model, I quickly realized nice 2d cartoonish drawings don't necessarily translate well to the 3d world. The very first wip was actually pretty scarry !
Left: Sally early WIP model
Above: JYL concepts for Sally
After some more work, I came up with this first version that was to be integrated into the first prototype of the game:
Above: Sally V1.0
At that point, I starter working on textures and shaders. More on that later. This was much better but still not appealing enough. Once integrated, she looked pretty ugly. That was partly due to her eyes half-closed and neutral mouth because there was no rig active yet. But the feedback was clear, Sony asked us to rework her to make her more cute.
I would better describe myself as an environment artist, so I needed some advice for this character. At that point, I talked with Francois, who was going to do the rigging and animation, and his experience in cartoon character design was very valuable. He helped me figure out how to improve the model and come up with version 2.0.
But again Sony would say this was not cute enough. Or rather, they asked us to just make her look like Boo (from the Pixar animated film Monsters Inc. ), something they probably had in mind from the very beginning. I figured we would later rework the model to give her our own design, but this would do for the prototype anyways. So I changed her hair shape and color and modified again the proportions slightly and came up with V3.0 that can be seen in the prototype video.
Above: Sally V2.0
Left: Sally V3.0 as can be seen in the prototype.
For her hair, I used alpha-blended polygon strips on top of painted hairs on the head itself. That way, the hairs had a sense of thickness. It was planned to have them rigged to the physical engine too, but it was not implemented in the prototype. The same goes for the dress.
Another character I worked on was Banzai, the little dog. I made both the model and textures. The pirate version of Sally was done by a student during his internship at our company.
It might surprise some of you, but I used Maya as a 3d painter as I was not convinced by thrid party applications at the time, and made most of the texture of my games with it. The tool can be seen in action here on some wip images of Banzai. I never quite understood why Autodesk did not push the tool further, an integrated painter is quite handy. This had great potential.
Above: Banzai 3d painting wip
Below: The final model features a tileable detail texture on top of the base painting.
Environment
One of the first task I started with was defining the overall look and feel of the game through a target render. For that I needed to create the environment where the game prototype would take place. The goal of the game was to see things through the eye of the little girl (or boy) as she played with her toys, so we figured the best place to start the game would be her bedroom. I started with a template to define space and kept working on refining the room and furniture models:
Lighting would be key to the environment, so I started playing with global illumination to generate a target render that we would later try to replicate with real-time techniques:
One of the key feature of the game was the ability for the environment to morph to show the imagination of the little girl as she change clothes and plays with her toys. There would be some costumes in her wardrobe and as she would be able to collect more.
As a demonstrator, we included a pirate outfit and the room morphing into a pirate ship's deck.
For the pirate version, I painted a large 360 image of a pirate bay that was mapped to a cubemap texture to bring some nice parallax effects when moving the camera.
To create the morphing effect I made use of programmable shaders.
Shaders
Prior to the Playstation 3, I had never worked with programmable shader. Both the PS2 and the PSP featured fixed pipelines (except for the PS2's vertex pipeline), that is the operations the gpu was capable of doing were pretty much fixed in the hardware and you could only change parameters or do multiple pass to create more complex effects.
Now with programmable shaders, although the performance limit was still there obviously, the functionnal limit was just the imagination.
This is the time I realized I should have put more effort into studying math when I was at school. Lighting and shading is all about modeling light and surface interractions, and like any model, it requires a mathematical description to work.
I did my homework and realized it is really not that difficult after all. I learned Cg, which is a high level shading language using the same syntax as C that includes a lot of helper intrinsic functions for manipulating vectors and other useful quantities.
Hopefully, Maya includes a Cg node, so you can write your shader and preview it while working inside Maya. Naturally, we decided to include the Cg support in our exporter so that we could seamlessly create content in Maya and export it to our viewer. Overtime, the original Cg node would be replaced by a custom FreshEngine node that included data driven properties of the shader such as blending states and the like.
The holy grail of 1080p
When the PS3 came out, I was eager to see all games in full HD glory running at 60 frames per second. Of course I wanted Strings attached to perform at the same highest resolution and framerate. And I wanted to have it runing with MSAA 4x, that is the frame buffer is 4 times the size of the screen for antialiasing purpose.
I quickly understood the gpu was largely underpowered for full HD rendering with per pixel lighting (and most games came out in 720p). Think about it, a pixel shader is a programm that executes at every pixel of the screen (2 millions in full HD), 60 times per second. So basically two million programs execution in about 16 milliseconds. With that in mind I chose to do much of the calculation per vertex. So only high frequency calculations such as specular highlignts are done per pixel, the rest is per vertex (such as diffuse lighting).
To avoid the use of normal mapping, I chose to have a decent tesselation on the models, enough to hide the linear interpolation between vertices.
Above: you can see how Sally (V2) interracts with the lighting (viewed here in Maya).
My shader supported 2 point lights and one directional light. The shadow casting was only enabled in our viewer.
We never quite managed to maintain our 60 fps target goal in all situation and the prototype was finally running at 40-45 fps 1080p with MSAA 4x.
Fabric
For the clothes and fur, I made use of a simplified version of BTF (bidirectional texture function) which is basically a texture prelit with different lighting angles. It provides good results for fabric because it captures all the subtles light interactions inside the fabric that are difficult to reproduce synthetically.
Here's an example of fabric I shot with 4 different light angles:
And then re-mapped onto Sally's model dependent on the real-time light angle:
Screenshot gallery
Trivia
A couple years after the development of this prototype, Sony came back to us to see if we would be interested in developing the game, this time for their move controller (which had absolute position tracking at last). We were on another project and unfortunately had to decline the offer.
Credits
Yann Robert: Engine / gameplay programming
Gregory Leblond: Engine programming
Guillaume Raffy: Physics programming
Franck Sauer: Art / Technical Art / Game design
Christophe Bauvir: Game design
Stephane Menardais: Animation engine
Francois De Bue: Rigging / Animation
JYL (Jean-Yves Leclercq): Concept art