Sunday, July 17, 2016

AR will improve how we live, VR will change how we live

I had an interesting conversation with a friend this past weekend that got me thinking about the promise of augmented reality vs. virtual reality. My friend, who's opinion of technology I respect a great deal and who has enough VR and AR experience to have an informed opinion, said that he could see how augmented reality will be big, but he still didn't understand virtual reality. This makes a lot of sense to me, and I still remain as bullish as ever on virtual reality. VR has the potential to change culture which, for me, is more exciting than augmenting it. Because of this potential, however, it's much harder to envision what the future of virtual reality looks like compared to augmented reality's.

When you think of the potential of AR, it's easy to think of how a heads up display would augment driving, navigating, classroom learning, cooking, meetings, shopping, etc. These are activities we do every day and to add an effortless layer of technology that improves that experience is a no brainer. But most of the uses I can think of for VR don't improve existing activities, they try to define a new way of doing things. Some of my favorite VR learning apps don't take you to a classroom, but transport you across the solar system or scale you down to the size of an atom. As support for a virtual desktop improves and eventually starts taking advantage of the depth dimension, it has the potential to redefine what an office looks like. If you follow our work at Oculus Story Studio, you can see how VR has required us to rethink how we tell stories and how the audience experiences them.

Immersion, the defining feature of VR, is a problem and a benefit: it's a problem because every VR experience needs to reward all of the user's attention; it's a benefit because if we succeed, all of that experience is something you've never had before. The leap your imagination needs to take to add technology to what you've already experience is a smaller one. Imagining a completely new experience that will redefine how you go about your day requires a leap of faith, and will never be fully convincing as an idea. You have to eventually see it for yourself.

I know many people who have found a compelling experience in VR-- whether it was from playing a game, seeing a story, watching a sporting event in a virtual space, or being in a virtual dance party. Of all the potential experiences you can have in VR right now, I totally get that a lot of people haven't found something that clicks for them. Talk of a single killer app that will convert everyone is the wrong approach. There are so many different uses of VR that I think everyone will come to it for their own reasons. I accept that my friend hasn't found that app that convinces him of a new way of doing things. I have faith that given time, the innovativeness of the VR community, and the inevitable improvement of the technology, enough "killer apps" will be created to cover a large range of interests, including my friend's.

Monday, January 18, 2016

A Culture of Critique

It's undeniably a good practice, as uncontroversial as eating well and getting more sleep. It's a skill that applies to every profession you can think of.  You can always get better at doing it, and when done right, it will always lead to making you and your team better.  And I believe it to be the difference between what rises to greatness and what falls into mediocrity.

Asking for critique and then mindfully acting on it is the best skill for improving everything you do, and it's also one of the hardest things to practice.  We're just not naturally wired to do it.  To ask for criticism is to overcome one of our base fears, to expose ourselves to being judged as less than we think we are.

It's hard to say what makes Pixar "a lightning in a bottle" studio, but one thing I did learn during my time there is how a strong feedback culture leads to better story.  I got to see many crappy "Pixar films" in my 10 years, but none of them were ever shown beyond the studio walls.  They were all internal screenings.  During production, all films are assembled as a complete viewable experience at regular intervals and shown to the crew and creative leadership.  These internal screenings are edited together using whatever assets are available at that time of production- rough story boards, scratch audio, and unlit renders– until the final screening includes all animated, lit and rendered shots.  After every screening, feedback is given and the director uses those notes to make the story better for the next iteration.

The directors hated these internal screenings.  They were forced to put their baby up for criticism every 4 months and let an audience of film making experts point out its flaws.  And even though the screenings could sometimes create doubt and disruption, they were done on every production without exception.  It was so entrenched in the process that it was just the cost of playing.

In the absence of a strong culture like Pixar's, it's very hard to ask a director of a creative project to be so naked in front of their team.  The stress of making anything creative is tough as is.  But I believe it was only when a film had gone through enough iterations and improvements that it rose to the level of excellence that audiences expect from Pixar.  And this is why Pixar's batting average is so much higher than the film industry's.

We're striving to build this culture at our studio and it's not easy.  Not only do you need to account for the extra process it takes to be producing a screening at regular intervals (we're calling them rough assemblies), but you also need to build trust between the crew and directors.  The crew needs to provide constructive notes while being sensitive to their director's vulnerability, and the director must convey receptiveness while taking criticism.

Something we've been trying to do is to create a "notes window" after a screening. After the screening, the entire team can send notes for fixed number of days.  Discussion threads spawn and can keep active until the window closes.  It's strongly encouraged that team members not only call out what isn't working, but to highlight what they love.  Once the window ends, all official notes traffic ends.  Our director can then digest, be creative and respond with a synopsis of the new direction in which she wants to go.  She doesn't need to acknowledge or defend her decisions against every note, but it's a tactful skill to make the team feel heard even if she might disagree with the majority.  The window provides some piece of mind as to when the director can expect to feel vulnerable and when she should feel protected.  We've found that it's too hard to be receptive and creative at the same time.

Given time and consistency, it's my hope that a culture of critique grows so fundamental to our process that it will just become the cost of playing.  So far, I'm very proud of the direction we're heading.  It doesn't hurt that a lot of us at Story Studio already come from a place like Pixar where this culture of critique is so strong and proven.

Thursday, November 19, 2015

Stop Climbing Out of the Uncanny Valley

I've been playing video games since my first NES.  I grew up with Mario, Sonic, Megaman and Fox McCloud.  Back then, the 8-bit machines forced stylization in character design: there wasn't enough pixels or compute power to render realism.  So the game makers worked with the constraints. Mario is a plumber because with only 16 pixel rows to work with, the designers needed the mustache and overalls to create definition for his nose, chin, and body.



Since that time, we have far outgrown 8-bit.  Games are now built on top of gigabytes of dedicated graphics memory and have a little over 2 million pixels of screen real estate.  Shouldn't more horsepower and more pixels lead to more beautiful realism and as a result, a more appealing character? And it's not just the tech that's getting better.  Today's artists are more skilled and have way more tools than those early developers.  Shouldn't the expertly crafted detail lead to a better product?

In my opinion, character design is getting worse, and it's all because of the Uncanny Valley.

In brief, the Uncanny Valley is a postulate widely accepted in robotics and computer animation that the more a fabricated character tries to act and look like a real human, the more we subconsciously notice what's wrong, and therefore reject the fabrication as creepy.

Here's an example of state of the art work in games from Ninja Theory.  The detail they're getting out of the high res scans is mind boggling - down to finger print accurate resolution.


And I have to admit that the still images of the CG character in neutral pose are looking pretty good. Where it all breaks down is when the character starts moving.  Even if the game makers capture a wide range of neutral and active poses, humans are so adept at discerning meaning from facial differences as little as millimeters apart, there's no algorithm that can blend between those poses that captures a completely believable human face.  Most who aspire to be actors have a hard time achieving believability, why do we think a computer could do any better?

What's frustrating is that even though today's game designers know they are making creepy characters, many of them have this unshakeable belief that with a little more technology and a few more clever ideas, they will eventually converge to the limit.  If you rephrase the Uncanny Valley premise from the creative's perspective, the more you try to climb out of the Valley, the harder it gets. Sounds like a classic case of diminishing returns.

Image grabbed from Manly Guys doing Manly Things.

I think there are two alternative paths that could lead towards better ways out of the Valley.

The first is obvious: don't even try climbing the difficult side.  Use our wide range of tools, talent and technology to make something you would never see in real world - like a stylized illustration come to life.  We are much better as humans in finding appeal in the fantastic than we are at forgiving flaws in the realistic.  In the 80s, tech limitations forced stylization.  Now, stylization is a choice worth making.

The second comes from thinking how film captures appealing character.  Humans are not very sensitive to temporal gaps in what we see and so flashing a moving picture 24 times a second is not distracting and has been used for over a century in the cinema.  We also know that a film of a good actor can be incredibly appealing.  So what if instead of capturing high resolution scans of static poses, we scan actors as they act at around 30 times per second.  There's already promising tech going in this direction coming out of teams like 8i.com, Uncorporeal, and Microsoft Research.


We can then play that 3D capture back at 30 poses per second without needing any blending for the in between frames, even if the game is rendering at 60 or 90 fps.  Where this becomes a problem is that game developers like a simplified model, usually in the form of a joint hierarchy.  With a simplified model, developers can drive a character that can be affected by interactive input, like when the player hits the "jump" button.  Furthermore, the joint hierarchy is a good way to compress movement data so you don't need to download 1TB of data to see a cutscene.

But I think these are the problems worth working on.  Instead of finding a way to fabricate acting with an algorithm, even if that algorithm is fed by scans and motion capture, find algorithms that solve the problems of using dynamically scanned acting.

Stop struggling to climb out of the Uncanny Valley on the hard side.  We should either walk back up the easier side towards more stylized characters, or avoid falling into the Valley all together by finding better ways to record actors in 3D.  In the latter case, the problems worth solving are around interactivity and compression, problems better solved with a computer anyways.

Saturday, October 17, 2015

Less Meetings, More Artifacts

I feel like I've been fighting the gravitational pull of "too many meetings" ever since I became a supervisor.  At heart, I'm a software engineer, a maker.  I'm naturally a meeting minimilast: code doesn't get written in meetings. But as projects get more complex and a team grows to match, good communication requires structure. And a well prepped meeting is often the best way to make communication work.

The problem is that meetings are addictive.  One good meeting makes you think the next will be just as effective. You forget that most of the work that makes a good meeting happens before it starts.  And as the meetings increase in frequency, it's hard to gear down when your prep time is overrun. You go from meeting to meeting without any prep and as a result they keep getting less and less effective.

In my experience, all of the best meetings I've ever attended were about reviewing an artifact - something that took time and skill to make.  An artifact can be a board of concept art, a digital sculpt, a build of the game, work in progress lighting captures, etc.  Technical artists are familiar with these types of artifacts and the dailies or weeklies where we review them.  Creating something, showing it, getting notes, and revising : these steps define a technical artist's work cycle.



What I want to see more of is an artifact for every meeting, not just for director reviews.  For example, a meeting should not be a place to setup context or provide new information.  That can be delivered by email.  That email and resulting thread is a great artifact to bring into a meeting where a decision is needed. A well organized and well paced slideshow is much more engaging than an improvised speech.  Production planning should revolve around well designed and maintained spreadsheets or database.

There are two keys to a better artifact.

  1. An artifact should be well crafted, designed to the needs of the meeting - which means it should take time to make it good.  Even an email takes crafting to make it cohesive, clear, and well formatted.  I even think it's better to err on overcrafting... to a point.  Putting more effort into an artifact than is needed can lead to a different kind of waste.
  2. Secondly, the easier it is to reproduce the artifact, the better.  For example, slideshows are fine artifacts but a recording of that presentation is much better.  A presented slideshow can change with each telling, but a recorded presentation plays the same every time and can be distributed to a larger audience.  The best part about a recording is that you don't even need a meeting to review it.  Furthermore, an edited video is easier to work with since you can use a tool like Premiere to finely craft the experience. A script is a good artifact, but an animatic reel is much better.  Two audience members can read a script very differently, but a video fills in a lot of what screenplay leaves to the imagination.  At Pixar, the animatic reel was the required foundation for all production planning, not the script. 

There are a few benefits to always requiring an artifact before a meeting.  You'll have less wasted meeting time because you'll be spending your time creating artifacts.  If there's a reproduceable artifact, especially a playable media of some kind, it's much easier to get people on the same page, get consistent feedback, and make revisions.

In the end, any creative team's final product is an artifact of some kind.  And the more we spend our time getting good at crafting artifacts at all stages of production, the more we improve on delivering a better final experience to our audience.

Tuesday, September 8, 2015

The Gender Disparity Problem - A Fishing Story


This past week, I visited my home town of St. Louis, Missouri.  Along with getting to be with my parents and sister,  I got to meet my first newborn niece who is just 6 weeks old.  During most of my stay, we spent the time talking in our pajamas, watching Netflix, and waiting for the baby to get hungry again. 

As I do on occasion when visiting home,  I also went fishing with my Dad.  We started our Sunday waking up at 4:30 AM.  We said our goodbye to my sister who was awake giving the baby a drowsy, early morning meal.  We drove for an hour out to a small bass lake in Wentzville, MO just in time for the sunrise.

Once we arrived and set all of the rods,  I started casting a top feeder lure, setting the reel, then winding and jigging the lure across the water surface making sure to keep the right amount of tension on the line, feeling for a hit.  When a fish didn't catch, I'd do it over again.  Again and again for 3 hours without much action.

At first, I wasn't awake enough to think about anything beyond casting and reeling.  But when the sky finally started glowing orange, and I was nearing what felt like my hundredth cast, a provocative thought bubbled up - a thought about gender disparity in Silicon Valley.  I know it's a strange moment for such a thought, but it's been on my mind as of late (we're in dire need of more females in the VR space).  For the past 2 years, whenever I get a pulpit opportunity, I have talked about how a woman's mentality is sorely lacking in the male dominated tech world.  I just know from experience that a mixed gender team performs better than a male dominated one.  But what is it that's missing about an all male team?  What is it about a male way of thinking that needs female balance?

One thing I've observed more in men than in women is an obsessiveness when solving a problem - a drive to keep going even when there's probably a good reason to stop or take a break.  Playing games late into the night passed a reasonable bed time, banging our head against a difficult problem set instead of changing context, coding past the point of dehydration and an inability to think straight – it's probably this kind of male dominated thinking that has fostered the unhealthy Silicon Valley culture of working late into the night beyond mental exhaustion.  Furthermore, this obsession with solving a problem can sometimes make us lose sight of whether we should be solving it in the first place.

I've been lucky to work with some great women problem solvers at Pixar (as well as be in a long term relationship with one), and I've found that women are better at considering the bigger picture, often augmenting technical thinking with other intelligences, like social engineering and logistics.  They are better at setting priorities and have the discipline to temper when needed and the patience to rethink an approach instead of drilling deep on the first one they tried.

And that brings me back to that provocative thought.  This all his seems to make sense from an evolutionary point of view.  The men worth their genetic salt should have been able to hunt or fish over long periods of time with an obsessive drive.  Even if the trail ran cold and the fish stopped biting, the men who stuck it out had a stronger chance statistically of bringing home some calories for the tribe.  And while the men were away on long hunts, women had to deal with everything else that made a tribe work including nursing the newborns and foraging for reserves in case the men came back meatless.  So it makes sense that task prioritization and big picture thinking is their natural strength.

Now, I'm not apologetic about my male way of thinking nor do I speak in absolutes.  Some great things have come out of an obsessive drive.  Men are capable of big picture thinking as much as women can be obsessive.  My concern for VR is that without mixing in more women, a room full of men are more naturally wired to have blind spots.   We are more prone to obsessing over barren rabbit holes or dead lakes.  It's a gender balanced work place that builds a healthy, long lasting, well focused product and team.

And it's around that moment of clarity that the bass finally started biting.

Saturday, August 1, 2015

Quick and Dirty Code Reviews

By now, the code review is well understood and respected as the vegetables of healthy programming.  And as we know from any meal eaten in haste, the vegetables are often the first thing to be left on the plate.  I've seen this neglect a lot in computer animation production where the code only needs to work until the final frames are rendered.  It's worth arguing that in these cases where code is tactical and only needs to work for a few clients for a brief time, it isn't worth the extra time for code reviews.

This post is a bit of a confession.  I admit I also have neglected my vegetables in the chaos of production.  Now that I've started building realtime VR experiences, however, I need to remind myself that my deliverable is no longer a bunch of rendered images, but an executable.  It makes me nervous to think that my code may be used many years beyond the launch of a project.  Hats off to the programmers of all of the games I've ever played.  So now, even tactical code can have huge ramifications to the end product.  All stages of checkin need code reviews.

But here's the problem.  When code needs to be quick and dirty, for prototyping, firefighting, or rushed deadlines, the rigorous code reviews will inevitably start getting left on the plate.  And so instead of losing the benefits of code review, introspection, shared knowledge, readability, I think the code reviews also need to be quick and dirty.

For me, a big benefit of reviews is having my brain process my code differently when having to explain it verbally to another programmer.  A lot of bugs I've found with a collaborator come from me talking through a section of code and then realizing the code doesn't actually do what I am saying.  This part of a review is easy and doesn't even need another person (see about the Rubber Duck).  But the reason I would prefer another person is that they can ask questions.  Why do it like that?  Is there a simpler way?  What does this section do?  It's in answering these questions or usually because I'm unable to answer these questions well that I find code that's more complicated than it needs to be, or code that needs to be rewritten for readability.  For core system code that will live longer than a production, I do appreciate feedback on syntax, line spacing, performance and convention, even when it can feel draconian.  But when I need to checkin a fix to unblock a lighter or animator,  I just want a quick sanity check - something just slightly more interactive than a rubber duck.  A reviewing pair should collectively understand where on the scale from zealot to rubber duck the reviewer should be based on the context of the code to be checked in.

To enforce a consistency of having reviews,  I don't think the rigor of code reviews needs to be consistent across an entire project.  There's nothing wrong with adding some butter when you're having a hard time eating your broccoli.

Friday, May 22, 2015

The Arcade Megaplex is coming back with VR

Image grabbed from Pinball Wizard Arcade


I miss the megaplex Arcades in America.  I'm talking arcades with a capital 'A'.  They were a childhood institution.  Every time I see a small gaming room tucked away in the dark corner of a cineplex or a flashing pinball machine at the back of bar, a moment of intense recall hits me with the smell of ionized air, the cacophony of 8-bit brawling and quarter falls, the taste of metallic adrenaline, and an excited tremor in my fingers.  Gone are the days of the dark, neon lit warehouses with rows of gaming machines capped with money exchanges.  Gone are the days when the best place to have a 10 year old birthday party was at places like Tilt at the Northwest Plaza or Exhilirama! at the Crestwood Mall.

When the home consoles started having competitive graphics and a killer local multiplayer, going over to a friend's house to play GoldenEye or Smash Bros. became the new thing to do on a Saturday.

Our generation hasn't forgotten as seen by the success of adult gaming bars like Dave & Busters.  But I do wish that today's kids had a place more magical than the living room to make new friends adventuring together in mind blowing experiences.

The Arcade was where entertainment based virtual reality attempted its disastrous debut with experiences like Dactyl Nightmare.  But with better display tech and affordable tracking solutions being at the level of maturity that Oculus has demonstrated, VR Arcades feels like an inevitability.

Arcades got big around the same time the Atari console was released in the late 70s.  The home console was a great appetizer but it was only at the Arcade where you could get access to the best experiences: experiences that felt many generations away from being in our home consoles.  The experience gap remained wide for so long, that the 6 foot tall game cabinets felt like they housed a completely different medium.  When that gap shrank, it just didn't make sense to leave your home when you could play Street Fighter on your TV.

Just like when the Atari home console came out, when VR becomes available to the market, it will drive us to want more.  And just like the Atari was heavily limited in the experience it could deliver at the price point it offered, consumer grade VR is still very limited in terms of resolution, performance, and the fact that you need to be tethered to a powerful machine.

When places like the VOID become a reality, these places will be able to deliver the kind of presence that is unrestricted by consumer constraints.  The head mounted displays and tracking solutions at these VR Arcades will be much more expensive and therefore be miles ahead of what you can afford at home. Instead of a warehouse of cabinets, these arcades will be a large space of corridors, rooms, stairs and platforms. You'll be able to move around untethered with a group of your friends all sharing a world and story projected over those physical walls and obstacles.  When you touch the wall of a 20s style mansion in a murder mystery or the alien ship corridor of a sci-fi space operetta, it will touch back.  These places will not just be for teenage boys; they will most likely have the audience diversity of cinemas.  They will be so much closer to the dream of the Holodeck, and so novel compared to what we have at home, our drive to want more will lead us here.

In the long run, VR Arcades will inspire the technology that we can bring home, just like the Arcades of the 80s and 90s inspired the Super Nintendo, Sega Genesis and Atari Jaguar.  But I wonder if there will be a time when the home experience will close the presence gap to the point where VR Arcades will become obsolete.  The big difference from what came before is that great presence requires a large physical space onto which virtual worlds can be projected.  So perhaps the killer app will be some form of augmented reality that overlays a virtual world over your neighborhood backyards.  Or maybe we'll need to wait for tech like the Matrix that will bypass our tactile sensory systems and allow us to explore and touch a virtual world from the comfort of a couch.

In any case, I know the presence gap will take a long time to close and I look forward to spending some of my upcoming Saturday's returning to the Arcade and having my mind blown.