Saturday, July 22, 2017

The Growing Skill Gap - VR can help here

For those of us lucky to be working in tech in the Bay Area, we get to work on some of the coolest problems in automation, artificial intelligence, robotics, and virtual and augmented reality. We are driven to innovate not just because they are cool problems but because we believe what we invent makes life better for everyone. Technology like self driving cars, robot driven assembly lines, image classification, speech recognition and translation all have the promise to make our lives easier and more cost efficient.

The hope of any enriching technology is that it frees our time to pursue a richer life. But if that is our goal, we need to be mindful of the skill gap we are creating and how this could actually be doing more harm than good. A lot of people contribute to society by taxiing passengers, working on a factory floor, labeling and sorting information, and translating foreign languages - skills that are being made obsolete by those same futuristic technologies we work on. As tech continues to get better we may invent ourselves out of usefulness. Considering how hard it is to learn new skills to catch up on a changing world, technological progress will lead towards a bankruptcy of purpose.

This bankruptcy can have dire consequences on progress if not addressed. I believe we are already seeing signs of this with the dramatic shift in world politics towards populism and nationalism and a fear of any change in the name of science. I think this is also why science backed messaging of climate change is still not getting through to everyone.

As a technology lover and innovator, I believe progress is the right path forward, but is there a way we can also solve the problem of the growing skill gap. This is where I think virtual reality and augmented reality may be able to help. My favorite feature of VR is how it has created the most intuitive interface for how we interact with computers. Up until now, our interaction has been a difficult abstraction to learn through mouse clicks and keyboard strokes. With VR, I see new users pick it up within seconds. We've all grown up knowing how to move, look around, reach out to touch and grab things; VR just taps into those instincts. We can convey so much complexity with the ease of natural gesturing and grok so much information through the ease of immersion.

It's my belief that we'll be able to take advantage of this facility by creating educational VR experiences that will make it much easier to learn new, relevant skills. Unfortunately, making virtual content also suffers from a skill gap. Those who know how to create in this new space are mostly game makers who are passionate about making games. Those who are passionate about education technology are still learning the skills to create useful virtual experiences.

That's why it should be an important goal for this new industry to make it easier to be a content developer. Heavily documented and community backed game engines like Unity and the Unreal Engine are a good start but are still designed around making games. We need the next generation of VR developer tools to make it easier for educators to get involved.

The progress of technology is inevitable, but we need to remember that technology should also be used to close the skill gap it creates. And it starts by closing the skill gap with the tools we use to teach.

Friday, July 7, 2017

Onward

About 2 months ago, my virtual reality studio was shut down. The closure was sudden and a stunning punch to the gut. It was a decision I very much disagreed with, but it was a decision I can understand considering the larger context of the VR industry and the difficult path ahead.

Before going onto what may be next, I've needed some time to recover from that blow, and reflect. In writing this public diary entry, my hope is that it will not only help me plant a sign post for the journey so far but perhaps serve as a way to leave behind what needs to be left.

Considering what we were able to accomplish in 3 years, I am proud of the work we did to see what this new medium could look like. Lost, Henry, and Dear Angelica were as much experiment as they were entertainment, and our mission was far from accomplished. But any pioneering work needs to start somewhere. When these pieces first premiered in 2015, 2016, and 2017 respectively, they contributed new nouns and verbs to how we talked about VR entertainment.

The thing I am proudest of is the team we assembled. Story Studio was a group of some of the best computer animation, visual effects and game makers in their industries who were all excited to stake new claims in a risky frontier. I know this because during the interview for every team member, I took a moment to warn of the dangers of being so far from civilization. I spoke of how I was excited to be discovering what no one had ever done before, but that as exciting as the discovery was, it was difficult to predict what was around the next corner, how long the journey would take, and what setbacks we'd need to overcome. I'd even be as honest to say that I couldn't guarantee the studio would still be around in 6 months. Some interviewees blanched at the uncertainty, but I knew those that came back with eyes wide open and excited about what lies ahead, they would make great team members.

I am confident that the seeds of this team, now having scattered out into the world, will go on to help grow the forests that will define our cultural and technical landscape.

Then there's what needs to be left behind. I've struggled with the guilt that we weren't able to convince our leadership we were a bet worth making. I've struggled with a bruised ego that inevitably comes from a fall like this despite how lucky we were to get the opportunity in the first place. I've struggled with the fear that capturing the kind of lightning we had only happens once in a lifetime. I need to constantly remind myself that all of these suffocating thoughts aren't worth my energy if I'm to get on with my journey.

In early 2014, I made the difficult decision to leave the comfort and renown of Pixar and embark on a journey to do something novel, scary and groundbreaking. I would have been happy to find a creative problem worth solving that may have taken 3 to 5 years and given me the opportunity to see what it was like to grow a team from the beginning. I was ecstatic to find virtual reality, a problem so big that it wasn't just about creating a new product on an existing platform, it was about figuring out a new medium-- a problem that would take the rest of my career. The horizon line stretched far beyond an unknown and perilous frontier. It was daunting, inspiring, and irresistible.

This closure is a particularly humbling setback and it's given me a moment to pause and reflect on if I should turn around. Am I so driven towards the promise of gold in California that I'm blind to the fact it may be a fool's journey?

I don't think so. There are enough brilliant, thoughtful, and future savvy people that believe in VR. It remains undeniable that VR has the potential to change our relationship with digital content, making it a richer, expansive and more meaningful experience. I continue to be awed and inspired by the innovation that continues to happen on a weekly basis.

As for what's next, I have decided to continue towards the horizon of what VR may become, even during this particular stretch of desert. I may plot a slightly different course than what we were doing at Story Studio, but my goal is still to give a visitor something that is immersive, memorable, and emotionally engaging. My sights have changed in that now I also believe that for VR to truly find itself as a medium, it must be an experience you share with your friends. But that's for a future post...

It's time to dust myself off and push onward.

Sunday, October 23, 2016

Leading via Questions

There's a pivotal moment in the lifetime of a growing team where its leadership no longer has enough information to be responsible for how that team gets things done. And if both team members and leadership don't recognize their roles need to change, it can be frustrating for everyone.

When a team is small and the leadership has the capacity to be in the trenches, there's a lot of efficiency in having your more experienced decision makers steering the work. By working so closely with everyone, leaders are operating with shared context that team members can trust and the leaders can course correct quickly based on all of the feedback they get.

There is a point, however, where the responsibilities of leading and protecting a larger team make it more difficult for leadership to spend time in the trenches. When that happens, if the leaders keep involving themselves in areas in which they are now so woefully uninformed compared to those they lead, it becomes meddling-- a serious impediment to getting things done.  Up until that point, the natural relationship was one of leader and followers and so it's not evident that there's a moment where the team knows a lot more on how stuff gets done.

Over the past year, with my studio having grown to more than 25 people, I've been working on this role change. There's a few interesting problems to solve. If I am accountable for, but no longer responsible for how stuff gets done, how do I pull that off? How do I help those team members that are not used to being decision makers develop this skill? What are the skills I need to grow in this new role and how do I improve? And how do I do all of this while making sure that trust flows both ways?

I have one solution that has had some success. I try to ask questions like I'm the student and they're the teacher. If I am now less informed on how things are getting done, it makes sense that I'm the one who needs to be taught.

This has had several effects:
  1. By making it clear that I respect and want to learn from what the team is doing, they feel trusted and empowered. 
  2. Because I have a lot of decision making experience and a 10,000 foot view, the type of questions I ask can probe and reveal problems the team hasn't seen from their angle. If I'm not satisfied with where things appear to be going, asking followup questions can either help the team realize they haven't thought something through or help them craft an argument that is satisfying for me. In both cases, the team is the one coming up with the answers. 
  3. Over time, I've noticed the team is getting better at anticipating the questions I would ask and in effect getting better at their own decision making. 
  4. For those moments when I still need to make the decision, there's now a clear relationship for how the team gets involved leading up to that decision and for when unexpected consequences may need correcting.
After a year, I still make mistakes where I'll give trench level direction that comes off as uninformed and meddling. It's easy to lapse back into the comfort of being the one who decides, a skill I've developed and gotten good at over the last 6 years. When this happens, it's been really exciting when my team, having developed leadership skills of their own, are now actively coming to me, letting me know what context I may have missed, and teaching me something about how I can be a better leader.

Sunday, October 2, 2016

The Finishing Trolls

We've recently finished our third project at the studio, and as proud as I am of it, I'm glad these final months are over. It's not the desk work that tired me out - that blood and sweat comes from my amazing team. The hard part about finishing is having the strength and discipline to know that it's good enough. As a producer, I was constantly tested in making the hard decisions necessary to ship it even when more time and effort could always make it better. And that's the haunting fact about making something creative— you can always make it better. But until you actually put it in front of an audience, it doesn't count. One of my favorite lessons from Pixar, expressed elegantly by John Lasseter, is that a film is never finished, it's just released.

The problem with working on a creative and highly technical product like what we make at Story Studio is that over the long amount of time it takes to develop, produce and finish a VR movie, you can lose what excited you about the project at the start. And then you start worrying if it's not just fatigue, but something more fundamentally wrong.

And this isn't new to anyone who creates. Self-doubt and angst are trolls that move in during those last few months of finishing. But in a medium as young as VR, I feel like the trolls are especially tricky. Not only do you begin doubting the project, but you also begin worrying if the audience will even get it. We've now spent so many more hours in VR than the average consumer that I worry my experience will mislead us into thinking we know what will entertain. One of the tricks to producing is to find the strength to ignore those trolls and push through that last mile.

I have to thank my experience at Pixar for helping me stay strong.  While there, I was lucky enough to see 3 feature films through to the end— Cars, Wall•e and Up.  I got to experience what it's like to spend so long on a project that you begin focusing on the flaws and forget what makes the film great. For each of those projects, as much as I and my fellow team members cringed to know all of the things we just didn't have time to fix, all of those films came out as beautiful, entertaining and critically acclaimed films. If I could be tricked into doubting Pixar films, and then be blown away by their reception, then those trolls aren't worth listening to.

As for knowing if the young VR audience will get it and be entertained, that's part of frontier life with this new medium. The only way we'll know if what we're doing works is to have the faith and guts to put something finished out into the world and see what happens. If we spent all of our time trying to perfect our experiences, we would never learn anything. Furthermore, with each VR movie we complete at Story Studio, it helps me build up a tolerance against those tricksy trolls.

That all being said, I can't wait to show you what we've done.  It's inspiring, unexpected, and I guarantee it's something different than anything you've experienced before.

Sunday, July 17, 2016

AR will improve how we live, VR will change how we live

I had an interesting conversation with a friend this past weekend that got me thinking about the promise of augmented reality vs. virtual reality. My friend, who's opinion of technology I respect a great deal and who has enough VR and AR experience to have an informed opinion, said that he could see how augmented reality will be big, but he still didn't understand virtual reality. This makes a lot of sense to me, and I still remain as bullish as ever on virtual reality. VR has the potential to change culture which, for me, is more exciting than augmenting it. Because of this potential, however, it's much harder to envision what the future of virtual reality looks like compared to augmented reality's.

When you think of the potential of AR, it's easy to think of how a heads up display would augment driving, navigating, classroom learning, cooking, meetings, shopping, etc. These are activities we do every day and to add an effortless layer of technology that improves that experience is a no brainer. But most of the uses I can think of for VR don't improve existing activities, they try to define a new way of doing things. Some of my favorite VR learning apps don't take you to a classroom, but transport you across the solar system or scale you down to the size of an atom. As support for a virtual desktop improves and eventually starts taking advantage of the depth dimension, it has the potential to redefine what an office looks like. If you follow our work at Oculus Story Studio, you can see how VR has required us to rethink how we tell stories and how the audience experiences them.

Immersion, the defining feature of VR, is a problem and a benefit: it's a problem because every VR experience needs to reward all of the user's attention; it's a benefit because if we succeed, all of that experience is something you've never had before. The leap your imagination needs to take to add technology to what you've already experience is a smaller one. Imagining a completely new experience that will redefine how you go about your day requires a leap of faith, and will never be fully convincing as an idea. You have to eventually see it for yourself.

I know many people who have found a compelling experience in VR-- whether it was from playing a game, seeing a story, watching a sporting event in a virtual space, or being in a virtual dance party. Of all the potential experiences you can have in VR right now, I totally get that a lot of people haven't found something that clicks for them. Talk of a single killer app that will convert everyone is the wrong approach. There are so many different uses of VR that I think everyone will come to it for their own reasons. I accept that my friend hasn't found that app that convinces him of a new way of doing things. I have faith that given time, the innovativeness of the VR community, and the inevitable improvement of the technology, enough "killer apps" will be created to cover a large range of interests, including my friend's.

Monday, January 18, 2016

A Culture of Critique

It's undeniably a good practice, as uncontroversial as eating well and getting more sleep. It's a skill that applies to every profession you can think of.  You can always get better at doing it, and when done right, it will always lead to making you and your team better.  And I believe it to be the difference between what rises to greatness and what falls into mediocrity.

Asking for critique and then mindfully acting on it is the best skill for improving everything you do, and it's also one of the hardest things to practice.  We're just not naturally wired to do it.  To ask for criticism is to overcome one of our base fears, to expose ourselves to being judged as less than we think we are.

It's hard to say what makes Pixar "a lightning in a bottle" studio, but one thing I did learn during my time there is how a strong feedback culture leads to better story.  I got to see many crappy "Pixar films" in my 10 years, but none of them were ever shown beyond the studio walls.  They were all internal screenings.  During production, all films are assembled as a complete viewable experience at regular intervals and shown to the crew and creative leadership.  These internal screenings are edited together using whatever assets are available at that time of production- rough story boards, scratch audio, and unlit renders– until the final screening includes all animated, lit and rendered shots.  After every screening, feedback is given and the director uses those notes to make the story better for the next iteration.

The directors hated these internal screenings.  They were forced to put their baby up for criticism every 4 months and let an audience of film making experts point out its flaws.  And even though the screenings could sometimes create doubt and disruption, they were done on every production without exception.  It was so entrenched in the process that it was just the cost of playing.

In the absence of a strong culture like Pixar's, it's very hard to ask a director of a creative project to be so naked in front of their team.  The stress of making anything creative is tough as is.  But I believe it was only when a film had gone through enough iterations and improvements that it rose to the level of excellence that audiences expect from Pixar.  And this is why Pixar's batting average is so much higher than the film industry's.

We're striving to build this culture at our studio and it's not easy.  Not only do you need to account for the extra process it takes to be producing a screening at regular intervals (we're calling them rough assemblies), but you also need to build trust between the crew and directors.  The crew needs to provide constructive notes while being sensitive to their director's vulnerability, and the director must convey receptiveness while taking criticism.

Something we've been trying to do is to create a "notes window" after a screening. After the screening, the entire team can send notes for fixed number of days.  Discussion threads spawn and can keep active until the window closes.  It's strongly encouraged that team members not only call out what isn't working, but to highlight what they love.  Once the window ends, all official notes traffic ends.  Our director can then digest, be creative and respond with a synopsis of the new direction in which she wants to go.  She doesn't need to acknowledge or defend her decisions against every note, but it's a tactful skill to make the team feel heard even if she might disagree with the majority.  The window provides some piece of mind as to when the director can expect to feel vulnerable and when she should feel protected.  We've found that it's too hard to be receptive and creative at the same time.

Given time and consistency, it's my hope that a culture of critique grows so fundamental to our process that it will just become the cost of playing.  So far, I'm very proud of the direction we're heading.  It doesn't hurt that a lot of us at Story Studio already come from a place like Pixar where this culture of critique is so strong and proven.

Thursday, November 19, 2015

Stop Climbing Out of the Uncanny Valley

I've been playing video games since my first NES.  I grew up with Mario, Sonic, Megaman and Fox McCloud.  Back then, the 8-bit machines forced stylization in character design: there wasn't enough pixels or compute power to render realism.  So the game makers worked with the constraints. Mario is a plumber because with only 16 pixel rows to work with, the designers needed the mustache and overalls to create definition for his nose, chin, and body.



Since that time, we have far outgrown 8-bit.  Games are now built on top of gigabytes of dedicated graphics memory and have a little over 2 million pixels of screen real estate.  Shouldn't more horsepower and more pixels lead to more beautiful realism and as a result, a more appealing character? And it's not just the tech that's getting better.  Today's artists are more skilled and have way more tools than those early developers.  Shouldn't the expertly crafted detail lead to a better product?

In my opinion, character design is getting worse, and it's all because of the Uncanny Valley.

In brief, the Uncanny Valley is a postulate widely accepted in robotics and computer animation that the more a fabricated character tries to act and look like a real human, the more we subconsciously notice what's wrong, and therefore reject the fabrication as creepy.

Here's an example of state of the art work in games from Ninja Theory.  The detail they're getting out of the high res scans is mind boggling - down to finger print accurate resolution.


And I have to admit that the still images of the CG character in neutral pose are looking pretty good. Where it all breaks down is when the character starts moving.  Even if the game makers capture a wide range of neutral and active poses, humans are so adept at discerning meaning from facial differences as little as millimeters apart, there's no algorithm that can blend between those poses that captures a completely believable human face.  Most who aspire to be actors have a hard time achieving believability, why do we think a computer could do any better?

What's frustrating is that even though today's game designers know they are making creepy characters, many of them have this unshakeable belief that with a little more technology and a few more clever ideas, they will eventually converge to the limit.  If you rephrase the Uncanny Valley premise from the creative's perspective, the more you try to climb out of the Valley, the harder it gets. Sounds like a classic case of diminishing returns.

Image grabbed from Manly Guys doing Manly Things.

I think there are two alternative paths that could lead towards better ways out of the Valley.

The first is obvious: don't even try climbing the difficult side.  Use our wide range of tools, talent and technology to make something you would never see in real world - like a stylized illustration come to life.  We are much better as humans in finding appeal in the fantastic than we are at forgiving flaws in the realistic.  In the 80s, tech limitations forced stylization.  Now, stylization is a choice worth making.

The second comes from thinking how film captures appealing character.  Humans are not very sensitive to temporal gaps in what we see and so flashing a moving picture 24 times a second is not distracting and has been used for over a century in the cinema.  We also know that a film of a good actor can be incredibly appealing.  So what if instead of capturing high resolution scans of static poses, we scan actors as they act at around 30 times per second.  There's already promising tech going in this direction coming out of teams like 8i.com, Uncorporeal, and Microsoft Research.


We can then play that 3D capture back at 30 poses per second without needing any blending for the in between frames, even if the game is rendering at 60 or 90 fps.  Where this becomes a problem is that game developers like a simplified model, usually in the form of a joint hierarchy.  With a simplified model, developers can drive a character that can be affected by interactive input, like when the player hits the "jump" button.  Furthermore, the joint hierarchy is a good way to compress movement data so you don't need to download 1TB of data to see a cutscene.

But I think these are the problems worth working on.  Instead of finding a way to fabricate acting with an algorithm, even if that algorithm is fed by scans and motion capture, find algorithms that solve the problems of using dynamically scanned acting.

Stop struggling to climb out of the Uncanny Valley on the hard side.  We should either walk back up the easier side towards more stylized characters, or avoid falling into the Valley all together by finding better ways to record actors in 3D.  In the latter case, the problems worth solving are around interactivity and compression, problems better solved with a computer anyways.