Monday, August 14, 2017

VR is not for story telling, it's for story making

3 years ago, I thought VR was the next great medium for story telling. My team and I worked hard to see if that thought proved out, and we did some outstanding work, work I'm proud of. But since my team was disbanded, I've had a moment to pause and reflect.

I now think I was wrong. VR is not for story telling, it's for story making.

We all love being told a great story, a hero's journey we can relate to, and perhaps apply something to be learned in our own lives. But the stories we are told are never our own. Whether we are told a story through a book, on stage, on screen, or in person, the order of events has already been decided and we the audience can have no influence over it.

Film and television have become the most popular story telling media because the tools that define them, composition and editing, can so easily convey a complex order of events in an accessible and readable way, especially considered how well the audience understands the language of those media. 

To think that VR, which takes away framing as a tool, could be a superior form of linear narration was naive. Now, I don't kick myself for that naivety. It wasn't obvious until we tried. At the beginning, any new medium is always underutilized or misused. Just like the first film makers thought the best use of a camera was to put it in front of a stage play, we early VR pioneers thought we would take away the 4th wall but could still tell the same kind of stories.

Over the past few months, I've taken a step back and thought about what VR is good at- the feeling of being there- and I've realized that instead of trying to tell stories, we need to have our visitors make their own.

The problem is that if you setup a world and then ask your visitors to craft their own story, that can be a blank canvas that's way too intimidating and inevitably unfulfilling. We need to find a way to define a road worth traveling that can lead to moments of awe, discovery and excitement but give the visitor enough agency and compulsion to walk the road for themselves.

I believe we can do this in 2 ways.

First, the compelling stories we make in our own lives are always with others and most often with our friends. VR must be a social experience.

Second, the power of a shared goal is the right kind of story hook. With well defined rules of interaction, a quest gives everyone in your party a common goal and its in the journey that you create memories together.

This is why companies are made, sports are played, teams are built. If we can capture that bond and excitement that comes from accomplishing something together, I think we'd be getting closer to what will make VR work.

In many ways, this is what multiplayer games try to do, but how my approach may differ is that I think the design of the experience should focus on the journey's emotional arc, and less on the feeling of accomplishment that comes at the end.

I am excited to be working in the social VR space and to be thinking up new ways of how friends can come together to go on virtual adventures.

Sunday, August 6, 2017

Siggraph 2017 marks a turning point

I came away from Siggraph this year inspired, and surprised! I haven't felt this way about Siggraph in a while.

I remember my first Siggraph in 2004 where I was giddy and overwhelmed by the amazing things happening in computer graphics. In fact, I think that first Siggraph was the reason I decided to make CG my life's work. Year after year, I thoroughly enjoyed watching the improvements in simulation, shading & lighting, geometry manipulation, and rendering.

But it was around 2011 when I began feeling like the paper titles were getting longer and more obscure to compensate for the fact that the contributions felt like they were making smaller and smaller steps forward. The production talks continued to be somewhat interesting, but they became more about efficient pipelines and entertaining anecdotes about adventures in technical creation. It felt like innovation in the field was plateauing.

It was also around this time where I had this sinking feeling that computer animation and visual effects had become a mature industry, and I found myself sneaking away to all of the realtime sessions. That feeling eventually grew into a decision to leave software rendering and go into realtime VR.

And I wasn't the only one, I felt like the "Advanced in Realtime Rendering" course kept getting bigger and bigger as attendance in path tracing and PBR rendering talks shrank. Worse than shifting attendance, many were opting out of the conference altogether as overall registration declined since its peak in the 2000s. More of a concern for the organizers, companies were deciding not to exhibit, and exhibitor fees are a serious revenue stream for Siggraph's governing body.

Since I've moved over to VR, I have also started attending GDC, where I found myself reinvigorated by all of the interesting problems still ahead in the area of realtime. In contrast, Siggraph just wasn't exciting any more.

But this year, it felt different. Despite the fact that this was the smallest exhibition I've seen to date, emerging venues like the VR Village and the new VR Theater were signs that Siggraph is pivoting in a new and promising direction. Furthermore, research and talk topics are swinging towards VR, AR and machine learning techniques, areas that have a lot of undiscovered promise.

I have a theory, most likely a well studied theory in Economics, that there is a cycle in technology development between spurts of innovation as new promising areas of research are uncovered and plateaus of maturity, and this cycle drives a lagging curve of commercialization.

time is not to scale and growth is conceptual
My idea is that as new areas of innovation show promise and gain momentum, ways of turning that rapid progress into thriving business soon follows. But as the innovation curve starts to plateau, the commercialized space gets over saturated and goes through a correction. I think what's happened over the last 7 years with the decline of exhibitors at Siggraph is a sign of that correction.

But what's so exciting about what I saw at this past Siggraph is that although the exhibitor attendance was its lowest in the past 12 years, individual attendance feels like it's creeping up and the areas of research felt fresher than ever.

I think Siggraph 2017 may mark a turning point, showing signs that we are in the first few years of an innovation spurt, and I think it will take a few more years before we see the market respond.

There were 4 particular moments that stood out for me at the conference.
  1. I got to experience Neurable's brain pattern recognition technology for VR. It was not only magical but incredibly stylish integration with the Vive. It does some interesting tricks with pattern matching, sensing voltaic patterns on the scalp, and using machine learning to turn that sensor data into intent. It's very comfortable and absolutely works.
  2. I've been excited about point based rendering as a way to stream volumetric capture and the new renderer from Nurulize, called AtomView, shows that you can see through the lack of topology and find appeal in simply displaying points. I think an audience enjoys "filling in the blanks" with their own imagination over being shown an approximation that is a little wrong, leading to the Uncanny Valley problem. Also, it's so easy to create stream based, dynamically optimizing solution when you're only throwing points to the GPU.
  3. The talk on how the designers of Google Earth VR made it a better feeling app had some great insights and design decisions. I'm especially proud of Per Karlsson who had some great ideas in the design of the app. I had the honor of supervising Per during my time at Pixar.
  4. One of my favorite traditions of Siggraph is the Electronic Theater. It's still one of my greatest wishes that I could have been in the crowd back in 1986 when Luxo Jr. was first shown to the world at the Electronic Theater, and a room full of scientists erupted to see their research turned into magic. Most of the projects are really inspiring and remind me why I love making a art with technology. My top 2 favorites were emotionally powerful and ingeniously executed. They were Scrambled by Polder Animation and "Happy Valentine's Day" by Neymarc Visuals.

Saturday, July 22, 2017

The growing skill gap

For those of us lucky to be working on the bleeding edge of tech, we get to work on some of the coolest problems in automation, artificial intelligence, robotics, and virtual and augmented reality. We are driven to innovate not just because they are cool problems but because we believe what we invent makes life better for everyone. Technology like self driving cars, robot driven assembly lines, image classification, speech recognition and translation all have the promise to make our lives better.

The hope of any enriching technology is that it frees our time to pursue a richer life. But if that is our goal, we need to be mindful of the skill gap we are creating between those who are early adopters of new technology and those who adopt later. And this gap could actually be doing more harm than good.

A lot of people contribute to society by taxiing passengers, working on a factory floor, labeling and sorting information, and translating foreign languages - skills that are being made obsolete by those same futuristic technologies we work on. As tech continues to get better we may invent ourselves out of usefulness. Considering how hard it is to learn new skills to catch up on an accelerating wo, technological progress will lead towards a bankruptcy of purpose.

If not addressed, this bankruptcy can have dire consequences on progress. I believe we are already seeing signs of problems with the dramatic shifts in world politics away from science backed thinking and towards populism, nationalism and a fear of reasoning. This is probably why messaging of climate change is still not getting through.

Progress is the right path forward, but is there a way we can also solve the problem of the growing skill gap?

This is where I think virtual reality may be able to help. My favorite feature of VR is how it has created the most intuitive interface for how we interact with computers. Up until now, our interaction has been a difficult abstraction through mouse clicks and keyboard strokes. With VR, I see new users pick it up within seconds. We've all grown up knowing how to move, look around, reach out to touch and grab things; VR just taps into those instincts. We can convey so much complexity with the ease of natural gesturing and soak up so much information through the ease of immersion.

It's my belief that we'll be able to take advantage of this facility by creating educational VR experiences that will make it much easier to learn new, relevant skills. Unfortunately, making virtual content also suffers from a skill gap. Those who know how to create in this new space are mostly game makers who are passionate about making games. Those who are passionate about education technology are still learning the skills to create useful virtual experiences.

That's why it should be an important goal for this new industry to make it easier to be a content developer. Heavily documented and community backed game engines like Unity and the Unreal Engine are a good start but are still designed around making games. We need the next generation of VR developer tools to make it easier for educators to get involved.

The progress of technology is inevitable, but we need to remember that technology should also be used to close the skill gap it creates. And it starts by closing the skill gap with the tools we use to teach.

Friday, July 7, 2017


About 2 months ago, my virtual reality studio was shut down. The closure was sudden and a stunning punch to the gut. It was a decision I very much disagreed with, but it was a decision I can understand considering the larger context of the VR industry and the difficult path ahead.

Before going onto what may be next, I've needed some time to recover from that blow, and reflect. In writing this public diary entry, my hope is that it will not only help me plant a sign post for the journey so far but perhaps serve as a way to leave behind what needs to be left.

Considering what we were able to accomplish in 3 years, I am proud of the work we did to see what this new medium could look like. Lost, Henry, and Dear Angelica were as much experiment as they were entertainment, and our mission was far from accomplished. But any pioneering work needs to start somewhere. When these pieces first premiered in 2015, 2016, and 2017 respectively, they contributed new nouns and verbs to how we talked about VR entertainment.

The thing I am proudest of is the team we assembled. Story Studio was a group of some of the best computer animation, visual effects and game makers in their industries who were all excited to stake new claims in a risky frontier. I know this because during the interview for every team member, I took a moment to warn of the dangers of being so far from civilization. I spoke of how I was excited to be discovering what no one had ever done before, but that as exciting as the discovery was, it was difficult to predict what was around the next corner, how long the journey would take, and what setbacks we'd need to overcome. I'd even be as honest to say that I couldn't guarantee the studio would still be around in 6 months. Some interviewees blanched at the uncertainty, but I knew those that came back with eyes wide open and excited about what lies ahead, they would make great team members.

I am confident that the seeds of this team, now having scattered out into the world, will go on to help grow the forests that will define our cultural and technical landscape.

Then there's what needs to be left behind. I've struggled with the guilt that we weren't able to convince our leadership we were a bet worth making. I've struggled with a bruised ego that inevitably comes from a fall like this despite how lucky we were to get the opportunity in the first place. I've struggled with the fear that capturing the kind of lightning we had only happens once in a lifetime. I need to constantly remind myself that all of these suffocating thoughts aren't worth my energy if I'm to get on with my journey.

In early 2014, I made the difficult decision to leave the comfort and renown of Pixar and embark on a journey to do something novel, scary and groundbreaking. I would have been happy to find a creative problem worth solving that may have taken 3 to 5 years and given me the opportunity to see what it was like to grow a team from the beginning. I was ecstatic to find virtual reality, a problem so big that it wasn't just about creating a new product on an existing platform, it was about figuring out a new medium-- a problem that would take the rest of my career. The horizon line stretched far beyond an unknown and perilous frontier. It was daunting, inspiring, and irresistible.

This closure is a particularly humbling setback and it's given me a moment to pause and reflect on if I should turn around. Am I so driven towards the promise of gold in California that I'm blind to the fact it may be a fool's journey?

I don't think so. There are enough brilliant, thoughtful, and future savvy people that believe in VR. It remains undeniable that VR has the potential to change our relationship with digital content, making it a richer, expansive and more meaningful experience. I continue to be awed and inspired by the innovation that continues to happen on a weekly basis.

As for what's next, I have decided to continue towards the horizon of what VR may become, even during this particular stretch of desert. I may plot a slightly different course than what we were doing at Story Studio, but my goal is still to give a visitor something that is immersive, memorable, and emotionally engaging. My sights have changed in that now I also believe that for VR to truly find itself as a medium, it must be an experience you share with your friends. But that's for a future post...

It's time to dust myself off and push onward.

Sunday, October 23, 2016

Leading via Questions

There's a pivotal moment in the lifetime of a growing team where its leadership no longer has enough information to be responsible for how that team gets things done. And if both team members and leadership don't recognize their roles need to change, it can be frustrating for everyone.

When a team is small and the leadership has the capacity to be in the trenches, there's a lot of efficiency in having your more experienced decision makers steering the work. By working so closely with everyone, leaders are operating with shared context that team members can trust and the leaders can course correct quickly based on all of the feedback they get.

There is a point, however, where the responsibilities of leading and protecting a larger team make it more difficult for leadership to spend time in the trenches. When that happens, if the leaders keep involving themselves in areas in which they are now so woefully uninformed compared to those they lead, it becomes meddling-- a serious impediment to getting things done.  Up until that point, the natural relationship was one of leader and followers and so it's not evident that there's a moment where the team knows a lot more on how stuff gets done.

Over the past year, with my studio having grown to more than 25 people, I've been working on this role change. There's a few interesting problems to solve. If I am accountable for, but no longer responsible for how stuff gets done, how do I pull that off? How do I help those team members that are not used to being decision makers develop this skill? What are the skills I need to grow in this new role and how do I improve? And how do I do all of this while making sure that trust flows both ways?

I have one solution that has had some success. I try to ask questions like I'm the student and they're the teacher. If I am now less informed on how things are getting done, it makes sense that I'm the one who needs to be taught.

This has had several effects:
  1. By making it clear that I respect and want to learn from what the team is doing, they feel trusted and empowered. 
  2. Because I have a lot of decision making experience and a 10,000 foot view, the type of questions I ask can probe and reveal problems the team hasn't seen from their angle. If I'm not satisfied with where things appear to be going, asking followup questions can either help the team realize they haven't thought something through or help them craft an argument that is satisfying for me. In both cases, the team is the one coming up with the answers. 
  3. Over time, I've noticed the team is getting better at anticipating the questions I would ask and in effect getting better at their own decision making. 
  4. For those moments when I still need to make the decision, there's now a clear relationship for how the team gets involved leading up to that decision and for when unexpected consequences may need correcting.
After a year, I still make mistakes where I'll give trench level direction that comes off as uninformed and meddling. It's easy to lapse back into the comfort of being the one who decides, a skill I've developed and gotten good at over the last 6 years. When this happens, it's been really exciting when my team, having developed leadership skills of their own, are now actively coming to me, letting me know what context I may have missed, and teaching me something about how I can be a better leader.

Sunday, October 2, 2016

The Finishing Trolls

We've recently finished our third project at the studio, and as proud as I am of it, I'm glad these final months are over. It's not the desk work that tired me out - that blood and sweat comes from my amazing team. The hard part about finishing is having the strength and discipline to know that it's good enough. As a producer, I was constantly tested in making the hard decisions necessary to ship it even when more time and effort could always make it better. And that's the haunting fact about making something creative— you can always make it better. But until you actually put it in front of an audience, it doesn't count. One of my favorite lessons from Pixar, expressed elegantly by John Lasseter, is that a film is never finished, it's just released.

The problem with working on a creative and highly technical product like what we make at Story Studio is that over the long amount of time it takes to develop, produce and finish a VR movie, you can lose what excited you about the project at the start. And then you start worrying if it's not just fatigue, but something more fundamentally wrong.

And this isn't new to anyone who creates. Self-doubt and angst are trolls that move in during those last few months of finishing. But in a medium as young as VR, I feel like the trolls are especially tricky. Not only do you begin doubting the project, but you also begin worrying if the audience will even get it. We've now spent so many more hours in VR than the average consumer that I worry my experience will mislead us into thinking we know what will entertain. One of the tricks to producing is to find the strength to ignore those trolls and push through that last mile.

I have to thank my experience at Pixar for helping me stay strong.  While there, I was lucky enough to see 3 feature films through to the end— Cars, Wall•e and Up.  I got to experience what it's like to spend so long on a project that you begin focusing on the flaws and forget what makes the film great. For each of those projects, as much as I and my fellow team members cringed to know all of the things we just didn't have time to fix, all of those films came out as beautiful, entertaining and critically acclaimed films. If I could be tricked into doubting Pixar films, and then be blown away by their reception, then those trolls aren't worth listening to.

As for knowing if the young VR audience will get it and be entertained, that's part of frontier life with this new medium. The only way we'll know if what we're doing works is to have the faith and guts to put something finished out into the world and see what happens. If we spent all of our time trying to perfect our experiences, we would never learn anything. Furthermore, with each VR movie we complete at Story Studio, it helps me build up a tolerance against those tricksy trolls.

That all being said, I can't wait to show you what we've done.  It's inspiring, unexpected, and I guarantee it's something different than anything you've experienced before.

Sunday, July 17, 2016

AR will improve how we live, VR will change how we live

I had an interesting conversation with a friend this past weekend that got me thinking about the promise of augmented reality vs. virtual reality. My friend, who's opinion of technology I respect a great deal and who has enough VR and AR experience to have an informed opinion, said that he could see how augmented reality will be big, but he still didn't understand virtual reality. This makes a lot of sense to me, and I still remain as bullish as ever on virtual reality. VR has the potential to change culture which, for me, is more exciting than augmenting it. Because of this potential, however, it's much harder to envision what the future of virtual reality looks like compared to augmented reality's.

When you think of the potential of AR, it's easy to think of how a heads up display would augment driving, navigating, classroom learning, cooking, meetings, shopping, etc. These are activities we do every day and to add an effortless layer of technology that improves that experience is a no brainer. But most of the uses I can think of for VR don't improve existing activities, they try to define a new way of doing things. Some of my favorite VR learning apps don't take you to a classroom, but transport you across the solar system or scale you down to the size of an atom. As support for a virtual desktop improves and eventually starts taking advantage of the depth dimension, it has the potential to redefine what an office looks like. If you follow our work at Oculus Story Studio, you can see how VR has required us to rethink how we tell stories and how the audience experiences them.

Immersion, the defining feature of VR, is a problem and a benefit: it's a problem because every VR experience needs to reward all of the user's attention; it's a benefit because if we succeed, all of that experience is something you've never had before. The leap your imagination needs to take to add technology to what you've already experience is a smaller one. Imagining a completely new experience that will redefine how you go about your day requires a leap of faith, and will never be fully convincing as an idea. You have to eventually see it for yourself.

I know many people who have found a compelling experience in VR-- whether it was from playing a game, seeing a story, watching a sporting event in a virtual space, or being in a virtual dance party. Of all the potential experiences you can have in VR right now, I totally get that a lot of people haven't found something that clicks for them. Talk of a single killer app that will convert everyone is the wrong approach. There are so many different uses of VR that I think everyone will come to it for their own reasons. I accept that my friend hasn't found that app that convinces him of a new way of doing things. I have faith that given time, the innovativeness of the VR community, and the inevitable improvement of the technology, enough "killer apps" will be created to cover a large range of interests, including my friend's.