Wednesday, September 27, 2017

stories I've experienced: Passengers (2016)

On an 11 hour direct flight from San Francisco to Tokyo, I find myself with the kind of time where I can catch up on movies that I wouldn't even consider watching on a lazy Sunday. There's something liberating about being crammed in a big flying metal tube for a long period of time. It's like the time is already forfeit so I don't mind if it's wasted on a potentially bad movie.

But I was pleasantly stunned to find an under appreciated gem in my selection, and unplanned by me, I watched it in the perfect setting.

Passengers (2016) is a small ensemble sci fi romance drama starring Jennifer Lawrence and Chris Pratt that passed through the theaters unnoticed. Two space faring passengers find themselves woken up way too early on a ship traversing an expanse that takes 100 years to cross, dooming both of them to live out their remaining years as castaways on a cruise ship. Pratt's character is woken because of misfortune; Lawrence's character is woken because of desperation. They eventually fall in love, but in a way that feels earned and meaningful.

I had very low expectations for this movie (it has a 30% on rotten tomatoes) and only had it on my list because I'm a sucker for space travel themes, high production value sci fi design, and Thomas Newman on score (there are some really sweet moments that remind me of Wall*e because of Newman's sound). But in the last act of the film, my jaw was dropping as much from the tense climax and heartfelt ending as from the complete under appreciation for this film. I cared for these characters. I felt for their decisions, their mistakes and their emotions. Their motivation, although simple, was believable and relatable in an unimaginable situation.

These two passengers start out feeling like it was a life sentence to live out a luxurious yet isolated existence traveling across the stars without others around them to provide purpose or validation. But by the end, they find all they needed was each other. Romance wise, it's a touching sentiment, but it got me thinking how this space ship, with around 5000 migrating passengers was already on a mission of isolation from an overpopulated Earth. Everyone on that ship felt like 5000 was enough other people to bring them meaning when they reached their final destination.

It's a very believable premise and it got me thinking how we all probably have a number in our mind of how big our community needs to be for us to feel fulfilled. In a time when technology allows us to create connections well beyond the Dunbar Number, I worry that we all think our numbers are bigger than they need to be, which causes inflation to the point where each relationship is so devalued that even the sum of many leaves us wanting. I see this story as a suggestion that the lower our number is, the more satisfied and connected we may feel with the community we share. Having a number as low as 2 may be a bit overly romantic, but I do feel like a lower number is something worth aspiring towards.

If the inevitable evolution of virtual reality is to create communities that mirror social value in real life, I wonder if it will be a technology that further dilutes our feeling of belonging or if it provides us a way to create communities sized to fit each one of us. As some one who is invested in seeing VR be a medium that makes us better people, I push and hope for the latter.

Tuesday, September 19, 2017

Taking off the manager hat. For now.

Over the past 7 years, I've been a manager and supervisor in various roles, first at Pixar and then from the ground floor at Oculus Story Studio. When I started down this path, I had an advantage. I approached managing as a challenge altogether different from the challenges I had overcome as a team member. For me, I wanted to know if I even had the skills to be a professional leader, and I knew that focusing on developing these skills meant growing in areas that were new to me.

The classic and natural mistake most new managers make is thinking that this transition is about scaling their influence with the skills they already possess. The fact is that as a leader, I needed to start at the beginning again. I needed to learn how to make sure my team is the strongest version of themselves. That usually means I don't get to do exactly what I want, but what my team needs. Being humble about this restart made me more receptive to the many hard challenges I've faced.

What makes management fun, and challenging, is the diversity of problems I've had to solve and the creativity required to solve them. What makes management hard is that I rarely get into flow for myself. I've had to play the role of the champion defending against forces from outside the walls. I've to be an advisor who can find inventive ways to unblock. I've had to be the parent who dishes out tough love. And I've switched between these roles many times in a day. I've come to find that as a manager, I've often needed to fall on the grenade of context switching so the team finds their flow.

I enjoy managing and have found that it to be incredibly fulfilling. But not wholly satisfying. What I've been missing is that feeling of making something with my hands, pointing at it and saying "I made this thing." Instead, I've pointed at my team and been sustained by pride. Pride goes a long way, and I've been blessed with great teams who filled my cup, but in the end, I realized I got into VR because I wanted to be on the ground floor of building a new medium.

With recent events in my career, I've been given an opportunity to take a step back from managing and start building as a creative engineer.

It hasn't been an easy choice. It's understandable to feel like once you become a manager, you're stuck on that track. After having played the role of champion, advisor, and father, you earn a kind of respect and appreciation that soaks into your identity, and the lack of it makes you feel less than.

But this is also where that initial mindset helps me again. Because I approached managing as an altogether different skillset, I can recognize it as a hat I chose to put on. Now, it's a hat I can choose to take off.

My skills and experience as a manager will still be there for the day that I want to return to it, but for now, I want to roll up my sleeves and make some amazing VR.

Monday, September 4, 2017

stories I've experienced: The Asylum Escape Room

I love puzzles. I love immersion. And I love a team effort. So this labor day weekend, I tried out one of the newest escape rooms by Clockwise Escape Rooms called "The Asylum". The experience takes place in a Nixon era mental ward where you have to solve your escape before the staff comes back from their one hour meeting. The premise was simple but evocative, and it was beautifully executed with smart era consistent set design. We even got to wear patient gowns to complete the immersion.

I look especially crazy on the far left

I went with a good friend and got paired with 2 other groups to make a total of 8 team members. We had a few experienced team members but most people were new to escape rooming. Right when the door shut behind us and the clock started, any feeling of being strangers melted away as the adrenaline kicked in and our shared task became our shared focus.

The puzzles themselves were clever, diverse, and rewarding. Never did I think that a puzzle felt too obscure and never did I feel like the clues were ham-fisted. Every time we solved a clue, it felt like we were in flow. A few of the puzzles and their inventive solutions were so good, you felt like giving a high five to the game master herself.

As much as I'd like to talk about some of the particularly cool solutions, spoilers would have their full effect, and so I'll focus on the high level. Here's my thoughts on what worked, how it could have been better, and how it can inspire VR experiences.

  • The lighting design decisions alone really gave a sense of progression through an amazing story experience. The first part of the experience takes place in a sterile white overly lit ward room and the second half takes place in an underlit dark wood paneled warden's office. And within those two acts, there are event based lighting changes as you progress through the clues. The contrasting acts and the variety in the event based lighting really kept a compelling pace which deepened the suspense and theme.
  • But what I really missed was a soundtrack. Having some brooding background music that would shift in tone and theme as we made progress or would crescendo as we got closer to solving a puzzle would have made for an epic shared story making experience. I wouldn't even need the music to be motivated by a set consistent device. In fact, having music be part of the ambiance would enhance my belief that I've stepped into a story.
  • Sound effects were used quite well across a variety of devices, but what I wanted was a clear sound effect for when a puzzle was solved. Our team was a clever bunch and did a pretty decent job of communicating all of the clues we were finding. But there were times where we were doing such a good job of plowing through the puzzles that we weren't communicating what we'd solved and what clues we'd use. Sure we wasted some time as some folks would pick up and ponder solved clues, but that's not what bugs me. The opportunity that was really missed is giving the group a clear moment to share in a team member's victory.
  • During the experience, I solved a clue here or there, but what I loved was how good I felt when another member of the team solved a puzzle. It especially felt good when everyone was thinking on the same clues and it was through talking it out that somewhat had a spark of insight. Having a shared goal, and then having shared victories on the path towards that goal created an immediate kinship. We were woohooing, fist bumping, and high fiving with folks we had just met 10 minutes ago.
  • I'm a big believer that a shared quest creates strong bonds, and I continue to believe that social adventures is the key to what will make VR exciting. But there's a design constraint we need to consider if we're to make escape room work in VR. Escape rooms are fun because everyone gets to use a full range of tools they're very familiar with in inventive ways - tools like seeking, opening, touching, pushing, talking, and listening. The problem we have with VR is that the tools we use to interact in a virtual environment are under developed. We're much closer to creating interactions that universally improve immersion over game controllers and keyboards, but the fact that we're still figuring out mechanics for simple actions like opening a drawer, flipping a switch, picking up an object, entering in a key combination, and even for interactions as simple as moving around means that the puzzles we'd make in VR need to be designed very differently if we hope to make them fun.
  • After 37 minutes, having used one hint, we solved our way to an escape and earned the #1 spot in the winner's circle! By that time, we were beaming, high fiving, and collecting phone numbers. And then we were then confronted with our last puzzle: "What should our team name be?" The funny thing is that we had become a team before we even had a name. That's how powerful a shared task is, especially under a time constraint.

Sunday, August 27, 2017

stories I've experienced: The Speakeasy

Inspired by Jesse Schell's blog "Things I finished", I'm starting to write down my thoughts on stories I've experienced. It's a great way to get ideas flowing and to archive all of the stuff my mind is soaking up. I've been thinking a lot about what story making would look like in VR and so a lot of my reading, TV, movies, theater, video games, and even D&D sessions have been focused around cracking this nut.

This past May on a Saturday night in an undisclosed location in San Francisco, I had the pleasure of dressing up in prohibition era outfit, made complete with a sharp looking fedora, to step inside the immersive theater experience called The Speakeasy. It was an amazing night, brilliantly acted, intelligently written, and it was my favorite immersive theater experience I've had, having seen Sleep No More and Then She Fell in NY.

The day after, I was so excited and inspired that I had to write down a bunch of thoughts. I'm finally getting around to posting them. Warning: there are some spoilers! Although considering that I probably only got to follow about 5% of the many story lines, it's not that much of a spoiler.
  • The complexity of the experience was in the number of story lines, and not the stories themselves. Each story line on its own was simple enough to describe in a few sentences and based on relatable tropes. It made it easier for the audience to feel empathy because they weren't spending their time piecing together a narrative puzzle. Sleep No More and Then She Fell are great experiences but poor narratives. It's so hard to read a story without dialogue.
  • The Speakeasy proves to me that dialogue in immersive theater works! Both Sleep No More and Then She Fell mostly use interpretive dance and I always wondered if the non-linear format just didn't work with words. Given a familiar setting of a prohibition era bar, even a setting all of us have only experienced through movies & books, and simple enough topics, it's okay if the audience misses 95% of the dialogue and walks into the middle of prose. We pick up so much on body language and familiar tropes that as long as the dialogue is smartly and clearly written, we can easily drop one story line and pick up midway in another.
  • The moments that felt the best in immersive theater are the ones where the actor improvises on the fly and the audience knows they were going off-script. Much like how a comedian may respond to a heckler with a great come back, audiences love on-the-fly wit. It feels like a display of mastery akin to watching a great athlete adapting to the field. It also feels like a special moment that could never be repeated. Everyone involved feels special. I wish this happened more but when it did, it was magic.
Dressed to impress and immerse
  • One of the problems with immersive theater is that you can feel like you're doing it wrong. There were times where I had to fight down a fear of missing out. Was I using my limited time wisely? Was this the most interesting thing going on at this particular moment? Certainly some story lines must be better than others. But when I did discover a story line that was really compelling, the feeling that I had discovered it felt really good. I'm not sure how to address the fear of missing out. Perhaps it's unavoidable and the other side of the coin of what makes immersive theater fun.
  • Some of the most memorable moments were when the actors engaged us. It could be as simple as looking us in the eye, asking a rhetorical question, patting us on the back, or handing us something. Handing off an object was especially effective. It makes the connection with the story tangible, literally. This resonates with what we found while developing story experiences in VR. 
  • The way the play began for us was inspired for the format. We started in a bar with assigned seating and so it felt more like a classic theater experience. It really takes advantage of the "antechamber" idea we found in VR story telling. Before giving us too much choice, they made sure to warm us up to the idea that the story is happening all around. To do that, the bar scene uses paradigms we were all familiar with: we had assigned seats; we were a captive audience; the lighting helped direct our eye. We were let loose to choose where to go and what to do only after we were given time to feel immersed.
  • The story line we followed the most was about this particular cabaret girl who felt enslaved to the owner of the Speakeasy as she worked to pay off her brother's gambling debts. To see her emotional arch play out in several venues was enthralling- as a voyeur behind the mirror, as a member of the crowd watching her lash out at her brother in the casino, and then as an audience member watching her as she performed on stage. It was especially powerful to see the anguish on her face while she was dancing and know the reason. It felt like a level of intimacy you cannot experience in any other medium because the choices we made to follow her made it our story too.
  • There was a particular moment where the cabaret girl with the deadbeat brother flipped out on stage and then ran into the cramped corridors. She began to tear off her revealing dress in an elaborate display of rage and despair all the way down to a body suit. The act was so elaborate that it became a dance performance. It helped frame the moment as something we were watching instead of something we were participating in. It would have been difficult for her to express her rage as well as improvise against the variability of the audience in this confined space. Unfortunately, this broke immersion for me. I would have loved it if the actress had a way to flip out in the context of whatever the audience was doing in that small space. If it was busy, she could shove passed people. If it was only a few people who stared at her, she could lash out and say "What are you looking at?" To see her emotionally and physically stripped bare without losing immersion would have left an impression that would have been unparalleled.
  • The act of chasing a story thread was really fun. I loved the moment where our cabaret girl ran out and we were so compelled to just get up out of our seats and follow her. I am wondering how you could capture this same idea in VR considering the difficulty with locomotion. My only thought at this time is to design an experience with zero-G locomotion mechanics so the audience feels like they can follow with grab and pull. Teleporting always breaks presence for me.
  • There were times where I could tell the theater design was working around the limitations of an audience who didn't know what they were doing. For example, the beginning scene at the bar was designed so everyone was seated in known spots the actors could work around. In VR, the story makers have a lot more control over a visitor's presence, how they interact ability, and whether their voice is audible, or even how it sounds. This gives a lot of power to a VR host to control the moments where agency may get in the way of readability. If the audience member violates the rules of the world with that agency, you can easily mute action and voice.
  • One of the most impactful scenes for me was a quick act of infidelity with Viola, an innocent cabaret girl learning how to hustle, and the "Hardware guy", an unremarkable bloke with a weak moral compass. We all watched this intimate moment behind the glass as what played out was heart wrenching. It was a type of moment that would have been told with a wide framed, long shot in cinema, letting the slow burn soak into us. But the fact that this wasn't shot behind a lens and that it was happening live right in front of us made it almost too real and incredibly memorable.
  • The mechanics of being able to see behind the dressing room mirror and being able to look into the owner's office through little windows felt fantastic (and fantastical), but broke our immersion in the space. It felt like we were stepping out of our character and into the role of a ghostly voyeur. Especially considering how dressed up we all got for it, it felt like all moments should have been grounded in the roles we were playing. That being said, the moments where the story blinked into dance or memory or theatrics where the actors no longer inhabited the physical space but acted out dreams, memories or flash backs were really cool. I was willing to suspend my disbelief for those context switches. What may have made the experience better is that when we were blinked into a detached scene, along with changing the lighting and sound, it would be amazing if the set mechanic changed as well. Imagine a moment where the audience didn't know there was a one way mirror in the set, and only during one of these blinks does the mirror suddenly become transparent. If the set could shift to have the physical space reflect the emotional space the actors were blinking into, it would be amazing. This is expensive and complicated to do in the physical world, but in VR ...

Monday, August 14, 2017

VR is not for story telling, it's for story making

3 years ago, I thought VR was the next great medium for story telling. My team and I worked hard to see if we were onto something. We did some outstanding work, work I'm proud of. But since my team was disbanded this past May, I've had a moment to pause and reflect on the first steps we took. Looking back, I now think I was wrong.

VR is not for story telling, it's for story making.

We all love being told a great story, a hero's journey we can relate to, and perhaps apply something to be learned in our own lives. But the stories we are told are never our own. Whether we are told a story through a book, on stage, on screen, or in person, the order of events has already been decided and we the audience can have no influence over it.

Film and television have become the most popular story telling media because the tools that define them, composition and editing, can so easily convey a complex order of events in an accessible and readable way, especially considered how well the audience understands those media after having grown up with them.

To think that VR, which takes away framing as a tool, could be a superior form of linear narration was naive. I don't kick myself for that naivety, because it wasn't obvious until we tried. At the beginning, any new medium is always underutilized or misused. Just like the first film makers thought the best use of a camera was to put it in front of a stage play, we early VR pioneers thought we would take away the 4th wall but could still tell the same kind of stories. It takes lots of trial and error before a medium has its first Citizen Kane.

So over the past few months, I've taken a step back and thought about what VR is truly good at, the feeling of being there, and I've realized that instead of trying to tell stories, we need to have our visitors make their own.

The problem is that if you create a world and then ask your visitors to craft their own stories within that world, that is a blank canvas that's way too intimidating and inevitably unfulfilling. We need to find a way to define a road worth traveling that can lead to moments of awe, discovery and excitement but give the visitor enough agency and compulsion to walk the road for themselves.

I believe we can do this in 2 ways.

First, the compelling stories we make in our own lives are always with others and most often with our friends. VR must be a social experience.

Second, give those group of friends a shared goal and a suggestion of how to get there. A quest is the right kind of story hook and its in the journey that you create memories together. If we can capture that excitement that comes from bonding together and accomplishing something as a team, I think we'd be getting closer to what will make VR work. In many ways, this is what multiplayer games try to do, but how my approach may differ is that I think the design of the experience should focus on the journey's arc and less on the feeling of accomplishment that comes at the end.

I am excited to be working in the social VR space and to be thinking up new ways of how friends can come together to go on virtual adventures.

Sunday, August 6, 2017

Siggraph 2017 marks a turning point

I came away from Siggraph this year inspired, and surprised! I haven't felt this way about Siggraph in a while.

I remember my first Siggraph in 2004 where I was giddy and overwhelmed by the amazing things happening in computer graphics. In fact, I think that first Siggraph was the reason I decided to make CG my life's work. Year after year, I thoroughly enjoyed watching the improvements in simulation, shading & lighting, geometry manipulation, and rendering.

But it was around 2011 when I began feeling like the paper titles were getting longer and more obscure to compensate for the fact that the contributions felt like they were making smaller and smaller steps forward. The production talks continued to be somewhat interesting, but they became more about efficient pipelines and entertaining anecdotes about adventures in technical creation. It felt like innovation in the field was plateauing.

It was also around this time where I had this sinking feeling that computer animation and visual effects had become a mature industry, and I found myself sneaking away to all of the realtime sessions. That feeling eventually grew into a decision to leave software rendering and go into realtime VR.

And I wasn't the only one, I felt like the "Advanced in Realtime Rendering" course kept getting bigger and bigger as attendance in path tracing and PBR rendering talks shrank. Worse than shifting attendance, many were opting out of the conference altogether as overall registration declined since its peak in the 2000s. More of a concern for the organizers, companies were deciding not to exhibit, and exhibitor fees are a serious revenue stream for Siggraph's governing body.

Since I've moved over to VR, I have also started attending GDC, where I found myself reinvigorated by all of the interesting problems still ahead in the area of realtime. In contrast, Siggraph just wasn't exciting any more.

But this year, it felt different. Despite the fact that this was the smallest exhibition I've seen to date, emerging venues like the VR Village and the new VR Theater were signs that Siggraph is pivoting in a new and promising direction. Furthermore, research and talk topics are swinging towards VR, AR and machine learning techniques, areas that have a lot of undiscovered promise.

I have a theory, most likely a well studied theory in Economics, that there is a cycle in technology development between spurts of innovation as new promising areas of research are uncovered and plateaus of maturity, and this cycle drives a lagging curve of commercialization.

time is not to scale and growth is conceptual
My idea is that as new areas of innovation show promise and gain momentum, ways of turning that rapid progress into thriving business soon follows. But as the innovation curve starts to plateau, the commercialized space gets over saturated and goes through a correction. I think what's happened over the last 7 years with the decline of exhibitors at Siggraph is a sign of that correction.

But what's so exciting about what I saw at this past Siggraph is that although the exhibitor attendance was its lowest in the past 12 years, individual attendance feels like it's creeping up and the areas of research felt fresher than ever.

I think Siggraph 2017 may mark a turning point, showing signs that we are in the first few years of an innovation spurt, and I think it will take a few more years before we see the market respond.

There were 4 particular moments that stood out for me at the conference.
  1. I got to experience Neurable's brain pattern recognition technology for VR. It was not only magical but incredibly stylish integration with the Vive. It does some interesting tricks with pattern matching, sensing voltaic patterns on the scalp, and using machine learning to turn that sensor data into intent. It's very comfortable and absolutely works.
  2. I've been excited about point based rendering as a way to stream volumetric capture and the new renderer from Nurulize, called AtomView, shows that you can see through the lack of topology and find appeal in simply displaying points. I think an audience enjoys "filling in the blanks" with their own imagination over being shown an approximation that is a little wrong, leading to the Uncanny Valley problem. Also, it's so easy to create stream based, dynamically optimizing solution when you're only throwing points to the GPU.
  3. The talk on how the designers of Google Earth VR made it a better feeling app had some great insights and design decisions. I'm especially proud of Per Karlsson who had some great ideas in the design of the app. I had the honor of supervising Per during my time at Pixar.
  4. One of my favorite traditions of Siggraph is the Electronic Theater. It's still one of my greatest wishes that I could have been in the crowd back in 1986 when Luxo Jr. was first shown to the world at the Electronic Theater, and a room full of scientists erupted to see their research turned into magic. Most of the projects are really inspiring and remind me why I love making a art with technology. My top 2 favorites were emotionally powerful and ingeniously executed. They were Scrambled by Polder Animation and "Happy Valentine's Day" by Neymarc Visuals.

Saturday, July 22, 2017

The growing skill gap

For those of us lucky to be working on the bleeding edge of tech, we get to work on some of the coolest problems in automation, artificial intelligence, robotics, and virtual and augmented reality. We are driven to innovate not just because they are cool problems but because we believe what we invent makes life better for everyone. Technology like self driving cars, robot driven assembly lines, image classification, speech recognition and translation all have the promise to make our lives better.

The hope of any enriching technology is that it frees our time to pursue a richer life. But if that is our goal, we need to be mindful of the skill gap we are creating between those who are early adopters of new technology and those who adopt later. And this gap could actually be doing more harm than good.

A lot of people contribute to society by taxiing passengers, working on a factory floor, labeling and sorting information, and translating foreign languages - skills that are being made obsolete by those same futuristic technologies we work on. As tech continues to get better we may invent ourselves out of usefulness. Considering how hard it is to learn new skills to catch up on an accelerating wo, technological progress will lead towards a bankruptcy of purpose.

If not addressed, this bankruptcy can have dire consequences on progress. I believe we are already seeing signs of problems with the dramatic shifts in world politics away from science backed thinking and towards populism, nationalism and a fear of reasoning. This is probably why messaging of climate change is still not getting through.

Progress is the right path forward, but is there a way we can also solve the problem of the growing skill gap?

This is where I think virtual reality may be able to help. My favorite feature of VR is how it has created the most intuitive interface for how we interact with computers. Up until now, our interaction has been a difficult abstraction through mouse clicks and keyboard strokes. With VR, I see new users pick it up within seconds. We've all grown up knowing how to move, look around, reach out to touch and grab things; VR just taps into those instincts. We can convey so much complexity with the ease of natural gesturing and soak up so much information through the ease of immersion.

It's my belief that we'll be able to take advantage of this facility by creating educational VR experiences that will make it much easier to learn new, relevant skills. Unfortunately, making virtual content also suffers from a skill gap. Those who know how to create in this new space are mostly game makers who are passionate about making games. Those who are passionate about education technology are still learning the skills to create useful virtual experiences.

That's why it should be an important goal for this new industry to make it easier to be a content developer. Heavily documented and community backed game engines like Unity and the Unreal Engine are a good start but are still designed around making games. We need the next generation of VR developer tools to make it easier for educators to get involved.

The progress of technology is inevitable, but we need to remember that technology should also be used to close the skill gap it creates. And it starts by closing the skill gap with the tools we use to teach.