There has been considerable discussion lately about the impact that 360° video will have on the world of business. When we say business, we think of the brands using this immersive storytelling medium for promotion of their products and services.

But this is actually the least exciting thing about 360° video -- the possibilities in the filmmaking arena or immersive journalism, for example, are far more creative and promising.

This post reflects on our experience in developing technology to empower immersive 360 experiences on different devices and form factors. We will cover technical challenges that we experienced during the development of a 360 interactive player that is bundled as SDK for 3rd parties to use.

So what is 360° video, anyway?

The definition of 360° video is tied to the content itself - a video that offers a 360 degree view around the viewer.

What is interesting to us as a digital product company is the technology that enables the viewing of 360° content. There are many ways to watch this type of content e.g. desktop browsers, mobile browsers, native apps, and something that is neglected at the moment - tv.

On mobile, the viewer can explore content by swiping the image around or by moving the device (smartphone, tablet). A 360° video can also be viewed with a VR headset, like Google cardboard, which makes the experience even more captivating.

The fact that online giants such as YouTube, Vimeo, and Facebook actively support 360° video is a good indicator of how popular this format has become.

6DOF, 3DOF?

DOF stands for degrees of freedom and relates to the directions you can move in the virtual space. Mobile phones are capable of tracking in 6DOF while not all headsets are 6DOF. And yes, many people will say that most of the VR experiences are sedentary and that 3DOF is enough.

However, while sitting, you move your head up and down, left and right. While doing so you are moving the central viewpoint but in the 3DOF experience the viewpoint center is not changed. This may be subtle but it is the cause of motion sickness for most of us.

6DOF can be achieved with the rendered scenes, but with 360 video you are limited to 3DOF which is ok as the sequences that are 360 are still short enough that you can fully immerse yourself into it without getting dizzy.

As the technology moves forward the volumetric video will change this, allowing us to explore the video scenes in full 3D while moving around the scene and seeing it from different angles. This is still an expensive process but there are some solutions out there trying to get it to the masses.

While volumetric video capture is still years ahead commercially, there is already enough 360 content available that can be used to provide additional experience to the viewers. Being able to interact while in 360 is something that can help them fully immerse into the virtual experience.

This is especially interesting for the advertising industry. Imagine making a scene where the car moves around and the viewer can activate/click on the areas of the car which he would like to explore more. Then it can render new overlaying video on top of that area, or expand into more info, or get him to another 360 video scene. Wow.. right?

How to create interactivity for 360 degree video content

Designing interaction for this new environment isn’t easy since you can’t just replicate the UI patterns that worked in a two-dimensional space. It’s a fairly new world after all.

Here are some of the things we learned:

  • Content is key - 360 video should tell the story to enable users to easily connect their actions with a storyline
  • Interaction components are difficult to place - Users need to stay oriented to follow the storyline, so UI elements need to be clear but also get out of the user’s way
  • It is hard to create good UX for navigating.

Defining interactive parts is challenging but the effects can be fully immersive and allow content creators to build scenery and interactions that are like point and click adventures.

Technical challenges

Though 360 experience is available on the mobile phones our goal was to provide it through different form factors and platforms: mobile, web, and tv, each with their own challenges. And even though there are libraries available to help you out, not all of them solve the challenges fully.

Here are the key areas you need to be aware of before stepping into developing 360 experiences:

  • Euler angles and Gimbal lock - handling the issue when two of the three gimbals are driven into a parallel configuration. This is intuitive but extremely complex.
  • Quaternions - an alternative approach to Euler angles that enables you to calculate orientation in 3D space.
  • Stereoscopy - used for the VR mode.
  • OpenGL - for some things you just need low level graphic interface.

These are the fundamentals needed for all platforms.

There are some commercial libraries available but they are not perfect (e.g. they avoided gimbal lock handling by not allowing simultaneous finger drag and giro movement), and there is currently only one free library that offers cross platform development - Google VR Kit (android and iOS).

In the web world there is A-Frame.

Sounds good, right? Like these guys solved the issues and there is no need to dive into the complex math. Unfortunately, this is not the case. Let’s go over this platform by platform.

Mobile

GVR looks very promising when you read through the website. But the reality is that it is poorly documented, core of the system is not open sourced, and some of the examples are stale and use their previous library version and can mislead you in how you should develop your solution.

When you start working with iOS version of the GVR it looks good. There are some really nice high level abstractions. And then when you switch to Android, you realize that Android version is not the same and you need to spend more time on the low level stuff. The only real help you get is a stereoscopic view for the scene that you need to construct.

We still needed OpenGL to write custom shaders to render semi-transparent layers. We also needed to develop UI orientation inertia ourselves in order to ensure smooth UX.

On iOS, GVR comes as a static library so there is no way of creating a dynamic framework based on GVR, but one can live with this limit. The dealbreaker is that GVR does not offer tvOS library which means there is no solution to use on tvOS.

It would be ideal to have the same code base for both iOS and tvOS but this is not possible with GVR.

We went on to create everything from the ground up on iOS and tvOS. SceneKit was a great alternative because it allowed us to abstract away 3D models and put the effort into improving user experience through orientation. Stereoscopic view has been implemented using Metal for compensating lens distortion and SceneKit for video content rendering.

The outcome:

  • Unified code base on iOS and tvOS
  • GVR on Android
  • Intuitive connection between Giro and touch control panning

The last one is important to point out as it was a real beast for our engineering team to get it right and one we are most proud of. Other solutions are using different approach and sacrificing some features which creates bad experience for the user.

For example, its not possible for the user to move the scene with their finger to a certain position and then move the phone around to see more details. With our implementation this is perfectly possible.

TV

As mentioned above, tvOS has been supported with the same codebase as iOS written from scratch. Of course the navigation is different and supporting Siri remote was one of the challenges here - adopting to Apple tv experience and handling different events and gestures coming from the remote instead of the touch screen.

Apple TV is just one drop in the sea of TV experiences. Supporting popular players is a must for wide adoption - FireTV, Android TV, Roku, etc. - but the market share is still small enough that it can be ignored for some time.

FireTV and Android TV are both based on Android, and GVR does the job. For the Apple TV the code needs to be adjusted to process inputs from the d-pad remotes.

The remaining one is Roku which is still in progress and requires writing everything from the ground up with Roku NDK which is available to selected partners only.

WEB

A-Frame is good library and helped us kickstart the project by eliminating low level components, e.g. placing the frames on a sphere and VR mode so we could focus on creating good user experiences by providing efficient controls in the player.

The downside is that WebGL needs to be enabled in the browser for the player to work. Plus, not all browsers process all the media formats or cannot use HLS stream as a material in WebGL. All of this need to be handled gracefully for the user.

Other important details

We have spent many hours working to provide the best media player capabilities and the best user experience, and the details outlined above are just one tiny piece of the puzzle.

Here are some other important details you need to be aware of before diving into this:

  • Different streaming formats available - HLS, MPEG-Dash, MSS, HDS
  • Devices have different hardware capabilities

Not all streaming formats can run on all platforms. Fortunately there are some libraries that enable this and we provide support for both HLS and MPEG-Dash on most platforms.

Rendering 360 video is an intensive operation and not all devices have enough power to run it or can render smaller frame sizes. For 360 video to look ok, even on the smaller screens, you need a 4k resolution of the equirectangular projection, but some devices cannot render frames this big.

This is true even for the latest generation devices, especially for TVs and media controllers. Fortunately, with the streaming formats content creators can specify different sizes so that the player can detect device capabilities and stream the best available size.

Conclusion

If you want to break into the 360 video, either as a content or technology maker, now would be the perfect time. You may not have all the resources available at your fingertips, but you’ll enjoy being part of this relatively new, unexplored frontier with many challenges to solve.

In this article we have presented some of the details of our journey of creating a 360 video player that works across as many devices as possible to bring the immersive experience to viewers everywhere.

We believe that technology should enable creators to create new worlds and this is why we got into this project.

Interested to learn more about designing for the 360 experience? Are you exploring this area and looking for a technology partner to help you shape a product? Feel free to Reach out _at any time! _