Rewind to 2017: Apple CEO Tim Cook was free-flowing in his conversations around augmented reality. He said he was excited about AR because “unlike virtual reality which closes the world out, AR allows individuals to be present in the world but hopefully allows an improvement on what’s happening presently”. Speaking to The Independent, he stressed on the practicality of the technology. “Most people don’t want to lock themselves out from the world for a long period of time and today you can’t do that because you get sick from it. With AR you can, not be engrossed in something, but have it be a part of your world, of your conversation. That has resonance.”
The man was onto something. When Apple works on a new technology, it’s not in the habit of leaking documents that become fodder for rumour-mongers on social media. Instead, the company works on a new technology and carefully hints at it over time as and when the technology firms up.
Since that interview, the other big development came in 2020 — LIDAR scanner on the iPad Pro. At first, many people laughed at the idea. Then they backed down from cracking jokes when it was included in the iPhone 12 Pro to improve photography in a big way, especially after twilight. It’s not a new technology per se but absolutely path-breaking when it arrived on the iPhone.
The third big announcement came a few days ago: As part of the promotion campaign for the new season of the Apple TV+ show For All Mankind (the 10-episode second season will debut globally with the first episode on February 19, 2021, followed by one new episode weekly, every Friday), an augmented reality app — For All Mankind: Time Capsule — was launched. Though it’s easy to label it “promo”, there is something deeper involved.
First, and obviously, it shows the importance of AR in the future scheme of things. Second, it offers a new way that television or streaming content can be presented before viewers. Third, it’s an indication of how Apple is tying its hardware and service content to offer a streamlined experience.
Read D-mail messages between Danny Stevens and his friend Kelly Baldwin (characters from the show For All Mankind) that hold secrets of the teens’ changing lives Picture: Apple
A marriage of technology and creativity
In a nutshell, For All Mankind has as its premise that instead of America, Soviet Union managed to place a couple of cosmonauts on the moon before the launch of Apollo 11. In the hands of its creator Ronald D. Moore, the alternate historical story comes across as entertaining as well as a tool for innovations beyond the TV screen. It opens up to augmented reality through your iPhone.
“We’ve been thinking about telling stories in this space for many years and we’re always looking for ways to expand the story beyond the episodes. Once we got to Apple, it was a perfect pairing with AR and you know you are creating a new form of storytelling, so you are looking for ways and kind of defining the language of storytelling in this new medium,” For All Mankind producer Ben McGinnis told us over a video call.
Probably the biggest challenge while developing the app has been transcribing a linear story to fit a virtual experience. “As writers, we are used to linear stories. It was a challenge to figure out how to translate our style of storytelling into this format…. The challenge for us was to figure out what kind of linear narrative one has to go through, at least the first time. You don’t want to get bored the next time. It was constructing a narrative that’s not like a traditional narrative where you’re following a character in a television series from scene to scene. That took a little while to figure out, but I think that we eventually cracked it and Stephanie Shannon, one of our writers on the show, wrote the creative pieces and wrote the letters and the newscasts and all those sorts of things. It was a marriage of technology and creativity... the best of all worlds,” says Ronald D. Moore.
There are a lot of segments in the app to keep you busy. ‘Time Capsule’ opens up the decade-long gap between seasons one and two, to trace the relationship between Danny Stevens and his parents, the astronauts Gordo and Tracy Stevens.
On Apple devices with a LIDAR scanner (iPhone 12 Pro, 12 Pro Max and iPad Pro), the show’s fans will be able to use an old slide projector (a segment in the app) that displays photos projected perfectly on any wall in their space. Picture: The Telegraph
Once you have installed the free iOS app, you will be able to interact with a variety of objects that have a realistic feel to them, such as a mixtape and an Apple II computer. Suddenly, you will find yourself checking ‘D-mail’ and play a text adventure game on the computer. If you have an Apple device with a LIDAR scanner — which means, the iPhone 12 Pro, iPhone 12 Pro Max or iPad Pro — you can use a virtual slide projector to project Danny’s family photos onto your own walls.
When there’s an alternate history storyline to work on, it may appear that the creator had no limitations to work within. Not so. “What we know today as the history of the ’70s, ’80s and ’90s… we always want the audience to recognise the decade that they were in, recognise some basic ideas of what happened in the ’70s, what happened into the 80s, some of the music things would be the same, so we decided that it wasn’t going to just branch out and be a completely unrecognisable history. We had to decide on the limits. So we were still going to have Ronald Reagan as president. Who was his opponent? We just had to build up our own limitations to give us sort of guiderails,” says Ron.
Returning to the usage of the LIDAR scanner — or Light Detection and Ranging — driverless cars have been relying on the laser sensors for years to detect objects and build 3D maps of the surroundings. Apple has put it on iPhones and the iPad so that by combining the depth information from the LIDAR scanner with camera data, motion sensors, and computer vision algorithms, the devices work faster and better at placing AR objects and tracking the location of people.
On the iPhone 12 Pro, low-light focus is on point. Try the Apple Arcade game, Hot Lava, which uses LIDAR to scan the room and the obstacles in it. Quite amazing. The promise AR combined with LIDAR is mindboggling, powering the next chapter in tech, social media and journalism. It has the potential to open up scenarios where you share data involving AR content. And then there are possibilities in the realms of AR headsets as well as automated cars.
Returning to Tim Cook’s 2017 interview, he said: “I regard it as a big idea like the smartphone. The smartphone is for everyone, we don’t have to think the iPhone is about a certain demographic, or country or vertical market: it’s for everyone. I think AR is that big, it’s huge. I get excited because of the things that could be done that could improve a lot of lives. And be entertaining. I view AR like I view the silicon here in my iPhone, it’s not a product per se, it’s a core technology.”
No wonder he made AR a passion project at Apple and it seems that we are not really far away from the time when augmented reality will give iPhone and iPads another reason to standout from competition.
The app For All Mankind: Time Capsule is currently available in the US and will be accessible in more markets soon
