GUEST BLOGGER:
Eric de Grasse
Chief Technology Officer
Project Counsel Media
( Eric is my CTO and business partner and heads our team at CES 2020 )
10 January 2019 (Las Vegas, NV) – What was Neon? It was shrouded in mystery leading up to CES 2020. All we knew from the pre-event media distributions was that Neon had something to do with artificial intelligence. Was it a Google Assistant competitor? A robot? Something more? Well, said Neon’s CEO Pranav Mistry at the start of his keynote presentation:
“It’s a preview of a wonderful technology we have, and a wonderful future we can create together.”
So what is it? It’s not hyperbole, for a start. Neon is a step closer to living with a digital creation that not only understands and emotes with us in a meaningful and relatable way, but is also able to create valuable memories with us and truly share our lives. Explaining exactly what Neon is, how it works, and the incredible depth of technology underlying it is a considerable challenge and one that Neon itself isn’t quite sure how to tackle. To help introduce Neon, Mistry started out by saying he wants to change the way we interact with machines, and no longer say just, “Stop,” “Next song,” or even, “Hey Google, Bixby, or Siri,” because it’s not how we talk to humans.
Hat tips to Andy Boxall of Business Insider and Jag Kumar of Samsung for the patience to walk me through the basics of the technology.
Mistry said he wants to “push the boundaries so machines understand more about us. Whether we are tired or happy, our expressions, and our emotions.” In turn, the more machines understand us, the more we will be able to connect with them on a deeper, human level. He believes the path to this means machines need to look and act more like us, and this is where Neon’s journey really began.
The CES demonstration came just four months after the project started. Mistry and the team began by creating a digital version of a friend, which closely emulated his facial movements during conversation. This evolved into larger, grander tests until eventually, the digital version began to do things on its own. It would make expressions the real person had not. It had “learned,” and become something individual.
The Neon booth in Central Hall at CES is covered in large screens showing people on them, all moving, smiling, laughing, or silently mouthing words to the audience. Except these aren’t videos. These are Neons. They are digital creations born from real people, and although they visually represent the model on which they’re based, the movements, expressions, and “emotions” are entirely automatically generated.
A Neon yoga teacher
Once you understood this, it was surreal walking around the booth, looking at the Neons who in turn are looking at you, and now understanding the movements they made were of their own doing, not a repeating video or animation. What was powering the Neon, and what did Mistry have in mind for their future?
The Neons are generated by the company’s own reality engine called Core R3. The R3 name refers to the principals on which the system is based — reality, real time, and responsiveness, and it’s the combination of all these that bring the Neon to life. It’s not an intelligent system, says Mistry, because it does not have the ability to learn or remember. Instead, it’s equal parts behavioral neural network and computational reality that independently generates the Neon’s “personality” by training it to emulate human behavior on a visual level — how your head moves when you’re happy, what your mouth does when you’re surprised, for example. I need to work through the tech sheets when time permits to give you a better view of this technology in a later post. It is mind-boggling. Said a Samsung rep:
Once it has been created, Core R3 does not then continually run a Neon. It generates it initially, then the Neon continually relies on its own information to react based on its interactions with the real world. However, it doesn’t know you or remember you. It uses a combination of the Core R3-generated Neon, cameras, and other sensors to interact with us in the moment — but once that moment is over, everything is forgotten. In the near future, the company has big plans to change that.
Despite only being worked on for four months, there was a live demonstration of what a Neon can do now. There are two “states” for Neons currently, an auto mode where it does what it wants, whether it is thinking, responding, idling, or greeting you, plus a “live” mode where the Neon can be controlled remotely.
The Neon has multiple ways to respond and can choose how to do so, even when instructed to do a particular action. Tell it to smile and be happy, and it does so, but it chooses the way it will look when it does. The level of granular control is impressive, right down to eyebrow movement and the closing of eyes, along with head movements and both visual and verbal responses. This all happens with a response time of 20 milliseconds (the real-time aspect of R3), which removes the barrier between human and machine even further during any interaction. Speech is not produced by Neon at the moment, and in the demo, voice was pulled from third-party APIs, giving life to artificially intelligent voice assistants and chatbots everywhere.
Yes, evil raises its head again. As Greg notes in an upcoming essay: “People worry about deep fakes. Ha. Chatbots have been skewing social-media discussions for years. About a fifth of all tweets about the 2016 U.S. presidential election were published by bots, according to one estimate, as were about a third of all tweets about that year’s Brexit vote”.
These are Neon prototypes, not yet ready for public consumption. There is still a lot of work the developers need to do. But for some “live” looks here are two videos :
The Neon is “domain independent.” A Neon could teach you yoga or it could help bridge language gaps around the world, for example. Potential uses for a Neon in business are obvious, such as in hotels, at the airport, or in public spaces. The Neon is an evolution of the clunky robots or lifeless video screens seen in these places around the world at the moment. But that’s not really very exciting, and certainly not the part of the Neon that’s truly groundbreaking.
Right now, a Neon cannot know who you are or remember you. Once your interaction is over, your relationship with it is lost to the digital ether. However, over the next year, the Neon team will work on the next version of Core R3, along with a project called Spectra that will add these important traits to Neon, and arguably bring it to life. “Spectra will provide memory and learning,” Mistry said, revealing the true direction of Neon.
By adding memory and the ability to learn, along with the advanced human-like visuals, a Neon has the potential to become a true digital companion. Speaking to Mistry after the presentation, his eyes lit up as he talked about the characters he loved as a child, and that the connection he had with them was not affected by the fact they were not “real.” A fully fledged Neon could bring similar joy to people, in a stronger and even more personal way.
What Neon showed at CES 2020 is very much the beginning, but there’s clearly a massive amount of investment, belief, and talent involved. Not many companies would have the guts to come to Las Vegas and show off a four-month-old demo after a few weeks of hyping it up. Mistry has worked with Microsoft on the Xbox, and with Samsung on the Gear VR in the past. He’s soft-spoken and charismatic, and everyone we spoke to at Neon had a similarly strong belief in what the company is doing.
It was contagious … and scary … especially if you’ve had sci-fi dreams or nightmares about artificial humans and digital companions all your life.