Have you heard about Phantom Intelligence yet?

If you haven’t, I bet you will. Phantom Intelligence is a startup that is looking to revolutionize LiDAR for automotive. I hadn’t heard of them either until QNX and Phantom Intelligence found themselves involved in a university project in 2014. They had some cool technology and are just all-around good guys, so we started to explore how we could work together at CES 2015. One thing led to another and their technology was ultimately featured in both the QNX reference vehicle and the new QNX technology concept car.

I knew little about LiDAR at the beginning of the partnership. But as I started to ramp up my knowledge I learned that LiDAR can provide valuable sensor input into ADAS systems. Problem is, LiDAR solutions are big, expensive, and have not, for the most part, provided the kind of sensitivity and performance that automakers look for.

Phantom Intelligence is looking to change all this with small, cost-effective LiDAR systems that can detect not just metal, but also people (handy if you are crossing the street and left your Tin Man costume at home) and that are impervious to inclement weather. As a frequent pedestrian this is all music to my ears.

I am still in no way qualified to offer an intelligent opinion on the pros and cons of competing LiDAR technology so I’m just going on the positive feedback I heard from customers and other suppliers into the ADAS space at CES. Phantom turned out to be one of the surprise hits this year and they are just getting started. That’s why I think you will be hear more about them soon.


Both QNX vehicles showcased at CES 2015 use a LiDAR system from Phantom Intelligence to detect obstacles on the road ahead.

Driving simulators at CES

CES was just 15 minutes from closing when I managed to slip away from the very busy QNX booth to try out an F1 simulator. Three screens, 6 degrees of freedom, and surround sound came together for the most exciting simulated driving experience I have ever had. I was literally shaking when they dragged me out of the driver’s seat (I didn’t want to stop :-). Mind you, at around $80K for the system, it seems unlikely I will ever own one.

The experience got me thinking about the types of vehicles currently in simulation or in the lab that I fully expect to drive in my lifetime: cars that are virtually impossible to crash, cars that make it painless to travel long distances, and, ultimately, cars that worry about traffic jams so I can read a book.

Re-incarnated: The QNX reference
vehicle.

QNX Software Systems had a very popular simulator of its own at CES this year. You may have seen some details on it already but to recap, it is a new incarnation of our trusty QNX reference vehicle, extended to demonstrate ADAS capabilities. We parked it in front of a 12 foot display and used video footage captured on California’s fabled Highway 1 to provide the closest thing to real-world driving we could create.

The resulting virtual drive showcased the capabilities not only of QNX technology, but of our ecosystem as well. Using the video footage, we provided camera inputs to Itseez’ computer vision algorithms to demonstrate a working example of lane departure warning and traffic sign recognition. By capturing GPS data synchronized with the video footage, and feeding the result through Elektrobit’s Electronic Horizon Solution, we were able to generate curve speed warnings. All this was running on automotive-grade Jacinto 6 silicon from Texas Instruments. LiDAR technology from Phantom Intelligence rounded out the offering by providing collision feedback to the driver.

The lane departure and curve speed warnings in action. Screen-grab from video by Embedded Computing Design.

Meeting the challenge
While at CES, I also had the opportunity to meet with companies that are working to make advanced ADAS systems commercially viable. Phantom Intelligence is one example but I was also introduced to companies that can provide thermal imaging systems and near-infrared cameras at a fraction of what these technologies cost today.

These are all examples of how the industry is rising up to meet the challenge of safer, more autonomous vehicles at a price point that allows for widespread adoption in the foreseeable future. Amazing stuff, really — we are finally entering the era of the Jetsons.

By the way, I can’t remember what booth I was in when I drove the simulator. But I’m willing to bet that the people who experienced the Jeep at CES will remember they were in the QNX booth, seeing technology from QNX and its key partners in this exciting new world.

My next car will know when I'm in a good mood

It will also know when I'm sad, grumpy, or just plain mad. Okay, it won't quite figure it out on its own — I will have to tell it. How will my car get so smart? Because Gracenote has been working to personalize the music experience in a way that dynamically adapts content to the driver’s mood and musical taste.

It's not just about play-listing locally stored content and displaying album art anymore. The folks at Gracenote can now merge multiple occupants’ music collections and find common ground. They can create stations that incorporate cloud-based content from multiple sources. And they can adapt the nature of the content played in the car to the mood of the occupants through their Mood Grid technology.

This not only makes the music experience intensely personal but it does it automagically, keeping the driver’s eyes on the road and hands on the wheel. Here’s a short video from 2014 CES that provides a quick overview — on a system powered by QNX technology, of course :-)

When is a road trip not a road trip?

The über-cool modified Mercedes-Benz CLA45 AMG that QNX unveiled at CES is on its inaugural road trip. Well, sort of. It's actually winging its way to Barcelona for Mobile World Congress 2014.

For those that didn't have a chance to see it at CES, the car incorporates state-of-the-art voice recognition; navigation from Elektrobit, Here, and Kotei; smartphone connectivity based on Miracast and on MirrorLink from RealVNC; advanced multi-media streaming, including iHeartRadio; and a reconfigurable digital instrument cluster, all delivered in a user-centric, multi-modal experience.

For those that have seen it, it is still worthwhile taking the time to check it out because this time around it's powered by Qualcomm’s Snapdragon S602A Automotive Solution. We announced the relationship with Qualcomm at CES and just over a month later we're showcasing the relationship in the Mercedes. Imagine what we'll do by Telematics Detroit.

You can check it out in the Qualcomm booth, Hall 3, Mobile World Congress 2014

Now powered by Qualcomm’s Snapdragon S602A Automotive Solution: the latest QNX technology
concept car, based on a Mercedes CLA45 AMG.

So many cores — what to do with them all?

Multi-core processors are clearly becoming the mainstream for automotive infotainment systems. TI’s OMAP processors and their automotive derivatives use dual A15 cores, Freescale's i.MX 6 boasts up to four A9 cores, and other companies also have multi-core architectures in production or on near-term roadmaps. Quad-core A15 processors are just around the corner. As a percentage of overall die area, the CPU core is relatively small, so SoC producers can lay down multiple cores with little impact on cost. GPUs, on the other hand, represent a large percentage of the die area and, as such, are typically instantiated only once per SoC.

Realistically, infotainment systems should no longer be CPU bound. In fact, when looking at leading-edge SoCs available today, many are asking what to do with all that extra CPU just sitting there, waiting to do something. At first blush, the more obvious areas to merge are infotainment and ADAS, or infotainment and digital instrument clusters. This is, at the highest level, pretty much a no-brainer so long as the safety requirements mandated for clusters and ADAS can be achieved.

Thing is, automotive safety standards like ISO 26262 require system-level certifications. As such, the entire system needs to be certified. Already a daunting task for a standalone ADAS system or digital instrument cluster, the problem can become unmanageable when you include the full infotainment stack.

Think about your car. Your cluster does a handful of operations whereas your infotainment system runs a full navigation system, voice recognition, multimedia, device connectivity, and, in the connected case, cloud services. People don't get frustrated trying to figure out how your cluster works (I hope), and they don't give up trying to figure out how fast the car is moving. The same cannot be said for many infotainment systems shipping today. Ask your mother to pair her cell phone to her car. I dare you! The complexity involved in attempting to certify a system that combines infotainment and cluster functions is orders of magnitude higher than certifying a cluster alone.

All is not lost, however. Virtualization offers an elegant way to isolate multiple systems running on a single multi-core SoC. By using virtualization you could seek certification on the cluster without burdening yourself with the infotainment problem and collapse two formerly discrete systems onto one SoC. You would save money and probably get a promotion to boot. Just one thing: there is still only one GPU on the die and both the infotainment system and the cluster rely heavily on that single GPU.

Enter Red Bend Software, a long-time QNX CAR Platform partner for FOTA. They have taken the challenge of virtualizing the GPU head-on and successfully demonstrated the QNX CAR Platform and a Crank Software-based digital instrument cluster running on dual displays driven by a single OMAP 5 at Telematics Munich. I saw it and was impressed with how snappy performance was on the infotainment side and how smooth the needles were (60+ fps) on the cluster.


Using virtualization to drive dual displays from a single TI OMAP 5 processor.

According to Red Bend, they have designed a GPU-sharing architecture that enables multiple guest operating systems to access hardware accelerators, including the GPU, providing isolation between the operating systems while having a minimal impact on overall performance (which probably isn't a huge deal considering how many CPU cores are going to be shipping on a single SoC in the near term). It sounds impressive, but seeing is believing.

Red Bend will next show this demo in the TI Suite at CES (N115 in the North Hall). If system consolidation is something that keeps you up at night, you should really stop by to see what they have done.

Distracted driving — the stats are alarming

I was driving to work the other day when I heard something on the radio that almost made me drop my smartphone. The Ontario Provincial Police (OPP) announced that, for the first time, deaths attributable to driver distraction outnumber those caused by impaired driving. So far this year, on roads patrolled by the OPP, distraction has led to 47 deaths, while impaired driving has led to 32.

This stat drives home the need for dramatically better head-unit integration of services that drivers would otherwise use their phones to access. This isn't anything new to QNX. We've been working with our partners to provide all the necessary elements to enable this integration through technologies such as HTML5, Qt, iPod out, MirrorLink, and Bluetooth. All these technologies can help create systems that minimize driver distraction but they represent only part of the solution. Pushing buttons on your head unit, combined with smart HMI design, does help, but it's not a panacea.

To truly help drivers keep their eyes on the road we have to minimize the time they spend looking at the infotainment display. Multi-modal HMIs built from the ground up with the assumption that high-quality speech recognition and text-to-speech are available will drastically change the way drivers interact with their infotainment systems. For instance, such HMIs could read your texts and emails aloud to you; they could even let you dictate responses at the appropriate time. But really, the possibilities are endless. And on the topic of talking to your car, we're constantly working with our partners to enrich the speech capabilities of the QNX CAR Platform. But more on that in an upcoming post.

By the way, I wasn't really using my smartphone while I was driving. That's illegal here. Not to mention incredibly dumb.

Seminar: managing the growing amount of software in cars

It’s no secret that the amount of software in automobiles is growing rapidly — as is the challenge of maintaining it reliably and efficiently. At QNX Software Systems we focus on areas like infotainment, telematics, clusters, and ADAS, but our long-term FOTA partner, Red Bend Software, takes a more holistic view, working with companies like Vector Informatik to extend FOTA all the way down to ECUs.

To help automakers and tier one suppliers manage their software deployments more efficiently, Red Bend is hosting a seminar Friday September 27 at the Westin Southfield Detroit. Speakers will include representatives from Strategy Analytics, Texas Instruments, and Vector, not to mention our own Andy Gryc. You can register on the Red Bend website.

Finally, I can throw away my 8 tracks

Okay, maybe I'm not old enough to have owned 8-track tapes. But I do remember that my uncle had an 8-track player in the dash of his station wagon when I was a kid, and I am old enough to have owned a car with a cassette player.

Music has been fundamental to the driving experience for about as long as cars have been on the road. Terrestrial radio dominated forever, supplemented by tape and then CD. XM radio came along in 2001 and connecting your iPod started to show up in the late 2000s. That's 5 formats since the Model T was introduced in 1908 (okay, so it didn't  have a radio) and 3 formats in the first 90 years.

Now, with connected cars becoming a reality, the rate of change is shifting into overdrive. Want Pandora – check. Want to listen to the top alternative radio station in Dublin while driving in California – check. Want to keep listening to your Songza programming as you move from the house to the car – check.

Today's announcement from QNX and 7Digital adds a whole new dimension. Having 7Digital in the car will unify the music ownership experience across the big three: car, pocket, and home. Want to listen to your own music programming in the car – check. Want to buy a song you just heard on that Dublin radio station – check.

Read the press release for details. And when you're done, check out the 7digital blog.

Pandora interview: Using HTML5 to deliver content to the car

At CES this year, our own Andy Gryc had a chance to sit down with Tom Conrad, CTO at Pandora, a long-time QNX CAR platform partner. Pandora is already in 85 vehicle models today and continues to grow their footprint, not only in automotive but in consumer as well.

Take a couple minutes to hear Tom's perspective on standardizing on HTML5 across markets and to get a glimpse of the future of Internet radio in automotive. And make sure you watch the whole thing — there's some fun outtakes at the end.

Recall? What recall?

Red Bend demonstrates firmware-over-the-air (FOTA) updates of QNX CAR 2 application platform at Telematics Munich

I think anyone with a passing knowledge of software development in automotive would agree that the infotainment systems currently under development are light years ahead of the systems that shipped only 5 years ago. The blurring of the automotive and the consumer experience is accelerating at an amazing pace. And the processing power being specified for next-gen infotainment aligns with what is expected in advanced smart phones.

It's no surprise, then, that the size of the code base and the complexity of the underlying software is growing at a similar pace. This complexity creates a maintenance challenge. On your phone, upgrades are pushed out regularly in a way that you barely notice: you get a notification of an update, push a couple buttons, and presto, you are up to date. In automotive, if we stick to the traditional methodology, this same type of upgrade would require a recall. You'd have to take your car to the dealership and they would reflash whatever needs to be updated. Expensive for the auto manufacturer and a big pain for the consumer.

Thankfully, people are thinking about this. Companies like Red Bend Software have cut their teeth in the mobile space, specializing in firmware-over-the-air updates, or FOTA for short. They can generate something called a delta file, which effectively encapsulates the difference (or delta) between what is currently on the end device and the new software build. In some cases, the file can be up to 50 times smaller than the new build. They also have the ability to track current load status of all the devices deployed.

So what does that get you? Using FOTA, OEMs will be able to minimize the network bandwidth required for upgrades and to manage the update process remotely, moving us all towards that Zen state of automagic. I don't know about you, but anything that saves me a trip to the dealer is a good thing.

Red Bend will demonstrate this capability by updating versions of the QNX CAR 2 application platform this week at Telematics Munich. So if you happen to be there, do stop to check it out.

QNX, NVIDIA team up to deliver infotainment solutions

Today, at SAE Convergence, QNX announced that it is working with graphics leader NVIDIA to bring infotainment solutions to the automotive market. As part of this initiative, the companies will integrate support for the NVIDIA Tegra processor into the QNX CAR 2 application platform.

The Tegra system-on-chip is the size of thumbnail, yet it incorporates a quad-core ARM CPU and a GeForce GPU, as well as dedicated audio, video, and image processors.

The NVIDIA Tegra visual
computing module

“QNX Software Systems and NVIDIA have a proven track record of delivering on production programs for Audi... and we’re excited to add support for Tegra to the latest generation of our automotive platform,” said Linda Campbell, QNX director of strategic alliances.

Speaking of Audi, NVIDIA is bringing an Audi A6 to SAE Convergence, equipped with an infotainment system powered by technology from QNX and NVIDIA. The system bristles with high-end features, including 3D navigation with Google Maps and Google Earth, as well as natural voice recognition.

For more information on this announcement, read the press release, and for more information on QNX activities at SAE Convergence, visit our Convergence overview page.

QNX CAR goes to school

A while back, my good friends at Freescale approached me about sponsoring a program called EcoCAR 2. Established by the U.S. Department of Energy (DOE) and General Motors, EcoCAR 2 challenges 15 North American universities to create a greener vehicle without compromising performance, safety, and consumer acceptability.

This initiative isn't really new as it builds on a 23-year history of DOE advanced vehicle technology competitions. What is new is that, for the first time, students are being asked to include an infotainment system in the vehicle.

This is an exciting opportunity for QNX. Being able to work with young minds and supporting university research has always been a priority, but this is the first time we've been able to engage so directly with the next generation of automotive engineers. We are enabling them in much the same way that we enable our customers, with full access to the QNX CAR application platform as the baseline for their systems.

This is year one of a three-year program. Given the innovation that has gone into the QNX CAR platform by QNX and its ecosystem in the last year alone, I cannot imagine what the bright young minds will come up with for 2014. Looking forward to it...

---

Resources

• Learn more about EcoCAR 2
  
• Follow EcoCAR 2 on Facebook
  
• Check out the EcoCAR 2 blog
  
• Tune in to the EcoCAR 2 YouTube channel

 

Cold beer (and a chance encounter) at Telematics Detroit

Last year I was part of QNX’s advance team for Telematics Detroit 2011, which had me in Detroit well before the show. I was just heading out to dinner when I saw a small bar set up on a terrace outside my hotel. I wandered by for a closer look, thinking I might stop by later to enjoy a cool drink in the warm evening air.

A few guys were sitting drinking beer and they invited me to join them. Ever social, I agreed. It turned out that the person who extended the invitation was from Cybercom — serendipitous, as Cybercom is a Bluetooth connectivity provider that we were considering for our next-gen QNX CAR application platform.

Fast forward one year exactly. We are getting ready to showcase that next-gen platform, QNX CAR 2, at Telematics Detroit 2012 — in a new reference vehicle, no less. Meanwhile, QNX and Cybercom are now engaged with a number of leading tier one suppliers. In fact, Cybercom has issued a press release about the partnership; read it here.

To Kristian at Cybercom: thanks for showing kindness to a guy on his lonesome, and for being a key contributor and partner in our mutual automotive initiatives.
 

A cool surprise at the Elektrobit auto summit

Recently, our good friends at Elektrobit invited the QNX team to participate in their German Executive Automotive Summit. It was an outstanding event with all of the leading OEMs and tier ones represented. The day was filled with engaging speakers and plenty of opportunities to network.

Elektrobit held the event in a small castle near Erlangen. In the courtyard, several cars featuring Elektrobit technology (and, in almost all cases, QNX technology) were on display. The car from Delphi was especially interesting. It's a full-blown race car, complete with everything you'd expect in a track vehicle — but it also has two rear seats. These seats allow mere mortals like you and I to vicariously share the racing experience with a professional driver at the wheel.



As I stood next to it, drooling, I noticed that it was equipped with an infotainment system, mounted on the back of the driver's seat. I leaned in to have a closer look and, to my delight, saw that it was running the QNX OS. Who knew?

TCS showcases hybrid navigation system in VW Polo

The Porsche 911 concept car that won the 2012 Best of CES award wasn't the only QNX-outfitted vehicle at Mobile World Congress this year. Our friends at TeleCommunication Systems (TCS), who provide hybrid navigation software for the QNX CAR application platform, showcased a VW Polo running the same QNX-powered navigation, maps, and local search capabilities as the Porsche.

Like the Porsche, the Polo featured a retrofitted infotainment system in place of the factory radio — and TCS did a very nice job integrating it into the car.

The Polo was a hit, driving discussions with mobile operators, handset manufacturers, automakers, and automotive tier ones. Seems the connected car is a hot topic just about everywhere you go these days.

Here's the Polo in the TCS booth:



And here's a close-up of the retrofitted infotainment system:


 

Listen to the music

The audio CD is on its last track... Internet radio, anyone?

I don’t think anyone will disagree with me when I say that music still represents the most important element in an infotainment system. Just look at the sound system capabilities in new cars. Base systems today have at least 6 speakers, while systems from luxury brands like Audi and BMW boast up to 16 speakers and almost 1000 watts of amplification.

For nearly as long as I can remember, cars have come with CD players. For many years they’ve provided a simple way to take your music on the road. But nothing lasts forever.

Earlier this year, Ford announced it would discontinue CD players in many of its vehicle models. Some industry pundits have predicted that CD players will have no place in cars in model year 2015 and beyond. A few weeks ago, Side-Line Music Magazine reported that major labels plan to abandon the CD format as early as 2012. This revelation has created a flurry of activity on the Net, but the labels have yet to confirm it’s true.

Steady decline
The fact is, CD sales have declined steadily for the last several years, down 16 percent in 2010 alone. Digital downloads (the legal kind), on the other hand, have been growing quickly and are expected to exceed CD sales for the first time in 2012.

Where does that leave us in the car? Obviously, media device integration will be key in the coming years. QNX Software Systems has long supported Apple iPod integration and supported Microsoft’s ill-fated Zune for a while. USB connectivity is a given, and soon you’ll be able to stream music from your phone.

Radio redefined

The QNX-powered Toyota Entune system 
supports both Pandora and iHeartRadio.

But what’s more exciting is how radio is evolving in the vehicle. Along with the steering wheel, radio has been a staple of car pretty much since day one. As the connected vehicle moves to the mainstream, internet radio will become a huge part of the automotive experience.

Companies like Slacker extend the concept of radio beyond audio to include artist bios, album art, photos, reviews, and more. Pandora, through its work with the Music Genome Project, expands the musical experience by playing songs it predicts you will appreciate. iHeartRadio aggregates American radio stations for replay throughout the US. TuneIn takes it one further with a global view. Driving down highway 101 in California, you’ll be able to tune in all your favorites from around the world.

Beyond entertainment
These services are changing the way people consume music. Today, I rely on my car radio not only to entertain but also to educate by constantly exposing me to new artists and content. Internet radio in the car will expand my horizons even further. And as online music stores like 7Digital integrate their service with the internet radio stations, I’ll be able to download the song I just heard at the push of a button. Not good for CD sales, but it seems that’s the way of the future anyway.

We are, of course, working with the leaders in internet radio and online music services to bring them to a car near you.
 

Building a hands-free future

The end of my street is governed by a three-way stop. The other morning I was backing out of my driveway when someone rolled past the stop sign and came within inches of hitting me. I stopped, glared at him, and resumed driving. Two stop signs later, the same guy squeezed past my car (in the same lane), completely oblivious to what he was doing.

Why was he driving like this? Probably because he was deeply engrossed in a conversation on his cell phone.

Where I live, using a handset while driving has been illegal for over a year. You cannot talk, you cannot text, you cannot “Facebook”, you cannot Tweet — even if you're stopped at a red light. This makes perfect sense to me. As a driver, your primary responsibility is to control the vehicle. And yet I see people texting on the freeway, talking on their cell phones, and doing who knows what else on an alarmingly regular basis.

The QNX-powered BMW
ConnectedDrive system

Society has become obsessed with mobile devices, and it will take more than legislation to change its behavior. The answer, I think, is to embrace the behavior in a way that makes it possible to interact socially while maintaining control of the car. We’ve seen great progress in hands-free/phone integration, and BMW ConnectedDrive offers an example of how drivers can access email and other smartphone services more safely.

This is the tip of the iceberg. Integrating the handset with the infotainment unit is going to change the way you interact with your car. Intelligently designed apps, combined with multi-modal human machine interfaces, will let you Tweet or update Facebook using speech recognition, keeping your eyes on the road.

Without taking your hands of the wheel, you’ll be able to call a friend and decide that you want to go to dinner, do a local search to find out what’s available, check a restaurant review on Yelp, make a reservation, text your friend back with the time and place, and aim your navigation system at the restaurant. And you’ll be able to do it using natural language. None of this “please say a name” stuff.

Seems futuristic? It’s not. People are working on it today. In fact, QNX-based systems, such as Toyota Entune, already offer a taste of this hands-free and highly personalized future.