Application of Edge AI in Smart Home Human Machine Interface (HMI)

Consumers have an insatiable appetite for improving their convenience, safety and user experience. We see that the human-machine interface (HMI) has developed in an obvious way, from pure tactile development to over the years, it has covered various input methods from voice to gestures to video and various computer vision functions. , From the point of sale to the smart home. The next step will be a device that not only understands direct commands but can infer intent.

At the same time, concerns about the security and latency of traditional cloud-based connected devices are increasing, paving the way for more edge-based processing. This is especially true in the human machine interface (HMI). However, local processing brings new challenges to technology developers. They must consider specific use case requirements, development options, and the cost of intelligent (machine learning-trained) equipment that needs to introduce new levels of automation to enhance perceptual intelligence and Environmental computing.

Edge AI is the foundation

The so-called Edge AI (Edge AI) is the basis for achieving a more complex, user-friendly and safer IoT experience. By definition, edge AI means that AI processing is performed inside the final product itself (such as a set-top box or smart display) rather than in the cloud. The reason for this is well-known, namely better privacy, less bandwidth and faster response time, and even eco-friendliness, because edge processing reduces the need for energy, water and other resources to run large data centers.

Edge AI has been adopted in many applications that we touch every day, but the initial use is largely limited to expensive products, such as smartphones and cars. Therefore, the implementation of edge AI for these products is also expensive and out of reach for consumer retail devices in smart homes. To a large extent, existing edge AI applications are one-dimensional in terms of the user experience they provide. For example, AI-enabled vision in ADAS (Advanced Driving Assistance System) applications or image quality enhancement in mobile phones.

What are the compelling reasons for creating and adopting edge AI solutions for smart homes?

HMI promotes edge AI in the home

We see that in the ubiquitous consumer IoT market segment, people’s interest is particularly strong and there are more and more application opportunities, such as various entertainment, communication, home automation, security and other various devices. Especially in the current era, consumers want an interconnected experience without the cost, privacy and performance issues of traditional interconnection. The demand for more immersive and perceptual human-computer interaction is a key factor driving the demand for edge AI in smart homes.

With the help of AI-based edge computing solutions for smart homes on the market, the performance required to create a more humane experience will be used in a wider range of products.

There are many practical application cases that benefit from edge AI in smart homes. Some have obvious practical benefits. For example, a home doorbell camera can tell the difference between a dropped package and a stolen package. Entertainment devices can automatically detect low-resolution video streams and upgrade them to higher resolutions, and have excellent perceptual quality, thereby making better use of high-resolution TV displays. Even the familiar, now almost ubiquitous video conferencing applications can be enhanced with higher-quality video and audio, and can be used on cost-effective devices.

Other examples seem more futuristic. A refrigerator that can provide dinner (menu) suggestions based on the contents of the ingredients in the refrigerator. The oven can tell you when the meal is cooked to perfection. A virtual private home yoga instructor can remind you to straighten your arms while posing. Home automation devices can work together to predict the needs of homeowners, from heating and preparing food to choosing what to watch on TV.

Such a solution can combine video, vision, and voice sensors with AI processing functions to bring enhanced functions to a new generation of familiar devices, such as smart displays and sound bars, set-top boxes, and security cameras.

What these applications have in common is the need for an edge-based, AI-based solution that is tailored specifically for smart homes rather than smart phones or car applications. To further democratize edge AI, the solution must be:

The ability to combine voice, video, and vision in an effective system to support multi-modal AI enhanced user experience; through standard tools, a wider range of AI developers and innovators can access; to ensure that security and privacy measures meet consumer expectations.

Edge AI advantages in smart homes

Smart home human-computer interaction requires a multi-modal approach

As we discussed before, edge AI-based solutions for smartphones and automotive applications mainly focus on camera vision applications. However, in smart homes, multi-mode human-computer interaction interfaces are a key element to enhance user experience in the new era of connected devices. Taking the set-top box as an example, this application will require video AI, perhaps in the form of video enhancements as described earlier. It also requires voice AI to be able to recognize the person watching TV through voice commands and configure the experience accordingly. For example, making it easier to choose favorite shows. It may even require visual AI and a built-in camera to provide an enhanced and intuitive video conferencing experience when chatting with family members remotely.

The ideal solution is a smart home-centric SoC (System-on-a-Chip), which can support high-performance video, voice and vision processing and integrated AI accelerators. Synaptics VS600 SoC series is an example of such a solution. This method is not only optimized to meet the requirements of smart home applications for multi-mode AI performance, but also can integrate all these functions into a single chip, so that ordinary household products sold at consumer market prices can be used.

The required solution starts from a SOC platform that integrates multiple types of processor engines: CPU, NPU, GPU, and ISP, as well as links to high-performance cameras and displays. This architecture achieves an ideal combination of high security, low-cost reasoning, and real-time multi-mode performance. Synaptics Edge AI series is a series of SoCs, each SoC is highly targeted at its given consumer application. Each SoC in the series integrates the required processing cores and the appropriate level of integrated AI performance for the application.

A complete stack tool approach simplifies AI development

As we have seen, the cost/performance trade-off is critical to successfully extending edge AI to more applications. In the highly competitive consumer electronics field, time to market and differentiation are also crucial. In order to meet the challenge of the widespread dissemination of edge AI, a full-stack approach is needed, which includes the necessary development tools to introduce AI innovation into edge AI SoCs.

Most importantly, the required toolset should be compatible with the large and growing user community of AI developers. For example, the toolkit will enable developers to import models created using industry standard frameworks such as TensorFlow, TensorFlow Lite, Caffe, and ONNX. This enables developers to take advantage of existing AI innovations and enables them to quickly and easily work on the target SoC.

Let’s use the personal home yoga coach app we discussed earlier. The AI ​​model on which the application is based will be a human pose estimation model, which is an industry standard concept used to detect the user’s relative bone position in the camera’s line of sight. If AI developers themselves use the implementation of the human pose estimation model created by industry standard tools (such as TensorFlow lite), they will use the toolkit to import it for use on the required SoC.

When developers are ready, the tool should enable them to optimize the performance of their AI model for the selected processor that will run on it. Developers can choose to use open frameworks, such as TensorFlow or TensorFlow Lite, but keep in mind the capabilities of the target processor when using them. Or they can use SoC-specific tools again, such as Synaptics’ SyNAP tool, which supports optimizations specifically for the processors in the VS600 SoC. In our example, developers can use the SyNAP optimization function to configure their body pose estimation model, for example, to enable it to run in real time on the VS600 SoC at 30 frames per second.

However, security and privacy need to meet consumer expectations

The future of human-machine interfaces sounds bright, but the biggest obstacle to adoption may be that users think their privacy and security will be compromised. There are many recent stories in the news that confirm this concern. Any meaningful human-machine interface solution must take this into consideration.

Fortunately, the fact that these video, voice, and visual data will be processed in the device rather than in the cloud has made great strides in privacy. In the video doorbell example, by adding AI intelligence to the doorbell itself, the video from the front door does not need to be streamed to the cloud 24/7, but only when there is a specific event. For example, the AI ​​engine only transmits video when it detects that an evil person is approaching the door. Or, take our home yoga instructor as an example. The app can run completely on the device as we showed before, without sending any images from home to the cloud server at all.

However, even if these images have never been sent to the cloud, users may worry that even temporarily, these images will still be captured and processed in your device. There is also a security risk, a malicious person may try to obtain this data from your device. Therefore, the ideal smart home-centric AI solution must also ensure that the content is captured and processed in a safe manner, which is crucial.

Smart home security


The new era of the Internet of Things will be driven by more “local intelligence” (edge AI), which will reduce the need and risk of always staying connected. AI-driven neural networks that process on edge devices are the key to accelerating the adoption of perceptual intelligence systems. By being able to implement this function at the edge, the system can operate with higher security and privacy and lower latency. Can support the high performance of multi-mode interface solutions, multi-processor SoC (available at the price of the consumer market) will help developers quickly use AI innovation and make their products stand out.

How machines use voice, video, and visual data, and how to use them to understand and predictively respond to what we do (such as speaking or touching), thereby improving how the Internet of Things provides unprecedented security in our lives. Convenience and productivity.

Post time: Dec-30-2020

WhatsApp Online Chat !