HUAWEI AI Open Platform

Abstract

This chapter is an introduction to HUAWEI HiAI, an open AI capability platform for smart devices. Firstly, Architecture of HUAWEI HiAI Platform, a three-layer open architecture based on “Service, Engine and Foundation” that provides capabilities at chip, application and service, as well as its three sub modules, is introduced. Then the development of Apps based on HUAWEI HiAI and some solutions of HUAWEI HiAI are briefly introduced too.

This chapter is an introduction to HUAWEI HiAI , an open AI capability platform for smart devices. Firstly, Architecture of HUAWEI HiAI Platform, a three-layer open architecture based on “Service, Engine and Foundation” that provides capabilities at chip, application and service, as well as its three sub modules, is introduced. Then the development of Apps based on HUAWEI HiAI and some solutions of HUAWEI HiAI are briefly introduced too.

7.1 Introduction to HUAWEI HiAI Platform

Currently most consumers are exposed to such AI Apps as voice assistant, AI photography, image beautifying, whose application scenarios are relatively single and limited. In fact, with the evolution of on-engine AI to distributed AI and the sharing of resources and computing power among multiple terminals, the application scenarios of on-engine AI will be greatly broadened, which will further empower developers to achieve more intelligent innovation and bring the marvelous experience to consumers.

Based on the above background, Huawei launched HUAWEI HiAI 3.0. The evolution of HUAWEI HiAI platform has experienced single-device version 1.0, multi-device version 2.0 and current distributed scenario version 3.0, as shown in Fig. 7.1.

Fig. 7.1

Evolution of HUAWEI HiAI platform

AWEI HiAI 3.0 was officially released at Software Green Alliance Developer Conference on November 19, 2019, marking that on-engine AI is officially moving towards distributed AI. HUAWEI HiAI 3.0 will bring users the marvelous full-scene intelligent life experience.

HUAWEI HiAI 3.0 provides one-access service and multi-terminal adaptive operation. Users can enjoy convenient services such as voice assistant and HiBoard on mobile phones, tablets, smart screens, smart speakers and other devices, so that the same service can be realized on different devices. Here are two cases: private coaching and driving experience.

• Case 1: Private coaching. HUAWEI HiAI 3.0 opens its distributed Computer Vision (CV) and Automatic Speech Recognition (ASR), which are designed to enable users to exercise at home as effectively as being guided by a gym private coach. Distributed Computer Vision can recognize 3D human body key points so that users can capture multiple angles of motion posture in real time through multiple cameras at home, and correct posture through multiple screen displays. In addition, Automatic Speech Recognition assists users to control their motional rhythm and further helps them to enjoy private coaching at home.

• Case 2: Driving experience. Combined with distributed technology, HUAWEI HiAI 3.0 enables users to connect smart phones with cars, so that safety detection of users’ driving behavior is carried out through the in-car camera, and the AI chip computing power provides safety reminders for dangerous behaviors such as fatigue driving. Through the in-car network environment and local data operation with lower delay, drivers can better protect themselves from accidents.

7.1.1 Architecture of HUAWEI HiAI Platform

HUAWEI HiAI platform builds a three-layer ecosystem of “Service, Engine and Foundation”. Service supports a rich front-end mainstream framework while Engine provides various upper functional business APIs, which can run efficiently on mobile devices, and Foundation flexibly schedules heterogeneous resources to meet the needs of developers so as to accelerate neural network model calculation and operator calculation. Moreover, HUAWEI HiAI provides a systematic tool chain, complete documents, various APIs and quick-start source code, which can enable rapid application development. The architecture of HUAWEI HiAI mobile computing platform is shown in Fig. 7.2.

Fig. 7.2

The architecture of HUAWEI HiAI mobile computing platform

HUAWEI HiAI is a mobile terminal oriented AI computing platform. Compared with on-service AI, on-engine AI has three core advantages: more security, lower cost and less delay. HUAWEI HiAI constructs three layers of AI Ecology: open service capability, open application capability and open chip capability. The three-layer open platform of “Service, Engine and Foundation” brings more marvelous experience to users and developers. Each layer features as follows.

• 1. Service: Create once, reuse many times.

• 2. Engine: Distributed, full scenario.

• 3. Foundation: Greater computing power, more operators; more frames, smaller models.

The three-layer AI ecology of HiAI is shown in Fig. 7.3.

Fig. 7.3

The three-layer AI ecology of HiAI

HUAWEI HiAI enables App to have the following values: real-time, timeliness, stability, security and cost.

The biggest feature of HUAWEI HiAI 3.0 platform is that AI enables distributed full scenarios. The architecture of HUAWEI HiAI consists of three layers: Service, Engine and Foundation. The corresponding sub module of Device is Service, which is mainly to open the service capability. It will push the service timely and appropriately according to the needs of the user, so that the service can actively find the user. What it brings to the user is to create once and reuse many times. The sub module corresponding to Engine is called HiAI Engine, which mainly provides API to open AI application capabilities. It can easily integrate a variety of AI capabilities with App, making App more intelligent and powerful. Through HiAI engine, you can call various algorithms in HiAI platform and integrate them in App. For example, if you want to achieve image recognition, character recognition, face recognition, speech recognition, natural language understanding, you can directly call the API in HiAI Engine. HiAI engine can be used in distributed and full scenarios. Foundation is a batch of chips, mainly based on Huawei’s Kirin chip, with the chip capability open. The sub module corresponding to Foundation is called HiAI Foundation, which is mainly responsible for providing operators. It can quickly transform and migrate existing models, and obtain the best performance with the help of heterogeneous scheduling and NPU acceleration. The chip provides more operators, greater computing power, and more frameworks to simplify the model. If you want to migrate some of AI applications that have been developed locally to terminal devices, you can use HiAI Foundation to transform the model to fit the terminal devices.

The three sub-modules are highlighted below.

7.1.2 HUAWEI HiAI Foundation

HiAI Foundation AP is an Artificial Intelligence Computing Library in mobile computing platform, which is designed for developers of artificial intelligence applications. It allows developers to easily and efficiently write artificial intelligence applications running on mobile devices. It features as follows.

• 1. Based on the constant improvement of high performance and high precision of Kirin chip, it provides better AI performance with greater computing power.

• 2. The number of supported operators exceeds 300, which is the largest in the industry, and more frameworks are supported, so the flexibility and compatibility have been greatly improved.

• 3. Honghu chip, Kirin chip, AI Camera chip and full scene chip enable more AI capabilities of devices.

HiAI Foundation API is going to be released as a unified binary file. The set of API aims to accelerate the calculation of neural network through HiAI heterogeneous computing platform. Currently, it only supports running on Kirin SoC.

By using HiAI Foundation API, developers can focus on novel AI application development rather than computing performance optimization.

HiAI Foundation API is integrated on Kirin SoC chip, which provides developers with running environment and debugging tools based on mobile devices. Developers can run neural network model in mobile devices and call HiAI Foundation API to accelerate computing. HiAI Foundation API does not need to be installed, for it supports relevant integration, development and verification by using the default image of mobile devices.

The following two main function are provided for AI application developers by HiAI Foundation API.

• 1. It provides a common AI business API, which can run efficiently on mobile devices.

• 2. It provides an acceleration API, independent of processor hardware, so that application manufacturers and developers can accelerate model computing and operator computing on HiAI heterogeneous acceleration system.

The following basic functions are supported by HiAI Foundation API.

• 1. It supports AI model management interfaces such as model compilation, model loading, model running and model destruction.

• 2. It supports basic operator computing interfaces, including convolution, pooling, full link.

HiAI Foundation supports special AI instruction set for neural network model operation, so that more neural network operators can be executed efficiently and in parallel with the least clock cycle.

HiAI Foundation does the offline compilation of various neural network operators, such as convolution, pooling, activation, full link into the special AI instruction sequence of NPU through tools. At the same time, it rearranges the data and weight, and fuse the instruction and data to generate the offline execution model. When compiling offline, the operators that can be fused between the front and back layers (convolution, activation function Relu, pooling) can be fused between layers. This method reduces the read-write bandwidth of DDR and improves the performance.

HiAI Foundation rearranges the relevant data (Batch, Channel, Height, Width) in the neural network model in an efficient way, especially the channel data of characteristic graph. During the convolution operation, the calculation efficiency of associated with the channel is greatly improved.

HiAI Foundation supports sparse model acceleration. Under the premise of no loss of calculation accuracy, the weight is set to zero and sparse optimization is carried out. NPU skips the multiplication and addition operation with zero coefficient, which greatly improves the calculation efficiency and reduces the bandwidth.

Figure 7.4 shows that the trained neural network model is generated by the compiler tool, which is executed efficiently on HiAI Foundation, and saved as a binary file offline model.

Fig. 7.4

Neural network model compiled into offline model

The standard neural network model, such as Caffe, is compiled and converted into an offline model. The purpose of compilation is to optimize the network configuration and generate the optimized object file, which is the offline model. The offline model is serially stored on the disk. When the neural network is used for forward calculation, the optimized object file is used for calculation directly, and the speed is faster.

Figure 7.5 shows that in offline model calculation, the offline model is loaded from the file, and the user’s input data (such as pictures) is copied to the NPU of HiAI for calculation. In the process of calculation, each inference only needs to import and export user data from DDR to NPU once.

Fig. 7.5

Loading calculation of offline model

HUAWEI HiAI Foundation supports a variety of intelligent platform frameworks, including Caffe and TensorFlow, and different intelligent platform frameworks are used. The third party needs to indicate the specific intelligent platform framework to be used in this calculation in the interface, and other interfaces and parameters need no modification.

HUAWEI HiAI Foundation supports most models and neural network operators, and continues to optimize and improve them.

7.1.3 HUAWEI HiAI Engine

HUAWEI HiAI Engine, as an open platform of application capabilities, easily integrates a variety of AI capabilities with Apps, making Apps more intelligent and powerful. HUAWEI HiAI Engine 3.0 adds some API recognition capabilities on the basis of the previous one, so that the number of underlying APIs exceeds 40. HUAWEI HiAI Engine not just allows users to directly call the existing API, but helps developers to focus on business development. To achieve image recognition, voice processing and other functions, users just need to put the integrated API into App. Moreover, in HUAWEI HiAI 3.0, APIs such as computer vision and speech recognition are distributed, which helps developers develop more full scene intelligent life experiences.

HiAI’s open application engines include CV engine, ASR engine, NLU engine, etc. According to the survey results of developers’ demands for HiAI’s capabilities, more than 60% of the respondents focused on CV, ASR, NLU.

• 1. Computer Vision (CV) features the capability of computer to simulate human visual system to perceive the surrounding environment, which is composed of judgment, recognition and understanding space. It includes image super resolution, face recognition, object recognition and so on.

• 2. Automatic Speech Recognition (ASR) designates the capability of converting human voice into text for further analysis and understanding by computer. It includes speech recognition, speech conversion and so on.

• 3. Natural Language Understanding (NLU) indicates the capability of computer to understand human voice or text, communicate or act naturally in combination with ASR. It includes word segmentation, text entity recognition, sentiment bias analysis, machine translation and so on.

The application scenarios and open engines of HUAWEI HiAI Engine are shown in Table 7.1, and the specific API introduction is shown in Appendix 1 in Chap. 8.

Table 7.1 The application scenarios and open engines of HUAWEI HiAI engine

7.1.4 HUAWEI HiAI Service

HUAWEI HiAI Service API achieves intelligent distribution of Pan terminal, so that developers only need to access the service once before they reuse it in mobile phones, tablet computers and other terminals to complete the distribution efficiently. HiAI Service API can timely and appropriately recommend AI applications or services to users, so that users quickly gets what they need in massive services. At the same time, AI applications also accurately connects users. With the help of HiAI Service API, each function or content in the application is split into individual atomic services for push. HiAI Service API has the function of multi-scene and multi-entry precise distribution. HiAI Service API is used in multiple portals such as HiBoard intelligent assistant, global search, HiVoice, HiTouch and HiVison to recommend and display relevant applications according to users’ habits or search contents, voice instructions and other operations, making the corresponding applications reach users more intelligently and accurately. HiAI Service API connects man and services intelligently, realizing the experience upgrading from “man hunting for services” to “services hunting for man”.

7.2 Application Development Based on HUAWEI HiAI Platform

HUAWEI HiAI also provides IDE, a development tool for rapid integration of HiAI capabilities, aiming to help developers use Huawei EMUI open capabilities quickly, conveniently and efficiently. Based on Android Studio function extension (provided in plug-in form), IDE supports HiAI Engine, HiAI Foundation (AI model analysis, AI model transformation, business class generation, AI model market), etc. IDE supports drag operation, fast and efficient integration, and provides free remote real machine service (3000 + AI real machine, 7 × 24 h remote one-click debugging).

IDE supports operating systems such as Android studio 2.3. X and above, Windows 7, Windows 10, Mac OS 10.12/10.13. If the operating system does not meet the requirements, it only affects the AI local model conversion function.

IDE selects corresponding functions according to actual scenarios: HUAWEI HiAI Engine is selected when using EMUI AI API, while HUAWEI HiAI Foundation is selected to convert TensorFlow/Caffe model into HUAWEI HiAI model, and then integrate model into App. And HUAWEI HiAI Service is used as a service provider for ordinary App.

HiAI is perfectly integrated with Android Studio, that is, HiAI is used as a plug-in of Android Studio, as shown in Fig. 7.6.

Fig. 7.6

HiAI IDE integrated with android studio

HiAI platform plug-in provides the functions of HiAI Engine and HiAI Foundation. HiAI Engine mainly provides API integrated with App, which can be called directly. HiAI Foundation integrates the trained models, which can be downloaded and used directly, as shown in Fig. 7.7.

Fig. 7.7

Function of HiAI integrated with android studio

After the development of APP is completed, it will enter the real machine debugging phase. Huawei provides a full range of Huawei remote real machine debugging services for developers. Developers can access the real machine of Huawei remote terminal laboratory with one click, conducting real-time remote control and single-step debugging, with provided profiling and log. Some Huawei models supported by HiAI are shown in Fig. 7.8.

Fig. 7.8

Huawei models supported by HiAI

The steps of App integrated into HiAI DDK are as follows: Firstly, we get the framework models such as trained Caffe/TensorFlow, and then use the provided OMG model conversion tool to convert the original open source framework model into OM model, which contains 8-bit quantization function, suitable for DaVinci platform. Finally App integration is carried out, which includes model preprocessing, model inference and other parts, as shown in Fig. 7.9.

Fig. 7.9

Procedure of App integrated into HiAI DDK

The integration process of App is as follows.

• 1. Step 1: Create a project.

  • (a) Create an Android Studio project and check the “Include C++ support” option.

  • (b) C++ Standard select C++ 11, check “Exceptions Support (-fexceptions)” option, check “Runtime Type Information Support(-frtti)” option.

• 2. Step 2: Compile JNI.

  • (a) Realize JNI, write Android.mk Documents.

  • (b) rite Application.mk File, copy sdk so to the resource library.

  • (c) In build.gradle File, specify NDK to compile C++ files.

• 3. Step 3: Model integration.

  • (a) Model preprocessing: application layer model preprocessing, JNI layer model preprocessing.

  • (b) Model Inference.

7.3 Part of the HUAWEI HiAI Solutions

7.3.1 HUAWEI HiAI Helping the Deaf and Dumb

Hearing-impaired children cannot enjoy normal good time because of their physical handicaps. They cannot hear the greetings from their family and friends. For them, the world is silent and lonely. According to the statistics, there are about 32 million hearing-impaired children in the world. They can neither hear beautiful voices nor speak out their inner thoughts i, so their way of communicating with the world is fraught with obstacles.

The cruel reality is that the parents of 90% of hearing-impaired children are able-bodied. However, 78% of them are unable to communicate with their children. Hearing impaired children have great difficulties in language learning and reading.

Language is the basis of listening, speaking, reading and writing, and listening is the only way to language learning. For example, when encountering a new word, normal children can understand its meaning by listening to the explanation of adults, and then they can master it through continuous listening, speaking, reading and writing. Hearing-impaired children, however, cannot do it, as for them language learning is carried out through sign language. Without the help of professional sign language teachers, they can not communicate with ordinary people.

Therefore, together with European Union for the Deaf (a non-profit organization), Penguin Group and Aardman (an animation master), Huawei has developed an application StorysSign. With the aid of Huawei HiAI platform’s some capabilities such as Image Recognition and Optical Character Recognition (OCR), if you use your mobile phone to face the text on the book, Huawei HiAI will animate the text immediately. Sister Xingxing uses sign language to express the text on the book, as shown in Fig. 7.10.

Fig. 7.10

Huawei HiAI animating the text

7.3.2 HUAWEI HiAI Enhancing the Visual Effects of Yuanbei Driving Test Application

Yuanbei driving test is a tailor-made driving learning application for beginners. It provides illustrated driving test services, including enrollment for driving school, booking learning, and simulation driving test. It is committed to establishing a convenient and practical one-stop driving test platform.

Simulation driving test, one of the main features of Yuanbei driving test, combines graphics, video, voice and other forms of the built-in installation package. It effectively helps learners quickly get familiar with the test content and specifications, so that they can pass the driving test quickly.

Simulation test contains a large number of pictures to assist learners to practice, but the practice might be impacted due to some low-quality pictures’ poor display and clarity lack on the ordinary mobile phones.

On most devices, image optimization program of simulation driving test relies on the Internet. Therefore, in the case of weak network signal or no network, the improvement of picture quality will be significantly hindered.

HUAWEI HiAI adopts intelligent noise reduction and 9× resolution amplification, which significantly improves the image quality, bringing users more clear image details, and comprehensively enhancing the user’s visual experience.

Relying on HUAWEI HiAI on-device learning model, Yuanbei driving test achieves the optimization and amplification of on-device pictures, and the same pictures are displayed more clearly on Huawei NPU models. Meanwhile, free from network dependence, users can still view high-quality amplified pictures when the network is unstable or disconnected, as shown in Fig. 7.11.

Fig. 7.11

HUAWEI HiAI enhancing the visual effects of Yuanbei driving test application

7.3.3 HUAWEI HiAI Empowering Ctrip Travel

Ctrip mobile app provides users with comprehensive travel services, including hotel reservation, air tickets, train tickets, travel guides, admission ticket discounts, travel insurance, etc.

During the journey, users often take many photos, hoping to capture the beautiful scenery for memory. But due to the lack of professional photography knowledge, most people cannot judge accurately whether the photos are good or not, and they always doubt whether the photos have taken the best effect. At the same time, the pictures taken by users are not clear enough, and the rendering effect is poor. Therefore, improving the image quality has become the appeal of many users.

By accessing the aesthetic scoring capability of HUAWEI HiAI Engine, it automatically integrates the technical factors such as defocus and jitter of the image, and the subjective aesthetic feeling such as skew, color and composition, evaluate and score the image. Users can quickly understand the photo quality through the score level, solve their doubts, and adjust accordingly, so as to take the most beautiful scenery.

In addition, with the help of HUAWEI HiAI, the application also realizes the functions of voice wake-up and poem writing in one-click, which provides users with a lot of conveniences, as shown in Fig. 7.12.

Fig. 7.12

HUAWEI HiAI empowering Ctrip with poem writing in one-click

7.3.4 HUAWEI HiAI Empowering WPS Document Error Detection and Correction

WPS App is an office software, which can edit and view common office documents such as text, forms, presentation. Meanwhile, users can use free cloud space and document templates.

With the rise and development of mobile devices, mobile phones are increasingly used in editing documents, sending and receiving e-mails. However, without the assistance of keyboard or mouse, only through the fingers on the screen to complete the operation, mobile phone office efficiency is extremely low. For example, when attending a class or attending a meeting or training, we see the key points and “essences” on the presentation, we will take out our mobile phone immediately to take photos and record them. However, the photos often need clipping on the computer before sorted into presentation, which is quite cumbersome and time-consuming.

• 1. Surroundings interference: In addition to the presentation, there are surrounding interference such as the screen, walls, desks and chairs in the pictures taken by users, which need to be cropped.

• 2. Distorted document image: When the shooting angle is not directly facing the document, the document imaging will be distorted to varying degrees, and the stretched or compressed image will affect the subsequent use.

• 3. Blur image: Restricted in light, distance and other factors, the images taken by users may be blur, which will subsequently affect the perception and information recognition.

• 4. Non-editable content: Many users want to edit or modify the content of the presentation when viewing the captured presentation pictures. However, the image content cannot be edited directly.

WPS can easily solve the above problems by accessing to HUAWEI HiAI ecology and the powerful performance of Huawei Kirin 970 processor. It only takes 3 s to generate a presentation from multiple pictures with one click.

• 1. Document perception for automatic identification of document valid area: By accessing to HUAWEI HiAI Engine document detection and correction capability, WPS accurately perceives the area where the document is located, and automatically clip the surroundings such as the screen, walls, desks and chairs, as shown in Fig. 7.13.

• 2. Document correction for quick adjustment to center view: It is an auxiliary enhancement in the process of document remaking, which automatically adjusts the camera angle directly to the document, with the maximum correction angle of 45°, as shown in Fig. 7.14.

• 3. Text super-resolution for clearer text in the document: HUAWEI HiAI magnifies the image that contains the text content by nine times of resolution (three times of height and width), as a result, the quality of the picture is significantly improved and the text is legible, as shown in Fig. 7.15.

• 4. OCR recognition for free editing of text content in pictures: By accessing to general OCR, WPS automatically recognizes and extracts text information in pictures, achieving free modification, clipping, copying and deleting of text content in presentation pictures, as shown in Fig. 7.16.

  Fig. 7.13

  

  WPS document perception

  Fig. 7.14

  

  WPS document correction

  Fig. 7.15

  

  Text super-resolution

  Fig. 7.16

  

  WPS OCR recognition

For more solutions, please pay attention to the official website of HiAI.

7.4 Chapter Summary

This chapter mainly introduces the three-layer ecological structure of HUAWEI HiAI platform: HUAWEI HiAI Foundation, HUAWEI HiAI Engine, and HUAWEI HiAI Service API. The relevant capabilities of each layer, and some HiAI solutions are introduced as well.

Finally, in order to connect developers in an all-round way, HUAWEI HiAI has adopted the following activities to encourage innovation and achieve ecological win-win.

• 1. HUAWEI HiAI has organized the following profound communication activities of offline connection.

  • (a) HUAWEI Developer Day salons.

  • (b) HUAWEI HiAI open classes.

  • (c) HUAWEI HiAI special technical meetings.

• 2. HUAWEI HiAI has organized the following activities worth of $1 billion to spur full scene innovation.

  • (a) Open innovation of terminal capability.

  • (b) Digital service innovation of full scene.

  • (c) Ecological co-construction of Cloud service.

• 3. HUAWEI HiAI has organized the following innovation contests.

  • (a) AI application innovation contest.

  • (b) Future application creativity contest.

  • (c) AR application innovation contest.

Huawei believes that AI can make users’ lives better. Whether it is the back-end or the terminal, AI can break through the imagination and bring users with unprecedented convenience. But all of these need to have practical application scenarios, in which more enterprises and developers can participate so that users can get substantial experience improvement. Huawei is willing to cooperate with more talents and enterprises, so as to jointly promote the implementation of industry intelligence based on HUAWEI HiAI 3.0 platform.

7.5 Exercises

• 1. HUAWEI HiAI 3.0 was officially released at Software Green Alliance Developer Conference on November 19, 2019, marking that on-engine AI is officially moving towards distributed, which will bring the ultimate full scene intelligent life experience. What is the three-layer AI ecology of HUAWEI HiAI?

• 2. Which layer of HUAWEI HiAI can compile the standard neural network model into offline model?

• 3. Which layer of HUAWEI HiAI can easily integrate multiple AI capabilities with App to make App more intelligent and powerful?

• 4. Which tool can HiAI integrate with perfectly?

• 5. What is the integration process of APP?