Is Your Mobile Phone Smart? Antutu AI Benchmark Public Beta Is Released
At present, the word AI is not new to us. Whether in terms of PC end, mobile end, mobile phone or car, it seems to be a bit out of date without mentioning AI. AI has become relevant to our lives, affecting every aspect of our lives.
Though AI is strictly an algorithm, the support from software and hardware has a close relationship to it, with the integration between them bringing a performance difference.
In order to provide you with an intuitive judgment on AI performance of your mobile phone, Antutu formally released the "Antutu AI Benchmark" public beta, giving you a quantifiable standard to determine the difference in AI performances between different platforms.
However, it is important to note that there is no unified standard about AI in the industry at present and different chip vendors have various understanding to AI. For example, Qualcomm processes AI-related operations through DSP inside SoC while Hisilicon Kirin performs AI operations through an internal separated NPU. In addition, Samsung and MTK also have added separated AI chips in their next generation chips, which are known as NPU for Samsung and APU for MTK.
Apart from differences in hardware, there is also no unified standard in SDK, and each vendor offers a SDK for its own AI chip, whereas SDK of Qualcomm is called SNPE while that of MTK, Hisilicon Kirin and NVIDIA are called NeuoroPilot, HiAI and TensorRT respectively (SDK of Samsung has not been published by far). These SDKs will be presented in the Antutu AI Benchmark.
Antutu has established a unified standard for testing AI Performance by working with the above manufacturers. The test is divided into two subcategories: image classification and object detection.
Among them, the image classification is based on the Inception v3 neural network, with 200 images as test data; and that object detection is based on the mobilenet ssd neural network, with a video of 600 frames as test data.
Antutu recommends users to test in Wi-Fi status as the software installation package size is only 33 MB, but the data package is up to 160 MB. Data package will be automatically downloaded when users click start to test for the first time (there will be a prompt in non-Wi-Fi network status). A verification for integrity of data package will be carried out after download is completed. Then the test will formally start.
Data package download is required only when first running Antutu AI Benchmark. In case of subsequent re-testing, the system will verify the integrity of data package each time other than downloading it again. But, in case of damage, the data package needs to be re-downloaded prior to test.
In addition, different platforms have data packages with different sizes so that Antutu converts the original neural network into the manufacturers supported neural networks with SDKs provided by each manufacturer. Although the data package size is different, the final test sample is identical and all chips are tested under a unified standard.
If the chip itself does not support AI-related algorithms, or the manufacturer provided SDK is temporarily unable to support the network of Antutu AI Benchmark, image classification and object detection will use TFLite to call CPU for operation, which is low in efficiency with poor performance.
Scoring Standard:1. The score is related to speed and accuracy, namely, the faster speed and higher accuracy enable a better final score.
2. When the speed is faster but with a poor accuracy, Antutu AI Benchmark will adopt targeted penalty measures to keep final score from being influenced by cheating that decreasing the accuracy to increase the speed, and vice versa.
Special Cases Description:
1. Because Antutu AI benchmark focuses on AI operational capability, there is no obvious difference in performance for the same AI processors (for example, the DSP models of Qualcomm 845 and 710 are the same, so the score difference between them is marginal).
2. Samsung has not yet released its AI SDK, so testing for Samsung chip is by its CPU, leading to poor performance. This situation will be improved after Samsung releases its SDK.
3. The Hisilicon's HiAI engine object recognition currently uses TFLite to perform operations on the CPU, resulting in low scores. This situation will also be improved after HiAI is upgraded by Hisilicon.
4. Mobile chip of NVIDIA completes AI algorithms through GPU with floating point algorithms by TensorRT engine.
5. Android versions also have an impact on test results, and theoretically the performance of the same chip on Android 9.0 will be higher than that in Android 8.0, as Google has been optimizing support for AI at the system level.
After the release of Antutu AI benchmark official version, Antutu will release Antutu AI Mobile Tool at appropriate time to make things convenient for developers to call hardware acceleration function on each platform and speed up the development of mobile AI application industry.