Enabling Distributed Inference of Large Neural Networks on Resource Constrained Edge Devices using Ad Hoc Networks

Abstract

Processing neural network inferences on edge devices, such as smartphones, IoT devices and smart sensors can provide substantial advantages compared to traditional cloud-based computation. These include technical aspects such as low latency or high data throughput – but also data sovereignty can be a concern in many applications. Even though general approaches of distributed inference have been developed recently, a transfer of these principles to the edge of the network is still missing. In an extreme edge setting, computations typically are severely constraint by available energy resources and communication limitations among individual devices, which makes the distribution and execution of large-scale neural networks particularly challenging. Moreover, since mobile devices are volatile, existing static networks are unsuited and instead a highly dynamic network architecture needs to be orchestrated and managed. We present a novel, multi-stage concept which tackles all associated tasks in one framework. Specifically, distributed inference approaches are complemented with necessary resource management and network orchestration in order to enable distributed inference in the field – hence paving the way towards a broad number of possible applications, such as autonomous driving, traffic optimization, medical applications, agriculture, and industry 4.0.