From Cloud to Edge: Enabling Offline IoT With Small-Scale Language Models

Large Language Models (LLMs) are increasingly becoming a central part of modern computing, simplifying complex tasks such as summarizing large documents, generating innovative ideas, and accelerating software development. Simultaneously, the proliferation of Internet of Things (IoT) devices in households, agriculture, and industrial settings has created new opportunities and challenges. A significant issue is the reliance on centralized servers for data analysis, which can lead to device malfunctions or outages during network disruptions. Advancements in LLMs, combined with the growing power of compact computing platforms, present a solution. Small-scale language models (SLMs) with 2–7 billion parameters can now run on devices as small as a Raspberry Pi. This enables the development of intelligent IoT edge systems capable of locally logging and analyzing sensor data, performing troubleshooting, and ensuring continuous operation without dependency on external servers. In this article, we propose an architecture for an IoT Edge system that integrates compact computing platforms with SLMs, machine-learning and embedded logic maintaining accuracy and efficient operation. This system leverages the natural language processing capabilities of SLMs to enable intuitive, human-like interactions with IoT data, while also utilizing native scripts to seamlessly interface with IoT devices. The proposed solution supports offline functionality, making it ideal for remote or rural deployments, and enhances usability for non-technical users through natural language interaction. We present a demonstration of an implementation of our proposed architecture and also studied the suitability of current off the shelf models for these proposed use case scenarios. In addition, we also present the future works and challenges that need to be addressed to rule out the concerns and list the possibilities for future work on this topic.