Edge-Level Rate Limiting in Influx DB: Enhancing Performance and Scalability

Abstract

In the realm of the Internet of Things (IoT), the volume of data being fed into time series databases (TSDBs) is continuously increasing. When deployed on the edge with limited resources, Influx DB may be unable to process all incoming requests if it exceeds its rate limit. Moreover, its memory usage will increase, resulting in longer response times or even process crashes. Vertical scaling and migration come with high costs. Several research efforts are dedicated to enhancing the Influx DB engine. In this study, a lightweight rate limiter layer is introduced to operate within the pipeline of insertion requests, aiming to regulate traffic flow and manage the requests directed to Influx DB more effectively. This mechanism significantly improves Influx DB's scalability and performance, particularly under conditions of imposed rate limiting. Experimental results demonstrate that the proposed rate limiter enhanced the number of successful ingestion requests while maintaining an acceptable level of increased request latency. Furthermore, the solution also optimized the memory usage pattern in Influx DB.