Beyond data deluge: cloud-grid architectures and end-to-end pipelines for big data processing in optical sensor systems: a scoping and narrative review

Abstract

There has been an unprecedented increase in volume, velocity, and variety of data, driven by the rapid evolution of optical sensor technology, including optical imaging, hyperspectral imaging, and fibre-optic sensors. The volume and complexity of data from modern optical sensors have now become beyond the capacity of current single -point or localised computing methods to efficiently process. As such, the need for scalable data processing, storage, and analysis through high-performance computing (HPC) infrastructure, cloud computing, or grid computing has gradually gained acceptance. However, to date, scattered research initiatives persist, lacking integration of architectural design patterns and end-to-end pipelines tailored to the needs of optical sensor data. Within the PRISMA-ScR framework, this research presents a scoping and narrative review of cloud- and grid-based architectures for processing large volumes of data from optical sensors. Only 8 of the 39 records uncovered after an exhaustive search of the Web of Science database met the inclusion criteria. Computing architectures are examined in the review for their mapping into cloud-focused, grid/HPC-focused, and hybrid cloud/grid models. The report reveals an emerging trend of reliance on distributed systems but also points out limitations in support for real-time processing and other aspects of transparency. To promote further research into computing frameworks for the next generation of optical sensors, the report concludes with a list of areas for further research.