A Unified Architecture for Real-Time Analytics Using Microsoft Fabric OneLake

Narendra Mangala

doi:10.5281/zenodo.18759771

Authors

Narendra Mangala Sogeti USA, 8075 Washington Village Drive, Washington Township, Dayton, Ohio 45458.

DOI:

https://doi.org/10.5281/zenodo.18759771

Keywords:

Microsoft Fabric, OneLake, Real-Time Analytics, Transformer Autoencoder, Anomaly Detection, Multivariate Time-Series, Data Integration

Abstract

In this study, in the day and age of digital disruption, there is a substantial task at hand regarding the volume and speed of data associated with the Internet of Things (IoT), financial transactions, and logs. A complex architecture with complex ETL (Extract, Transform, Load) and disconnected storage schemes typically causes substantial data latency, hindering real-time decision support. To overcome these challenges, this paper presents an integrated architecture that leverages Microsoft Fabric OneLake, a common data lake infrastructure that requires no physical data transfer via shortcuts, serving as a starting point for fast, real-time analytics. The backbone of this architecture consists of an intelligent analytics layer incorporating a Transformer Autoencoder. Contrary to traditional approaches that use Linear Regression (LR), Long Short-Term Memory (LSTM), and Gated Recurrent Units (GRU), which rely on self-attention in Transformers to directly compute the temporal importance of all timestamps, this model achieves revolutionary accuracy in characterizing seasonality and long-term dependencies in multivariable time series data. The algorithm includes an unsupervised learning-based rebuilding strategy that identifies anomalies when corresponding activities in a system fall above or below a data-driven threshold for normal operations. Experiments performed on a smart grid real-time load data test set reveal that the proposed strategy using a Transformer Autoencoder attains maximum accuracy, precision, and F1 scores of 96.0%. This result clearly beats all comparison algorithms, namely Linear Regression (88.5), Isolation Forest (90.8), LSTM (92.3), and GRU (93.4). Additionally, this strategy is integrated with Microsoft Fabric Real-Time Intelligence workload.

References

Shaheen, K. (2024). Microsoft Fabric: The All-in-One Analytics Solution for the Era of AI. International Journal of Computer Science and Information Technology Research, 12(1), 45-58.

Chen, Z., Chen, D., Zhang, X., Chakravarty, S., & Wu, Z. (2024). Learning Graph Structures with Transformer for Multivariate Time-Series Anomaly Detection. IEEE Transactions on Neural Networks and Learning Systems.

Nie, Y., Nguyen, N. H., Sinthong, P., & Kalagnanam, J. (2023/2024 reprint). A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. International Conference on Learning Representations (ICLR).

Zhang, Y., & Chen, H. (2024). An Improved Isolation Forest Algorithm Based on Transformer Embedding for High-Dimensional Data. Journal of Cloud Computing, 13(1), 12-28.

Bondar, S., et al. (2024). Real-Time Data Processing Architectures in Cloud Environments: A Comparative Study of Microsoft Fabric and AWS. Journal of Big Data Research, 34, 100412.

Xu, J., et al. (2024). Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI).

D. Schaffer et al., “Data Fabric Architectures for Real-Time Enterprise Analytics,” IEEE Software, vol. 41, no. 1, pp. 45–53, 2024.

A. Zimek et al., “Challenges of Classical Anomaly Detection in High-Dimensional Data Streams,” ACM Computing Surveys, 2024.

H. Wen et al., “Scalability Limits of RNN-Based Models for Real-Time Time-Series Analytics,” IEEE TNNLS, 2024.

Y. Li et al., “Transformer-Based Time-Series Modeling with Long-Range Dependency Awareness,” arXiv, 2025.

S. Kumar et al., “Industrial-Scale Transformer Autoencoder for Real-Time Anomaly Detection,” IEEE IoT Journal, 2025.

Choppa, Narendra Kumar Reddy, and Mark Knipp. "Advancing Generative AI with GraphQL API: Unified Data Access in Microsoft Fabric Ecosystem." Journal of Computer Science and Technology Studies 7, no. 5 (2025): 438-450.

Lu, Daixing, Kun Wang, Yubo Wang, and Ye Shen. "The Proposal and Validation of a Distributed Real-Time Data Management Framework Based on Edge Computing with OPC Unified Architecture and Kafka." Applied Sciences 15, no. 12 (2025): 6862.

Petchiappan, Venkateswaran. "AI-Powered Data Load Automation from SAP HANA to Cloud Platforms with Instant Error-Handling Techniques." Journal of Computer Science and Technology Studies 7, no. 5 (2025): 272-282.

Tera, Sougandhika. "Overcoming Early-Stage Adoption Challenges of Microsoft Fabric in Enterprise Environments." International Journal of AI, BigData, Computational and Management Studies 6, no. 4 (2025): 82-84.

Gallucci, Guillermo Jose. "Adaptation and Implementation of an automated Corporate ETL Framework on Microsoft Fabric: A Technical Approach to Integrating Data Workflows into a Modern Cloud Platform." PhD diss., Politecnico di Torino, 2025.

Ahmed, S. T., Mahesh, T. R., Srividhya, E., Vinoth Kumar, V., Khan, S. B., Albuali, A., & Almusharraf, A. (2024). Towards blockchain based federated learning in categorizing healthcare monitoring devices on artificial intelligence of medical things investigative framework. BMC Medical Imaging, 24(1), 105.

Fathima, A. S., Basha, S. M., Ahmed, S. T., Khan, S. B., Asiri, F., Basheer, S., & Shukla, M. (2025). Empowering consumer healthcare through sensor-rich devices using federated learning for secure resource recommendation. IEEE Transactions on Consumer Electronics.

Pasha, A., Ahmed, S. T., Painam, R. K., Mathivanan, S. K., Mallik, S., & Qin, H. (2024). Leveraging ANFIS with Adam and PSO optimizers for Parkinson's disease. Heliyon, 10(9).

Kumar, V. N., Sivaji, U., Kanishka, G., Devi, B. R., Suresh, A., Madhavi, K. R., & Ahmed, S. T. (2023). A framework for tweet classification and analysis on social media platform using federated learning. Malaysian Journal of Computer Science, 90-98.