Arseam

International Journal of Advances in Engineering & Scientific Research

BUILDING RELIABLE AND SECURE DATA PIPELINES: STRATEGIES FOR SCALABLE DATA ENGINEERING

Digvijay Waghela

Digvijay Waghela Big Data Architect India.

DOI : https://doi.org/10.53882/IJAESR.2024.1101006 Page No : 79-92

Published Online : 2024-12-25

Download Full Article : PDF Check for Updates

The evolving nature of cloud computing changes data management and storage. Thus, there is a need to have an ensured safe way of managing data. Currently, most security frameworks have the problem of imposing significant computational overhead, thus reducing efficiency. This paper proposes a lightweight data security system designed especially for cloud environments. Advanced encryption techniques, streamlined key management processes, and real-time threat monitoring characterize this system for the safety of data while minimizing resource consumption. This approach addresses the unique challenges of scalability and performance in cloud computing by balancing robust protection with operational efficiency. The effectiveness of the proposed solution in securing sensitive data is demonstrated, making it a practical and efficient alternative to traditional security methods.

Keywords: Cloud Security, Data Encryption, Lightweight Framework, Key Management and Scalability.

REFERENCES

Ansari, S. et. al (2018, December). A Review on Multisource Data Analysis using soft computing Techniques. In 2018 4th International Conference on Computing Communication and Automation (ICCCA) (pp. 1-6). IEEE." DOI: http://dx.doi.org/10.1109/CCAA.2018.8777551
Ansari, S. et. al (2022). A fusion of dolphin swarm optimization and improved sine cosine algorithm for automatic detection and classification of objects from surveillance videos. Measurement, 192, 110921. DOI: https://doi.org/10.1016/j.measurement.2022.110921
Ansari, S. et. al (2023). Multi video summarization using query based deep optimization algorithm. International Journal of Machine Learning and Cybernetics, 14(10). DOI: http://dx.doi.org/10.1007/s13042-023-01852-3
Ansari, S. et. al (2023, March). A Comprehensive Study on Video Captioning Techniques, Benchmark Datasets and QoS Metrics. In 2023 10th International Conference on Computing for Sustainable Global Development (INDIACom) (pp. 1598-1603). IEEE. Retrieved From: https://vufind-test.katalog.k.utb.cz/SummonRecord/FETCH-ieee_primary_101125243?sid=77480263
Ansari, S.A.,et. al (2024). MetaV: A Pioneer in feature Augmented Meta-Learning Based Vision Transformer for Medical Image Classification. Interdiscip Sci Comput Life Sci (2024). DOI: https://doi.org/10.1007/s12539-024-00630-1
Bharathy, S. et. al (2017). Hand Gesture Recognition for Physical Impairment Peoples. SSRG International Journal of Computer Science and Engineering (SSRG-IJCSE), 6-10. DOI: https://doi.org/10.14445/23488387/IJCSE-V4I10P102
F. Cremer, et. al (2022). Cyber risk and cyber security: a systematic review of data availability, Geneva Pap. Risk Insur. - Issues Pract. 47 (3) (2022) 698–736. DOI: https://doi.org/10.1057/s41288-022-00266-6
F. Thabit, et. al (2021). A new lightweight cryptographic algorithm for enhancing data security in cloud computing, Global Transitions Proceedings 2 (1) (2021) 91–99. DOI: https://doi.org/10.1016/j.gltp.2021.01.013
J. Chen, et. al (2023) Is cyber security risk factor disclosure informative? Evidence from disclosures following a data breach, J. Bus. Ethics (2022) 1–26. Retrieved From: https://ideas.repec.org/a/kap/jbuset/v187y2023i1d10.1007_s10551-022-05107-z.html
K. Huang, et. al (2020). A lightweight privacy-preserving CNN feature extraction framework for mobile sensing, IEEE Trans. Dependable Secure Comput. 18 (3) (2019) 1441–1455. Retrieved From: https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=6934&context=sis_research
K.M. Caramancion, et. al (2022). The missing case of disinformation from the cyber security risk continuum: a comparative assessment of disinformation with other cyber threats, Data 7 (4) (2022) 1–18. DOI: https://doi.org/10.3390/data7040049
P.K. Roy et. al (2023). Healthcare data security using lightweight protocol for cyber physical system, IEEE Transactions on Network Science and Engineering (2022) 1–10. DOI: https://doi.org/10.1109/TNSE.2022.3186437
S. Achar, (2022). Cloud computing security for multi-cloud service providers: controls and techniques in our modern threat landscape, International Journal of Computer and Systems Engineering 16 (9) (2022) 379–384. DOI: http://dx.doi.org/10.5281/zenodo.7084251
S. Ismail, et. al (2022). A lightweight multilayer machine learning detection system for cyber-attacks in WSN, in: IEEE 12th Annual Computing and Communication Workshop and Conference (CCWC), 2022, pp. 481–486. DOI: http://dx.doi.org/10.1109/CCWC54503.2022.9720891
S. Namasudra, et. al (2020). Towards DNA based data security in the cloud computing environment, Comput. Commun. 151 (2020) 539–547. DOI: https://doi.org/10.1016/j.comcom.2019.12.041
Sujithra, M., et. al (2022, October). A Study on Psychological Stress Of Working Women In Educational Institution Using Machine Learning. In 2022 13th International Conference on Computing Communication and Networking Technologies (ICCCNT) (pp. 1-7). IEEE. DOI: https://doi.org/10.1109/ICCCNT54827.2022.9984460
Suresh, K., et. al (2019). A novel key exchange algorithm for security in internet of things. Indones. J. Electr. Eng. Comput. Sci, 16(3), 1515-1520. DOI: http://dx.doi.org/10.11591/ijeecs.v16.i3.pp1515-1520
W. Iqbal, H. et. al (2020). ALAM: anonymous lightweight authentication mechanism for SDN-enabled smart homes, IEEE Internet Things J. 8 (12) (2020) 9622–9633. DOI: https://doi.org/10.1109/JIOT.2020.3024058
W. Xiaoyu, et. al (2020). Research and development of data security multidimensional protection system in cloud computing environment, in: International Conference on Advance in Ambient Computing and Intelligence (ICAACI), 2020, pp. 67–70. DOI: http://dx.doi.org/10.1109/ICAACI50733.2020.00019

Waghela D. (2024), “Building Reliable And Secure Data Pipelines: Strategies For Scalable Data Engineering”, International Journal of Advances in Engineering & Scientific Research,Volume 11, Issue 2, 2024, pp 79-92. DOI : https://doi.org/10.53882/IJAESR.2024.1101006

Article View: 76
PDF Download: 5

REFERENCES

Ansari, S. et. al (2018, December). A Review on Multisource Data Analysis using soft computing Techniques. In 2018 4th International Conference on Computing Communication and Automation (ICCCA) (pp. 1-6). IEEE." DOI: http://dx.doi.org/10.1109/CCAA.2018.8777551
Ansari, S. et. al (2022). A fusion of dolphin swarm optimization and improved sine cosine algorithm for automatic detection and classification of objects from surveillance videos. Measurement, 192, 110921. DOI: https://doi.org/10.1016/j.measurement.2022.110921
Ansari, S. et. al (2023). Multi video summarization using query based deep optimization algorithm. International Journal of Machine Learning and Cybernetics, 14(10). DOI: http://dx.doi.org/10.1007/s13042-023-01852-3
Ansari, S. et. al (2023, March). A Comprehensive Study on Video Captioning Techniques, Benchmark Datasets and QoS Metrics. In 2023 10th International Conference on Computing for Sustainable Global Development (INDIACom) (pp. 1598-1603). IEEE. Retrieved From: https://vufind-test.katalog.k.utb.cz/SummonRecord/FETCH-ieee_primary_101125243?sid=77480263
Ansari, S.A.,et. al (2024). MetaV: A Pioneer in feature Augmented Meta-Learning Based Vision Transformer for Medical Image Classification. Interdiscip Sci Comput Life Sci (2024). DOI: https://doi.org/10.1007/s12539-024-00630-1
Bharathy, S. et. al (2017). Hand Gesture Recognition for Physical Impairment Peoples. SSRG International Journal of Computer Science and Engineering (SSRG-IJCSE), 6-10. DOI: https://doi.org/10.14445/23488387/IJCSE-V4I10P102
F. Cremer, et. al (2022). Cyber risk and cyber security: a systematic review of data availability, Geneva Pap. Risk Insur. - Issues Pract. 47 (3) (2022) 698–736. DOI: https://doi.org/10.1057/s41288-022-00266-6
F. Thabit, et. al (2021). A new lightweight cryptographic algorithm for enhancing data security in cloud computing, Global Transitions Proceedings 2 (1) (2021) 91–99. DOI: https://doi.org/10.1016/j.gltp.2021.01.013
J. Chen, et. al (2023) Is cyber security risk factor disclosure informative? Evidence from disclosures following a data breach, J. Bus. Ethics (2022) 1–26. Retrieved From: https://ideas.repec.org/a/kap/jbuset/v187y2023i1d10.1007_s10551-022-05107-z.html
K. Huang, et. al (2020). A lightweight privacy-preserving CNN feature extraction framework for mobile sensing, IEEE Trans. Dependable Secure Comput. 18 (3) (2019) 1441–1455. Retrieved From: https://ink.library.smu.edu.sg/cgi/viewcontent.cgi?article=6934&context=sis_research
K.M. Caramancion, et. al (2022). The missing case of disinformation from the cyber security risk continuum: a comparative assessment of disinformation with other cyber threats, Data 7 (4) (2022) 1–18. DOI: https://doi.org/10.3390/data7040049
P.K. Roy et. al (2023). Healthcare data security using lightweight protocol for cyber physical system, IEEE Transactions on Network Science and Engineering (2022) 1–10. DOI: https://doi.org/10.1109/TNSE.2022.3186437
S. Achar, (2022). Cloud computing security for multi-cloud service providers: controls and techniques in our modern threat landscape, International Journal of Computer and Systems Engineering 16 (9) (2022) 379–384. DOI: http://dx.doi.org/10.5281/zenodo.7084251
S. Ismail, et. al (2022). A lightweight multilayer machine learning detection system for cyber-attacks in WSN, in: IEEE 12th Annual Computing and Communication Workshop and Conference (CCWC), 2022, pp. 481–486. DOI: http://dx.doi.org/10.1109/CCWC54503.2022.9720891
S. Namasudra, et. al (2020). Towards DNA based data security in the cloud computing environment, Comput. Commun. 151 (2020) 539–547. DOI: https://doi.org/10.1016/j.comcom.2019.12.041
Sujithra, M., et. al (2022, October). A Study on Psychological Stress Of Working Women In Educational Institution Using Machine Learning. In 2022 13th International Conference on Computing Communication and Networking Technologies (ICCCNT) (pp. 1-7). IEEE. DOI: https://doi.org/10.1109/ICCCNT54827.2022.9984460
Suresh, K., et. al (2019). A novel key exchange algorithm for security in internet of things. Indones. J. Electr. Eng. Comput. Sci, 16(3), 1515-1520. DOI: http://dx.doi.org/10.11591/ijeecs.v16.i3.pp1515-1520
W. Iqbal, H. et. al (2020). ALAM: anonymous lightweight authentication mechanism for SDN-enabled smart homes, IEEE Internet Things J. 8 (12) (2020) 9622–9633. DOI: https://doi.org/10.1109/JIOT.2020.3024058
W. Xiaoyu, et. al (2020). Research and development of data security multidimensional protection system in cloud computing environment, in: International Conference on Advance in Ambient Computing and Intelligence (ICAACI), 2020, pp. 67–70. DOI: http://dx.doi.org/10.1109/ICAACI50733.2020.00019

Submission Instruction

Call for Paper

Submit Paper Online

Join Editorial Board

Author Guidelines

Impact Factor: 6.013

Indexed in

BUILDING RELIABLE AND SECURE DATA PIPELINES: STRATEGIES FOR SCALABLE DATA ENGINEERING

Aims and Objectives

Ethics Policy

Peer Review Policy

Call for Paper

Conference

Instruction To Review

Guide To Authors