The Repository @ St. Cloud State

Open Access Knowledge and Scholarship

Date of Award

6-2025

Culminating Project Type

Starred Paper

Styleguide

apa

Degree Name

Information Assurance: M.S.

Department

Information Assurance and Information Systems

College

Herberger School of Business

First Advisor

Susantha Herath

Second Advisor

Erich Rice

Third Advisor

Jieyu Wang

Creative Commons License

Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.

Keywords and Subject Headings

Data Pipeline, Chaos Engineering, Reliability & Resiliency

Abstract

Data engineering plays a key role in how businesses function today. Whether it's generating reports or supporting real-time applications, companies depend on data pipelines to move and process information efficiently. However, ensuring that these pipelines remain reliable and resilient is becoming more difficult. As systems grow more complex, even minor issues can cause delays, data mistakes, or failures that are not easily detected through traditional testing methods.

When data pipelines fail, the impact on a business can be serious. It can lead to lost revenue, missed opportunities, and poor decisions caused by missing or incorrect data. These issues are often not noticed until they start affecting day-to-day operations by which time, the damage has already occurred.

This paper shows how Chaos Engineering can be applied to ETL data pipelines to test how they behave during failures. By introducing controlled issues in a test environment, we can see how the pipeline reacts, where it fails, and how well it recovers. These experiments help uncover problems that may not be found during regular testing and offer useful insights for creating more reliable and production-ready data pipelines.

Comments/Acknowledgements

I would like to thank everyone at the Department of Information Assurance at St. Cloud State University for their support and guidance throughout my studies. A special thank you to Prof. Susantha Herath, my committee chair, for all his valuable feedback and encouragement during this research.

I am also grateful to Prof. Erich Rice and Prof. Jieyu Wang for their helpful advice and support, which played a big role in the developing of this work.

Lastly, I want to thank my family and friends for their constant support and encouragement. Their understanding and motivation kept me going throughout this journey.

Download

Share

COinS