The Repository @ St. Cloud State

Open Access Knowledge and Scholarship

Date of Award


Culminating Project Type

Starred Paper



Degree Name

Computer Science: M.S.


Computer Science and Information Technology


School of Science and Engineering

First Advisor

Dr. Maninder Singh

Second Advisor

Dr. Andrew A. Anda

Third Advisor

Dr. Aleksandar Tomovic

Creative Commons License

Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.

Keywords and Subject Headings

Object detection, Text Extraction, UML diagram, Faster-RCNN, Requirement Verification, Software Design


Unified Modeling Language (UML) class diagrams are widely used throughout software design lifecycle to model the Software Requirement Specifications in developing any software. In many cases these class diagrams are initially drawn, as well as subsequently revised using hand in a piece of paper, or a whiteboard. Although these hand-drawn class diagrams capture most of the specifications, they need a lot of revision and visual inspection by the software architects for verification of the captured requirements, of the system being modeled. Manually verifying the correctness and completeness of the class diagrams involves a lot of redundant work, and can raise issues due to human errors, for e.g., diagrams not drawn to scale, typeface issues, unclear handwriting, and even missing requirements etc. In this paper, we propose a state-of-the-art technique that pipelines the object detection, text extraction and replication phase to parse requirements incorporated within the user-drawn class diagrams. The parsed output from the proposed system, can be used to keyword-match the requirements in the Software Requirement Specification (SRS) document. We show that the proposed system can localize the UML classes and relationships in the diagram with 100% accuracy and can identify the localized objects of each type with maximum mean Average Precision of 0.8584. We also show that the text can be efficiently parsed from the diagrams with the character error rate of 0.3043.


I would like to acknowledge and give my sincerest thanks to my advisor Dr. Maninder Singh, whose expertise in the subject matter and unparalleled guidance made this work possible. I’d also like to thank the members of committee, Dr. Andrew A. Anda, and Dr. Aleksandar Tomovic for providing their invaluable suggestions and support throughout the course of this work. All the committee members have had tremendous positive impact in offering timely response, best practices for writing paper, recommendations and much more than what can be brought into light.

A debt of gratitude is also owed to the dean of Computer Science Department, Dr. Ramnath Sarnath, and all the faculty members of Computer Science Department at St. Cloud State University, who has contributed to my academic development and competency. St. Cloud State University, along with the Department of Graduate Studies undoubtedly deserves a recognition for providing me the platform to learn and succeed in my studies. A special thanks to Mr. Cliff Moran, for making the operational work such as scheduling meetings, registering classes etc. smoother, and most of all setting me on the track to graduate by providing all the information about the guidelines of the department, and the university.

It goes without saying, but above all I am grateful to my parents Dr. Ishwori Prasad Gautam and Mrs. Amrita Gautam, and all my family members for their everlasting support. I would like to thank my friends, for helping to me to collect the data and for being there for me.

uml_detection_extraction.ipynb (414 kB)
Implementation in Jyputer interactive notebook (148246 kB)
Training Dataset for Object Detection (1693 kB)



To view the content in your browser, please download Adobe Reader or, alternately,
you may Download the file to your hard drive.

NOTE: The latest versions of Adobe Reader do not support viewing PDF files within Firefox on Mac OS and if you are using a modern (Intel) Mac, there is no official plugin for viewing PDF files within the browser window.