Regression and Classification Machine Learning Model Management Software Tool for Energy Toolbase

Our design addresses ETB’s issues of model management and comparison using a two-pronged approach. First, a model library for storing ETB’s models was created with an interface to integrate it with existing software systems created by ETB’s data science team. Second, an interactive web application for managing models was created, allowing ETB’s data scientists to easily upload, search, edit and delete models. Additionally, collections of graphs can be created and saved, simultaneously comparing the performance of several different models. Furthermore, several machine learning models were developed to demonstrate the functionality of the system and investigate different types of models.

Our design is innovative in that it added two entirely new software tools to ETB’s toolkit for model comparison and storage. The first tool acts as an API into a database which tracks models. This provides the majority of the model management functionality. The second tool is a UI which allows users to easily interact with the database and to compare models.

Our design is also innovative in the fact it approached (with the counsel of ETB’s data science team) energy usage as a function of weather as a classification problem rather than a regression problem. Regression machine learning models try to output a value such as 5, meanwhile classification machine learning models try to find a bucket in which this value would fit, such as between 3 and 7. The vast majority of literature approaches this problem as a regression problem as well. The classification method would be used to show the confidence about a prediction, allowing ETB to choose safer energy strategies over optimal strategies.

Our solution is effective as it addresses the issues faced by ETB’s data science team.

It standardizes and centralizes models that ETB creates. It allows members of ETB’s data science team to upload models into a shared environment with versioning and additional metadata to keep track of models. As well as the ability to search for these models and then download them for use.

The solution also allows for quick comparison of models by creating shareable dashboards that any user can see. This comparison tool allows ETB to make decisions about the based on various performance metrics. Such as whether a model is better than another or even whether certain classes of models perform better than another class of models.

Our team validated our solution using a multi-faceted approach.

• Our team created our own models, which helped with the classification model investigation, as well as using them to validate various parts of the system. Pushing these models through the system quickly unearthed any issues our system had, and helped prove our system worked with real data.
• Our team also created 200 unit tests to ensure functionality and code quality remains constant through project iterations.
• Our team enlisted the help of ETB’s data science team to use our system and provide feedback for the functionality and ease of use of the system.
• We also measured key performance metrics that were outlined in the design of the project. These measured values were compared to these outlined values to determine the validity of our solution.

Our key focus in this project was to attain a usable system for ETB. This focus was accomplished through careful planning and prioritizing of the different aspects of the project. We used a JIRA board to manage and prioritize our work amongst the many project components. We consulted with our academic advisor and sponsor representatives to discuss the possibility of certain features. We constantly triaged the bugs and features and included ETB on discussions for directions in the project. For example the classification model investigation was of low priority compared to the model registry and comparison features.