Statistical Analysis of HDD Failure¶

Matthew Unrue, Spring 2020¶

Western Govenors University MSDA Capstone¶

Website Version Note:¶

This notebook is so large and works with so much that data that it was run in multiple settings with the kernel reset for memory management each time. As such, the code cell blocks have execution numbers that are not perfectly in order. Though these do not match up perfectly in this version, the code was and should only be excecuted from top to bottom.

Additional Resources:¶

The 5-page project Executive Summary can be found here.
The reveal.js based multimedia presentation notes can be found here.
The 87-page report write-up of this project can be found here.

Table of Contents:¶

• I: Dataset Preparation
• II: Data Tidying
• III: Nan Value Management
• IV: Analysis of Potential Predictors
• V: Dataset Preparation
• VI: PCA and MCA
• VII: Model Creation
• VIII: Conclusions

Introduction¶

What factors indicate impending hard disk drive failure?¶

H0: Study factors do not significantly indicate impending hard disk failure.¶

H1: Study factors do significantly indicate impending hard disk failure.¶

Context¶

Data helps businesses solve problems, make better decisions, and understand consumers, but a lot of data needs to be stored and available to enable these benefits. Hard drive failure is the most common form of data loss, which is one of the most impactful problems that businesses can experience today as simple drive recovery can cost up to $7,500 per drive (Painchaud, 2018). For cloud-based data centers, keeping multitudes of businesses’ data intact for their own operations is crucial. Being able to predict which hard drives are at the highest risk of failure based on understanding of the combinations of routine diagnostics test results is an ideal solution to backup and replace failing drives before the data is lost.

Data¶

The dataset used is Backblaze’s 4th quarter data from 2019 (Backblaze, 2020). All of the needed data is contained within the .zip file that Backblaze provides to the public as .csv files split by day.

The dataset contains .csv files for each day of its corresponding quarter, from 2019-10-01 to 2019-12-31. As an example, the subsection of the dataset for 2019-10-01 contains 115,259 rows of data. However, as this data contains recorded readings from a live data center, the number of hard drives and thus rows, changed daily as failed drives were taken out and new drives were installed. The 129 column attributes are Date, Serial Number, Model Number, Capacity, Failure, 62 Self-Monitoring, Analysis and Reporting Technology (SMART) test results, and 62 normalized values of the SMART test values. The Failure attribute is the dependent variable of this study and is a qualitative binary categorical variable. The Date, Serial Number, and Model are nominal qualitative independent variables. Finally, and the SMART value columns are continuous quantitative independent variables.

As stated in Backblaze’s Hard Drive Data and Stats page (Backblaze, n.d.), this dataset is free for any use as long as Backblaze is cited as the data source, that users accept that they are solely responsible for how the data is used, and that the data cannot be sold to anybody as it publicly available.

Data Analytics Tools and Techniques¶

Python, pandas, and the scikit-learn stack are extensively used for the loading, tidying, manipulation, and analysis of the datasets. PyTorch is used for all neural network related tasks of the analysis and model production. Matplotlib and seaborn are used to create charts and graphics for analysis and presentation of project findings. A needed algorithm, namely Fisher's Exact test for contingency tables greater than 2x2 dimensions, is unavailable in the scikit-learn ecosystem, and R.stats is used for this by using rpy2 to run the R code by embedding it in the Python process. Prince is used for factor analysis, and imbalanced-learn is used for the implementation of SMOTE.

Like R and unlike SAS, all of these packages are easily available, free, and open-source with Python. These methods have been chosen over R for ease of explanation, as Python code is often understood more readily than R, and because of the potential of integrating this project directly into a program or software for future use. While R is highly specialized for statistics and mathematics, Python is a general-purpose programming language with specialized libraries for the needed tools, and this facilitates project expansion in the future.

Synthetic Minority Over-Sampling Technique (SMOTE) is used specifically to handle the imbalanced classes for training and testing splits. PCA is used for dimensionality reduction. Predictor variables are examined through correlation coefficients and Fisher's exact test, as well as graphed univariate and bivariate distributions. A logistic regression model and a decision tree model are examined along with the results of the PCA to find predictor variables as well. For building a predictive model for future use, the logistic regression model, a random forest ensemble model, and neural networks are compared to determine which can produce the most useful model.

As HDD failure is an extremely rare event, the dependent variable class is extremely imbalanced and failing to control for the imbalance through techniques like boosting or oversampling would lead to ineffective models. As the dependent variable is a Boolean value, this task is a binary classification task. Logistic regression is an ideal predictive model for binary classification tasks that gives a probability for classification while also having a simplistic interpretation of coefficients that can be used for feature selection. Decision trees are also simple to understand and work well for classification tasks. Given the complexity of the various fields in the dataset, a more complicated model may work better for predictive power. Random forests and neural networks work very well for classification tasks under these circumstances.

Project Outcomes¶

The key project outcomes are a deep understanding of the risk of hard drive failure based on the results of SMART test values regardless of manufacturer and predictive models that will be able to flag hard drives that are at high risk of failing. The understanding of the risk of failure based on test values will empower better business decisions by optimizing the choice of storage used based on projected lifetime. The predictive models will allow the business to proactively backup data from storage onto new storage devices before failure while also allowing hard drives to continue working closer to their end of life, minimizing waste from constantly replacing hard drives before it is needed. The combination of these two products will also enable the future creation of a more automated system that protects data from hard drive failure.

Dataset Preparation¶

The dataset provided by Backblaze is made up of 92 .csv files, 1 for each day in the 2019 4th quarter, totaling 3.13GB of text data. As hard drive failure is an extremely rare event, all of these days will need to be considered together in order to have enough failures to draw conclusions. The project begins by combining all parts of the dataset from their .csv files into a single file.

Out of 10,991,209 hard drive days, there were only 678 failures, which gives a failure rate of 0.006169%.

Weiss (2013) defined the imbalance ratio as the ratio between majority and minority classes with a modestly imbalanced dataset having an imbalance ratio of 10:1, and extremely imbalanced datasets as having an imbalance ratio of 1000:1 or greater (pg. 15). This dataset has an imbalance ratio of approximately 16,210:1 and as such will require very careful cultivation in order for any predictive model to successfully learn from. The rarity of the positive failure cases is also the reason that the entire 4th quarter dataset is required.

Unfortunately, this combined file requires too much memory to load all at once for current hardware restraints. It needs 13.5GB for just the data, not including the memory needed for the OS and other software, nor memory for calculations.

As this dataset contains both raw and normalized values for all of the SMART values, a simple way to deal with the memory issues is to divide the dataset into a raw form and a normalized form.

Data Tidying¶

The considerably smaller raw value subset of data is the main dataset of this project. As with nearly all real-world datasets, this one needs considerable cleaning and tidying in order to use for analysis.

All SMART test columns have null values in some rows. The dataset notes state that this comes from differing manufacturer's standards despite the standardized nature of SMART tests.

Deriving the manufacturer from the model column will allow the dataset to be easily divided by manufacturer.

The "DELLBOSS VD" model value seems the be the only value potentially out of place.

None of the SMART values exist for this hard drive model, but 60 of the drives have this model value. Additionally, no failures for this model exist in the dataset. Any row with this model value should be removed from the training data before any predictive analysis. Some searching online leads to the belief that it may be a RAID controller. (https://www.dell.com/support/manuals/au/en/aubsd1/boss-s-1/boss_s1_ug_publication/overview?guid=guid-b20ef25b-b7e3-40f2-b7cd-e497358cd10a&lang=en-us)

Additionally the "Seagate SSD" model seems to be missing information. Like the "DELLBOSS VD" model rows, this one also does not have any failures and will need to be removed before predictive analysis is performed.

The rows not appropriate for analysis are deleted.

Dtype Conversion¶

Given the size of the dataset, a few minor changes to the columns may free up a considerable amount of memory. The date and capacity_bytes columns are two easy places to improve.

Here we can see that 1108 drive days have an error value rather than their actual capacity. These rows may need to be removed, but may also be an excellent signal for a failing drive.

Unfortunately, all drives experiencing this error do not fail and this can introduce problems in the final model. As it only affects 0.01% of the dataset, removing the affected rows seems best.

The capacity_bytes column is converted from bytes to terabytes to condense the information on disk.

Raw Data Univariate Distributions¶

With these things finished, the univariate distributions can be examined to gain a better sense of the data.

The first column, date shows some sort of testing or operational failure on November 5th.

Drive capacities are mostly 4, 8, and 12 TB, likely coinciding with large investments in new drives for the datacenter and possibly alongside the price lowering of specific models.

The manufacturer of the most drives in this dataset is Seagate at 72.59%. HGST is the second highest at 24.24%. Western Digital is the least represented manufacturer in the dataset with only 0.23%, but as HGST was acquired by Western Digital in 2012 (Sanders, 2018), the drives in this dataset will likely be quite similar between the two manufacturers given the seven-year timespan between then and the time of dataset recording and creation. Finally, Toshiba is the other manufacturer, with 2.94% of the dataset. This amount is quite low and may make it difficult to accurately predict their drives in comparison.

The SMART values vary greatly from the number of different types of drives that exist in this dataset. Before the columns can be graphed appropriately, the NaN/null values need to be examined. It's most likely that the missing data is most related to the hard drive's manufacturer or model.

Every single SMART figure column has null values.