Statistical Analysis of Machine Learning Methods for Knee Osteoarthritis
Live Citation Count
DOI: 10.26685/urncst.564
For this project, I co-authored a peer-reviewed paper investigating how machine learning models use biomechanical gait data to diagnose knee osteoarthritis and predict how the condition will likely progress. Knee osteoarthritis is a disease that damages the cartilage in the entire knee joint, which will slowly affect how a person walks. We can measure these changes by looking at a patient’s walking speed, stride length, and joint angles and use these as inputs to ML models. Our review focused on comparing camera-based gait analysis with wearable sensor-based gait analysis. We also evaluated how well different ML algorithms performed by looking at various existing studies. My contributions focused on the analytical part of this project. I helped develop the study design and conducted statistical tests, as well as drafted sections of the manuscript.
Below is a PRISMA flow diagram of our study screening process. In the end, we had 22 studies where 16 used camera-based methods and 6 used wearable sensors.
One of the main challenges I encountered was that most of the papers had different methods and metrics of reporting their results, so they were difficult to compare quantitatively. Only 11 out of 22 of these studies reported their results using accuracy values that could be compared, so these were the studies I used in my final analysis. In general, camera based models had a mean accuracy of 92.05% while sensor-based models had a mean accuracy of 67.96%. Out of the algorithms used, Support Vector Machines had the highest mean accuracy of 90.28% while Random Forests and Decision Trees had a mean of 78.95%. However, I couldn’t just compare these averages since they didn’t tell us whether they’re actually statistically meaningful. My main technical contribution was conducting statistical tests to figure out whether these comparisons were actually statistically significant.
Statistical Analysis
Since the dataset was quite small and the group sizes were uneven, I used non parametric tests which don’t rely on the data to follow a bell shaped distribution and instead compare how the values rank relative to each other. For both of the tests I conducted, I used a significance level of 0.05, where I would reject the null hypothesis if p < 0.05.
To compare the five machine learning algorithms used in the included studies, I used a Kruskal-Wallis test. This kind of test is useful when comparing more than two independent groups, and it's especially useful in this case because it's less sensitive to irregularities and can handle group sizes being different. See below for more details on how this test works:
The null (H0) and alternative (H1) hypotheses were:
H0: The five machine learning algorithm groups have similar accuracy distributions.
H1: At least one algorithm group is different.
The test found that H = 2.200 and p = 0.699 > 0.05, and thus the null hypothesis was not rejected. This means we did not have enough statistical evidence to say that a certain algorithm would make a difference in accuracy.
To compare camera and sensor based methods, I used a Mann-Whitney U test. This test basically combines values from both groups, ranks them and then checks whether one group is consistently higher in the ranking or not. It is suitable for comparing two groups with small samples, which is why I chose it. See below for more details on how this test works:
The null (H0) and alternative (H1) hypotheses were:
H0: Camera and sensor accuracies have similar distributions.
H1: Their accuracy distributions are different.
The test found that U = 90.0 and p = 0.0032 < 0.05, and thus the null hypothesis was rejected. This means there was a statistically significant difference in terms of what approach the study used, and camera-based approaches had higher model accuracy compared to sensor-based studies. This makes sense since cameras can usually capture more spatial and biomechanical data while wearable sensors are generally more specific. Of course, traditional camera systems are often more expensive than wearable sensors so our results don’t mean cameras are always the best method.