Welcome to my Social Blog
Most ML interpretation methods have a common enemy:
Correlated features.
They ruin interpretation both on a technical and a philosophical level.
Why correlation is problematic, how to patch it, and why we have no cure.
A thread 🧵
Correlated features are the rule, not the exception.
Predicting bike rentals? Season and temperature are correlated.
Credit scoring? Income correlates with age, job, ...
Diagnosing patients? Blood values are correlated, like multiple markers of inflammation, ...
We'll use 3 points of view to understand the effect of correlation on interpretability:
Extrapolation
Entanglement
Latent variables
Note: correlation here includes more general dependencies, not only linear correlation.
Extrapolation
Most model-agnostic interpretation methods require independent features.
Partial dependence plots
Shapley values
Permutation feature importance
...
What happens when we ignore correlation and apply the interpretation methods anyways?
We extrapolate data to unlikely regions.
The method produces new data points that lie outside of the distribution, predicts them, and produces the interpretation with those points.
Extrapolation is bad.
The model wasn't trained with data from extrapolated regions: the quality of predictions might be bad or unknowable.
Data might even be undefined in this region: 100kg baby
Unrealistic data points become part of the interpretation
We have the means to address the correlation problem.
Analyze correlated features in groups, like grouped importance
Decorrelate features, e.g. with PCA
Remove correlated features
Use conditional interpretation methods
...
Unfortunately, none of these patches is perfect.
A grouped analysis is often too crude
Decorrelation destroys feature interpretability
Removing features worsens model performance
Conditional interpretation is difficult.
The conditional interpretation is worth looking deeper into. What is it?
Many interpretation methods have a "marginal" and a "conditional" version. The marginal versions pretend that features are independent.
Conditional interpretation methods manipulate data but keep the conditional distribution intact.
See conditional importance: Values are sampled based on the feature's distribution conditional on all other features.
Also: M-Plot (cond. PDP), TreeSHAP (“tree_path_dependent”), ...
Conditional interpretation fixes the extrapolation problem, but it has another problem: entanglement.
Conditioning changes the interpretation, because it entangles the interpretation of a feature with its correlated features.
Example: Permutation feature importance: The increase in loss when we permute the values of a feature.
Conditioning entangles the interpretation.
Conditional feature importance: The increase in loss when we sample a feature, given that we know the values of all other features.
Entanglement is bad because we usually want an isolated interpretation.
We want to understand the isolated effect of temperature on bike rentals.
We have a hard time interpreting some mashed-up effect of temperature, season, humidity, ...
Similar to PCA versus origin features.
You Can’t Have Your Cake And Eat It Too
If you avoid extrapolation by using conditional interpretation, you entangle your interpretation.
We shouldn't treat correlation as a technical issue. Instead, correlation indicates that features share information. They might be based on the same latent variable.
Temperature and humidity are physically "correlated". They both have as latent variables the weather conditions.
Isolated interpretation is desirable, but out of reach. We artificially create features in separate columns even though they might share a latent variable.
You can duplicate a feature, train, interpret in isolation. Clearly meaningless. For correlation it's similar.
To summarize:
Correlation worsens interpretation
Correlation can cause extrapolation
We have patches to address correlation
Conditional interpretation entangles interpretation
Isolated interpretation is desirable, but artificially constructed
For more content on interpretable machine learning and being a mindful modeler, subscribe to my newsletter:
https://mindfulmodeler.substack.com/
In my book Interpretable Machine Learning, correlated features is also a constant theme, and for each interpretation method I discuss this issue: