y = b1x+b0. If x is 1D, this linear model has 2 parameters. In contrast, recent Deep Neural Network(DNN) model 'GPT-3' powering ChatGPT has 175 Billion parameters. 'GPT-4' of ChatGPT Plus has 1 Trillion. Beyond the explosion in number of parameters, DNNs learn complex non-linear functions and are said to be non-interpretable in how they learn to map input(x) to output(y).

But what does interpretability in DNNs mean? (this linked article attempts definitions). For this post, we consider a model as interpretable if we can perfectly predict the output, and can infer how input data changes affect predicted output. Is the model's decision path hard to find or is it that the theoretical framework and design choices in training the models do not easily account for interpretability?

Below I describe 3 aspects of the underlying learning theory for a large section of DNNs (excluding Bayesian Deep learning, etc) that bake in non-interpretability.

Illustration: Imagine we train a DNN to predict the agricultural crop yield of land plots. The model takes as input the raw sensor data on water levels and weather of the land.

• Deep learning does not learn statistical relationships between dependent target variable (y) and input variables (x). DNNs do not rely on explicit input features, but learn their own 'latent' features and representations of input.
  Illustration: The crop yield prediction model does not learn explicit relations linking water levels to crop yield, but hunts for its own patterns in the sensor data (and learns its own input features) that give best performance in predicting the yield.

• Deep learning does not explicitly account for relationships between input variables.

  Illustration: The DNN does not model how weather (e.g. excess temperature) might affect water levels. It thus can not predict how much the yield would change as a function of interlinked changes between inputs. It only predicts on observed data.

• Most importantly, Deep learning does not assume probabilistic processes behind the data generation of either input or output.

  Illustration: Input variables weather and water levels are collected across different days and have distributions. High temperatures and low water levels might require a different crop yield prediction function, while a period with moderate rainfall follows a different yield trajectory. The likeliness or unlikeliness of the draw of a certain input event (tsunami?) should change the crop yield function. Our model does not consider the data to be generated or related in a probabilistic manner.