Causal Inference: Why is it so hard?

In 2013 I was working at a machine learning company called NICTA and I learned about an amazing field called causal inference.

The promise is huge, imagine if you could learn cause and effect relationships from purely data alone.

Amazingly smart people like @yudapearl wrote great books espousing how awesome it is.

Yet, I struggle to figure out how to actually formulate problems.

It's one thing to hear causal language, but its entirely another to actually formulate a causal inference problem.

The biggest thing is we often want to know the causal impact of what might have happened.

Say for example, how bad things could have been if we didn't apply a particular a safety control.

Why is it so hard to formulate outside of health? Is it the data is lacking? The signals are sparse? What is the big problem?

Perhaps the issues is that causal inference needs a big mindset shift from an AI practitioner calling fit just doesn't work anymore.

But if you need to model a system completely to use causal inference have you actually just built an expert system?

Would it have been better to just try and fully model the system instead of using all the fancy causal inference mathematics? It would be more understandable too.

Causal Inference: Why is it so hard?

Causal Inference: Why is it so hard? In 2013 I was working at a machine learning company

The all-in-one writing platform.

Write, publish everywhere, see what works, and become a better writer - all in one place.