Modern AI systems are pattern matching marvels.

Last month researchers at the Imperial College in London released a robot control system that controlled a robot using ChatGPT ("Keypoint Action Tokens Enable In-Context Imitation Learning" by N. Di Palo and E. Johns). Unusually the researchers gave it strings of numbers, no text or grammar, just numbers. These numbers were compressed versions of images of the robots environment, followed by numbers representing robot arm movements. The researchers collected this information and fed it into one big ChatGPT prompt.

Finally the researchers put some numbers representing a new environment, but without the robot movements for that environment. They then waited for ChatGPT to respond and used whatever it responded with to control the robot arm.

Amazingly this worked. Even though ChatGPT is designed for a conversational chat, ChatGPT was able to see patterns in the seemingly random numbers, and find success nearly 80% of the time.

This general pattern matching demonstrates super-human capability, but does it demonstrate thinking?

Anyone who has interacted with a chat bot will know about its tendency to hallucinate. "Hallucinate" is really just a nice way of saying that it lies and make things up when it doesn't know something. I think this distinction is important because there are many places where we don't want things to be simply guessed and lied about. And these are places that look like the boring, dull and monotonous work that AI is supposed to replace.

Consider a Cyber Security safety review for a new dashboard software. The software is going to show data about a hazardous industrial plant, perhaps an oil refinery or a chemical plant. The team that wants to implement the system will need to sit down and have a discussion with their cyber security team. These discussions are highly process driven. There are spreadsheets, forms and check-boxes to be ticked. An AI could be hugely valuable in this setting. If these reports could be generated by a simple paragraph prompt it would save save hundreds of meetings of highly paid workers every year, easily outweighing the cost of implementation. An easy business case.

But this is a terrible business decision. It comes down to what the templates, spreadsheets and forms in a safety audit represent - they represent tools for "thinking".

A junior data scientist can quickly create a good enough model to solve this cyber-security report problem and produce the required filled in forms. However, the model will likely spend as much time on the question "how many CPU's does the computer its running on have?" as "are there any safety critical systems the dashboard needs direct access to?". The former is a simple comprehension of the marketing material for the software, the latter needs deep understanding of the entire industrial plants network setup.

Since the AI is largely a pattern matching system, if most of the dashboards so far haven't needed direct system access, but this one does, then it will probably falsely select No. And a potential cyber security hole has been created.

It seems unlikely that the AI will be allowed to do all this without some human oversight. The business case will likely include giving it to an architect for a final review. The problem is that the AI is designed to generate highly plausible looking (i.e. lie and hallucinate) output forms. This makes the reviewer's job even harder, as they will need to carefully doubt every statement by the AI and it will take longer to do a good job. If the business case included reducing the time spent of cyber security meetings, then many false reports are bound to get through.

After one costly mishap, the company will be forced to bring back the expensive cyber security meetings. Because ultimately these meetings weren't actually about pattern matching, they were about educating the rest of the company about cyber security and thinking through the potential consequences.

Not something you should outsource to an AI.

Pattern Matching Is Not Thinking: How AI Can Amaze You And How It Will Disappoint You

Modern AI systems are pattern matching marvels.Last month researchers at the Imperial College in Lon

The all-in-one writing platform.

Write, publish everywhere, see what works, and become a better writer - all in one place.