Is it possible to build a C-3PO "protocol" droid with a ChatGPT like system?

The venerable star wars droid, with its stiff walking style and awkward interactions seems like a good fit for the strangely human, yet not quite human, interactions we have with modern chat bots. But to actually interact in the real world, our droid will need more than just good manners. It will need to understand the world and what it can do in it.

There are many examples of humanoid robots that appear to fit the bill. The most well known is Boston Dynamics Atlas robot, a large, tethered humanoid that is often seen in YouTube videos performing backflips or jumping through obstacle courses. Tesla's Optimus humanoid robot has also been seen moving things and waving to the audience. Agility Robotics has a bipedal robot for moving parcels in warehouses. But many of these robots are operating in highly controlled, pre-mapped spaces.

If we want a C-3PO robot that can land on a moon and evade capture from an evil empire, we will need a robot that can operate anywhere.

A Picture Says A Thousand Words

The amazing thing about ChatBots is that they can be asked anything, and will usually return a reasonable sounding result.

This is the holy grail for an AI system, as it shows a large capability to deal with the unpredictability of the real world. The problem is that text is not always a good way to describe something, sometimes a picture is much better. Fortunately researchers have solved this problem too with an AI technique called CLIP. If you've ever asked an AI to make a picture for you, then it's probably using the CLIP system in some way.

The amazing thing about CLIP is that it can work in reverse, instead of making a picture from text, you can make a text description of a picture. Sometimes this is also called an "embedding", which is a precursor to text creation and can be more easily used by AI systems. With these image embeddings you can now compare voice commands such as "find Obi-Wan Kenobi" with images from what your robot sees.

Using CLIP, a robot can understand what is in any picture coming from its camera.

Translating Into The Language Of Movement

Even if a robot can understand text and sight, it needs to be able to move.

An interesting property of ChatBots and the Large Language Models that they are built upon is that they can translate between different languages such as Japanese and English. Recent work by Google, named RT-1, asked "what if movement was just another language?", and this had a surprising result.

The RT-1 system was able to translate an instruction like "pickup rice chips from top drawer onto counter" into the necessary robot movements to perform that task. The system was focused on pick and place tasks in a kitchen style environment, but was able to achieve up to 67% accuracy, a remarkable result.

Putting It All Together - OK Robot

In January 2024 an exciting new robot demo spread through the internet.

The team based out of New York University showcased a robot that could do a quick scan of any bedroom and then accept open vocabulary commands. The "Ok Robot" system was able to start anywhere in a bedroom and complete tasks such as "move the soda can to the box" with up to 82% accuracy.

The best thing about this system is that it was an engineering solution, combining existing AI systems and Robotics algorithms rather than training things from scratch.

Modern Droids

So is it possible to build a C-3PO with modern tools?

In the movies, C-3PO and R2-D2 act like fully conscious robot humans, and are able to think and plan. Our modern robots aren't that sophisticated, but demonstrations like OK Robot take us closer and closer to having household droids that can understand us and do what we ask.

Can ChatGPT evolve into a Star Wars droid?

Is it possible to build a C-3PO "protocol" droid with a ChatGPT like system? The venerable star wars

The all-in-one writing platform.

Write, publish everywhere, see what works, and become a better writer - all in one place.