ChatGPT vs Github Copilot

And the winner is...

ChatGPT is not only good for summarizing some text or explaining how to divide integer numbers:

It’s also capable of writing code. In C# or Java or Typescript or Rust and many other languages. That’s pretty awesome.

But how good is it compared to other services, even more tailored to programmers Like Github Copilot.

Copilots promise sounds very interesting for always overloaded programmers: Get help from your good AI friend at Github right in your IDE. That would be so cool, wouldn’t it? The $10/month would be a tiny price to pay.

But does it work as advertised? And is it as good as ChatGPT?

I took Copilot for a short test drive. The scenario was simple: a spiced-up version of the notorious Hello-World challenge.

Copilot

I’m using Jetbrains Rider as my C# IDE. Installing Copilot as a plugin was a no-brainer after registering for the trial on the Github site.

You then ask your AI buddy for help either through “meaningful identifiers” of functions or classes or describe the problem to solve in a comment.

Getting the requirements right, i.e. clear and comprehensive etc., is kinda of an art, I quickly realized. But I’m sure these aren’t too far off:

You as a human would be able to come up with code doing that right away, I’m sure. But does Copilot?

Unfortunately not. I at least couldn’t get it to respond in a way I expected. Maybe the requirements where too complicated?

What about this then?

It did not work, either. No suggestions for code were made.

Maybe using comments wasn’t the best way to trigger my AI buddy’s intelligence? Maybe an “intention revealing” function name would help?

Yes, indeed, there is a response from Copilot. It’s not matching C# as my language… but at least I can see some good will. The suggestion is matching the problem at least.

But can’t Copilot do better? I really want more than just a “Hello, world!” message. I’d like the code to react “to people coming back” with different greeting messages.

After some trial and error I at least was able to get this from Copilot. But I had to first put in the function signature and then add the comments above:

The result, though, is not very convincing. Just a bunch of if-statements with all made up names? Seriously? No, I’m not willing to spend any money on this kind of help. Sorry, AI buddy.

(Ok, le me admit, maybe I haven’t tried hard enough. Maybe I’ve overlooked some way to motivate Copilot to engage more with my requirements. Maybe it can do better. But then… if I couldn’t get a better result for such a trivial problem right off the bat while being very interested… there is something missing. The UX then is less than perfect. And to be honest, whenever I got a result it was delivered with a noticeable lag. Hence I’m sticking with my judgement: not a tool I would bet on right now.)

ChatGPT

Is Copilot the paragon of AI support for coding? How does ChatGPT compare? Can it at least keep up with Copilot?

Well, the answer to me is very obvious. See for yourself: I prompted ChatGPT with the original requirements on which Copilot choked.

This is so much better than Copilot!

What I find especially remarkable is that ChatGPT realized it needed a class to wrap around the function to keep the state. And it did not make a fuzz about object orientation but stayed with a static class. Because it does the job just well.

To actually be able to use the code I just needed to add a final return or throw an exception.

While writing the code to exercise ChatGPT’s solution, to my surprise Copilot tried to help me. See the grey line with the lightbulb? That’s Copilot suggestion for the empty line I inserted:

Cute, isn’t it? Thank’s buddy… but too late. This kind of code completion on steroids I don’t find that enticing, I must say.

Presentation of Prizes

It’s not hard for a jury to decide who should be the winner in this contest.

The winner is… ChatGPT

Hands down ChatGPT convinced me to stick with it (for now). It’s clearly the more helpful AI. Even though I need to copy&paste its code into the IDE. That’s a small price to pay for its more thought through solutions.

And not only that: ChatGPT is a real buddy I can talk to. We can have a dialog about the code. There is no conversation with Copilot, though. But that’s a story for another article.