Renneder@sh.itjust.worksM to BecomeMe@sh.itjust.works · 1 year agoChatGPT gets code questions wrong 52% of the timewww.theregister.comexternal-linkmessage-square10fedilinkarrow-up143arrow-down14cross-posted to: technology@lemmy.ml
arrow-up139arrow-down1external-linkChatGPT gets code questions wrong 52% of the timewww.theregister.comRenneder@sh.itjust.worksM to BecomeMe@sh.itjust.works · 1 year agomessage-square10fedilinkcross-posted to: technology@lemmy.ml
minus-squareQfuiyh@lemm.eelinkfedilinkarrow-up3arrow-down1·1 year agoTitle feels misleading, it gets stack overflow questions wrong 52% of the time However it got 77% of easy Leetcode questions correct. Also I believe that’s first try, which is not generally how chatgpt should be used. Also also, you should probably be using a coding specific model if you want good coding results
minus-squarenanoUFO@sh.itjust.workslinkfedilinkarrow-up5·1 year agoEvery leetcode question has been answered a billion times and you train it on those billions of answers, it should get those right.
minus-squareOskarAxolotl@lemmy.worldlinkfedilinkarrow-up3·1 year agoProbably because the model has seen thousands of possible solutions to those exact Leetcode problems. Actual questions people ask on StackOverflow tend to be much more specialized.
minus-squareHackerJoe@sh.itjust.workslinkfedilinkarrow-up2·1 year agoBut it confidently explains the wrong answers. I just hope politicians don’t find out how to use it. It’ll be our doom.
Title feels misleading, it gets stack overflow questions wrong 52% of the time
However it got 77% of easy Leetcode questions correct. Also I believe that’s first try, which is not generally how chatgpt should be used.
Also also, you should probably be using a coding specific model if you want good coding results
Every leetcode question has been answered a billion times and you train it on those billions of answers, it should get those right.
Probably because the model has seen thousands of possible solutions to those exact Leetcode problems. Actual questions people ask on StackOverflow tend to be much more specialized.
But it confidently explains the wrong answers.
I just hope politicians don’t find out how to use it. It’ll be our doom.