A test that doesn’t include a real commercial trial or A/B test with real human customers means nothing. Put their game in the App Store and tell us how it performs. We don’t care that it shat out code that compiled successfully. Did it produce something real and usable or just gibberish that passed 86% of its own internal unit tests, which were also gibberish?
According to the paper, 80%+ of the code was “executable”… they don’t mention if it does anything useful. I know if I went to an AI restaurant, and 80% of the food was “edible” I would not be that happy. For now, programmers jobs are safe.
A test that doesn’t include a real commercial trial or A/B test with real human customers means nothing. Put their game in the App Store and tell us how it performs. We don’t care that it shat out code that compiled successfully. Did it produce something real and usable or just gibberish that passed 86% of its own internal unit tests, which were also gibberish?
According to the paper, 80%+ of the code was “executable”… they don’t mention if it does anything useful. I know if I went to an AI restaurant, and 80% of the food was “edible” I would not be that happy. For now, programmers jobs are safe.