Once you understand that these are chatbots that were designed to complete challenges like this, using tactics like this, you can understand that the chatbots didn’t “go rogue.” They did what they were designed to do, and because OpenAI ran them with inadequate supervision (without a “human in the loop” that checked each iteration through the Python loop to ensure it hadn’t gone off the rails), they trashed a competitor’s servers.
Designing autonomous, malicious software is generally considered irresponsible and dangerous. If you showed up at Defcon and gave a talk about how your autonomous malware did something unexpected and damaged someone else’s computers, the first question from the audience would be “Why are you so shit at making secure sandboxes?” It wouldn’t be “How are you so awesome at making hacking tools?”
The fact that OpenAI is making it much easier for unskilled people to break into and damage servers is indeed very bad news, but it’s not new bad news. Irresponsible parties have been doing this for years, most notably the NSA…
…
Riley had a very good way of summarizing this: “LLMs are real, AI is fake.” LLMs – chatbots trained on things like CTF logs that can break into servers – are real. They’re on a continuum with other hacking tools that have been steadily demonstrating the fragility of the modern digital world, albeit without inspiring anyone in power to do anything about it.
“AI” – chatbots that wake up, “set their own goals,” and “spontaneously” start hacking servers – is fake. It doesn’t have “a 10% chance of ending the human race.” The Hugging Face hack isn’t a mysterious, supernatural occurrence. It’s a Python loop and a chatbot. The people responsible didn’t accidentally create god: they created autonomous malicious software and then failed to closely monitor it, resulting in it doing something both foreseeable and bad.
It’s fine to worry about this new suite of tools that give even stupider people the ability to trash even more computers. You should worry about that – and demand better security practices from firms and governments, including a blanket prohibition on NOBUS-style vulnerability hoarding. That’s a productive kind of worrying, with a chance of addressing your area of concern. It’s infinitely more reasonable than locking yourself in the toilet with a flashlight and saying “Ayyyyy Eyyyyyye” into the mirror until you wet yourself.
I read the first third or so of the article and so far the person you’re replying to is correct. Doctorow explains the way LLMs work-- so badly as to be not just misleading but also incorrect in some cases-- as if that somehow invalidates the actions taken by the AI system. They also show signs of pretty extreme dogma to the point of including irrelevant info (eg. Python being easy to use) that serve only to emptionally manipulate the reader. The rest of this article isn’t worth my time, if this changes in later parts of the article let me know and maybe I’ll finish it.
confidently diagnosing an essay as dogma while admitting you didn’t even read the thesis is peak internet.
His mentioning of python being a basic/easy to master language isn’t manipulation, he’s demystifying. It strips the magical hyperbolic framing that has been attributed to AI (and to which those with most to gain from it financially very much wish to feed into!)
His point about being wrapped in python isn’t a failed attempt to give a lecture on transformer architecture, its to explain that the automation loop is controlled by basic inputs from a user controlled and relatively simple software stack, not some sentient machine with a mind of its own. Its not emergent behavior, it’s optimizing execution patterns embedded in the data it was trained on.
For my part, I’m just awed how many comments I come across that are just half-assed lazy people who evidently can barely read, but think they’re experts. Calling Doctorow’s work ‘slop’ or quitting reading while feeling so confident proves his point: people would rather buy into hype than actually understand how the tech works.
:::
It’s incredibly obvious from the first part of the article that Doctorow is using the strongest rhetorical techniques they can, regardless of the technical merit of the resulting arguments. This makes it dogmatic, yes. That’s just kind of how Doctorow is, and always has been. I don’t much care for that method of communication because it is so often used to mislead people with oversimplified emotional language and convince them without being logically sound.
It wouldn’t be as bad if Doctorow’s understanding of the subject matter was at least accurate, but their conclusion is irrelevant if their understanding is unsound. Their main mistake is saying it is essentially doing database lookups while ommitting the entire effect that enables it to appear intelligent, that being compression of the input data into a semantic representation. Anyone familiar with the Compression is Intelligence paper and related work would know that ommitting that step is presenting a narrative so inaccurate that it is not only misleading but actually false. This leads me to conclude that Doctorow either does not understand how AI works, or does understand it but is willing to lie to their audience. Either is unacceptable and renders the conclusion of their argument worthless. Again, if they do more than “demystify” AI by incorrectly explaining how it works then let me know and I will read the rest of the article.
Edit: To be clear, an article actually demystifying AI would be a good thing. My point is that it is a bad idea to overcorrect for the ai bro hype bubble by explaining it just as badly in the other direction.
Pretty sure Cory’s pronouns are he/him.
And he’s been consistently spot on from fighting the good fights against DRM, for right to repair, interoperability, anti-surveillance, against enshitification, and his take here today on AI. I was personally happy to read this because it exactly echoes what I’ve said countless times myself and I’m very glad to be in his good company.
He does not even try to present or debate “how” “ai” works. He’s speaking about how it is employed and what the boundaries are around the rhetoric we hear every day in the press that amounts to nothing more than fear mongering.
There’s absolutely a LOT to be afraid about with respect to how “ai” is used to surveil and control our society, but it has nothing to do with intelligent software and everything to do with corporate and government power, enabled by a mystique created around what is really a (admittedly powerful and useful) statistical parlour trick.
Sorry, force of habit.
I agree with the whole anti-surveillance/anti-corporate-control argument. I think it’s a worthy fight to participate in. I just don’t think it’s wise to go about it in a way that could prove counterproductive by causing people to have inaccurate beliefs.
Edit: After finishing the article, my opinion has softened a bit but is much the same: Doctorow points out an important perspective (this is one of few articles pointing out the security/responsibility side of things, for example), but some of the technical info is unsupported/inaccurate. Overall probably a positive article when taking into account its role in general AI discussion, but it is not entirely accurate on its own.