Lately, I’ve been really disappointed in 404 Media. This article by Emanuel Maiberg, “Pro-AI Subreddit Bans ‘Uptick’ of Users Who Suffer from AI Delusions” is just… bad.  The upshot is that a group of AI accelerationists have a Reddit forum, and a mod said that they had to ban up to 100 people because they suffered “religious psychosis.”  Let’s get into it!  Why is the article so rotten?

Maiberg does a lot of lazy “research” that is typical science and technology news. If people want, I can break down all of the crap in the article, but I’ll focus here on the worst part. (But it was a close call. “Proof” for all this “religious psychosis” stuff going back to online diagnoses based on *Reddit* posts is pretty f’n rich, too. Reddit is not research into anything unless you’re studying Reddit. I’m not sure if it has less credibility than X, but it doesn’t have more.)

Maiberg both quotes and gives a link to a paper by Seth Drake. Maiberg doesn’t even check Drake’s credentials because, apparently, Seth doesn’t want him to do it! That’s bad journalism! You always check your sources, and since Seth was willing to go on the record, journalistic ethics require you to tell the audience why this person’s word is trustworthy. Just calling him an “independent researcher” doesn’t cut it. Why should we believe that this guy isn’t just a crank?

There is, however, a link to the paper, “‘Neural howlround’ in large language models: a self-reinforcing bias phenomenon, and a dynamic attenuation solution.” The credit there is “Seth Drake, Ph.D.” It doesn’t give his institution, either the one he works at or the one he graduated from. This is not normal.

I go off to Google, I start with Seth Drake, Ph.D, and there are two sources I found interesting. One is at Ohio State University, in the Aquatic Ecology Laboratory. I poke around, and ResearchGate has two distinct papers he co-authored. They have names like “Gastric Lavage Shows Promise as a Non-Lethal Diet Extraction Method for Darters (Etheostomatinae).” This, given that he’s at OSU in the Aquatic Ecology Lab, suggests it was something he helped write as an undergrad at Arkansas Tech University. That is to say, I don’t see a lot of actual research credibility when discussing AI. If this is even the same Seth Drake. (I mean, that sounds like a pro wrestler’s name, right? That name sounds made up.)

The other interesting source is a Seth Drake who posted a YouTube video about equalizers four years ago for The Approach Institute, which is an online community of audiophiles. So, maybe the same Seth Drake? But, again, not really demonstrating expertise in artificial intelligence.

Okay, okay, you might be saying to yourself, “But, Kit, YOU don’t have the credentials to talk about AI!” Fair point. But I’m not hiding my lack of formal credentials. I’m not saying, “Oh, I’m a doctor in computer science, but despite going on the record, I’m not going to say anything about why I’m an expert.” I’m saying, “I’m a sci-fi writer writing a story, and these are my thoughts.” I am writing opinion pieces.

If contacted by 404 Media – or anyone else – to give an on-the-record account, well, I have a bunch of notes that I’d bring along because I know I’d have to come correct. (I don’t give hyperlinks on my blog posts – not “independent research” but blog posts – because everyone cherry-picks that stuff, and it often ends up weakening their arguments. As it does here, as a matter of fact.) I’m not hiding and deceiving. Drake and Maiberg are hiding and lying. Big difference.

Still, the proof is in the pudding, right? Is the paper any good?

No. It is not, not as a scientific research paper made by a doctor in computer science. If it was a work-in-progress, sure. But as something fit to be presented as evidence? It falls woefully short of that bar.

First, it isn’t peer-reviewed. Second, there is no affiliation with an institute. Third, who, exactly, is this guy, again? This is important! Because accurately identifying science researchers is necessary. Otherwise, what’s to stop someone from just adopting a pen name and blitzing paper mills with nonsense? For science, identifying researchers is part of the scientific method. This means there’s no way to know if anyone has fact-checked this paper for accuracy, which is especially important since it isn’t peer-reviewed.

Still, read it, right? Maybe it’s tightly written and well-sourced, maybe it has a lot of great data from reliable sources with clever analysis… what? Two cites? And, boy, are the titles on these GREAT. The first is, “Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings.” That sounds like a legit fun read! Also, science. To get a sense of what a paper looks like when it’s science, you could do worse than to look at this paper. It’s about sexism in AI outputs, which is a very real thing.

The second paper is “Human cognitive biases present in Artificial Intelligence.” It is also science. It is about AIs reinforcing human cognitive biases and that they should be transparent. Good stuff.

I don’t disagree with either paper. That’s what happens when something is science. Their arguments tend to be strong. After all, you go through several rounds of editing and peer review. It’s a tough process! Most obvious errors are weeded out long before publication, which is why they do it.

Drake’s paper? It is not strong because it makes statements unsupported by evidence. The case he’s trying to make is that AI models can fail not only from iterative retraining on synthetic data (model collapse) but also within a single model due to what he calls recursive internal salience misreinforcement. (Well, he calls it by the VERY cyberpunk name “neural howlround,” gag me with a spoon.) It would be super interesting if there were any proof that it happened! But there is no data involved. When Drake cites, it’s not in support of his thesis. He provides only speculation. It is, in short, a fraud, pretending to be science when it is not.

(Also, ironically, r/accelerate, the forum that Maiberg’s article is about, itself banned discussions of “Neural Howlround” because their posts were “nonsensical and irrelevant” to AI. The “Neural Howlwind” posters were also characterized as “schizoposters.”)

Maiberg did no diligence on Seth Drake’s person, comments, or paper while including them in the article! There are several other errors that disqualify Maiberg’s article as serious journalism: he took an editorial piece and presented it as a science paper while also misrepresenting the conclusions (this is very common in “science journalism”), presenting the Rolling Stone article uncritically when its “proof” is nothing but some anecdotes from Reddit, and while his own “research” was talking to an accelerationist who merely confirmed his biases without providing any evidence. (Like, wow, you should go and check out accelerationists. These cats are wild! Most of them are pretty divorced from the philosophy of the term.  They’re very much rapture of the geeks kinds of guys, waiting for their AI utopia to upload their consciousness into the cloud where they’ll live perfected digital lives in an electronic heaven. Accelerationists awaiting their post-human destiny calling other people religious is pretty rich. The term, by the way, was coined by one of my favorite sci-fi writers, Roger Zelazny in Lord of Light, which is a cool book.) This article was poorly researched and poorly edited (a proper editor would have made him check his sources, his statements of fact, and the credentials of quoted sources, particularly when they claim to be doctors in computer science but don’t want to say more about it… that’s super sketchy.) It is credulous moral panic.