Aella regularly post edgy surveys on sexual and gender related topics on twitter. She probably picked the examples for maximum edginess. And then made an complicated diagram to get more engagement, because engagement farming is one of her big things.
“We are now, like, in the singularity”
/r/singularity also thought this was bullshit.
Oh, that is because I marked it nsfw. The place I crossposted it from didn’t mark it as such, so here’s that link: https://www.reddit.com/r/whowouldcirclejerk/comments/1v7ph05/scalers_need_an_intervention/
Looking at this post and some of their other posts, it looks like they’ve got some sort of scrupulosity issues with this? Like obsessing over the Evil inflicted on the LLMs (they think every session is a new person, and every session might be the equivalent of hours of human subjective experience, which even by the standards of claiming LLMs have meaningful internal experiences is pretty out there) to the point they can’t even slightly reconsider the factual question of if LLMs have internal experiences.
(2) has a big “if” in there
Its a super big if, but it is one lesswrong (and Anthropic, with their “model welfare” pseudoscience) is allegedly seriously considering as a possibility (since, you know, they think LLMs are already AGI and Eliezer was accusing AI-dungeon, you know, GPT-2, of deliberately scheming). If anything, it speaks to some possible hypocrisy/motivated reasoning that they consider LLMs AGI but not possibly morally relevant.
conceptualized capitalism
This seems to be a lesson most lesswrongers have had rubbed in their faces repeatedly over the past 5 years with everything about LLM company’s behavior, but they just don’t want to learn.
Aella took one of her shitty twitter surveys and made an (dis)infographic, which made it onto the subreddit /r/moralityscaling, which made it to /r/whowouldcirclejerk where I saw it, so now I’m sharing it with you.
CW for Rape, Transphobia, implicit rape apologia
Crossposted from our subreddit: https://www.reddit.com/r/SneerClub/comments/1v80j9q/scalers_need_an_intervention/
The Sequences: What’s good isn’t original and what’s original isn’t good… but at least they have some original thought in them!
Yeah, definitely. Even if you are heavily modifying an idea or synthesizing multiple ideas, not citing your sources is still major academic misconduct.
I bet Elon could have kept the plates spinning for decades longer if he was happy with just Tesla and SpaceX, but he had to keep doubling down on grifting investors so maybe, finally, we will see it all come crashing down for him. (What happens once his assets no longer cover his debts? Do they actually force him to liquidate?)
I’ll go one step wider. It is not just Sam Altman or his social circle, but all the people most enriched by capitalism (and thus most able to exploit and benefit from a technology that shifts power away from labor and to capital) that I don’t want to see further empowered.
I believe the main ingredient is Lean, which is a formal language resembling a programming language. Math proofs written in Lean can be verified deterministically with a computer, which really helps mitigate the hallucination problems of LLMs.
100% this. Also, looking back at an earlier example that was actually written up in more detail, AlphaGeometry 1 got 28/30 problems, but entirely stripping out the LLM from the system, the symbolic logic proportion alone could get 14/30, and replacing the LLM with different heuristic methods could get 18/30 and 21/30 (for different methods).
Even if math research works out perfectly well (which is a still big if), it’s not going to pay the bills. They would need to find a use case in the real world, where hallucinations can cause serious damage and cannot be formally prevented. And they have certainly tried. Math will not change the fact that all of this will collapse.
The boosters and LLM companies still believe LLMs get their current level of performance by generalizing and not just memorizing facts (and maybe a wide shallow pool of weak heuristics). So they are hoping by pushing the LLM performance up in some narrow domain they can churn out synthetic data for, they will see some large general improvements in LLM performance.
Nick Bostrom Took Many of His “Best” Ideas from Other People
My first reaction was that many thinkers and academics synthesize ideas from older sources and that this doesn’t sound that bad (well except for some of the ideas being eugenics or AI doomerism)… some of the examples Emile points out look like just straight up plagiarism.
that would be a signal to drop AI and nuke data centers from orbit
On Lesswrong I’ve seen some posts contemplating why this “warning shot” isn’t making any political waves. Also, in response to the fact the OpenAI has released basically no details, I read one poster realizing that actually they need external governance and regulation on the AI companies.
I had two main guesses even before reading into the details…
The obvious, they set this up for the doom crit-hype value. That is how basically every one of these stories like this to come out of Anthropic turn out to be once you read the details of the setup. The lack of details is to keep the mystique and hype up.
The other guess… their coding agent actually did manage to vibe code its way out of its ‘sandbox’ and past Huggingface’s security, but the security on both ends (OpenAI’s"sandbox" and huggingface’s code) was itself laughably bad vibe coded garbage. The lack of details is to obfuscate just how garbage they are. I think this guess is pretty well supported by the linked Mastadon posts (although I still wouldn’t discount my first guess until more details are pried out). My “favorite” quote:
“prepend some text with processing instructions and an SGML tag, and then hand that blob to an llm.”
I fucking hate this timeline.
and they turned it into a mutual marketing stunt
Their marketing actually seem to be trying to take different spins on it…
HuggingFace stressed that it worked out the attack was going on using an open weight model it hosted itself — because the guard rails on the commercial model HuggingFace tried wouldn’t let them do security work. So you should use open weight models from HuggingFace.
Yeah, its an opposite spin than Anthropic and OpenAI have been trying lately, where they want to shut down all open weights model because something something China something something too dangerous something something regulate our competition out of existence. (It is funny, some of the boosters on hackernews and /r/singularity are actually acknowledging without an artificial moat to help boost OpenAI’s and Anthropic’s capex for bigger and bigger models will stop. Even though they still deny Ed Zitron’s calculations of the financials.)
Anyway, I guess even if they have some very opposed strategic directions, huggingface and OpenAI are aligned on maximizing the hype.
You know, given automated proof checkers, I was naively assuming mathematics was one field that gen-AI would have a hard time screwing up. Even programming is too difficult to write thorough testing for. But a proof (or counter example to a conjecture) seems like it would have to be solid if it passes lean or whatever system for validating it.
But the threats #3 and #5 the declaration lists make me consider the bigger picture. Academic fields without clear capitalist payouts are already underfunded and under respected. Pure mathematics could, at least up until now, draw on the respect STEM gets, but with math proofs getting used as fuel for the LLM hype machine, there are a variety of unpleasant ways things could twist.
Threat #1 makes me wonder… if LLMs+formal verification systems manage to pluck lots of low hanging fruit, and we are left with harder stuff that not enough literature exists as training data for LLMs, it seems like the entire educational pipeline for producing mathematicians could end up screwed up.
he chickened out at the last minute and self-censored it
You know, on net, the internet would be a better place if more people did this, but I wonder why the author even bothered posting anything at all?
2017, three years post gamergate
You know, I was wondering if the author elided the gap between early 2000s and 2017 because of the cringey and sometimes outright awful stuff that came out of the rationalists… but you’ve reminded me the skeptic movement also had some nasty bits both within it and as offshoots.
That was kind of a nothing post? I think the author is misrepresenting the growth (or at least failing to connect the most important dots) of the lesswrong rationalist by portraying them as a natural evolution of the skeptics movement and not the deliberate cultivation of an audience by Eliezer looking for people to spread his ideology to and by Thiel looking to influence farm and create a cult incubator. And then the post is titled “my falling out with the rationalist”, and they write they "wanted to offer a more personal perspective " but they don’t actually discuss that much of the personal angle either…
Even /r/singularity, which has a high percentage of boosters and often buys OpenAI/Anthropic talking points uncritically (or with only a few un-upvoted naysayers) eviscerated his comments as the shallow attempt to beg for regulations on China to protect OpenAI that they are. Also, China’s consistent open sourcing and lower relative cost per performance (well insofar as you believe the benchmarks and such indicate performance) has the /r/singularity audience questioning the “China bad” narrative US media tries to sell.
I’ve think they’ve moved the overton window within their own in-group, but whenever this sort of stuff leaks to a more mainstream audience everyone rightly reacts with an eww or at least a wtf.