Every Cause Wants To Be A Cult

Author: Eliezer Yudkowsky. Link to original: http://lesswrong.com/lw/lv/every_cause_wants_to_be_a_cult/ (English).
Tags: lesswrong, Rationality Submitted by Elspet 11.09.2013. Public material.

Translations of this material:

into Russian: Любая причина хочет быть культом. Translation complete.
Submitted for translation by Remlin 12.09.2013 Published 3 years, 5 months ago.


Cade Metz at The Register recently alleged that a secret mailing list of Wikipedia's top administrators has become obsessed with banning all critics and possible critics of Wikipedia. Including banning a productive user when one administrator—solely because of the productivity—became convinced that the user was a spy sent by Wikipedia Review. And that the top people at Wikipedia closed ranks to defend their own. (I have not investigated these allegations myself, as yet. Hat tip to Eugen Leitl.)

Is there some deep moral flaw in seeking to systematize the world's knowledge, which would lead pursuers of that Cause into madness? Perhaps only people with innately totalitarian tendencies would try to become the world's authority on everything—

Correspondence bias alert! (Correspondence bias: making inferences about someone's unique disposition from behavior that can be entirely explained by the situation in which it occurs. When we see someone else kick a vending machine, we think they are "an angry person", but when we kick the vending machine, it's because the bus was late, the train was early and the machine ate our money.) If the allegations about Wikipedia are true, they're explained by ordinary human nature, not by extraordinary human nature.

The ingroup-outgroup dichotomy is part of ordinary human nature. So are happy death spirals and spirals of hate. A Noble Cause doesn't need a deep hidden flaw for its adherents to form a cultish in-group. It is sufficient that the adherents be human. Everything else follows naturally, decay by default, like food spoiling in a refrigerator after the electricity goes off.

In the same sense that every thermal differential wants to equalize itself, and every computer program wants to become a collection of ad-hoc patches, every Cause wants to be a cult. It's a high-entropy state into which the system trends, an attractor in human psychology. It may have nothing to do with whether the Cause is truly Noble. You might think that a Good Cause would rub off its goodness on every aspect of the people associated with it—that the Cause's followers would also be less susceptible to status games, ingroup-outgroup bias, affective spirals, leader-gods. But believing one true idea won't switch off the halo effect. A noble cause won't make its adherents something other than human. There are plenty of bad ideas that can do plenty of damage—but that's not necessarily what's going on.

Every group of people with an unusual goal—good, bad, or silly—will trend toward the cult attractor unless they make a constant effort to resist it. You can keep your house cooler than the outdoors, but you have to run the air conditioner constantly, and as soon as you turn off the electricity—give up the fight against entropy—things will go back to "normal".

On one notable occasion there was a group that went semicultish whose rallying cry was "Rationality! Reason! Objective reality!" (More on this in future posts.) Labeling the Great Idea "rationality" won't protect you any more than putting up a sign over your house that says "Cold!" You still have to run the air conditioner—expend the required energy per unit time to reverse the natural slide into cultishness. Worshipping rationality won't make you sane any more than worshipping gravity enables you to fly. You can't talk to thermodynamics and you can't pray to probability theory. You can use it, but not join it as an in-group.

Cultishness is quantitative, not qualitative. The question is not "Cultish, yes or no?" but "How much cultishness and where?" Even in Science, which is the archetypal Genuinely Truly Noble Cause, we can readily point to the current frontiers of the war against cult-entropy, where the current battle line creeps forward and back. Are journals more likely to accept articles with a well-known authorial byline, or from an unknown author from a well-known institution, compared to an unknown author from an unknown institution? How much belief is due to authority and how much is from the experiment? Which journals are using blinded reviewers, and how effective is blinded reviewing?

I cite this example, rather than the standard vague accusations of "Scientists aren't open to new ideas", because it shows a battle line—a place where human psychology is being actively driven back, where accumulated cult-entropy is being pumped out. (Of course this requires emitting some waste heat.)

This post is not a catalog of techniques for actively pumping against cultishness. Some such techniques I have said before, and some I will say later. Today I just want to point out that the worthiness of the Cause does not mean you can spend any less effort in resisting the cult attractor. And that if you can point to current battle lines, it does not mean you confess your Noble Cause unworthy. You might think that if the question were "Cultish, yes or no?" that you were obliged to answer "No", or else betray your beloved Cause. But that is like thinking that you should divide engines into "perfectly efficient" and "inefficient", instead of measuring waste.

Contrariwise, if you believe that it was the Inherent Impurity of those Foolish Other Causes that made them go wrong, if you laugh at the folly of "cult victims", if you think that cults are led and populated by mutants, then you will not expend the necessary effort to pump against entropy—to resist being human.