the Evil Line

Jim Manzi says experiments (randomized field trials, or RFTs) in the social sciences are hard:

Criminologists at the University of Cambridge have done the yeoman’s work of cataloging all 122 known criminology RFTs with at least 100 test subjects executed between 1957 and 2004. By my count, about 20 percent of these demonstrated positive results—that is, a statistically significant reduction in crime for the test group versus the control group. That may sound reasonably encouraging at first. But only four of the programs that showed encouraging results in the initial RFT were then formally replicated by independent research groups. All failed to show consistent positive results.

The whole article is very interesting and thoughtful. I'm a little more optimistic than Manzi about the prospects of learning about society, though.