Monday, February 11, 2013

Six Big Problems With "Why Can Some Kids Handle Pressure ..."

Surely how kids handle pressure is an important and interesting question. And surely how we perform in pressure situations has a lot to do with our genes. But the recent New York Times article "Why Can Some Kids Handle Pressure While Others Fall Apart?" by Po Bronson and Ashley Merryman is shot through with the most basic mistakes in science writing about behavior genetics. This makes me sad, because I have liked the authors' previous books, and because I think it is quite possible to communicate research on genetics accurately for an intelligent general audience. Here, unfortunately, they appear to have taken no note of what has happened in behavior genetics in the past 5–10 years, which ought to have been a prerequisite for this piece. A few examples:

Exaggerated claims: "One particular gene, referred to as the COMT gene, could to a large degree explain why one child is more prone to be a worrier, while another may be unflappable" [emphasis added]. In reality, what kind of COMT gene you have, if it is relevant, is an extremely minor influence by itself on how much you worry. The particular variant of the COMT gene being discussed here is very common, and like all other common genetic variants, it has never been shown to have a large, or even medium-sized, influence on any behavioral traits.


Cherrypicking the study with the most dramatic results: "Other research has found that those with the slow-acting enzymes have higher IQs, on average. One study of Beijing schoolchildren calculated the advantage to be 10 IQ points." In 2013 it should be regarded as journalistic malpractice to write things like this when the average of all the studies on this gene and IQ show the effect to be, at best, a tiny fraction of 10 IQ points. In an analysis that included almost 10,000 subjects from two countries, in fact, a team of colleagues and myself found virtually no evidence of any effect of COMT on IQ.


Idealizing your favorite study: "In other words, the exam was a perfect, real world experiment for studying the effects of genetics on high-stakes competition." In reality, there are no "perfect" experiments, and the one Bronson and Merryman report on had only 779 subjects, which might seem like a lot, but is almost certainly too small to learn anything reliable about genetic effects. About 100 times more participants are needed to really answer these questions.


Labeling genes with behaviors and pretending that possessing a genetic variant makes you a particular type of lucky or unlucky person: The two variants of the COMT gene are labelled "warrior" and "worrier" (for the different responses to stress they supposedly cause people to have—get it??), and then people are in turn labelled as Warriors or Worriers based on their genotypes. That's tantamount to calling the variants of APOE the "Doofus" and "Genius" genes because one makes you more likely to develop Alzheimer's disease while the other offers some protection against dementia. No, wait, it's not, because APOE has a highly significant effect on Alzheimer risk that has been replicated over and over by independent researchers, but COMT's links to the behaviors discussed in this article are smaller and more tenuous. Later we are told that the Worriers' "genetically blessed working memory and attention advantage kicked in. And their experience meant they didn't melt under the pressure of their genetic curse." I thought we gave up on this kind of superficial genes-as-personality-types-and-blessings-or-curses kind of science writing years ago.


Contradicting your own point: "... we are all Warriors or Worriers ... In truth, because we all get one COMT gene from our father and one from our mother, about half of all people inherit one of each gene variation, so they have a mix of the enzymes and are somewhere in between the Warriors and the Worriers." (Is anyone else reminded of the camp 1970s film "The Warriors," about gangs that roam the New York City subways?) We can't all be one type or the other if half of us are both. And incidentally, the pattern of 25%-50%-25% of the three genotypes does not arise only because we get one allele from each parent. It also depends on the frequency of the two variants being about 50% each in the population, which it happens to be in the case of this COMT polymorphism.


Pretending that what has been known for generations is a new discovery: "Stress turns out to be far more complicated than we've assumed ... short-term stress can actually help people perform ..." And later: "It may be difficult to believe ... that stress can benefit your performance." But psychology textbooks have long taught that the level of arousal for optimal performance is moderate, with too much arousal or too little leading to lower performance. This is called the Yerkes-Dodson Law, and it was originally proposed in 1908. Perhaps worth a mention?


The article makes much of findings that "those with Worrier-genes can still handle incredible stress." This would only be surprising if COMT had such a strong effect that it could determine what kind of person you are. But COMT doesn't have that effect. It's surprising when someone with the genotype for brown eyes has blue eyes instead, because the relevant genes almost completely determine the phenotype. It's not surprising that people with one of hundreds or thousands of genes that make one susceptible to stress turn out to be able to handle themselves just fine.


If the authors were conversant with—and showed concern for—the relevant literature and the background science, they would not have made these mistakes. I understand that they are writers, not researchers, but people who write about research for the public have a simple obligation to communicate not just good stories, but reliable facts.

5 comments:

  1. I am aware that you feel that journals should stop publishing candidate gene research, but I think a more appropriate perspective is that research findings do not deserve trust without replication. I know GWAS are expensive, but they should not be the only approach to genetics research.

    "...like all other common genetic variants, it has never been shown to have a large, or even medium-sized, influence on any behavioral traits."

    Well, never is a strong word. Caspi et al in 2002 found that the 12% of men with MAOA-3R accounted for 44% of those violent convictions. MAOA-2R's interaction with grade retention increased violent delinquency 21 times as much as retention alone.

    "In 2013 it should be regarded as journalistic malpractice to write things like this when the average of all the studies on this gene and IQ show the effect to be, at best, a tiny fraction of 10 IQ points."

    Did you actually read the study in question? I could be wrong, but I believe it is referring to Qian et al. It was not a study of COMT's effect on IQ. That was a study of MAOA's effect on IQ in boys with ADHD when the COMT ValVal allele is held constant. So, that study found a 8.7 IQ point advantage, and the average of the one study that studied this is 8.7 IQ points.

    If genes that influence neurotransmitters are not important, why should we consider the drugs that influence neurotransmitters important? Because they are.

    ReplyDelete
    Replies
    1. Thanks very much for your thoughtful comment. Let me say a couple of things in response to clarify what I was trying to say.

      First, regarding whether there have ever been any variants "shown" to have big effects. Instead of "shown" I probably should have said "shown consistently across multiple studies with large samples" or something like that. I didn't mean to say that there weren't studies out there that seem convincing by themselves—there are many, of course. Without very large samples, however, it is really hard to tell whether the results of published candidate gene studies are reasonable estimates of the true population effects, since publication bias and p-hacking could be at work in determining what results get into the literature. I acknowledge that for repeats, insertions, deletions, and other non-SNP polymorphisms this will be a much higher hurdle than for SNPs, since it is more expensive to assay repeats and you can't do a million of them at a time. I also think that non-SNPs have the potential to have larger effects than SNPs. But I still think the totality of the evidence should make us very skeptical whenever a researcher or writer says that one INDIVIDUAL COMMON genetic variant (e.g. not a rare mutation like the Huntington's gene, or a haplotype like APOE), whether SNP, repeat, or other, has a large effect on a behavioral trait. A verified result that it did would be of great interest, of course. At the very minimum, I think Bronson and Merryman should have addressed these issues in their writing, rather than ignore them. Even for a general audience, it's not right to pretend there is no complexity or uncertainty behind the headline findings.

      Second, I went back and re-read the portion of Bronson and Merryman's article about the 10-IQ-point study. It seemed pretty clear to me from the context of the surrounding paragraphs that they were talking about a study of COMT. I don't see MAOA mentioned anywhere in the article, which seems to be entirely about COMT. So if they are actually referring to a study of MAOA, then they should have made that clear. I can see how they might have taken an MAOA study to be about fast- and slow-acting enzymes, and I see that they referred to enzymes, not genes, when mentioning the 10-IQ-point study. But let's assume the Qian et al. study is what they were referring to. I have not read it recently, but I would maintain on statistical grounds that a study of 779 participants is even less likely to give an accurate estimate of a gene-gene interaction than it is of a main effect. I doubt it will replicate with anything close to this effect size in multiple independent large samples. Bronson and Merryman should realize this and not latch onto it as a way to make their story seem stronger than it is. Finding an isolated gene-gene interaction with a large effect size in boys with ADHD (and calling 8.7 points "10," for which there is no excuse) is not how good science writers should be explaining the research.

      Finally, I actually think that genes that influence neurotransmitter systems are very important! Your point about drugs is one very good reason why. But we shouldn't get confused by the large effects of drugs on behavior into thinking that genetic variants will necessarily have the same effect sizes, or even close. The effect sizes of genetic variants (COMMON genetic variants, it's important to note) are at issue here, not whether the proteins those genes produce are related to the behaviors (they are). But the fact that COMT is involved in dopamine metabolism does not entail variants of COMT making us into "Warriors" or "Worriers," as the article implies. We need a more sophisticated approach to the science from our best writers and publications.

      Delete
  2. First, you say “…candidate gene studies of complex traits were commonplace in medical genetics research. Such studies are now rarely published in leading journals…. In our view, excitement over the value of behavioral and molecular genetic studies in the social sciences should be tempered—as it has been in the medical sciences…” Now, you call for “multiple studies with large samples.” Should America invest heavily in candidate-gene studies, or should journals rarely publish them? Popular science writing has egregious inaccuracy with both genetics hype and dismissive critique, but I seem to be the only one pointing out the latter. Critique benefits from the popular pseudoscientific belief in tabula rasa, which succeeded in cancelling academic conferences and funding in the 90s. I am in the process of quantifying the lack of a decline effect in MAOA research. The relative reliability of the findings hasn’t prevented inexplicable wild mood swings from some scientists. (“[T]he evidence for an association between the [MAOA] VNTR variant and antisocial behaviour is substantially more consistent than most of these associations. This may well be one of the rare cases of genuine associations.” – Daniel MacArthur, 2009 “[M]ost if not all of this [MAOA] literature is wrong, and will soon be forgotten.” – Daniel MacArthur, 2011) Critics also seem to be fond of satire that suggests that an individual “molecule” cannot influence “complex human behavior.” This belief seems to apply to neurotransmitters but not epigenetic moieties.

    I find it to be an odd loophole for “a rare mutation” to be able to have “a large effect on a behavioral trait.” In the case of the MAOA promoter, abundance and influence are on a continuum. Brunner syndrome is both extremely rare in humans and nearly deterministic. We can still study it in mice very easily. MAOA-3R, the original “warrior gene,” is the most common MAOA VNTR allele worldwide, and it falls far short of being deterministic, but it has been well studied in humans. MAOA-2R falls in between with preliminary evidence suggesting a powerful main effect, and it has the academic advantage of not being rare in African Americans, (and, therefore, probably quite common in Africans). The bottleneck for investigating MAOA-2R seems to be a lack of “excitement,” as you say, among potential researchers. I have never met anyone who accepts that high dosages of a psychotropic drug are profoundly helpful but who viscerally opposes the notion that somewhat lower dosages have any effect whatsoever (excluding homeopathic dosage), so I am unable to comprehend why genetic promoter dosing should be different. The fact that many of these candidate genes share the same neurotransmitter targets allow for multiple-gene index studies, like Beaver et al, 2010 and Beaver and Chaviano, 2011, that quantify high levels of predicted probability with increasing numbers of risk alleles. Even the relatively small number of GWAS-identified risk alleles for the complex phenotype of BMI have useful predictive value, per Belsky et al, 2012.

    ReplyDelete
  3. The fact that I would be more convinced of candidate gene findings if there were replicated by multiple independent groups in large samples does not mean that I am also advocating that such research be funded by any government or institution. I do think, however, that funds would be better spent on checking the existing candidate gene associations than on looking for more of them. The field rests on a shaky foundation that should be shored up before more building is undertaken. Lots of people pursue COMT and other SNPs, and I think resources are being misdirected if they are basing their studies on previous false positive findings.

    That said, I don't have anything to say about MAOA research in particular, since I have not studied that literature. You might be right -- this gene could turn out to be a success story for candidate gene approaches. I don't think that whether it is a success or not is related to whether Bronson & Merryman have characterized the science of COMT correctly.

    Finally, the fact that we can construct predictive indices is interesting, but again I am concerned about publication bias, p-hacking, and replication. There is so much in this field -- including my own findings (see Benjamin et al., 2012, Annual Review of Economics) -- that does not replicate despite how promising and sensible it seems. It is very easy to fool ourselves about "discoveries" that I think journalists should be cautious before passing them along to the public.

    Thanks again for your comments. I really do appreciate the interaction!

    ReplyDelete