Thoughts on Pagel et al. on ultra-conserved words

Here is what I posted on FB two weeks ago:

It seems I have read a different article than the Washington Post author, but also than the many linguists who reacted to Pagel et al.’s recent PNAS article on ultraconserved words (see for instance Geocurrents.info).

Here is what I found in the PNAS article by Pagel et al.

Background assumptions:

  • A: The cognacy relations across various Eurasiatic language families that the Tower of Babel project proposes are largely correct.
  • B: In previous work, the authors (or, rather, three of the four authors of the recent paper) found statistical support for a relation between the replacement rate of words in Indo-European languages and their usage frequency. According to the uniformitarian principle, this relation also held for language change in Eurasia 15 ky – 8 ky before present.

Findings:

  • C: There is strong statistical support showing that the distribution of Tower-of-Babel cognacy relations are pretty much as we would expect if A and B were true.

The main finding of the paper is C. [Aside: So criticism of the paper should actually attack C, not A or B.]

Now to the role of A and B. We are talking about a conditional statement of the form

A & B -> C

All Pagel et al. claim is that they have shown C to be true. Of course this does not prove A and B to be true. However, according to standard scientific practice, proving the consequent of a conditional increases the plausibility of the antecedent. (See for instance Jaynes 2003, Probability Theory: The Logic of Science, page 2:

“If A is true, then B is true
B is true
Therefore, A becomes more plausible”)

So Pagel et al. provide evidence making A and B more plausible. No matter how much prior plausibility you attached to A and B, as long as you did not consider them logical contradictions, after reading Pagel et al. you should assign them a somewhat higher posterior plausibility (which might still be low) if you are a rational person, unless you find a flaw in their statistical arguments.

Another way to put it is to see this as a Popperian piece of research. If C could be shown to be false, this would be a falsification (or at least a serious blow) to A, i.e. to the claims of the Tower of Babel people. Since falsification failed, the hypothesis is – of course – not proven, but it has received further empirical support.

This is certainly not as earth-shattering as the Washington Post claims, but it seems to me that it is rock-solid empirical research. It is ground-breaking in the sense that it proposes a novel way to derive empirically testable predictions from statements about language history, so it extends the methodological tool box of historical linguistics. Good work.

After some discussion, I later added:

An interesting point was brought up in the discussion on languagelog. It is conceivable that the ToB-people spent most effort in their hunt for deep cognates on stable, i.e. frequently used concepts. Therefore most false positive would be in this group, which would explain the bias that Pagel et al. found equally well.
To test this alternative hypothesis, we would have to run automatic cognate recognition algorithms (along the lines of Greg Kondrak’s or Mattis List’s work) on the raw data that ToB uses. Such an approach would also help to test some of the critical points that have been raised against Atkinson & Gray (2003) and Bouckaert et al. (2012).
Bottom line: Historical linguistics needs more applications of statistical and algorithmic techniques, not less.

After mulling over this a bit, I finally extracted the numbers from the supplementary information files of the two papers (which are, somewhat inconveniently, in PDF and Word, rather than in csv or some other machine readable format…). Here is what I found:

A little addendum: I tried to replicate the statistics in Pagel et al. 2013, and in their Nature paper from 2007, where they established the correlation between frequency of use and rate of change. Starting about the latter, they overstate their results a bit, but it still holds up. When they write that R = 0.7 in a linear regression model, what they should say is that R-squared = 0.48, ie. 48% of the variance in the data is explained by the predictors. This is still a lot, but 0.7 is of course a more impressive number than 0.48…
Also, these 48% are the reduction in variance if you predict the rate of change from frequency of use and from part of speech! Part of speech information already explains 45% of the variance, and frequency of use covers the remaining 3%. So the correlation is quite weak, but it is still significant (p = 0.01).

(Btw, this correlation only holds for content words. For function words, there is no significant correlation in either direction.)

About the recent paper, regarding the correlation between their estimated rate of change in Indo-European and the number of families that supposedly preserve a Proto-Eurasian word: part of speech explains 24% of the variance. The estimated rate of change accounts for another 13%, which strikes me as quite substantial.

The alleged correlation between number of families and frequency of use turns out to be non-significant (p=0.61) if you factor in part of speech.

Bottom line: they are overselling their results at various places, but their estimates for Indo-European rates of change are in fact good predictors for cognacy relations as proposed in the Tower of Babel data base. This holds even if we control for part of speech. This makes the possible circularity mentioned above much less likely, I think. I can easily imagine that Starostin and his disciples searched harder for possible cognates among pronouns than among nouns, say. However, the mentioned correlation also holds within the nouns, and within the verbs, and within the adjectives… and its quite a stretch to assume that this is due to some bias in the process of identifying/stipulating cognacy relations.

P.S. After some email exchange with Martin Haspelmath: The fact that there is no significant correlation between frequency and cognate class size once we control for part of speech shows, I think, that the patterns that we see in Tower of Babel cannot be explained as a frequency bias on the side of the researchers. It is conceivable though that they, being brought up in the tradition of Indo-European comparative linguistics, have a bias towards concepts that are stable within the Indo-European family.


Print This Post Print This Post
This entry was posted in Article comments. Bookmark the permalink.

One Response to Thoughts on Pagel et al. on ultra-conserved words

  1. Very constructive contribution. For people wondering about the “ToB” term, here’s a link to Starostin’s followers’ Tower of Babel website.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <embed style="" type="" id="" height="" width="" src="" object="" allowfullscreen="" allowscriptaccess="" cachebusting="" bgcolor="" quality="" flashvars=""> <iframe width="" height="" frameborder="" scrolling="" marginheight="" marginwidth="" src=""> <object style="" height="" width="" param="" embed=""> <param name="" value="">