Epiwonk sent a polite email telling us of the posting, (linked on our list of blog sites) and then indicating an interest in comments. We have indicated to Epiwonk that we think the post makes a substantive contribution, is better done than some blog postings, and have thanked him/her for the interest in our work. Nonetheless, here are our thoughts, as requested.
There is one actual error in your posting.... and a related
point that may be leading several of your readers to be confused. It is written
on the posting in regards to why I contacted the editor:
"Dr. DeSoto’s specific concern related to the statistical interpretation of the data."
This is not accurate at all. My specific concern was that the means and standard deviations published by Ip and Wong in 2004 resulted in a t value that easily reached statistical significance. Please run the means and SD from the 2004 publication with reported sample sizes in any online t stat calculator and put yourself in my position (Ip et al 2004: blood level 19.53 + 5.7 vs 17.68 + 2.48 does not give a non significant t stat). This is most assuredly the error that was of concern and that I emailed the editor about. A statistical interpretation, something about which learned persons may disagree, would not result in an erratum (examples of this type of debatable points that would not result in an erratum would include one-tailed versus two tailed statistical significance, how to deal with outliers, whether to transform the data, whether to report an odds ratio, etc etc). On the other hand—straight up errors in a t-value calculation that lead to a completely different conclusion are clearly of a different magnitude. Again -- it was the unequivocal error in calculation that was astonishing and not a statistical interpretation of the data that led to the contact, the erratum, and eventually to the reanalysis from scratch of the data set Ip and Wong provided in 2007. We hope you will strive to make this clear to your readers.
A related point that appears to be confused by some is that the data set published in 2007 (the one that was provided by Wong in 2007 after the error came to light) is not a data set that results in the same means Ip and Wong published in 2004. This is a rather important point to be clear on. Without making this clear, someone who runs the numbers from the 2007 data and does not compare the means to Ip’s 2004 publication may not realize the error in 2004 goes way beyond most of the issues the blog posts I have read seem to focus on.
It might be of interest to you and/or your readers that another article authored Dr. Wong using a closely related data set has recently been recalled by a different journal, and she has resigned from an editorial board on which she served. I say this because some posts seem to have the mistaken impression that there was nothing seriously wrong with Ip and Wong’s 2004 report, that the error we brought ot light is essentially nitpicking over statistical interpretations, and that Wong has been above reproach. Maybe I am defensive. To be honest, I do grow weary of reading that my motives are suspect. Overall, we wonder if a lack of clarity on why the journal author was contacted, combined with confusion about the true extent of the original error by Ip and Wong contributes to some readers attributing false pernicious motives to us ... such as "One does not have to do an extensive evaluation of the DeSoto and Hitlan analysis to show that they did everything possible to show correlation between mercury and autism." Nothing could be further from the truth, actually. Again, the data set Ip and Wong provided in 2007 does not match up with their original stated means from 2004-- and the data set they provided still results in a difference between the two groups (unless you do odd things such as leave in clear outliers and / or falsely dichotomize a continuous variable.) Statistical experts have opinions on which statistics they prefer, and there is some variation in preference across fields of study. But readers should not lose sight of the forest for the trees, and I think your website, among other valuable things, serves to make it clear that the difference between autistic and control subjects shows up using a variety of statistical techniques, yes? In hindsight, we agree that publishing confidence intervals would have been a wise addition. On the other hand, we have provided this analysis when asked to do so by an interested party, we are not trying to hide or misrepresent anything. Most to the point -- one can do nothing more transparent than to provide the complete data set so that interested statisticians can do their own preferred analyses. This was done by the editor in the same issue and we referenced this in our article. Surely no one would advocate publishing the data set twice in the same issue? (?)
We would like to see the following point clarified: According to the abstract, DeSoto & Hitlan “found that the original p value was in error and that a significant relation does exist between the blood levels of mercury and diagnosis of an autism spectrum disorder.” As written on your website, this might be seen as being taken out of context. Please clarify that this conclusion was actually qualified – the full sentence makes clear we are speaking about within this data set. It does matter to us. We would not want it to appear to anyone that we are drawing conclusions beyond our data set. We think Ip et al were somewhat guilty of this in 2004, and were actually careful to avoid doing so ourselves. It may seem like a fine line, but to us it matters.
We have offered this reply because it was solicited by you
(the author EpiWonk who appears to add some meaningful contribution to the
debate about this issue). We are happy to reply to those who email us directly,
and in fact always do so. Having said this, we have stopped reading the postings
and did not read all of the postings on your website. Therefore, do not take a
failure to comment on specific comments within the blog postings as agreement.
Thank you for your efforts and we hope you will help your readers maintain an
accurate view of the related issues.
Epiwonk penned in a follow up post, DIET (SEAFOOD): This is the reason I put in the quote from Ip et al: “This study is limited by the sample size and culture because Hong Kong Chinese are famous for eating seafood.” I agree with all of you that this should be be taken into account in any discussion of these data. As a scientist, it upsets me that that the DeSoto & Hitlan paper made it almost straight from the pages of the journal in November 2007 to the mouth of Congressman Dan Burton in December as firm evidence that ethymercury in vaccines is causing autism in “our” kids.
Epiwonk-- We had no way of controlling for this, did we? Not asking about the seafood intake of the participants seafood question would be an oversight of Ip and Wong in their data collection. Please note the strong conclusions that Ip and Wong made in 2004 about this data set "not causally related". Do you not think they overstated their results back then? Why did no one call them on this if they had such obvious confounds? Does it make logical sense to attack our reanalysis for their potential failure to collect information on a variable that could confound the results? We have been clear that these results do not prove vaccines cause autism and would be happy to help anyone who has questions understand exactly what we think this data set means. Are we responsible for all the blogging and opinions based on our research? Much of it much farther astray than Burton. I think we have gone above and beyond to try and help the public understand what these results say and do not say.
Epiwonk wondered, "Did Catherine DeSoto make any attempt to contact
Congressman Burton or his staff to tell them that these were data from Hong Kong
children and correct any of the misinformation? I’ve dealt with Congressional
staff during my career; this an be done."
We have stated something similar in on line posts and in journal publication. I was invited by an attorney to testify/get involved in the vaccine court proceedings, but declined as I feel I do not have anything to add at this point (beyond removing the Ip and Wong as evidence against a vaccine link and/or mercury link). I respond to every query I get and would do so for congressional staff as well. Congressman Burton appears to be trying to understand the data set – and like many of the postings I have read by non-scientists – overstates the results to some extent. Again, I try to be as clear as humanly possible, and hope that you yourself will revise your point three to avoid giving readers on either side of the issue from the impression that we feel we have proven a relationship between mercury and autism with this one data set…. The data set shows a relationship, but any single data set should be taken as a single data set. Had this standard been applied to the 2004 publication, some in the scientific community may have realized that the possibility of a relationship should not have been entirely written off when only one single case control study of blood mercury levels existed (why is that by the way?) …. The irony is seeing those who have long argued for a causal link to vaccines try to turn the tables on Shattuck or Fombonne (examples of scientific experts who cited Ip and Wong as proof vaccines have been ruled out as a cause of autism) by saying that a statistically significant result serves as proof that vaccines have caused autism. The logic in both conclusions may be essentially faulty, but those who now cry foul over this approach but failed to do so when Fombonne or Shattuck used it, appear logically inconsistent. To be clear, the proclamation that Ip and Wong 2004’s results (as they were published) would have proven that vaccines do not cause autism was faulty. To say the 2007 reanalysis provides proof positive that a link exists outside this data set is faulty. But with the 2004 data having been fundamentally a whole series of errors, and a new analysis showing some relationship does exist in the data set (even Epiwonk's alternate analysis shows a statistically significant relationship)-- the weight of evidence should properly be seen as having shifted a bit. Come on EpiWonk – step up tell your readers you agree.
“The majority of recent studies have failed to establish a connection between measles-mumps-rubella vaccination or the use of mercury-based vaccine preservative and autism,” Shattuck 2006 with Ip 2004 as a cite.
“By and large, biological studies of ethylmercury exposure have also failed to support the thimerosal hypothesis,” (includes Ip 2004) and then goes on to chide those who continue to consider the possibility that vaccines could have any bearing on autism (Fombonne, 2006).