Epiwonk sent a polite email telling us of the posting, (linked on our list of blog sites) and then indicating an interest in comments. We have indicated to Epiwonk that we think the post makes a substantive contribution, is better done than some blog postings, and have thanked him/her for the interest in our work. Nonetheless, here are our thoughts, as requested.
-------------
There is one actual error in your posting.... and a related
point that may be leading several of your readers to be confused. It is written
on the posting in regards to why I contacted the editor:
"Dr. DeSoto’s specific concern related to the statistical interpretation of
the data."
This is not accurate at all. My specific concern was that the means and
standard deviations published by Ip and Wong in 2004 resulted in a t value that
easily reached statistical significance. Please run the means and SD from the
2004 publication with reported sample sizes in any online t stat calculator and
put yourself in my position (Ip et al 2004: blood level 19.53 + 5.7 vs
17.68 + 2.48 does not give a non significant t stat). This is most
assuredly the error that was of concern and that I emailed the editor about. A
statistical interpretation, something about which learned persons may disagree,
would not result in an erratum (examples of this type of debatable points that
would not result in an erratum would include one-tailed versus two tailed
statistical significance, how to deal with outliers, whether to transform the
data, whether to report an odds ratio, etc etc). On the other hand—straight up
errors in a t-value calculation that lead to a completely different conclusion
are clearly of a different magnitude. Again -- it was the unequivocal error in
calculation that was astonishing and not a statistical interpretation of the
data that led to the contact, the erratum, and eventually to the reanalysis from
scratch of the data set Ip and Wong provided in 2007. We hope you will strive to
make this clear to your readers.
A related point that appears to be confused by some is that the data set
published in 2007 (the one that was provided by Wong in 2007 after the error
came to light) is not a data set that results in the same means Ip and
Wong published in 2004. This is a rather important point to be clear on. Without
making this clear, someone who runs the numbers from the 2007 data and does not
compare the means to Ip’s 2004 publication may not realize the error in 2004
goes way beyond most of the issues the blog posts I have read seem to focus on.
It might be of interest to you and/or your readers that another article authored
Dr. Wong using a closely related data set has recently been recalled by a
different journal, and she has resigned from an editorial board on which she
served. I say this because some posts seem to have the mistaken impression that
there was nothing seriously wrong with Ip and Wong’s 2004 report, that the error
we brought ot light is essentially nitpicking over statistical interpretations,
and that Wong has been above reproach. Maybe I am defensive. To be honest, I do
grow weary of reading that my motives are suspect. Overall, we wonder if a lack
of clarity on why the journal author was contacted, combined with confusion
about the true extent of the original error by Ip and Wong contributes to some
readers attributing false pernicious motives to us ... such as "One does not
have to do an extensive evaluation of the DeSoto and Hitlan analysis to show
that they did everything possible to show correlation between mercury and
autism." Nothing could be further from the truth, actually. Again, the data
set Ip and Wong provided in 2007 does not match up with their original stated
means from 2004-- and the data set they provided still results in a difference
between the two groups (unless you do odd things such as leave in clear outliers
and / or falsely dichotomize a continuous variable.) Statistical experts have
opinions on which statistics they prefer, and there is some variation in
preference across fields of study. But readers should not lose sight of the
forest for the trees, and I think your website, among other valuable things,
serves to make it clear that the difference between autistic and control
subjects shows up using a variety of statistical techniques, yes? In hindsight,
we agree that publishing confidence intervals would have been a wise addition.
On the other hand, we have provided this analysis when asked to do so by an
interested party, we are not trying to hide or misrepresent anything. Most to
the point -- one can do nothing more transparent than to provide the complete
data set so that interested statisticians can do their own preferred analyses.
This was done by the editor in the same issue and we referenced this in our
article. Surely no one would advocate publishing the data set twice in the same
issue? (?)
We would like to see the following point clarified: According to the abstract, DeSoto & Hitlan “found that the original p value was in error and that a significant relation does exist between the blood levels of mercury and diagnosis of an autism spectrum disorder.” As written on your website, this might be seen as being taken out of context. Please clarify that this conclusion was actually qualified – the full sentence makes clear we are speaking about within this data set. It does matter to us. We would not want it to appear to anyone that we are drawing conclusions beyond our data set. We think Ip et al were somewhat guilty of this in 2004, and were actually careful to avoid doing so ourselves. It may seem like a fine line, but to us it matters.
We have offered this reply because it was solicited by you
(the author EpiWonk who appears to add some meaningful contribution to the
debate about this issue). We are happy to reply to those who email us directly,
and in fact always do so. Having said this, we have stopped reading the postings
and did not read all of the postings on your website. Therefore, do not take a
failure to comment on specific comments within the blog postings as agreement.
Thank you for your efforts and we hope you will help your readers maintain an
accurate view of the related issues.
---
Epiwonk penned in a follow up post, DIET (SEAFOOD): This is the reason I put in the quote from Ip et al: “This study is limited by the sample size and culture because Hong Kong Chinese are famous for eating seafood.” I agree with all of you that this should be be taken into account in any discussion of these data. As a scientist, it upsets me that that the DeSoto & Hitlan paper made it almost straight from the pages of the journal in November 2007 to the mouth of Congressman Dan Burton in December as firm evidence that ethymercury in vaccines is causing autism in “our” kids.
Epiwonk-- We had no way of controlling for this, did we? Not asking about the seafood intake of the participants seafood question would be an oversight of Ip and Wong in their data collection. Please note the strong conclusions that Ip and Wong made in 2004 about this data set "not causally related". Do you not think they overstated their results back then? Why did no one call them on this if they had such obvious confounds? Does it make logical sense to attack our reanalysis for their potential failure to collect information on a variable that could confound the results? We have been clear that these results do not prove vaccines cause autism and would be happy to help anyone who has questions understand exactly what we think this data set means. Are we responsible for all the blogging and opinions based on our research? Much of it much farther astray than Burton. I think we have gone above and beyond to try and help the public understand what these results say and do not say.
Epiwonk wondered, "Did Catherine DeSoto make any attempt to contact
Congressman Burton or his staff to tell them that these were data from Hong Kong
children and correct any of the misinformation? I’ve dealt with Congressional
staff during my career; this an be done."
We have stated something similar in on line posts and in journal publication. I
was invited by an attorney to testify/get involved in the vaccine court
proceedings, but declined as I feel I do not have anything to add at this point
(beyond removing the Ip and Wong as evidence against a vaccine link and/or
mercury link). I respond to every query I get and would do so for
congressional staff as well. Congressman Burton appears to be trying to
understand the data set – and like many of the postings I have read by
non-scientists – overstates the results to some extent. Again, I try to be as
clear as humanly possible, and hope that you yourself will revise your point
three to avoid giving readers on either side of the issue from the impression
that we feel we have proven a relationship between mercury and autism with this
one data set…. The data set shows a relationship, but any single data set
should be taken as a single data set. Had this standard been applied to the 2004
publication, some in the scientific community may have realized that the
possibility of a relationship should not have been entirely written off when
only one single case control study of blood mercury levels existed (why is that
by the way?) …. The irony is seeing those who have long argued for a causal link
to vaccines try to turn the tables on Shattuck or Fombonne (examples of
scientific experts who cited Ip and Wong as proof vaccines have been ruled out
as a cause of autism) by saying that a statistically significant result serves
as proof that vaccines have caused autism. The logic in both conclusions may be
essentially faulty, but those who now cry foul over this approach but failed to
do so when Fombonne or Shattuck used it, appear logically inconsistent. To be
clear, the proclamation that Ip and Wong 2004’s results (as they were published)
would have proven that vaccines do not cause autism was faulty. To say the 2007
reanalysis provides proof positive that a link exists outside this data set is
faulty. But with the 2004 data having been fundamentally a whole series of
errors, and a new analysis showing some relationship does exist in the data set
(even Epiwonk's alternate analysis shows a statistically significant
relationship)-- the weight of evidence should properly be seen as having shifted
a bit. Come on EpiWonk – step up tell your readers you agree.
“The majority of recent studies have failed to establish a connection between measles-mumps-rubella vaccination or the use of mercury-based vaccine preservative and autism,” Shattuck 2006 with Ip 2004 as a cite.
“By and large, biological studies of ethylmercury exposure have also failed to support the thimerosal hypothesis,” (includes Ip 2004) and then goes on to chide those who continue to consider the possibility that vaccines could have any bearing on autism (Fombonne, 2006).