Information Quality, Peer Review, and Consumer Reports
Misleading child safety seat tests withdrawn
26 Jan 2007 in Regulatory Science, Information Quality, Peer Review
A recent story involving an erroneous test report about child safety seats published by Consumer Reports shows how information quality is not just a concern related to information disseminated by the federal government.
It also illustrates the value of genuinely independent peer review. Consumer Reports is published by Consumers Union, a nonprofit organization that does much more than product testing. It is an activist organization that routinely takes strong positions on a wide range of public policy issues, including child safety seats.
CU's activism creates an inherent conflict of interest with its product testing functions. Credibility as a product safety tester requires, at a minimum, an extraordinarily rigorous program of independent peer review. Currently, CU relies solely on internal peer review.
Consumers Union, the activist nonprofit organization that publishes Consumer Reports, announced on January 18 that it has withdrawn a report on child safety seats published in its February 2007 issue. CU also has posted a letter from president Jim Guest on on its web site and sent that letter via email to subscribers on January 26.
CU had promoted its report on January 4. A subsequent report in the Salt Lake Tribune by Jeremiah Stettler and Dawn House quotes from the now-withdrawn report and leads with CU's public policy advocacy:
Consumer Reports, a product watchdog, called on the National Highway Traffic Safety Administration (NHTSA) this week to toughen its standards. The magazine reported that 10 out of 12 popular car seats failed at speeds as low as 35 mph.
"Most failed disastrously," the report stated. "The car seats twisted violently or flew off their bases, in one case hurling a test dummy 30 feet across the lab."
The test is a first for the car seat industry. Instead of measuring the seats' performance against the NHTSA standard of a 30-mph front-end collision, Consumer Reports exposed them to the same tests afforded to new cars - a 35-mph frontal crash and 38-mph side collision.
CU apparently withdrew its report after receiving information from the National Highway Traffic Safety Administration proving that CU's testing was faulty. Instead of simulating side impacts at 38.5 mph as CU had claimed, its tests corresponded to impacts exceeding 70 mph -- an obvious and significant error. The statement from NHTSA administrator Nicole Nason is here.
NBC Nightly News reporter Tom Costello says (see sidebar video) CU "is indeed in damage control" over this episode. Costello interviews a mother who after being informed that CU's tests were at double the reported speed continued to express concern because she drives at 70 mph on the freeway.
This short vignette illustrates the power of misinformation when it comes from a trusted source. The tests in question concern side impacts at 70 mph, a scenario that is virtually impossible on a controlled access divided highway such as a freeway. NHTSA's side impact test concerns the rare event when one driver on a major major city street or highway ignores a red light at a major intersection and strikes another vehicle broadside. For the impact to occur at 70 mph, the driver violating the red light must also be driving recklessly.
An organization that takes positions on consumer product safety issues has an obvious conflict of interest when it conducts purportedly independent tests on those same consumer products. Because of this conflict of interest, it is susceptible to the temptation to conduct tests in a manner that support the political positions it advocates. It is also more likely to uncritically believe that test results showing product failure are valid and reliable. External independent peer review is crucial for organizations that are conflicted, and CU does not have such a program.
Meanwhile, Costello reports that CU continues to insist that certain child seats that performed poorly in its 70 mph test should be recalled. By continuing to maintain this policy position after withdrawing its test results, CU is proving that it does not even recognize that its political activism conflicts with scientific and technical integrity.
CONSUMERS UNION PRESS RELEASE
January 18, 2007
CONSUMER REPORTS WITHDRAWS INFANT CAR SEAT REPORT
Move is made pending additional testing now underway
NEW YORK (Jan. 18) -- Consumer Reports is withdrawing its recent report on infant car seats pending further tests of the performance of those seats in side-impact collisions.
A new report will be published with any necessary revisions as soon as possible after the new tests are complete.
We withdrew the report immediately upon discovering a substantive issue that may have affected the original test results. The issue came to light based on new information received Tuesday night and Wednesday morning from the National Highway Traffic Safety Administration (NHTSA) concerning the speed at which our side-impact tests were conducted.
The original study, published in the February issue of Consumer Reports, was aimed at discovering how infant seats performed in tests at speeds that match those used in the government’s New Car Assessment Program (NCAP). This program tests most new vehicles in crashes at speeds of 35 mph for frontal impact and 38 mph for side impact. Child safety seats, in contrast, are currently tested only in front-impact crashes at speeds of 30 mph.
Our tests were intended to simulate side crashes at the NCAP speed of 38 mph. The new information raises a question about whether the tests accurately simulated that speed, however, so we are now reviewing our tests and the resulting article.
To those who may have seen the report earlier in print, on the Web, or in broadcasts, we urge you to remember that use of any child seat is safer than no child seat, but to suspend judgment on the merits of individual products until the new testing has been completed and the report re-published.
We appreciate that manufacturers and particularly NHTSA are engaging directly with us on this article, and we applaud NHTSA for giving serious consideration to development of side-impact child seat tests. Consumer Reports has long advocated adoption of such tests, since government data shows that side crashes account for a significant number of child fatalities.
We look forward to re-issuing guidance on child-seat safety as soon as possible.
WITHDRAWAL ANNOUNCEMENT
January 2007
We always strive to be accurate and fair, and I regret this error. Going forward, I want to make sure that our actions are as thorough and transparent as possible so that we preserve your trust as we continue to test, inform, and protect consumers. To that end, I'm writing you and the millions of other Consumer Reports readers to tell you what I know about the situation and what we're doing about it.
Here's what I know so far: One of our tests was intended to simulate how infant car seats perform in a side-impact crash at 38 mph. That's the speed at which many new vehicles are tested in side crashes by the government's auto safety agency. But upon reevaluating our data, we believe our tests simulated crashes that were much more severe than that.
Some of the questions I've heard involve our use of an outside lab to conduct the crash tests. While the vast majority of product testing by Consumers Union occurs in our own labs, we sometimes use outside contractors that have special test equipment or other expertise that we don't. This enables us to inform you about the safety, reliability, and performance of important products that we couldn't otherwise test.
That said, we expect all our testing to meet the same high standards, and our own staff oversees all projects. The board of directors and I are appointing a panel of experts to review this incident and determine what went wrong.
We're also retesting the infant car seats featured in our article as thoroughly and quickly as possible, so that we can publish our findings and help parents who are making this important buying decision. I've directed that we suspend the article's Ratings and other recommendations regarding specific car-seat models until this retesting is completed. In any case, I again stress the importance of what we say in the article: Any child car seat is better than no seat at all.
For 71 years, the staff of Consumers Union has worked hard to earn the trust of readers like you and to build the stellar reputation we have enjoyed. We test more than 3,000 products each year, and errors like this one are rare. I apologize on behalf of Consumers Union and I promise you we're working hard to ensure that such an error does not happen again.
Sincerely,
 |
|
Jim Guest |
| NHTSA 01-07 Thursday, January 18, 2007 |
Contact: Heather Hopkins Telephone: (202) 366-9550 |
|---|
Statement From National Highway Traffic Safety Administrator Nicole Nason Regarding Consumer Reports’ Withdrawal of Its Infant Car Seat Test Report
“Consumer Reports was right to withdraw its infant car seat test report and I appreciate that they have taken this corrective action. We are always eager to work with Consumer Reports and other organizations to improve child safety and ensure that consumers continue to have access to accurate and credible data. I was troubled by the report because it frightened parents and could have discouraged them from using car seats. It is absolutely essential for every parent to understand that the safest place in an automobile for an infant is in a car seat. Simply put, car seats are the best defense for a child in a crash.
“Our initial review of the Consumer Reports testing procedures showed a significant error in the manner in which it conducted and reported on its side-impact tests. The organization’s data show its side-impact tests were actually conducted under conditions that would represent being struck in excess of 70 mph, twice as fast as the group claimed. When NHTSA tested the same child seats in conditions representing the 38.5 mph conditions claimed by Consumer Reports, the seats stayed in their bases as they should, instead of failing dramatically.”
Video footage of NHTSA’s side-impact tests can be found here (Windows Media format).
A high-resolution version in MP2 format of the Compass safety seat can be downloaded from NHTSA's FTP server here.
A high-resolution version in MP2 format of the Evenflo Discovery safety seat can be downloaded from NHTSA's FTP server here.
