Counting Jobs Created or Saved by the "Stimulus" Bill:
A lesson in information quality
20 Nov 2009 in Information Quality
A scandal has erupted over the federal government's reporting of the number of jobs created or saved by the "stimulus" bill (formally the American Recovery and Reinvestment Act of 2009).
This scandal would have been avoided if the government had complied with the Information Quality Act.
The Information Quality Act, and implementing guidelines issued by the Office of Management and Budget in 2002, established three main substantive principles regarding information disseminated by the federal government.
- It must be objective, which is defined as clear, accurate and unbiased, and presented in a clear, complete, and unbiased manner.
- It must be the product of data generation, management, and storage processes that have documented integrity.
- It must be so transparent that competent third-parties can reproduce it within a reasonable margin of error.
Agencies must adhere to these principles using the following methods:
- Apply sufficient internal administrative controls to prevent the dissemination of non-compliant information to the extent practicable.
- Provide for a timely and effective petition process whereby anyone can seek and obtain the correction of information that violates one or more of the information quality standards set forth above.
The web portal where the government discloses and promotes the number of jobs created or saved is recovery.gov. This portal is very nearly anonymous, however. Buried at the bottom of the "About" page is says:
Recovery.gov is operated by the Recovery Accountability and Transparency Board, which was also created by the Recovery Act.
The Recovery Accountability and Transparency Board consists of 12 agency inspectors general and chairman Earl E. Devaney, "a veteran public official dedicated to accountability and transparency in government." The Board has not issued information quality guidelines of its own, nor is information quality mentioned in its statement of Powers and Functions, its Bylaws, or its operating Guidelines.
In short, the Board has ignored applicable law and guidance. In recent congressional testimony, Devaney explicitly denied that the Board had any responsibility with respect to information quality:
A distinction needs to be made between data quality and data integrity. Although the Board and Inspectors General (IGs) will play a role in data quality – chiefly by reviewing agencies’ processes for ensuring quality of the data – the Board’s main goal will be one of data integrity. That is, the Board will strive to ensure that the data on Recovery.gov is a true reflection of what recipients report, including any subsequent modifications made to that data.
That Devaney would try to distinguish data "quality" from data "integrity," and define "integrity" in a way that expressly permits the propagation of error by the government so long as it originates elsewhere, strongly suggests that he is unfamiliar with the Information Quality Act and its guidelines, even though he was responsible for complying with them at his previous job as inspector general of the Department of the Interior.
Further, in Devaney's mind, information quality is the responsibility of those who receive federal dollars. The Board, in his view, has no responsibility for validating any of the data they submit or which the agencies forward to them:
The responsibility for data quality, however, rests with the recipients of the funds and the agencies distributing the funds, as they are in the best position to know the details associated with these funds.
Moreover, Devaney asserts that inspectors general are somehow statutorily prohibited from implementing the Information Quality Act and its associated guidelines:
IGs’ involvement in ensuring data quality could run afoul of the Inspector General Act’s longstanding prohibition on program operating responsibilities – as well as the Comptroller General’s Government Auditing Standards for performing audits, also known as the Yellow Book.
The Board's violation of the Information Quality Act and OMB's implementing guidelines is the source of the growing scandal. If the public is likely to infer governmental endorsement of information an agency disseminates, then the agency has a legal obligation to take reasonable and prudent steps to detect and prevent error, even if the information was originally generated or produced by others. The agency also has an obligation to establish an administrative process whereby the public can seek and obtain the correction of error.
The Recovery Accountability and Transparency Board has failed on both counts. As a matter of policy, it has no pre-dissemination review procedures to prevent the reporting of erroneous data. And it has no administrative procedure enabling the public to seek and obtain corrections.
As of October 30, 2009, the Board says on its home page that 640,329 jobs were created or saved by ARRA, as reported by recipients. This may be technically correct, because the Board has done nothing to attempt to validate recipients' claims. But we know from the controversy that the public reasonably infers that the Board has validated these claims. For that reason, when many of these claims were exposed as technically impossible or extraordinarily unlikely, the objectivity, integrity, and transparency of the Board's work became suspect.
Devaney now acknowledges that the figures the Board reports are unreliable and likely incorrect, but according to New York Times reporter Michael Cooper, he blames the Administration for using the information improperly. He's certainly correct that the White House has promoted the Board's figures without caution or caveat. But Recovery.Org's reporting of these figures also is light on caution or caveat. The box nearby displays the aggregate figure much less prominently than the caveat, yet we know know that the caveat is more important than the aggregate figure.
Government Accountability Office auditors, who work for Congress, are now busy auditing the work of the auditors who comprise the Recovery Accountability and Transparency Board. GAO notes that in its June 2009 guidance to agencies, the Office of Management and Budget included a section titled "Data Quality Requirements" (Sec. 4). Peculiarly, OMB's guidance failed to even mention its own information quality guidelines. OMB imposed a minimal oversight duty on agencies focused primarily on "material omissions and significant reporting errors." However, OMB allowed agencies to shift the responsibility for information quality to recipients, while at the same time declining to specify the methodology that recipients had to use to estimate jobs saved or created. If there is no set of accepted methodologies, then there is no objective basis for determining that recipients have reported erroneously.
In short, while the Information Quality Act and OMB's implementing guidelines apply to the government's reporting of job created or saved, the process put in place enabled all governmental entities in the chain to evade responsibility for ensuring and maximizing the quality of the information to be reported. This includes the agencies disbursing the money, the Recovery Accountability and Transparency Board that agglomerated them, the Office of Management and Budget which issued guidance on how to implement the reporting function, and the White House which promoted the results.
Information quality has been regularly assailed by liberal interest groups, but its intuitive appeal -- even to liberals -- is clearly evident in the context of stimulus jobs created or saved. Here's Washington Post writer Alec MacGillis:
House Appropriations Committee Chairman David R. Obey (D-Wis.) released a blistering statement Tuesday, taking to task flawed data in the federal figures that show $160 billion in spending had created or saved at least 640,000 jobs.
His criticism came in response to two ABC News stories. The first reported that the administration deleted 60,000 jobs from its list after it became clear that the figures were overstated. Then, on Monday, the network reported that the government had claimed $761,420 in spending and 30 jobs created in Arizona's 15th Congressional District -- a district that does not exist.
"The inaccuracies . . . are outrageous, and the administration owes itself, the Congress and every American a commitment to work night and day to correct the ludicrous mistakes," Obey said. "We designed the Recovery Act to be open and transparent. . . . Whether the numbers are good news or bad news, I want the honest numbers and I want them now."
The quest for "honest numbers" is the purpose of the Information Quality Act and the government's information quality guidelines. The government's decision to ignore these guidelines is the root of today's controversy. The public is unlikely to believe anything the government says unless and until it rigorously complies.
Orszag, PR. Implementing Guidance for the Reports on Use of Funds Pursuant to the American Recovery and Reinvestment Act of 2009, June 22, 2009 (excerpt, pp. 28-29).
4.3 Does this Guidance mandate a specific methodology for conducting data quality reviews?
No. However, the relevant party conducting a data quality review required by this Guidance (i.e., recipients, sub-recipients, Federal agencies) must use its discretion in determining the optimal method for detecting and correcting material omissions or significant reporting errors. At a minimum, Federal agency, recipients, and sub-recipients should establish internal controls to ensure data quality, completeness, accuracy and timely reporting of all amounts funded by the Recovery Act.
