by Beth Jarosz
My house has a window that sticks. Locking that window requires perfectly aligned closure and a bit of arm strength. That window is annoying. I use it sparingly.
But I'm not going to burn the house down because of an imperfect window.
In May I was quoted in a news story expressing disgust that DOGE was indiscriminately canceling federal surveys. In a somewhat bold move, a reader took issue with my comments and emailed me directly to complain. I'll spare you the hysterics, but the essence is that the surveys are imperfect and therefore should be trashed.
And this is not an isolated viewpoint. The week before, someone argued to me that because the federal government has historically undercounted some population groups, all federal data are worthless and should be destroyed.
Seems a tad extreme, to me.
Let's assume the surveys are imperfect–because they are. I've been working with data for more than 25 years and have yet to meet a dataset that is “perfect.” Some are excellent. Some are middling. And some are downright terrible (someday, maybe, I'll tell stories about deduplicating records from a private vendor database). Even the decennial census–the gold standard for data quality–misses some people and counts others twice.
Imperfect data, from any source, means that data users need to understand the strengths and limitations of the dataset they’re working with. As one example, healthcare records are an incredible source of data on motor vehicle crashes. But those records miss cases where there is no emergency department visit or hospitalization, so minor injury and non-injury crashes are missed. To understand more types of crashes we may need to layer in police records, insurance claims, and/or surveys. That doesn’t mean we never use healthcare data for studying vehicle crashes. It just means we need to use our data wisely.
In truth, while robust data are fundamental to good policymaking and a sound democracy–individual datasets are not meant to stand alone. We need multiple sources to triangulate toward Truth. If we cancel one because it's imperfect, it's like pulling the side mirror off of a car–we lose a key perspective that helps us understand what’s going on around us.
Federal surveys provide many key observations that help us make sense of the world–imperfect though any one of them may be.
How do we know federal surveys provide key observations?
First and foremost, no federal survey would have been allowed to proceed if there was not a clear “need to know.” Surveys can’t be created on a whim. Questions can’t be added just because someone thinks they’re interesting. These data collection efforts need to provide mission critical data for the agency that’s collecting the information. (Might there be other ways to fill those data gaps? Sure. But making any major change should be done with research, testing, and public input. And that’s another blog for another time.)
Perhaps most importantly, federal surveys are vetted, repeatedly. OMB is also “required to report annually to Congress on ‘major activities’ under the” Paperwork Reduction Act (PRA), which OMB does through an annual "information collection budget” (details of the current paperwork burden, efforts to reduce the burden, violations of the PRA, and improvements in the use of information). In addition, federal data collection efforts are periodically re-opened for full public review. If you’ll permit me to extend the sticky window analogy, the periodic public review is like a cross between an open house and a home inspection–it’s a chance for anyone to point out what works and what needs fixing.
So the solution to a quibble about accuracy or coverage isn’t to just burn it all down. The solution is to fix what seems broken. Maybe things are so broken that a teardown and rebuild makes sense. If so, let’s plan what the new system looks like first and let’s be sure we still have a roof over our heads throughout the process.