I have been working on an analysis of a data set containing 622,493 Federal Freedom of Information Act (FOIA) requests over the last 12 years. The data set covered who requested the data, the agency and department the request was set to, the dates the cases were opened and closed, what data the request was for and what the agency response was. Unfortunately the dataset was very large and fairly messy, which made it challenging to work with. There are some queries that I would have like to run on the data too see how certain factors influenced each other, but it some cases there were simply too many holes in the data. However, I was able to come up with three visualizations of the data that shed some light on the landscape of FOIA requests. (Links to the interactive version of these graphics coming as soon as I can solve some technical issues.)
Table 1: Yearly Federal FOIA Requests. Table 2: FOIA Data Requesters. Table 3: Most FOIAed Federal Agencies.
To gain some insight and context for my findings, I sat down and talked with Northeastern University journalism professor Laurel Leff to learn more about FOIA, how it has changed over the years and why it is so important now.
I the data I found a huge increase in the number of FOIA requests, starting in 2003 but continuing to increase over the years with noticeable growth in 2009 and 2012. (Table 2). According to Professor Leff, there were two main forces that affected the FOIA act in the past few decades.
“One is the very obvious. Now we have the possibility of making this stuff available digitally, so that means that even though there might not be a philosophical change there needs to be a change in the way statutes are written,” Leff said.
“Being able to get access to [data] online makes a difference and being able to get the data in [digital] form makes an enormous difference so then you can do the kind of calculations that people do with data. Even though the government was collecting data way before you could get an electronic version, it was obviously a lot harder to manipulate it any use it in that sense. There has been a really profound transition as a result of that.”
The other change that Professor Leff discussed had the opposite affect, making data less available and the government less transparent.
“In the wake of 9/11 there was clearly a crackdown on the part of the Bush administration to take information that would have been available and say that it now presents some sort of national security threat,” Leff said. “Certainly the courts that heard these requests … were very willing to say ‘there is nothing we can do about it, it is classified information and the requester isn’t going to be able to get it.’ There are a whole slew of those kinds of cases that occur in the post- 9/11 period.”
As a journalist, I have always learned about FOIA in the context of news organizations filing FOIA requests, but unsurprisingly it is investment managers and lawyers who actually file the most FOIA requests. The difference being, Leff said, that when journalists file requests they often turn into big stories that the public learns about, where as when other types of organizations request the information it is typically kept for internal use.
“It is generally assumed that the biggest users of FOIA are businesses, not news organizations and not the public. The government is a huge repository of data about business practices, and so they do so to get a competitive edge, not using trade secrets but using any kind of data you can get that could illustrate some aspect of the economy.
“For law firms and for public interest law firms it would be it is not competitive in the sense of getting a business advantage, but if you’re filing a law suit and you’re trying to determine what the status of a group of people – either for a class action or just to bolster a claim – the government in general is the best storehouse of information about pretty much anything.”
However, in the uncertain times we are living the future of open government, and by extension the Freedom of Information Act, hangs in the balance. There is a lot of concern with federal data, particularly scientific data, being classified, altered or even deleted and causing huge set backs in research and government accountability around the world, not just in America.
“The intent of FOIA is to provide openness in government, but as I think we’ve seen in so many other areas a lot of the actual strength of our institutions is that we do it and we believe in it, not just in the fine letter of the law,” Leff said. “Certainly if someone who is skeptical of the press and of public scrutiny could do a great deal to try and get information, documents from getting to the public. And certainly that could be done fairly easily in the national security context, and probably in other contexts if you turn your attention to it. And if nothing else you can muck things up.”
But for once the size of the federal bureaucracy may benefit the people.
“The federal government is a vast enterprise and it’s hard to imagine doing that for everything,” Leff said. “There are just so many requests and so much information to be able to have control over all of it, to necessarily know what is more important is pretty hard to do. I think to some extent we can count on the size of the federal bureaucracy to mean that even with someone who might have the intent to keep information from getting to the public it would be pretty hardtop do.”