Archives for April 2014

Earlier today, Emily Kopp of Federal News Radio interviewed Congressional Data Coalition chairman Daniel Schuman about the launch of the coalition and a recent victory in the House of Representatives. Listen here.

“Doesn’t Congress already make its information publicly accessible?”

That’s the question I hear most frequently when I tell people about the Congressional Data Coalition’s mission to get Congress to provide open access to its data. “Open access” is a complicated and loaded term in the digital information world, but at its core it involves three main components:

1. The ability to find the data.
2. The ability to use the data.
3. The ability to repurpose the data.

Truly achieving open access to congressional data will require more than just posting the information online: the information has to be in the correct format. Presenting data as gobs of text is seriously problematic because machines cannot read it.

In today’s information ecosystem, information that cannot be parsed and read by machines is like building an all-terrain vehicle that can only drive straight forward. It might be able to get you where you need to go, but only if your destination lies straight ahead. And it completely defeats the purpose of being able to drive off-road.

So what can congressional data that is machine-readable do that facilitates open access?

1. Finding the data: Search engines can search for the content stored within documents.
2. Using the data: A variety of programs can access and display the data. Mobile apps can provide to-the-minute updates, APIs can scrape it and immediately display it on another website, programs can download it into spreadsheets, etc.
3. Repurposing the data: Data can be run through programs that display it in charts, graphs, or elegant visualizations. Journalists and engaged citizens can also get timely access to the data that informs their output and ideas.

Plus there is a multitude of extra benefits. Machine-readable data is more accessible to people with disabilities because screen readers can read it. It is also easier to preserve because the data is not dependent on the software we use to access it, which will most likely become obsolete within the next ten years (remember floppy disks? Word Perfect?). Because laws passed by Congress can remain in effect for decades, we have to keep the data that allows us to put those laws in context.

Earlier today, the House of Representatives’ Appropriations Committee made a major move towards improving public access to legislative information. In layman’s terms, the committee said that by the beginning of the next Congress information about the disposition of bills—where they are in the legislative process and who authored or co-sponsored the legislation—will be published in a way that computers can easily process, and thus can be easily reused by apps and websites.

Americans access legislative information through third-party sites. This change in publication policy will help guarantee that accurate, timely, and complete legislative information is directly available from the official source. Congress already publishes the text of legislation in a structured format that is downloadable in bulk.

The committee specifically directed the Clerk of the House to work with the Librarian of Congress and the Public Printer to publish bill status information for bulk data downloads by the beginning of the next congress. This has been a long-standing request of the public interest community and was the subject of a recent letter sent by CREW and GovTrack.us on behalf of the newly formed Congressional Data Coalition.

The report language came at the behest of Rep. Mike Quigley (D-IL), who recommended the committee adopt this language in its report. His recommendation was the culmination of many years of hard work by legislative transparency advocates in both parties, including (but not limited to) Speaker John Boehner (R-OH), Majority Leader Eric Cantor (R-VA), Minority Leader Steny Hoyer (D-MD), and Reps. Darrell Issa (R-CA), Mike Quigley (D-IL), Mike Honda (D-CA), and Ander Crenshaw (R-FL).

In June 2012, Speaker Boehner, Majority Leader Cantor, and then-Legislative Branch Appropriations Subcommittee Chairman Ander Crenshaw issued a letter on the occasion of the establishment of a Legislative Bulk Data Task Force charged with looking into improved public access to legislative information, stating “our goal is to provide bulk access to legislative information the American people without further delay.” Rep. Issa had offered an amendment to put that requirement into law, but withdrew it pending the report of the Task Force. In its December 2013 report, the Task Force recommended “that it be a priority for Legislative Branch agencies to publish legislative information in XML and provide bulk access to that data.” While the issue was not raised during the recent Legislative Branch Appropriations Subcommittee hearings, Ranking Member Debbie Wasserman-Schultz (D-FL) singled out Rep. Quigley at the full committee hearing for making the recommendation.

With the report language in the final committee report, it is unclear what additional action, if any, is necessary to put it into effect. The House Appropriations Committee has tremendous sway over legislative branch agencies, who may spring to comply even in the absence of floor action in the House. The Senate, in its own committee report, may not address the issue (thus perhaps giving tacit approval) or may expressly agree or disagree to bulk publication of bill status information. Indeed, the Senate’s Legislative Branch Appropriations Subcommittee is still reviewing its appropriation bill, having met just yesterday.

Regardless, today’s action in the House is a significant win for transparency. Public interest advocates have been fighting for bulk access to legislative information at least since May 2007, and the House has now put its full weight on the side of legislative transparency.

Here is the report language:

The Committee request that the Clerk of the House, the Librarian of Congress and the Public Printer work together to make available to the public through Congress.gov or FDsys bulk data downloads of bill status by the beginning of the next Congress.

One of the benefits of the Congressional Data Coalition has been our ability to collaborate on mutual projects of interest. CDC members recognize that reusable, cleaned-up legislative information, especially the laws themselves, is essential for both the legislative data community and the public. Unfortunately, at least some information will likely not be provided by Congress or will not be provided in a timely manner.

Almost 3½ years ago, in November 2010, GPO and the Library of Congress were authorized by the Joint Committee on Printing to make the following three document sets available on the Internet: Statutes at Large, the Congressional Record (1878-1998), and the Constitution of the United States: Analysis and Interpretation (CONAN). Quoting from the JCP letter: “These are key primary research sources, essential to understanding our laws and legislative history, and they should all be readily available online in electronic format.”

So far, volumes 65 through 124 (1951-2010) of the Statutes at Large and PDF files only of CONAN have been published by the Legislative Branch per the November 2010 authorization.

Why are the Statutes at Large important?

The United States Statutes at Large is the legal and permanent evidence of all the laws enacted during a session of Congress (1 U.S.C. 112). Every law, public and private, is published in the order of its passage. The set contains treaties and international agreements before 1948, concurrent resolutions, proposed and ratified amendments to the Constitution, and proclamations by the President. Pretty much the whole enchilada – and before you ask about the Constitution, yes, volume 1 includes the Declaration of Independence, the Articles of Confederation, and the Constitution of the United States.

But isn’t the US Code the law? Only a subset of the laws in the Statutes at Large are contained within the U.S. Code and many of those laws have been modified by subsequent laws to the point that the original language is difficult to discern. Hundreds of laws have been enacted that never made it into the United States Code. For example, of the 440 laws enacted in 1949, 235 made it into the US Code.

The importance of Internet accessibility to the laws enacted before 1951 should be obvious. The Law Revision Counsel (the organization responsible for putting together the U. S. Code) in their Table of Acts Cited by Popular Name have identified almost 2,100 laws that were enacted before 1951. Searching legislative text from the 112th Congress (2011-2012) shows that the past is not completely forgotten. About 6 percent of Statute at Large citations reference pre-1951 volumes.

So, while some of these laws are cited in current bills, they remain in 2014, officially available only as paper documents and, unofficially, there are scanned versions of the volumes at the Constitution Society’s website, but these volume files have not been broken down into individual laws, treaties, Presidential proclamations, etc. until now.

Making more laws available

Starting in January 2014, the Congressional Data Coalition and citizens joined together to make the individual laws and other documents of the US Statutes at Large available as discreet PDF files. We’re a little over half way through the initiative but we need volunteers to help for the final push.

Rather than attempting to produce a full-text table of contents for each volume as was accomplished by GPO for the post-1950 volumes, we’ve extracted the page number where each component (public law, resolution, etc.) begins by reusing the OCRed text from the constitution.org PDF files. We then crowdsource the proofreading and correcting of the data which is where we need your help. Once the simple table of contents is completed, software extracts the individual PDF files for each sub-document. The software to do all this is open source and available online.

As of April 2014, volumes 28 through 64 (1893-1951) have been processed. We’ve also begun extracting the text from the tables of contents from the volume files and combined it with the simple table of contents data being used to create the files (sort of like a final QA check). By combining the two data sources (the text from the tables of contents along with the public law number and stat page data, we’ve been able to build more usable tables of contents. See the U. S. Statutes at Large Pre-1951 Directory.

The future of legislative data collaboration

Our approach has combined crowd-sourcing, manual editing, and automated processes. We’ve received help from a variety of outstanding volunteers. In two months, we have expanded the availability of laws by 50 years and over 15,000 acts, treaties, and international agreements.

Similar approaches should be strongly considered for publishing other historical documents on the Internet. The best example of the elephant in the room of course is the Congressional Record – only available on the Internet back to 1994 but published since 1873. As software developers, both inside and outside of government, we should be thinking in terms of how crowdsourcing can help us build the necessary document repositories for the 21st century.

Our role, as the Congressional Data Coalition, includes supporting public initiatives that provide improved legislative information for ourselves and the public. Tom Bruce, Director of the Cornell Law Information Institute, said it eloquently in his hangout session when he talked about the dream of having an open-access Westlaw or LexisNexis with layered access to information providing legal/legislative services housed under many roofs – a federation of services and data.

We should not shy away from identifying data anomalies and provide corrected data in a fully transparent and constructive way to support the public need for accurate and timely legislative information. It might not seem that having all of these laws as discreet files on the Internet would mean much. We’ve lived without them as discreet electronic files for a long time without any apparent problems. My hope for now is that these documents will extend our electronic legislative library so that our history can be read and referenced over the Internet.

Please consider helping our effort and volunteering along with us at http://legisworks.org/sal. Special thanks to Owen Ambur, Daniel Schuman, Sara S. Frug, Joe Jerome, and Matt Steinberg.

Daniel Schuman and I have a new op-ed on legislative data in The Hill:

Nearly two decades ago, Congress began publishing some of its activities online, revolutionizing access to essential public information. The system was called THOMAS, after our third president. Managed by the Library of Congress, it aimed to serve as a central hub to find bills and resolutions, the Congressional Record, committee reports, treaties and so on. There’s no doubt that, for 1995, this was a huge leap forward.

While technology has changed a lot since the mid 1990s, the quality of data coming from Congress has not kept up. Complicating matters, the THOMAS website is set to be retired and replaced with Congress.gov by the end of 2014. Once this happens, applications that have been developed with this data, and that are used extensively by congressional members and their staff, interest groups and citizens, will stop working.

Read the rest here.