BillScraper.js 📝📝📝

A NPM module to scrape, analyze, and cache Congressional bills. Works for bills from the 113th, 114th, and 115th congress.

docs: Holds markdown files that explain the project in more detail.
examples: Holds example code.
example_outputs: Holds example outputs.
lib: Holds all Javascript files
misc: Hold all graphics for the README.md
test: Holds all unit tests.

Getting Started

To get started, visit GET_STARTED.md.
For examples of how to use the library, visit examples.
A variety of guides on how to use the library are available in docs.
Documentation for all the methods is available here.

Why?

The government hosts bulk data for all congressional bills at gpo.gov/fdsys/bulkdata. However, this repository is horrible to use for developers, as bills are uploaded to unpredictable URLS as XML files. Each bill file is too structured, as XML style elements like paragraphs, bold text, and quotes are preserved, making it hard to parse the actual text of a bill.

What does this project do

Returns full Congressional bill text and metadata from Congress.gov as JSON. This library can scrape both the actual bill text (bills) and meta information such as cosponsors, amendments, and committees (bill statuses). For example, the following code will convert the XML (on the top) into the JSON (on the bottom)
```
var billScraper = require("bill-scraper");
billScraper.getBill("BILLS-113hr1033rs", function(res){
 billScraper.printBill(res, function(parsedResult){
   console.log(JSON.stringify(parsedResult));
 });
}, {session: 2});
```
XML | JSON
:———————————-:|:———————————-:
|

For more detailed examples, visit examples/get_bills.js.

Parses bills for key points and ideas: This project generates “tags” for bills, allowing them to more easily searched. For instance, searching for all bills tagged with “battlefield”.

var billScraper = require("bill-scraper");
billScraper.getBill("BILLS-113hr1033rs", function(res){
billScraper.printBill(res, function(parsedResult){
  billScraper.tagBill(parsedResult, function(tags){
     console.log(tags);
  });
});
}, {session: 2});
//Will return ['battlefield', 'sites','war','section','land','subsection','period','funds','appropriation','protection','revolutionary','public','sub','term','nations','civil','preservation','service','sellers','prohibition','purposes','person','entity','matter','influence','law','ratification','policy'];

For more detailed examples, visit examples/tag_bills.js.

Generates a searchable Mongo database of bills: For example, the following code will locally cache a database (see photo below) of parsed, searchable bills.
```
var billScraper = require("bill-scraper");
var mongoose = require("mongoose");
mongoose.connect("mongodb://localhost/bills");
billScraper.generateDB("hr", 115, 1);
```
For documentation on the Bill struct stored in the database, visit BILL_TEMPLATE.md in docs, for more detailed examples, visit examples/database.js, and for information on searching bills, see the examples in examples/search_bills.js.

Community

This repository encourages all users to contribute. Please read the Code of Conduct and Contributing.md. For things to contribute, check out the issues or TODO.md.

Datasets used

GPO Bill bulk data: I am using the “Congressional Bills” and “Bill Statuses” folder.

Relevant libraries used

chai and mocha: To run unit tests
jsdoc: To generate function-level documentation
istanbul and codecov: To generate code coverage reports
mongoose and mongo: For the database
retext and retext-keywords: To automatically tag bills by subject
htmlparser, request, soupselect, and xml2js for web scraping.

View the package.json for the full list of dependencies.

Contributors

Jeffrey Shen: Initial development. Currently maintains the code.
Amanda Li
Malcolm Essaid
Ryan Greenberg
Herbie Turner
Kevin Sun

View the full list of contributors here.

License

This repository is licensed as CC0. See the LICENSE for more information.

To the extent possible under law,

Jeffrey Shen
has waived all copyright and related or neighboring rights to
BillScraper.js.
This work is published from:

United States.