January 24, 2015

Introduction to the EIA Website and API

Energy economics has always been an interest of mine, and I love keeping up to date on current developments in technology, as well as understanding historical trends in the economics of energy infrastructure, development, and commerce. This project looks into historical trends in the energy industry, and involves a bit of data mining to learn more about energy. In particular, this project looks at the production fuel sources, the generation of energy, and the consumption of energy.

The US Energy Information Administration's website is an excellent resource for anything energy data-related. The site hosts a wide range of data covering virtually every aspect of the energy industry, and serves as a federal hub for public energy data. This project uses Python (2.7) and Requests to remotely tap into the EIA's data via its API. Once data is compiled, Pandas, Json, and Datetime are used to reorganize the data into a manageable data frame. Using EIA's state-level data, heatmaps can be created with the help of a blank map of the United States in SVG format, along with the BeautifulSoup library.

My imports in Python for this project were as follows:

import pandas as pd
from pandas.io.json import json_normalize
import requests
import json
from bs4 import BeautifulSoup as bs
import datetime
import pickle

Some things that might be helpful for fellow beginners in Python that are covered in this project include:

• How to pull in website data with Requests
• Making use of helper functions
• How to manipulate data frames with Pandas
• What json_normalize does
• How to create a heatmap using an SVG file

Before getting started in Python, it's probably better to get registering for an EIA API key out of the way. The process is quite simple and can be done here. Once an API key is acquired, the next step is to start exploring what data the EIA site has to offer.

Next →