Using APIs

What are APIs?

Structured ways people can give you their data.

Why?

Usually because they want to help web/mobile developers attract more users to their service.

Twitter doesn't have an API to help you out.

They want developers to build apps to drive more eyeballs to their service.

No love with the scrape!!!

In [1]: 
import requests

Do me a favor and sign up to be a developer with the New York Times and get your own API key.

In [1]: 
my_times_api_key = 'YOUR-API-GOES-HERE'

APIs can be accessed like a normal URL, but they are often very long, complicated, and involve variables you want to change. For example, you can get the information about the first 10 articles published in the New York Times that used the word inequality with:

http://api.nytimes.com/svc/search/v2/articlesearch.json?sort=newest&begin_date=20130101&end_date=20131231&api-key=d20bc9ac37156ecc4cb3d78eb956201d%3A0%3A54059647&q=inequality&page=0

Requests allows you to do this in a more civilized way.

In [16]: 
payload = {'q'         : 'inequality', 
           'begin_date': '20130101' ,
           'end_date'  : '20131231',
           'api-key'   :  my_times_api_key,
           'sort'      : 'oldest' ,
           'offset'    :  20}

base_url = 'http://api.nytimes.com/svc/search/v2/articlesearch.json?'
In [17]: 
r = requests.get(base_url, params = payload)

r.url
Out[17]:
u'http://api.nytimes.com/svc/search/v2/articlesearch.json?sort=oldest&begin_date=20130101&end_date=20131231&api-key=d20bc9ac37156ecc4cb3d78eb956201d%3A0%3A54059647&q=inequality&offset=20'
In [18]: 
r.text
Out[18]:
u'{"response":{"meta":{"hits":1255,"time":371,"offset":0},"docs":[{"web_url":"http:\\/\\/www.nytimes.com\\/2013\\/01\\/01\\/opinion\\/struggling-to-get-a-college-degree.html","snippet":"Readers react to a front-page article describing the travails of three low-income students attending college.","lead_paragraph":"Readers react to a front-page article describing the travails of three low-income students attending college.","abstract":null,"print_page":"18","blog":[],"source":"The New York Times","multimedia":[],"headline":{"main":"Struggling to Get a College Degree","kicker":"Letters"},"keywords":[{"value":"Student Loans","name":"subject","rank":"4"},{"value":"Graduation Rates","name":"subject","rank":"2"},{"value":"Income","name":"subject","rank":"5"},{"value":"Colleges and Universities","name":"subject","rank":"1"},{"value":"Income Inequality","name":"subject","rank":"6"},{"value":"Scholarships and Fellowships","name":"subject","rank":"3"}],"pub_date":"2013-01-01T00:00:00Z","document_type":"article","news_desk":null,"section_name":"Opinion","subsection_name":null,"byline":[],"type_of_material":"Letter","_id":"50e2389500315214fbb81e4c","word_count":602},{"web_url":"http:\\/\\/www.nytimes.com\\/aponline\\/2013\\/01\\/01\\/world\\/europe\\/ap-eu-vatican-new-years.html","snippet":"Pope Benedict XVI said Tuesday he is convinced that peace will prevail in 2013, despite the inequality, terrorism and \\"unregulated financial capitalism\\" that afflict the world today.","lead_paragraph":"Pope Benedict XVI said Tuesday he is convinced that peace will prevail in 2013, despite the inequality, terrorism and \\"unregulated financial capitalism\\" that afflict the world today.","abstract":null,"print_page":null,"blog":[],"source":"AP","multimedia":[],"headline":{"main":"Pope Convinced of Peace in 2013 Despite World Woes"},"keywords":[],"pub_date":"2013-01-01T04:31:19Z","document_type":"article","news_desk":null,"section_name":"World","subsection_name":null,"byline":{"organization":"THE ASSOCIATED PRESS","original":"By THE ASSOCIATED PRESS","person":[]},"type_of_material":"","_id":"50e2af0f00315214fbb81ecb","word_count":210},{"web_url":"http:\\/\\/www.nytimes.com\\/reuters\\/2013\\/01\\/01\\/world\\/europe\\/01reuters-pope.html","snippet":"Pope Benedict said in his New Year\'s message on Tuesday he hoped 2013 would be a year of peace and that the world was under threat from unbridled capitalism, terrorism and criminality.","lead_paragraph":"Pope Benedict said in his New Year\'s message on Tuesday he hoped 2013 would be a year of peace and that the world was under threat from unbridled capitalism, terrorism and criminality.","abstract":null,"print_page":null,"blog":[],"source":"Reuters","multimedia":[],"headline":{"main":"Pope Hopes for 2013 of Peace, Slams Unbridled Capitalism"},"keywords":[],"pub_date":"2013-01-01T07:20:08Z","document_type":"article","news_desk":null,"section_name":"World","subsection_name":null,"byline":{"organization":"REUTERS","original":"By REUTERS","person":[]},"type_of_material":"","_id":"50e2d62b00315214fbb81ee7","word_count":442},{"web_url":"http:\\/\\/www.nytimes.com\\/reuters\\/2013\\/01\\/02\\/world\\/middleeast\\/02reuters-oman-abortion.html","snippet":"When unmarried 19-year-old Sama got pregnant, she ran away from home to have an abortion rather than face family wrath.","lead_paragraph":"When unmarried 19-year-old Sama got pregnant, she ran away from home to have an abortion rather than face family wrath.","abstract":null,"print_page":null,"blog":[],"source":"Reuters","multimedia":[],"headline":{"main":"Taboo on Premarital Sex Can Lead to Tragedy in Changing Oman"},"keywords":[],"pub_date":"2013-01-02T16:15:08Z","document_type":"article","news_desk":null,"section_name":"World","subsection_name":null,"byline":{"organization":"REUTERS","original":"By REUTERS","person":[]},"type_of_material":"","_id":"50e4a7c600315214fbb82197","word_count":1141},{"web_url":"http:\\/\\/www.nytimes.com\\/2013\\/01\\/03\\/us\\/politics\\/for-obama-fiscal-deal-is-a-victory-that-also-holds-risks.html","snippet":"The new fiscal deal holds small victories for President Obama and his Democratic allies in Congress that further their policy aims, but Mr. Obama lost some leverage for future battles.","lead_paragraph":"The new fiscal deal holds small victories for President Obama and his Democratic allies in Congress that further their policy aims, but Mr. Obama lost some leverage for future battles.","abstract":null,"print_page":"16","blog":[],"source":"The New York Times","multimedia":[{"subtype":"xlarge","url":"images\\/2013\\/01\\/03\\/us\\/OBAMA\\/OBAMA-articleLarge-v2.jpg","height":388,"width":600,"legacy":{"xlargewidth":"600","xlarge":"images\\/2013\\/01\\/03\\/us\\/OBAMA\\/OBAMA-articleLarge-v2.jpg","xlargeheight":"388"},"type":"image"},{"subtype":"thumbnail","url":"images\\/2013\\/01\\/03\\/us\\/OBAMA\\/OBAMA-thumbStandard.jpg","height":75,"width":75,"legacy":{"thumbnailheight":"75","thumbnail":"images\\/2013\\/01\\/03\\/us\\/OBAMA\\/OBAMA-thumbStandard.jpg","thumbnailwidth":"75"},"type":"image"}],"headline":{"seo":"For Obama, Fiscal Deal Is a Victory That Also Holds Risks","main":"For Obama, a Victory That Also Holds Risks","kicker":"News Analysis"},"keywords":[{"value":"Income Tax","is_major":"N","rank":"5","name":"subject"},{"value":"National Debt (US)","is_major":"N","rank":"3","name":"subject"},{"value":"Obama, Barack","is_major":"Y","rank":"1","name":"persons"},{"value":"Federal Budget (US)","is_major":"Y","rank":"2","name":"subject"},{"value":"Income Inequality","is_major":"N","rank":"4","name":"subject"}],"pub_date":"2013-01-03T00:00:00Z","document_type":"article","news_desk":"National","section_name":"U.S.","subsection_name":null,"byline":{"contributor":"Jonathan Weisman contributed reporting.","person":[{"organization":"","role":"reported","rank":1,"firstname":"David","lastname":"LEONHARDT"}],"original":"By DAVID LEONHARDT"},"type_of_material":"An Analysis; News Analysis","_id":"50e4ef3600315214fbb8223c","word_count":"1111"},{"web_url":"http:\\/\\/bittman.blogs.nytimes.com\\/2013\\/01\\/03\\/a-months-worth-of-links-for-new-year\\/","snippet":"Some farming, health and GMO news to usher you into 2013.","lead_paragraph":null,"abstract":"Some farming, health and GMO news to usher you into 2013.","print_page":null,"blog":[],"source":"The New York Times","multimedia":[],"headline":{"main":"A Month\'s Worth of Links for New Year","kicker":"Mark Bittman"},"keywords":[{"rank":"1","name":"persons","value":"Boxer, Barbara"},{"rank":"2","name":"persons","value":"Philpott, Tom"},{"rank":"1","name":"glocations","value":"Colorado River"},{"rank":"4","name":"glocations","value":"National Petroleum Reserve (Alaska)"},{"rank":"3","name":"glocations","value":"India"},{"rank":"2","name":"glocations","value":"Doha (Qatar)"},{"rank":"6","name":"glocations","value":"Wyoming"},{"rank":"5","name":"glocations","value":"Pakistan"},{"rank":"5","name":"organizations","value":"Kansas City Star"},{"rank":"6","name":"organizations","value":"Monsanto Company"},{"rank":"8","name":"organizations","value":"PepsiCo Inc"},{"rank":"7","name":"organizations","value":"Obesity (Journal)"},{"rank":"9","name":"organizations","value":"United Nations"},{"rank":"2","name":"organizations","value":"Dominos Pizza Inc"},{"rank":"3","name":"organizations","value":"Environmental Protection Agency"},{"rank":"4","name":"organizations","value":"Food and Drug Administration"},{"rank":"1","name":"organizations","value":"AquaBounty Technologies"},{"rank":"10","name":"organizations","value":"Wal-Mart Stores Inc"},{"rank":"3","name":"subject","value":"Diet and Nutrition"},{"rank":"2","name":"subject","value":"Aurora, Colo, Shooting (2012)"},{"rank":"8","name":"subject","value":"Soft Drinks"},{"rank":"5","name":"subject","value":"Newtown, Conn, Shooting (2012)"},{"rank":"4","name":"subject","value":"Greenhouse Gas Emissions"},{"rank":"1","name":"subject","value":"Antibiotics"},{"rank":"7","name":"subject","value":"Poverty"},{"rank":"6","name":"subject","value":"Obesity"}],"pub_date":"2013-01-03T16:29:38Z","document_type":"blogpost","news_desk":null,"section_name":"Opinion","subsection_name":null,"byline":{"person":[{"organization":"","role":"reported","firstname":"Mark","rank":1,"lastname":"BITTMAN"}],"original":"By MARK BITTMAN"},"type_of_material":"Blog","_id":"50e5f9c600315214fbb8247e","word_count":"1182"},{"web_url":"http:\\/\\/www.nytimes.com\\/roomfordebate\\/2013\\/01\\/03\\/venezuela-post-chavez\\/","snippet":"What sorts of social, economic and foreign relations issues must Venezuela\'s next president confront in order to guarantee a smooth transition?","lead_paragraph":"","abstract":"What sorts of social, economic and foreign relations issues must Venezuela\'s next president confront in order to guarantee a smooth transition?","print_page":null,"blog":[],"source":"The New York Times","multimedia":[],"headline":{"main":"The Future of Venezuela","kicker":"Room For Debate"},"keywords":[{"value":"News","name":"type_of_material","rank":"1"}],"pub_date":"2013-01-03T19:11:49Z","document_type":"blogpost","news_desk":null,"section_name":"Opinion","subsection_name":null,"byline":null,"type_of_material":"Blog","_id":"5136a39dcf28d009de003a55","word_count":"3881"},{"web_url":"http:\\/\\/www.nytimes.com\\/aponline\\/2013\\/01\\/03\\/us\\/ap-us-obit-lerner.html","snippet":"Gerda Lerner, a pioneer in the field of women\'s history and a founding member of the National Organization for Women, has died in Wisconsin, her son said Thursday. She was 92.","lead_paragraph":"Gerda Lerner, a pioneer in the field of women\'s history and a founding member of the National Organization for Women, has died in Wisconsin, her son said Thursday. She was 92.","abstract":null,"print_page":null,"blog":[],"source":"AP","multimedia":[],"headline":{"main":"Women\'s History Pioneer Gerda Lerner Dies at 92"},"keywords":[],"pub_date":"2013-01-03T20:26:09Z","document_type":"article","news_desk":"None","section_name":"U.S.","subsection_name":null,"byline":{"organization":"THE ASSOCIATED PRESS","original":"By THE ASSOCIATED PRESS","person":[]},"type_of_material":"","_id":"50e6312d00315214fbb8251a","word_count":795},{"web_url":"http:\\/\\/www.nytimes.com\\/2013\\/01\\/04\\/opinion\\/kurgman-battles-of-the-budget.html","snippet":"Let\\u2019s put the cliff deal, which was part of a larger class war, in perspective.","lead_paragraph":"Let\\u2019s put the cliff deal, which was part of a larger class war, in perspective.","abstract":null,"print_page":"23","blog":[],"source":"The New York Times","multimedia":[{"subtype":"wide","url":"images\\/2010\\/09\\/16\\/opinion\\/Krugman_New\\/Krugman_New-thumbWide.jpg","height":126,"width":190,"legacy":{"wide":"images\\/2010\\/09\\/16\\/opinion\\/Krugman_New\\/Krugman_New-thumbWide.jpg","wideheight":"126","widewidth":"190"},"type":"image"},{"subtype":"xlarge","url":"images\\/2010\\/09\\/16\\/opinion\\/Krugman_New\\/Krugman_New-articleLarge.jpg","height":900,"width":600,"legacy":{"xlargewidth":"600","xlarge":"images\\/2010\\/09\\/16\\/opinion\\/Krugman_New\\/Krugman_New-articleLarge.jpg","xlargeheight":"900"},"type":"image"},{"subtype":"thumbnail","url":"images\\/2010\\/09\\/16\\/opinion\\/Krugman_New\\/Krugman_New-thumbStandard-v2.jpg","height":75,"width":75,"legacy":{"thumbnailheight":"75","thumbnail":"images\\/2010\\/09\\/16\\/opinion\\/Krugman_New\\/Krugman_New-thumbStandard-v2.jpg","thumbnailwidth":"75"},"type":"image"}],"headline":{"seo":"Battles of the Budget","main":"Battles of the Budget","kicker":"Op-Ed Columnist","print_headline":"Battles Of The Budget"},"keywords":[{"value":"United States Politics and Government","is_major":"N","rank":"5","name":"subject"},{"value":"Senate","is_major":"N","rank":"7","name":"organizations"},{"value":"Obama, Barack","is_major":"N","rank":"3","name":"persons"},{"value":"House of Representatives","is_major":"N","rank":"6","name":"organizations"},{"value":"Federal Budget (US)","is_major":"Y","rank":"1","name":"subject"},{"value":"National Debt (US)","is_major":"N","rank":"2","name":"subject"},{"value":"Law and Legislation","is_major":"N","rank":"4","name":"subject"}],"pub_date":"2013-01-04T00:00:00Z","document_type":"article","news_desk":"Editorial","section_name":"Opinion","subsection_name":null,"byline":{"contributor":"","person":[{"organization":"","role":"reported","rank":1,"firstname":"Paul","lastname":"KRUGMAN"}],"original":"By PAUL KRUGMAN"},"type_of_material":"Op-Ed","_id":"50e63a7f00315214fbb82536","word_count":"816"},{"web_url":"http:\\/\\/www.nytimes.com\\/aponline\\/2013\\/01\\/04\\/us\\/ap-us-obit-lerner.html","snippet":"Gerda Lerner spent her 18th birthday in a Nazi prison, sharing a cell with two gentile women arrested for political work who shared their food with the Jewish teenager because jailers restricted rations for Jews.","lead_paragraph":"Gerda Lerner spent her 18th birthday in a Nazi prison, sharing a cell with two gentile women arrested for political work who shared their food with the Jewish teenager because jailers restricted rations for Jews.","abstract":null,"print_page":null,"blog":[],"source":"AP","multimedia":[],"headline":{"main":"Women\'s History Pioneer Gerda Lerner Dies at 92"},"keywords":[],"pub_date":"2013-01-04T03:07:26Z","document_type":"article","news_desk":"None","section_name":"U.S.","subsection_name":null,"byline":{"organization":"THE ASSOCIATED PRESS","original":"By THE ASSOCIATED PRESS","person":[]},"type_of_material":"","_id":"50e68ee400315214fbb825a6","word_count":835}]},"status":"OK","copyright":"Copyright (c) 2013 The New York Times Company.  All Rights Reserved."}'
In [19]: 
r.json()
Out[19]:
{u'copyright': u'Copyright (c) 2013 The New York Times Company.  All Rights Reserved.',
 u'response': {u'docs': [{u'_id': u'50e2389500315214fbb81e4c',
    u'abstract': None,
    u'blog': [],
    u'byline': [],
    u'document_type': u'article',
    u'headline': {u'kicker': u'Letters',
     u'main': u'Struggling to Get a College Degree'},
    u'keywords': [{u'name': u'subject',
      u'rank': u'4',
      u'value': u'Student Loans'},
     {u'name': u'subject', u'rank': u'2', u'value': u'Graduation Rates'},
     {u'name': u'subject', u'rank': u'5', u'value': u'Income'},
     {u'name': u'subject',
      u'rank': u'1',
      u'value': u'Colleges and Universities'},
     {u'name': u'subject', u'rank': u'6', u'value': u'Income Inequality'},
     {u'name': u'subject',
      u'rank': u'3',
      u'value': u'Scholarships and Fellowships'}],
    u'lead_paragraph': u'Readers react to a front-page article describing the travails of three low-income students attending college.',
    u'multimedia': [],
    u'news_desk': None,
    u'print_page': u'18',
    u'pub_date': u'2013-01-01T00:00:00Z',
    u'section_name': u'Opinion',
    u'snippet': u'Readers react to a front-page article describing the travails of three low-income students attending college.',
    u'source': u'The New York Times',
    u'subsection_name': None,
    u'type_of_material': u'Letter',
    u'web_url': u'http://www.nytimes.com/2013/01/01/opinion/struggling-to-get-a-college-degree.html',
    u'word_count': 602},
   {u'_id': u'50e2af0f00315214fbb81ecb',
    u'abstract': None,
    u'blog': [],
    u'byline': {u'organization': u'THE ASSOCIATED PRESS',
     u'original': u'By THE ASSOCIATED PRESS',
     u'person': []},
    u'document_type': u'article',
    u'headline': {u'main': u'Pope Convinced of Peace in 2013 Despite World Woes'},
    u'keywords': [],
    u'lead_paragraph': u'Pope Benedict XVI said Tuesday he is convinced that peace will prevail in 2013, despite the inequality, terrorism and "unregulated financial capitalism" that afflict the world today.',
    u'multimedia': [],
    u'news_desk': None,
    u'print_page': None,
    u'pub_date': u'2013-01-01T04:31:19Z',
    u'section_name': u'World',
    u'snippet': u'Pope Benedict XVI said Tuesday he is convinced that peace will prevail in 2013, despite the inequality, terrorism and "unregulated financial capitalism" that afflict the world today.',
    u'source': u'AP',
    u'subsection_name': None,
    u'type_of_material': u'',
    u'web_url': u'http://www.nytimes.com/aponline/2013/01/01/world/europe/ap-eu-vatican-new-years.html',
    u'word_count': 210},
   {u'_id': u'50e2d62b00315214fbb81ee7',
    u'abstract': None,
    u'blog': [],
    u'byline': {u'organization': u'REUTERS',
     u'original': u'By REUTERS',
     u'person': []},
    u'document_type': u'article',
    u'headline': {u'main': u'Pope Hopes for 2013 of Peace, Slams Unbridled Capitalism'},
    u'keywords': [],
    u'lead_paragraph': u"Pope Benedict said in his New Year's message on Tuesday he hoped 2013 would be a year of peace and that the world was under threat from unbridled capitalism, terrorism and criminality.",
    u'multimedia': [],
    u'news_desk': None,
    u'print_page': None,
    u'pub_date': u'2013-01-01T07:20:08Z',
    u'section_name': u'World',
    u'snippet': u"Pope Benedict said in his New Year's message on Tuesday he hoped 2013 would be a year of peace and that the world was under threat from unbridled capitalism, terrorism and criminality.",
    u'source': u'Reuters',
    u'subsection_name': None,
    u'type_of_material': u'',
    u'web_url': u'http://www.nytimes.com/reuters/2013/01/01/world/europe/01reuters-pope.html',
    u'word_count': 442},
   {u'_id': u'50e4a7c600315214fbb82197',
    u'abstract': None,
    u'blog': [],
    u'byline': {u'organization': u'REUTERS',
     u'original': u'By REUTERS',
     u'person': []},
    u'document_type': u'article',
    u'headline': {u'main': u'Taboo on Premarital Sex Can Lead to Tragedy in Changing Oman'},
    u'keywords': [],
    u'lead_paragraph': u'When unmarried 19-year-old Sama got pregnant, she ran away from home to have an abortion rather than face family wrath.',
    u'multimedia': [],
    u'news_desk': None,
    u'print_page': None,
    u'pub_date': u'2013-01-02T16:15:08Z',
    u'section_name': u'World',
    u'snippet': u'When unmarried 19-year-old Sama got pregnant, she ran away from home to have an abortion rather than face family wrath.',
    u'source': u'Reuters',
    u'subsection_name': None,
    u'type_of_material': u'',
    u'web_url': u'http://www.nytimes.com/reuters/2013/01/02/world/middleeast/02reuters-oman-abortion.html',
    u'word_count': 1141},
   {u'_id': u'50e4ef3600315214fbb8223c',
    u'abstract': None,
    u'blog': [],
    u'byline': {u'contributor': u'Jonathan Weisman contributed reporting.',
     u'original': u'By DAVID LEONHARDT',
     u'person': [{u'firstname': u'David',
       u'lastname': u'LEONHARDT',
       u'organization': u'',
       u'rank': 1,
       u'role': u'reported'}]},
    u'document_type': u'article',
    u'headline': {u'kicker': u'News Analysis',
     u'main': u'For Obama, a Victory That Also Holds Risks',
     u'seo': u'For Obama, Fiscal Deal Is a Victory That Also Holds Risks'},
    u'keywords': [{u'is_major': u'N',
      u'name': u'subject',
      u'rank': u'5',
      u'value': u'Income Tax'},
     {u'is_major': u'N',
      u'name': u'subject',
      u'rank': u'3',
      u'value': u'National Debt (US)'},
     {u'is_major': u'Y',
      u'name': u'persons',
      u'rank': u'1',
      u'value': u'Obama, Barack'},
     {u'is_major': u'Y',
      u'name': u'subject',
      u'rank': u'2',
      u'value': u'Federal Budget (US)'},
     {u'is_major': u'N',
      u'name': u'subject',
      u'rank': u'4',
      u'value': u'Income Inequality'}],
    u'lead_paragraph': u'The new fiscal deal holds small victories for President Obama and his Democratic allies in Congress that further their policy aims, but Mr. Obama lost some leverage for future battles.',
    u'multimedia': [{u'height': 388,
      u'legacy': {u'xlarge': u'images/2013/01/03/us/OBAMA/OBAMA-articleLarge-v2.jpg',
       u'xlargeheight': u'388',
       u'xlargewidth': u'600'},
      u'subtype': u'xlarge',
      u'type': u'image',
      u'url': u'images/2013/01/03/us/OBAMA/OBAMA-articleLarge-v2.jpg',
      u'width': 600},
     {u'height': 75,
      u'legacy': {u'thumbnail': u'images/2013/01/03/us/OBAMA/OBAMA-thumbStandard.jpg',
       u'thumbnailheight': u'75',
       u'thumbnailwidth': u'75'},
      u'subtype': u'thumbnail',
      u'type': u'image',
      u'url': u'images/2013/01/03/us/OBAMA/OBAMA-thumbStandard.jpg',
      u'width': 75}],
    u'news_desk': u'National',
    u'print_page': u'16',
    u'pub_date': u'2013-01-03T00:00:00Z',
    u'section_name': u'U.S.',
    u'snippet': u'The new fiscal deal holds small victories for President Obama and his Democratic allies in Congress that further their policy aims, but Mr. Obama lost some leverage for future battles.',
    u'source': u'The New York Times',
    u'subsection_name': None,
    u'type_of_material': u'An Analysis; News Analysis',
    u'web_url': u'http://www.nytimes.com/2013/01/03/us/politics/for-obama-fiscal-deal-is-a-victory-that-also-holds-risks.html',
    u'word_count': u'1111'},
   {u'_id': u'50e5f9c600315214fbb8247e',
    u'abstract': u'Some farming, health and GMO news to usher you into 2013.',
    u'blog': [],
    u'byline': {u'original': u'By MARK BITTMAN',
     u'person': [{u'firstname': u'Mark',
       u'lastname': u'BITTMAN',
       u'organization': u'',
       u'rank': 1,
       u'role': u'reported'}]},
    u'document_type': u'blogpost',
    u'headline': {u'kicker': u'Mark Bittman',
     u'main': u"A Month's Worth of Links for New Year"},
    u'keywords': [{u'name': u'persons',
      u'rank': u'1',
      u'value': u'Boxer, Barbara'},
     {u'name': u'persons', u'rank': u'2', u'value': u'Philpott, Tom'},
     {u'name': u'glocations', u'rank': u'1', u'value': u'Colorado River'},
     {u'name': u'glocations',
      u'rank': u'4',
      u'value': u'National Petroleum Reserve (Alaska)'},
     {u'name': u'glocations', u'rank': u'3', u'value': u'India'},
     {u'name': u'glocations', u'rank': u'2', u'value': u'Doha (Qatar)'},
     {u'name': u'glocations', u'rank': u'6', u'value': u'Wyoming'},
     {u'name': u'glocations', u'rank': u'5', u'value': u'Pakistan'},
     {u'name': u'organizations', u'rank': u'5', u'value': u'Kansas City Star'},
     {u'name': u'organizations', u'rank': u'6', u'value': u'Monsanto Company'},
     {u'name': u'organizations', u'rank': u'8', u'value': u'PepsiCo Inc'},
     {u'name': u'organizations',
      u'rank': u'7',
      u'value': u'Obesity (Journal)'},
     {u'name': u'organizations', u'rank': u'9', u'value': u'United Nations'},
     {u'name': u'organizations',
      u'rank': u'2',
      u'value': u'Dominos Pizza Inc'},
     {u'name': u'organizations',
      u'rank': u'3',
      u'value': u'Environmental Protection Agency'},
     {u'name': u'organizations',
      u'rank': u'4',
      u'value': u'Food and Drug Administration'},
     {u'name': u'organizations',
      u'rank': u'1',
      u'value': u'AquaBounty Technologies'},
     {u'name': u'organizations',
      u'rank': u'10',
      u'value': u'Wal-Mart Stores Inc'},
     {u'name': u'subject', u'rank': u'3', u'value': u'Diet and Nutrition'},
     {u'name': u'subject',
      u'rank': u'2',
      u'value': u'Aurora, Colo, Shooting (2012)'},
     {u'name': u'subject', u'rank': u'8', u'value': u'Soft Drinks'},
     {u'name': u'subject',
      u'rank': u'5',
      u'value': u'Newtown, Conn, Shooting (2012)'},
     {u'name': u'subject',
      u'rank': u'4',
      u'value': u'Greenhouse Gas Emissions'},
     {u'name': u'subject', u'rank': u'1', u'value': u'Antibiotics'},
     {u'name': u'subject', u'rank': u'7', u'value': u'Poverty'},
     {u'name': u'subject', u'rank': u'6', u'value': u'Obesity'}],
    u'lead_paragraph': None,
    u'multimedia': [],
    u'news_desk': None,
    u'print_page': None,
    u'pub_date': u'2013-01-03T16:29:38Z',
    u'section_name': u'Opinion',
    u'snippet': u'Some farming, health and GMO news to usher you into 2013.',
    u'source': u'The New York Times',
    u'subsection_name': None,
    u'type_of_material': u'Blog',
    u'web_url': u'http://bittman.blogs.nytimes.com/2013/01/03/a-months-worth-of-links-for-new-year/',
    u'word_count': u'1182'},
   {u'_id': u'5136a39dcf28d009de003a55',
    u'abstract': u"What sorts of social, economic and foreign relations issues must Venezuela's next president confront in order to guarantee a smooth transition?",
    u'blog': [],
    u'byline': None,
    u'document_type': u'blogpost',
    u'headline': {u'kicker': u'Room For Debate',
     u'main': u'The Future of Venezuela'},
    u'keywords': [{u'name': u'type_of_material',
      u'rank': u'1',
      u'value': u'News'}],
    u'lead_paragraph': u'',
    u'multimedia': [],
    u'news_desk': None,
    u'print_page': None,
    u'pub_date': u'2013-01-03T19:11:49Z',
    u'section_name': u'Opinion',
    u'snippet': u"What sorts of social, economic and foreign relations issues must Venezuela's next president confront in order to guarantee a smooth transition?",
    u'source': u'The New York Times',
    u'subsection_name': None,
    u'type_of_material': u'Blog',
    u'web_url': u'http://www.nytimes.com/roomfordebate/2013/01/03/venezuela-post-chavez/',
    u'word_count': u'3881'},
   {u'_id': u'50e6312d00315214fbb8251a',
    u'abstract': None,
    u'blog': [],
    u'byline': {u'organization': u'THE ASSOCIATED PRESS',
     u'original': u'By THE ASSOCIATED PRESS',
     u'person': []},
    u'document_type': u'article',
    u'headline': {u'main': u"Women's History Pioneer Gerda Lerner Dies at 92"},
    u'keywords': [],
    u'lead_paragraph': u"Gerda Lerner, a pioneer in the field of women's history and a founding member of the National Organization for Women, has died in Wisconsin, her son said Thursday. She was 92.",
    u'multimedia': [],
    u'news_desk': u'None',
    u'print_page': None,
    u'pub_date': u'2013-01-03T20:26:09Z',
    u'section_name': u'U.S.',
    u'snippet': u"Gerda Lerner, a pioneer in the field of women's history and a founding member of the National Organization for Women, has died in Wisconsin, her son said Thursday. She was 92.",
    u'source': u'AP',
    u'subsection_name': None,
    u'type_of_material': u'',
    u'web_url': u'http://www.nytimes.com/aponline/2013/01/03/us/ap-us-obit-lerner.html',
    u'word_count': 795},
   {u'_id': u'50e63a7f00315214fbb82536',
    u'abstract': None,
    u'blog': [],
    u'byline': {u'contributor': u'',
     u'original': u'By PAUL KRUGMAN',
     u'person': [{u'firstname': u'Paul',
       u'lastname': u'KRUGMAN',
       u'organization': u'',
       u'rank': 1,
       u'role': u'reported'}]},
    u'document_type': u'article',
    u'headline': {u'kicker': u'Op-Ed Columnist',
     u'main': u'Battles of the Budget',
     u'print_headline': u'Battles Of The Budget',
     u'seo': u'Battles of the Budget'},
    u'keywords': [{u'is_major': u'N',
      u'name': u'subject',
      u'rank': u'5',
      u'value': u'United States Politics and Government'},
     {u'is_major': u'N',
      u'name': u'organizations',
      u'rank': u'7',
      u'value': u'Senate'},
     {u'is_major': u'N',
      u'name': u'persons',
      u'rank': u'3',
      u'value': u'Obama, Barack'},
     {u'is_major': u'N',
      u'name': u'organizations',
      u'rank': u'6',
      u'value': u'House of Representatives'},
     {u'is_major': u'Y',
      u'name': u'subject',
      u'rank': u'1',
      u'value': u'Federal Budget (US)'},
     {u'is_major': u'N',
      u'name': u'subject',
      u'rank': u'2',
      u'value': u'National Debt (US)'},
     {u'is_major': u'N',
      u'name': u'subject',
      u'rank': u'4',
      u'value': u'Law and Legislation'}],
    u'lead_paragraph': u'Let\u2019s put the cliff deal, which was part of a larger class war, in perspective.',
    u'multimedia': [{u'height': 126,
      u'legacy': {u'wide': u'images/2010/09/16/opinion/Krugman_New/Krugman_New-thumbWide.jpg',
       u'wideheight': u'126',
       u'widewidth': u'190'},
      u'subtype': u'wide',
      u'type': u'image',
      u'url': u'images/2010/09/16/opinion/Krugman_New/Krugman_New-thumbWide.jpg',
      u'width': 190},
     {u'height': 900,
      u'legacy': {u'xlarge': u'images/2010/09/16/opinion/Krugman_New/Krugman_New-articleLarge.jpg',
       u'xlargeheight': u'900',
       u'xlargewidth': u'600'},
      u'subtype': u'xlarge',
      u'type': u'image',
      u'url': u'images/2010/09/16/opinion/Krugman_New/Krugman_New-articleLarge.jpg',
      u'width': 600},
     {u'height': 75,
      u'legacy': {u'thumbnail': u'images/2010/09/16/opinion/Krugman_New/Krugman_New-thumbStandard-v2.jpg',
       u'thumbnailheight': u'75',
       u'thumbnailwidth': u'75'},
      u'subtype': u'thumbnail',
      u'type': u'image',
      u'url': u'images/2010/09/16/opinion/Krugman_New/Krugman_New-thumbStandard-v2.jpg',
      u'width': 75}],
    u'news_desk': u'Editorial',
    u'print_page': u'23',
    u'pub_date': u'2013-01-04T00:00:00Z',
    u'section_name': u'Opinion',
    u'snippet': u'Let\u2019s put the cliff deal, which was part of a larger class war, in perspective.',
    u'source': u'The New York Times',
    u'subsection_name': None,
    u'type_of_material': u'Op-Ed',
    u'web_url': u'http://www.nytimes.com/2013/01/04/opinion/kurgman-battles-of-the-budget.html',
    u'word_count': u'816'},
   {u'_id': u'50e68ee400315214fbb825a6',
    u'abstract': None,
    u'blog': [],
    u'byline': {u'organization': u'THE ASSOCIATED PRESS',
     u'original': u'By THE ASSOCIATED PRESS',
     u'person': []},
    u'document_type': u'article',
    u'headline': {u'main': u"Women's History Pioneer Gerda Lerner Dies at 92"},
    u'keywords': [],
    u'lead_paragraph': u'Gerda Lerner spent her 18th birthday in a Nazi prison, sharing a cell with two gentile women arrested for political work who shared their food with the Jewish teenager because jailers restricted rations for Jews.',
    u'multimedia': [],
    u'news_desk': u'None',
    u'print_page': None,
    u'pub_date': u'2013-01-04T03:07:26Z',
    u'section_name': u'U.S.',
    u'snippet': u'Gerda Lerner spent her 18th birthday in a Nazi prison, sharing a cell with two gentile women arrested for political work who shared their food with the Jewish teenager because jailers restricted rations for Jews.',
    u'source': u'AP',
    u'subsection_name': None,
    u'type_of_material': u'',
    u'web_url': u'http://www.nytimes.com/aponline/2013/01/04/us/ap-us-obit-lerner.html',
    u'word_count': 835}],
  u'meta': {u'hits': 1255, u'offset': 0, u'time': 371}},
 u'status': u'OK'}
In [20]: 
json = r.json()
In [21]: 
json.keys()
Out[21]:
[u'status', u'response', u'copyright']
In [22]: 
json.keys()
Out[22]:
[u'status', u'response', u'copyright']
In [23]: 
json['status']
Out[23]:
u'OK'

Output from json['response'] ommitted because it was really long.

In [28]: 
json['response']['docs']
Out[28]:
10
In [27]: 
json['response']['meta']['hits']
Out[27]:
1255
In [29]: 
from time import sleep

base_url = 'http://api.nytimes.com/svc/search/v2/articlesearch.json?'

years = [2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013]

for year in years:
    year_string = str(year)
    payload = { 'q'         : 'inequality', 
                'begin_date':  year_string + '0101',
                'end_date'  :  year_string + '1231',
                'api-key'   :  my_times_api_key,
                'sort'      : 'newest' ,
                'page'      :  0}
    
    r = requests.get(base_url, params = payload)
    json = r.json()
    count  = json['response']['meta']['hits']
    print year, count
    sleep(.1)

2006 231
2007 298
2008 258
2009 235
2010 338
2011 674
2012 1022
2013 1255

Writing a CSV file

In [30]: 
import csv

outfile = open('inequality_annual.csv', 'w')
csv_writer = csv.writer(outfile)

header = ['year','article_count']
csv_writer.writerow(header)

base_url = 'http://api.nytimes.com/svc/search/v2/articlesearch.json?'

years = [2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013]

for year in years:
    year_string = str(year)
    payload = { 'q'         : 'inequality', 
                'begin_date':  year_string + '0101',
                'end_date'  :  year_string + '1231',
                'api-key'   :  my_times_api_key,
                'sort'      : 'newest' ,
                'page'      :  0}
    
    sleep(.1)
    
outfile.close()
In [31]: 
!cat inequality.csv

year,article_count

2006,231

2007,298

2008,258

2009,235

2010,338

2011,674

2012,1023

2013,1256
In [15]: 
from time import sleep

base_url = 'http://api.nytimes.com/svc/search/v2/articlesearch.json?'

payload = { 'q'         : 'inequality', 
            'api-key'   :  my_times_api_key,
            'sort'      : 'newest' ,
            'page'      :  0}
    
years = [2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013]
counts = []
for year in years:
 
    
    year_string = str(year)
    payload['begin_date'] = year_string + '0101'
    payload['end_date']   = year_string + '1231'
    r = requests.get(base_url, params = payload)
    json = r.json()
    count  = json['response']['meta']['hits']
    counts.append(count)
    sleep(.1)

This can be plotted in Python.

In [16]: 
%pylab inline
import matplotlib.pyplot as plt

plt.scatter(years,counts)
plt.ticklabel_format(useOffset=False)

Populating the interactive namespace from numpy and matplotlib

Your turn. Modify the script below to output a csv with the monthly total of inequality articles. For an extra challenge, add an additional column with the count of the number of "unemployment" articles.

In [39]: 
outfile = open('inequality_annual.csv', 'w')
csv_writer = csv.writer(outfile)


base_url = 'http://api.nytimes.com/svc/search/v2/articlesearch.json?'

def last_day_calculator(month):
    '''A little function that tells you the last day of the month.'''
    if month in [1,3,5,7,8,10,13]:
        last_day = '31'
    elif month in [4,6,9,11]:
        last_day = '30'
    else:
        last_day = '28'
    return last_day


#Loop through each year 2006-2013
for year in range(2006,2013):
    #turn the interger into string
    year_string = str(year)
    
    #Loop through each month
    for month in range(1,3):
        
        
        
        #figure out the last day of that month
        last_day_string = last_day_calculator(month)
            
        month_string =  "%02d" % month

        
        payload = { 'q'         : 'inequality',
                    'begin_date':  year_string + month_string + '01',
                    'end_date'  :  year_string + month_string + last_day_string,
                    'api-key'   :  my_times_api_key,
                    'sort'      : 'newest' ,
                    'page'      :  0}
        
         
        r = requests.get(base_url, params = payload)
        json = r.json()
        count  = json['response']['meta']['hits']
        csv_writer.writerow( [year_string, month_string, count] )

        
        sleep(.5)
        
outfile.close()
In [40]: 
!cat inequality_annual.csv

2006,01,17

2006,02,16

2007,01,23

2007,02,14

2008,01,19

2008,02,11

2009,01,14

2009,02,24

2010,01,20

2010,02,16

2011,01,43

2011,02,21

2012,01,127

2012,02,90
In []: