Web scraping with Python using Beautiful Soup

---

Load in packages

#Packages
#--Web scraping packages
from bs4 import BeautifulSoup
import requests
#Pandas/numpy for data manipulation
import pandas as pd
import numpy as np

Load URLs we want to scrape into an array

#load URLs we want to scrape into an array
BASE_URL = [
'http://www.reuters.com/finance/stocks/company-officers/GOOG.O',
'http://www.reuters.com/finance/stocks/company-officers/AMZN',
'http://www.reuters.com/finance/stocks/company-officers/AAPL'
]

Loop through our URLs, scrape table, pass information to array

#loading empty array for board members
board_members = []
#Loop through our URLs we loaded above
for b in BASE_URL:
    html = requests.get(b).text
    soup = BeautifulSoup(html, "html.parser")
    #identify table we want to scrape
    officer_table = soup.find('table', {"class" : "dataTable"})
    #try clause to skip any companies with missing/empty board member tables
    try:
        #loop through table, grab each of the 4 columns shown (try one of the links yourself to see the layout)
        for row in officer_table.find_all('tr'):
            cols = row.find_all('td')
            if len(cols) == 4:
               board_members.append((b, cols[0].text.strip(), cols[1].text.strip(), cols[2].text.strip(), cols[3].text.strip()))
    except: pass  

Create new array, check length to ensure things pulled in correctly

#convert output to new array, check length
board_array = np.asarray(board_members)
len(board_array)

49

Convert new array to dataframe

#convert new array to dataframe
df = pd.DataFrame(board_array)

Rename columns, preview output

#rename columns, check output
df.columns = ['URL', 'Name', 'Age','Year_Joined', 'Title']
df.head(10)

	URL	Name	Age	Year_Joined	Title
0	http://www.reuters.com/finance/stocks/company-...	Eric Schmidt	61	2015	Executive Chairman of the Board of Director
1	http://www.reuters.com/finance/stocks/company-...	Sergey Brin	43	2015	President, Director
2	http://www.reuters.com/finance/stocks/company-...	Lawrence Page	44	2015	Chief Executive Officer, Director
3	http://www.reuters.com/finance/stocks/company-...	Ruth Porat	59	2015	Chief Financial Officer, Senior Vice President
4	http://www.reuters.com/finance/stocks/company-...	Sundar Pichai	45	2017	Director, Chief Executive Officer, Google Inc.
5	http://www.reuters.com/finance/stocks/company-...	David Drummond	54	2015	Senior Vice President - Corporate Development,...
6	http://www.reuters.com/finance/stocks/company-...	John Hennessy	64	2007	Lead Independent Director
7	http://www.reuters.com/finance/stocks/company-...	Diane Greene	61	2015	Director
8	http://www.reuters.com/finance/stocks/company-...	L. John Doerr	65	2016	Independent Director
9	http://www.reuters.com/finance/stocks/company-...	Roger Ferguson	65	2016	Independent Director

Export data to CSV

#export data
df.to_csv('/Users/yourname/desktop/board_members.csv')

That's it! If you're interested in seeing how I used this data check out my visualization on the interconnectedness of companies through shared board members here.

Ace your next data science interview

Get better at data science interviews by solving a few questions per week

Learn more

Find a bug? Submit a suggested change on Github, or message me on Twitter.