From Facebook, Amazon, Microsoft, Apple, and more!
Machine learning is an interdisciplinary field that focuses on extracting actionable insights from data in any format - to enhance decision-making skills or predicting future trends. Since it covers a lot of sub-fields that are pretty detailed as well, it's amongst the hardest field to master of this age.
Consequently, you can imagine how stressful machine learning interviews are. So, today we’ll be going through the most frequently asked questions that you’re likely to get asked when appearing for an ML-based job.
I'll divide the article into segments, so you can conveniently go through different concepts involved in machine learning. Here's how it's split up:
<ul>
<li>Machine Learning Fundamentals</li>
<li>Data Analysis &#x26; Visualization</li>
<li>Statistics &#x26; Probability</li>
</ul>
So, let's start without any further ado!
<h2>Machine Learning Fundamentals</h2>
<h3>Q: What steps to take when dealing with unbalanced binary classification?</h3>
There are multiple ways we use to deal with this:
<ul>
<li>We can start considering other evaluation metrics such as precision and recall. Using accuracy in unbalanced classes is never a great idea, and other metrics could provide an appropriate overview even if the classes are imbalanced.</li>
<li>We can increase the cost factor associated with misclassifying the minority class. This penalty will result in higher accuracy when it comes to the prediction of the minority class.</li>
<li>Lastly, more sampling could be done from the minority as compared to the majority class. It will automatically reduce the imbalance.</li>
</ul>
<h3>Q: What are regularization methods? Provide an example of the regularization method.</h3>
Regularization methods are used to reduce the overfitting of a model and discourage them from learning a very complex model, which is not easy to generalize on new testing data.
L2 regularization is a commonly used regularization method, also called ridge regression.
<h3>Q: Differentiate between supervised and unsupervised learning using appropriate examples.</h3>
<h4>Supervised learning</h4>
It's referred to the training of machine learning models where you have a labeled dataset and know what classes the data instances belong to while training the model. This way, the machine learns the data according to the labels.
Example: We can show some dogs to a child and tell him that they're called dogs. Based on this, the child can learn how a dog looks; so, the next time he sees a dog, he can identify it.
<h4>Unsupervised learning</h4>
This technique involves training machine learning models when the data is not label. The model figures out the labels of the data by itself. So basically, the model is not being supervised. The technique involves grouping similar data with the assumption that it belongs to the same class.
Example: You have data about customer data but not the labels. You plot the data based on some features of customers such as their preferred products, budget, age, etc. Based on all these factors, you group similar customers and assign them the same class.
<h3>Q: What are the three stages of building a machine learning model?</h3>
The three stages of building machine learning models are:
<h4>Model Building</h4>
Choosing the dataset and algorithm to use to train a machine learning model. And processes like Preprocessing, EDA, Hyperparameter tuning, etc.
<h4>Model Testing</h4>
Using the test dataset to calculate different evaluation metrics and checking if the model performs according to the expectations.
<h4>Model Deployment</h4>
Deploying the model upon the suitable platform or service.
<h3>Q: Differentiate classification and regression. How will you choose what technique to use for a dataset?</h3>
Classification and regression are separated by the nature of the target variable. Classification is used when the target variable is categorical such as predicting colors or genders, while regression is used the target variable is continuous, for example, predicting house prices or scores.
So, all I have to do is check the type of the target variable to check whether I’ll use a classification algorithm or a regression algorithm.
<h3>Q: What's the difference between a generative and discriminative model?</h3>
The models that learn the categories of data and train upon them are referred to as generative models. On the other hand, a discriminative model does the learning by only learning the distinction between the data categories. However, discriminative models usually outperform the former.
<h3>Q: A given machine learning model has a high training data accuracy but a much lower accuracy when it comes to testing data. Why could be the reason? Explain what’s happening.</h3>
It's because the model is likely to be overfitted. It means that the model has 'learned' the training data far too well. Even the minor fluctuations in the data are considered by the model so it's not generalized and hence struggles to label the new data correctly, even though it was learning the training data very accurately
<h3>Q: What do you understand by the term ‘Naïve’ in the Naïve Bayes algorithm?</h3>
The Naïve Bayes algorithm is referred to as 'naïve' because it makes a naïve assumption that all the features available are independent of each other, but this is hardly the case in real-world problems since most input features are related to one another.
<h3>Q: Do you think 100 small decision trees are better than a single larger one? If so, why?</h3>
Hint: This is just another way of asking if a random forest model is better than a decision tree.
Well, 100 small decision trees are definitely better than a single larger one. The ensemble method takes away many weak decision trees, which makes it a strong learner. A large number of trees are more robust, less prone to overfitting, and generally more accurate as well, since the average of outputs is employed.
<h2>Data Analysis &#x26; Visualization</h2>
<h3>Q: How do you handle datasets that have missing or corrupted values?</h3>
We can either remove the values if there aren't many. Or replace them with measures such as the mean or median. Pandas library provides various built-in functions to find missing and treat missing values like isnull() and fillna().
<h3>Q: What are some of the best visualization tools or libraries one can conveniently use for EDA?</h3>
Choosing the best visualization tool or library depends upon the specific task or dataset. Some of the most used ones are Seaborn, matplotlib, or ggplot if R is used. For the tools, Tableau and Plot.ly are amongst the best.
<h3>Q: What’s the difference between a histogram and a box plot?</h3>
A histogram is used to depict the frequency distributions of variables that help us understand the shape of the distribution. However, a box plot cannot be used to estimate the shape of a distribution. Instead, it shows different measures about the data distribution, such as the range that the data is split upon, the quartiles, mean, outliers, or so forth.
<h3>Q: Differentiate Bivariate and Univariate analysis? Explain with examples.</h3>
The univariate analysis involves statistical analysis involving only a single variable. Techniques involve average, sum, variance, and so on. However, bivariate analysis is when two variables are involved, and we're finding their relationship. Such as scatter plots, correlation, and so on.
<h3>Q: Provide some techniques that can be used to visualize data that has more than three dimensions.</h3>
Various techniques are available to cope with data having more than three dimensions, such as using plot facets, different colors, and depths. Also, dimensionality reduction can be applied if the dimensions are too much to visualize. It reduces the dimensions while keeping its integrity.
<h3>Q: What is data transformation? Is it necessary?</h3>
Data transformation is known as converting the data from an available format to the desired format. It's a crucial technique that's required before the data is stored in a data warehouse. Raw data is available in various formats; we must transform it according to the requirements of the destination to ensure consistency.
<h2>Statistics &#x26; Probability</h2>
<h3>Q: What’s the difference between probability and likelihood?</h3>
While probability is simply calculating the odds of an event to happen, likelihood, on the other hand, is aimed at maximizing the chances of some event happening.
<h3>Q: How do you perform hypothesis testing?</h3>
There are 5 five main steps in hypothesis testing given below:
<ol>
<li>Classify your hypothesis as null or alternative.</li>
<li>Collect the data required to test the hypothesis.</li>
<li>Perform the statistical test to get the results.</li>
<li>Calculate if the null hypothesis is supported or not.</li>
<li>Present your findings corresponding to your hypothesis setting.</li>
</ol>
<h3>Q: What is known as selection bias?</h3>
Selection bias occurs during the sampling of data from a given dataset that doesn't show the true nature of the distribution of data. The bias doesn't occur by chance, and there is a systematic approach behind it which is why it's referred to as 'active'.
<h3>Q: Differentiate between a Type I and Type II error.</h3>
Type I errors occur when the null hypothesis is true - but rejected. In contrast, a type II error occurs when the null hypothesis is false but fails to get rejected. Further details on both types of errors can be found <a href="https://www.datasciencecentral.com/understanding-type-i-and-type-ii-errors/">here.</a>
<h3>Q: What meant by binomial probability distribution? Explain.</h3>
The binomial probability distribution is a kind of distribution that comprises the probabilities for the possible numbers of successes on a total of N trials - that are completely independent and have the probability of 'pi'.
<h2>Conclusion</h2>
That's it for today! I hope you found the questions helpful. However, that's not all. As I have already mentioned, machine learning is a vast field, and you need to keep learning to become better at it since it's still in its evolving phase.

Wondering what machine learning is and what it can do for your business? Check out our top 20 machine learning questions for starters!

Cracking Amazon's Data Science Interview

When you're interviewing for a data scientist or data analyst role, it's highly likely you'll encounter SQL questions in your interview. Additionally, it's likely one or more of those SQL questions will require a window function to be solved. Window functions are a core concept in intermediate/advanced SQL, and mastering them will put you one step closer to landing your analytics role. In this post we'll give a quick overview of what window functions are, and then we'll dive into an example interview question along with our solution.
What are SQL window functions?
A window function defines a frame or ‘window’ of rows with a given length around the current row, and performs a calculation across the set of data in the window.
If you’re struggling with the definition above, don’t worry, it should become more clear as you put it into practice. Below we'll step through an example window function interview question that uses a simple aggregation.
Example question - RANK() in SQL window function
Suppose you're given the following tables called 'orders' and 'order_info'. The table 'orders' shows revenue values for unique orders along with the associated channel ('online' or 'in_store') while the table 'order_info' shows the order's ID along with its location.
Table: orders
<table>
<thead>
<tr>
<th>order_id</th>
<th>channel</th>
<th>date</th>
<th>month</th>
<th>revenue</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>online</td>
<td>2020-09-01 00:00:00</td>
<td>9</td>
<td>100</td>
</tr>
<tr>
<td>2</td>
<td>online</td>
<td>2020-09-03 00:00:00</td>
<td>9</td>
<td>125</td>
</tr>
<tr>
<td>3</td>
<td>in_store</td>
<td>2020-10-11 00:00:00</td>
<td>10</td>
<td>208</td>
</tr>
<tr>
<td>4</td>
<td>in_store</td>
<td>2020-08-21 00:00:00</td>
<td>8</td>
<td>80</td>
</tr>
<tr>
<td>5</td>
<td>online</td>
<td>2020-08-13 00:00:00</td>
<td>8</td>
<td>200</td>
</tr>
<tr>
<td>6</td>
<td>online</td>
<td>2020-08-16 00:00:00</td>
<td>8</td>
<td>210</td>
</tr>
<tr>
<td>7</td>
<td>in_store</td>
<td>2020-08-16 00:00:00</td>
<td>8</td>
<td>205</td>
</tr>
<tr>
<td>8</td>
<td>online</td>
<td>2020-10-11 00:00:00</td>
<td>10</td>
<td>215</td>
</tr>
<tr>
<td>9</td>
<td>online</td>
<td>2020-08-16 00:00:00</td>
<td>8</td>
<td>203</td>
</tr>
<tr>
<td>10</td>
<td>in_store</td>
<td>2020-09-01 00:00:00</td>
<td>9</td>
<td>400</td>
</tr>
<tr>
<td>11</td>
<td>online</td>
<td>2020-08-01 00:00:00</td>
<td>8</td>
<td>107</td>
</tr>
</tbody>
</table>
Table: order_info
<table>
<thead>
<tr>
<th>order_id</th>
<th>location</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>NYC</td>
</tr>
<tr>
<td>2</td>
<td>NYC</td>
</tr>
<tr>
<td>3</td>
<td>LAX</td>
</tr>
<tr>
<td>4</td>
<td>LAX</td>
</tr>
<tr>
<td>5</td>
<td>SEA</td>
</tr>
<tr>
<td>6</td>
<td>AUS</td>
</tr>
<tr>
<td>7</td>
<td>LON</td>
</tr>
<tr>
<td>8</td>
<td>LAX</td>
</tr>
<tr>
<td>9</td>
<td>BLD</td>
</tr>
<tr>
<td>10</td>
<td>SEA</td>
</tr>
<tr>
<td>11</td>
<td>AUS</td>
</tr>
</tbody>
</table>
Using these tables, write a SQL query to return the top 3 'online' orders and their associated locations based on revenue generated. You can assume that each order has a unique revenue value, but you should be able to highlight the implications of ties in revenue values and how you would handle that.
Click <a href="https://www.db-fiddle.com/f/pNnzDC1gDDNYEMcD6LcYAk/1">here</a> to view these tables in an interactive SQL fiddle.
Solution:
Click <a href="https://www.db-fiddle.com/f/cKPGJwFHWrE2PFwu4YMK6m/6">here</a> to view this solution in an interactive SQL fiddle.
<pre><code># SQL query to return the top 
# 3 'online' orders and their 
# associated locations based on revenue generated

# Pulling from our subquery below, 
# filtering on just the top 3 records

SELECT * FROM ( 
 # Subquery to pull online orders and
 # add ranking next to each based on 
 # revenue generated
 SELECT 
 o.order_id as order_id,
 o.revenue as revenue,
 oi.location as location,
 # Add a ranking in descending order, 
 # highest revenue first
 RANK() OVER (ORDER BY sum(o.revenue) DESC) as my_rank
 FROM orders o
 # Join in order info (for location) 
 # on order_id
 LEFT JOIN order_info oi
 ON o.order_id = oi.order_id
 # Filter where the channel is online
 WHERE o.channel = "online"
 group by 1,2,3) stg
WHERE my_rank</code></pre>

Learn how to do SQL joins. Provided by InterviewQs, a mailing list for coding and data interview problems.

SQL Window Function Example Interview Question - RANK()

When you're interviewing for a data scientist or data analyst role, it's highly likely you'll encounter SQL questions in your interview. Additionally, it's likely one or more of those SQL questions will require a window function to be solved. Window functions are a core concept in intermediate/advanced SQL, and mastering them will put you one step closer to landing your analytics role. In this post we'll give a quick overview of what window functions are, and then we'll dive into an example interview question along with our solution.
What are SQL window functions?
A window function defines a frame or ‘window’ of rows with a given length around the current row, and performs a calculation across the set of data in the window.
If you’re struggling with the definition above, don’t worry, it should become more clear as you put it into practice. Below we'll step through an example window function interview question that uses a simple aggregation.
Example question - LAG and LEAD in SQL window function
You are given the following table containing historical employee salaries for company XYZ:
Table: EmployeeSalaries
<table>
<thead>
<tr>
<th>employee_ID</th>
<th>salary</th>
<th>year</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>80000</td>
<td>2020</td>
</tr>
<tr>
<td>1</td>
<td>70000</td>
<td>2019</td>
</tr>
<tr>
<td>1</td>
<td>60000</td>
<td>2018</td>
</tr>
<tr>
<td>2</td>
<td>65000</td>
<td>2020</td>
</tr>
<tr>
<td>2</td>
<td>65000</td>
<td>2019</td>
</tr>
<tr>
<td>2</td>
<td>60000</td>
<td>2018</td>
</tr>
<tr>
<td>3</td>
<td>65000</td>
<td>2019</td>
</tr>
<tr>
<td>3</td>
<td>60000</td>
<td>2018</td>
</tr>
</tbody>
</table>
Given the above table, can you write a SQL query to return the employees who have received at least 3 year over year raises based on the table's data?
<h3>Solution:</h3>
<pre><code>SELECT 
 a.employee_ID as employee_ID
FROM 
 # First, we'll build a subquery with employeeID, current salary, and prior year salary
 (SELECT 
 employee_ID, 
 salary, 
 # Using window function lead, we can grab the prior year's salary
 # Note we could alternatively use LAG and sort our year in ascending order to achieve the same result
 LEAD(salary) OVER (PARTITION BY employee_ID ORDER BY year DESC) as previous_year_sal 
 FROM Employee ) a

# Only pull records where the current salary is > the prior year salary
WHERE a.salary > a.previous_year_sal
GROUP BY employee_ID
# Since we want 3 consecutive years of raises, 
# there should be at least 2 records (e.g. yr 1 -> yr 2, yr 2 -> yr 3)
HAVING count(*) = 2</code></pre>

SQL Window Function Example Interview Question - LEAD/LAG

When you're interviewing for a data scientist or data analyst role, it's highly likely you'll encounter SQL questions in your interview. Additionally, it's likely one or more of those SQL questions will require a window function to be solved. Window functions are a core concept in intermediate/advanced SQL, and mastering them will put you one step closer to landing your analytics role. In this post we'll give a quick overview of what window functions are, and then we'll dive into an example interview question along with our solution.
What are SQL window functions?
A window function defines a frame or ‘window’ of rows with a given length around the current row, and performs a calculation across the set of data in the window.
If you’re struggling with the definition above, don’t worry, it should become more clear as you put it into practice. Below we'll step through an example window function interview question that uses a simple aggregation.
Example question - Aggregation in SQL window function
Supose you're given the following table that shows spend by keyword, advertiser, and unique ad ID:
Table: keyword_summary
<table>
<thead>
<tr>
<th>keyword</th>
<th>ad_id</th>
<th>advertiser_id</th>
<th>total_spend</th>
</tr>
</thead>
<tbody>
<tr>
<td>bicycle</td>
<td>1243213</td>
<td>1234</td>
<td>95</td>
</tr>
<tr>
<td>bike</td>
<td>1243213</td>
<td>1234</td>
<td>71</td>
</tr>
<tr>
<td>bike tires</td>
<td>1243213</td>
<td>1234</td>
<td>22</td>
</tr>
<tr>
<td>bike brakes</td>
<td>1243213</td>
<td>1234</td>
<td>95</td>
</tr>
<tr>
<td>bike accessories</td>
<td>1243213</td>
<td>1234</td>
<td>28</td>
</tr>
<tr>
<td>...</td>
<td>...</td>
<td>...</td>
<td>...</td>
</tr>
</tbody>
</table>
Using the table above, write a SQL query that returns each keyword along with the advertiser_id of the top spending advertiser, and total spend on the keyword (agnostic of advertiser). Sort the results in descending order by the total spend on the keyword.
You can view/query the data in an interactive SQL fiddle <a href="https://www.db-fiddle.com/f/7S9t3MwFBXBVQ1qZ89FDN9/1"> here</a>.
Solution
<a href="https://www.db-fiddle.com/f/o4Ze4uK1ZAEZ9hYxyzPjVB/2">Click here</a> to view this solution in an interactive SQL fiddle.
<pre><code>## First we separate the base query to get (1) the total spend on the 
## keywords and (2) the advertiser spend ranking by keyword. 
## Next, we get the total spend by keyword and join to the subquery 
## that will provide the top spend by advertiser
## Last, we sort by the total spend on the keyword

WITH 
 base AS (
 # separating the base sub-query to get the baseline calculations that we can 
 # use in our subqueries
 SELECT 
 keyword,
 advertiser_id,
 ## calculating the total spend on each keyword
 SUM(total_spend) OVER(PARTITION BY keyword) AS spend_keyword,
 ## ranking the advertisers based on how much they spend on a given keyword
 RANK() OVER (PARTITION BY keyword ORDER BY total_spend DESC) as advertiser_rank
 FROM keyword_summary
)
SELECT 
 q1.keyword,
 q2.advertiser_id,
 SUM(q1.spend_keyword) AS spend_keyword
# we need to query the "base" table 2 times, 
# 1. to get the total spend on each keyword
# 2. to get the top advertiser spend on each keyword
FROM base AS q1
LEFT JOIN (
 SELECT 
 keyword,
 advertiser_id
 FROM base
 WHERE advertiser_rank = 1
 ) AS q2
ON q1.keyword = q2.keyword
GROUP BY 
 q1.keyword,
 q2.advertiser_id
# sorting the results by the total spend
ORDER BY spend_keyword DESC</code></pre>
Output:
<table>
<thead>
<tr>
<th>keyword</th>
<th>advertiser_id</th>
<th>spend_keyword</th>
</tr>
</thead>
<tbody>
<tr>
<td>bike</td>
<td>21781</td>
<td>3608</td>
</tr>
<tr>
<td>bicycle</td>
<td>1234</td>
<td>2520</td>
</tr>
<tr>
<td>bike brakes</td>
<td>1234</td>
<td>266</td>
</tr>
<tr>
<td>bike accessories</td>
<td>3829</td>
<td>196</td>
</tr>
<tr>
<td>bike tires</td>
<td>3829</td>
<td>170</td>
</tr>
<tr>
<td>mountain bike</td>
<td>3829</td>
<td>59</td>
</tr>
</tbody>
</table>

SQL Window Function Example Interview Question - Aggregation

InterviewQs' goal is to get you prepared for the quantitative side of a Data Science interview. However when you are interviewing for a DS role, you’re most likely going to be asked both quantitative and qualitative questions. The goal of this post is to take you through:
<ol>
<li>Examples of qualitative questions</li>
<li>How you should answer these types of questions (STAR Method)</li>
<li>How you can prepare for this part of your interview (practice, practice, practice)</li>
</ol>
1. Examples of qualitative interview questions [more listed at bottom of post]
<ul>
<li>Tell me a time where you failed at a task</li>
<li>Tell me about a time where you influenced a decision with data</li>
<li>Tell me about a time when you disagreed with a coworker</li>
<li>Tell me about a time when you had to lead without authority</li>
<li>Tell me about a time you had to explain a technical analysis to a non-technical audience</li>
</ul>
2. STAR Method
If you are asked a question when you are asked to tell about a past experience, you should frame the question using the STAR methodology. STAR is an acronym for (Situation, Task, Action, and Result).
Situation: What is the context to your story? Set the scene for your interviewer. Be sure to include who, what, when, where, and how of the situation. Provide enough context early on so that the interviewer will be able to follow your story
Task: Describe the task at hand and your involvement, keep it specific and concise and be sure to highlight the challenges with the task at hand.
Action: Describe what you did to accomplish the task. Specify skills, tools, characteristics, behaviors, and any conflicts.
Result: Share the outcome of the situation and specifically how you contributed to the outcome.
You'll most likely get follow up questions during the interview to clarify points you’ve made, and that’s fine! Answer the question as clearly as possible.
3. Preparation for behavioral questions
The best way to prepare for behavioral questions is to have curated a list of situations you’ve been in the past. I typically write out 10-12 scenarios I’ve experienced and clearly label the Situation, Task, Action, and Result. I rehearse these with another person and ensure that my language is clear, my scenario makes sense, and clearly highlights the impact of my contribution to the situation/task.
Find some additional behavioral questions, and ensure that the scenarios you’ve outlined capture a majority of the questions in your list. It’s okay if one of your scenarios covers 3-4 questions, as long as you’re prepared to answer the question, you should be fine!
Bonus example behavioral questions:
• Describe a situation in which you were able to use persuasion to successfully convince someone to see things your way.
• Describe a time when you were faced with a stressful situation that demonstrated your coping skills.
• Give me a specific example of a time when you used good judgment and logic in solving a problem.
• Give me an example of a time when you set a goal and were able to meet or achieve it.
• Tell me about a time when you had to use your presentation skills to influence someone's opinion.
• Give me a specific example of a time when you had to conform to a policy with which you did not agree.

How to crack behaviorial interview questions using the STAR Method

Overview of Nested Queries
A SQL nested query, also known as a subquery or subselect, is a SELECT query method embedded within another query. It’s an efficient way to get the desired output that may require multiple query steps or to handle queries that are interdependent.
When to use Nested Subqueries
You can use nested subqueries in a couple different ways:
<ul>
<li>You can query over subqueries (e.g. selecting from a subquery)</li>
<li>You can replace 1D arrays (e.g. a typical list of items) and single field joins with a single subquery in the WHERE or HAVING clause</li>
</ul>
In order to use a subquery, there are a couple of syntactical rules that need to be followed:
<ul>
<li>The subquery needs to be enclosed within parentheses</li>
<li>Depending on the SQL engine you are using, you might need to alias a given subquery</li>
<li>If using in a WHERE or HAVING clause, the SELECT statement of an subquery can only return the single field being evaluated</li>
</ul>
Example SQL interview question using a nested subquery
Suppose you are given the following table showing company sales:
Table: sales_info
<table>
<thead>
<tr>
<th>date</th>
<th>sale_id</th>
<th>sale_usd</th>
</tr>
</thead>
<tbody>
<tr>
<td>2020-01-05</td>
<td>1111</td>
<td>93695</td>
</tr>
<tr>
<td>2020-01-07</td>
<td>1112</td>
<td>879617</td>
</tr>
<tr>
<td>2020-01-07</td>
<td>1113</td>
<td>752878</td>
</tr>
<tr>
<td>...</td>
<td>...</td>
<td>...</td>
</tr>
</tbody>
</table>
Calculate the cumulative percent of total sales on a given day. The output of the table should look like the example table below.
<table>
<thead>
<tr>
<th>date</th>
<th>pct_total_sales</th>
</tr>
</thead>
<tbody>
<tr>
<td>2020-01-05</td>
<td>X%</td>
</tr>
<tr>
<td>2020-01-07</td>
<td>Y%</td>
</tr>
</tbody>
</table>
You can work through this example using the interactive SQL fiddle <a href="https://www.db-fiddle.com/f/tG8kzU91Nt1TQpTGpNY3VF/0">here</a>.
Before we start writing SQL, we'll break the question into steps:
<ul>
<li>Calculate the daily sales total</li>
<li>Calculate the cumulative sum of the daily sales total and total sales for all days</li>
<li>Divide the daily total sales by the cumulative sum</li>
</ul>
1. Calculate the daily sales total
First, we'll write the base query, which will become a subquery in the next step. The below query calculates the daily sales total and you can interact with the query with this <a href="https://www.db-fiddle.com/f/tG8kzU91Nt1TQpTGpNY3VF/2">SQL fiddle</a>.
<pre><code>SELECT 
 #we'll need to sum sale_usd by the date 
 date, 
 sum(sale_usd) as total_usd
FROM sales_info
#because we're aggregating sale_usd by date, we need 
#need to group by date
GROUP BY date</code></pre>
2. Calculate the cumulative sum of the daily sales total and total sales for all days
The query below calculates the cumulative sum of the daily sales total and total sales for all days. You can interact with the query below with this <a href="https://www.db-fiddle.com/f/tG8kzU91Nt1TQpTGpNY3VF/8">SQL fiddle</a>.
You'll notice in this step that we build a query around the previous query from the step above. In order to successfully build the subquery, we needed to enclose the subquery in parentheses and ensure that we alias the table (since we're using MySQL, note that this isn't a requirement for some SQL variants). We also need to ensure that we include all fields needed for the outer query (e.g. date and total sales).
<pre><code>SELECT 
#in this query, we do not group by since we're using a windowing function
 date, 
 SUM(total_usd) OVER (
 ORDER BY date ASC rows 
 BETWEEN unbounded preceding and current row) 
 as cum_total, # this is a windowing function to 
 # calculate the cumulative sum
 SUM(total_usd) OVER () as total # this is a windowing function to 
 # calculate the total
FROM(
 SELECT 
 #we'll need to sum sale_usd by the date 
 date, 
 sum(sale_usd) as total_usd
 FROM sales_info
 #because we're aggregating sale_usd by date, we need 
 #need to group by date
 GROUP BY date
) as q1 #we create an alias for this table as required by MySQL</code></pre>
3. Divide the cumulative total sales by the cumulative sum
The last step is to divide the <code>cum_total</code> by the total. We can accomplish this in the same step as above (just dividing the two windowing functions), or we can build a subquery over the previous step. The query below uses another subquery resulting in the end query having two nested subqueries. You can interact with the query below with this <a href="https://www.db-fiddle.com/f/tG8kzU91Nt1TQpTGpNY3VF/10">SQL fiddle</a>.
<pre><code>SELECT 
 date,
 100 * cum_total / total as
FROM(
 #in this query, we do not group by 
 #since we're using a windowing function
 SELECT
 date, 
 SUM(total_usd) OVER (ORDER BY date ASC 
 rows BETWEEN unbounded preceding and current row)
 as cum_total, # this is a windowing function to 
 # calculate the cumulative sum
 SUM(total_usd) OVER () as total # this is a windowing function
 # to calculate the total
 FROM(
 SELECT 
 #we'll need to sum sale_usd by the date 
 date, 
 sum(sale_usd) as total_usd
 FROM sales_info
 #because we're aggregating sale_usd by date, we need 
 #need to group by date
 GROUP BY date
 ) as q1 #we create an alias for this table as required by MySQL
) as q2</code></pre>

A curated list of 70 data science and analyst interview questions.

SQL Nested Subquery Example Interview Question

<h2>Intro</h2>
Like many of the tech giants, Google leverages data to the fullest extent in all parts of its business. This provides many ongoing opportunities for those looking to get hired into an Analyst role at Google! This blog post will take you through the role, required skills, the interview process, a few example questions, and some tips to help you start preparing for your interview.
The Role
Analysts are embedded throughout Google, meaning there are many opportunities and analytics roles available. Below are just a few types of analysts that Google hires for:
<ul>
<li>Financial Analysts - Folks in this role ensure that Google is spending and making money for the business according to plan. Financial Analysts can be aligned to a product area (think Youtube, Google Playstore, Google Cloud) or can be a part of the core finance function.</li>
<li>People Operations Analysts - These analysts help monitor and track activity across HR, spanning from Googler’s compensation to analyzing work habits to making sure Google has the right amount of recruiters in order to hit their hiring targets across the business.</li>
<li>Product Analysts - These analysts work on understanding how users interact with the different Google products and provide analysis to help improve the products, working closely with Engineering.</li>
<li>Trust and Safety Analyst - These analysts help ensure Google's content-focused products (such as Ads) have content that are compliant with Google’s policies. These teams work heavily with Engineering to build/improve machine learning models centered around flagging content.</li>
</ul>
Required Skills
An average Google analyst typically has 3-4 years of experience in a quantitative role. However, there are definitely instances when employees have less or more than that. Other qualifications for Google Analysts include:
<ul>
<li>Bachelor’s or higher in a technical program such as Mathematics, Statistics, Computer Science, Physics, Engineering, Operations Research, and Economics</li>
<li>Experience with SQL</li>
<li>Experience with relational databases and business intelligence tools</li>
<li>Basic experience with scripting languages (e.g. Python, Java)</li>
<li>Experience with stats software such as Numpy/Pandas/Scikit Learn or R (Colab Notebooks are a great way to practice this skill)</li>
<li>Experience working with cross-functional teams</li>
</ul>
These skills vary from role to role, so be sure to read into the specifics for a particular role.
Interview Process
The typical interview process at Google goes as follows:
<ul>
<li>Initial screening by a recruiter (~30 minutes). This is more of a formality to ensure the candidate fits the criteria for the role. The recruiter will ask about past experiences, your interests, and will provide you more details about the role.</li>
<li>Initial phone interview with a member of the current team (45 minutes). This interview will cover past experiences, but with more of a focus on technical skills and problem solving ability. It’s important to use stories/past experiences that highlight your technical expertise here.</li>
<li>Potential case study (24-48 hours to complete). Depending on the role you are applying for, you might have to do a home case study. Typically, this case study will be reviewed by a member of the team; provided you pass this step, you may then have to discuss it during your onsite interviews.</li>
<li>Onsite interview. Here you'll run through a slate of interviews with the current team (it's likely one of the interviews will be with the hiring manager) and cross-functional team members. These interviews will take about 1/2 a day - interviewing with 3-4 people for about 45 minutes each. These interviews can be structured in many different ways depending on what team you’re interviewing for, but generally onsite interviews focus on 4 main areas:
<ul>
<li>General Cognitive Ability: This area is graded based on general problem solving and critical thinking, along with the ability to communicate your line of thinking.</li>
<li>Leadership: This will focus on how you’re able to help lead a group to a positive outcome, either with or without direct leadership authority.</li>
<li>Role-related knowledge: Here you'll have a more technical interview, likely covering SQL along with your technical approach to hypothetical problems.</li>
<li>Googleyness: “Googleyness” is a set of qualities like fun, intellectual humility, conscientiousness, and a record of having done interesting things. Be yourself and if you have any side projects outside of your core “experience” this will help you do well in this area.</li>
</ul>
</li>
</ul>
Example Interview Questions
<ul>
<li>If you were the owner of a flower shop, how would you target your customers online? (Who? When? Where? How?)</li>
<li>How would you measure customer check out rates?</li>
<li>Name a Google product you use. What would you change about it and how would you improve it?</li>
<li>Give me an example of a project you’ve led. What was the result? Which difficulties did you meet?</li>
<li>How would you measure how much a given user liked a video?</li>
<li>Explain confidence intervals to someone with no statistics background.</li>
</ul>
Tips to Help You Prepare for Your Interview
<ul>
<li>
Brush up on your SQL along with any other technical skills listed in the requirements.
</li>
<li>
Come prepared with 6-7 stories based on past experience that cover the following:
<ul>
<li>A time you disagreed with a team member and how you dealt with it.</li>
<li>A time a client had unreasonable demands and how you responded.</li>
<li>A time you took on a leadership role (formally or informally).</li>
<li>A time you took a risk and what made it notable.</li>
<li>A time you worked on a particularly difficult project and what made the project difficult.</li>
<li>An example of a time you were leading a group and it wasn’t going well and how you turned it around.</li>
</ul>
</li>
<li>
Ensure that you have spare copies of your resume.
</li>
<li>
Use the <a href="/blog/star_method">STAR Method</a> when drawing from past experience.
</li>
<li>
Relax and be yourself! Many folks interview at big tech companies multiple times before landing the role, so don't sweat it if you don't land a role on the first try. You'll gain great experience either way and be better prepared for future interviews!
</li>
</ul>

The Google Data Analyst Interview

Intro
Like many of the Tech giants, Facebook, leverages data to the fullest extent in all parts of its business. This provides many ongoing opportunities for those looking to get hired into a Data Scientist role at Facebook! This blog post will take you through the role, required skills, the interview process, a few example questions, and some tips to help you start prepping for your interview.
The Role
Data Scientist roles are embedded throughout Facebook, affording many opportunities. Below are a few types of data scientists Facebook hires for:
<ul>
<li>Financial Data Scientists - Folks in this role ensure that Facebook is spending and making money for the business according to plan. Financial Data Scientists help build models to understand what revenue and spending will look like in the years to come.</li>
<li>Product Data Scientists - These data scientists work on understanding how users interact with the different Facebook products and provide analysis to help improve products and user experience.</li>
<li>Community Operations and Business Integrity Data Scientists - Folks here ensure Facebook’s content-focused products (such as Ads, Facebook Posts, Marketplace Posts) have content compliant with Facebook’s policies. These teams work heavily with Engineering to build/improve machine learning models which help proactively detect non-compliant content.</li>
</ul>
Required Skills
An average Facebook Data Scientist typically has 3-6 years of experience in a quantitative role. Other qualifications include:
<ul>
<li>Degree in a technical program such as Mathematics, Statistics, Computer Science, Physics, Engineering, Operations Research, Economics, Engineering</li>
<li>Experience with SQL</li>
<li>Experience with applied statistics</li>
<li>Experience with quantitative analysis</li>
<li>Experience with relational databases and business intelligence tools</li>
<li>Experience with scripting languages (e.g. Python, Java)</li>
<li>Experience with stats software such as Numpy/Pandas (Colab Notebooks are a great way to practice this skill)</li>
<li>Experience working with cross-functional teams</li>
</ul>
These skills vary from role to role, so be sure to read into the specifics for a particular role.
Interview Process
The typical interview process at Facebook follows the steps below:
<ul>
<li>Initial screening by a recruiter (~30 minutes). This is more of a formality to ensure the candidate fits the criteria for the role. The recruiter will ask about your past experiences, your interests, and will provide you more details about the role.</li>
<li>Initial phone interview with a member of the current team (45 minutes). This interview will again typically cover past experiences, but with more of a focus on technical skills and problem solving ability. It’s important to use stories / past experiences that highlight your technical expertise here.</li>
<li>Onsite interview. These interviews will be with members of the current team (it's likely one of interviews will be with the hiring manager) and cross-functional team members. The interviews will take about 1/2 a day - interviewing with 3-4 people for about 45 minutes each. These interviews can be structured in many different ways depending on what team you’re interviewing for (your recruiter will provide you with the specifics when you reach this stage), but generally expect 1-2 technical interviews and the remaining be case-like/behavioral interviews.</li>
</ul>
Example Facebook Data Scientist Interview Questions
<ul>
<li>What are the challenges you faced in a previous project and how did you resolve it?</li>
<li>Write pseudo code for a dot product of two sparse vectors, ensuring it’s time and space efficient.</li>
<li>How would you measure the effectiveness of Facebook Lite?</li>
<li>How would you measure the effectiveness of Facebook Marketplace?</li>
<li>How do you work with cross-functional teams? Can you give me an example?</li>
<li>What is a F-test? What is a T-test? How are these different?</li>
<li>How does maximum likelihood relate to OLS? Under what conditions are they the same?</li>
<li>If you are the owner of a flower shop, how would you target your customers online? (who? when? where? how?)</li>
<li>How would you measure customer check out rates? How would you go about doing this?</li>
<li>How would you measure how much users liked videos?</li>
<li>Explain confidence intervals to someone with no statistics background.</li>
</ul>
Tips to help prepare for your interview
<ul>
<li>
Come prepared with 6-7 stories that cover the following:
</li>
<li>
A time you disagree with a team member, how did you deal with this?
</li>
<li>
A time where a client had unreasonable demands, how did you respond?
</li>
<li>
A time where you took on a leadership role formally or informally.
</li>
<li>
A time where you took a risk, what made the risk notable?
</li>
<li>
A time where you worked on a particularly difficult project. What made the project difficult?
</li>
<li>
An example of a time you were leading a group and it wasn’t going well, how did you turn it around?
</li>
<li>
Ensure that you have spare copies of your resume.
</li>
<li>
Use the <a href="/blog/star_method">STAR Method</a>
</li>
<li>
Be yourself!
</li>
</ul>

Like many of the Tech giants, Facebook, leverages data to the fullest extent in all parts of its business. This provides many ongoing opportunities...

The Facebook Data Scientist Interview

<h3>1) The rules of The Price is Right / the setup</h3>
Suppose we're a contestant on a popular game show called <a href="https://en.wikipedia.org/wiki/The_Price_Is_Right">The Price is Right</a>. We've made it to 'The Showcase' round where we're tasked with guessing the price of a group of prizes (typically trips, cars, furniture, etc.). The rules for the round (simplified a bit for our example) are as follows:
<ul>
<li>There are two contestants competing in a showcase round</li>
<li>Both contestants are shown unique groups of prizes (two groups of prizes shown in total)</li>
<li>After seeing the prizes, each contestant is asked to bid on what they think the value is of their unique set of prizes</li>
<li>Whichever bid is closer, without going over, wins. If a contestant bids over, then their bid is disqualified from winning</li>
</ul>
This game provides an interesting example of balancing uncertainty with risk (since if we bid even 1 dollar over the true price of the items we are disqualified).
Given this information, we'll step through using <a href="https://en.wikipedia.org/wiki/Bayesian_inference">Bayesian Inference</a> to build a model and select our final bid in the showcase. Below are some details for our hypothetical showcase round (the exact #s are not important):
<ul>
<li>We'll assume we have data on the 'true prices' of past The Price is Right Showcase prizes, and they are normally distributed with a mean of 23,000 and a standard deviation of 4,500</li>
<li>The two prizes in our prize package are:
<ul>
<li>A trip to Hawaii</li>
<li>A set of living room furniture</li>
</ul>
</li>
</ul>
<h3>2) Visualizing the historical 'true price' distribution, along with our guesses</h3>
First, we'll set up a quick visualization to illustrate the historical 'true price,' as well as our guesses for each sub-prize in our package:
<pre><code># Import libraries
%matplotlib inline
import scipy.stats as stats
from IPython.core.pylabtools import figsize
import numpy as np
import matplotlib.pyplot as plt

figsize(12.5, 9)

### Plot historical total price distribution ###

##################################################
# Subplot 1: historical total package true prices
# 3 x 1 grid, first sub plot
plt.subplot(311)
# Set range for x-axis
# 0-40000, split into 200 intervals
x = np.linspace(0, 40000, 200)
# Set up plot, normally distributed pdf 
# with mean of 23000, std. dev of 4500
# set up line color, weight, alpha (opacity), 
# line label
sp1 = plt.fill_between(x , 0, stats.norm.pdf(x, 23000, 4500), 
 color = "#4b2cde", lw = 3, alpha = 0.6,
 label = "Historical total prices")
# Plot subplot inside rectangle, starting at 0,0 width/height of 1
p1 = plt.Rectangle((0, 0), 1, 1, fc=sp1.get_facecolor()[0])
# Add legend
plt.legend([p1], [sp1.get_label()])

##################################################

# Subplot 2: Our guess for Hawaii trip cost
# Comments repeat for historical price distribution above
plt.subplot(312)
x = np.linspace(0, 40000, 200)
sp2 = plt.fill_between(x , 0, stats.norm.pdf(x, 4500, 800), 
 color = "#62cb73", lw = 3, alpha = 0.6,
 label="Hawaii trip guess")

p2 = plt.Rectangle((0, 0), 1, 1, fc=sp2.get_facecolor()[0])
plt.legend([p2], [sp2.get_label()])

##################################################

# Subplot 3: Our guess for living room furniture
plt.subplot(313)
x = np.linspace(0, 40000, 200)
sp3 = plt.fill_between(x , 0, stats.norm.pdf(x, 12000, 3000), 
 color = "#ed5259", lw = 3, alpha = 0.6,
 label = "Furniture set guess")
plt.autoscale(tight=True)
p3 = plt.Rectangle((0, 0), 1, 1, fc=sp3.get_facecolor()[0])
plt.legend([p3], [sp3.get_label()]);</code></pre>
<a href="/images/blog/q286_p1.png"><img src="/images/blog/q286_p1.png" alt="q286 p1"></a>
<h3>3) Plug our guesses + historical price information into a Bayesian statistical model</h3>
Next, we'll feed our guesses, as well as the historical total price mean and standard deviation, into <a href="https://docs.pymc.io/">PyMC3</a> (a Python package for Bayesian statistical modeling and probabilistic machine learning).
<pre><code># Import library
import pymc3 as pm
# Mean guesses: Hawaii trip, furniture
data_mu = [4500, 12000]
# Our error estimate on guesses: Hawaii trip, furniture
data_std = [800, 3000] 
# Historical mean
mu_prior = 23000
# Historical standard deviation
std_prior = 4500

# If new to pymc3, would recommend exploring quickstart/docs:
# https://docs.pymc.io/notebooks/api_quickstart.html
with pm.Model() as model:
 # Feed the variables into our model
 # you can read more about the required variables/structure
 # in the docs above
 true_price = pm.Normal("true_price", mu=mu_prior, sd=std_prior)
 prize_1 = pm.Normal("first_prize", mu=data_mu[0], sd=data_std[0])
 prize_2 = pm.Normal("second_prize", mu=data_mu[1], sd=data_std[1])
 price_estimate = prize_1 + prize_2
 # Here we're evaluating a normal distribution w/ mean of price_estimate,
 # standard deviation of 3800, at the values provided by true_price
 # PyMC3 expects the logp() method to return a log-probability 
 # evaluated at the passed value argument. This method is used 
 # internally by all of the inference methods to calculate the model 
 # log-probability that is used for fitting models. 
 logp = pm.Normal.dist(mu=price_estimate, sd=(3800)).logp(true_price)
 error = pm.Potential("error", logp)

 # Set step method, pull sample
 # Here we selected NUTS, 'No-U-Turn Sampler' most commonly used
 # in pymc3, Markov chain Monte Carlo sampling algorithm
 trace = pm.sample(50000, step=pm.NUTS())
 # Sampler usually converge to the typical set slowly in high dimension, 
 # generally good practice to discard first few thousand samples that are 
 # not yet in the typical set. Less of an issue with modern sampler like NUTS
 burned_trace = trace[10000:]

true_price_trace = burned_trace["true_price"]</code></pre>
<h3>4) Plot our new price estimate along with the prior distribution of historical prices</h3>
Next we can plot the output of our 'price_trace' which represents our new price estimate. Additionally, we'll plot the distribution and mean of the prior (historical) estimate so we can visually see how things shifted:
<pre><code>import scipy.stats as stats
# Build linespace from 5000-40000 for 
# x axis
x = np.linspace(5000, 40000)
# Plot normal pdf for historical prior distribution
plt.plot(x, stats.norm.pdf(x, 23000, 4500), c = "k", lw = 2, 
 label = "prior dist. of suite price")
# Plot price trace into histogram, add lines for mean, prior's mean
_hist = plt.hist(true_price_trace, bins = 25, normed= True, color = "#4b2cde", alpha=0.6, histtype= "stepfilled")
plt.title("Posterior of the true price estimate")
plt.vlines(mu_prior, 0, 1.1*np.max(_hist[0]), label = "prior's mean",
 linestyles="--")
plt.vlines(true_price_trace.mean(), 0, 1.1*np.max(_hist[0]), \
 label = "posterior's mean", linestyles="-.")
plt.legend(loc = "upper right");</code></pre>
<a href="/images/blog/q286_p2.png"><img src="/images/blog/q286_p2.png" alt="q286 p2"></a>
Here we can see that due to our two observed prizes and guesses (including our uncertainty estimates expressed as standard deviation), we shifted our mean price estimate down a few thousand dollars from the prior mean price.
<h3>5) Account for the asymmetric reward system for our guesses</h3>
Now that we've adjusted our bid from just pulling the mean of prior distributions (23,000), one might be tempted to just take our new posterior mean that incorporates our estimates for each prize into the overall bid.
However, we can incorporate one more factor - risk - into our thinking before arriving at a final bid.
Since the rules of the game are unique in the sense that any bid over the value results in earnings of $0, we can create a loss function to incorporate a 'risk' parameter, which is meant to represent how bad it is for us if we guess higher than the true price. (A proxy for our risk aversion)
For every possible bid, we calculate the expected loss associated with that bid. We then vary the risk parameter to see how it affects our loss:
<pre><code>figsize(12.5, 7)
#numpy friendly showdown_loss

def showdown_loss(guess, true_price, risk = 80000):
 loss = np.zeros_like(true_price)
 ix = true_price &#x3C; guess
 loss[~ix] = np.abs(guess - true_price[~ix])
 loss[ix] = risk
 return loss

# x-axis labeling
guesses = np.linspace(5000, 40000, 70) 
# y-axis labeling, risk band
risks = np.linspace(20000, 90000, 6)
expected_loss = lambda guess, risk: \
 showdown_loss(guess, true_price_trace, risk).mean()

# Loop through each risk parameter, calculate the expected 
# loss, plot 
for _p in risks:
 results = [expected_loss(_g, _p) for _g in guesses]
 plt.plot(guesses, results, label = "%d"%_p)

plt.title("Expected loss of different guesses, \nvarious risk-levels of \
overestimating")
plt.legend(loc="upper left", title="Risk parameter")
plt.xlabel("price bid")
plt.ylabel("expected loss")
plt.xlim(5000, 30000);</code></pre>
<a href="/images/blog/q286_p3.png"><img src="/images/blog/q286_p3.png" alt="q286 p3"></a>
<h3>6) Minimize our expected loss, calculate our bid across various risk levels</h3>
Here we can visually see approximations for where the minimum expected loss is relative to each risk parameter (line). However, we can get more precise by plugging this into an optimization function and minimizing our expected loss subject to our risk.
The function in 'fmin in scipy.optimize' module uses an intelligent search to find a minimum (not necessarily a global minimum) of any uni or multivariate function. For most purposes, 'fmin' will provide you with a good answer. Note there are several ways to pull the rough minimum out of this function, this is just a quick approach:
<pre><code>import scipy.optimize as sop
ax = plt.subplot(111)
for _p in risks:
 _color = next(ax._get_lines.prop_cycler)
 # https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.fmin.html
 # 10k is our x0, starting point
 _min_results = sop.fmin(expected_loss, 10000, args=(_p,),disp = False)
 _results = [expected_loss(_g, _p) for _g in guesses]
 plt.plot(guesses, _results , color = _color['color'])
 plt.scatter(_min_results, 0, s = 60, \
 color= _color['color'], label = "%d"%_p)
 plt.vlines(_min_results, 0, 120000, color = _color['color'], linestyles="--")
 print("minimum at risk %d: %.2f" % (_p, _min_results))

plt.title("Expected loss &#x26; Bayes actions of different guesses, \n \
various risk-levels of overestimating")
plt.legend(loc="upper left", scatterpoints = 1, title = "Bayes action at risk:")
plt.xlabel("price guess")
plt.ylabel("expected loss")
plt.xlim(6000, 30000)
plt.ylim(0, 100000);</code></pre>
<pre><code>minimum at risk 20000: 15507.21
minimum at risk 34000: 14336.10
minimum at risk 48000: 13817.60
minimum at risk 62000: 13353.86
minimum at risk 76000: 12932.99
minimum at risk 90000: 12930.03</code></pre>
<a href="/images/blog/q286_p4.png"><img src="/images/blog/q286_p4.png" alt="q286 p4"></a>
As expected, when we decrease the risk threshold, our optimal bid increases. Additionally, we can see that the overall bid is shifted lower (compared to the posterier mean above) when accounting for risk, which is in line with a realistic strategy one would undertake in this game. In other words, we'd expect someone with a level of risk aversion to slightly undercut their normal price estimate to avoid overbidding. So depending on our personal risk level, we can now select our best estimate for the showcase!

Suppose we're a contestant on a popular game show called The Price is Right. We've made it to 'The Showcase' round where we're tasked with guessing...

Bayesian Analysis of The Price is Right Showcase

If you're new to the data space, recently learned a new skill, or are trying to build a more robust data science/analyst portfolio, a perfect way of solidifying your skills is to do some mini-projects focused on your new skills. Below we outline a few places you can find publicly available data for your next project.
If you're interested in practicing real data scientist and analyst interview questions, feel free to sign up for our <a href="?ref=free_public_data_sets">email newsletter</a>, where we send a few curated questions per week to help you prepare for interviews at top companies.
<h2>1. <a href="https://data.fivethirtyeight.com/">FiveThirtyEight</a></h2>
FiveThirtyEight is an interactive news and sports site that has some incredible data visualizations (<a href="https://fivethirtyeight.com/">which you should totally check out</a>). They make a lot of their data open to the public, meaning you can download and play with the source data yourself!
Here are some examples:
<ul>
<li><a href="https://github.com/fivethirtyeight/data/tree/master/airline-safety">Airline Safety</a> — contains information on accidents from each airline.</li>
<li><a href="https://github.com/fivethirtyeight/data/tree/master/us-weather-history">US Weather History</a> — historical weather data for the US.</li>
<li><a href="https://github.com/fivethirtyeight/data/tree/master/study-drugs">Study Drugs</a> — data on who's taking Adderall in the US.</li>
</ul>
<h2>2. <a href="https://github.com/BuzzFeedNews">BuzzFeed News</a></h2>
BuzzFeed makes the data sets, analysis, libraries, tools, and guides used in its articles available on GitHub. Check them out to learn from some of the best!
Here are some examples:
<ul>
<li><a href="https://github.com/BuzzFeedNews/2016-04-federal-surveillance-planes">Federal Surveillance Planes</a> — contains data on planes used for domestic surveillance.</li>
<li><a href="https://github.com/BuzzFeedNews/zika-data">Zika Virus</a> — data about the geography of the Zika virus outbreak.</li>
<li><a href="https://github.com/BuzzFeedNews/nics-firearm-background-checks">Firearm Background Checks</a> — data on background checks of people attempting to buy firearms.</li>
</ul>
<h2>3. <a href="https://www.kaggle.com/datasets">Kaggle</a></h2>
Kaggle, <a href="https://techcrunch.com/2017/03/08/google-confirms-its-acquisition-of-data-science-community-kaggle/">recently acquired by Google</a>, is a place where you can learn, practice, and fine-tune your data science/analytics skills. They have tons of data that’s open to the public, and allow users of the platform to share code so you can learn best practices within the data space. They also host <a href="https://www.kaggle.com/competitions">competitions</a> where you can win real money if you have a top ranking model!
Here are some examples:
<ul>
<li><a href="https://github.com/BuzzFeedNews/2016-04-federal-surveillance-planes">Federal Surveillance Planes</a> — contains data on planes used for domestic surveillance.</li>
<li><a href="https://github.com/BuzzFeedNews/zika-data">Zika Virus</a> — data about the geography of the Zika virus outbreak.</li>
<li><a href="https://github.com/BuzzFeedNews/nics-firearm-background-checks">Firearm background checks</a> — data on background checks of people attempting to buy firearms.</li>
</ul>
<h2>4. <a href="https://opendata.socrata.com/">Socrata</a></h2>
Socrata hosts cleaned open source data sources ranging from government, business, and education data sets.
Here are some examples:
<ul>
<li><a href="https://opendata.socrata.com/Government/2010-Report-to-Congress-on-White-House-Staff/vedg-c5sb">White House Staff Salaries</a> — data on the salary of each White House staffer in 2010.</li>
<li><a href="https://opendata.socrata.com/Government/Milk-RadNet-Laboratory-Analysis/pkfj-5jsd">Radiation Analysis</a> — data on which milk products in the US were radioactive based on location.</li>
<li><a href="https://opendata.socrata.com/Government/2012-Workplace-Fatalities-by-State/vcx3-xxtb">Workplace Fatalities by US State</a> — the number of workplace deaths across the US.</li>
</ul>
<h2>5. <a href="https://github.com/awesomedata/awesome-public-datasets">Awesome-Public-Datasets on GitHub</a></h2>
This github hosts a library of awesome, public datasets! They are all sorted by category and link you straight to the hosting website.
Here are some examples:
<ul>
<li>
<a href="https://en.tutiempo.net/climate">Global Climate Data</a> — climate information for every country in the world with historical data (some date back to 1929).
</li>
<li>
<a href="http://ecg.mit.edu/time-series/">Heart Rate Time Series Data</a> — two series of data contains 1800 evenly-spaced measurements of instantaneous heart rate from a single subject.
</li>
<li>
<a href="http://www.planecrashinfo.com/database.htm">Plane Crash Database</a> — plane crash data dating from 1929.
</li>
</ul>
<h2>6. <a href="https://cloud.google.com/bigquery/public-data/">Google Public Datasets</a></h2>
Google lists all of the data sets on a page. Google has a cloud hosting service called Google Cloud Platform (GCP), and you can query using a tool called BigQuery to explore these datasets. You'll need to sign up for a GCP account, but the first 1TB of queries you make are free!
Here are some examples:
<ul>
<li><a href="https://cloud.google.com/bigquery/public-data/usa-names">US Name Data Set</a> — contains all names from social security card applications from births that occur after 1879.</li>
<li><a href="https://cloud.google.com/bigquery/public-data/baseball">Major League Baseball Data</a> — data includes pitch-by-pitch data for Major League Baseball (MLB) games in 2016.</li>
</ul>
<h2>7. <a href="http://archive.ics.uci.edu/ml/index.php">UCI Machine Learning Repository</a></h2>
University of California Irvine hosts 440 data sets as a service to the machine learning community. These data sets are nice because most of them are squeaky clean and ready for modeling!
Here are some examples:
<ul>
<li>
<a href="http://archive.ics.uci.edu/ml/datasets/Iris">Iris Data Set</a> — the most famous pattern recognition dataset.
</li>
<li>
<a href="http://archive.ics.uci.edu/ml/datasets/Wine">Wine Data Set</a> — using chemical analysis to determine the origin of wine.
</li>
<li>
<a href="http://archive.ics.uci.edu/ml/datasets/Wine">Forest Fires</a> — try to predict the burn area of forest fires using this dataset.
</li>
</ul>
<h2>8. <a href="https://www.data.gov/">Data.gov</a></h2>
Data.gov allows you to download and explore data from multiple US government agencies. Data can range from government budgets to climate data. The data is very well documented so you should have an easy time navigating the sources. You can browse the data sets on Data.gov directly without registering. You can browse by topic area or search for a specific data set.
Here are some examples:
<ul>
<li><a href="https://catalog.data.gov/dataset/food-environment-atlas-f4a22">Food Environment Atlas</a> — contains data on how local food choices affect diet in the US.</li>
<li><a href="https://catalog.data.gov/dataset/annual-survey-of-school-system-finances">School System Finances</a> — a survey of the finances of school systems in the US.</li>
<li><a href="https://catalog.data.gov/dataset/u-s-chronic-disease-indicators-cdi-e50c9">Chronic Disease Data</a> — data on chronic disease indicators in areas across the US.</li>
</ul>
<h2>9. <a href="http://academictorrents.com/browse.php">Academic Torrents</a></h2>
Academic Torrents is a site that is geared around sharing the data sets from scientific papers. It has tons of interesting data sets. You can browse the data sets directly on the site and download them.
Here are some examples:
<ul>
<li><a href="http://academictorrents.com/details/4697a6e1e7841602651b087d84f904d43590d4ff">Enron Emails</a> — a set of many emails from executives at Enron, a company that famously went bankrupt.</li>
<li><a href="http://academictorrents.com/details/e24e083cc337695bb84a2b68707695579c0ab4d8">Student Learning Factors</a> — a set of factors that measure and influence student learning.</li>
<li><a>News Articles</a> — contains news article attributes and a target variable.</li>
</ul>
<h2>10. <a href="https://www.quandl.com/search">Quandl</a></h2>
Quandl is a repository of economic and financial data. Some of the datasets are free, while others are up for purchase.
Here are some examples:
<ul>
<li><a href="https://www.quandl.com/search?query=KAUFFMAN">Entrepreneurial Activity by Race and Other Factors</a> — contains data from the Kauffman foundation on entrepreneurs in the US.</li>
<li><a href="https://www.quandl.com/search?query=NBSC">Chinese Macroeconomic Data</a> — indicators of Chinese economic health.</li>
<li><a href="https://www.quandl.com/search?query=FRED">US Federal Reserve Data</a> — US economic indicators from the Federal Reserve.</li>
</ul>
<h2>11. <a href="https://tinyletter.com/data-is-plural">Jeremy Singer-Vine</a></h2>
Jeremy Singer-Vine collects awesome data sets across multiple sources. If you're interested in getting data sets straight to your inbox, you should consider signing up for his <a href="https://tinyletter.com/data-is-plura">newsletter</a>.

The 11 best websites to find interesting, free data sets to analyze. Provided by InterviewQs, a mailing list for coding and data interview problems.

11 Websites to Find Free, Interesting Datasets

<h3>1) Background - The Efficient Frontier</h3>
Modern portfolio theory (MPT) is a mathematical framework for assembling a portfolio of assets such that risk-averse investors can construct portfolios to maximize expected return. This is based on a given level of market risk, emphasizing that higher risk is an inherent part of higher reward.
MPT uses weighted returns of equities to calculate a portfolio-wide return. Then it uses statistical measures such as variance (standard deviation) and correlation to determine 'risk' of a portfolio. It's important to note that for a portfolio comprised of 'n' assets you can't simply sum the standard deviation of each asset to calculate the total risk, but must also account for the correlation between assets.
Using MPT, one can calculate what is called the 'efficient frontier,' which is the set of 'optimal' portfolios that offer the highest expected return for a defined level of risk. In this post, we'll run thousands of portfolio simulations to highlight where the efficient frontier is, given a set of equities. The efficient frontier can also be calculated formulaically, but simulating and visualizing portfolio runs can help better illustrate what's going on.
If the above is all sounding like a lot, don't worry; it will become clearer as I step through the code example below!
<h3>2) Setting up a random portfolio of equities</h3>
First, let's grab some historical return data for a set of equities. We can then use this to generate random portfolios in our simulation:
<pre><code>import pandas as pd
from pandas_datareader import data as pdr
import fix_yahoo_finance as yf
import numpy as np
import datetime as dt
import matplotlib.pyplot as plt
plt.style.use('seaborn-colorblind') 
%matplotlib inline

# Create our portfolio of equities
tickers = ['AAPL','FB', 'C', 'DIS']

# Download closing prices
data = pdr.get_data_yahoo(tickers, start="2018-01-01", end=dt.date.today())['Close']

# From the closing prices, calculate periodic returns
returns = data.pct_change()

# Preview the data
data.tail()</code></pre>
<table>
<thead>
<tr>
<th>Symbols</th>
<th>AAPL</th>
<th>FB</th>
<th>C</th>
<th>DIS</th>
</tr>
</thead>
<tbody>
<tr>
<td>Date</td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>2019-12-24</td>
<td>284.269989</td>
<td>205.119995</td>
<td>78.589996</td>
<td>145.289993</td>
</tr>
<tr>
<td>2019-12-26</td>
<td>289.910004</td>
<td>207.789993</td>
<td>79.830002</td>
<td>145.699997</td>
</tr>
<tr>
<td>2019-12-27</td>
<td>289.799988</td>
<td>208.100006</td>
<td>79.669998</td>
<td>145.750000</td>
</tr>
<tr>
<td>2019-12-30</td>
<td>291.519989</td>
<td>204.410004</td>
<td>79.510002</td>
<td>143.770004</td>
</tr>
<tr>
<td>2019-12-31</td>
<td>293.649994</td>
<td>205.250000</td>
<td>79.889999</td>
<td>144.630005</td>
</tr>
</tbody>
</table>
Next, just for general context into each stock's historical performance and to help sense check our numbers, we'll take a look at the price trends:
<pre><code>plt.figure(figsize=(14, 7))
for i in data.columns.values:
 plt.plot(data.index, data[i], lw=2, alpha=0.8,label=i)
plt.legend(loc='upper left', fontsize=12)
plt.ylabel('price ($)')</code></pre>
<a href="/images/blog/efficient_frontier_p1.png"><img src="/images/blog/efficient_frontier_p1.png" alt="efficient frontier p1"></a>
Plotting this using daily % change in return can help illustrate the stocks' volatility.
<pre><code>returns = data.pct_change()

plt.figure(figsize=(14, 7))
for i in returns.columns.values:
 plt.plot(returns.index, returns[i], lw=2, alpha=0.8,label=i)
plt.legend(loc='lower center', fontsize=14)
plt.ylabel('daily returns')</code></pre>
<a href="/images/blog/efficient_frontier_p2.png"><img src="/images/blog/efficient_frontier_p2.png" alt="efficient frontier p2"></a>
This chart is a bit noisy with all of the equities plotted at once, but we can see Disney has a positive spike around April 2019, Facebook has one large negative spike along with some smaller positive/negative spikes, etc. Remember, in MPT the spikes = volatility which is then used as a proxy for risk.
<h3>3) Defining functions to simulate random portfolios</h3>
Now that we have the historical price data pulled, we'll define functions to assign random weights to each stock in the portfolio, then calculate the portfolio's overall annualized returns and annualized volatility.
We'll first create a function to calculate our portfolio returns and volatility:
<pre><code>
# Define function to calculate returns, volatility
def portfolio_annualized_performance(weights, mean_returns, cov_matrix):
 # Given the avg returns, weights of equities calc. the portfolio return
 returns = np.sum(mean_returns*weights ) *252
 # Standard deviation of portfolio (using dot product against covariance, weights)
 # 252 trading days
 std = np.sqrt(np.dot(weights.T, np.dot(cov_matrix, weights))) * np.sqrt(252)
 return std, returns</code></pre>
Next we can create a function to generate random portfolios with random weights assigned to each equity:
<pre><code>def generate_random_portfolios(num_portfolios, mean_returns, cov_matrix, risk_free_rate):
 # Initialize array of shape 3 x N to store our results, 
 # where N is the number of portfolios we're going to simulate
 results = np.zeros((3,num_portfolios))
 # Array to store the weights of each equity
 weight_array = []
 for i in range(num_portfolios):
 # Randomly assign floats to our 4 equities
 weights = np.random.random(4)
 # Convert the randomized floats to percentages (summing to 100)
 weights /= np.sum(weights)
 # Add to our portfolio weight array
 weight_array.append(weights)
 # Pull the standard deviation, returns from our function above using 
 # the weights, mean returns generated in this function
 portfolio_std_dev, portfolio_return = portfolio_annualized_performance(weights, mean_returns, cov_matrix)
 # Store output
 results[0,i] = portfolio_std_dev
 results[1,i] = portfolio_return
 # Sharpe ratio
 results[2,i] = (portfolio_return - risk_free_rate) / portfolio_std_dev
 return results, weight_array</code></pre>
Next, let's set up our final input variables, calculating the mean returns of each stock, as well as a covariance matrix to be used in our risk calculation below:
<pre><code>returns = data.pct_change()
mean_returns = returns.mean()
cov_matrix = returns.cov()
# Number of portfolios to simulate
num_portfolios = 10000
# Risk free rate (used for Sharpe ratio below)
# anchored on treasury bond rates
risk_free_rate = 0.018</code></pre>
<h3>4) Simulating portfolio returns/variance, visualizing the efficient frontier</h3>
Finally, we can toss everything together in one final function below. The below function is generating a random portfolio, obtaining the returns, volatility, and weights. We'll also go ahead and add an annotation showing the maximum <a href="https://www.investopedia.com/terms/s/sharperatio.asp">Sharpe ratio</a> (the average return earned in excess of the risk-free rate per unit of volatility or total risk). In general, a higher Sharpe ratio is better.
All of the randomly generated portfolios will be plotted with a color map applied to them based on the Sharpe ratio. The more green, the higher the Sharpe ratio.
For the 'optimal' portfolio with the highest Sharpe ratio (highest returns relative to risk), we'll also print the % allocation across the portfolio:
<pre><code>def display_simulated_portfolios(mean_returns, cov_matrix, num_portfolios, risk_free_rate):

 # pull results, weights from random portfolios
 results, weights = generate_random_portfolios(num_portfolios,mean_returns, cov_matrix, risk_free_rate)

 # pull the max portfolio Sharpe ratio (3rd element in results array from 
 # generate_random_portfolios function)
 max_sharpe_idx = np.argmax(results[2])

 # pull the associated standard deviation, annualized return w/ the max Sharpe ratio
 stdev_portfolio, returns_portfolio = results[0,max_sharpe_idx], results[1,max_sharpe_idx]

 # pull the allocation associated with max Sharpe ratio
 max_sharpe_allocation = pd.DataFrame(weights[max_sharpe_idx],index=data.columns,columns=['allocation'])
 max_sharpe_allocation.allocation = [round(i*100,2)for i in max_sharpe_allocation.allocation]
 max_sharpe_allocation = max_sharpe_allocation.T

 print("-"*100)
 print("Portfolio at maximum Sharpe Ratio\n")
 print("--Returns, volatility--\n")
 print("Annualized Return:", round(returns_portfolio,2))
 print("Annualized Volatility:", round(stdev_portfolio,2))

 print("\n")
 print("--Allocation at max Sharpe ratio--\n")
 print(max_sharpe_allocation)
 print("-"*100)

 plt.figure(figsize=(16, 9))
 # x = volatility, y = annualized return, color mapping = sharpe ratio
 plt.scatter(results[0,:],results[1,:],c=results[2,:], cmap='winter', marker='o', s=10, alpha=0.3)
 plt.colorbar()
 # Mark the portfolio w/ max Sharpe ratio
 plt.scatter(stdev_portfolio, returns_portfolio, marker='x',color='r',s=150, label='Max Sharpe ratio')
 plt.title('Simulated portfolios illustrating efficient frontier')
 plt.xlabel('annualized volatility')
 plt.ylabel('annualized returns')
 plt.legend(labelspacing=1.2)

display_simulated_portfolios(mean_returns, cov_matrix, num_portfolios, risk_free_rate)</code></pre>
<a href="/images/blog/efficient_frontier_p3.png"><img src="/images/blog/efficient_frontier_p3.png" alt="efficient frontier p3"></a>
<a href="/images/blog/efficient_frontier_p4.png"><img src="/images/blog/efficient_frontier_p4.png" alt="efficient frontier p4"></a>

Learn how to calculate Value at Risk (VaR) of a stock portfolio using Python. Provided by InterviewQs, a mailing list for coding and data interview problems.

Simulating Thousands of Random Portfolios to Illustrate the Efficient Frontier using Python

The St. Petersburg Paradox is a paradox related to probability and decision theory in economics. It is based on a lottery game that has an infinite expected value (one should be willing to play at whatever cost). However, for the participants, each game is only worth a small amount (hence the paradox). This post will go over the expected value of a game, the reality of the expected value, and a Python simulation to emphasize the game's reality.
The standard version of the St. Petersburg Paradox is derived from the St. Petersburg game. The game is played by flipping a fair coin until a heads appears (at which point the game is over). The player wins $2 per flip. Given the rules of the game, what would be a fair price to enter the game?
<h3>Solving for cost using expected value</h3>
We can use probability theory to solve for the expected value of the game, then assume any price less than the expected value is a fair price. The expected value of the game is straightforward to calculate. All that is needed is the monetary values of the outcomes and their probabilities. If the coin lands heads on the first flip, winnings are $2. If it lands heads on the second flip, winnings are $4. If this happens on the third flip, winnings are $8, and so on.
The probabilities of the outcomes are (\frac{1}{2}\ ), (\frac{1}{4}\ ), (\frac{1}{8}\ ), etc.
Given this, the expected monetary value of the St. Petersburg game is:
(E(X) = (\frac{1}{2}\ \times2) + (\frac{1}{4}\ \times4) + (\frac{1}{8}\ \times8) + (\frac{1}{16}\ \times16) + \ldots )
( E(X) = \sum_{n=1}^{\infty} (\frac{1}{2})^{n} \cdot 2^n )
(E(X) = \infty )
The probabilistic expected value of the game is infinite, meaning any cost would be acceptable to play the game. For many reasons this does not make sense, and we'll go into the reality of the game in the next section.
<h3>Reality of the expected value</h3>
The expected value of the game indicates that it would be rational to pay any amount for an opportunity to play the game, even though average payout of the game is quite low. For example, there's a 50% chance of winning $2, and a 75% chance that no more than $4 will be won. Given the two statistics and potential winnings, the reality is that individuals are probably not willing to pay more than $2-4 dollars to play this game. We can support this hypothesis in the next section with a simulation.
<h3>Simulating the game</h3>
Here is a <a href="https://colab.research.google.com/drive/1I5aeWjS7MZMyoUnQSV-VZXMmnpn7EJ52#scrollTo=e3n8DjG50h2p">link to the simulation of the game in a Python notebook.</a> You can create a copy of this notebook and run the simulation yourself.
In the simulation, 100,000 iterations of the game were run and yielded the following outputs:
Total Winnings: 751340
Expected Winnings: 7.5134
Median Winnings: 2.0
The output of the simulation provides a framework to come up with a more reasonable cost per game. From the simulation, the expected winnings per game is $7.51, however the median expected winnings are $2.00 per game. As we stated above, we have a 50% chance of winning $2, and the median winnings of $2 from our simulation make sense. The expected winnings are much higher than the median, but again this is after playing the game 100,000 times (which a person probably will not be playing the game for that long). The chart below plots the first 1000 runs and the cumulative winnings. At 200 runs, our simulation has us at ~$800 of winnings, and at 1000 runs, our simulation has us at $5000. This places the value of our game between $4-$5 per game. We would want to err on the conservative side of $4 per game, and we can estimate that one would only want to pay less than $4 per game.
<a href="/images/blog/stpetersburg.png"><img src="/images/blog/stpetersburg.png" alt="stpetersburg"></a>
Below is the code that is used in the notebook and an example of the output for 100,000 runs:
<pre><code>#importing packages
import numpy as np
from scipy import stats

def GameTime(n_simulations):
 win = 0
 win_list = []
 for i in range(n_simulations):
 coin = 1
 n_consecutive_heads = 0

 while coin == 1:
 #creating binomial distribution for flip
 coin = np.random.binomial(1, 0.5)
 #if coin toss is = 1, then we keep going and
 # count the # of consecutive tails
 if coin == 1:
 n_consecutive_heads += 1

 #we enter this when we break our streak
 # if we get more than 1 head...
 if n_consecutive_heads != 0:
 #...we calculate our winnings
 new_win = 2 ** n_consecutive_heads
 #and append it to our running total for the simulation
 win += new_win
 #and append the instance to a list
 win_list.append(new_win)
 else:
 #if we lose, we append $0
 win_list.append(0)

 win_list = np.array(win_list, dtype=np.int32)
 print("Total Winnings:", win)
 print("Expected Winnings:", win / n_simulations)
 print("Median Winnings:", np.median(win_list), "\n")

 bins = np.bincount(win_list)
 unique = np.unique(win_list)

 for u in unique:
 print("Won ${}: {} times".format(u, bins[u]))

#driver code
n_simulations = 100000
GameTime(n_simulations)</code></pre>
Output:
<pre><code>Total Winnings: 751340
Expected Winnings: 7.5134
Median Winnings: 2.0

Won $0: 49769 times
Won $2: 25216 times
Won $4: 12515 times
Won $8: 6334 times
Won $16: 3075 times
Won $32: 1586 times
Won $64: 756 times
Won $128: 370 times
Won $256: 200 times
Won $512: 94 times
Won $1024: 52 times
Won $2048: 13 times
Won $4096: 9 times
Won $8192: 7 times
Won $16384: 2 times
Won $32768: 1 times
Won $65536: 1 times</code></pre>

The St. Petersburg Paradox is a paradox related to probability and decision theory in economics. It is based on a lottery game that has an infinite...

St. Petersburg Paradox

Using Monte Carlo methods, we'll write a quick simulation to predict future stock price outcomes for Apple ($AAPL) using Python. You can read more about Monte Carlo simulation (in a finance context) <a href="https://www.investopedia.com/terms/m/montecarlosimulation.asp">here</a>.
1) Pull the data
First, we can import the libraries, and pull the historical stock data for Apple. For this example, I picked the last ~10 years, although it would be valuable to test sensitivities of different ranges as this alone is subjective:
<pre><code># Import required libraries
import math
import matplotlib.pyplot as plt
import numpy as np
from pandas_datareader import data
apple = data.DataReader('AAPL', 'yahoo',start='1/1/2009')
apple.head()</code></pre>
<table>
<thead>
<tr>
<th></th>
<th>High</th>
<th>Low</th>
<th>Open</th>
<th>Close</th>
<th>Volume</th>
<th>Adj Close</th>
</tr>
</thead>
<tbody>
<tr>
<td>Date</td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr>
<td>2009-01-02</td>
<td>13.005714</td>
<td>12.165714</td>
<td>12.268572</td>
<td>12.964286</td>
<td>186503800.0</td>
<td>11.357091</td>
</tr>
<tr>
<td>2009-01-05</td>
<td>13.740000</td>
<td>13.244286</td>
<td>13.310000</td>
<td>13.511429</td>
<td>295402100.0</td>
<td>11.836406</td>
</tr>
<tr>
<td>2009-01-06</td>
<td>13.881429</td>
<td>13.198571</td>
<td>13.707143</td>
<td>13.288571</td>
<td>322327600.0</td>
<td>11.641174</td>
</tr>
<tr>
<td>2009-01-07</td>
<td>13.214286</td>
<td>12.894286</td>
<td>13.115714</td>
<td>13.001429</td>
<td>188262200.0</td>
<td>11.389630</td>
</tr>
<tr>
<td>2009-01-08</td>
<td>13.307143</td>
<td>12.862857</td>
<td>12.918571</td>
<td>13.242857</td>
<td>168375200.0</td>
<td>11.601127</td>
</tr>
</tbody>
</table>
<pre><code>#Next, we calculate the number of days that have elapsed in our chosen time window
time_elapsed = (apple.index[-1] - apple.index[0]).days</code></pre>
2) Calculate the compounded annualized growth rate over the length of the dataset + standard deviation (to feed into simulation)
<pre><code>#Current price / first record (e.g. price at beginning of 2009)
#provides us with the total growth %
total_growth = (apple['Adj Close'][-1] / apple['Adj Close'][1])

#Next, we want to annualize this percentage
#First, we convert our time elapsed to the # of years elapsed
number_of_years = time_elapsed / 365.0
#Second, we can raise the total growth to the inverse of the # of years
#(e.g. ~1/10 at time of writing) to annualize our growth rate
cagr = total_growth ** (1/number_of_years) - 1

#Now that we have the mean annual growth rate above,
#we'll also need to calculate the standard deviation of the
#daily price changes
std_dev = apple['Adj Close'].pct_change().std()

#Next, because there are roughy ~252 trading days in a year,
#we'll need to scale this by an annualization factor
#reference: https://www.fool.com/knowledge-center/how-to-calculate-annualized-volatility.aspx

number_of_trading_days = 252
std_dev = std_dev * math.sqrt(number_of_trading_days)

#From here, we have our two inputs needed to generate random
#values in our simulation
print ("cagr (mean returns) : ", str(round(cagr,4)))
print ("std_dev (standard deviation of return : )", str(round(std_dev,4)))
</code></pre>
cagr (mean returns) : 0.3746
std_dev (standard deviation of return : ) 0.2878
3) Generate random values, run Monte Carlo simulation
<pre><code>#Generate random values for 1 year's worth of trading (252 days),
#using numpy and assuming a normal distribution
daily_return_percentages = np.random.normal(cagr/number_of_trading_days, std_dev/math.sqrt(number_of_trading_days)number_of_trading_days)+1

#Now that we have created a random series of future
#daily return %s, we can simply apply these forward-looking
#to our last stock price in the window, effectively carrying forward
#a price prediction for the next year

#This distribution is known as a 'random walk'

price_series = [apple['Adj Close'][-1]]

for j in daily_return_percentages:
 price_series.append(price_series[-1] * j)

#Great, now we can plot of single 'random walk' of stock prices
plt.plot(price_series)
plt.show()</code></pre>
<a href="/images/blog/q184_p1.png"><img src="/images/blog/q184_p1.png" alt="q184 p1"></a>
<pre><code>#Now that we've created a single random walk above,
#we can simulate this process over a large sample size to
#get a better sense of the true expected distribution
number_of_trials = 3000

#set up an additional array to collect all possible
#closing prices in last day of window.
#We can toss this into a histogram
#to get a clearer sense of possible outcomes
closing_prices = []

for i in range(number_of_trials):
 #calculate randomized return percentages following our normal distribution
 #and using the mean / std dev we calculated above
 daily_return_percentages = np.random.normal(cagr/number_of_trading_days, std_dev/math.sqrt(number_of_trading_days),number_of_trading_days)+1
 price_series = [apple['Adj Close'][-1]]

 for j in daily_return_percentages:
 #extrapolate price out for next year
 price_series.append(price_series[-1] * j)

 #append closing prices in last day of window for histogram
 closing_prices.append(price_series[-1])

 #plot all random walks
 plt.plot(price_series)

plt.show()

#plot histogram
plt.hist(closing_prices,bins=40)

plt.show()</code></pre>
<a href="/images/blog/q184_p2.png"><img src="/images/blog/q184_p2.png" alt="q184 p2"></a>
<a href="/images/blog/q184_p3.png"><img src="/images/blog/q184_p3.png" alt="q184 p3"></a>
4) Analyze results
<pre><code>#from here, we can check the mean of all ending prices
#allowing us to arrive at the most probable ending point
mean_end_price = round(np.mean(closing_prices),2)
print("Expected price: ", str(mean_end_price))</code></pre>
Expected price: 193.85
<pre><code>#lastly, we can split the distribution into percentiles
#to help us gauge risk vs. reward

#Pull top 10% of possible outcomes
top_ten = np.percentile(closing_prices,100-10)

#Pull bottom 10% of possible outcomes
bottom_ten = np.percentile(closing_prices,10);

#create histogram again
plt.hist(closing_prices,bins=40)
#append w/ top 10% line
plt.axvline(top_ten,color='r',linestyle='dashed',linewidth=2)
#append w/ bottom 10% line
plt.axvline(bottom_ten,color='r',linestyle='dashed',linewidth=2)
#append with current price
plt.axvline(apple['Adj Close'][-1],color='g', linestyle='dashed',linewidth=2)

plt.show()</code></pre>
<a href="/images/blog/q184_p4.png"><img src="/images/blog/q184_p4.png" alt="q184 p4"></a>
Here, we can see (based solely on using Monte Carlo simulation, of course) there looks to be more upside than downside for the next year, with the expected price running about $193 and only a 10% chance of the price landing below $128.
There are of course many, many things that drive a stock price beyond historical percent changes (meaning I would not make investment decisions solely off Monte Carlo simulation!), but this offers a thorough example of using Monto Carlo simulation to better understand the distribution of possible outcomes.

Walk you through a quick simulation of future stock returns for AAPL using Python + Monte Carlo methods.

Simulating Future Stock Prices using Monte Carlo Methods in Python

<h3>What is Value at Risk (VaR)?</h3>
Value at Risk (VaR) is a statistic used to try and quantify the level of financial risk within a firm or portfolio over a specified time frame. VaR provides an estimate of the maximum loss from a given position or portfolio over a period of time, and you can calculate it across various confidence levels.
Estimating the risk of a portfolio is important to long-term capital growth and risk management, particularly within larger firms or institutions. VaR is typically framed as something like this:
<ul>
<li>"We have a portfolio VaR of 250,000 USD over the next month at 95% confidence."</li>
<li>This means that, with 95% confidence, we can say that the portfolio's loss will not exceed 250,000 USD in a month.</li>
</ul>
In this post I'll walk you through the steps to calculate this metric across a portfolio of stocks.
<h3>How is VaR calculated?</h3>
There are two main ways to calculate VaR:
<ul>
<li>Using Monte Carlo simulation</li>
<li>Using the variance-covariance method</li>
</ul>
In this post, we'll focus on using the second method, variance-covariance. In short, the variance-covariance method looks at historical price movements (standard deviation, mean price) of a given equity or portfolio of equities over a specified lookback period. It then uses probability theory to calculate the maximum loss within your specified confidence interval. You can read more details <a href="https://www.investopedia.com/ask/answers/041715/what-variancecovariance-matrix-or-parametric-method-value-risk-var.asp">here</a>, but we'll calculate it step by step below using Python.
Before we get started, note that the standard VaR calculation assumes the following:
<ul>
<li>Normal distribution of returns - VaR assumes the returns of the portfolio are normally distributed. This is of course not realistic for most assets, but allows us to develop a baseline using a much more simplistic calculation.</li>
<li>(Modifications can be made to VaR to account for different distributions, but here we'll focus on the standard VaR calculation).</li>
<li>Standard market conditions - Like many financial instruments, VaR is best used for considering loss in standard markets, and is not well-suited for extreme/outlier events.</li>
</ul>
<h3>Steps to calculate the VaR of a portfolio</h3>
In order to calculate the VaR of a portfolio, you can follow the steps below:
<ul>
<li>Calculate periodic returns of the stocks in the portfolio</li>
<li>Create a covariance matrix based on the returns</li>
<li>Calculate the portfolio mean and standard deviation</li>
</ul>
(weighted based on investment levels of each stock in portfolio)
<ul>
<li>Calculate the inverse of the normal cumulative distribution (PPF) with a specified confidence interval, standard deviation, and mean</li>
<li>Estimate the value at risk (VaR) for the portfolio by subtracting the initial investment from the calculation in step (4)</li>
</ul>
<h3>1) Calculate periodic returns of the stocks in the portfolio</h3>
<pre><code>import pandas as pd
from pandas_datareader import data as pdr
import fix_yahoo_finance as yf
import numpy as np
import datetime as dt

# Create our portfolio of equities
tickers = ['AAPL','FB', 'C', 'DIS']

# Set the investment weights (I arbitrarily picked for example)
weights = np.array([.25, .3, .15, .3])

# Set an initial investment level
initial_investment = 1000000

# Download closing prices
data = pdr.get_data_yahoo(tickers, start="2018-01-01", end=dt.date.today())['Close']

#From the closing prices, calculate periodic returns
returns = data.pct_change()

returns.tail()</code></pre>
<a href="/images/blog/var_returns.png"><img src="/images/blog/var_returns.png" alt="var returns"></a>
<h3>2) Create a covariance matrix based on the returns</h3>
<pre><code># Generate Var-Cov matrix
cov_matrix = returns.cov()
cov_matrix
</code></pre>
<a href="/images/blog/cov_matrix.png"><img src="/images/blog/cov_matrix.png" alt="cov matrix"></a>
This will allow us to calculate the standard deviation and mean of returns across the entire portfolio.
<h3>3) Calculate the portfolio mean and standard deviation</h3>
<pre><code># Calculate mean returns for each stock
avg_rets = returns.mean()

# Calculate mean returns for portfolio overall, 
# using dot product to 
# normalize individual means against investment weights
 # https://en.wikipedia.org/wiki/Dot_product#:~:targetText=In%20mathematics%2C%20the%20dot%20product,and%20returns%20a%20single%20number.
port_mean = avg_rets.dot(weights)

# Calculate portfolio standard deviation
port_stdev = np.sqrt(weights.T.dot(cov_matrix).dot(weights))

# Calculate mean of investment
mean_investment = (1+port_mean) * initial_investment

# Calculate standard deviation of investmnet
stdev_investment = initial_investment * port_stdev</code></pre>
Next, we can plug these variables into our percentage point function (PPF) below:
<h3>4) Calculate the inverse of the normal cumulative distribution (PPF) with a specified confidence interval, standard deviation, and mean</h3>
<pre><code># Select our confidence interval (I'll choose 95% here)
conf_level1 = 0.05

# Using SciPy ppf method to generate values for the
# inverse cumulative distribution function to a normal distribution
# Plugging in the mean, standard deviation of our portfolio
# as calculated above
# https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html
from scipy.stats import norm
cutoff1 = norm.ppf(conf_level1, mean_investment, stdev_investment)</code></pre>
<h3>5) Estimate the value at risk (VaR) for the portfolio by subtracting the initial investment from the calculation in step 4</h3>
<pre><code>#Finally, we can calculate the VaR at our confidence interval
var_1d1 = initial_investment - cutoff1
var_1d1

#output
#22347.7792230231</code></pre>
Here we are saying with 95% confidence that our portfolio of 1M USD will not exceed losses greater than 22.3k USD over a one day period.
<h3>Value at risk over n-day time period</h3>
What if we wanted to calculate this over a larger window of time? Below we can easily do that by just taking our 1 day VaR and multiplying it by the <a href="https://www.investopedia.com/articles/04/101304.asp">square root of the time period</a> (this is due to the fact that the standard deviation of stock returns tends to increase with the square root of time).
<pre><code># Calculate n Day VaR
var_array = []
num_days = int(15)
for x in range(1, num_days+1): 
 var_array.append(np.round(var_1d1 * np.sqrt(x),2))
 print(str(x) + " day VaR @ 95% confidence: " + str(np.round(var_1d1 * np.sqrt(x),2)))

# Build plot
plt.xlabel("Day #")
plt.ylabel("Max portfolio loss (USD)")
plt.title("Max portfolio loss (VaR) over 15-day period")
plt.plot(var_array, "r")
</code></pre>
<pre><code>1 day VaR @ 95% confidence: 20695.24
2 day VaR @ 95% confidence: 29267.49
3 day VaR @ 95% confidence: 35845.21
4 day VaR @ 95% confidence: 41390.49
5 day VaR @ 95% confidence: 46275.97
6 day VaR @ 95% confidence: 50692.79
7 day VaR @ 95% confidence: 54754.47
8 day VaR @ 95% confidence: 58534.99
9 day VaR @ 95% confidence: 62085.73
10 day VaR @ 95% confidence: 65444.11
11 day VaR @ 95% confidence: 68638.36
12 day VaR @ 95% confidence: 71690.43
13 day VaR @ 95% confidence: 74617.76
14 day VaR @ 95% confidence: 77434.51
15 day VaR @ 95% confidence: 80152.33</code></pre>
<a href="/images/blog/var_return_temporal.png"><img src="/images/blog/var_return_temporal.png" alt="var return temporal"></a>
<h3>(Extra) Checking distributions of our equities against normal distribution</h3>
As mentioned in the calculation section, we are assuming that the returns of the equities in our portfolio are normally distributed when calculating VaR. Of course, we can't predict that moving forward, but we can at least check how the historical returns have been distributed to help us assess whether VaR is suitable to use for our portfolio.
<pre><code>import matplotlib.mlab as mlab
import matplotlib.pyplot as plt
# Repeat for each equity in portfolio
returns['AAPL'].hist(bins=40, normed=True,histtype="stepfilled",alpha=0.5)
x = np.linspace(port_mean - 3*port_stdev, port_mean+3*port_stdev,100)
plt.plot(x, scipy.stats.norm.pdf(x, port_mean, port_stdev), "r")
plt.title("AAPL returns (binned) vs. normal distribution")
plt.show()</code></pre>
AAPl returns vs. normal distribution
<a href="/images/blog/aapl_returns.png"><img src="/images/blog/aapl_returns.png" alt="aapl returns"></a>
FB returns vs. normal distribution
<a href="/images/blog/fb_returns.png"><img src="/images/blog/fb_returns.png" alt="fb returns"></a>
C returns vs. normal distribution
<a href="/images/blog/c_returns.png"><img src="/images/blog/c_returns.png" alt="c returns"></a>
DIS returns vs. normal distribution
<a href="/images/blog/dis_returns.png"><img src="/images/blog/dis_returns.png" alt="dis returns"></a>
From the above we can see the returns have all been fairly normally distributed for our chosen stocks since 2018.
//You can view the code in Google Colab <a href="https://colab.research.google.com/drive/1IiB9ufjavgrCV_VCpbtIGSBuiAsJ-PN3">here</a> - to run it just make a copy of the notebook.

Calculating Value at Risk (VaR) of a Stock Portfolio using Python

This post will walk through a practice problem which analyzes NBA players' season stats. We've also provided the practice problem's <a href="https://colab.research.google.com/drive/1UTzahnEZ2j1vqpfJEctv4NxV1T2K9S0g#scrollTo=xo0zBHonsD3r">Colab Notebook</a> so you can follow along. Just copy the notebook and the two Google sheets (<a href="https://docs.google.com/spreadsheets/d/15NKCZNdFNe914qCnyDamuLj6Nq_-WyeBZ0IXCmvCm20/edit#gid=1473318657">season stats</a> and
<a href="https://docs.google.com/spreadsheets/d/19Xay4wMd2vxDTbz6n36y8RVPdbr1_o83nvFJhPQXPdA/edit">player data</a>) to your Google Drive.
This practice problem has a data set that contains NBA players and their invidiual season stats. This practice will take you through fitting a linear model using player stats to estimate <a href="https://www.basketball-reference.com/about/ws.html">win shares</a> per 48 minutes.
Topics covered in this tutorial:
<ul>
<li>Importing data from Google Sheets into Colab</li>
<li>Basic data cleaning / analysis</li>
<li>Splitting data into test and training sets</li>
<li>Training a single variable regression model</li>
<li>Training a multivariate regression model</li>
<li>Manually calculating Adjusted R-squared and R-squared</li>
<li>Calculating Root Mean Square Error</li>
<li>Comparing results of 2 regression models</li>
</ul>
Importing and cleaning data
<pre><code># Importing packages.
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sb
import numpy as np
import scipy.stats as ss
from sklearn.linear_model import LinearRegression
import seaborn as sns
sns.set(color_codes = True)
%matplotlib inline

# Setting figure sizes
plt.rcParams['figure.figsize'] = (8, 6)
plt.rcParams['font.size'] = 14</code></pre>
<pre><code># Code to connect Colab to Google Drive (save the spreadsheet provided into your drive)
# https://colab.research.google.com/notebooks/io.ipynb#scrollTo=JiJVCmu3dhFa

!pip install --upgrade -q gspread
from google.colab import auth
auth.authenticate_user()

import gspread
from oauth2client.client import GoogleCredentials

gc = gspread.authorize(GoogleCredentials.get_application_default())</code></pre>
<pre><code>### Importing sheet

# Open our new sheet and read some data.
worksheet = gc.open('Seasons_Stats').sheet1
# get_all_values gives a list of rows.
rows = worksheet.get_all_values()
# Convert to a DataFrame and render.
seasonPlayerStats = pd.DataFrame.from_records(rows)

### Fixing header of dataframe

# our header isn't named correctly lets fix our header
seasonPlayerStats.columns = seasonPlayerStats.iloc[0]
#drop the first row after we rename the header
seasonPlayerStats = seasonPlayerStats.reindex(seasonPlayerStats.index.drop(0))</code></pre>
<pre><code>### Importing sheet

# Open our new sheet and read some data.
worksheet = gc.open('player_data').sheet1
# get_all_values gives a list of rows.
rows = worksheet.get_all_values()
# Convert to a DataFrame and render.
playerData = pd.DataFrame.from_records(rows)

### Fixing header of dataframe
#rename header so it's the first row
playerData.columns = playerData.iloc[0]
#drop the first row after we rename the header
playerData = playerData.reindex(playerData.index.drop(0))</code></pre>
<pre><code># Merging dataframes and dropping null values so we avoid errors
# joining 
playerAllStats = pd.merge(seasonPlayerStats, playerData, how='left', on='Player')
# dropping null value columns to avoid errors 
playerAllStats = playerAllStats[pd.notnull(playerAllStats['height'])]
playerAllStats = playerAllStats[pd.notnull(playerAllStats['Points'])]
playerAllStats = playerAllStats[pd.notnull(playerAllStats['Games'])]
playerAllStats.head()</code></pre>
Output:
<a href="/images/blog/img1.png"><img src="/images/blog/img1.png" alt="img1"></a>
<pre><code>#Viewing all columns that we have in the dataframe. I'll be referencing this figure out which fields I need to clean
playerAllStats.columns</code></pre>
Output:
<pre><code>
Index(['', 'Year', 'Player', 'Position', 'Age', 'Team', 'Games',
 'GamesStarted', 'MinutesPlayed', 'PlayerEfficiencyRating',
 'TrueShootingPerct', 'ThreePointAttemptRate', 'FreeThrowRate',
 'OffensiveReboundPercentage', 'DefensiveReboundPercentage',
 'TotalReboundPercentage', 'AssistPercentage', 'StealPercentage',
 'BlockPercentage', 'TurnoverPercentage', 'UsagePercentage',
 'OffesniveWinShares', 'DefensiveWinShares', 'WinShares',
 'WinsSharesPer48Minutes', 'OffensiveBoxPlusMinus',
 'DefensiveBoxPlusMinus', 'BoxPlusMinus', 'ValueOverReplacement',
 'FieldGoals', 'FieldGoalAttempts', 'FieldGoalPercentage',
 'ThreePointFieldGoals', 'ThreePointFieldGoalAttempts',
 'ThreePointFieldGoalPercentage', 'TwoPointFieldGoals',
 'TwoPointFieldGoalAttempts', 'TwoPointFieldGoalPercentage',
 'EffectiveFieldGoalPercentage', 'FreeThrows', 'FreeThrowAttempts',
 'FreeThrowPercentage', 'OffensiveRebounds', 'DefensiveRebounds',
 'TotalRebounds', 'Assists', 'Steals', 'Blocks', 'Turnovers',
 'PersonalFouls', 'Points', 'year_start', 'year_end', 'position',
 'height', 'weight', 'birth_date', 'college', 'feet', 'inches',
 'heightInches', 'PointsPerGame', 'TotalReboundsPerGame',
 'AssistsPerGame'],
 dtype='object', name=0)</code></pre>
Cleaning data even more
<ul>
<li>Converting height to inches (so we can use it if we decide to)</li>
<li>Converting some of the stats to a numeric value so we can do analysis</li>
<li>Calculating points per game, total rebounds per game, and assists per game</li>
</ul>
<pre><code>playerAllStats['feet'], playerAllStats['inches'] = playerAllStats['height'].str.split('-', 1).str
playerAllStats['feet'], playerAllStats['inches'], playerAllStats['weight'] = pd.to_numeric(playerAllStats.feet), \
 pd.to_numeric(playerAllStats.inches), pd.to_numeric(playerAllStats.weight)
playerAllStats['heightInches'] = playerAllStats['feet']*12 + playerAllStats['inches']

playerAllStats['Games'], playerAllStats['Points'] = pd.to_numeric(playerAllStats.Games), pd.to_numeric(playerAllStats.Points)
playerAllStats['MinutesPlayed'], playerAllStats['FieldGoalPercentage']= pd.to_numeric(playerAllStats.MinutesPlayed), pd.to_numeric(playerAllStats.FieldGoalPercentage)
playerAllStats['Games'], playerAllStats['TotalRebounds']= pd.to_numeric(playerAllStats.Games), pd.to_numeric(playerAllStats.TotalRebounds)
playerAllStats['Assists'], playerAllStats['WinsSharesPer48Minutes']= pd.to_numeric(playerAllStats.Assists), pd.to_numeric(playerAllStats.WinsSharesPer48Minutes)
playerAllStats['WinShares'], playerAllStats['Year'] = pd.to_numeric(playerAllStats.WinShares), pd.to_numeric(playerAllStats.Year)

playerAllStats['PointsPerGame'] = playerAllStats['Points']/ playerAllStats['Games']
playerAllStats['TotalReboundsPerGame'] = playerAllStats['TotalRebounds']/ playerAllStats['Games']
playerAllStats['AssistsPerGame'] = playerAllStats['Assists']/ playerAllStats['Games']
</code></pre>
Question: Can we create a linear regression model using points per game to estimate win shares per 48 minutes?
1. Create a plot with the 2 statistics
Before we create the model, let's plot the points per game vs win shares per 48 minutes. This will let us understand the data set and see if we need to remove outliers to improve model accuracy.
<pre><code>sns.regplot(x="WinsSharesPer48Minutes", y = "PointsPerGame", data=playerAllStats)</code></pre>
Output:
<a href="/images/blog/img2.png"><img src="/images/blog/img2.png" alt="img2"></a>
There's obviously some outliers and data that doesn't belong. Let's see what kind of data we're working with.
<pre><code>playerAllStats.describe()</code></pre>
Output:
<a href="/images/blog/img3.png"><img src="/images/blog/img3.png" alt="img3"></a>
There's a few things I see immediately that would make our data hard to plot.
<ul>
<li>
We have seasons from 11400...to minimize noise, we're only going to look at recent player's seasons. So I'm going to limit my analysis from 2014 to present.
</li>
<li>
We have players who've played 1 game all season. We're also going to limit the number of games a player has played in a given season to be >70% of all games. A given season has 82 games.
</li>
<li>
This isn't in the descriptive statistics above, but there are different positions in basketball. For the purpose of this analysis let's limit our data to include players who play Point Guard (PG) (or other positions along with PG). These are typically the players who score the most points, and it makes sense for us to group players who have even scoring potential to reduce skewness.
</li>
</ul>
<pre><code>#filtering on players who have played since 2014 and have played at least 70% of the games in a given season (assuming 82 games per season)
playerAllStatsSince2014 = playerAllStats.loc[(playerAllStats['Year'] >= 2014) &#x26;(playerAllStats['Games'] >= 58)] 
#filteirng on PG players
playerAllStatsSince2014PG = playerAllStatsSince2014[playerAllStatsSince2014['Position'].str.contains('PG')].reset_index(drop=True)
sns.regplot(x="WinsSharesPer48Minutes", y = "PointsPerGame", data=playerAllStatsSince2014PG)</code></pre>
Output:
<a href="/images/blog/img4.png"><img src="/images/blog/img4.png" alt="img4"></a>
The plot looks much better between points per game vs win shares per 48 minutes. Our data is in a format we can use. Let's move on to creating a training and test data set.
2. Splitting data into training and test set
We're going to split our data into dependent variables and an independent varaibles for both training and test sets using SKLearn's 'train_test_split' function.
<pre><code>playerAllStatsSince2014PG_WinShare = playerAllStatsSince2014PG['WinsSharesPer48Minutes']
playerAllStatsSince2014PG_PPG = playerAllStatsSince2014PG['PointsPerGame']

 #importing packages and splitting out data into training + testing sets
from sklearn.model_selection import train_test_split
train_x, test_x, train_y, test_y = train_test_split(playerAllStatsSince2014PG['PointsPerGame'], \
 playerAllStatsSince2014PG['WinsSharesPer48Minutes'], test_size=0.20, random_state=42)</code></pre>
3. Fitting the data to a linear regression and getting R-squared
Here we're creating a linear regression model called 'model_1.' We're going to fit the training data to this model. After that we're going to get the R-squared value. R-squared is a value between 0 and 1 which is a statistical measure of how close the data are to the fitted regression line. 1 = perfect fit, 0 = no fit
<pre><code>model_1 = LinearRegression()
model_1.fit(train_x.reshape(-1, 1), train_y.reshape(-1, 1))
model_1.score(train_x.reshape(-1, 1), train_y.reshape(-1, 1))</code></pre>
Output:
<pre><code>0.49632072484943907</code></pre>
An r-squared of 0.496 tells us we have a moderately effective fit. Since we're also going to be creating a multi-variate model, let's also calculate Adjusted R-squared and Root Mean Square Error (RMSE) so we have more statistics to compare. The below code calculates R-squared and Adjusted R-squared.
<pre><code>X = train_x.values.reshape(-1, 1)
y = train_y.values.reshape(-1, 1)

# compute with formulas from the theory
yhat = model_1.predict(X)
SS_Residual = sum((y-yhat)**2)
SS_Total = sum((y-np.mean(y))**2)
r_squared = 1 - np.divide(SS_Residual,SS_Total)
#r_squared = 1 - np.exp((float(SS_Residual))/SS_Total)
adjusted_r_squared = 1 - (1-r_squared)*(len(y)-1)/(len(y)-X.shape[1]-1)
print(r_squared, adjusted_r_squared)</code></pre>
Output:
<pre><code>[0.49632072] [0.49326812]</code></pre>
4. Estimate values for the test data
Last thing to do is to estimate Win Shares with our test data. We'll compare our estimate to the true values and calculate the Root Mean Square Error.
<pre><code>model_1_predict_values = model_1.predict(test_x.values.reshape(-1, 1))
model_1_predict = pd.DataFrame({'Points Per Game': test_x.values, 'Actual Win Share': test_y.values,'Predicted Win Share': model_1_predict_values.reshape(1, -1)[0]})</code></pre>
The code below calculate Root Mean Square Error. The lower the Root Mean Square Error the closer our estimations are to the real values.
<pre><code>#Root Mean square error on test dataset
np.sqrt(np.mean(np.square(model_1_predict['Actual Win Share'] - \
 model_1_predict['Predicted Win Share'])))</code></pre>
Output:
<pre><code>0.0523034857831081</code></pre>
Question: Can we create a linear regression model using points per game, total rebounds per game, and assists per game to estimate win shares per 48 minutes?
We'll go through the same process as above (less the data exploration, which we do not need to do 2 times). We'll create the model and generate the statistics. Then we'll compare the single variable to the multivariable model.
<pre><code>playerAllStatsSince2014PG_Ys = playerAllStatsSince2014PG['WinsSharesPer48Minutes']
playerAllStatsSince2014PG_Xs = playerAllStatsSince2014PG[['PointsPerGame', 'TotalReboundsPerGame', 'AssistsPerGame']]

#splitting test and training data
train_x, test_x, train_y, test_y = train_test_split(playerAllStatsSince2014PG_Xs, playerAllStatsSince2014PG_Ys, test_size=0.20, random_state=42)

#fitting model
model_1 = LinearRegression()
model_1.fit(train_x, train_y)
model_1.score(train_x, train_y)
</code></pre>
Output:
<pre><code>0.5060498809063979</code></pre>
It looks like this model's R-square value is already higher that our single variable. Next we'll calculate the model's estimate for Win Shares.
<pre><code>model_1_predict_values = model_1.predict(test_x)
model_1_predict = pd.DataFrame({'Actual Win Share': test_y.values,'Predicted Win Share': model_1_predict_values.reshape(1, -1)[0]})
#Root Mean square error on test dataset
np.sqrt(np.mean(np.square(model_1_predict['Actual Win Share'] - model_1_predict['Predicted Win Share'])))</code></pre>
Output:
<pre><code>0.05174714479057624</code></pre>
<pre><code>X = train_x.values
y = train_y.values

# compute with formulas from the theory
yhat = model_1.predict(X)
SS_Residual = sum((y-yhat)**2)
SS_Total = sum((y-np.mean(y))**2)
r_squared = 1 - np.divide(SS_Residual,SS_Total)
#r_squared = 1 - np.exp((float(SS_Residual))/SS_Total)
adjusted_r_squared = 1 - (1-r_squared)*(len(y)-1)/(len(y)-X.shape[1]-1)
print(r_squared, adjusted_r_squared)</code></pre>
Output:
<pre><code>0.5060498809063977 0.49695877442001235</code></pre>
Comparing single variable to multivariate linear regression model
Below is a summary of the test statistics of the two models we created. Across all 3 statistics, the multivariate model performs slightly better than the single variable.
<table>
<thead>
<tr>
<th>statistic</th>
<th>single</th>
<th>multi</th>
</tr>
</thead>
<tbody>
<tr>
<td>r-squared</td>
<td>0.49632</td>
<td>0.50604</td>
</tr>
<tr>
<td>adjusted r-squared</td>
<td>0.49326</td>
<td>0.49695</td>
</tr>
<tr>
<td>RMSE</td>
<td>0.0523</td>
<td>0.0517</td>
</tr>
</tbody>
</table>

Which news publications have the most positive headlines? (simple sentiment analysis with Textblob)

Introduction to Linear Regression using Python

<a href="#collapseExample">+ Click for more information</a>
Why did we do this?
A lot of interviews have a take-home case component. This is different than a lot of the prepping we provide because this is challenging you to think end-to-end, on what is probably a real business problem the organization you're interviewing for is facing.
How can I use this?
We have provided the data (downloaded from Kaggle, thank you Kaggle!), a mock prompt (similar to ones experienced at top-tier companies), a potential presentation (which could be thought of as a solution), and the work we did to get to the solution. We hope that if you run into a similar scenario or want to extra practice, you can use our work to guide you in the right direction!
The data
It was really hard to find a real dataset that we could use as a "practice case." We ended up finding data on <a href="https://www.kaggle.com/olistbr/brazilian-ecommerce#olist_orders_dataset.csv">Kaggle!</a> You can download the data there, or you can visit the links below. We didn't use all of the data provided in this practice set by Kaggle, so below we only include ones we used to get to the solution.
<ul>
<li><a href="https://github.com/erood/interviewqs.com_code_snippets/blob/master/Case_1/olist_orders_dataset.csv">Orders dataset</a>: Provide information for each item ordered</li>
<li><a href="https://github.com/erood/interviewqs.com_code_snippets/blob/master/Case_1/olist_order_items_dataset.csv">Order items dataset</a>: Information for items within each order and the cost to ship and price broken out for each item within an order.</li>
<li><a href="https://github.com/erood/interviewqs.com_code_snippets/blob/master/Case_1/olist_order_payments_dataset.csv">Order payments dataset</a>: Provides information regarding payments made on each order. Make sure you aggregate total payment for each order to get unique order price.</li>
<li><a href="https://github.com/erood/interviewqs.com_code_snippets/blob/master/Case_1/olist_products_dataset.csv">Product dataset</a>: Provides information about the product.</li>
<li><a href="https://github.com/erood/interviewqs.com_code_snippets/blob/master/Case_1/product_category_name_translation.csv">Product category name translated dataset</a>: This dataset is Brazilian, so all categories are in Portuguese, join category name on this table to get the translated category.</li>
<li><a href="https://github.com/erood/interviewqs.com_code_snippets/blob/master/Case_1/olist_order_reviews_dataset.csv">Order reviews dataset</a>: This table has review information for each order.</li>
<li><a href="https://github.com/erood/interviewqs.com_code_snippets/blob/master/Case_1/olist_customers_dataset.csv">Customers dataset dataset</a>: This dataset has information regarding customer_id, which links directly to order_id in the orders dataset. However, to get the unique customer id for each order you need to link to this table.</li>
</ul>
<h3>The prompt</h3>
You’re a Data Scientist / Business Analyst working for a new eCommerce company called A&#x26;B Co. (similar to Amazon) and you’ve been asked to prepare a presentation for the Vice President of Sales and the Vice President of Operations that summarizes sales and operations thus far. The summary should include (at a minimum) a summary of current state the business, current customer satisfaction, and a proposal of 2-3 areas where the company can improve.
Here are some facts:
<ul>
<li>It’s currently September 2018 (e.g., you can ignore all data after September 2018)</li>
<li>The company’s inception was January 2017 (so you can ignore all data before January 2017)</li>
<li>Company is US-based, but launched in Brazil (which is why some information is in Portuguese)</li>
<li>You can assume all orders are delivered (so ignore the order state field)</li>
</ul>
Your presentation should not have more than 10 slides of content, and the presentation itself should only take ~15 minutes.
<h3>The data</h3>
Note all data was provided by <a href="https://www.kaggle.com/olistbr/brazilian-ecommerce#olist_orders_dataset.csv">Kaggle</a>. Feel free to read about each file on Kaggle (or by clicking the "Click for more infomation" button above), however you can download the data by clicking on each link below.
<ul>
<li><a href="https://github.com/erood/interviewqs.com_code_snippets/blob/master/Case_1/olist_orders_dataset.csv">Orders dataset</a></li>
<li><a href="https://github.com/erood/interviewqs.com_code_snippets/blob/master/Case_1/olist_order_items_dataset.csv">Order items dataset</a></li>
<li><a href="https://github.com/erood/interviewqs.com_code_snippets/blob/master/Case_1/olist_order_payments_dataset.csv">Order payments dataset</a></li>
<li><a href="https://github.com/erood/interviewqs.com_code_snippets/blob/master/Case_1/olist_products_dataset.csv">Product dataset</a></li>
<li><a href="https://github.com/erood/interviewqs.com_code_snippets/blob/master/Case_1/product_category_name_translation.csv">Product category name translated dataset</a></li>
<li><a href="https://github.com/erood/interviewqs.com_code_snippets/blob/master/Case_1/olist_order_reviews_dataset.csv">Order reviews dataset</a></li>
<li><a href="https://github.com/erood/interviewqs.com_code_snippets/blob/master/Case_1/olist_customers_dataset.csv">Customers dataset dataset</a></li>
</ul>
<h3>A solution</h3>
This is a potential solution to address the problems outlined in the prompt. Note this is not the only solution, nor is it necessarily the best. The goal of providing you this is to give you an idea of how we would go about solving this problem and how we would present this to an interview pannel.
<h3>The work!</h3>
Below are the code from our iPython notebook, the work we did in Google Sheets (how we created most of the charts), and some thoughts we had on solving case.
Python notebook
You can view the Jupyter notebook we used <a href="https://github.com/erood/interviewqs.com_code_snippets/blob/master/Case_1/interviewqs_case_1.ipynb">here.</a>
Google sheets
A link to the <a href="https://docs.google.com/spreadsheets/d/1CSi8aWCDiC5weTmFOek1vFVtvLcgriB92HRxVJeLR2E/edit?usp=sharing">spreadsheet</a>.
Most of the charts / quick side analysis I did was using Google sheets. Python charts are nice, but I find Google sheets generally quicker/more flexible for basic charts. For a case, you typically get 24-48 hours and you might end up spending a lot of time trying to make charts look "pretty". I found it's best to use Python for data aggregation/manipulation and use a spreadsheet software to make the more simple charts (unless otherwise specified in prompt).
Our thought process
Build a framework to answer the questions. If you’re not sure what the questions are, create questions for yourself to answer. It makes the process of digging for data so much easier.
It’s hard to come up with an answer if you don’t know what the questions are. This point seems like a no-brainer, but it’s good to make sure that you’ve created a structure to answer the key points. If you’re not sure what the question at hand is, you might need to play with the data a bit to understand what seems to be the problem at hand.
For this particular case the ask is really clear, we need to create a summary which includes current state the business, current customer satisfaction, and a proposal for an area where the company can improve. The questions we came up with (and some answers listed below) given the ask are:
<ul>
<li>
What should be shown to reflect current state of business? (Probably something to do with product growth and amount of money)
</li>
<li>
Revenue → how much money are we making?
</li>
<li>
Volume of sales → how many orders are we getting?
</li>
<li>
Customer summary based on spend + behavior
</li>
<li>
How should we present customer satisfaction?
</li>
<li>
Is customer satisfaction a problem? Do initial analysis to figure it out?
</li>
<li>
Does customer satisfaction reflect our current business state? (e.g. if business is down, does customer satisfaction also go down?, if business is boomin’ is customer satisfaction higher?)
</li>
<li>
Should I focus on sales area of improvement or customer satisfaction area of improvement?
</li>
<li>
This depends on what bullet points 1 + 2 yield, also depends on customer churn
</li>
</ul>
Questions don’t always have analytical answers, sometimes questions lead to more questions. This is of course a high-level framework, but the above should give you and idea of how we went through solving this! (the details of course are provided in the code/presentation material)

You’re a Data Scientist / Business Analyst working for a new eCommerce company called A&B Co. (similar to Amazon) and you’ve been asked to prepare...

Case Study Example 1:An eCommerce Company Evaluation

<h3>Inspiration/base dataset</h3>
Kaggle provides a great <a href="https://www.kaggle.com/snapcrack/all-the-news">dataset</a> containing news headlines for most major publications. I decided to run some simple sentiment analysis using <a href="https://textblob.readthedocs.io/en/dev/">Textblob</a>, a Python library for processing textual data, that comes with some pre-trained sentiment classifiers. One could of course train their own model, and probably obtain more accurate results overall, but I wasn't able to quickly fine a clean dataset of news headlines tagged with sentiment. Textblob should work fine for comparing the publications relative to each other.
To start, from Kaggle we're given a table as follows:
<table>
<thead>
<tr>
<th></th>
<th>author</th>
<th>content</th>
<th>date</th>
<th>month</th>
<th>publication</th>
<th>title</th>
<th>url</th>
<th>year</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>Avi Selk</td>
<td>Uber driver Keith Avila picked up a p...</td>
<td>2016-12-30</td>
<td>12.0</td>
<td>Washington Post</td>
<td>An eavesdropping Uber driver saved his 16-year...</td>
<td><a href="https://web.archive.org/web/20161231004909/htt">https://web.archive.org/web/20161231004909/htt</a>...</td>
<td>2016.0</td>
</tr>
<tr>
<td>1</td>
<td>Sarah Larimer</td>
<td>Crews on Friday continued to search L...</td>
<td>2016-12-30</td>
<td>12.0</td>
<td>Washington Post</td>
<td>Plane carrying six people returning from a Cav...</td>
<td><a href="https://web.archive.org/web/20161231004909/htt">https://web.archive.org/web/20161231004909/htt</a>...</td>
<td>2016.0</td>
</tr>
<tr>
<td>2</td>
<td>Renae Merle</td>
<td>When the Obama administration announced a...</td>
<td>2016-12-30</td>
<td>12.0</td>
<td>Washington Post</td>
<td>After helping a fraction of homeowners expecte...</td>
<td><a href="https://web.archive.org/web/20161231004909/htt">https://web.archive.org/web/20161231004909/htt</a>...</td>
<td>2016.0</td>
</tr>
<tr>
<td>3</td>
<td>Chelsea Harvey</td>
<td>This story has been updated. A new law in...</td>
<td>2016-12-30</td>
<td>12.0</td>
<td>Washington Post</td>
<td>Yes, this is real: Michigan just banned bannin...</td>
<td><a href="https://web.archive.org/web/20161231004909/htt">https://web.archive.org/web/20161231004909/htt</a>...</td>
<td>2016.0</td>
</tr>
<tr>
<td>4</td>
<td>Christopher Ingraham</td>
<td>The nation’s first recreational marijuana...</td>
<td>2016-12-29</td>
<td>12.0</td>
<td>Washington Post</td>
<td>What happened in Washington state after voters...</td>
<td><a href="https://web.archive.org/web/20161231004909/htt">https://web.archive.org/web/20161231004909/htt</a>...</td>
<td>2016.0</td>
</tr>
</tbody>
</table>
After poking around the n-counts by year, it looks like 2016-2017 are the only years with decent data >~2000 headlines across each publication. So, I filtered the dataset to these two years:
<pre><code>#years came in string.0 format, just filtering them as they are for now
years = ['2016.0','2017.0']
df = df.loc[df['year'].isin(years)] </code></pre>
<h3>Running our data through Textblob to determine news headline sentiment</h3>
Great, so now we have a pretty clean dataframe containing headlines by news publication. Next, I'll create a utility function to log the polarity (ranging from -1 as most negative to 1 as most positive) of each headline
<pre><code>#used to analyze sentiment w/ textblob, leaving the raw output
def analize_sentiment_raw(headline):
 '''
 Function to lassify the polarity of a tweet
 using textblob.
 '''
 analysis = TextBlob(headline)
 #polarity = negative (-1) to positive (1)
 return analysis.sentiment.polarity </code></pre>
Next, we'll just feed the headlines into the function we created above:
<pre><code>df['SA'] = np.array([ analize_sentiment_raw(headline) for headline in df['title']])</code></pre>
We can now see the polarity score (again ranging from -1 to 1) reflected in our dataframe:
<table>
<thead>
<tr>
<th></th>
<th>title</th>
<th>publication</th>
<th>author</th>
<th>date</th>
<th>year</th>
<th>month</th>
<th>url</th>
<th>content</th>
<th>SA</th>
<th>publish_year</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>An eavesdropping Uber driver saved his 16-year...</td>
<td>Washington Post</td>
<td>Avi Selk</td>
<td>2016-12-30</td>
<td>2016.0</td>
<td>12.0</td>
<td><a href="https://web.archive.org/web/20161231004909/htt">https://web.archive.org/web/20161231004909/htt</a>...</td>
<td>Uber driver Keith Avila picked up a p...</td>
<td>0.00</td>
<td>2016</td>
</tr>
<tr>
<td>1</td>
<td>Plane carrying six people returning from a Cav...</td>
<td>Washington Post</td>
<td>Sarah Larimer</td>
<td>2016-12-30</td>
<td>2016.0</td>
<td>12.0</td>
<td><a href="https://web.archive.org/web/20161231004909/htt">https://web.archive.org/web/20161231004909/htt</a>...</td>
<td>Crews on Friday continued to search L...</td>
<td>-0.40</td>
<td>2016</td>
</tr>
<tr>
<td>2</td>
<td>After helping a fraction of homeowners expecte...</td>
<td>Washington Post</td>
<td>Renae Merle</td>
<td>2016-12-30</td>
<td>2016.0</td>
<td>12.0</td>
<td><a href="https://web.archive.org/web/20161231004909/htt">https://web.archive.org/web/20161231004909/htt</a>...</td>
<td>When the Obama administration announced a...</td>
<td>-0.05</td>
<td>2016</td>
</tr>
<tr>
<td>3</td>
<td>Yes, this is real: Michigan just banned bannin...</td>
<td>Washington Post</td>
<td>Chelsea Harvey</td>
<td>2016-12-30</td>
<td>2016.0</td>
<td>12.0</td>
<td><a href="https://web.archive.org/web/20161231004909/htt">https://web.archive.org/web/20161231004909/htt</a>...</td>
<td>This story has been updated. A new law in...</td>
<td>0.20</td>
<td>2016</td>
</tr>
<tr>
<td>4</td>
<td>What happened in Washington state after voters...</td>
<td>Washington Post</td>
<td>Christopher Ingraham</td>
<td>2016-12-29</td>
<td>2016.0</td>
<td>12.0</td>
<td><a href="https://web.archive.org/web/20161231004909/htt">https://web.archive.org/web/20161231004909/htt</a>...</td>
<td>The nation’s first recreational marijuana...</td>
<td>0.00</td>
<td>2016</td>
</tr>
</tbody>
</table>
<h3>Pivoting the data to rank news publications by headline sentiment</h3>
Now that we have sentiment attached to each news headline as reflected in col 'SA' above, we can pivot to summarize the publications with the most positive/negative headlines:
<pre><code>#grab a slice of the original df with just publication and polarity score
df_clean = df[['SA', 'publication']]
#pivoting to grab the average sentiment by publication
df_clean2 = df_clean.groupby(['publication']).mean().reset_index()
#preview the dataframe, sorting by most positive
df_clean2.sort_values(['SA'],ascending=False).head(15)</code></pre>
Below, we can see that (according to Textblob's pre-trained classifier) Business Insider has the most positive headlines on average, with Breitbart coming in at the bottom with the most negative headlines.
<table>
<thead>
<tr>
<th></th>
<th>publication</th>
<th>SA</th>
</tr>
</thead>
<tbody>
<tr>
<td>2</td>
<td>Business Insider</td>
<td>0.060214</td>
</tr>
<tr>
<td>12</td>
<td>Vox</td>
<td>0.051213</td>
</tr>
<tr>
<td>7</td>
<td>NPR</td>
<td>0.043666</td>
</tr>
<tr>
<td>11</td>
<td>Talking Points Memo</td>
<td>0.035564</td>
</tr>
<tr>
<td>6</td>
<td>Guardian</td>
<td>0.032476</td>
</tr>
<tr>
<td>3</td>
<td>Buzzfeed News</td>
<td>0.028023</td>
</tr>
<tr>
<td>0</td>
<td>Atlantic</td>
<td>0.026798</td>
</tr>
<tr>
<td>13</td>
<td>Washington Post</td>
<td>0.022752</td>
</tr>
<tr>
<td>10</td>
<td>Reuters</td>
<td>0.020006</td>
</tr>
<tr>
<td>9</td>
<td>New York Post</td>
<td>0.019992</td>
</tr>
<tr>
<td>5</td>
<td>Fox News</td>
<td>0.016228</td>
</tr>
<tr>
<td>4</td>
<td>CNN</td>
<td>0.015503</td>
</tr>
<tr>
<td>8</td>
<td>National Review</td>
<td>0.014110</td>
</tr>
<tr>
<td>1</td>
<td>Breitbart</td>
<td>0.009544</td>
</tr>
</tbody>
</table>
Next, lets check out the n-counts on these, and add a view that shows the % of articles that are positive (score >0), negative (score 0, 1, 0)
df_clean['neutral'] = np.where(df['SA'] == 0, 1, 0)
df_clean['negative'] = np.where(df['SA'] &#x3C; 0, 1, 0)
df_clean2 = df_clean[['publication','positive','neutral','negative']]
df_clean3 = df_clean2.groupby(['publication']).sum().reset_index()
df_clean3.sort_values(['positive'],ascending=False).head(15)
<pre><code>
| | publication | positive | neutral | negative |
| -- | ------------------- | -------- | ------- | -------- |
| 0 | Atlantic | 1435 | 4781 | 963 |
| 1 | Breitbart | 5469 | 13536 | 4701 |
| 2 | Business Insider | 2585 | 2852 | 1320 |
| 3 | Buzzfeed News | 1354 | 2549 | 951 |
| 4 | CNN | 1591 | 5056 | 1279 |
| 5 | Fox News | 990 | 2493 | 839 |
| 6 | Guardian | 2366 | 4558 | 1700 |
| 7 | NPR | 3451 | 6359 | 2010 |
| 8 | National Review | 1203 | 4064 | 904 |
| 9 | New York Post | 4677 | 9196 | 3573 |
| 10 | Reuters | 2423 | 6600 | 1664 |
| 11 | Talking Points Memo | 670 | 1499 | 393 |
| 12 | Vox | 1730 | 2134 | 1014 |
| 13 | Washington Post | 3175 | 5555 | 2348 |

Great, we can now see the # of headlines Textblob identified as positive/neutral/negative. Lets add some %s into this df too:

```python
#fields for positive/neutral/negative percents
df_clean3['positive_perc'] = \
df_clean3['positive']/(df_clean3['positive']+df_clean3['neutral']+df_clean3['negative'])
df_clean3['neutral_perc'] = \
df_clean3['neutral']/(df_clean3['positive']+df_clean3['neutral']+df_clean3['negative'])
df_clean3['negative_perc'] = \
df_clean3['negative']/(df_clean3['positive']+df_clean3['neutral']+df_clean3['negative'])
#sort and preview dataframe
df_clean3.sort_values(['positive_perc'],ascending=False).head(15)
</code></pre>
<table>
<thead>
<tr>
<th></th>
<th>publication</th>
<th>positive</th>
<th>neutral</th>
<th>negative</th>
<th>positive_perc</th>
<th>neutral_perc</th>
<th>negative_perc</th>
</tr>
</thead>
<tbody>
<tr>
<td>2</td>
<td>Business Insider</td>
<td>2585</td>
<td>2852</td>
<td>1320</td>
<td>0.382566</td>
<td>0.422081</td>
<td>0.195353</td>
</tr>
<tr>
<td>12</td>
<td>Vox</td>
<td>1730</td>
<td>2134</td>
<td>1014</td>
<td>0.354654</td>
<td>0.437474</td>
<td>0.207872</td>
</tr>
<tr>
<td>7</td>
<td>NPR</td>
<td>3451</td>
<td>6359</td>
<td>2010</td>
<td>0.291963</td>
<td>0.537986</td>
<td>0.170051</td>
</tr>
<tr>
<td>13</td>
<td>Washington Post</td>
<td>3175</td>
<td>5555</td>
<td>2348</td>
<td>0.286604</td>
<td>0.501444</td>
<td>0.211952</td>
</tr>
<tr>
<td>3</td>
<td>Buzzfeed News</td>
<td>1354</td>
<td>2549</td>
<td>951</td>
<td>0.278945</td>
<td>0.525134</td>
<td>0.195921</td>
</tr>
<tr>
<td>6</td>
<td>Guardian</td>
<td>2366</td>
<td>4558</td>
<td>1700</td>
<td>0.274351</td>
<td>0.528525</td>
<td>0.197124</td>
</tr>
<tr>
<td>9</td>
<td>New York Post</td>
<td>4677</td>
<td>9196</td>
<td>3573</td>
<td>0.268084</td>
<td>0.527112</td>
<td>0.204803</td>
</tr>
<tr>
<td>11</td>
<td>Talking Points Memo</td>
<td>670</td>
<td>1499</td>
<td>393</td>
<td>0.261514</td>
<td>0.585090</td>
<td>0.153396</td>
</tr>
<tr>
<td>1</td>
<td>Breitbart</td>
<td>5469</td>
<td>13536</td>
<td>4701</td>
<td>0.230701</td>
<td>0.570995</td>
<td>0.198304</td>
</tr>
<tr>
<td>5</td>
<td>Fox News</td>
<td>990</td>
<td>2493</td>
<td>839</td>
<td>0.229061</td>
<td>0.576816</td>
<td>0.194123</td>
</tr>
<tr>
<td>10</td>
<td>Reuters</td>
<td>2423</td>
<td>6600</td>
<td>1664</td>
<td>0.226724</td>
<td>0.617573</td>
<td>0.155703</td>
</tr>
<tr>
<td>4</td>
<td>CNN</td>
<td>1591</td>
<td>5056</td>
<td>1279</td>
<td>0.200732</td>
<td>0.637901</td>
<td>0.161368</td>
</tr>
<tr>
<td>0</td>
<td>Atlantic</td>
<td>1435</td>
<td>4781</td>
<td>963</td>
<td>0.199889</td>
<td>0.665970</td>
<td>0.134141</td>
</tr>
<tr>
<td>8</td>
<td>National Review</td>
<td>1203</td>
<td>4064</td>
<td>904</td>
<td>0.194944</td>
<td>0.658564</td>
<td>0.146492</td>
</tr>
</tbody>
</table>
<h3>Visualizing news publication sentiment</h3>
Using plotly, making a quick bar chart to visualize the news publication sentiment shown in our pivoted data above:
<pre><code>#bar chart
x = df_clean2['publication']
y = df_clean2['SA']

data = [
 go.Bar(
 x=x,
 y=y,
 marker=dict(
 color='#8bffc0f2',
 line=dict(
 color='#d8ffea96',
 width=1.5
 ),
 ),
 opacity=1
 )
]
layout = go.Layout(
 title='News publications with the most/least positive headlines',
 xaxis=dict(
 title='News publication'
 ),
 yaxis=dict(
 title='Sentiment (higher = more positive',
 ),
 margin=go.Margin(
 l=140,
 r=90,
 b=140,
 t=60,
 pad=4
 ),
)
fig = go.Figure(data=data, layout=layout)
py.iplot(fig, filename='news_pub_sentiment')</code></pre>
Voila! We were able to quickly use Textblob to classify and rank sentiment across hundreds of thousands of news headlines.
<a href="/images/blog/pic1.png"><img src="/images/blog/pic1.png" alt="pic1"></a>
For those interested, the jupyter notebook with the complete code can be found in our <a href="https://github.com/erood/interviewqs.com_code_snippets/blob/master/iPython_Tutorial_Notebooks/News_sentiment_analysis.ipynb">github repository</a>.

Interviewing for a role as a analyst? You've come to the right place! Below we've curated a list of data analyst interview questions from multiple sources to help make your preparation easier.
For a data analyst interview, the interviewer will ask a wide range of topics covering statistics, programming (Python and/or SQL), data modeling, and overall business acumen. This guide contains 45 data analyst interview questions, broken out by high-level topics.
If you're interested in practicing further (and having the option to receive solutions), sign up for our <a href="/">email newsletter</a>, where we send a few interview questions per week.
<h2>Statistics</h2>
Statistics are the guiding principles to collection, organizing, and interpreting data. A general familiarity with stats will be necessary for any data analyst. You don't necessarily have to be an expert in statistics, but use the job description to gauge the level of familiarity necessary.
<ul>
<li>What is r^2? What are some other metrics that could be better than r^2 and why?</li>
<li>What is sampling? Can you provide an example of a sampling method? Can you provide an example of a time in the past where you needed to use sampling?</li>
<li>What is Type I error and how is it different from Type II error?</li>
<li>How would you explain a linear regression to a non-technical person?</li>
<li>What are the assumptions of linear regression?</li>
<li>What is selection bias?</li>
<li>How would you explain a logistic regression to a non-technical person?</li>
<li>Be prepared to answer conditional probability questions and Bayesian probability questions.</li>
<li>What are the assumptions of a logistic regression?</li>
<li>What is a p-value?</li>
<li>What is a confidence interval and how do you interpret it?</li>
<li>How would you estimate the disease probability in a city, given the probability is very low nationwide?</li>
<li>In any 15-minute interval, there is a 20% probability that you will see at least one shooting star. What is the probability that you see at least one shooting star in the period of an hour?</li>
<li>You have an 50-50 mixture of two normal distributions with the same standard deviation. How far apart do the means need to be in order for this distribution to be bimodal?</li>
<li>A couple tells you that they have two children, at least one of which is a girl. What is the probability that they have two girls?</li>
<li>What is the probability of rolling a 4 or 7 for two 6 sided dice?</li>
<li>On a dating site, users can select 1 out of 10 adjectives to describe themselves. A match is declared between two users if they match on at least 4 adjectives. Given a pool of 10 independent users, what is the probabilty that at least two will be a match?</li>
<li>You have two coins, one of which is fair and comes up heads with a probability 1/2, and the other which is biased and comes up heads with probability 3/4. You randomly pick coin and flip it twice, and get heads both times. What is the probability that you picked the fair coin?</li>
</ul>
<h2>Programming</h2>
Another large portion of data analytics is aggregating and processing data. As a data analyst, having a familiarity with programming can provide you an advantage in the interview process. It's common for data analysts to be fluent in SQL/Excel, but often Python is not an explicit requirement. By knowing at least a little bit of programming you can stand out both in the interview process and on the job (e.g. using programming to automate work/analysis will go a long way). Below are some practice data science interview questions covering Python and general programming.
<ul>
<li>How would you clean a data set in Python (or any other language)?</li>
<li>How would you drop null values in a Pandas dataframe?</li>
<li>Given a list of tweets, determine the top 10 most used hashtags. You can assume the list is provided in a Python dataframe.</li>
<li>Write an algorithm that can calculate the square root of a number, using Python.</li>
<li>Explain some examples of automating data analysis/related processes using programming.</li>
<li>How would you replace all NaN values in a Pandas dataframe?</li>
<li>{Whiteboard} Create a simple Python function to take in an integer value, multiply it by 2, and append it to an array.</li>
<li>{Whiteboard} Create a simple Python function to determine if a given Python string contains a baseline string, 'as'. If the given string does contain 'as', print 'yes' otherwise print 'no'.</li>
</ul>
<h2>SQL</h2>
As a data analyst, being able to pull information directly from datatables will be very important to a lot of employers. SQL is the most common language for the job, and is becoming a standard requirement for many data analyst roles. If you can master basic/intermediate SQL this portion of the interview will be a breeze.
<ul>
<li>
What does the GROUP BY function in SQL do?
</li>
<li>
What is the difference between an inner join and a union?
</li>
<li>
How do you eliminate duplicate rows from a query result?
</li>
<li>
Given a impressions table with ad_id, click (a boolean indicator that the ad was clicked), and date, can you write a SQL query that will tell me the monthly click-through-rate all ads?
</li>
<li>
Can you write a query that returns the name of each department and a count of the number of employees in each? Below are the tables' schema:
</li>
<li>
EMPLOYEES containing: Emp_ID (Primary key) and Emp_Name
</li>
<li>
EMPLOYEE_DEPT containing: Emp_ID (Foreign key) and Dept_ID (Foreign key)
</li>
<li>
DEPTS containing: Dept_ID (Primary key) and Dept_Name
</li>
<li>
Explain to me a common use case of windowing functions.
</li>
<li>
What's the difference between a right join and a left join?
</li>
<li>
What is a subquery?
</li>
</ul>
<h2>Behavioral/Business scenarios</h2>
You can be a great data analyst, but if you're unable to communicate the results of your analysis, or you don't mesh well culturally that could cost you the job. Because of this, it's important to have a few situations prepared prior to interviewing that highlight your communication skills, character, and capabilities! Keep it cool, and let your personality shine through!
<ul>
<li>Tell me about a time where you had to present a detailed analysis to a high-level executive.</li>
<li>Introduce me to something you’re passionate about.</li>
<li>Tell me about a time you failed, and what you have learned from it.</li>
<li>Tell me about a time you had to influence someone without authority.</li>
<li>What have you done in your previous job that you are really proud of?</li>
<li>What’s a project you would want to work on at our company?</li>
<li>Tell me about a time where you resolved a conflict.</li>
<li>Tell me about a time when you took initiative.</li>
<li>Suppose you encounter a tedious, boring task. How would you deal with it and motivate yourself to complete it?</li>
<li>What are your top 5 predictions for the next 20 years?</li>
<li>What personality traits do you conflict with?</li>
</ul>

45 Data Analyst Interview Questions

College is obviously expensive, but is it still a wise investment?
We've all heard how expensive college is getting, along with plenty of criticism surrounding its value in a changing job market. Of course, there are many benefits beyond the monetary ones that should be considered when exploring college options, but for the purpose of this post I'm going to limit the scope and purely assess the financial benefit of attending college.
The main financial benefit of attending college is the earnings differential received by a college graduate over a high school graduate; <a href="http://www.payscale.com/college-roi/">Payscale</a> provides 20-year return on investment (ROI) figures for exactly that. The website compares the gain in median pay from graduating over a high school graduate across ~1,250 4 and 5-year educational institutions in the United States.
Below I'll compare the ROI of college to the return generated from
simply joining the workforce after high school, but investing college tuition costs* into the stock market (using S&#x26;P 500 as a proxy).
Option A - Skip college, enter workforce, and invest your saved tuition costs
Lets say we skip college and dump our saved money from avoided tuition costs into the stock market. The S&#x26;P 500 (our stock market proxy) has risen from ~435 at the beginning of 1993 to ~2,260 at the beginning of 2017. This represents a 520% return, which equates to an annualized return of ~7.1%** for 24 years. Note that we get 24 years for our investment return, rather than the 20 years college gets, since we can make our investment upon high school graduation and immediately realize returns.
Option B - Attend college, pay tuition, generate higher levels of income if you graduate
Here, we'll attend college as usual, and realize the average ROI across U.S. colleges. Using our Payscale ROI data mentioned above, and normalizing for graduation rates (ROI*graduation rate = Expected ROI), the average annualized ROI with financial aid is ~4.5%** (median 4.1%).
The winner?
As shown, on average, forgoing college costs, investing that money in equities, and beginning work would generate significantly higher returns than going to college. Below you can see the distribution of ROI across U.S. colleges, along with our 'Option A' comparison point.
<a href="/images/blog/College_roi_histogram.png"><img src="/images/blog/College_roi_histogram.png" alt="College roi histogram"></a>
A couple of additional stats:
<ul>
<li>Only ~10% of the ~1250 colleges listed on Payscale generate an average ROI higher than the 7.1% generated by foregoing college and investing that money. In other words, a student planning on enrolling in college would generate higher returns entering the labor force and investing tuition into the stock market 90% of the time.</li>
<li>If we remove financial aid, only ~2.5% of colleges generate a better ROI than our 'skip college invest in stock market' option!</li>
</ul>
Footnotes/disclaimers:
<ul>
<li>Of course, lenders won't give out large loans for an individual to skip college and dump tuition into the stock market, this is just a hypothetical scenario.</li>
<li>One could incorporate undergraduate population size of each school to volume-weight the ROIs provided by Payscale, for an improved average across the pool. (I did not feel like scraping undergrad population size for all schools listed).</li>
<li>Returns vary significantly by school and major. If you're trying to decide whether or not</li>
</ul>
to attend college from a financial point of view, I'd recommend being thoughtful about your school choice, major, potential financial aid, and ability to graduate enrolled with a degree from said school in said major.
As mentioned above, there are also many additional (non-financial) factors to consider.
<ul>
<li>I went to college, and would recommend it.</li>
</ul>

On average, foregoing college in favor of entering the workforce + investing tuition costs nets higher returns

When you’re querying most data sets, you’re most likely going to need to use a JOIN clause. This article will take you through when you want to use a join, the requirements of using a join, and the types of joins.
<h2>What is a SQL Join?</h2>
A SQL join is an instruction used within the SQL (Structured Query Language) to combine data from two table on a key. Let’s take you through an example of using a join in practice.
<h2>Example of when you want to use a SQL join</h2>
Say you have two tables relating to menu performance at a restaurant. The first table, daily_menu, provides you with information what’s on the menu for the day, the price of the item on the menu, and the number of unit sold. The primary key for this table is a combination of the date and item_id (these are the two fields that uniquely identify each record in the table). The second table, menu_items, provides you with information regarding the menu item name, if it’s vegetarian, and the average price of the item. The primary key for this table is the item_id. This is a simple example of a relational database.
<a href="/images/blog/sql_joins_1.png"><img src="/images/blog/sql_joins_1.png" alt="sql joins 1"></a>
Let us pretend you are an analyst for the restaurant chain, and you want to know how well the vegetarian items on the menu are selling doing compared to the non-vegetarian items. Given these two tables, you realize you cannot determine the performance based on daily_menu items alone, you also need to pull in the is_veg field from menu_items. You’ve identified a relationship between menu_items and daily_menu that can help you pull is_veg from menu_items and map it onto daily_menu. See graphic below.
<a href="/images/blog/sql_joins_2.png"><img src="/images/blog/sql_joins_2.png" alt="sql joins 2"></a>
How can you join daily_menu with menu_items in SQL? A perfect opportunity for a join! The next dilemma you run into is trying to figure out what type of join to use.
#We know we have 2 sets of data daily_menu and menu_items, we can pretend that daily_menu is Table A and menu_items is Table B. In a venn diagram, we essentially want to bring the two sets of data together.
<a href="/images/blog/sql_joins_3.png"><img src="/images/blog/sql_joins_3.png" alt="sql joins 3"></a>
There are 4 basic ways we can join this data together: inner join, left join, right join, or full join. We’ll go into each on in detail below.
<a href="/images/blog/sql_joins_4.png"><img src="/images/blog/sql_joins_4.png" alt="sql joins 4"></a>
<h2>INNER JOIN</h2>
An inner join lets us output records referenced in both Table A and Table B. If there is a mismatch of records in Table A and Table B, those records will not show up in the output of the query.
To do this in practice, if we take our example of daily_menu and menu_items here’s an example query of an inner join and the corresponding output.
<pre><code>SELECT 
 d.date, 
 i.name, 
 i.is_veg, 
 d.price, 
 d.units_sold 
FROM daily_menu d 
 INNER JOIN menu_items i 
 ON d.item_id = i.item_id </code></pre>
<a href="/images/blog/sql_joins_5.png"><img src="/images/blog/sql_joins_5.png" alt="sql joins 5"></a>
<h2>FULL OUTER JOIN</h2>
FULL JOIN or FULL OUTER JOIN, lists all records from both table, regardless of whether the records have a match in the opposite table. Again, let's use the daily_menu and menu_item tables from above to practice. If we do a full outer join on item_id, we’re keeping all primary ids of both tables as well as the corresponding columns we select. The SQL and output table are below.
<pre><code>SELECT 
 d.date, 
 i.name, 
 i.is_veg, 
 d.price, 
 d.units_sold 
FROM daily_menu d 
 FULL OUTER JOIN menu_items i 
 ON d.item_id = i.item_id </code></pre>
<a href="/images/blog/sql_joins_6.png"><img src="/images/blog/sql_joins_6.png" alt="sql joins 6"></a>
<h2>LEFT JOIN</h2>
A left join, as illustrated in our SQL joins picture above, will return all records from Table A, and any matching records from Table B. Using the daily_menu and menu_item tables, below is an example query for a left join and the resulting output.
<pre><code>SELECT 
 d.date, 
 i.name, 
 i.is_veg, 
 d.price, 
 d.units_sold 
FROM daily_menu d 
 LEFT JOIN menu_items i 
 ON d.item_id = i.item_id </code></pre>
<a href="/images/blog/sql_joins_7.png"><img src="/images/blog/sql_joins_7.png" alt="sql joins 7"></a>
<h2>RIGHT JOIN</h2>
A right join, contrast to a left join, will return all records from Table B and any matching records in Table A. Again, we’re going to use daily_menu and menu_item tables to write an example query for a right join and the resulting output.
<pre><code>SELECT 
 d.date, 
 i.name, 
 i.is_veg, 
 d.price, 
 d.units_sold 
FROM daily_menu d 
 RIGHT JOIN menu_items i 
 ON d.item_id = i.item_id </code></pre>
<a href="/images/blog/sql_joins_8.png"><img src="/images/blog/sql_joins_8.png" alt="sql joins 8"></a>
<h2>What’s next?</h2>
The basic SQL joins described above let you tie different pieces of data together, and allow you to start asking and answering more challenging questions! You’ll be using these often, especially if you have multiple data sources, or data that lives in different places.

SQL Joins

The goal of this post is to introduce you to SQL and take you through the minimum requirements of building a SQL query to pull data. We'll be coming out with SQL 201 soon, which will cover more advanced topics.
<h2>What is SQL and how can I use it?</h2>
SQL stands for Structured Query Language. SQL lets you access and manipulate data within databases. Not only can you pull data with SQL, you can also do most modifying actions to a database including, inserting, deleting, updating records.
<h2>Database tables</h2>
A database typically contains one or more tables, each can be uniquely identified by a table name. Below is an example table that we’ll be using to practice our SQL skills. The table below has the daily menu at a restaurant.
<a href="/images/blog/sql_101_1.png"><img src="/images/blog/sql_101_1.png" alt="sql 101 1"></a>
<h2>A basic SQL query</h2>
The query below is the most basic SQL query you can write to select data from a table.
<pre><code>SELECT * 
FROM daily_menu; </code></pre>
The SELECT statement indicates that you’d like to select data. The “*” operator indicates all, meaning that you want to select all columns from the table. FROM indicates where you want to select this data from.
<h2>SELECT statement</h2>
Within the “A basic SQL query” section above, you’ve learned that “*” operator allows you to select all of the columns from a given table. What if you only want to select 1-2 columns? You can specify that within the SELECT clause. See the example below, we’re only going to select all menu item names and their id’s within the daily_menu table.
<pre><code>SELECT item_name, item_id 
FROM daily_menu; </code></pre>
<a href="/images/blog/sql_101_2.png"><img src="/images/blog/sql_101_2.png" alt="sql 101 2"></a>
Note that within the SELECT statement we can also do data aggregations (e.g. sum, if clauses, windowing functions), we don’t just need to select unmanipulated columns.
<h2>SELECT DISTINCT statement</h2>
As you can see from above, when we select item_name and item_id we get duplicate records. If we want to remove the duplicate records we can use the DISTINCT statement to remove duplicates.
<pre><code>SELECT DISTINCT item_name, item_id 
FROM daily_menu; </code></pre>
<a href="/images/blog/sql_101_3.png"><img src="/images/blog/sql_101_3.png" alt="sql 101 3"></a>
Note that within the SELECT statement we can also do data aggregations (e.g. sum, if clauses, windowing functions), we don’t just need to select unmanipulated columns.
<h2>WHERE Clause</h2>
Now what if we wanted to select only items and their id’s that were served on a particular day? We can use the WHERE clause to do this! The WHERE clause is used to extract only records that fulfill a specified condition. Let's say we only want to select the items and their id’s if they were served on “2017-07-27” (comparing two strings, assuming date is string data type). We can do that with the query below.
<pre><code>SELECT item_name, item_id 
FROM daily_menu 
WHERE date = ‘2017-07-27’ </code></pre>
<a href="/images/blog/sql_101_4.png"><img src="/images/blog/sql_101_4.png" alt="sql 101 4"></a>
We can also do numeric comparisons with a where clause. Let’s return item_id’s and item_name’s where unit sold is less than 100.
<pre><code>SELECT item_name, item_id 
FROM daily_menu 
WHERE units_sold </code></pre>
<a href="/images/blog/sql_101_5.png"><img src="/images/blog/sql_101_5.png" alt="sql 101 5"></a>
<h2>WHERE Clause Operators</h2>
There are a few operators we can use within the WHERE clause. The below chart outlines all of the operators you can use to compare values.
<table>
<thead>
<tr>
<th>Operator</th>
<th>Description</th>
<th>Example</th>
</tr>
</thead>
<tbody>
<tr>
<td>=</td>
<td>Equal</td>
<td>WHERE units_sold or !=</td>
</tr>
<tr>
<td>></td>
<td>Greater than</td>
<td>WHERE units_sold > 100</td>
</tr>
<tr>
<td>=</td>
<td>Greater than or equal to</td>
<td>WHERE units_sold >= 100</td>
</tr>
<tr>
<td></td>
<td></td>
<td></td>
</tr>
</tbody>
</table>
<pre><code>SELECT item_name, item_id 
FROM daily_menu 
WHERE date = ‘2017-07-27’ 
 AND units_sold > 200 </code></pre>
<a href="/images/blog/sql_101_6.png"><img src="/images/blog/sql_101_6.png" alt="sql 101 6"></a>
We will select all item_name’s and item_id’s that were sold on 2017-07-27 OR has a price over $20.00.
<pre><code>SELECT item_name, item_id 
FROM daily_menu 
WHERE date = ‘2017-07-27’ 
 OR price > 20.00 </code></pre>
<a href="/images/blog/sql_101_7.png"><img src="/images/blog/sql_101_7.png" alt="sql 101 7"></a>
<h2>Conclusion</h2>
Hopefully now you feel confident in pulling simple data requests from a table. If you want more content like this, visit SQL 201 for a deeper dive into SQL commands.

Learn SQL fundamentals. Focused on SELECT, FROM, and WHERE clauses. Provided by InterviewQs, a mailing list for coding and data interview problems.

SQL 101 - SELECT, FROM, WHERE Clauses

Interviewing for a role as a data scientist or analyst? You've come to the right place! Below we've curated a list of data science interview questions from multiple sources to help make your preparation easier.
For a data science or data analyst interview, the interviewer will ask a wide range of topics covering statistics, programming (Python and SQL), data modeling (including machine learning), and overall business acumen. This guide contains 70 data science interview questions, broken out by high-level topics.
If you're interested in practicing further (and having the option to receive solutions), sign up for our <a href="/">email newsletter</a>, where we send a few interview questions per week.
<h3>Statistics</h3>
Statistics are the guiding principles to collection, organizing, and interpreting data, sounds pretty core to data science huh? Below are some data science interview questions covering statistics.
<ul>
<li>What is the importance of the Central Limit Theorem?</li>
<li>What is sampling? Can you provide an example of a sampling method? Can you provide an example of a time in the past where you needed to use sampling?</li>
<li>What is Type I error and how is it different from Type II error?</li>
<li>How would you explain a linear regression to a non-technical person?</li>
<li>What are the assumptions of linear regression?</li>
<li>What is selection bias?</li>
<li>How would you explain a logistic regression to a non-technical person?</li>
<li>Be prepared to answer conditional probability questions and Bayesian probability questions.</li>
<li>What are the assumptions of a logistic regression?</li>
<li>Explain collinearity to me.</li>
<li>What are key factors to running a successful A/B test?</li>
<li>What is a confidence interval and how do you interpret it?</li>
<li>How would you estimate the disease probability in a city given the probability is very low nationwide?</li>
<li>If you have a monthly collection of time series data, how can you tell if there is a "significant" difference between this month and previous month data?</li>
<li>In any 15-minute interval, there is a 20% probability that you will see at least one shooting star. What is the probability that you see at least one shooting star in the period of an hour?</li>
<li>You have an 50-50 mixture of two normal distributions with the same standard deviation. How far apart do the means need to be in order for this distribution to be bimodal?</li>
<li>A couple tells you that they have two children, at least one of which is a girl. What is the probability that they have two girls?</li>
<li>What is the probability of rolling a 4 or 7 for two 6 sided dice?</li>
<li>On a dating site, users can select 1 out of 10 adjectives to describe themselves. A match is declared between two users if they match on at least 4 adjectives. Given a pool of 10 independent users, what is the probabilty that at least two will be a match?</li>
<li>You have two coins, one of which is fair and comes up heads with a probability 1/2, and the other which is biased and comes up heads with probability 3/4. You randomly pick coin and flip it twice, and get heads both times. What is the probability that you picked the fair coin?</li>
</ul>
<h3>Modeling/Machine Learning</h3>
Understanding the fundamentals of modeling data will be important for your data science interview. A lot of data science is interpreting large data sets, and making sense of what is happening. Modeling can play a pivotal role in efficiently making sense of large, multidimensional data sets.
<ul>
<li>Can you explain the process of developing a ML algorithm.</li>
<li>How would you effectively represent data with 5 dimensions?</li>
<li>How is kNN different from k-means clustering?</li>
<li>How would you create a logistic regression model?</li>
<li>What is precision and recall? How do they relate to the ROC curve?</li>
<li>Explain the difference between L1 and L2 regularization methods.</li>
<li>What is one way that you would handle an imbalanced dataset that’s being used for prediction?</li>
<li>Do you think 50 small decision trees are better than a large one? Why?</li>
<li>Is it better to spend 5 days developing a 90% accurate solution, or 10 days for 100% accuracy?</li>
<li>What is random forest?</li>
<li>If you have two algorithms, how do you know one performs better than the other?</li>
<li>Given location data of golf balls in games, how would construct a model that can advise golfers where to aim?</li>
<li>You have data on all purchases of customers at a grocery store. Describe to me how you would program an algorithm that would cluster the customers into groups. How would you determine the appropriate number of clusters to include?</li>
<li>What error metric would you use to evaluate how good a binary classifier is? What if the classes are imbalanced? What if there are more than 2 groups?</li>
<li>What are various ways to predict a binary response variable? Can you compare two of them and tell me when one would be more appropriate? What’s the difference between these? (SVM, Logistic Regression, Naive Bayes, Decision Tree, etc.)</li>
<li>Given a database of all previous alumni donations to your university, how would you predict which recent alumni are most likely to donate?</li>
</ul>
<h3>Programming</h3>
Another large portion of data science is aggregating and processing the data. Most data sets are super messy, and require a lot of pre-processing, and this is why employers want data scientists to understand the fundamentals of programming. For these problems, explaining your logic outloud to the interviewer is the most important thing to do. Once you come up with a solution, you’ll probably be asked to explain the run-time of your algorithm and come up with a way to make your algorithm run faster. Below are some practice data science interview questions covering Python and general programming.
<ul>
<li>What is your favorite programming language and why?</li>
<li>Tell me about an algorithm you’ve recently created.</li>
<li>How would you clean a data set in Python (or any other language)?</li>
<li>{Whiteboard} Given a list A of objects, and another list B which is identical to A except that one element is removed, how would you find which element is missing?</li>
<li>{Whiteboard} Write a Python function that displays the first n Fibonacci numbers.</li>
<li>{Whiteboard} Write a function that prints the least integer that is not present in a given list and cannot be represented by the summation of the sub-elements of the list.</li>
<li>Given a binary tree and a node, print all cousins of given node. Note that siblings should not be printed.</li>
<li>Given a list of tweets, determine the top 10 most used hashtags.</li>
<li>You have a stream of data coming in of size n, but you don’t know what n is ahead of time. Can you write an algorithm that will take a random sample of k elements?</li>
<li>Can you write an algorithm that can calculate the square root of a number.</li>
</ul>
<h3>SQL</h3>
As a data scientist, being able to pull data from the source is very important to a lot of employers. Knowing basic SQL will help you glide right through the database questions of your interview with ease.
<ul>
<li>
What does the GROUP BY function in SQL do?
</li>
<li>
What is the difference between an inner join and a union?
</li>
<li>
How do you eliminate duplicate rows from a query result?
</li>
<li>
Given a impressions table with ad_id, click (a boolean indicator that the ad was clicked), and date, can you write a SQL query that will tell me the monthly click-through-rate all ads?
</li>
<li>
Can you write a query that returns the name of each department and a count of the number of employees in each? Below are the tables' schema:
</li>
<li>
EMPLOYEES containing: Emp_ID (Primary key) and Emp_Name
</li>
<li>
EMPLOYEE_DEPT containing: Emp_ID (Foreign key) and Dept_ID (Foreign key)
</li>
<li>
DEPTS containing: Dept_ID (Primary key) and Dept_Name
</li>
</ul>
<h3>Behavioral + Cultural fit</h3>
You can be a great data scientist or data analyst, but if you don’t mesh with the culture that could prevent you from succeeding in the role, or being hired in the first place. Because of this, it's important to have a few situations prepared prior to interviewing that highlight your character and capabilities! Keep it cool, and let your personality shine through!
<ul>
<li>Introduce me to something you’re passionate about.</li>
<li>Tell me about a time you failed, and what you have learned from it.</li>
<li>What have you done in your previous job that you are really proud of?</li>
<li>What’s a project you would want to work on at our company?</li>
<li>Tell me about a time where you resolved a conflict.</li>
<li>Tell me about a time when you took initiative.</li>
<li>Suppose you encounter a tedious, boring task. How would you deal with it and motivate yourself to complete it?</li>
<li>What are your top 5 predictions for the next 20 years?</li>
<li>What personality traits do you butt heads with?</li>
<li>If you had one superpower, what would it be?</li>
</ul>

A curated list of 70 data science interview questions

A step-by-step Python code example that shows how to add new column to Pandas DataFrame with default value. Provided by InterviewQs, a mailing list for coding and data interview problems.

Introduction to Machine Learning -- evaulating chemical composition of wine

Office space per person is shrinking
Offices have gotten more crowded 'open' in recent years, with the <a href="https://www.cbsnews.com/news/companies-are-packing-workers-in-like-sardines/">average square feet per person</a> shrinking ~33% from 225 in 2010 to just ~150 in 2017. This can generally be attributed to companies adopting 'open offices', citing benefits of 'increased productivity/collaboration/face time'.
Of course, there <a href="https://arstechnica.com/science/2018/07/in-open-offices-workers-chat-70-less-are-less-productive-and-email-more/">are</a> <a href="https://www.inc.com/geoffrey-james/science-just-proved-that-open-plan-offices-destroy-productivity.html">plenty</a> of <a href="https://bigthink.com/robby-berman/open-office-plans-actually-reduce-face-to-face-interaction">studies</a> analyzing the effect open offices have on worker productivity, with generally negative results. Despite the research, companies are still marching towards increasingly crowded 'open' offices. Why?
As mentioned, average square feet per person in an office has dropped from 225 feet to 150 feet. This shrinkage in space directly translates to real estate cost savings for companies. Conversely, it's difficult to pin down the $ impact open offices are having on a given company's output, despite the studies showing that it's making folks less productive. In other words, we can't say "because of open office productivity decline, Company ABC is generating X% less profits than it would under a closed office scenario", but we can say "if we adopt an open office, we'll save a ton on real estate costs".
How much money are companies saving by using open offices?
I get into detail down at the bottom for those curious, but we can estimate this by multipying 3 variables together:
(Square feet per person saved via open offices * # of employees * Cost/square foot of office space)
Using this formula, we are able to estimate savings tied to open offices for a few of the larger companies in the S&#x26;P 500:
Total amount of money saved/year by opting for open offices:
<a href="/images/blog/annual_real_estate_savings.png"><img src="/images/blog/annual_real_estate_savings.png" alt="annual real estate savings"></a>
Open office savings as a % of annual profit:
<a href="/images/blog/annual_real_estate_savings_profit.png"><img src="/images/blog/annual_real_estate_savings_profit.png" alt="annual real estate savings profit"></a>
As we can see, the largest companies are saving 100s of millions per year in office space expenses by offering less square footage per person, resulting in multiple % points of extra profit. To put this into a larger context, if we carry forward 1% open office profit savings across all companies in the S&#x26;P 500 (<a href="https://craft.co/reports/s-p-500-revenue-per-employee-perspective">w/ 25 million employees</a>), that translates to over $200B/year in extra profits! Not all companies in the S&#x26;P 500 necessarily have open offices, but this shows the magnitude of the potential cost savings.
Based on this, we can see see a clear, quantitative incentive for companies to 'open' their offices up. Whether or not these real estate savings are offset by reduced productivity or increased employee attrition is something that remains to be seen.
Footnotes/assumptions:
<ul>
<li>
Cost savings = (75 square feet per person saved by using open office$<a href="https://www.entrepreneur.com/article/246857">50/square foot</a> annual office cost*# employees)*
</li>
<li>
Assumes all employees (as defined in company financial data) occupy office space. Of course, some employees are in the field or working remotely and don't necessitate office space. However, I'm also only counting employees in this analysis (e.g., excluding temps/vendors who often occupy office space as well), so this should provide a good approximation.
</li>
<li>
Assuming annual office space cost of $50/square foot. Depending on the city, whether company owning or renting, the type of building, that # may skew higher or lower.
</li>
</ul>

Offices have gotten more 'open' in recent years, with the average square feet per person shrinking ~33% from 225 in 2010 to just ~150 in 2017. This...

Why offices are becoming more 'open'

In this tutorial I'll walk you through a simple methodology to correlate various stocks against each other. We'll grab the prices of the selected stocks using python, drop them into a clean dataframe, run a correlation, and visualize our results.
(1) Import libraries, select our list of stocks to correlate
<pre><code> import numpy as np
 import pandas as pd
 #used to grab the stock prices, with yahoo
 import pandas_datareader as web
 from datetime import datetime
 #to visualize the results
 import matplotlib.pyplot as plt
 import seaborn

 #select start date for correlation window as well as list of tickers
 start = datetime(2017, 1, 1)
 symbols_list = ['AAPL', 'F', 'TWTR', 'FB', 'AAL', 'AMZN', 'GOOGL', 'GE']</code></pre>
(2) Pull stock prices, push into clean dataframe
<pre><code>#array to store prices
symbols=[]

#pull price using iex for each symbol in list defined above
for ticker in symbols_list: 
 r = web.DataReader(ticker, 'yahoo', start)
 # add a symbol column
 r['Symbol'] = ticker 
 symbols.append(r)

# concatenate into df
df = pd.concat(symbols)
df = df.reset_index()
df = df[['Date', 'Close', 'Symbol']]
df.head()
</code></pre>
We're now left with a table in this format:
<table>
<thead>
<tr>
<th>date</th>
<th>close</th>
<th>symbol</th>
</tr>
</thead>
<tbody>
<tr>
<td>2017-01-03</td>
<td>113.4101</td>
<td>AAPL</td>
</tr>
<tr>
<td>2017-01-04</td>
<td>113.2832</td>
<td>AAPL</td>
</tr>
<tr>
<td>2017-01-05</td>
<td>113.8593</td>
<td>AAPL</td>
</tr>
</tbody>
</table>
3) However, we want our symbols represented as columns so we'll have to pivot the dataframe:
<pre><code> df_pivot = df.pivot('date','symbol','close').reset_index()
 df_pivot.head()</code></pre>
<table>
<thead>
<tr>
<th></th>
<th>date</th>
<th>AAL</th>
<th>AAPL</th>
<th>AMZN</th>
<th>F</th>
<th>FB</th>
<th>GE</th>
<th>GOOGL</th>
<th>TWTR</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>2017-01-03</td>
<td>45.7092</td>
<td>113.4101</td>
<td>753.67</td>
<td>11.4841</td>
<td>116.86</td>
<td>30.1370</td>
<td>808.01</td>
<td>16.44</td>
</tr>
<tr>
<td>1</td>
<td>2017-01-04</td>
<td>46.1041</td>
<td>113.2832</td>
<td>757.18</td>
<td>12.0132</td>
<td>118.69</td>
<td>30.1465</td>
<td>807.77</td>
<td>16.86</td>
</tr>
<tr>
<td>2</td>
<td>2017-01-05</td>
<td>45.3044</td>
<td>113.8593</td>
<td>780.45</td>
<td>11.6483</td>
<td>120.67</td>
<td>29.9754</td>
<td>813.02</td>
<td>17.09</td>
</tr>
<tr>
<td>3</td>
<td>2017-01-06</td>
<td>45.6203</td>
<td>115.1286</td>
<td>795.99</td>
<td>11.6392</td>
<td>123.41</td>
<td>30.0610</td>
<td>825.21</td>
<td>17.17</td>
</tr>
<tr>
<td>4</td>
<td>2017-01-09</td>
<td>46.4792</td>
<td>116.1832</td>
<td>796.92</td>
<td>11.5206</td>
<td>124.90</td>
<td>29.9183</td>
<td>827.18</td>
<td>17.50</td>
</tr>
</tbody>
</table>
(4) Next, we can run the correlation. Using the <a href="https://pandas.pydata.org/pandas-docs/version/0.21/generated/pandas.DataFrame.corr.html"> Pandas 'corr' function</a> to compute the <a href="https://www.investopedia.com/terms/p/pearsoncoefficient.asp"> Pearson correlation coeffecient</a> between each pair of equities
<pre><code> corr_df = df_pivot.corr(method='pearson')
 #reset symbol as index (rather than 0-X)
 corr_df.head().reset_index()
 del corr_df.index.name
 corr_df.head(10)</code></pre>
<table>
<thead>
<tr>
<th>symbol</th>
<th>AAL</th>
<th>AAPL</th>
<th>AMZN</th>
<th>F</th>
<th>FB</th>
<th>GE</th>
<th>GOOGL</th>
<th>TWTR</th>
</tr>
</thead>
<tbody>
<tr>
<td>AAL</td>
<td>1.000000</td>
<td>0.235239</td>
<td>0.226061</td>
<td>0.068024</td>
<td>0.356063</td>
<td>-0.332681</td>
<td>0.494075</td>
<td>0.169487</td>
</tr>
<tr>
<td>AAPL</td>
<td>0.235239</td>
<td>1.000000</td>
<td>0.868763</td>
<td>0.184501</td>
<td>0.911380</td>
<td>-0.895210</td>
<td>0.903191</td>
<td>0.781755</td>
</tr>
<tr>
<td>AMZN</td>
<td>0.226061</td>
<td>0.868763</td>
<td>1.000000</td>
<td>0.108351</td>
<td>0.744732</td>
<td>-0.937415</td>
<td>0.864455</td>
<td>0.955373</td>
</tr>
<tr>
<td>F</td>
<td>0.068024</td>
<td>0.184501</td>
<td>0.108351</td>
<td>1.000000</td>
<td>0.206055</td>
<td>-0.216064</td>
<td>0.189753</td>
<td>0.161078</td>
</tr>
<tr>
<td>FB</td>
<td>0.356063</td>
<td>0.911380</td>
<td>0.744732</td>
<td>0.206055</td>
<td>1.000000</td>
<td>-0.814703</td>
<td>0.900033</td>
<td>0.650404</td>
</tr>
<tr>
<td>GE</td>
<td>-0.332681</td>
<td>-0.895210</td>
<td>-0.937415</td>
<td>-0.216064</td>
<td>-0.814703</td>
<td>1.000000</td>
<td>-0.882526</td>
<td>-0.866871</td>
</tr>
<tr>
<td>GOOGL</td>
<td>0.494075</td>
<td>0.903191</td>
<td>0.864455</td>
<td>0.189753</td>
<td>0.900033</td>
<td>-0.882526</td>
<td>1.000000</td>
<td>0.789379</td>
</tr>
<tr>
<td>TWTR</td>
<td>0.169487</td>
<td>0.781755</td>
<td>0.955373</td>
<td>0.161078</td>
<td>0.650404</td>
<td>-0.866871</td>
<td>0.789379</td>
<td>1.000000</td>
</tr>
</tbody>
</table>
5) Finally, we can plot a heatmap of the correlations (with Seaborn and Matplotlib) to better visualize the results:
<pre><code> #take the bottom triangle since it repeats itself
 mask = np.zeros_like(corr_df)
 mask[np.triu_indices_from(mask)] = True
 #generate plot
 seaborn.heatmap(corr_df, cmap='RdYlGn', vmax=1.0, vmin=-1.0 , mask = mask, linewidths=2.5)
 plt.yticks(rotation=0) 
 plt.xticks(rotation=90) 
 plt.show()</code></pre>
<a href="/images/blog/py_corr_output.png"><img src="/images/blog/py_corr_output.png" alt="py corr output"></a>
Here we can see that, as expected, the tech companies are generally pretty highly correlated (as indicated by dark green -- AMZN/GOOGL, FB/AAPL, etc). Conversely, Ford (F) and General Electric (GE) are either not correlated or negatively correlated with the rest of the group.

Learn how to pull stock price data with python and analyze correlations between 2 different companys' stock returns data using a Seaborn heatmap in Python. Provided by InterviewQs, a mailing list for coding and data interview problems.

 Correlating stock returns using Python

Many executives in public companies sit on multiple boards of directors, meaning they have a vested interest in multiple companies. For example, if a board member sits on both Company A and Company B, then it's in this individual's best interest to ensure that Company A's decisions don't negatively impact Company B, and vice versa. This creates a 'link' or 'connection' between the two (or three, or four) companies. Within the largest 50 companies in the S&#x26;P 500 (the scope of this analysis), 39/50 (or 78%), are directly connected to one another through 1 or more board members.
It's worth noting that the # of connections get much larger when you expand the scope beyond 50 companies or expand to second and third degree connections, but it becomes pretty difficult to fit into any sort of visualization, so I've limited scope significantly for this analysis.
The below visualization shows the board member connections within the 50 largest public companies (if two companies share at least one board member, then they are flagged as connected). You can hover over a specific company to see its connection(s), and the # of connections for each company are shown in parenthesis next to the name.
The above visualization is best viewed on desktop, but mobile users should be able to click in and out of the companies (you may need to long pause or refresh page to click out on mobile). A static image of the chart can also be found <a href="/Board_members/static_board.png">here</a>.

Many executives in public companies sit on multiple boards of directors, meaning they have a vested interest in multiple companies. For example, if...

Interconnectedness of companies through shared board members

Load in packages
<pre><code>
#Packages
#--Web scraping packages
from bs4 import BeautifulSoup
import requests
#Pandas/numpy for data manipulation
import pandas as pd
import numpy as np</code></pre>
Load URLs we want to scrape into an array
<pre><code>#load URLs we want to scrape into an array
BASE_URL = [
'http://www.reuters.com/finance/stocks/company-officers/GOOG.O',
'http://www.reuters.com/finance/stocks/company-officers/AMZN',
'http://www.reuters.com/finance/stocks/company-officers/AAPL'
]</code></pre>
Loop through our URLs, scrape table, pass information to array
<pre><code>#loading empty array for board members
board_members = []
#Loop through our URLs we loaded above
for b in BASE_URL:
 html = requests.get(b).text
 soup = BeautifulSoup(html, &#x26;quot;html.parser&#x26;quot;)
 #identify table we want to scrape
 officer_table = soup.find('table', {&#x26;quot;class&#x26;quot; : &#x26;quot;dataTable&#x26;quot;})

 #try clause to skip any companies with missing/empty board member tables
 try:
 #loop through table, grab each of the 4 columns shown (try one of the links yourself to see the layout)
 for row in officer_table.find_all('tr'):
 cols = row.find_all('td')
 if len(cols) == 4:
 board_members.append((b, cols[0].text.strip(), cols[1].text.strip(), cols[2].text.strip(), cols[3].text.strip()))
 except: pass </code></pre>
Create new array, check length to ensure things pulled in correctly
<pre><code>#convert output to new array, check length
board_array = np.asarray(board_members)
len(board_array)</code></pre>
49
Convert new array to dataframe
<pre><code>#convert new array to dataframe
df = pd.DataFrame(board_array)</code></pre>
Rename columns, preview output
<pre><code>#rename columns, check output
df.columns = ['URL', 'Name', 'Age','Year_Joined', 'Title']
df.head(10)</code></pre>
<table>
<thead>
<tr>
<th></th>
<th>URL</th>
<th>Name</th>
<th>Age</th>
<th>Year_Joined</th>
<th>Title</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td><a href="http://www.reuters.com/finance/stocks/company-">http://www.reuters.com/finance/stocks/company-</a>...</td>
<td>Eric Schmidt</td>
<td>61</td>
<td>2015</td>
<td>Executive Chairman of the Board of Director</td>
</tr>
<tr>
<td>1</td>
<td><a href="http://www.reuters.com/finance/stocks/company-">http://www.reuters.com/finance/stocks/company-</a>...</td>
<td>Sergey Brin</td>
<td>43</td>
<td>2015</td>
<td>President, Director</td>
</tr>
<tr>
<td>2</td>
<td><a href="http://www.reuters.com/finance/stocks/company-">http://www.reuters.com/finance/stocks/company-</a>...</td>
<td>Lawrence Page</td>
<td>44</td>
<td>2015</td>
<td>Chief Executive Officer, Director</td>
</tr>
<tr>
<td>3</td>
<td><a href="http://www.reuters.com/finance/stocks/company-">http://www.reuters.com/finance/stocks/company-</a>...</td>
<td>Ruth Porat</td>
<td>59</td>
<td>2015</td>
<td>Chief Financial Officer, Senior Vice President</td>
</tr>
<tr>
<td>4</td>
<td><a href="http://www.reuters.com/finance/stocks/company-">http://www.reuters.com/finance/stocks/company-</a>...</td>
<td>Sundar Pichai</td>
<td>45</td>
<td>2017</td>
<td>Director, Chief Executive Officer, Google Inc.</td>
</tr>
<tr>
<td>5</td>
<td><a href="http://www.reuters.com/finance/stocks/company-">http://www.reuters.com/finance/stocks/company-</a>...</td>
<td>David Drummond</td>
<td>54</td>
<td>2015</td>
<td>Senior Vice President - Corporate Development,...</td>
</tr>
<tr>
<td>6</td>
<td><a href="http://www.reuters.com/finance/stocks/company-">http://www.reuters.com/finance/stocks/company-</a>...</td>
<td>John Hennessy</td>
<td>64</td>
<td>2007</td>
<td>Lead Independent Director</td>
</tr>
<tr>
<td>7</td>
<td><a href="http://www.reuters.com/finance/stocks/company-">http://www.reuters.com/finance/stocks/company-</a>...</td>
<td>Diane Greene</td>
<td>61</td>
<td>2015</td>
<td>Director</td>
</tr>
<tr>
<td>8</td>
<td><a href="http://www.reuters.com/finance/stocks/company-">http://www.reuters.com/finance/stocks/company-</a>...</td>
<td>L. John Doerr</td>
<td>65</td>
<td>2016</td>
<td>Independent Director</td>
</tr>
<tr>
<td>9</td>
<td><a href="http://www.reuters.com/finance/stocks/company-">http://www.reuters.com/finance/stocks/company-</a>...</td>
<td>Roger Ferguson</td>
<td>65</td>
<td>2016</td>
<td>Independent Director</td>
</tr>
</tbody>
</table>
Export data to CSV
<pre><code>#export data
df.to_csv('/Users/yourname/desktop/board_members.csv')</code></pre>
That's it! If you're interested in seeing how I used this data check out my visualization on the interconnectedness of companies through shared board members <a href="/blog/board_members">here</a>.

Learn to scrape multiple web pages with Python using BeautifulSoup and requests through this step-by-step tutorial. Provided by InterviewQs, a mailing list for coding and data interview problems.

Web scraping example using Python and Beautiful Soup

<h2>Overview</h2>
Below, we'll show you how to scrape Reddit using <a href="http://praw.readthedocs.io/en/latest/getting_started/quick_start.html">Praw</a> (Python Reddit API Wrapper). For this example, our goal will be to scrape the top submissions for the year across a few subreddits, storing the following: submission URL, domain (website URL), submission score. Ultimately, we want to be able to see which domains (urls) generate the highest scoring posts across a given subreddit.
<h2>1) Import packages, set up PRAW, select subreddits</h2>
Here we can set up our <a href="http://praw.readthedocs.io/en/latest/getting_started/quick_start.html">Praw credentials</a> and select the list of subreddits we want to analyze.
<pre><code>#packages
import pandas as pd
import praw
import operator
import pandas as pd

#set up praw - setup here: http://praw.readthedocs.io/en/latest/getting_started/quick_start.html
reddit = praw.Reddit(client_id='my client id',
client_secret='my client secret',
user_agent='my user agent')

#create list of subreddits to include
s_list = \
[
'enter subreddits you want to include here as comma separated strings - e.g. 'news', 'datascience', etc']</code></pre>
<h2>2) Grab the score, domain (url), and subreddit for each top yearly submission</h2>
In this section we're looping through our array of subreddits from above, and storing the score, domain, and subreddit; we'll store each of these attributes in 3 separate dataframes, and merge together using the submission ID.
<pre><code>#set up dictionaries to store submission information
domains_sub = {}
domains = {}
domains_score = {}
domains_url = {}

#Loop through our selected list of subreddits
for i in s_list:

#--Grab the score for a given submission--#
#pull in top submissions for the year for subreddit specified in list above
subreddit = reddit.subreddit(i) 
submissions = subreddit.top('year', limit=50)
#sum score across submissions
for s in submissions:
 if s.id in domains_score.keys():
 domains_score[s.id] += s.score
 else:
 domains_score[s.id] = s.score

df_score = pd.DataFrame.from_dict(domains_score, orient='index').reset_index()
df_score.columns = ['id','score']

#--Grab domain for given submission ID--#
subreddit = reddit.subreddit(i) #input('enter subreddit name: /r/'))
submissions = subreddit.top('year', limit=50)

for s in submissions:
 if s.id in domains.keys():

 domains[s.id] = s.domain
 else:

 domains[s.id] = s.domain

df_domain = pd.DataFrame.from_dict(domains, orient='index').reset_index()
df_domain.columns = ['id','domain']

#--Grab subreddit for given submission ID--#
subreddit = reddit.subreddit(i)
submissions = subreddit.top('year', limit=50)

for s in submissions:
 if s.id in domains_sub.keys():

 domains_sub[s.id] = s.subreddit.display_name
 else:

 domains_sub[s.id] = s.subreddit.display_name

df_subreddit = pd.DataFrame.from_dict(domains_sub, orient='index').reset_index()
df_subreddit.columns = ['id','subreddit']</code></pre>
<h2>Merge dataframes</h2>
Now that we have dataframes containing score, domain (url), and subreddit we can merge the three tables together, using submission ID as the primary key.
<pre><code>#merge the three tables together, using submission ID as primary key
df_sub_score = df_subreddit.merge(df_score, how='left', on=&#x26;quot;id&#x26;quot;)
df_final = df_sub_score.merge(df_domain, how='left', on='id')

# Add in submission URL using the 'id' 
df_final['url'] = ['www.reddit.com/']+df_final['id'].astype(str)

df_final.head()</code></pre>
<table>
<thead>
<tr>
<th></th>
<th>id</th>
<th>subreddit</th>
<th>score</th>
<th>domain</th>
<th>url</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>78tulq</td>
<td>todayilearned</td>
<td>42729</td>
<td>atlasobscura.com</td>
<td><a href="http://www.reddit.com/78tulq">www.reddit.com/78tulq</a></td>
</tr>
<tr>
<td>1</td>
<td>76bn5s</td>
<td>science</td>
<td>25024</td>
<td>ns.umich.edu</td>
<td><a href="http://www.reddit.com/76bn5s">www.reddit.com/76bn5s</a></td>
</tr>
<tr>
<td>2</td>
<td>7871xy</td>
<td>science</td>
<td>30642</td>
<td>acsh.org</td>
<td><a href="http://www.reddit.com/7871xy">www.reddit.com/7871xy</a></td>
</tr>
<tr>
<td>3</td>
<td>77pnk6</td>
<td>science</td>
<td>13176</td>
<td>jech.bmj.com</td>
<td><a href="http://www.reddit.com/77pnk6">www.reddit.com/77pnk6</a></td>
</tr>
<tr>
<td>4</td>
<td>75eydj</td>
<td>gaming</td>
<td>64510</td>
<td>i.redd.it</td>
<td><a href="http://www.reddit.com/75eydj">www.reddit.com/75eydj</a></td>
</tr>
</tbody>
</table>
Done! Explore the <a href="https://docs.google.com/spreadsheets/d/1B_JXeitjMH0IuvbgUUnk0k7OK1eryQfFRZXQt6StQ80/edit#gid=1539053097">output</a>
We now have a nice clean dataframe of the top yearly posts from each chosen subreddit, allowing us to see which domains racked up the highest total scores. I dumped the dataframe into a <a href="https://docs.google.com/spreadsheets/d/1B_JXeitjMH0IuvbgUUnk0k7OK1eryQfFRZXQt6StQ80/edit#gid=1539053097">Google Sheet</a> for you to explore.

Learn how use the Python Reddit API Wrapper (PRAW) to scrape key pieces of information from Reddit using Python with this step-by-step tutorial. Provided by InterviewQs, a mailing list for coding and data interview problems.

Scraping Reddit to find the most popular domains

(1) Google Drive API, credentials
In order to follow along, you're going to need a spreadsheet. Head over to <a href="https://docs.google.com/spreadsheets/u/0/">Google Sheets</a> create one. Put a few dummy columns in there so you can programatically access it later.
Next, you'll need to create a service account and OAuth2 credentials from the <a href="https://console.developers.google.com/">Google API Console</a>. Follow the steps below to enable the API and grab your credentials.
<ul>
<li>Head over to the <a href="https://console.developers.google.com/">Google API Console</a>.</li>
<li>Create a new project by selecting My Project -> + button</li>
<li>SearchInput for 'Google Drive API', enable it.</li>
<li>Head over to 'Credentials' (sidebar), click 'Create Credentials' -> 'Service Account Key'</li>
</ul>
<a href="/images/blog/create_credentials.png"><img src="/images/blog/create_credentials.png" alt="create credentials"></a>
<ul>
<li>Select Compute Engine service default, JSON, hit create.</li>
<li>Open up the JSON file, share your spreadsheet with the "<a href="mailto:XXX-compute@developer.gserviceaccount.com">XXX-compute@developer.gserviceaccount.com</a>" email listed.</li>
<li>Save the JSON file wherever you're hosting your project, you'll need to load it in through Python later.</li>
</ul>
(2) Connecting Python to Google Sheets, writing a dataframe
First, you'll need to install <a href="https://github.com/nithinmurali/pygsheets">pygsheets</a>, which allows us to actually read/write to the sheet through Python. Once that's installed, you're all set. Here's an example of importing the credentials and writing some dummy data to the sheet using a Pandas dataframe:
<pre><code>import pygsheets
import pandas as pd
#authorization
gc = pygsheets.authorize(service_file='/Users/erikrood/desktop/QS_Model/creds.json')

# Create empty dataframe
df = pd.DataFrame()

# Create a column
df['name'] = ['John', 'Steve', 'Sarah']

#open the google spreadsheet (where 'PY to Gsheet Test' is the name of my sheet)
sh = gc.open('PY to Gsheet Test')

#select the first sheet 
wks = sh[0]

#update the first sheet with df, starting at cell B2. 
wks.set_dataframe(df,(1,1))</code></pre>
(3) Done - Check your sheet for the data we pushed in the above step
Voila! The spreadsheet you created should now be populated with your dataframe:
<a href="/images/blog/py_gsheet_output.png"><img src="/images/blog/py_gsheet_output.png" alt="py gsheet output"></a>

Learn how to read from and write to a Google Spreadsheet using Python and the Google Spreadsheets API through this step-by-step tutorial. Provided by InterviewQs, a mailing list for coding and data interview problems.

Reading and writing to Google Spreadsheets using Python

(1) Set up an application password through Google
In order to send emails using Gmail, you'll need to set up an app password. Fortunately, this is pretty simple:
<ul>
<li>Head over to <a href="https://myaccount.google.com/security">myaccount.google.com/security</a>, and then click into the 'app passwords section (as shown in image below).</li>
</ul>
<a href="/images/blog/py_email_img_1.png"><img src="/images/blog/py_email_img_1.png" alt="py email img 1"></a>
<ul>
<li>
Select the app ('mail' in this case) and device (I used 'mac').
</li>
<li>
Click 'generate', save the password (we will feed it into the python script below).
</li>
</ul>
(2) Build a Python script to send email from our gmail account
<pre><code> import smtplib
 from email.mime.multipart import MIMEMultipart
 from email.mime.text import MIMEText

 from_address = "from_email@gmail.com"
 to_address = "to_email@gmail.com"

 # Create message container - the correct MIME type is multipart/alternative.
 msg = MIMEMultipart('alternative')
 msg['Subject'] = "Test email"
 msg['From'] = from_address
 msg['To'] = to_address

 # Create the message (HTML).
 html = """\
 We are sending an email using Python and Gmail, how fun! We can fill this with html, and gmail supports a decent range of css style attributes too - https://developers.google.com/gmail/design/css#example.
 """

 # Record the MIME type - text/html.
 part1 = MIMEText(html, 'html')

 # Attach parts into message container
 msg.attach(part1)

 # Credentials
 username = 'example_email@gmail.com' 
 password = 'your_password'

 # Sending the email
 ## note - this smtp config worked for me, I found it googling around, you may have to tweak the # (587) to get yours to work
 server = smtplib.SMTP('smtp.gmail.com', 587) 
 server.ehlo()
 server.starttls()
 server.login(username,password) 
 server.sendmail(from_address, to_address, msg.as_string()) 
 server.quit()
</code></pre>
(3) Voila! We can now use python to send emails through our gmail account
This is a pretty simple example, but hopefully gives you a sense of some of the cool things that can be done with this functionality. For example, you can build out complex html templates (using Python to build unique templates for each individual if desired). If you end up building something with this, <a href="">let me know!</a>

Learn how use send emails using Python through Gmail with this step-by-step tutorial. Provided by InterviewQs, a mailing list for coding and data interview problems.

 Sending email using Python and Gmail

Cereal tenure:
The below chart shows how long each cereal has lasted on the market. We can see the ~half of all cereals have been on the market for 5 years or less before being discontinued.
<a href="/images/blog/cereal_tenure.png"><img src="/images/blog/cereal_tenure.png" alt="cereal tenure"></a>
Note that 5 (3.4%) cereals in the 0-5 yr bin are still currently available, meaning they could last longer than 5 years before being discontinued. However, more new cereals will likely also be created by that time, keeping the % of cereals active 5 yrs or less ~the same.
New cereals launched by decade:
Here we can see the over half of all cereals have been launched after 1980. We can also see that cereal biz was booming in the 2000s, posting a record ~88 new cereals launched. Things seem to be cooling off 2010-2020, with 42 new cereals launched through 2018 (putting us on pace for ~53 total launches in decade).
<a href="/images/blog/cereal_launches.png"><img src="/images/blog/cereal_launches.png" alt="cereal launches"></a>
Why are manufacturers pushing out endless varieties of new cereals?
The next logical question to ask when coming across all if this information is ‘why are manufacturers producing a seemingly endless amount of cereal variety’?
Most speculate that this is due to manufacturers trying to combat the general stagnation in the breakfast cereal market. Below are a few strategies that seem to be in place to try and keep cereal sales high:
(1) Bring the millennials back by re-launching their old favorites:
<a href="https://www.nytimes.com/2016/02/24/dining/breakfast-cereal.html?_r=1">NY times</a> posted a popular article earlier this year discussing the declining interest, particularly amongst millennials, in breakfast cereal; the article explained that in recent surveys, over 40% of millennials opted out of breakfast cereal because of the hassle (time + mess to clean up after). Manufacturers, as a result, have been trying to lure millennials back in with nostalgic favorites, such as <a href="https://blog.generalmills.com/2014/12/weve-got-big-news-about-french-toast-crunch/">French Toast Crunch</a>.
(2) Join the health/psuedo-health wave by offering healthier varieties:
As <a href="https://qz.com/263968/as-protein-crazed-americans-shun-cold-cereal-for-other-breakfast-options-restaurants-are-pouncing/">qz.com</a> writes, yogurt sales have skyrocketed in the past ~10 years whereas breakfast cereal sales have remained flat. Many speculate this is due to yogurt being a natural convenient and healthy breakfast choice. Since cereal again cannot really change its convenience level (unless you’re on team dry cereal), it seems they may be taking the health angle, rebranding and renaming many existing cereals with buzzwords such as gluten-free, organic, all-natural, etc.
(3) Design short-tenured cereals intentionally for promotional reasons
If you check some of the more recent short-tenured cereals in the list below, you’ll see a lot of them are intentionally designed to be on the market for a limited time; they are either being promoted as ‘limited edition’ items, or are associated with some TV show/movie. I’d guess these types of cereals generate a nice jump in sales and lure consumers to buy outside of regular habits.

The below chart shows how long each cereal has lasted on the market. We can see the half of all cereals have been on the market for 5 years or less...

The never-ending variety of breakfast cereal

Lots of people opt for McDonald's as a cheap/convenient source of (questionably healthy) food. We thought it would be interesting to pair the nutritional data with prices to see which menu items actually get folks the most calories for their dollar.
Which menu items get you the most calories per dollar?
Assuming we're optimizing for just calories (energy), these items will provide you the most per dollar:
<a href="/images/blog/mcd_most_cal_dollar.png"><img src="/images/blog/mcd_most_cal_dollar.png" alt="mcd most cal dollar"></a>
Conversely, these foods offer you the least calories per dollar spent:
<a href="/images/blog/mcd_least_cal_dollar.png"><img src="/images/blog/mcd_least_cal_dollar.png" alt="mcd least cal dollar"></a>
Which menu items get you the most protein calories per dollar?
Of the 3 macronutrients, protein <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2954450/"> is the most expensive</a>. There are a ton of benefits to getting adequate protein in your diet, so instead of optimizing for just calories/dollar, it may be in your interest to look for foods with the most protein calories per dollar.
<a href="/images/blog/mcd_most_pro_cal_dollar.png"><img src="/images/blog/mcd_most_pro_cal_dollar.png" alt="mcd most pro cal dollar"></a>
Conversely, these foods offer you the least protein calories per dollar spent:
<a href="/images/blog/mcd_least_pro_cal_dollar.png"><img src="/images/blog/mcd_least_pro_cal_dollar.png" alt="mcd least pro cal dollar"></a>
Check out the complete menu here:
Here are the key stats for the items I was able to quickly gather information on. Grabbed the prices <a href="http://www.fastfoodmenuprices.com/mcdonalds-prices/"> here</a> (set for Michigan so could be higher/lower depending on your area, but will give you relative guidance), and the nutrition information <a href="http://www.nutritionsheet.com/facts/restaurants/fast-food/mcdonalds"> here</a>.
//csv to html dump here: <a href="http://www.convertcsv.com/csv-to-html.htm">http://www.convertcsv.com/csv-to-html.htm</a>
<table>
<thead>
<tr>
<th>Food</th>
<th>Price</th>
<th>Calories</th>
<th>Protein Calories</th>
<th>Calories/Dollar</th>
<th>Protein Calories/Dollar</th>
</tr>
</thead>
<tbody>
<tr>
<td>Sausage Biscuit (Regular Size Biscuit)</td>
<td>$1.27</td>
<td>430</td>
<td>44</td>
<td>339</td>
<td>35</td>
</tr>
<tr>
<td>McChicken</td>
<td>$1.07</td>
<td>360</td>
<td>56</td>
<td>336</td>
<td>52</td>
</tr>
<tr>
<td>Sausage McMuffin</td>
<td>$1.27</td>
<td>370</td>
<td>56</td>
<td>291</td>
<td>44</td>
</tr>
<tr>
<td>Cheeseburger</td>
<td>$1.07</td>
<td>300</td>
<td>60</td>
<td>280</td>
<td>56</td>
</tr>
<tr>
<td>Cinnamon Melts</td>
<td>$1.70</td>
<td>460</td>
<td>24</td>
<td>271</td>
<td>14</td>
</tr>
<tr>
<td>Baked Hot Apple Pie</td>
<td>$0.95</td>
<td>250</td>
<td>8</td>
<td>263</td>
<td>8</td>
</tr>
<tr>
<td>McDouble</td>
<td>$1.49</td>
<td>390</td>
<td>88</td>
<td>262</td>
<td>59</td>
</tr>
<tr>
<td>Double Cheeseburger</td>
<td>$1.70</td>
<td>440</td>
<td>100</td>
<td>259</td>
<td>59</td>
</tr>
<tr>
<td>Vanilla Triple Thick Shake (16 fl oz cup)</td>
<td>$2.77</td>
<td>693</td>
<td>52</td>
<td>250</td>
<td>19</td>
</tr>
<tr>
<td>Sausage Burrito</td>
<td>$1.27</td>
<td>300</td>
<td>48</td>
<td>236</td>
<td>38</td>
</tr>
<tr>
<td>Sausage Burrito</td>
<td>$1.27</td>
<td>300</td>
<td>48</td>
<td>236</td>
<td>38</td>
</tr>
<tr>
<td>French Fries (Medium)</td>
<td>$1.70</td>
<td>340</td>
<td>16</td>
<td>200</td>
<td>9</td>
</tr>
<tr>
<td>Ranch Snack Wrap (Crispy)</td>
<td>$1.81</td>
<td>360</td>
<td>56</td>
<td>199</td>
<td>31</td>
</tr>
<tr>
<td>Big Breakfast (Regular Size Biscuit)</td>
<td>$3.73</td>
<td>740</td>
<td>112</td>
<td>198</td>
<td>30</td>
</tr>
<tr>
<td>Chicken McNuggets (40 pc.)</td>
<td>$9.62</td>
<td>1880</td>
<td>384</td>
<td>195</td>
<td>40</td>
</tr>
<tr>
<td>Mac Snack Wrap</td>
<td>$1.81</td>
<td>330</td>
<td>60</td>
<td>182</td>
<td>33</td>
</tr>
<tr>
<td>Ranch Snack Wrap (Grilled)</td>
<td>$1.81</td>
<td>290</td>
<td>72</td>
<td>160</td>
<td>40</td>
</tr>
<tr>
<td>Double Cheeseburger</td>
<td>$2.76</td>
<td>440</td>
<td>100</td>
<td>159</td>
<td>36</td>
</tr>
<tr>
<td>Sausage, Egg &#x26; Cheese McGriddles</td>
<td>$3.52</td>
<td>560</td>
<td>80</td>
<td>159</td>
<td>23</td>
</tr>
<tr>
<td>Vanilla Reduced Fat Ice Cream Cone</td>
<td>$1.07</td>
<td>170</td>
<td>16</td>
<td>159</td>
<td>15</td>
</tr>
<tr>
<td>Double Quarter Pounder with Cheese</td>
<td>$5.13</td>
<td>740</td>
<td>192</td>
<td>144</td>
<td>37</td>
</tr>
<tr>
<td>Sausage McMuffin with Egg</td>
<td>$3.20</td>
<td>450</td>
<td>84</td>
<td>141</td>
<td>26</td>
</tr>
<tr>
<td>Sausage McGriddles</td>
<td>$2.99</td>
<td>420</td>
<td>44</td>
<td>140</td>
<td>15</td>
</tr>
<tr>
<td>Fruit 'n Yogurt Parfait (7 oz)</td>
<td>$1.07</td>
<td>150</td>
<td>16</td>
<td>140</td>
<td>15</td>
</tr>
<tr>
<td>Hash Brown</td>
<td>$1.07</td>
<td>150</td>
<td>4</td>
<td>140</td>
<td>4</td>
</tr>
<tr>
<td>Big Mac</td>
<td>$4.27</td>
<td>540</td>
<td>100</td>
<td>126</td>
<td>23</td>
</tr>
<tr>
<td>Quarter Pounder with Cheese</td>
<td>$4.06</td>
<td>510</td>
<td>116</td>
<td>126</td>
<td>29</td>
</tr>
<tr>
<td>Bacon, Egg &#x26; Cheese Biscuit</td>
<td>$3.41</td>
<td>420</td>
<td>60</td>
<td>123</td>
<td>18</td>
</tr>
<tr>
<td>McFlurry with Oreo Cookies (12 fl oz cup)</td>
<td>$2.56</td>
<td>315</td>
<td>52</td>
<td>123</td>
<td>20</td>
</tr>
<tr>
<td>Chicken Selects Premium Breast Strips (3 pc)</td>
<td>$3.20</td>
<td>384</td>
<td>92</td>
<td>120</td>
<td>29</td>
</tr>
<tr>
<td>Bacon, Egg &#x26; Cheese McGriddles</td>
<td>$3.52</td>
<td>420</td>
<td>60</td>
<td>119</td>
<td>17</td>
</tr>
<tr>
<td>Southern Style Crispy Chicken Sandwich</td>
<td>$4.16</td>
<td>430</td>
<td>96</td>
<td>103</td>
<td>23</td>
</tr>
<tr>
<td>Cheeseburger</td>
<td>$2.99</td>
<td>300</td>
<td>60</td>
<td>100</td>
<td>20</td>
</tr>
<tr>
<td>Egg McMuffin</td>
<td>$2.99</td>
<td>300</td>
<td>72</td>
<td>100</td>
<td>24</td>
</tr>
<tr>
<td>Orange Juice (Medium)</td>
<td>$2.02</td>
<td>200</td>
<td>12</td>
<td>99</td>
<td>6</td>
</tr>
<tr>
<td>Chicken McNuggets (10 piece)</td>
<td>$4.80</td>
<td>470</td>
<td>96</td>
<td>98</td>
<td>20</td>
</tr>
<tr>
<td>Hamburger</td>
<td>$2.66</td>
<td>250</td>
<td>48</td>
<td>94</td>
<td>18</td>
</tr>
<tr>
<td>Filet-O-Fish</td>
<td>$4.06</td>
<td>380</td>
<td>60</td>
<td>94</td>
<td>15</td>
</tr>
<tr>
<td>Premium Bacon Ranch Salad with Crispy Chicken</td>
<td>$4.91</td>
<td>450</td>
<td>116</td>
<td>92</td>
<td>24</td>
</tr>
<tr>
<td>McDouble</td>
<td>$4.27</td>
<td>390</td>
<td>88</td>
<td>91</td>
<td>21</td>
</tr>
<tr>
<td>Egg White Delight McMuffin</td>
<td>$2.99</td>
<td>250</td>
<td>72</td>
<td>84</td>
<td>24</td>
</tr>
<tr>
<td>Apple Slices</td>
<td>$1.07</td>
<td>70</td>
<td>0</td>
<td>65</td>
<td>0</td>
</tr>
<tr>
<td>Premium Southwest Salad with Grilled Chicken</td>
<td>$5.13</td>
<td>330</td>
<td>120</td>
<td>64</td>
<td>23</td>
</tr>
<tr>
<td>Side Salad</td>
<td>$1.7</td>
<td>20</td>
<td>4</td>
<td>12</td>
<td>2</td>
</tr>
</tbody>
</table>
Hope you enjoyed the post.

Learn which McDonald's menu items get you the most calories per dollar. Provided by InterviewQs, a mailing list for coding and data interview problems.

 Which McDonald's menu items get you the most calories/dollar?

I was curious which areas of the U.S. afforded folks the highest 'savings potential', defined for our purposes as [median income - 'minimum' cost of living], so I gathered the data and created the below map, detailing savings potential by U.S. county.
Want to see an interactive version? <a href="/blog/location-col-interactive">Click here</a>.
<a href="/images/blog/us_plot.png"><img src="/images/blog/us_plot.png" alt="us plot"></a>
<h3>Which counties offer the highest savings potential?</h3>
The chart below shows the top 20 counties in the U.S. with the highest savings potential.
<a href="/images/blog/highest_net_savings.png"><img src="/images/blog/highest_net_savings.png" alt="highest net savings"></a>
<h3>Which counties have the lowest savings potential?</h3>
The chart below shows the 20 counties in the US with lowest median annual savings. In other words, these counties have low median income relative to their high cost of living.
<a href="/images/blog/lowest_net_savings.png"><img src="/images/blog/lowest_net_savings.png" alt="lowest net savings"></a>
<h3>Where should I live to mimize costs? (regardless of income)</h3>
The chart below shows the 20 counties offering the lowest potential cost of living. Rent is the primary driver for these counties' low COL, as the other costs are ~similar across regions. For example, to rent a 1 bedroom in Jim Hogg County, TX it will run you an avg. of ~$188 per month. If Texas isn't your scene, don't worry -- there are ~200 counties in the U.S. that all offer 1 bedroom rent for less than $300/month. Take a dive into the raw data <a href="https://docs.google.com/spreadsheets/d/1EnX1o4m4jmU5D5zB5D5DAU9f8NQDyClW7u_wtLS-uPU/edit?usp=sharing">here</a>.
<a href="/images/blog/lowest_cost.png"><img src="/images/blog/lowest_cost.png" alt="lowest cost"></a>
As you can see from the charts above there are a large amount of places where it's difficult to save money, so it isn't particularly surprising that <a href="https://www.cnbc.com/2018/05/22/fed-survey-40-percent-of-adults-cant-cover-400-emergency-expense.html">40% of Americans can't cover a $400 emergency expense.</a>
h3(style="color:#4b2cde") Calculations/data:
<h3>Calculating 'savings potential'</h3>
I calculated median income post tax and subtracted by the total “minimum” costs of living. Below I outline how I calculated each input to create the chart above. Raw calculations and data are in this <a href="https://docs.google.com/spreadsheets/d/1EnX1o4m4jmU5D5zB5D5DAU9f8NQDyClW7u_wtLS-uPU/edit?usp=sharing"> Google Sheet</a>.
Income
I was able to find median income per household by US county (you can find the data in the "Sources and raw data" section below. I then reduce the median income per household by the average US tax rate of (~31.5%). This value provides me the median income post tax.
Rent
One of the largest costs that vary by region is rent, which I was lucky enough to have found the data of median rent per US county. For this exercise, we’re assuming median rent for a 1 bedroom apartment (in order to have consistency across all regions, (e.g. comparing a 1 bedroom to a 2 bedroom wouldn't be fair because a 2 bedroom costs more). Plus, if you're really trying to save, you'll minimize the amount of space you are paying for.
Costs associated with living
The next cost I factored in was the average cost of living per region across the US (e.g. West, Midwest, South, and Northeast). Unfortunately, I was not able to get cost of living at the county level. The costs of living value take into account average costs for food, alcohol, utilities, supplies, clothing, transportation, healthcare, and entertainment. However, for my calculation, I reduced cost of food, utilities + supplies + equipment, and transportation by 20%, and I completely removed entertainment, clothing, and alcohol. If you’re really trying to save, you would try to be frugal in these areas.
<h3>Sources and raw data</h3>
I used <a href="https://factfinder.census.gov/faces/nav/jsf/pages/index.xhtml">this website</a> and used data sets B25031 (rent) and S1901 (income). To get cost of living I used <a href="https://www.bls.gov/cex/csxregion.htm">this website</a>. You can see all the raw data <a href="https://docs.google.com/spreadsheets/d/1EnX1o4m4jmU5D5zB5D5DAU9f8NQDyClW7u_wtLS-uPU/edit?usp=sharing"> here</a>.

Learn which US Counties with the highest annual savings potential. Provided by InterviewQs, a mailing list for coding and data interview problems.

US Counties with the highest annual savings potential

<a href="/blog/location-col">View original post</a>

Blog

Top 20 Machine Learning Interview Questions You Should Know