Instacart Market Basket Analysis¶
Catie Fuentes
Introduction¶
The goal is to analyze the Instacart Online Grocery Shopping Dataset 2017. (https://www.instacart.com/datasets/grocery-shopping-2017) using the market basket analysis technique in order to find common patterns in the the customer's behaviors.¶
Market basket analysis consists of analyzing large datasets that include purchase history and products that are likely to be purchased together. Retailers use this to understand their customer's needs & behaviors in order to increase sales.¶
The key question in the market basket analysis is what products are most frequently purchased together.¶
https://runestone.academy/ns/books/published/DSA501_2022_HTTLADS/Instacart/market_basket.html¶
Objective¶
Find patterns based on the products ordered to understand the customer's needs and behaviors.¶
Determine if there are patterns in the kinds of products customers order and if they are telling of the lifestlye/diet-type (plant-based/vegetarian) of the customer- in order to market similar products to them.¶
Overview of the Datasets¶
Features of Aisles Dataframe:
aisle_id: the aisle identification number
aisle: the corresponding name of the aisle
Features of Departments Dataframe:
department_id: the department identification number
department: the corresponding name of the department
Features of Products Dataframe:
product_id: product identification number
product_name: the corresponding product name
aisle_id: aisle identification number where the product is located
department_id: department identification number that the product belongs to
Features of Orders Dataframe:
order_id: order identification number
user_id: user identification number
eval_set: whether the order was a prior order or a train order
order_number: order number
order_dow: day of the week the order was placed
order_hour_of_day: hour of day the order was placed
days_since_prior_order: days since the users last order
Features of Prior Order Dataframe:
order_id: order identification number
product_id: product identification number
add_to_cart_order: the order in which each product was added to the cart
reordered: whether or not the product was reordered
Features of Train Order Dataframe:
order_id: order identification number
product_id: product identification number
add_to_cart_order: the order in which each product was added to the cart
reordered: whether or not the product was reordered
Importing Libraries & Data¶
In [1]:
%matplotlib inline
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import altair as alt
import requests
matplotlib.style.use('ggplot')
sns.set_style("whitegrid")
import json
import pickle
import scipy
alt.data_transformers.enable('json')
Out[1]:
In [2]:
#!pip install kaggle # <- This is only needed the first time, I believe
# Install needed libraries
import json
# API credentials
token={"username":"catiefuentes","key":"4583b186134810a0769888ac7c442d00"}
# Create a folder in your Google account using the unix command "mkdir"
!mkdir ~/.kaggle
# Place the API credentials in the right place
with open('/root/.kaggle/kaggle.json','w') as file:
json.dump(token, file)
# Change the permissions of the file kaggle.json
!chmod 600 /root/.kaggle/kaggle.json
# Get a list of the datasets in kaggle with "instacart" in its name
!kaggle datasets list -s instacart
In [3]:
# Download the appropriate dataset and put it in the content folder
!kaggle datasets download -d psparks/instacart-market-basket-analysis -p /content
In [4]:
#Decript (unzip) all the encrypted files (with the ending ".zip")
!unzip \*.zip
In [23]:
#### COMMENTED OUT FOR USE ONLY IF KAGGLE API IS NOT WORKING ###
#url='http://136.183.139.117/dsa501data/instacart/'
#aisles=pd.read_csv(url+'aisles.csv')
#departments=pd.read_csv(url+'departments.csv')
#order_products_prior=pd.read_csv(url+'order_products__prior.csv')
#order_products_train=pd.read_csv(url+'order_products__train.csv')
#orders=pd.read_csv(url+'orders.csv')
#products=pd.read_csv(url+'products.csv')
#files=['aisles','departments','order_products__prior','order_products__train','orders','products']
#for file in files:
#globals()[file]=pd.read_csv(url+file+'.csv')
Instacart Datasets Displayed Side-by-side¶
In [5]:
# created dataframes for each dataset
orders = pd.read_csv('orders.csv')
products = pd.read_csv('products.csv')
order_products_prior = pd.read_csv('order_products__prior.csv')
order_products_train = pd.read_csv('order_products__train.csv')
aisles = pd.read_csv('aisles.csv')
departments = pd.read_csv('departments.csv')
files=['aisles','departments','order_products__prior','order_products__train','orders','products']
for file in files:
globals()[file]=pd.read_csv(file+'.csv')
In [6]:
# Got this code from JVP book. It displays frames side by side
class display(object):
"""Display HTML representation of multiple objects"""
template = """<div style="float: left; padding: 10px;">
<p style='font-family:"Courier New", Courier, monospace'>{0}</p>{1}
</div>"""
def __init__(self, *args):
self.args = args
def _repr_html_(self):
return '\n'.join(self.template.format(a, eval(a)._repr_html_())
for a in self.args)
def __repr__(self):
return '\n\n'.join(a + '\n' + repr(eval(a))
for a in self.args)
In [7]:
display(files[0],files[1],files[2])
Out[7]:
In [8]:
display(files[3],files[4],files[5])
Out[8]:
The Features of the Dataset¶
In [9]:
features_dict={}
for file in files:
features_dict[file]=[globals()[file].columns.values]
df=pd.DataFrame(features_dict).T
df.rename(columns = {0:'Features'}, inplace=True)
df.index.name = "File"
df = df.style.set_properties(**{'text-align': 'left'})
df
Out[9]:
Head() for each dataset¶
In [10]:
aisles.head()
Out[10]:
In [11]:
departments.head()
Out[11]:
In [12]:
orders.head()
Out[12]:
In [40]:
orders.describe()
Out[40]:
In [63]:
orders['user_id'].value_counts()
Out[63]:
In [13]:
products.set_index('product_id', inplace=True)
products.head()
Out[13]:
In [41]:
products.describe()
Out[41]:
In [15]:
order_products__prior.head()
Out[15]:
In [16]:
order_products__train.head()
Out[16]:
Data Description Visualizations¶
The plot below shows the frequency of orders placed throughout the week, with Saturday labeled as 0 and Friday labeled as 6. Most orders were placed during the weekend on Saturdays & Sundays and the least amount of orders were placed in the middle of the week on Tuesdays & Wednesdays. This is most likely due to customers wanting to prepare for the week ahead & avoid busy grocery stores on the weekend.¶
In [17]:
plt.figure(figsize=(12,8))
sns.countplot(x="order_dow", data=orders)
plt.ylabel('Count', fontsize=12)
plt.xlabel('Day of week', fontsize=12)
plt.xticks(rotation='vertical')
plt.title("Frequency of orders by day of week", fontsize=15)
plt.show()
The plot below shows the frequency of orders by hour of day. It is labeled based on 24 hours in a day, with 0 being 12am and 23, being 11pm. Most orders were placed from 10am to 4pm. This is likely because customers may be busier during this time, since most people work from 9am-5pm.¶
In [18]:
plt.figure(figsize=(12,8))
sns.countplot(x="order_hour_of_day", data=orders)
plt.ylabel('Count', fontsize=12)
plt.xlabel('Hour of day', fontsize=12)
plt.xticks(rotation='vertical')
plt.title("Frequency of orders by hour of day", fontsize=15)
plt.show()
Exploratory Data Analysis¶
In [19]:
order_products = order_products_prior.append(order_products_train)
order_products.shape
Out[19]:
In [20]:
order_products.head()
Out[20]:
In [21]:
order_products.product_id.nunique()
Out[21]:
In [22]:
product_counts = order_products.groupby('product_id')['order_id'].count().reset_index().rename(columns = {'order_id':'frequency'})
product_counts = product_counts.sort_values('frequency', ascending=False)[0:100].reset_index(drop = True)
product_counts = product_counts.merge(products, on = 'product_id', how = 'left')
product_counts.head(10)
Out[22]:
The top 10 products that were most frequently ordered were: Banana, Bag of Organic Bananas, Organic Strawberries, Organic Baby Spinach, Organic Hass Avocado, Organic Avocado, Large Lemon, Strawberries, Limes, and Organic Whole Milk. 9 out of the 10 products come from the produce section. The high frequency of bananas may be skewed due to customers ordering one individual banana at a time, rather than a bundle.¶
In [23]:
ax = sns.barplot(x="frequency", y="product_name", data=(product_counts).head(10))
plt.title("Frequency of top 10 ordered products", fontsize=15)
Out[23]:
In [24]:
product_departments = product_counts.merge(departments)
product_departments
Out[24]:
The highest frequency of products ordered came from the produce department and the lowest came from the canned goods department. This could be due to produce going bad quickly & needing to be replaced more frequently. And contrastly, canned goods may not be as in-demand because they have a longer shelf-life and do not need to be replaced often.¶
In [25]:
ax = sns.barplot(x="frequency", y="department", data=product_departments)
plt.title("Frequency of products ordered from each department", fontsize=15)
Out[25]:
Exploring Patterns in Plant-based/Vegetarian Options¶
I merged the datasets into one table.¶
To look for plant-based/vegetarian-friendly products:¶
I dropped any departments that contained: dairy eggs/meat seafood/deli & any aisles that contained: meat/dairy/poultry/chicken/beef/fish/jerky.¶
I filtered the dataframe to only show plant-based/vegetarian products that users added to cart 1st & reordered.¶
In [26]:
#merge tables: orders and order_products__prior
df=orders.merge(order_products__prior)
In [27]:
veg_df=df.merge(product_departments)
In [28]:
veg_df
Out[28]:
In [29]:
#dropping any departments that would not be vegetarian-friendly
vegetarian = veg_df[veg_df["department"].str.contains("dairy eggs|meat seafood|deli") == False]
In [30]:
vegetarian = vegetarian.merge(aisles)
In [31]:
plant_df = vegetarian[vegetarian["aisle"].str.contains("meat|dairy|poultry|chicken|beef|fish|jerky") == False]
In [32]:
#filter to see only add_to_cart_order 1 & reorders
plant_df[(veg_df.add_to_cart_order==1) & (veg_df.reordered==1)]
Out[32]:
In [33]:
plant_df.fillna(0, inplace = True)
plant_df.isnull().sum()
Out[33]:
In [34]:
len(plant_df['user_id'].unique())
Out[34]:
The plot below shows the frequency of the most popular plant-based products. Most products are from the Produce department, with the exception of: soda, coconut water, extra virgin olive oil, and wheat bread.¶
In [119]:
plt.rcParams['figure.figsize'] = (18, 7)
sns.countplot(plant_df['product_name'])
plt.title('Frequency of most popular products',fontsize = 30)
plt.xticks(rotation = 90 )
plt.show()
In [50]:
plant_df2 = plant_df[["order_hour_of_day","order_dow"]].mean()
plant_df2
Out[50]:
The plot below shows the frequency of plant-based orders by day of week. It is similar to the frequency of orders for the original dataset (above), with weekends (day 0 & 1) being the most popular days to order. However, there is a more significant decrease in frequency of orders from Monday-Thursday.¶
In [58]:
plt.figure(figsize=(12,8))
sns.countplot(x="order_dow", data=plant_df)
plt.ylabel('Count', fontsize=12)
plt.xlabel('Day of week', fontsize=12)
plt.xticks(rotation='vertical')
plt.title("Frequency of plant-based orders by day of week", fontsize=15)
plt.show()
The plot below shows the frequency of plant-based orders by hour of day. It looks almost identical to the original plot (above) containing all of the products.¶
In [59]:
plt.figure(figsize=(12,8))
sns.countplot(x="order_hour_of_day", data=plant_df)
plt.ylabel('Count', fontsize=12)
plt.xlabel('Hour of day', fontsize=12)
plt.xticks(rotation='vertical')
plt.title("Frequency of plant-based orders by hour of day", fontsize=15)
plt.show()
In [115]:
plant_products = plant_df['product_name'].value_counts()
plant_products
Out[115]:
This word cloud shows the most popular plant-based products:¶
Banana 472565
Bag of Organic Bananas 379450
Organic Strawberries 264683
Organic Baby Spinach 241921
Organic Hass Avocado 213584
In [117]:
from wordcloud import WordCloud, STOPWORDS
text = plant_products
wordcloud = WordCloud(
width = 3000,
height = 2000,
background_color = 'white',
stopwords = STOPWORDS).generate(str(text))
fig = plt.figure(
figsize = (20, 20),
facecolor = 'k',
edgecolor = 'k')
plt.imshow(wordcloud, interpolation = 'bilinear')
plt.axis('off')
plt.tight_layout(pad=0)
plt.show()
In [65]:
from mlxtend.frequent_patterns import apriori
from mlxtend.frequent_patterns import association_rules
In [66]:
pb_counts = plant_df.groupby('product_id')['order_id'].count().reset_index().rename(columns = {'order_id':'frequency'})
pb_counts = plant_df.sort_values('frequency', ascending=False)[0:100].reset_index(drop = True)
pb_counts = plant_df.merge(products, on = 'product_id', how = 'left')
pb_counts.head(10)
Out[66]:
In [67]:
freq_products = list(product_counts.product_id)
freq_products[1:10]
Out[67]:
In [68]:
pb_products = plant_df[plant_df.product_id.isin(freq_products)]
pb_products.shape
Out[68]:
In [69]:
pb_products = pb_products.merge(products, on = 'product_id', how='left')
pb_products.head()
Out[69]:
In [70]:
basket_analysis = plant_df.groupby(['order_id', 'product_name'])['reordered'].count().unstack().reset_index().fillna(0).set_index('order_id')
basket_analysis.head()
Out[70]:
In [71]:
def encode_units(x):
if x <= 0:
return 0
if x >= 1:
return 1
basket_analysis = basket_analysis.applymap(encode_units)
basket_analysis.head()
Out[71]:
In [72]:
basket_analysis.size
Out[72]:
In [73]:
basket_analysis.shape
Out[73]:
In [74]:
frequent_items = apriori(basket_analysis, min_support=0.01, use_colnames=True)
frequent_items.head()
Out[74]:
In [75]:
frequent_items.tail()
Out[75]:
In [76]:
frequent_items.shape
Out[76]:
I used the Mlxtend python library and found out associations between frequent plant-based product pairings:¶
Apriori Algorithm:
Apriori algorithm assumes that any subset of a frequent itemset must be frequent.
Association Rule Mining:
a technique to identify underlying relations between different items
used when we want to find an associations between different items or find frequent patterns in a dataset
Matrices
Support: the number of transactions containing product X divided by the total number of transactions
Confidence: Measures how often items in Y appear in transactions that contain X
Conviction: A high conviction value means that the consequent is highly depending on the antecedent
In [78]:
rules = association_rules(frequent_items, metric="lift", min_threshold=1)
rules.sort_values('lift', ascending=False)
Out[78]:
The plot below shows the support & confidence for the plant-based product associations.¶
In [118]:
import random
support = rules['support']
confidence = rules['confidence']
for i in range (len(support)):
support[i] = support[i] + 0.0025 * (random.randint(1,10) - 5)
confidence[i] = confidence[i] + 0.0025 * (random.randint(1,10) - 5)
plt.scatter(support, confidence, alpha=0.5, marker="*")
plt.title("Association of plant-based products")
plt.xlabel('support')
plt.ylabel('confidence')
plt.show()
Conclusion¶
Plant-based Diet Type Pattern¶
Most common products purchased:¶
Banana 472565
Bag of Organic Bananas 379450
Organic Strawberries 264683
Organic Baby Spinach 241921
Organic Hass Avocado 213584
Most common product pairings:¶
(Organic Strawberries, Organic Blueberries)
(Organic Garlic, Organic Yellow Onion)
(Organic Raspberries, Organic Hass Avocado)
(Organic Strawberries, Organic Hass Avocado)
(Organic Strawberries, Organic Raspberries)
The time of day and day of week that customers with a plant-based/vegetarian diet typically order is about the same as the general population: 10am-4pm/Saturdays & Sundays.¶
The most popular department ordered from was Produce.¶
In [ ]:
%%shell
jupyter nbconvert --to html /content/Fuentes_Final_Project.ipynb