"I don't know, what I don't know"
We want to first visually explore the data to see if we can confirm some of our initial hypotheses as well as make new hypothesis about the problem we are trying to solve.
For this we will start by loading the data and understanding the data structure of the dataframe we have.
In [1]:
# Import the library we need, which is Pandas
import pandas as pd
You will find the variable df used quite often to store a dataframe
In [2]:
# Read the csv file of Monthwise Quantity and Price csv file we have.
df = pd.read_csv('MonthWiseMarketArrivals_clean.csv')
In [3]:
df.shape
Out[3]:
In [4]:
df.head()
Out[4]:
So we have ten columns in our dataset. Let us understand what each one is.
Three are about the location of the Wholesale Market where Onion where sold.
Three are related to the
Four are about quantity and price in these wholesale market.
We would expect the following the columns to be of the following type
Let us see what pandas dataframe is reading these columns as.
In [5]:
# Get the typeof each column
df.dtypes
Out[5]:
So we are getting the quantitive columns are correctly being shown as integers and the categorical columns are showing as objects(strings) which is fine.
However, the date columns is being read as an object and not a Time-Interval. Let us at least fix the date column and make it into a datetime object
In [6]:
# Changing the date column to a Time Interval columnn
df.date = pd.DatetimeIndex(df.date)
In [13]:
df.shape
Out[13]:
In [14]:
# Now checking for type of each column
df.dtypes
Out[14]:
In [15]:
# Let us see the dataframe again now
df.head()
Out[15]:
In [21]:
# df.city.unique()
# Functional Approach
pd.unique(df.city)
Out[21]:
In [22]:
df2010 = df[df.year == 2010]
In [23]:
df2010.head()
Out[23]:
In [24]:
# We can also subset on multiple criterias
df2010Bang = df[(df.year == 2010) & (df.city == 'BANGALORE')]
In [25]:
df2010Bang.head()
Out[25]:
In [35]:
# Group by using city
df2010City = df2010.groupby(['city']).sum()
In [36]:
df2010City.head()
Out[36]:
In [37]:
type(df2010City)
Out[37]:
In [38]:
# If we only want to apply the sum function on quantity, then we specify the quantity column
df2010City = df2010.groupby(['city']).quantity.sum()
In [39]:
# Let us see this dataframe
df2010City.head()
Out[39]:
In [40]:
type(df2010City)
Out[40]:
In [41]:
# To create a dataframe again, it is best to specify index as false
df2010City = df2010.groupby(['city'], as_index=False).quantity.sum()
In [42]:
df2010City.head()
Out[42]:
In [45]:
sorted(df2010City.quantity)
Out[45]:
In [46]:
# Sort the Dataframe by Quantity to see which one is on top
df2010City = df2010City.sort_values(by = "quantity", ascending = False)
df2010City.head()
Out[46]:
In [47]:
%timeit sorted(df2010City.quantity)
In [49]:
%timeit df2010City.quantity.sort_values()
In [48]:
%timeit df2010City.sort_values(by = "quantity", ascending = False)
In [ ]:
In [50]:
# Load the visualisation libraries - Matplotlib
import matplotlib.pyplot as plt
In [51]:
# Let us see the output plots in the notebook itself
%matplotlib inline
In [52]:
# Set some parameters to get good visuals - style to ggplot and size to 15,10
plt.style.use('ggplot')
plt.rcParams['figure.figsize'] = (15, 10)
In [54]:
# Plot the Data
df2010City.plot(kind ="barh", x = 'city', y = 'quantity')
Out[54]:
In [ ]:
df2015 = df[]
In [ ]:
In [ ]:
In [ ]:
In [ ]:
In [55]:
df.head()
Out[55]:
In [56]:
dfBang = df[df.city == 'BANGALORE']
In [57]:
dfBang.head()
Out[57]:
In [58]:
dfBang.describe()
Out[58]:
In [59]:
# Reduce the precision of numbers - so that it is easy to read
pd.set_option('precision', 0)
In [60]:
dfBang.describe()
Out[60]:
In [61]:
dfBang.head()
Out[61]:
In [62]:
dfBang.index
Out[62]:
In [63]:
# Set the index as date
dfBang = dfBang.sort_values(by = "date")
dfBang.head()
Out[63]:
In [64]:
# Set the Index for the Dataframe
dfBang.index = pd.PeriodIndex(dfBang.date, freq='M')
In [65]:
dfBang.head()
Out[65]:
In [66]:
dfBang.priceMod.plot()
Out[66]:
In [67]:
dfBang.plot(kind = "line", y = ['priceMin', 'priceMod', 'priceMax'])
Out[67]:
To calculate the range of change, we will create a new price difference variable - which is the difference between the priceMin and priceMax
In [68]:
dfBang['priceDiff'] = dfBang['priceMax'] - dfBang['priceMin']
In [69]:
dfBang.head()
Out[69]:
In [70]:
dfBang.plot(kind = 'line', y = 'priceDiff')
Out[70]:
In [71]:
# Create new variable for Integer Month
dfBang['monthVal'] = pd.DatetimeIndex(dfBang['date']).month
In [72]:
dfBang.head()
Out[72]:
In [73]:
dfBangPivot = pd.pivot_table(dfBang, values = "priceDiff",
columns = "year", index = "monthVal")
In [75]:
dfBangPivot
Out[75]:
In [76]:
dfBangPivot.plot()
Out[76]:
In [77]:
dfBangPivot.plot(subplots = True, figsize=(15, 15), layout=(3, 5), sharey=True)
Out[77]:
In [78]:
dfBangPivot.plot(subplots = True, figsize=(15, 15), layout=(3, 5),
sharey=False)
Out[78]:
In [ ]: