The library is oriented towards table-like data structures that can be manipulated by a collection of methods:
In [1]:
# Import Pandas and Numpy
import pandas as pd
import numpy as np
In [2]:
# Make Series of count data and visaulize series
counts = pd.Series([223, 43, 53, 24, 43])
counts
Out[2]:
In [3]:
# What datatype is the counts object?
type(counts)
Out[3]:
In [4]:
# Make Series of count data with Gene Symbols
rna_counts = pd.Series([50, 10, 12, 29, 4], index=['BRCA2', 'GATA2', 'Myc', 'FOXA1', 'ERCC2'])
rna_counts
Out[4]:
In [5]:
# Construct second sample RNA-counts dict
cell2_counts = {'BRCA2':5, 'GATA2':20, 'Myc':45, 'FOXA1':10, 'ERCC2':0, 'BRCA1': 20}
cell2_counts
Out[5]:
In [6]:
# Make pandas Series from RNA-counts dict
rna_counts_cell2 = pd.Series(cell2_counts)
rna_counts_cell2
Out[6]:
In [7]:
# Access the 1st element of counts data
counts[0]
Out[7]:
In [8]:
# Get the 2nd through 4th elements
counts[1:4]
Out[8]:
In [9]:
# Get the counts for Myc Gene
rna_counts['Myc']
Out[9]:
In [10]:
# Get the Counts for FOXA1, GATA2 and BRCA2
rna_counts[['FOXA1', 'GATA2', 'BRCA2']]
Out[10]:
In [11]:
# Get the values in the counts matrix
counts.values
Out[11]:
In [12]:
# Get the index of the rna_counts matrix
rna_counts.index
Out[12]:
In [13]:
rna_counts.name = 'RNA Counts'
rna_counts.index.name = 'Symbol'
rna_counts
Out[13]:
In [14]:
# Select Genes that have greater than 20 counts
rna_counts > 20
Out[14]:
In [15]:
# Select genes that have greater than 20 counts
rna_counts[rna_counts > 20]
Out[15]:
In [16]:
# Make Movie Database with missing values
mcu_opening = {'Black Panther': 202003951, 'Thor: Ragnarok': 122744989, 'Spider-Man: Homecoming': 117027503,
'Guardians of the Galaxy Vol. 2': 146510104, 'Doctor Strange': 85058311,
'Captain America: Civil War': 179139142}
mcu_movies = ['Ant-Man and the Wasp', 'Avengers: Infinity War', 'Black Panther', 'Thor: Ragnarok',
'Spider-Man: Homecoming', 'Guardians of the Galaxy Vol. 2', 'Doctor Strange', 'Captain America: Civil War']
mcu_series = pd.Series(mcu_opening, index=mcu_movies)
mcu_series
Out[16]:
In [17]:
# Find movies with no opening revenue
pd.isnull(mcu_series)
Out[17]:
In [18]:
# Good opportunity to use Boolean filter get index and only movie names
mcu_series[pd.isnull(mcu_series)].index.values
Out[18]:
In [19]:
# Find movies with opening revenue
pd.notnull(mcu_series)
Out[19]:
In [20]:
# Display only movies with no opening revenue
mcu_series[pd.isnull(mcu_series)].index.values
Out[20]:
In [21]:
rna_counts
Out[21]:
In [22]:
rna_counts_cell2
Out[22]:
In [23]:
# Combine counts for 2 cells
rna_counts + rna_counts_cell2
Out[23]:
Create a Series from a specified list data which has the index labels as student names and test values.
Which Students have scores greater than 15?
Bonus: How would you use get the Students scores greater than 15 and less than 20?
| Names | Value |
|---|---|
| Anastasia | 12.5 |
| Dima | 9 |
| Katherine | 16.5 |
| James | NaN |
| Emily | 9 |
| Michael | 20 |
| Matthew | 14.5 |
| Laura | NaN |
| Kevin | 8 |
| Jonas | 19 |
In [24]:
# Sample Python data and labels:
students = ['Anastasia', 'Dima', 'Katherine', 'James', 'Emily', 'Michael', 'Matthew', 'Laura', 'Kevin', 'Jonas']
test_scores = [12.5, 9, 16.5, np.nan, 9, 20, 14.5, np.nan, 8, 19]
s_scores = pd.Series(test_scores, index=students)
# Which Students have scores greater than 15?
s_scores[s_scores > 15].index.values
Out[24]:
In [25]:
# Bonus: How would you use get the Students scores greater than 15 and less than 20?
s_scores[(s_scores > 15) & (s_scores < 20)]
Out[25]:
What is the mean, median and max test scores?
In [26]:
# What is the mean, median and max test scores?
s_scores.mean()
Out[26]:
In [27]:
s_scores.median()
Out[27]:
In [28]:
s_scores.max()
Out[28]:
| Title | Year | Studio | Rating |
|---|---|---|---|
| Ant-Man and the Wasp | 2018 | Beuna Vista | NaN |
| Avengers: Infinity War | 2018 | Beuna Vista | NaN |
| Black Panther | 2018 | Beuna Vista | 0.96 |
| Thor: Ragnarok | 2017 | Beuna Vista | 0.92 |
| Spider-Man: Homecoming | 2017 | Sony | 0.92 |
| Guardians of the Galaxy Vol. 2 | 2017 | Beuna Vista | 0.83 |
In [29]:
# Make Dataframe of Marvel data
mcu_data = {'Title': ['Ant-Man and the Wasp', 'Avengers: Infinity War', 'Black Panther', 'Thor: Ragnarok',
'Spider-Man: Homecoming', 'Guardians of the Galaxy Vol. 2'],
'Year':[2018, 2018, 2018, 2017, 2017, 2017],
'Studio':['Beuna Vista', 'Beuna Vista', 'Beuna Vista', 'Beuna Vista', 'Sony', 'Beuna Vista'],
'Rating': [np.nan, np.nan, 0.96, 0.92, 0.92, 0.83]}
df_mcu = pd.DataFrame(mcu_data)
df_mcu
Out[29]:
In [30]:
# What datatype is df_mcu?
type(df_mcu)
Out[30]:
In [31]:
# Assign column order and index based on Marvel Cinemetic Universe Movie Number
mcu_index = ['mcu_20','mcu_19', 'mcu_18', 'mcu_17', 'mcu_16', 'mcu_15']
mcu_columns = ['Title', 'Year', 'Studio', 'Rating']
df_mcu = pd.DataFrame(mcu_data, columns = mcu_columns, index = mcu_index)
df_mcu
Out[31]:
In [32]:
# Make Dataframe of population
pop = {'Nevada': {2001: 2.9, 2002: 2.9}, 'Ohio': {2002: 3.6, 2001: 1.7, 2000: 1.5}}
df_pop = pd.DataFrame(pop)
df_pop
Out[32]:
In [33]:
# Get the number of rows in a Dataframe
len(df_mcu)
Out[33]:
In [34]:
# Get the (rows, cols) of the Dataframe
df_mcu.shape
Out[34]:
In [35]:
# Get the column headers
df_mcu.columns
Out[35]:
In [36]:
# Get the row index values
df_mcu.index
Out[36]:
In [37]:
# Get values of the Dataframe only
df_mcu.values
Out[37]:
In [38]:
# Select values in a single column
df_mcu['Title']
Out[38]:
In [39]:
# Select values in a list of columns
df_mcu[['Title', 'Rating']]
Out[39]:
In [40]:
# Use slice to get the first n rows (NumPy style indexing)
df_mcu[:2]
Out[40]:
In [41]:
# Can combine slice and column selection to select the first n rows
df_mcu['Title'][:2]
Out[41]:
In [42]:
df_mcu[:4]['Year']
Out[42]:
In [43]:
# Return values in the first row
df_mcu.iloc[0]
Out[43]:
In [44]:
# Return values in the first row and second column
df_mcu.iloc[0,1]
Out[44]:
In [45]:
# Return values in the 3,5 and 6th rows
df_mcu.iloc[[2,4,5]]
Out[45]:
In [46]:
# Return values in the first row and columns 2 and 3
df_mcu.iloc[:2, 1:3]
Out[46]:
In [47]:
# Select all values of the 20th Movie
df_mcu.loc['mcu_20']
Out[47]:
In [48]:
# Select all values of the 20th, 17th and 15th movie, which uses row index values,
# Not to be confused with df_mcu[['Title', 'Rating']] which uses column headers
df_mcu.loc[['mcu_20', 'mcu_17', 'mcu_15']]
Out[48]:
In [49]:
# Select the Year and Rating
df_mcu.loc[:, ['Year', 'Rating']]
Out[49]:
In [50]:
# Filter for Rating < .95
df_mcu.loc[df_mcu['Rating'] < .95, :]
Out[50]:
In [51]:
# Filter for Rating < .95 and Sudio is Sony
# Reuse the bitwise comparator seen earlier but with OR instead of AND.
df_mcu.loc[(df_mcu['Rating'] < .95) | (df_mcu['Studio'] == 'Sony'), :]
Out[51]:
In [52]:
# Add new predicted rating to Dataframe
df_mcu['Predicted Rating'] = np.random.random(len(df_mcu))
df_mcu
Out[52]:
In [53]:
# Add a new row for a new movie
new_row = pd.Series(['Captain Marvel', 2019, 'BeunaVista', np.nan, np.random.random(1)[0]], index=df_mcu.columns, name= 'mcu_21' )
df_mcu.append(new_row)
Out[53]:
In [54]:
# Drop the Rating Column
df_mcu.drop('Rating', axis=1)
Out[54]:
In [55]:
# Drop the 17 and 19th movies
df_mcu.drop(['mcu_15', 'mcu_17'])
Out[55]:
Create a Dataframe from a specified dictionary data of
Which Students have scorescore is between 15 and 20?
| Names | Scores | Attempts | Qualify |
|---|---|---|---|
| Anastasia | 12.5 | 1 | 'yes' |
| Dima | 9 | 3 | 'no' |
| Katherine | 16.5 | 2 | 'yes' |
| James | NaN | 3 | 'no' |
| Emily | 9 | 2 | 'no' |
| Michael | 20 | 3 | 'yes' |
| Matthew | 14.5 | 1 | 'yes' |
| Laura | NaN | 1 | 'no' |
| Kevin | 8 | 2 | 'no' |
| Jonas | 19 | 1 | 'yes' |
In [56]:
# Sample Python data:
exam_data = {'Names': ['Anastasia', 'Dima', 'Katherine', 'James', 'Emily', 'Michael', 'Matthew', 'Laura', 'Kevin', 'Jonas'],
'Scores': [12.5, 9, 16.5, np.nan, 9, 20, 14.5, np.nan, 8, 19],
'Attempts': [1, 3, 2, 3, 2, 3, 1, 1, 2, 1],
'Qualify': ['yes', 'no', 'yes', 'no', 'no', 'yes', 'yes', 'no', 'no', 'yes']}
student_data = pd.DataFrame(exam_data)
student_data
Out[56]:
In [57]:
# Select the students the qualify based on scores using the Qualify column 'yes'
student_data[student_data['Qualify'] == 'yes']
Out[57]:
Add a new column of, Grade Level, to indicate which Grade in High school the students are in?
Add a new student named, Jack? (Hint: Need to use ignore_index=True)
In [58]:
# Add a new column of, Grade Level, to indicate which Grade in High school the students are in?
grade_level = ['9th', '9th', '10th', '11th', '12th', '9th', '10th', '11th', '12th', '11th']
student_data['Grade'] = grade_level
student_data
Out[58]:
In [59]:
student_data.columns
Out[59]:
In [60]:
# Add a new student named, Jack? (Hint: Need to use ignore_index=True)
jack = pd.Series([2, 'Jack', 'yes' , 17, '9th'],index=student_data.columns )
student_data.append(jack, ignore_index=True)
Out[60]:
Add a new column of Pass that is either 0 or 1 based on the column Qualify. (Hint: use numpy.where)
In [61]:
# Add a new column of Pass that is either 0 or 1 based on the column Qualify. (Hint: use numpy.where)
student_data['Pass'] = np.where(student_data['Qualify'] == 'yes', 1, 0)
student_data
Out[61]:
The first step in any problem is identifying what format your data is in, and then loading it into whatever framework you're using. Common formats are:
Read in Winter Olympic Medal Winners form Kaggle
In [62]:
# Read in Winter Olympic Medal Winners
winter_olympics = pd.read_csv('data/winter_olympics.csv')
winter_olympics.head()
Out[62]:
In [63]:
# Get the First 3 lines of a Dataframe
winter_olympics.head(3)
Out[63]:
In [64]:
# Sort Dataframe by rows in ascending order
df_mcu.sort_index(axis=0, ascending=True)
Out[64]:
In [65]:
# Sort Dataframe by column in descending order
df_mcu.sort_values(by=['Rating', 'Predicted Rating'], ascending=False)
Out[65]:
- for numeric data: mean, std, max, min, 25%, 50%, 75%, etc.
- For non-numeric data: count, uniq, most-frequent item, etc
In [66]:
# Summary Statistics for the Dataframe
df_mcu.describe()
Out[66]:
In [67]:
# Mean of the Rating and Predicted Rating Columns
df_mcu.loc[:,['Rating', 'Predicted Rating']].mean()
Out[67]:
In [68]:
# Get the variance of the Rating column
df_mcu.loc[:,['Rating']].var()
Out[68]:
- Drop rows with Nan values
In [69]:
# Drop rows with NaN values
df_mcu.dropna()
Out[69]:
- Fill Nan with default value
In [70]:
# File Nan in Dataframe with default value
df_mcu.fillna(0)
Out[70]:
- Use inplace to modify the dataframe instead of retunring a new object
In [71]:
# File Nan in Dataframe with default value in place
df_mcu.fillna(0, inplace=True)
df_mcu
Out[71]:
What is the median score of the students on the exam?
In [72]:
# What is the median score of the students on the exam?
student_data['Scores'].median()
Out[72]:
Deduct 4 points from everyone that attempted the exam 2 or more times. Replace all Nan scores with 0. (Passing is 12 points)
Compute the mean. Would the class as a whole pass the test?
Are there any students that will fail now?
In [73]:
# Deduct 4 points from everyone that attempted the exam 2 or more times. Replace all Nan scores with 0. (Passing is 12 points)
student_data.loc[student_data['Attempts'] >= 2, 'Scores'] -= 4
student_data
Out[73]:
In [74]:
# Compute the mean. Would the class as a whole pass the test?
student_data['Scores'].mean()
Out[74]:
In [75]:
# Are there any students that will fail now?
student_data[(student_data['Qualify'] == 'yes')]
Out[75]:
In [76]:
# Groupby year of release and get mean Rating
df_mcu.groupby('Year').mean()
Out[76]:
- Apply square to every value in a dataframe
In [77]:
# Apply square to every value in a dataframe
test_data = np.arange(9).reshape(3,-1)
df_test = pd.DataFrame(test_data, index=['r1', 'r2', 'r3'], columns=['c1', 'c2', 'c3'])
df_test
Out[77]:
In [78]:
df_test.applymap(np.square)
Out[78]:
- Apply a new function that subtract max from 2 times min in every column
In [79]:
# Define max minus min function
def max_minus_min(x):
return max(x)-(2*min(x))
# Apply a new function that subtract max from min in every column
df_test.apply(max_minus_min)
Out[79]:
Group students by attempts and find the average score?
In [80]:
# Group students by attempts and find the average score?
student_data.groupby('Attempts')['Scores'].mean()
Out[80]:
Group students by their pass result and report the variance in scores?
In [81]:
# Group students by their pass result and report the variance in scores?
student_data.groupby('Pass')['Scores'].var()
Out[81]:
In [82]:
# Import maplotlib and setup to display plots notebook
import matplotlib.pyplot as plt
%matplotlib inline
In [83]:
# Plot the Number of Medals in barchart by country using: plot.bar()
winter_olympics.groupby(['Country'])['Medal'].count().plot.bar()
Out[83]:
Typical normalization approach:
Further Resources:
In the Winter olympics which country has the most Biathlon medals?
In the Winter olympics which country has the most Skiing medals? And in which event do they have the most Gold medals?
In [84]:
# In the Winter olympics which country has the most Biathlon medals?
winter_sport_medal = winter_olympics.groupby(['Country','Sport'])['Medal'].count()
winter_sport_medal.head(10)
Out[84]:
In [85]:
winter_sport_medal.loc[:,'Biathlon',:].sort_values(ascending=False).head()
Out[85]:
In [86]:
# In the Winter olympics which country has the most Skiing medals?
winter_sport_medal.loc[:,'Skiing',:].sort_values(ascending=False).head()
Out[86]:
In [87]:
# And in which event do they have the most Gold medals?
winter_sport_disc_medal = winter_olympics.groupby(['Country','Sport', 'Discipline', 'Medal'])['Medal'].count()
winter_sport_disc_medal.head()
Out[87]:
In [88]:
winter_sport_disc_medal['NOR','Skiing'].sort_values(ascending=False)
Out[88]:
Import the Summer Olympic dataset located in ('data/summer_olypmics.csv')
Which Olympian has the most medals?
Which Olympian has the most Gold medals and for which Country and Sport?
Which rows have no values and why?
In [89]:
# Import the Summer Olympic dataset located in ('data/summer_olypmics.csv')
summer_olympics = pd.read_csv('data/summer_olympics.csv')
#Which Olympian has the most medals?
summer_olympics.groupby(['Athlete'])['Medal'].count().sort_values(ascending=False).head()
Out[89]:
In [90]:
# Which Olympian has the most Gold medals and for which Country?
summer_olympics.groupby(['Athlete','Medal','Country'])['Medal'].count().sort_values(ascending=False).head()
Out[90]:
In [91]:
# Which Olympian has the most Gold medals and for which Sport?
summer_olympics.groupby(['Athlete','Medal','Discipline'])['Medal'].count().sort_values(ascending=False).head()
Out[91]:
In [92]:
# Which rows have no values and why?
summer_olympics[pd.isnull(summer_olympics).any(axis=1)]
Out[92]:
Import the example RNA-seq Count Data in ('../data/RNAseq_count_table.txt')
Calculate CPM for each Sample. (CPM, Counts Per Million)
Formula for CPM = readsMappedToGene x 1/totalNumReads x 10^6
totalNumReads - total number of mapped reads of a sample readsMappedToGene - number of reads mapped to a selected gene
Which Gene has the highest average CPM?
What is the Correlation between SRR1550986 SRR1550987?
In [93]:
# Import the example RNA-seq Count Data in ('data/RNAseq_count_table.txt')
rna_counts = pd.read_csv('data/RNAseq_count_table.txt', sep = '\t', index_col = 0)
rna_counts.head()
Out[93]:
In [94]:
# Calculate CPM for each Sample.
rna_cpm = rna_counts.divide(rna_counts.sum(axis=0)).multiply(1000000)
rna_cpm.head()
Out[94]:
In [95]:
# Which Gene has the highest average CPM?
rna_cpm.mean(axis=1).sort_values(ascending=False).head()
Out[95]:
In [96]:
# What is the Correlation between SRR1550986 SRR1550987?
rna_cpm.corr()
Out[96]: