In this notebook, you'll
List the team members contributing to this notebook, along with their responsabilities:
I advise you to work at least in pairs for each project notebook, as you did for the homework assignments. Of course, all team members may participate to each notebook.
In markdown cells, you'll
Remark: If your data is not directly available through the web (for instance, you requested it from a company, or got it from Inna or Henry), place you data into your personal bdrive, and provide a link to it, that you'll use in this notebook to download it into the data folder, as explained below).
In code cells, you'll
create the following directories:
./data
./data/raw # to store your raw data
./data/cleaned # to store the cleanned data
./data/simulated # to store simulated data
./visualizations # to store your plotshead method)Write scripts into the ./script directory using the he magic command
%%file ./script/file_name
Your scripts should contain functions and objects allowing you to
./data/rawThis way, the functions and objects contained in your scripts will be accessible in other notebooks through the import command in Python (see example below).
Remark 1: In the code cells, you may use Python, R, or Bash, as you find more convenient.
Remark 2: Try to make your notebook as readable and usable (by others) as possible.
Here is an example of how to package your code into scripts reusable in other notebook.
You'll need to do something similar to that. You'll also need to write explanations as outlined above in markdown cells.
In [1]:
%%bash
mkdir ./script ./data ./data/raw ./data/cleaned ./data/simulated ./visualizations
ls -r
In this example, I am using R. You want to use Python instead.
In [2]:
%%bash
curl "http://www.ope.ed.gov/security/dataFiles/Crime2013EXCEL.zip" > file.zip
In [3]:
%%bash
cp file.zip ./data/raw/data.zip
cd ./data/raw
unzip -o data.zip
In [4]:
%%bash
cd ./data/raw
#cp oncampuscrime101112.xls oncampuscrime101112.csv
In [4]:
TO DO: list sources, display data sample, explain data content, etc.
We now download an XML source containing plant data, and display an typical plant entry, represented as a XML node:
In [5]:
%load_ext rmagic
Our form of raw data is .xls, so we will use read.xls to read our raw data into R as a DataFrame.
In [6]:
%%R
install.packages("gdata")
library(gdata)
In [ ]:
#data <- read.csv("~/project2/stat133-project2/examples/data/raw/oncampuscrime101112.csv",header = TRUE)
Given a column name, we create a data frame column containing the column values:
In [17]:
%%R
print(dim(data))
print(names(data))
print(typeof(data))
print(head(data))
We package the code above into a function that creates the data frame column corresponding from a tag name and a list of plant nodes:
In [12]:
%%R
get_column = function(tag, plants) sapply(plants, get_value, tag)
cat((get_column('COMMON', plant_nodes)))
We are now ready to retrieve all the columns of our data frame into a list of vectors, which we will use to construct our data frame:
In [13]:
%%R
data = lapply(column_names, get_column, plant_nodes)
plant_df = data.frame(data, stringsAsFactors = FALSE)
print(head(plant_df))
Using the data frame we just created, we save the plant data into a csv file into the raw data directory:
In [14]:
%%R
write.csv(plant_df, './data/raw/plants_to_be_cleaned.csv')
In [ ]:
!head ./data/raw/plants_to_be_cleaned.csv
In [15]:
%%file ./script/plant_df-R
get_value = function(plant_node, tag) xmlValue(plant_node[[tag]])
get_column = function(tag, plants) sapply(plants, get_value, tag)
create_df_from_plant_xml = function(plant_file){
require(XML)
xml_doc = xmlParse(plant_file)
root_node = xmlRoot(xml_doc)
plant_nodes = xmlChildren(root_node)
column_names = names(plant_nodes[[1]])
data = lapply(column_names, get_column, plant_nodes)
return(data.frame(data, stringsAsFactors = FALSE))
}
Now, our function converting a raw plant XML file into a R data frame can be used in other notebooks, using the source command:
In [16]:
%%R
source('./script/plant_df-R')
print(head(create_df_from_plant_xml(plant_file)))
In [ ]: