The following code creates two csv files using data from the Boston Housing dataset which is built into scikit-learn and adds them to the local directory
In [1]:
import pandas as pd
import numpy
from numpy.random import choice
from sklearn.datasets import load_boston
import h2o
h2o.init()
In [2]:
# transfer the boston data from pandas to H2O
boston_data = load_boston()
X = pd.DataFrame(data=boston_data.data, columns=boston_data.feature_names)
X["Median_value"] = boston_data.target
X = h2o.H2OFrame.from_python(X.to_dict("list"))
In [3]:
# select 10% for valdation
r = X.runif(seed=123456789)
train = X[r < 0.9,:]
valid = X[r >= 0.9,:]
h2o.export_file(train, "Boston_housing_train.csv", force=True)
h2o.export_file(valid, "Boston_housing_test.csv", force=True)
Enable inline plotting in the Jupyter Notebook
In [4]:
%matplotlib inline
import matplotlib.pyplot as plt
Read csv data into H2O. This loads the data into the H2O column compressed, in-memory, key-value store.
In [5]:
fr = h2o.import_file("Boston_housing_train.csv")
View the top of the H2O frame.
In [6]:
fr.head()
Out[6]:
View the bottom of the H2O Frame
In [7]:
fr.tail()
Out[7]:
Select a column
fr["VAR_NAME"]
In [8]:
fr["CRIM"].head() # Tab completes
Out[8]:
Select a few columns
In [9]:
columns = ["CRIM", "RM", "RAD"]
fr[columns].head()
Out[9]:
Select a subset of rows
Unlike in Pandas, columns may be identified by index or column name. Therefore, when subsetting by rows, you must also pass the column selection.
In [10]:
fr[2:7,:] # explicitly select all columns with :
Out[10]:
Key attributes:
* columns, names, col_names
* len, shape, dim, nrow, ncol
* types
Note:
Since the data is not in local python memory there is no "values" attribute. If you want to pull all of the data into the local python memory then do so explicitly with h2o.export_file and reading the data into python memory from disk.
In [11]:
# The columns attribute is exactly like Pandas
print("Columns:" + str(fr.columns) + "\n")
print("Columns:" + str(fr.names) + "\n")
print("Columns:" + str(fr.col_names) + "\n")
# There are a number of attributes to get at the shape
print("length:" + str( len(fr) ) + "\n")
print("shape:" + str(fr.shape) + "\n")
print("dim:" + str(fr.dim) + "\n")
print("nrow:" + str(fr.nrow) + "\n")
print("ncol:" + str(fr.ncol) + "\n")
# Use the "types" attribute to list the column types
print("types:" + str(fr.types) + "\n")
Select rows based on value
In [12]:
fr.shape
Out[12]:
Boolean masks can be used to subselect rows based on a criteria.
In [13]:
mask = fr["CRIM"]>1
fr[mask,:].shape
Out[13]:
Get summary statistics of the data and additional data distribution information.
In [14]:
fr.describe()
Set up the predictor and response column names
Using H2O algorithms, it's easier to reference predictor and response columns by name in a single frame (i.e., don't split up X and y)
In [15]:
x = fr.names[:]
y="Median_value"
x.remove(y)
H2O is a machine learning library built in Java with interfaces in Python, R, Scala, and Javascript. It is open source and well-documented.
Unlike Scikit-learn, H2O allows for categorical and missing data.
The basic work flow is as follows:
In [16]:
model = h2o.random_forest(x=fr[:400,x],y=fr[:400,y],seed=42) # Define and fit first 400 points
In [17]:
model.predict(fr[400:fr.nrow,:]) # Predict the rest
Out[17]:
The performance of the model can be checked using the holdout dataset
In [18]:
perf = model.model_performance(fr[400:fr.nrow,:])
perf.r2() # get the r2 on the holdout data
perf.mse() # get the mse on the holdout data
perf # display the performance object
Out[18]:
Instead of taking the first 400 observations for training, we can use H2O to create a random test train split of the data.
In [19]:
r = fr.runif(seed=12345) # build random uniform column over [0,1]
train= fr[r<0.75,:] # perform a 75-25 split
test = fr[r>=0.75,:]
model = h2o.random_forest(x=train[x],y=train[y],seed=42)
perf = model.model_performance(test)
perf.r2()
Out[19]:
There was a massive jump in the R^2 value. This is because the original data is not shuffled.
H2O's machine learning algorithms take an optional parameter nfolds to specify the number of cross-validation folds to build. H2O's cross-validation uses an internal weight vector to build the folds in an efficient manner (instead of physically building the splits).
In conjunction with the nfolds parameter, a user may specify the way in which observations are assigned to each fold with the fold_assignment parameter, which can be set to either:
* AUTO: Perform random assignment
* Random: Each row has a equal (1/nfolds) chance of being in any fold.
* Modulo: Observations are in/out of the fold based by modding on nfolds
In [20]:
model = h2o.random_forest(x=fr[x],y=fr[y], nfolds=10) # build a 10-fold cross-validated model
In [22]:
scores = numpy.array([m.r2() for m in model.xvals]) # iterate over the xval models using the xvals attribute
print("Expected R^2: %.2f +/- %.2f \n" % (scores.mean(), scores.std()*1.96))
print("Scores:" + str(scores.round(2)))
However, you can still make use of the cross_val_score from Scikit-Learn
In [23]:
from sklearn.cross_validation import cross_val_score
from h2o.cross_validation import H2OKFold
from h2o.estimators.random_forest import H2ORandomForestEstimator
from h2o.model.regression import h2o_r2_score
from sklearn.metrics.scorer import make_scorer
You still must use H2O to make the folds. Currently, there is no H2OStratifiedKFold. Additionally, the H2ORandomForestEstimator is analgous to the scikit-learn RandomForestRegressor object with its own fit method
In [24]:
model = H2ORandomForestEstimator(seed=42)
In [26]:
scorer = make_scorer(h2o_r2_score) # make h2o_r2_score into a scikit_learn scorer
custom_cv = H2OKFold(fr, n_folds=10, seed=42) # make a cv
scores = cross_val_score(model, fr[x], fr[y], scoring=scorer, cv=custom_cv)
print("Expected R^2: %.2f +/- %.2f \n" % (scores.mean(), scores.std()*1.96))
print("Scores:" + str(scores.round(2)))
There isn't much difference in the R^2 value since the fold strategy is exactly the same. However, there was a major difference in terms of computation time and memory usage.
Since the progress bar print out gets annoying let's disable that
In [27]:
h2o.__PROGRESS_BAR__=False
h2o.no_progress()
Grid search in H2O is still under active development and it will be available very soon. However, it is possible to make use of Scikit's grid search infrastructure (with some performance penalties)
In [28]:
from sklearn import __version__
sklearn_version = __version__
print sklearn_version
If you have 0.16.1, then your system can't handle complex randomized grid searches (it works in every other version of sklearn, including the soon to be released 0.16.2 and the older versions).
The steps to perform a randomized grid search:
All the steps will be repeated from above.
Because 0.16.1 is installed, we use scipy to define specific distributions
ADVANCED TIP:
Turn off reference counting for spawning jobs in parallel (n_jobs=-1, or n_jobs > 1). We'll turn it back on again in the aftermath of a Parallel job.
If you don't want to run jobs in parallel, don't turn off the reference counting.
Pattern is:
>>> h2o.turn_off_ref_cnts()
>>> .... parallel job ....
>>> h2o.turn_on_ref_cnts()
In [30]:
%%time
from h2o.estimators.random_forest import H2ORandomForestEstimator # Import model
from sklearn.grid_search import RandomizedSearchCV # Import grid search
from scipy.stats import randint, uniform
model = H2ORandomForestEstimator(seed=42) # Define model
params = {"ntrees": randint(20,50),
"max_depth": randint(1,10),
"min_rows": randint(1,10), # scikit's min_samples_leaf
"mtries": randint(2,fr[x].shape[1]),} # Specify parameters to test
scorer = make_scorer(h2o_r2_score) # make h2o_r2_score into a scikit_learn scorer
custom_cv = H2OKFold(fr, n_folds=10, seed=42) # make a cv
random_search = RandomizedSearchCV(model, params,
n_iter=30,
scoring=scorer,
cv=custom_cv,
random_state=42,
n_jobs=1) # Define grid search object
random_search.fit(fr[x], fr[y])
print("Best R^2:" + str(random_search.best_score_) + "\n")
print("Best params:" + str(random_search.best_params_))
We might be tempted to think that we just had a large improvement; however we must be cautious. The function below creates a more detailed report.
In [31]:
def report_grid_score_detail(random_search, charts=True):
"""Input fit grid search estimator. Returns df of scores with details"""
df_list = []
for line in random_search.grid_scores_:
results_dict = dict(line.parameters)
results_dict["score"] = line.mean_validation_score
results_dict["std"] = line.cv_validation_scores.std()*1.96
df_list.append(results_dict)
result_df = pd.DataFrame(df_list)
result_df = result_df.sort("score", ascending=False)
if charts:
for col in get_numeric(result_df):
if col not in ["score", "std"]:
plt.scatter(result_df[col], result_df.score)
plt.title(col)
plt.show()
for col in list(result_df.columns[result_df.dtypes == "object"]):
cat_plot = result_df.score.groupby(result_df[col]).mean()[0]
cat_plot.sort()
cat_plot.plot(kind="barh", xlim=(.5, None), figsize=(7, cat_plot.shape[0]/2))
plt.show()
return result_df
def get_numeric(X):
"""Return list of numeric dtypes variables"""
return X.dtypes[X.dtypes.apply(lambda x: str(x).startswith(("float", "int", "bool")))].index.tolist()
In [32]:
report_grid_score_detail(random_search).head()
Out[32]:
Based on the grid search report, we can narrow the parameters to search and rerun the analysis. The parameters below were chosen after a few runs:
In [34]:
%%time
params = {"ntrees": randint(30,40),
"max_depth": randint(4,10),
"mtries": randint(4,10),}
custom_cv = H2OKFold(fr, n_folds=5, seed=42) # In small datasets, the fold size can have a big
# impact on the std of the resulting scores. More
random_search = RandomizedSearchCV(model, params, # folds --> Less examples per fold --> higher
n_iter=10, # variation per sample
scoring=scorer,
cv=custom_cv,
random_state=43,
n_jobs=1)
random_search.fit(fr[x], fr[y])
print("Best R^2:" + str(random_search.best_score_) + "\n")
print("Best params:" + str(random_search.best_params_))
report_grid_score_detail(random_search)
Rule of machine learning: Don't use your testing data to inform your training data. Unfortunately, this happens all the time when preparing a dataset for the final model. But on smaller datasets, you must be especially careful.
At the moment, there are no classes for managing data transformations. On the one hand, this requires the user to tote around some extra state, but on the other, it allows the user to be more explicit about transforming H2OFrames.
Basic steps:
First let's normalize the data using the means and standard deviations of the training data. Then let's perform a principal component analysis on the training data and select the top 5 components. Using these components, let's use them to reduce the train and test design matrices.
In [35]:
from h2o.transforms.preprocessing import H2OScaler
from h2o.transforms.decomposition import H2OPCA
In [36]:
y_train = train.pop("Median_value")
y_test = test.pop("Median_value")
In [37]:
norm = H2OScaler()
norm.fit(train)
X_train_norm = norm.transform(train)
X_test_norm = norm.transform(test)
In [38]:
print(X_test_norm.shape)
X_test_norm
Out[38]:
Then, we can apply PCA and keep the top 5 components.
In [39]:
pca = H2OPCA(k=5)
pca.fit(X_train_norm)
X_train_norm_pca = pca.transform(X_train_norm)
X_test_norm_pca = pca.transform(X_test_norm)
In [40]:
# prop of variance explained by top 5 components?
In [41]:
print(X_test_norm_pca.shape)
X_test_norm_pca[:5]
Out[41]:
In [42]:
model = H2ORandomForestEstimator(seed=42)
model.fit(X_train_norm_pca,y_train)
y_hat = model.predict(X_test_norm_pca)
In [43]:
h2o_r2_score(y_test,y_hat)
Out[43]:
Although this is MUCH simpler than keeping track of all of these transformations manually, it gets to be somewhat of a burden when you want to chain together multiple transformers.
"Tranformers unite!"
If your raw data is a mess and you have to perform several transformations before using it, use a pipeline to keep things simple.
Steps:
In [44]:
from h2o.transforms.preprocessing import H2OScaler
from h2o.transforms.decomposition import H2OPCA
from h2o.estimators.random_forest import H2ORandomForestEstimator
In [45]:
from sklearn.pipeline import Pipeline # Import Pipeline <other imports not shown>
model = H2ORandomForestEstimator(seed=42)
pipe = Pipeline([("standardize", H2OScaler()), # Define pipeline as a series of steps
("pca", H2OPCA(k=5)),
("rf", model)]) # Notice the last step is an estimator
pipe.fit(train, y_train) # Fit training data
y_hat = pipe.predict(test) # Predict testing data (due to last step being an estimator)
h2o_r2_score(y_test, y_hat) # Notice the final score is identical to before
Out[45]:
This is so much easier!!!
But, wait a second, we did worse after applying these transformations! We might wonder how different hyperparameters for the transformations impact the final score.
"Yo dawg, I heard you like models, so I put models in your models to model models."
Steps:
In [46]:
pipe = Pipeline([("standardize", H2OScaler()),
("pca", H2OPCA()),
("rf", H2ORandomForestEstimator(seed=42))])
params = {"standardize__center": [True, False], # Parameters to test
"standardize__scale": [True, False],
"pca__k": randint(2, 6),
"rf__ntrees": randint(50,80),
"rf__max_depth": randint(4,10),
"rf__min_rows": randint(5,10), }
# "rf__mtries": randint(1,4),} # gridding over mtries is
# problematic with pca grid over
# k above
from sklearn.grid_search import RandomizedSearchCV
from h2o.cross_validation import H2OKFold
from h2o.model.regression import h2o_r2_score
from sklearn.metrics.scorer import make_scorer
custom_cv = H2OKFold(fr, n_folds=5, seed=42)
random_search = RandomizedSearchCV(pipe, params,
n_iter=30,
scoring=make_scorer(h2o_r2_score),
cv=custom_cv,
random_state=42,
n_jobs=1)
random_search.fit(fr[x],fr[y])
results = report_grid_score_detail(random_search)
results.head()
Out[46]:
Currently Under Development (drop-in scikit-learn pieces):
* Richer set of transforms (only PCA and Scale are implemented)
* Richer set of estimators (only RandomForest is available)
* Full H2O Grid SearchIt is useful to save constructed models to disk and reload them between H2O sessions. Here's how:
In [47]:
best_estimator = random_search.best_estimator_ # fetch the pipeline from the grid search
h2o_model = h2o.get_model(best_estimator._final_estimator._id) # fetch the model from the pipeline
In [48]:
save_path = h2o.save_model(h2o_model, path=".", force=True)
print(save_path)
In [49]:
# assumes new session
my_model = h2o.load_model(path=save_path)
In [50]:
my_model.predict(X_test_norm_pca)
Out[50]: