This notebook demonstrates how to import a machine learning model file from the S3 location and deploy the Models + Analysis on the Redis Labs Cloud (https://redislabs.com/redis-cloud) endpoint named "CACHE".
Download the S3 IRIS Model archive from the configured S3 Bucket + Key and decompress + extract the pickled historcial analysis and previously built models. This includes examples from the IRIS sample dataset and requires you to have a valid S3 Bucket storing the models and are comfortable paying for the download costs for downloading the objects from S3 (https://aws.amazon.com/s3/pricing/).
Once uploaded to the S3 Bucket you should be able to view the files have a similar disk size:
After importing, the Models and Analysis are available to any other Sci-pype instance with connectivity to the same Redis Labs Online Cloud cache instance.
This notebook was built from the importer command line examples:
https://github.com/jay-johnson/sci-pype/tree/master/bins/ml/importers/rl_import_iris_regressor.py
https://github.com/jay-johnson/sci-pype/tree/master/bins/ml/importers
In [1]:
# Setup the Sci-pype environment
import sys, os
# Only Redis Labs is needed for this notebook:
os.environ["ENV_DEPLOYMENT_TYPE"] = "RedisLabs"
# Load the Sci-pype PyCore as a named-object called "core" and environment variables
from src.common.load_ipython_env import *
Import the Models from S3 and store the extracted Models + Analysis in the Redis Labs Online Cloud cache endpoint named "Cache" listening on port 16005.
Please make sure the environment variables are set correctly and the S3 Bucket exists:
ENV_AWS_KEY=<AWS API Key>
ENV_AWS_SECRET=<AWS API Secret>For docker containers make sure to set these keys in the correct Jupyter env file and restart the container:
<repo base dir>/justredis/redis-labs.env
<repo base dir>/local/jupyter.env
<repo base dir>/test/jupyter.env
In [2]:
ds_name = "iris_regressor"
In [3]:
data_dir = str(os.getenv("ENV_DATA_SRC_DIR", "/opt/work/data/src"))
if not os.path.exists(data_dir):
os.mkdir(data_dir, 0777)
In [4]:
s3_bucket = "unique-bucket-name-for-datasets" # name this something under your AWS Account (This might be open to the public in the future...stay tuned)
s3_key = "dataset_" + core.to_upper(ds_name) + ".cache.pickle.zlib"
s3_loc = str(s3_bucket) + ":" + str(s3_key)
In [5]:
ml_file = data_dir + "/" + str(s3_key)
In [6]:
lg("-------------------------------------------------", 6)
lg("Importing Models and Analysis from S3 into Caching Models from CACHE - S3Loc(" + str(s3_loc) + ") File(" + str(ml_file) + ")", 6)
lg("", 6)
if os.path.exists(ml_file) == False:
s3_loc = str(s3_bucket) + ":" + str(s3_key)
lg("Downloading ModelFile S3Loc(" + str(s3_loc) + ")", 6)
download_results = core.s3_download_and_store_file(s3_loc, ml_file, core.get_rds(), core.get_dbs(), debug)
if download_results["Status"] != "SUCCESS":
lg("ERROR: Stopping processing for errror: " + str(download_results["Error"]), 0)
else:
lg("", 6)
lg("Done Downloading ModelFile S3Loc(" + str(s3_loc) + ") File(" + str(download_results["Record"]["File"]) + ")", 5)
ml_file = download_results["Record"]["File"]
else:
lg("", 6)
lg("Continuing with the existing file.", 5)
lg("", 6)
# end of downloading from s3 if it's not locally available
In [7]:
ra_name = "CACHE"
lg("Importing(" + str(ml_file) + ") Models and Analysis into Redis(" + str(ra_name) + ")", 6)
cache_req = {
"RAName" : ra_name,
"DSName" : str(ds_name),
"TrackingID": "",
"ModelFile" : ml_file,
"S3Loc" : s3_loc
}
upload_results = core.ml_load_model_file_into_cache(cache_req, core.get_rds(), core.get_dbs(), debug)
if upload_results["Status"] == "SUCCESS":
lg("", 6)
lg("Done Loading Model File for DSName(" + str(ds_name) + ") S3Loc(" + str(cache_req["S3Loc"]) + ")", 5)
lg("", 6)
lg("Importing and Caching Completed", 5)
lg("", 6)
else:
lg("", 6)
lg("ERROR: Failed Loading Model File(" + str(cache_req["ModelFile"]) + ") into Cache for DSName(" + str(ds_name) + ")", 6)
lg(upload_results["Error"], 6)
lg("", 6)
# end of if success
Now that the XGB models from the S3 artifact are deployed to the Machine Learning data store within the Redis Labs Cloud endpoint, you can run the following notebooks to checkout how the Sci-pype workflow continues using this data science artifact:
This notebook was built from the importer command line examples:
https://github.com/jay-johnson/sci-pype/tree/master/bins/ml/importers/rl_import_iris_regressor.py
https://github.com/jay-johnson/sci-pype/tree/master/bins/ml/importers