1. Exploring natality dataset

This notebook illustrates:

Exploring a BigQuery dataset using Datalab



In [ ]:

    
!sudo chown -R jupyter:jupyter /home/jupyter/training-data-analyst



In [ ]:

    
# change these to try this notebook out
BUCKET = 'cloud-training-demos-ml'   # CHANGE this to a globally unique value. Your project name is a good option to try.
PROJECT = 'cloud-training-demos'     # CHANGE this to your project name
REGION = 'us-central1'               # CHANGE this to one of the regions supported by Cloud AI Platform https://cloud.google.com/ml-engine/docs/tensorflow/regions



In [ ]:

    
import os
os.environ['BUCKET'] = BUCKET
os.environ['PROJECT'] = PROJECT
os.environ['REGION'] = REGION



In [ ]:

    
%%bash
if ! gsutil ls | grep -q gs://${BUCKET}/; then
  gsutil mb -l ${REGION} gs://${BUCKET}
fi

Explore data

The data is natality data (record of births in the US). My goal is to predict the baby's weight given a number of factors about the pregnancy and the baby's mother. Later, we will want to split the data into training and eval datasets. The hash of the year-month will be used for that -- this way, twins born on the same day won't end up in different cuts of the data.



In [ ]:

    
# Create SQL query using natality data after the year 2000
query = """
SELECT
  weight_pounds,
  is_male,
  mother_age,
  plurality,
  gestation_weeks,
  FARM_FINGERPRINT(CONCAT(CAST(YEAR AS STRING), CAST(month AS STRING))) AS hashmonth
FROM
  publicdata.samples.natality
WHERE year > 2000
"""



In [ ]:

    
# Call BigQuery and examine in dataframe
from google.cloud import bigquery
df = bigquery.Client().query(query + " LIMIT 100").to_dataframe()
df.head()

Lab Task #1

Using the above code as an example, write a query to find the unique values for each of the columns and the count of those values for babies born after the year 2000. For example, we want to get these values:

is_male num_babies  avg_wt
False   16245054    7.104715
 True   17026860    7.349797

This is important to ensure that we have enough examples of each data value, and to verify our hunch that the parameter has predictive value.

Hint (highlight to see):

Use COUNT(), AVG() and GROUP BY. For example:

SELECT
  is_male,
  COUNT(1) AS num_babies,
  AVG(weight_pounds) AS avg_wt
FROM
  publicdata.samples.natality
WHERE
  year > 2000
GROUP BY
  is_male

</p>

Lab Task #2

Which factors seem to play a part in the baby's weight?

Bonus: Draw graphs to illustrate your conclusions

Hint (highlight to see):

The simplest way to plot is to use Pandas' built-in plotting capability

from google.cloud import bigquery
df = bigquery.Client().query(query).to_dataframe()
df.plot(x='is_male', y='num_babies', logy=True, kind='bar');
df.plot(x='is_male', y='avg_wt', kind='bar');

Copyright 2017-2018 Google Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License