Exploring the Natality Dataset

Learning Objectives

Explore a BigQuery dataset inside Jupyter notebook
Read data from BigQuery into Pandas dataframe
Examine the distribution of average baby weight across various features

Introduction

In this notebook we'll read data from BigQuery into our notebook to begin some preliminary data exploration of the natality dataset. To beign we'll set environment variables related to our GCP project.



In [ ]:

    
PROJECT = "cloud-training-demos"  # Replace with your PROJECT
BUCKET = "cloud-training-bucket"  # Replace with your BUCKET
REGION = "us-central1"            # Choose an available region for Cloud MLE
TFVERSION = "1.14"                # TF version for CMLE to use



In [ ]:

    
import os
os.environ["BUCKET"] = BUCKET
os.environ["PROJECT"] = PROJECT
os.environ["REGION"] = REGION
os.environ["TFVERSION"] = TFVERSION



In [ ]:

    
%%bash
if ! gsutil ls | grep -q gs://${BUCKET}/; then
    gsutil mb -l ${REGION} gs://${BUCKET}
fi

Explore data

The data is natality data (record of births in the US). My goal is to predict the baby's weight given a number of factors about the pregnancy and the baby's mother. Later, we will want to split the data into training and eval datasets. The hash of the year-month will be used for that -- this way, twins born on the same day won't end up in different cuts of the data.



In [ ]:

    
# Create SQL query using natality data after the year 2000
query_string = """
SELECT
    weight_pounds,
    is_male,
    mother_age,
    plurality,
    gestation_weeks,
    FARM_FINGERPRINT(CONCAT(CAST(YEAR AS STRING), CAST(month AS STRING))) AS hashmonth
FROM
    publicdata.samples.natality
WHERE
    year > 2000
"""



In [ ]:

    
# Call BigQuery and examine in dataframe
from google.cloud import bigquery
bq = bigquery.Client(project = PROJECT)

df = bq.query(query_string + "LIMIT 100").to_dataframe()
df.head()

Let's write a query to find see how the number of babies and their average weight is distributed across a given field. This is important to ensure that we have enough examples of each data value, and to verify our hunch that the parameter has predictive value.

Exercise 1

Consider the field is_male. Write a query that will return the unique values for that field (i.e. False or True) as well as the count of the number of babies for each value and the average weight for each value.

For example, your query should return the following result:

is_male	num_babies	avg_wt
False	16245054	7.104715
True	17026860	7.349797

Hint: Look at the usage of operations like COUNT(), AVG() and GROUP BY in SQL.



In [ ]:

    
query = """
SELECT
    # TODO: Your code goes here
FROM
    publicdata.samples.natality
WHERE
    year > 2000
"""

df = # TODO: Your code goes here
df.head()

Exercise 2

Now write a function that will generalize your query above. That is, write a function that will take a column name from the natality dataset and return a dataframe containing the unique values for that field as well as the count of the number babies for each value and the average weight for each value.



In [ ]:

    
def get_distinct_values(column_name):
    sql_query = """
    # TODO: Your code goes here
    """.format(column_name)
    return # TODO: Your code goes here

Exercise 3

Use the get_distinct_values function you created above to examine the distributions for the fields is_male, mothers_age, plurality and gestation_weeks. After creating the dataframe, plot your results using df.plot. Have a look at the documentation for df.plot here.



In [ ]:

    
# Use a line plots to see mother_age with avg_wt linear and num_babies logarithmic
df = # TODO: Your code goes here

df = df.sort_values("mother_age")
df.plot(# TODO: Your code goes here
df.plot(# TODO: Your code goes here



In [ ]:

    
# Use a bar plot to see plurality(singleton, twins, etc.) with avg_wt linear and num_babies logarithmic
df = # TODO: Your code goes here

df = df.sort_values("plurality")
df.plot(# TODO: Your code goes here
df.plot(# TODO: Your code goes here



In [ ]:

    
# Use a bar plot to see gestation_weeks with avg_wt linear and num_babies logarithmic
df = # TODO: Your code goes here

df = df.sort_values("gestation_weeks")
df.plot(# TODO: Your code goes here
df.plot(# TODO: Your code goes here

Conclusion

All these factors seem to play a part in the baby's weight. Male babies are heavier on average than female babies. Teenaged and older moms tend to have lower-weight babies. Twins, triplets, etc. are lower weight than single births. Preemies weigh in lower as do babies born to single moms. In addition, it is important to check whether you have enough data (number of babies) for each input value. Otherwise, the model prediction against input values that doesn't have enough data may not be reliable.

In the next notebook, we will develop a machine learning model to combine all of these factors to come up with a prediction of a baby's weight.

Copyright 2017-2018 Google Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License