The Data Observatory is a spatial data repository that enables data scientists to augment their data and broaden their analysis. It offers a wide range of datasets from around the globe.
This guide is intended for those who want to start augmenting their own data using CARTOframes and wish to explore CARTO's public Data Observatory catalog to find datasets that best fit their use cases and analyses.
Note: The catalog is public and you don't need a CARTO account to search for available datasets
In this guide we walk through the Data Observatory catalog looking for demographics data in the US.
The catalog is comprised of thousands of curated spatial datasets, so when searching for data the easiest way to find what you are looking for is to make use of a faceted search. A faceted (or hierarchical) search allows you to narrow down search results by applying multiple filters based on faceted classification of catalog datasets.
Datasets are organized in three main hierarchies:
For our analysis we are looking for demographic datasets in the US with a spatial resolution at the block group level.
We can start by discovering which available geographies (or spatial resolutions) we have for demographic data in the US, by filtering the catalog by country and category and listing the available geographies.
Let's start exploring the available categories of data for the US:
In [1]:
from cartoframes.data.observatory import Catalog
Catalog().country('usa').categories
Out[1]:
For the case of the US, the Data Observatory provides six different categories of datasets. Let's discover the available spatial resolutions for the demographics category (which at a first sight will contain the population data we need).
In [2]:
from cartoframes.data.observatory import Catalog
geographies = Catalog().country('usa').category('demographics').geographies
geographies
Out[2]:
Let's filter the geographies by those that contain information at the level of blockgroup. For that purpose we are converting the geographies to a pandas DataFrame and search for the string blockgroup in the id of the geographies:
In [3]:
df = geographies.to_dataframe()
df[df['id'].str.contains('blockgroup', case=False, na=False)]
Out[3]:
We have three available datasets, from three different providers: Michael Bauer International, Open Data and AGS. For this example, we are going to look for demographic datasets for the MBI blockgroups geography mbi_blockgroups_535aed6d:
In [4]:
datasets = Catalog().country('usa').category('demographics').geography('mbi_blockgroups_535aed6d').datasets
datasets
Out[4]:
Let's continue with the data discovery. We have 6 datasets in the US with demographics information at the level of MBI blockgroups:
In [5]:
datasets.to_dataframe()
Out[5]:
They comprise different information: consumer spending, retail potential, consumer profiles, etc.
At a first sight, it looks the dataset with data_source_id: sociodemographic might contain the population information we are looking for. Let's try to understand a little bit better what data this dataset contains by looking at its variables:
In [6]:
from cartoframes.data.observatory import Dataset
dataset = Dataset.get('ags_sociodemogr_a7e14220')
variables = dataset.variables
variables
Out[6]:
In [7]:
from cartoframes.data.observatory import Dataset
vdf = variables.to_dataframe()
vdf
Out[7]:
We can see there are several variables related to population, so this is the Dataset we are looking for.
In [8]:
vdf[vdf['description'].str.contains('pop', case=False, na=False)]
Out[8]:
The Data Observatory catalog is not only a repository of curated spatial datasets, it also contains valuable information that helps on understanding better the underlying data for every dataset, so you can take an informed decision on what data best fits your problem.
Some of the augmented metadata you can find for each dataset in the catalog is:
head and tail methods to get a glimpse of the actual data. This helps you to understand the available columns, data types, etc. To start modelling your problem right away.geom_coverage to visualize on a map the geographical coverage of the data in the Dataset.counts, fields_by_type and a full describe method with stats of the actual values in the dataset, such as: average, stdev, quantiles, min, max, median for each of the variables of the dataset.You don't need a subscription to a dataset to be able to query the augmented metadata, it's just publicly available for anyone exploring the Data Observatory catalog.
Let's overview some of that information, starting by getting a glimpse of the ten first or last rows of the actual data of the dataset:
In [9]:
from cartoframes.data.observatory import Dataset
dataset = Dataset.get('ags_sociodemogr_a7e14220')
In [10]:
dataset.head()
Out[10]:
Alternatively, you can get the last ten ones with dataset.tail()
An overview of the coverage of the dataset
In [11]:
dataset.geom_coverage()
Out[11]:
Some stats about the dataset:
In [12]:
dataset.counts()
Out[12]:
In [13]:
dataset.fields_by_type()
Out[13]:
In [14]:
dataset.describe()
Out[14]:
Every Dataset instance in the catalog contains other useful metadata:
In [15]:
dataset.to_dict()
Out[15]:
There's also some intersting metadata, for each variable in the dataset:
Variables are the most important asset in the catalog and when exploring datasets in the Data Observatory catalog it's very important that you understand clearly what variables are available to enrich your own data.
For each Variable in each dataset, the Data Observatory provides (as it does with datasets) a set of methods and attributes to understand their underlaying data.
Some of them are:
head and tail methods to get a glimpse of the actual data and start modelling your problem right away.counts, quantiles and a full describe method with stats of the actual values in the dataset, such as: average, stdev, quantiles, min, max, median for each of the variables of the dataset.histogram plot with the distribution of the values on each variable.Let's overview some of that augmented metadata for the variables in the AGS population dataset.
In [16]:
from cartoframes.data.observatory import Variable
variable = Variable.get('POPCY_4534fac4')
variable
Out[16]:
In [17]:
variable.to_dict()
Out[17]:
There's also some utility methods ot understand the underlying data for each variable:
In [18]:
variable.head()
Out[18]:
In [19]:
variable.counts()
Out[19]:
In [20]:
variable.quantiles()
Out[20]:
In [21]:
variable.histogram()
In [22]:
variable.describe()
Out[22]:
Once you have explored the catalog and have detected a dataset with the variables you need for your analysis and the right spatial resolution, you have to look at the is_public_data to know if you can just use it from CARTOframes or you first need to subscribe for a license.
Subscriptions to datasets allow you to use them from CARTOframes to enrich your own data or to download them. See the enrichment guides for more information about this.
Let's see the dataset and geography in our previous example:
In [23]:
dataset = Dataset.get('ags_sociodemogr_a7e14220')
In [24]:
dataset.is_public_data
Out[24]:
In [25]:
from cartoframes.data.observatory import Geography
geography = Geography.get(dataset.geography)
In [26]:
geography.is_public_data
Out[26]:
Both dataset and geography are not public data, that means you need a subscription to be able to use them to enrich your own data.
To subscribe to data in the Data Observatory catalog you need a CARTO account with access to Data Observatory
In [27]:
from cartoframes.auth import set_default_credentials
set_default_credentials('creds.json')
In [28]:
dataset.subscribe()
In [29]:
geography.subscribe()
Licenses to data in the Data Observatory grant you the right to use the data subscribed for the period of one year. Every dataset or geography you want to use to enrich your own data, as long as they are not public data, require a valid license.
You can check the actual status of your subscriptions directly from the catalog.
In [30]:
Catalog().subscriptions()
Out[30]:
In this guide you've seen how to explore the Data Observatory catalog to identify variables of datasets that you can use to enrich your own data.
You've learned how to:
Geography, Dataset and their Variables.We also recommend checking out the resources below to learn more about the Data Observatory catalog: