Down sampling is typically done when the input tables are large (e.g. each containing more than 100K tuples). This IPython notebook illustrates how to down sample two large tables that are loaded in memory.
First, we need to import py_entitymatching package as follows:
In [2]:
import py_entitymatching as em
Next, we need to read in the input tables. For the purposes of this notebook we will use two large datasets: Citeseer and DBLP. You can download Citeseer dataset from http://pages.cs.wisc.edu/~anhai/data/falcon_data/citations/citeseer.csv and DBLP dataset from http://pages.cs.wisc.edu/~anhai/data/falcon_data/citations/dblp.csv. Once downloaded, save these datasets as 'citeseer.csv' and 'dblp.csv' in the current directory.
In [5]:
# Read the CSV files
A = em.read_csv_metadata('./citeseer.csv',low_memory=False) # setting the parameter low_memory to False to speed up loading.
B = em.read_csv_metadata('./dblp.csv', low_memory=False)
In [6]:
len(A), len(B)
Out[6]:
In [7]:
A.head()
Out[7]:
In [8]:
B.head()
Out[8]:
In [9]:
# Set 'id' as the keys to the input tables
em.set_key(A, 'id')
em.set_key(B, 'id')
Out[9]:
In [10]:
# Display the keys
em.get_key(A), em.get_key(B)
Out[10]:
Now, the input tables can be down sampled as shown below:
In [12]:
# Downsample the datasets
sample_A, sample_B = em.down_sample(A, B, size=1000, y_param=1, show_progress=False)
In the down_sample command, set the size
to the number of tuples that should be sampled from B (this would be the size of sampled B table) and set the y_param
to be the number of tuples to be picked from A (for each tuple in sampled B table). In the above, we set the number of tuples to be sampled from B to be 1000. We set the y_param
to 1.
In [13]:
# Display the number of tuples in the sampled datasets
len(sample_A), len(sample_B)
Now, the input tables A
and B
have been down sampled to smaller tables sample_A
and sample_B
.
In [ ]:
# Show the metadata of sample_A, sample_B
em.show_properties(sample_A)
In [ ]:
em.show_properties(sample_B)
Note that the sampled tables retain the same properies as the input tables.