How to make a clustered heatmap

Now we'll break down how to read the clustered heatmap we made in 1.3_explore_gene_dropout_via_distance_correlation_linkage_clustering

Read Expression data using pandas. Notice that pandas can read URLs (!), not just files on your computer!

Exercise 1

Now use pd.read_csv to read the csv file of the cell metadata

To correlate columns of dataframes in pandas, you use the function .corr. Let's look at the documentation of .corr

1. Is the default method Pearson or Spearman correlation?
2. Can you correlate between rows, or only between columns?

Since .corr only correlates between columns, we need to transpose our dataframe. Here's a little animation of matrix transposition from Wikipedia:

Exercise 2

Transpose the expression matrix so the cells are the columns, which makes it easy to calculate correlations. How do you transpose a dataframe in pandas? (hint: google knows everything)

Exercise 3

Use .corr to calculate the Spearman correlation of the transposed expression dataframe. Make sure to print the shape, and show the head of the resulting dataframe.

Pro tip: if your matrix is really big, here's a trick to make spearman correlations faster

Remember that spearman correlation is equal to performing pearson correlation on the ranks? Well, that's exactly what's happening inside the .corr(method='spearman') function! Every time it's calculating spearman, it's converting each row to ranks, which means that it's double-converting to ranks since it has to do it for each pair. Let's cut the work in half by converting to ranks FIRST. Let's take a look at the options for .rank:

Notice we can specify axis=1 or axis=0, but what does that really mean? Was this ascending along rows, or ascending along columns?

To figure this out, let's use a small, simple dataframe:

Exercise 4

Try axis=0 when using rank on this df

Did that make ranks ascending along columns or along rows?

Exercise 5

Now try axis=1 when using rank on this df

Did that make ranks ascending along columns or along rows?

Exercise 6

To get the gene (row) ranks for each cell (column), do we want axis=1 or axis=0? Perform .rank on the transposed expression matrix (expression_t), print the shape of the resulting ranks, and show the head() of it.

Exercise 6

Now that you're armed with all this information, we'll calculate the ranks. While you're at it, let's compare the time it takes to run ("runtime") of .corr(method="pearson") on the ranks matrix vs .corr(method="spearman") on the expression matrix.

1. Perform pearson correlation on the ranks
2. Check that it is equal to the expression spearman correlation.
3. Use the %timeit magic to check the runtimes of .corr on the ranks and expression matrices. (Feel free to calculate the expression correlation again, below)
   1. Note that when you use timeit, you cannot assign any variables -- using an equals sign doesn't work here.
4. How much time did it take, in comparison? What's the order of magnitude difference?

Use as many cells as you need.

Use inequality to see if any points are not the same. If this is equal to zero, then we know that they are ALL the same.

This is a flip of checking for equality, which is a little trickier because then you have to know exactly how many items are in the matrix. Since we have a 300x300 matrix, that multiplication is a little easier to do in your head and know that you got the right answer.

Make a heatmap!!

Now we are ready to make a clustered heatmap! We'll use seaborn's sns.clustermap. Let's read the documentation for sns.clustermap. What is the default distance metric and linkage method?

Exercise 7

Now run sns.clustermap on either the ranks or expression correlation matrices, since they are equal :)

How can we add the colors labeling the rows and columns? Check the documentation for sns.clustermap again:

Exercise 8

Since I am not a color design expert, I defer to color design experts in choosing my color palettes. One such expert is Cynthia Brewer, who made a ColorBrewer (hah!) list of color maps for both increasing quantity (shades), and for categories (differing colors).

As a reference, I like using this demo of every ColorBrewer scale. Hover over the palette to see its name.

Thankfully, seaborn has the ColorBrewer color maps built-in. Let's see what this output is

Remember -- we never make a variable without looking at it first!!

Huh that's a bunch of weird numbers. What do they mean? Turns out it's a value from 0 to 1 representing the red, green, and blue (RGB) color channels that computers understand. But I'm not a computer .... what am I supposed to do??

Turns out, seaborn also has a very convenient function called palplot to plot the entire palette. This lets us look at the variable without having to convert from RGB

Exercise 9

• Get the color palette for the "Set2" colormap and specify that you want 6 colors (read the documentation of sns.color_palette)
• Plot the color palette

If you are more advanced and want access to more colormaps, I recommend checking out palettable.

Assign colors to clusters

To set a specific color to each cluster, we'll need to see the unique clusters here. For an individual column (called a "Series" in pandas-speak), how can we get only the unique items?

Exercise 10

Get the unique values from the column "cluster_celltype_with_id". Remember, always look at the variable you created!

Detour: zip and dict

To map colors to each cluster name, we need to talk about some built-in functions in Python, called zip and dict

For this next part, we'll use the built-in function zip which is very useful. It acts like a zipper (like for clothes) to glue together the pairs of items in two lists:

To be memory efficient, this doesn't show us what's inside right away. To look inside a zip object, we can use list:

Exercise 11

What happened to "please" and "thank you" from english? Make another list, called spanish2, that contains the Spanish words for "please" and "thank you" (again, google knows everything), then call zip on english and spanish2. Don't forget to use list on them!

Now we'll use a dictionary dict to make a lookup table that uses the pairing made by zip, using the first item as the "key" (what you use to look up) and the second item as the "value" (the result of the lookup)

You can think of it as a translator -- use the word in English to look up the word in Spanish.

Now we can use English words to look up the word in Spanish! We use the square brackets and the english word we want to use, to look up the spanish word.

Exercise 12

Make an spanish_to_english dictionary and look up the English word for "por favor"

Okay, detour over! Switching from linguistics back to biology :)

Exercise 13

Use dict and zip to create a variable called id_to_color that assigns labels in cluster_ids_unique to a color in set2

Now we want to use this id_to_color lookup table to make a long list of colors for each cell.

As an example, let's use the celltypes column to make a list of each celltype color first. Notice that we can use cell_metadata.celltype or cell_metadata['celltype'] to get the column we want.

We can only use the 'dot' notation because our column name has no unfriendly characters like spaces, dashes, or dots -- characters that mean something special in Python.

Now we'll use the existing column cell_metadata.celltype to make a list of colors for each celltype

Exercise 14

Make a variable called per_cell_cluster_color that uses the id_to_color dictionary to look up the color for each value in the cluster_celltype_with_id column of cluster_metadata

Exercise 15

Now use the cluster colors to label the rows and columns in sns.clustermap. How can

We can also combine the celltype and cluster colors we created to create a double-layer colormap!