Overview:
The DSA database stores annotations in an (x,y) coordinate list format. Some object localization algorithms like Faster-RCNN take coordinate formats whereas others (eg Mask R-CNN) require some form of object segmentation mask image whose pixel values encode not only class but instance information (so that individual objects of the same class can be distinguished).
This notebook demonstrates tools to convert annotations into contours or masks that can be used with algorithms like Mask-RCNN. There are two approaches for generating these data:
Generate contours or an object segmentation mask image from a region defined by user-specified coordinates.
Generate contours or object segmentation mask images from annotations contained within region-of-interest (ROI) annotations. This involves mapping annotations to these ROIs and creating one image per ROI.
The examples below extend approaches described in Amgad et al, 2019:
Mohamed Amgad, Habiba Elfandy, Hagar Hussein, ..., Jonathan Beezley, Deepak R Chittajallu, David Manthey, David A Gutman, Lee A D Cooper, Structured crowdsourcing enables convolutional segmentation of histology images, Bioinformatics, , btz083, https://doi.org/10.1093/bioinformatics/btz083
A csv file like the one in
histomicstk/annotations_and_masks/tests/test_files/sample_GTcodes.csv is needed to define what group each pixel value corresponds to in the mask image, to define the overlay order of various annotation groups, and which groups are considered to be ROIs. Note that the term "group" here comes from the annotation model where each group represents a class like "tumor" or "necrosis" and is associated with a an annotation style.
What is the difference between this and annotations_to_masks_handler?
The difference between this and version 1, found at
histomicstk.annotations_and_masks.annotations_to_masks_handler
is that this (version 2) gets the contours first, including cropping
to wanted ROI boundaries and other processing using shapely, and then
parses these into masks. This enables us to differentiate various objects
to use the data for object localization/classification/segmentation
tasks. If you would like to get semantic segmentation masks instead, i.e. you do
not care about individual objects, you can use either version 1
or this handler using the semantic run mode.
They re-use much of the same code-base, but some edge cases maybe better handled
by version 1. For example, since this version uses shapely first to crop, some
objects may be incorrectly parsed by shapely. Version 1, using PIL.ImageDraw
may not have these problems.
Bottom line is: if you need semantic segmentation masks, it is probably
safer to use version 1 (annotations to masks handler), whereas if you need object segmentation masks, this handler should be used in object run mode.
Where to look?
|_ histomicstk/
|_annotations_and_masks/
| |_annotation_and_mask_utils.py
| |_annotations_to_object_mask_handler.py
|_tests/
|_ test_annotation_and_mask_utils.py
|_ test_annotations_to_object_mask_handler.py
In [1]:
import os
import copy
import girder_client
from pandas import DataFrame, read_csv
import tempfile
from imageio import imread
import matplotlib.pyplot as plt
%matplotlib inline
from histomicstk.annotations_and_masks.annotation_and_mask_utils import (
get_bboxes_from_slide_annotations,
scale_slide_annotations, get_scale_factor_and_appendStr)
from histomicstk.annotations_and_masks.annotations_to_object_mask_handler import (
annotations_to_contours_no_mask, contours_to_labeled_object_mask,
get_all_rois_from_slide_v2)
#Some nice default configuration for plots
plt.rcParams['figure.figsize'] = 7, 7
titlesize = 16
In [2]:
CWD = os.getcwd()
APIURL = 'http://candygram.neurology.emory.edu:8080/api/v1/'
SAMPLE_SLIDE_ID = '5d586d57bd4404c6b1f28640'
GTCODE_PATH = os.path.join(CWD, '../../tests/test_files/sample_GTcodes.csv')
# connect to girder client
gc = girder_client.GirderClient(apiUrl=APIURL)
# gc.authenticate(interactive=True)
gc.authenticate(apiKey='kri19nTIGOkWH01TbzRqfohaaDWb6kPecRqGmemb')
# just a temp directory to save masks for now
BASE_SAVEPATH = tempfile.mkdtemp()
SAVEPATHS = {
'mask': os.path.join(BASE_SAVEPATH, 'masks'),
'rgb': os.path.join(BASE_SAVEPATH, 'rgbs'),
'contours': os.path.join(BASE_SAVEPATH, 'contours'),
'visualization': os.path.join(BASE_SAVEPATH, 'vis'),
}
for _, savepath in SAVEPATHS.items():
os.mkdir(savepath)
# What resolution do we want to get the images at?
# Microns-per-pixel / Magnification (either or)
MPP = 2.5 # <- this roughly translates to 4x magnification
MAG = None
This contains the ground truth codes and information dataframe. This is a dataframe that is indexed by the annotation group name and has the following columns:
group: group name of annotation (string), eg. "mostly_tumor"overlay_order: int, how early to place the annotation in the
mask. Larger values means this annotation group is overlayed
last and overwrites whatever overlaps it.GT_code: int, desired ground truth code (in the labeled mask)
Pixels of this value belong to corresponding group (class)is_roi: Flag for whether this group marks 'special' annotations that encode the ROI boundaryis_background_class: Flag, whether this group is the default
fill value inside the ROI. For example, you may descide that
any pixel inside the ROI is considered stroma.NOTE:
Zero pixels have special meaning and do not encode specific ground truth class. Instead, they simply mean 'Outside ROI' and should be ignored during model training or evaluation.
In [3]:
# read GTCodes file
GTCodes_dict = read_csv(GTCODE_PATH)
GTCodes_dict.index = GTCodes_dict.loc[:, 'group']
GTCodes_dict = GTCodes_dict.to_dict(orient='index')
In [4]:
GTCodes_dict.keys()
Out[4]:
In [5]:
GTCodes_dict['mostly_tumor']
Out[5]:
Algorithms like Mask-RCNN consume coordinate data describing the boundaries of objects. The function annotations_to_contours_no_mask generates this countour data for user-specified region. These coordinate data in these contours is relative to the region frame instead of the whole-slide image frame.
In [6]:
print(annotations_to_contours_no_mask.__doc__)
In [7]:
# common params for annotations_to_contours_no_mask()
annotations_to_contours_kwargs = {
'MPP': MPP, 'MAG': MAG,
'linewidth': 0.2,
'get_rgb': True,
'get_visualization': True,
'text': False,
}
As shown in the example for generating semantic segmentation masks, this method can be run in four run modes: 1. wsi 2. min_bounding_box 3. manual_bounds 4. polygonal_bounds. Here we test the basic 'manual_bounds' mode where the boundaries of the region you want are provided at base/scan magnification.
In [8]:
bounds = {
'XMIN': 58000, 'XMAX': 63000,
'YMIN': 35000, 'YMAX': 39000
}
In [9]:
# get specified region, let the method get and scale annotations
roi_out = annotations_to_contours_no_mask(
gc=gc, slide_id=SAMPLE_SLIDE_ID,
mode='manual_bounds', bounds=bounds,
**annotations_to_contours_kwargs)
The result is an rgb image, contours and a visualization. Let's take a look at these below.
In [10]:
roi_out.keys()
Out[10]:
In [11]:
roi_out['bounds']
Out[11]:
In [12]:
for imstr in ['rgb', 'visualization']:
plt.imshow(roi_out[imstr])
plt.title(imstr)
plt.show()
In [13]:
DataFrame(roi_out['contours']).head()
Out[13]:
Note that if the above function call is made repeatedly for the same slide
(e.g. to iterate over multiple regions), multiple get requests would be created to retrieve annotations from the server. To improve efficiency when handling multiple regions in the same slide, we could manually get annotations
and scale them down/up to desired resolution, and pass them to annotations_to_contours_no_mask().
In [14]:
# get annotations for slide
slide_annotations = gc.get('/annotation/item/' + SAMPLE_SLIDE_ID)
# scale up/down annotations by a factor
sf, _ = get_scale_factor_and_appendStr(
gc=gc, slide_id=SAMPLE_SLIDE_ID, MPP=MPP, MAG=MAG)
slide_annotations = scale_slide_annotations(slide_annotations, sf=sf)
# get bounding box information for all annotations
element_infos = get_bboxes_from_slide_annotations(slide_annotations)
In [15]:
# get specified region -- manually providing scaled annotations
roi_out = annotations_to_contours_no_mask(
gc=gc, slide_id=SAMPLE_SLIDE_ID,
mode='manual_bounds', bounds=bounds,
slide_annotations=slide_annotations, element_infos=element_infos,
**annotations_to_contours_kwargs)
In [16]:
roi_out['bounds']
Out[16]:
In [17]:
# get ROI bounding everything
minbbox_out = annotations_to_contours_no_mask(
gc=gc, slide_id=SAMPLE_SLIDE_ID,
mode='min_bounding_box', **annotations_to_contours_kwargs)
In [18]:
minbbox_out['bounds']
Out[18]:
In [19]:
plt.imshow(minbbox_out['visualization'])
Out[19]:
wsi mode creates a scaled version of the entire whole-slide image and all annotations contained within.
NOTE:
This does not rely on tiles and processes the image at whatever magnification you want. You can supress the RGB or visualization outputs and to just fetch the contours or object segmentation mask (see below), providing a bigger magnification range before encountering memory problems.
In [20]:
# get entire wsi region
get_kwargs = copy.deepcopy(annotations_to_contours_kwargs)
get_kwargs['MPP'] = 5.0 # otherwise it's too large!
wsi_out = annotations_to_contours_no_mask(
gc=gc, slide_id=SAMPLE_SLIDE_ID,
mode='wsi', **get_kwargs)
In [21]:
wsi_out['bounds']
Out[21]:
In [22]:
plt.imshow(wsi_out['visualization'])
plt.show()
In [23]:
plt.imshow(wsi_out['visualization'][1500:2000, 2800:3300])
plt.show()
This function utilizes the polygonal_bounds mode of the get_image_and_mask_from_slide() method to generate a set of outputs for each ROI annotation.
In the example above we focused on generating contour data to represent objects. Below we focus on generating object segmentation mask images.
In [24]:
print(get_all_rois_from_slide_v2.__doc__)
The above method mainly relies on contours_to_labeled_object_mask(), described below.
In [25]:
print(contours_to_labeled_object_mask.__doc__)
In [26]:
get_all_rois_kwargs = {
'gc': gc,
'slide_id': SAMPLE_SLIDE_ID,
'GTCodes_dict': GTCodes_dict,
'save_directories': SAVEPATHS,
'annotations_to_contours_kwargs': annotations_to_contours_kwargs,
'slide_name': 'TCGA-A2-A0YE',
'mode': 'object',
'verbose': True,
'monitorprefix': 'test',
}
savenames = get_all_rois_from_slide_v2(**get_all_rois_kwargs)
In [27]:
savenames[0]
Out[27]:
In [28]:
# visualization of contours over RGBs
for savename in savenames:
vis = imread(savename["visualization"])
plt.imshow(vis)
plt.title(os.path.basename(savename["visualization"]))
plt.show()
In [29]:
mask = imread(savename["mask"])
maskname = os.path.basename(savename["mask"])
plt.imshow(mask[..., 0])
plt.title(maskname + ': LABELS')
plt.show()
plt.imshow(mask[..., 1] * mask[..., 2])
plt.title(maskname + ': OBJECTS')
plt.show()