Working with objects

working_with_objects

Working with objects¶

This notebook will show how to exchange objects (e.g. annotations, detections) between QuPath and Python.

As we will communicate with QuPath, it is recommended to go through the communicating_with_qupath.ipynb notebook first. Also, as we will work with images, it is recommended to go through the opening_images.ipynb notebook first.

A few classes and functions of the QuBaLab package will be presented throughout this notebook. To have more details on them, you can go to the documentation on https://qupath.github.io/qubalab/ and type a class/function name in the search bar. You will then see details on parameters functions take.

Before running this notebook, you should launch QuPath, start the Py4J gateway, and open an image that has at least one annotation.

In [1]:

from qubalab.qupath import qupath_gateway

token = None   # change the value of this variable if you provided a token while creating the QuPath gateway
port = 25333   # change the value of this variable if you provided a different port while creating the QuPath gateway
gateway = qupath_gateway.create_gateway(auth_token=token, port=port)

print("Gateway created")

Gateway created

Get objects from QuPath¶

We will first see how to get objects from QuPath. As mentionned in the communicating_with_qupath.ipynb notebook, there are two ways to communicate with QuPath:

Use one of the functions of qubalab.qupath.qupath_gateway.
If no function exists for your use case, use gateway.entry_point.

In our case, the qubalab.qupath.qupath_gateway file has a get_objects() function that suits our goal, so we will use it.

This function has an object_type parameter to define which type of object to retrieve. We will work with annotations here:

In [2]:

from qubalab.objects.object_type import ObjectType

object_type = ObjectType.ANNOTATION    # could be DETECTION, TILE, CELL, TMA_CORE

Get annotations as `JavaObject`¶

In [3]:

annotations = qupath_gateway.get_objects(object_type = object_type)

for annotation in annotations:
    print(annotation)

Hello from Python (Region*)
Annotation

By default, the returned objects are Java objects.

This is useful if we want to do something that isn't supported by the qubalab.qupath.qupath_gateway module, but it isn't very convenient... we end up needing to write Python code that looks a lot like Java code. We can also get stuck when things get complicated (e.g. due to threading issues) because we don't have the ability to do everything Java can do.

We can make changes though, like setting names and classifications, which is nice.

If we do, we should remember to call qupath_gateway.refresh_qupath() to update the interface accordingly.

In [4]:

annotation = annotations[0]

# Change the QuPath annotation
annotation.setName("Hello from Python")

# Refresh the QuPath interface. You should see the changes in QuPath
qupath_gateway.refresh_qupath()

Get annotations as `GeoJSON`¶

There's another approach we can take. Rather than directly accessing the QuPath objects, we can request them as GeoJSON. This does not give direct access, but rather imports a more Python-friendly representation that is no longer connected to QuPath.

In [5]:

annotations = qupath_gateway.get_objects(object_type = object_type, converter='geojson')

print(type(annotations[0]))

<class 'qubalab.objects.image_feature.ImageFeature'>

In practice, it's really a slightly 'enhanced' GeoJSON representation, called ImageFeature, because it includes a few extra fields that are relevant for QuPath.

This includes any classification, name, color and object type. It also includes a plane, which stores z and t indices.

But because it is still basically GeoJSON, we can use it with other Python libraries that supports GeoJSON... such as geojson.

We can also use it with Shapely, which is particularly useful. Shapely gives us access to lots of useful methods - and shapely objects can be displayed nicely in a Jupyter notebook.

In [6]:

from shapely.geometry import shape

shape(annotations[0].geometry)

Out[6]:

No description has been provided for this image

Add and remove objects from QuPath¶

The GeoJSON representation doesn't give us direct access to the QuPath objects, but we can still make changes and send them back.

The easiest way to see this in action is to begin by deleting the annotations and then adding them back again - but this time with a different color.

We only assign colors to annotations that aren't classified, so that we don't override the colors that QuPath uses for classification.

In [7]:

import random

for annotation in annotations:
    if not annotation.classification:
        annotation.color = (random.randint(0, 255), random.randint(0, 255), random.randint(0, 255))

qupath_gateway.delete_objects(object_type = object_type)
qupath_gateway.add_objects(annotations)

Create masks & labeled images¶

One reason to use Python rather than QuPath/Java is that NumPy/SciPy/scikit-image and other tools make working with pixels easier and more fun.

To begin, let's use the GeoJSON representation to create masks and labeled images.

To create masks and labeled images, Qubalab has a LabeledImageServer class. This class is an implementation of the qubalab ImageServer which is described in the opening_images.ipynb notebook, so it is recommended that you go through this notebook first. In short, ImageServer is a class to access metadata and pixel values of images.

This server needs:

Some metadata representing the image containing the objects. Since we are working with the image that is opened in QuPath, we can read the metadata of the QuPathServer, as described in communicating_with_qupath.ipynb.
The objects to represent. We will give the annotations we've been working with.
A downsample to apply to the image.

Once the server is created, all functions described in opening_images.ipynb (such as read_region() to read the image) are also available.

In [8]:

from qubalab.images.qupath_server import QuPathServer
from qubalab.images.labeled_server import LabeledImageServer

# Create a QuPathServer that represents the image currently opened in QuPath
qupath_server = QuPathServer(gateway)

# Set a downsample. The labeled image will be 20 times smaller than the image currently opened in QuPath
downsample = 20

# Create the LabeledImageServer. This doesn't create labeled image yet
labeled_server = LabeledImageServer(qupath_server.metadata, annotations, downsample=downsample)

# Request the pixel values of the entire labeled image. Pixel values will be created as they are requested 
label_image = labeled_server.read_region()

# label_image is an image of shape (c, y, x), not easily plottable
# We use a utility function from qubalab to plot the image
from qubalab.display.plot import plotImage
import matplotlib.pyplot as plt
_, ax = plt.subplots()
plotImage(ax, label_image)

Out[8]:

<Axes: >

By default, the LabeledImageServer will return a single channel image where all objects are represented by integer values (labels). It's a labeled image.

Another option is to create a multi channel image where each channel is a mask indicating if an annotation is present

In [9]:

# Create another LabeledImageServer, which will represent a stack of masks here
mask_server = LabeledImageServer(qupath_server.metadata, annotations, downsample=downsample, multichannel=True)

# Compute and return the masks
masks = mask_server.read_region()

# masks contains (n+1) channels, where n is the number of annotations
# The i channel corresponds to the mask representing the i annotation
# Let's plot the first mask corresponding to the first annotation
_, ax = plt.subplots()
plotImage(ax, masks, channel=1)

Out[9]:

<Axes: >

Image processing & creating objects¶

This whole thing becomes more useful when we start to use Python for image processing.

Here we'll use scikit-image to help find objects using two different thresholding methods. We'll then convert them to QuPath objects and add them to the current QuPath viewer for visualization.

We will:

Get pixels of the image opened in QuPath. We can use the qupath_server variable created before and use the read_region() function with a downsample.
Convert the previous image to greyscale, and apply a gaussian filter to it.
For each threshold method:
- Apply the threshold, and create a mask from it.
- Create annotations from the created mask. This uses the qubalab ImageFeature class, which represents a GeoJSON object. We actually used this class before: the qupath_gateway.get_objects(converter='geojson') function returned a list of ImageFeature.
- Add the annotations to QuPath.
- Plot the mask.

In [10]:

import numpy as np
import matplotlib.pyplot as plt
from skimage.filters import gaussian, threshold_otsu, threshold_triangle
from skimage.color import rgb2gray
from qubalab.objects.image_feature import ImageFeature


# Set a downsample. The labeled image will be 20 times smaller than the image currently opened in QuPath
downsample = 20

# Set different threshold methods to apply to the image
threshold_methods = {
    'Otsu' : threshold_otsu,
    'Triangle' : threshold_triangle
}

# Read image opened in QuPath
image = qupath_server.read_region(downsample=downsample)

# Convert the image to greyscale
if qupath_server.metadata.is_rgb:
    # If the image is RGB, we convert it to grayscale
    # read_region() returns an image with the (c, y, x) shape.
    # To use rgb2gray, we need to move the channel axis so that
    # the shape becomes (y, x, c)
    image = np.moveaxis(image, 0, -1)
    image = rgb2gray(image)
else:
    # Else, we only consider the first channel of the image
    image = image[0, ...]

# Apply a gaussian filter
image = gaussian(image, 2.0)

# Iterate over threshold methods
for i, (name, method) in enumerate(threshold_methods.items()):
    # Apply threshold to image
    threshold = method(image)

    # Create mask from threshold
    mask = image < threshold

    # Create annotations from mask
    annotations = ImageFeature.create_from_label_image(
        mask,   
        scale=downsample,   # mask is 20 times smaller than the QuPath image, so we scale
                            # the annotations to fit the QuPath image
        classification_names=name,  # set a single classification to the detected annotations
    )

    # Add annotations to QuPath
    qupath_gateway.add_objects(annotations)

    # Plot mask
    plt.subplot(1, len(threshold_methods), i+1)
    plt.imshow(mask)
    plt.title(f'{name} (threshold={threshold:.2f})')
    plt.axis(False)
plt.show()

Here, we used ImageServer.readRegion() to get pixels as a numpy array. It is also possible to use ImageServer.level_to_dask() or ImageServer.to_dask (as explained in the opening_images.ipynb notebook), which return Dask arrays. Using Dask, we can get access to the entire image as a single, NumPy-like array, at any pyramid level - even if it's bigger than our RAM could handle.

Let's delete these annotations as we won't use them anymore:

In [11]:

qupath_gateway.delete_objects()

Displaying objects¶

We don't need QuPath to visualize GeoJSON features.

QuBaLab also includes functionality for generating matplotlib plots that look a lot like QuPath plots... but that don't use QuPath.

Before running the next notebook cell, you should draw a few small annotations in QuPath and detect cells within them.

The plotting code will show the annotations and cells - randomly recoloring the cells, to demonstrate that they are distinct from QuPath's rendering.

In [12]:

from qubalab.display.plot import plotImageFeatures

# Get QuPath annotations and detections
annotations = qupath_gateway.get_objects(object_type = ObjectType.ANNOTATION, converter='geojson')
detections = qupath_gateway.get_objects(object_type = ObjectType.DETECTION, converter='geojson')

if len(detections) == 0:
    print("No detections found. Please run cell detection from QuPath before running this cell.")

# Set a random color for each detection
for detection in detections:
    detection.color = [random.randint(0, 255) for _ in range(3)]

# Plot every annotations and their detections
fig = plt.figure(figsize=(10, 8))
for i, annotation in enumerate(annotations):
    ax = fig.add_subplot(len(annotations), 1, i+1)

    # Invert y axis. This is needed because in QuPath, the y-axis is going down
    ax.invert_yaxis()

    # Set title of graph from annotation name. You won't see it if the annotation doesn't have a name
    ax.set_title(annotation.name)

    # Plot annotation
    plotImageFeatures(ax, [annotation], region=annotation.geometry, fill_opacity=0.1)

    # Plot detections that are located below the annotation
    plotImageFeatures(ax, detections, region=annotation.geometry, fill_opacity=0.25)

If you combine that with ImageServer.readRegion(), you can plot both the image and the objects:

In [13]:

from shapely.geometry import shape

# An offset in pixels to also see the image around the annotations
offset = 100

# Plot every annotations, their detections, and the image below it
fig = plt.figure(figsize=(10, 8))
for i, annotation in enumerate(annotations):
    ax = fig.add_subplot(len(annotations), 1, i+1)

    # Set title of graph from annotation name. You won't see it if the annotation doesn't have a name
    ax.set_title(annotation.name)

    # Compute bounds of the image. We add a small offset to see the image around the annotation
    bounds = shape(annotation.geometry).bounds
    min_x = max(0, bounds[0] - offset)
    min_y = max(0, bounds[1] - offset)
    max_x = min(qupath_server.metadata.width, bounds[2] + offset)
    max_y = min(qupath_server.metadata.height, bounds[3] + offset)

    # Get pixel values of image
    image = qupath_server.read_region(x=min_x, y=min_y, width=max_x-min_x, height=max_y-min_y)

    # Plot annotation
    plotImageFeatures(ax, [annotation], region=annotation.geometry, fill_opacity=0.1)

    # Plot detections that are located below the annotation
    plotImageFeatures(ax, detections, region=annotation.geometry, fill_opacity=0.25)

    # Plot image
    plotImage(ax, image, offset=[min_x, min_y, max_x, max_y])