Developer Blog

AI / Deep Learning |

Fast AI Assisted Annotation and Transfer Learning Powered by the Clara Train SDK

pasted image 0 28

The growing volume of clinical data in medical imaging slows down identification and analysis of specific features in an image. This reduces the annotation speed at which radiologists and imaging technicians capture, screen, and diagnose patient data.

The demand for artificial intelligence in medical image analysis has drastically grown in the last few years. AI-assisted solutions have emerged to speed up annotation workflows and increase productivity.

NVIDIA announced the AI-Assisted Annotation and Transfer Learning at the Radiological Society of North America (RSNA) 2018, a premier conference which showcases the latest research and technology advances making headlines in radiology. Clara Train SDK contains these tools to speed up AI-assisted workflows for medical imaging.

This post introduces to developers & data scientists how fast annotation works with transfer learning workflow for Medical Imaging. 

Annotation tools in medical imaging applications help radiologists annotate organs and abnormalities. These applications consist of software tools to draw boundaries around the organs of interest, isolating them from the rest of the dataset. These applications typically operate on 2D slices. A radiologist takes a single 2D image slice from a patient’s scan and marks annotation boundaries and corrections as needed. This manual process repeats for the next slice in which the organ of interest is annotated and corrected again. Some applications support 3D tools for annotation with region growing techniques that perform fairly well with certain organs but poorly on other organs. Moreover, both these 2D and 3D tools for annotation do not learn and adapt to the dataset being annotated which becomes crucial for medical imaging applications as the datasets grow.

Since these workflows are completely manual, the annotation process takes time. A normal Computed Tomography (CT) or Magnetic Resonance Imaging (MRI) scan of a patient has hundreds of 2D slices to cover a particular organ in 3D; new generation scanners producing increasingly higher resolution scans.

AI Assisted Annotation

NVIDIA brings its leadership position in modern artificial intelligence and deep learning to help automate the processing and understanding of images generated by medical scanners. The NVIDIA AI-assisted Annotation enables deep learning based applications by providing developers with tools that make it possible to speed up the annotation process, helping radiologists save time, and increase productivity, as figure 1 shows.

Annotation performance increase chart
Figure 1. Annotation speedup for different organs.

Clara Train SDK’s AI Assisted Annotation accelerates the annotation process by enabling application developers to integrate deep learning tools built into the SDK with existing medical imaging viewers , such as MITK, ITK-Snap, 3D Slicer or a custom-built application viewer. This uses a simple API and requires no prior deep learning knowledge.

As a result, radiologists can increase their productivity by analyzing more patient data while still using their existing workflows and familiar tools.

Generating Segmentation with Deep Learning Models

With AI-assisted annotation, radiologists don’t have to manually draw on each 2D slice to annotate an organ or an abnormality. Instead, they simply click a few extreme points on a particular organ of interest in a 3D MRI or CT scan and receive auto-annotated results for all the 2D slices of that particular organ.

When a user sends the extreme points to the SDK, a deep learning model receives this as input and returns the inference results of the segmented organ or abnormality Figure 2 highlights the steps in the process.

Auto-annotated results image
Figure 2. Auto-Annotated results showing liver as an example. Reference to algorithm:

Model Registry for Organs

The SDK incorporates deep learning models to perform AI-assisted annotation on a particular organ or abnormality. NVIDIA provides 13 different organ models for the early access release which have been pre-trained on public datasets by NVIDIA’s data scientists. These models can be used for annotation and can kick start your development effort by enabling faster annotation of datasets for AI algorithms. Additional organ models are under development.

Polygon Editing 

Since deep learning models, by their nature, are sensitive to the data used to train them, annotation accuracy might be lower than originally achieved with the training data.

If in case there are 2D slices with incorrect or partial annotation, 2D smart polygon editing feature helps in correcting these slices. When a user moves a single polygon point on the 2D slice, all other points around a radius will automatically snap to the organ boundaries thereby making it easier and efficient when correcting, shown in figure 3.

Corrections with smart polygon editing image
Figure 3. Corrections made easy with smart polygon editing

The following section describes how to save these corrected slices as new ground-truth and used to retrain the annotation model, thereby consistently increasing annotation accuracy.

Transfer Learning Workflow

Deep learning models are sensitive to the data used to train them, as described earlier. Factors such as varying scanner configurations, age differences of patients, and so on, need to be taken into account. This makes it hard to train the deep learning models on a specific dataset and deploy them on a different dataset.

Using transfer learning is well suited for medical image analysis. Since medical image analysis is a computer vision task, CNNs represent the best performing methods for this. Yet getting a large well-annotated dataset is considerably harder in the medical domain compared to general computer vision domain because of the domain expertise required to annotate the medical images. This makes transfer learning a natural fit for medical image analysis, use pre-trained CNN on larger database and then apply transfer learning to a target domain of medical images with limited availability.

As a solution, Transfer Learning for medical imaging helps users adapt pre-trained models into their datasets. Deep learning models used for annotation can be tuned and improved by re-training these pre-trained models based on new datasets. Figure 4 highlights this process.

TLT Workflow diagram
Figure 4. Workflow with Transfer Learning Toolkit to improve annotation accuracy

Clara Train SDK contains Annotation SDK and Transfer Learning Toolkit and is available in a docker container.

NVIDIA Transfer Learning Toolkit (TLT) provides domain specific pre-trained models for Intelligent Video Analytics. Same underlying technology allows users to adapt Medical Imaging Segmentation DNNs. Clara Train SDK for Medical Imaging contains several models from the public domain, optimized in-house to achieve high accuracy. Medical imaging developers can choose from one of the provided models as a starting point and use a high level python SDK to retrain and adapt these models with custom data.

We provide python wrappers for model retraining and exporting. These models are trained using Tensorflow framework on public datasets. Complexity of adaptation is considerably reduced because of easy to use shell commands, data conversion guidance, and an export wrapper for converting trained models to a TensorRT optimized graph.

A deep learning image segmentation approach is used for fine-grained predictions needed in medical imaging. Image segmentation algorithms partition input image into multiple segments. Images become divided down to the voxel level (volumetric pixel is the 3-D equivalent of a pixel) and each pixel gets assigned a label or is classified. For every pixel in the image, the network is trained to predict the pixel class. This allows the network to both identify several object classes in each image and determine the location of objects. Image segmentation generates a label image the same size as the input in which the pixels are color-coded according to their classes. 

Accurate segmentation is key in various medical applications.

Using Pre-trained Models

The medical segmentation decathlon challenge site provides a reliable dataset starting point for segmentation model development. All segmentation models in the SDK are trained from the identical segmentation training pipeline, with configurations for brain, heart, pancreas and spleen segmentation. Since several models have been trained with images 1x1x1mm resolution, data needs to be converted to 1x1x1mm NIfTI format. The site provides easy-to-use conversion utilities are provided

Models can be integrated into other applications. The toolkit supports applications built using NVIDIA’s Clara developer platform and deployed for inference with the TensorRT Inference Server.

To learn more about how to our integrate AI-assisted annotation into your existing application, visit

Annotate Faster with Clara Train

Get started using Clara Train SDK today. Easily develop, train and adapt your deep learning models. You can begin with the AI Assisted Annotation Client and Reference Plugin by downloading the client software. The docker Container that includes Python SDK and Annotation Server can be downloaded from NGC. Model Applications can also be downloaded from NGC.