Multi-Target Landmark Detection with Incomplete Images via Reinforcement Learning and Shape Prior

by   Kaiwen Wan, et al.

Medical images are generally acquired with limited field-of-view (FOV), which could lead to incomplete regions of interest (ROI), and thus impose a great challenge on medical image analysis. This is particularly evident for the learning-based multi-target landmark detection, where algorithms could be misleading to learn primarily the variation of background due to the varying FOV, failing the detection of targets. Based on learning a navigation policy, instead of predicting targets directly, reinforcement learning (RL)-based methods have the potential totackle this challenge in an efficient manner. Inspired by this, in this work we propose a multi-agent RL framework for simultaneous multi-target landmark detection. This framework is aimed to learn from incomplete or (and) complete images to form an implicit knowledge of global structure, which is consolidated during the training stage for the detection of targets from either complete or incomplete test images. To further explicitly exploit the global structural information from incomplete images, we propose to embed a shape model into the RL process. With this prior knowledge, the proposed RL model can not only localize dozens of targetssimultaneously, but also work effectively and robustly in the presence of incomplete images. We validated the applicability and efficacy of the proposed method on various multi-target detection tasks with incomplete images from practical clinics, using body dual-energy X-ray absorptiometry (DXA), cardiac MRI and head CT datasets. Results showed that our method could predict whole set of landmarks with incomplete training images up to 80 error 2.29 cm on body DXA), and could detect unseen landmarks in regions with missing image information outside FOV of target images (average distance error 6.84 mm on 3D half-head CT).


page 19

page 20

page 24


Learn Fine-grained Adaptive Loss for Multiple Anatomical Landmark Detection in Medical Images

Automatic and accurate detection of anatomical landmarks is an essential...

Multiple Landmark Detection using Multi-Agent Reinforcement Learning

The detection of anatomical landmarks is a vital step for medical image ...

Communicative Reinforcement Learning Agents for Landmark Detection in Brain Images

Accurate detection of anatomical landmarks is an essential step in sever...

X-ray-transform Invariant Anatomical Landmark Detection for Pelvic Trauma Surgery

X-ray image guidance enables percutaneous alternatives to complex proced...

An Attention-Guided Deep Regression Model for Landmark Detection in Cephalograms

Cephalometric tracing method is usually used in orthodontic diagnosis an...

Automatic View Planning with Multi-scale Deep Reinforcement Learning Agents

We propose a fully automatic method to find standardized view planes in ...

PEANUT: Predicting and Navigating to Unseen Targets

Efficient ObjectGoal navigation (ObjectNav) in novel environments requir...

Please sign up or login with your details

Forgot password? Click here to reset