Abstract
Medical image segmentation is often ambiguous: unclear bound-
aries, image artifacts, and genuine inter-observer disagreement
mean that no single mask is always the only valid target. Determin-
istic segmentation models collapse this ambiguity into one output,
discarding clinically meaningful variability. We present Ambigu-
ousTextDiff, a text-guided diffusion framework that generates a
diverse set of plausible segmentation masks per image, conditioned
jointly on the image and natural-language descriptors assembled
from structured clinical metadata (e.g., nodule texture and spic-
ulation, lesion site, and pathology metadata). For each image, a
single case-level text description is held fixed while repeated sto-
chastic diffusion sampling generates multiple plausible masks, so
the ensemble can represent the range of segmentations different
radiologists may provide for the same case. The model applies
diffusion only to the mask, fuses the image via channel-wise con-
catenation, and fuses text via cross-attention injected at multiple
U-Net resolutions; a latent KL term, adapted from the Probabilistic
U-Net, regularizes the model toward the distribution of annota-
tor masks rather than a single sampled target. We evaluate on the
complete, unfiltered LIDC-IDRI lung-CT test split (3,072 images)
and on the IMA++ skin-dermoscopy benchmark (2,394 images),
reporting both ambiguity-aware metrics (Generalized Energy Dis-
tance, Collective Insight) and standard overlap metrics. On full
LIDC-IDRI, AmbiguousTextDiff improves GED from 0.306 to 0.152
and CI from 0.470 to 0.835 over the CIMD diffusion baseline; on
IMA++, it improves GED from 0.139 to 0.072 and CI from 0.796
to 0.885 over the strongest baseline. The framework targets set-
tings where structured descriptors are already available before or
alongside pixel-level delineation (e.g., report-conditioned contour-
ing, retrospective cohort curation, or multidisciplinary planning),
and we report controlled ablations, a same-prompt TextDiff base-
line, a descriptor ablation, paired-bootstrap confidence intervals, a
regularization-weight sweep, and a DDPM/DDIM inference-cost
comparison.