An Empirical Investigation into the Domain Generalization of Nuclei Segmentation and Phenotyping Models in Computational Histopathology

Overview & Motivation

While deep learning models have achieved expert-level performance in computational pathology, their clinical translation is hindered by a critical lack of generalizability. Models trained on data from one source often fail dramatically when applied to unseen data from a different domain. This project frames the model as a scientific instrument to rigorously investigate and quantify this "domain shift" phenomenon. The core objective is not perfect generalization, but to measure its absence and analyze the reasons for failure, providing evidence-based insights for building robust AI tools for pathology.

The Problem: The Domain Gap

The central challenge is domain shift. A model can identify cancer nuclei in one hospital’s dataset but may collapse on images from another hospital or organ type. Subtle variations in tissue preparation, staining, and scanners create this domain gap. Quantifying this gap is the first step toward developing reliable models for clinical use.

Approach

The thesis is structured as a two-phase experiment:

• Baseline validation on a specialist task


• A rigorous generalization test on an unseen dataset

The project uses HoVer-Net/U-Net architectures trained on the CoNSeP dataset and evaluated on MoNuSeg for cross-domain performance. Metrics include Panoptic Quality (PQ) for segmentation and macro F1-score for classification.

Expected Outcomes & Scientific Contribution

• Quantitative Benchmark: Statistically significant drop in performance expected on out-of-domain data.


• Key Contribution: Characterization of the domain shift problem in computational pathology.


• Actionable Insights: Analysis of failure modes to guide future generalization strategies.


• Foundation for Future Work: Provides a basis for domain adaptation research in clinical AI applications.