RenalCLIP: A Disease-Centric Vision-Language Foundation Model for Precision Oncology in Kidney Cancer

🎉 RenalCLIP has been published in Nature Communications!

RenalCLIP is a 3D Vision-Language Model (VLM) for computed tomography that leverages a novel, two-stage knowledge-enhancement pre-training strategy to excel at the comprehensive assessment of renal masses. This repository provides the official pre-trained model weights for the image and text encoders used in our study.

For the full implementation, usage examples, and downstream task evaluation scripts, please visit our official GitHub repository.

GitHub: https://github.com/dt-yuhui/RenalCLIP

Paper (Nature Communications): https://doi.org/10.1038/s41467-026-74175-w

Preprint (arXiv): https://arxiv.org/abs/2508.16569

News

• 2026-07-27: The version of record was published in Nature Communications.
• 2026-06-08: The article was published online.
• 2026-06-01: The paper was accepted by Nature Communications.
• 2025-08-22: The preprint was released on arXiv.

Model Description

The RenalCLIP model consists of two main components: a specialized image encoder and a text encoder.

Image Encoder

The image encoder is designed to process 3D kidney CT volumes. Its architecture consists of a 3D ResNet-18 backbone followed by a projection layer to create embeddings suitable for cross-modal alignment.

• Checkpoint: RenalCLIP-image-encoder-model-best-acc.pt

Text Encoder (LLM2Vec)

The text encoder is built upon the Meta-Llama-3-8B-Instruct model and adapted into a powerful medical language expert using the LLM2Vec methodology. We provide two sets of LoRA weights corresponding to the two final stages of its pre-training:

1. MNTP (Masked Next Token Prediction) Stage:

   • LoRA Weights: LLM2Vec/Meta-Llama-3-8B-Instruct-radiology-ext-long/
   • This stage fine-tunes the base model on a large corpus of medical text (MIMIC-CXR) to enhance its general medical domain comprehension.

2. SimCSE (Contrastive Learning) Stage:

   • LoRA Weights: LLM2Vec/Meta-Llama-3-8B-Instruct-radiology-simcse/
   • This stage further refines the text encoder's understanding of kidney cancer-related terminology using our in-house pre-training corpus.

Important: The provided text encoder checkpoints are LoRA weights only. To use them, you must download the base Llama-3-8B-Instruct model and place its files within the same directory as these LoRA weights. You may change base_model_name_or_path in adapter_config.json to load Llama correctly.

How to Use

For detailed instructions on how to load the model weights, prepare data, and run the pre-training, fine-tuning, and inference scripts, please refer to our official GitHub repository:

https://github.com/dt-yuhui/RenalCLIP

Citation

If you find RenalCLIP useful in your research, please cite our paper:

@article{Tao2026RenalCLIP,
  title   = {A disease-centric vision-language foundation model for precision oncology in kidney cancer},
  author  = {Tao, Yuhui and Zhao, Zhongwei and Wang, Zilong and Luo, Xufang and Chen, Feng and Wang, Kang and Wu, Chuanfu and Zhang, Xue and Zhang, Shaoting and Yao, Jiaxi and Jin, Xingwei and Jiang, Xinyang and Yang, Yifan and Li, Dongsheng and Qiu, Lili and Shao, Zhiqiang and Guo, Jianming and Yu, Nengwang and Xiong, Ying and Wang, Shuo},
  journal = {Nature Communications},
  volume  = {17},
  pages   = {7313},
  year    = {2026},
  doi     = {10.1038/s41467-026-74175-w},
  url     = {https://doi.org/10.1038/s41467-026-74175-w}
}