DeltaNet

简洁版

训练测试代码主入口：

main_basic.py: basic model

main_conditional.py: 使用一个condition的model

main_multi_conditional.py: 使用多个condition的model

所有代码运行所需参数已经在config文件中设置好

运行命令只需指定配置文件和所需使用的gpu id

示例： python main<model>.py --cfg config/iu_con_1.yml --gpu 0

详细版

Requirements

• Python >= 3.6
• Pytorch >= 1.7
• torchvison
• Microsoft COCO Caption Evaluation Tools
• CheXpert

Data

Download IU and MIMIC-CXR datasets, and place them in data folder.

• IU dataset from here
• MIMIC-CXR dataset from here

Folder Structure

• config : setup training arguments and data path
• data : store IU and MIMIC dataset
• Models: basic model and all our models
  • the layer define of our model
  • loss function
  • metrics
  • some utils
• data_loader.py: dataloader
• build_vocab.py: tokenizer
• preprocessing: data preprocess
• pycocoevalcap: Microsoft COCO Caption Evaluation Tools

Data preparation

1. data

   • Download and unzip IU dataset to here

2. preprocess_data.py

   • Create dataset split for IU dataset.
   • Preprocess report for normalizion, such as lower-case, replacing nonsense tokens(e.g., 'xxxx-a-xxxx').
   • Build vocabulary file.

3. extract_feature.py

   • Extract visual feature for retrieve conditional image.

4. retieve_conditional_pair.py

   • Retrieve conditional image using cosine similarity.

Training

• dataset: iu / mimic

1. main_<model>.py

   • The model will be trained using command python main_<model>.py --cfg config/<dataset_N> --expe_name <experiment name> --gpu <GPU_ID>
   • More options can be found in config/opts.py file.

2. Models

   • Basic.py: Basic Model
   • Conditional.py: DeltaNet condition generation model
   • MultiConditional.py: DeltaNet multiple conditional generation model
   • Generator.py and Beam.py: Generate report using beam search
   • Modules.py: Implement of scaled dot-product attention
   • SubLayers.py: Implement of multi-head attention
   • misc.py and utils.py: Utils function

Testing

test.py: Generate report from trained model using command python test.py --pretrained <path to the checkpoint file>

Evaluate

1. metrics.py

   • Evaluate the generated report.

2. pycocoevalcap

   • Microsoft COCO Caption Evaluation Tools
   • Need clone from GitHub and modified the code to work with Python 3

Cite

Xian Wu, Shuxin Yang, Zhaopeng Qiu, Shen Ge, Yangtian Yan, Xingwang Wu, S.Kevin Zhou, Li Xiao. DeltaNet: Conditional Medical Report Generation for COVID-19 Diagnosis. Coling, 2022

If you have any problem with the code, please contact Xian Wu(kevinxwu@tencent.com), Shuxin Yang(aspenstarss@gmail.com) or Li Xiao(andrew.lxiao@gmail.com).

Thanks

Joseph Paul Cohen, Joseph D. Viviano, Paul Bertin, Paul Morrison, Parsa Torabian, Matteo Guarrera, Matthew P Lungren, Akshay Chaudhari, Rupert Brooks, Mohammad Hashir, Hadrien Bertrand. TorchXRayVision: A library of chest X-ray datasets and models. Medical Imaging with Deep Learning. https://github.com/mlmed/torchxrayvision, 2020