About training time

[dle666]

I am reproducing the training process of the Tokenizer. I am using the same code, with data from ImageNet1K totaling 1.28 million samples. On a single A100 GPU with a batch size of 16, it shows that a single epoch takes 260 hours.

This is quite different from what the paper says, that VFMTok requires 1.5 days of training on 16 Nvidia H800 GPUs. Could you please tell me what your setup is?

[dle666]

I processed the ImageNet data I downloaded from Hugging Face according to my own understanding and converted it into a format that can be used in code. Below is my conversion script. Could this be the reason why my training is slower?

def convert_parquet_to_lmdb(parquet_dir, lmdb_path, split):
"""
将指定分类的 parquet 文件（如 train、val、test）转换为符合数据加载格式的 LMDB。

Args:
    parquet_dir (str): parquet 文件所在的目录路径。
    lmdb_path (str): LMDB 文件存储路径。
    split (str): 指定处理的分类（train, val, test）。
"""
# 获取指定分类的 parquet 文件
parquet_files = [os.path.join(parquet_dir, f) for f in os.listdir(parquet_dir) 
                 if f.endswith('.parquet') and split in os.path.basename(f)]

if not parquet_files:
    print(f"No {split} files found in {parquet_dir}.")
    return

print(f"Processing {split} files...")
os.makedirs(lmdb_path, exist_ok=True)

# 创建LMDB环境
env = lmdb.open(lmdb_path, map_size=1099511627776)  # 1TB容量
txn = env.begin(write=True)

total = 0
for parquet_file in parquet_files:
    # 读取parquet文件
    df = pd.read_parquet(parquet_file)

    for idx, row in tqdm(df.iterrows(), total=len(df), desc=f"Processing {parquet_file}"):
        # 从parquet中提取图像字节和标签
        img_bin = row['image']['bytes']
        label = row['label']

        # 构造存储格式：字典包含 'image' 和 'label'
        data = {
            'image': img_bin,
            'label': label
        }

        # 构造键值对 (索引, 数据字典)
        key = f'{total}'.encode('ascii')  # 使用简单的数字键
        value = pickle.dumps(data)
        txn.put(key, value)

        total += 1

        # 每处理10000个样本，提交一次事务
        if total % 10000 == 0:
            txn.commit()
            txn = env.begin(write=True)
            print(f'Processed {total} images')

# 存储数据集大小信息
txn.put('num_samples'.encode('ascii'), str(total).encode('ascii'))
txn.commit()
env.close()

print(f"Finished processing {split} files. Total images: {total}")

[yexiguafuqihao]

LMDB has a warming-up process where to optimize a better working load for tokenizer training. It is suggested to make the training process last for 1 hour before observing the required training time, while also increasing the batch size to 32.

[dle666]

Thank you for your reply. I discovered that it was because I had set num-work to 0. However, when I set it to 8 or 16, I encountered a new error: RuntimeError: DataLoader worker (pid 1470152) is killed by signal: Segmentation fault.

This error occurs at unpredictable steps, sometimes at 4000 and sometimes at 60000.

[yexiguafuqihao]

Just set worker=4.

[dle666]

I tried setting num-work to 4, but I still encountered the same error, and setting it to 4 limits the training speed. Is there a better solution?

[dle666]

In addition, I also want to know whether this problem is caused by lmdb, or by some other reason.

[Pengjie-W]

LMDB has a warming-up process where to optimize a better working load for tokenizer training. It is suggested to make the training process last for 1 hour before observing the required training time, while also increasing the batch size to 32.LMDB 有一个预热过程，旨在优化更好的分词器训练工作负载。建议将培训过程延长 1 小时，然后再遵守规定的训练时间，同时将批次人数增加到 32 人。

The epoch time increases from 2 hours to 30 hours with each passing hour. It seems like the data loading is getting stuck. How can this be fixed?

[yexiguafuqihao]

This is possibly attributed to the server environment issues: insufficient communication bandwidth or slow disk read speeds. A simple solution is to set the batch size to 512, which can also accelerate tokenizer training by fully utilizing the GPU memory.