[BUG] Failed to run mig.sh on MIG dataproc-2.1-ubuntu20

[yinqingh]

Describe the bug
Observed following error while running mig.sh on dataproc-2.1-ubuntu20 with runtime version "2.1.72-ubuntu20" and kernel version "5.15.0-1067-gcp".

 make -f ./scripts/Makefile.modpost
   sed 's/\.ko$/\.o/' /var/lib/dkms/nvidia/495.29.05/build/modules.order | scripts/mod/modpost -m -a  -o /var/lib/dkms/nvidia/495.29.05/build/Module.symvers -e -i Module.symvers   -T -
 ERROR: modpost: GPL-incompatible module nvidia.ko uses GPL-only symbol 'rcu_read_unlock_strict'
 make[2]: *** [scripts/Makefile.modpost:133: /var/lib/dkms/nvidia/495.29.05/build/Module.symvers] Error 1

Tried with some old dataproc runtime versions. It works with runtime version "2.1.40-ubuntu20" and kernel version "5.15.0-1049-gcp".

Steps/Code to reproduce bug

1. Create dataproc cluster using MIG with nvidia-tesla-a100 gpu and runtime version "2.1.72-ubuntu20"
2. ssh to gpu node
3. download mig.sh
4. sudo bash mig.sh

Expected behavior
succeed to run mig.sh

Environment details (please complete the following information)

• Environment location: Dataproc, version 2.1.72-ubuntu20

[pxLi]

thanks for the investigation!

@sameerz This is the reason why mig-on-dataproc-2.1-ubuntu20 has been failing to initialize recently.

[SurajAralihalli]

Hello @yinqingh, I think you're using a different version of /gpu/mig.sh
Can you try with /spark-rapids/mig.sh?

I’ll inform the repository maintainers about this inconsistency.

Edit: Created issue GoogleCloudDataproc/initialization-actions#1259

[yinqingh]

Hi @SurajAralihalli , I tried with spark-rapids/mig.sh but it still failed in installing nvidia driver (535.104.05) with the same error. The dataproc runtime version is "2.1.73-ubuntu20".

 make -f ./scripts/Makefile.modpost
   sed 's/\.ko$/\.o/' /var/lib/dkms/nvidia/535.104.05/build/modules.order | scripts/mod/modpost -m -a  -o /var/lib/dkms/nvidia/535.104.05/build/Module.symvers -e -i Module.symvers   -T -
 ERROR: modpost: GPL-incompatible module nvidia.ko uses GPL-only symbol 'rcu_read_unlock_strict'
 make[2]: *** [scripts/Makefile.modpost:133: /var/lib/dkms/nvidia/535.104.05/build/Module.symvers] Error 1
 make[2]: *** Deleting file '/var/lib/dkms/nvidia/535.104.05/build/Module.symvers'
 make[1]: *** [Makefile:1829: modules] Error 2
 make[1]: Leaving directory '/usr/src/linux-headers-5.15.0-1070-gcp'
 make: *** [Makefile:82: modules] Error 2
DKMSKernelVersion: 5.15.0-1070-gcp
Date: Fri Nov  8 09:07:43 2024
Package: nvidia-dkms-535 535.104.05-0ubuntu1
PackageVersion: 535.104.05-0ubuntu1
SourcePackage: nvidia-graphics-drivers-535
Title: nvidia-dkms-535 535.104.05-0ubuntu1: nvidia kernel module failed to build

[pxLi]

more context at: GoogleCloudDataproc/initialization-actions#1259

[SurajAralihalli]

I've drafted GoogleCloudDataproc/initialization-actions#1269 to fix the MIG scripts. I can see mig is enabled by querying nvidia-smi --query-gpu=mig.mode.current --format=csv,noheader | grep Enabled on the cluster. Can you please use https://github.com/SurajAralihalli/initialization-actions/blob/fix_mig/spark-rapids/mig.sh and confirm if it works for your use case.

[cjac]

Hello @yinqingh - I've been working on the mig script recently. Here's some news:

• Open kernel drivers do not support A100
• Proprietary kernel drivers do not compile on new kernels
• I have been exercising the mig script with H100 devices, and these work the same way on GCP as the A100s as far as my tests are concerned.

So this is to ask: do you need to use A100 GPUs? If I provided documentation on how to use H100 GPUs and got the mig script working with those instead, would that meet your needs?

If you want a preview of the new script, I have a PR open here:
https://github.com/GoogleCloudDataproc/initialization-actions/pull/1284/files#diff-dbc9224bc43a19c64fa878694edbef41698d2bf1133af921d7b2a2e0da417f83

It's a complete re-write. I am generating the script from templates with this revision so that the code shared between the action scripts in the initialization-actions repository stays in sync. You can fetch the current state of the script, which I've recently exercised here:

https://github.com/GoogleCloudDataproc/initialization-actions/raw/883e6a2664c077edab176614a14d774c23009497/spark-rapids/mig.sh

Please keep in touch. I would like to have more contact with the users of these scripts.

[cjac]

I'm about to run some tests to exercise the MIG script on H100s

[yinqingh]

Hi @cjac , thanks for the new script! But we actually need to use A100. We are currently using the mig.sh in this PR in our test jobs and it works well so far.

[cjac]

Okay. A100, aye. Tell me your image? 2.2-rocky9? either of the rocky8 images?

I think it will be easiest on 2.1-rocky8 or 2.0-rocky8 but very briefly before upgrading to 2.2-rocky9 please.

I will use the script in that pr as a guide, then thank you!