[Bug] `tpu-info` can not access metrics

[steveepreston]

Minimal working code is Here. Code follows GoogleCloudPlatform example

Code run completed successfully on TPU VM v3-8, but while call !tpu-info at the end, it shows:

TPU Chips                                    
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━┓
┃ Chip        ┃ Type        ┃ Devices ┃ PID ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━┩
│ /dev/accel0 │ TPU v3 chip │ 2       │ 13  │
│ /dev/accel1 │ TPU v3 chip │ 2       │ 13  │
│ /dev/accel2 │ TPU v3 chip │ 2       │ 13  │
│ /dev/accel3 │ TPU v3 chip │ 2       │ 13  │
└─────────────┴─────────────┴─────────┴─────┘
Libtpu metrics unavailable. Is there a framework using the TPU? See https://github.com/google/cloud-accelerator-diagnostics/tree/main/tpu_info for more information

[steveepreston]

it shows same error on repo example:

import torch
import torch_xla
t = torch.randn((300, 300), device=torch_xla.device())

!tpu-info

Libtpu metrics unavailable.

[jcole75]

Minimal working code is Here. Code follows GoogleCloudPlatform example

Code run completed successfully on TPU VM v3-8, but while call !tpu-info at the end, it shows:

TPU Chips                                    
┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━┓
┃ Chip        ┃ Type        ┃ Devices ┃ PID ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━┩
│ /dev/accel0 │ TPU v3 chip │ 2       │ 13  │
│ /dev/accel1 │ TPU v3 chip │ 2       │ 13  │
│ /dev/accel2 │ TPU v3 chip │ 2       │ 13  │
│ /dev/accel3 │ TPU v3 chip │ 2       │ 13  │
└─────────────┴─────────────┴─────────┴─────┘
Libtpu metrics unavailable. Is there a framework using the TPU? See https://github.com/google/cloud-accelerator-diagnostics/tree/main/tpu_info for more information

Same issue for me on different TPUs (v3 or v4)

[steveepreston]

@will-cromar @SurbhiJainUSC @bvandermoon Please take a look, Thanks

[peregilk]

I am seeing the same issue. Using jax, and creating a v4-8 from --runtime-version tpu-ubuntu2204-base.

I see the comment about "Releases from before 2024 may not be compatible.". However, it is not clear to me what that refers to, and what needs to be updated.