runner updates

[tomeichlersmith]

I need to update the self-hosted runner at UMN. The download from pythia8.org was hanging in the most recent build attempts and I couldn't figure out why. After updating all the packages it seems to have worked, but it is still very slow.

I also have gained access to an old ARM chip (a Kria KR260). It only has 4GB of DDR memory, but there is a chance the bare-metal image building on it could be faster than the emulation currently standing at ~50hrs.

[tomeichlersmith]

I've updated all the software, but the pythia8 download is still way longer on the runner than on my computer (now sharing the same base Ubuntu version). The only differences that I can think of are

• Being in the VM
• The server install of ubuntu does not have some certificates that allow the fast path to be taken?

[tomeichlersmith]

It seems to be an issue with the host itself and not the runner inside the VM.

The computer on which I'm hosting the VMs seem to have an order-of-magnitude slower network connection.

Edit: Talking to our IT, there are some internal network speed/connection issues that UMN is seeing right now. I will need to come back and check again when they are resolved.

Personal Computer ping time averages 4ms for umn.edu and 200ms for pythia.org

tom@appa:~$ ping -c 3 umn.edu
PING umn.edu (134.84.223.68) 56(84) bytes of data.
64 bytes from www-v.oit.umn.edu (134.84.223.68): icmp_seq=1 ttl=248 time=2.45 ms
64 bytes from www-v.oit.umn.edu (134.84.223.68): icmp_seq=2 ttl=248 time=5.58 ms
64 bytes from www-v.oit.umn.edu (134.84.223.68): icmp_seq=3 ttl=248 time=3.79 ms

--- umn.edu ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 2.453/3.940/5.579/1.280 ms
tom@appa:~$ ping -c 3 pythia.org
PING pythia.org (130.235.189.138) 56(84) bytes of data.
64 bytes from william.thep.lu.se (130.235.189.138): icmp_seq=1 ttl=45 time=125 ms
64 bytes from william.thep.lu.se (130.235.189.138): icmp_seq=2 ttl=45 time=125 ms
64 bytes from william.thep.lu.se (130.235.189.138): icmp_seq=3 ttl=45 time=208 ms

--- pythia.org ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 124.563/152.347/207.898/39.280 ms

VM Host ping time averages 200ms for umn.edu and 5000ms for pythia.org

bob@spa-cms-docker-build:~$ ping -c 3 umn.edu
PING umn.edu (134.84.223.68) 56(84) bytes of data.
64 bytes from www-v.oit.umn.edu (134.84.223.68): icmp_seq=1 ttl=248 time=247 ms
64 bytes from www-v.oit.umn.edu (134.84.223.68): icmp_seq=2 ttl=248 time=185 ms
64 bytes from www-v.oit.umn.edu (134.84.223.68): icmp_seq=3 ttl=248 time=176 ms

--- umn.edu ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 175.625/202.287/246.679/31.600 ms
bob@spa-cms-docker-build:~$ ping -c 3 pythia.org
PING pythia.org (130.235.189.138) 56(84) bytes of data.
64 bytes from william.thep.lu.se (130.235.189.138): icmp_seq=1 ttl=46 time=4103 ms
64 bytes from william.thep.lu.se (130.235.189.138): icmp_seq=2 ttl=46 time=5708 ms
64 bytes from william.thep.lu.se (130.235.189.138): icmp_seq=3 ttl=46 time=4684 ms

--- pythia.org ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2045ms
rtt min/avg/max/mdev = 4102.559/4831.687/5708.284/663.776 ms, pipe 3
bob@spa-cms-docker-build:~$ ssh runner-1
github@runner-1:~$ ping -c 3 umn.edu
PING umn.edu (134.84.223.68) 56(84) bytes of data.
64 bytes from www-v.oit.umn.edu (134.84.223.68): icmp_seq=1 ttl=247 time=210 ms
64 bytes from www-v.oit.umn.edu (134.84.223.68): icmp_seq=2 ttl=247 time=257 ms
64 bytes from www-v.oit.umn.edu (134.84.223.68): icmp_seq=3 ttl=247 time=325 ms

--- umn.edu ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 209.951/263.953/324.563/47.022 ms
github@runner-1:~$ ping -c 3 pythia.org
PING pythia.org (130.235.189.138) 56(84) bytes of data.
64 bytes from william.thep.lu.se (130.235.189.138): icmp_seq=1 ttl=45 time=4942 ms
64 bytes from william.thep.lu.se (130.235.189.138): icmp_seq=2 ttl=45 time=3885 ms
64 bytes from william.thep.lu.se (130.235.189.138): icmp_seq=3 ttl=45 time=4194 ms

--- pythia.org ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2081ms
rtt min/avg/max/mdev = 3885.420/4340.592/4942.035/443.587 ms, pipe 3

[tomeichlersmith]

I updated the runner which is good and it seems like our IT fixed the internal network slowdown so I'm going to undo the wget debugging stuff I did on main in order to trigger another build.

[tomeichlersmith]

I was able to successfully flash an Ubuntu 24.04 image onto the KR260's SD Card, but it is failing to boot. I recorded the full boot log using the serial port (linked below), it is able to load the Linux kernel and begin the Linux boot but it fails with a bunch of I/O errors (device offline error, Buffer I/O error, and EXT4-fs error) all on the same device. The boot "completes" by Remounting filesystem read-only which, as far as I can tell, makes the system unresponsive.

The SD Card that I flashed does not appear to be the problem. The Raspberry Pi Imager that flashed it also verified that the imaging was successful and poking around manually myself did not reveal any corruption or filesystem issues on it.

Not sure what the next steps are, but its not looking like we'll have an ARM builder anytime soon.

• KR260 Linux Boot
• kria-boot-serial-capture.log