support non-KVM, non-privileged runners (currently works with ARM emulators)

[eighthave]

Its great to see Google officially support running the emulator in Docker containers!

The big, public CI services like Travis, GitLab-CI, etc. do not provide KVM or privileged runners by default. That means people have to run their own runners, which is a big chunk of work. The ARM emulators work in Docker without KVM or privileged. Yes, they are very slow, but they work.

[741g]

Got it, thanks. But is it necessary to have it be an ARM emulator? One can run the x86 emulators without KVM via the command line flag -no-accel.

[eighthave]

x86 would be great, I've never had any luck with -no-accel, that's why I suggested ARM, because it works now.

[741g]

Interesting; what problems were you having with -no-accel?

[eighthave]

The emulator didn't start. I only ever tried in gitlab-ci, and maybe in Travis, so I didn't really have a way to debug it.

[eighthave]

Here are two example runs with android-29, x86_64 and -no-accel, both fail to start:

• emulator: https://gitlab.com/eighthave/ci-images-client/-/jobs/324970532
• emulator-headless: https://gitlab.com/eighthave/ci-images-client/-/jobs/325001681

Here is one example run with android-22, armeabi, without -no-accel:

• emulator: https://gitlab.com/eighthave/ci-images-client/-/jobs/325012819

[pokowaka]

Not sure what is happening with the first one. The android-22 avd you are trying to boot with arm is missing the kernel-ranchu kernel. We cannot boot this image with the latest emulator.

[eighthave]

The android-22 armeabi emulator runs with a pinned emulator version, something like 28.0.23, I don't know exactly off the top of my head. That one works fine in gitlab-ci, that's why I included it here.

[eighthave]

Here's a simple example using emulator v29.2.1 without Docker in a user account with no access to /dev/kvm:

$ echo no | /opt/android-sdk/tools/bin/avdmanager -v create avd --name avd28 --tag default --package "system-images;android-28;default;x86_64"
Auto-selecting single ABI x86_64========] 100% Fetch remote repository...       
Do you wish to create a custom hardware profile? [no]
$ /opt/android-sdk/emulator/emulator-headless -avd avd28 -no-accel
statvfs('/home/android-sdk/.android/avd/avd28.avd/snapshots/default_boot/ram.img') failed: No such file or directory
emulator: WARNING: x86_64 emulation may not work without hardware acceleration!
Segmentation fault

[eighthave]

Ok, one more point of clarity: emulator vs. emulator-headless doesn't stop the crashing. Adding -no-window makes it work:

$ /opt/android-sdk/emulator/emulator-headless -avd avd28 -no-accel
emulator: WARNING: x86_64 emulation may not work without hardware acceleration!
Segmentation fault
$ /opt/android-sdk/emulator/emulator -avd avd28 -no-accel
statvfs('/home/android-sdk/.android/avd/avd28.avd/snapshots/default_boot/ram.img') failed: No such file or directory
emulator: WARNING: x86_64 emulation may not work without hardware acceleration!
Warning: could not connect to display  ((null):0, (null))
emulator: INFO: QtLogger.cpp:66: Warning: could not connect to display  ((null):0, (null))


emulator: INFO: QtLogger.cpp:66: Info: Could not load the Qt platform plugin "xcb" in "/opt/android-sdk-linux/emulator/lib64/qt/plugins" even though it was found. ((null):0, (null))


Fatal: This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: xcb.
 ((null):0, (null))
emulator: INFO: QtLogger.cpp:66: Fatal: This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: xcb.
 ((null):0, (null))


Aborted
$ /opt/android-sdk/emulator/emulator -avd avd28 -no-accel -no-window &
statvfs('/home/android-sdk/.android/avd/avd28.avd/snapshots/default_boot/ram.img') failed: No such file or directory
emulator: WARNING: x86_64 emulation may not work without hardware acceleration!
pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
$ adb devices 
List of devices attached
emulator-5556	device
$ adb -e shell
generic_x86_64:/ $ 

[741g]

Thanks for the find there; this seems to be a bug in the headless build. Will look into it.

[eighthave]

Ok, so this is not quite there. I've confirmed I can get emulator-headless -no-accel -no-window is running on my machine. But in GitLab-CI's limited Docker environment, the emulator still fails to start. Here's the test log:

https://gitlab.com/eighthave/ci-images-client/-/jobs/326821281

The "Waiting for emulator to start" lines from the wait-for-emulator script, which polls using adb -e shell getprop init.svc.bootanim. You can see that the emulator starts booting, since there is the message "Waiting for emulator to start: device offline". But then after a little while, the emulator seems to vanish, and the polling turns to "Waiting for emulator to start: error: no emulators found", which comes from adb -e shell.

The debug when it fails looks like this:

Waiting for emulator to start: device offline
emulator: onGuestSendCommand: [0x109dd200] Adb connected, start proxing data
emulator: Error while connecting to socket '127.0.0.1:1970': 111 -> Connection refused
emulator: ASC 127.0.0.1:1970: Retrying connection. Connector FD = 25
emulator: (metrics::reportConditional) ignoring
emulator: MemoryReport: uptime: 63766, Res/ResMax/Virt/VirtMax: 1446064128 1446064128 3606990848 3606990848
emulator: Error while connecting to socket '127.0.0.1:1970': 111 -> Connection refused
emulator: ASC 127.0.0.1:1970: Retrying connection. Connector FD = 25
emulator: _camera_service_connect: Connecting camera client 'Factory'
emulator: _factory_client_recv Camera factory query 'list'
emulator: Error while connecting to socket '127.0.0.1:1970': 111 -> Connection refused
emulator: ASC 127.0.0.1:1970: Retrying connection. Connector FD = 25
emulator: MemoryReport: uptime: 68766, Res/ResMax/Virt/VirtMax: 1498120192 1498120192 3606990848 3606990848
emulator: Error while connecting to socket '127.0.0.1:1970': 111 -> Connection refused
emulator: ASC 127.0.0.1:1970: Retrying connection. Connector FD = 25
Waiting for emulator to start: error: no emulators found

When the emulator is started in a GitLab-CI runner that supports KVM, this is that segment of the debug log:
https://gitlab.com/eighthave/fdroidclient/-/jobs/320356215

Waiting for emulator to start: device offline
emulator: MemoryReport: uptime: 30024, Res/ResMax/Virt/VirtMax: 129703936 129703936 1518415872 1518415872
emulator: Error while connecting to socket '127.0.0.1:1970': 111 -> Connection refused
emulator: ASC 127.0.0.1:1970: Retrying connection. Connector FD = 18
emulator: MemoryReport: uptime: 31024, Res/ResMax/Virt/VirtMax: 141033472 141033472 1566810112 1566810112
VERBOSE: AdbHostServer.cpp:49: Send [0012host:emulator:5555] to adb daemon.
emulator: onGuestSendCommand: [0x4ee5ea0] Adb connected, start proxing data
VERBOSE: AdbHostServer.cpp:49: Send [0012host:emulator:5555] to adb daemon.
emulator: MemoryReport: uptime: 32024, Res/ResMax/Virt/VirtMax: 213372928 213372928 1607671808 1607671808
emulator: _camera_service_connect: Connecting camera client 'Factory'
emulator: _factory_client_recv Camera factory query 'list'
<< ATE0Q0V1
>> OK
<< AT+CTEC?
handleTech. cmd: +CTEC?
>> +CTEC: 0,f<CR>OK
<< AT+CTEC=?
>> +CTEC: 0,1,2,3<CR>OK
<< ATE0Q0V1
>> OK
<< ATS0=0
>> OK
<< AT+CMEE=1
>> OK
<< AT+CREG=2
>> OK
<< AT+CGREG=1
>> OK
<< AT+CCWA=1
>> OK
<< AT+CMOD=0
>> OK
<< AT+CMUT=0
>> OK
<< AT+CSSN=0,1
>> OK
<< AT+COLP=0
>> OK
<< AT+CSCS="HEX"
>> OK
<< AT+CUSD=1
>> OK
<< AT+CGEREP=1,0
>> OK
<< AT+CMGF=0
>> %CTZV: 19/10/14:13:32:40+0:0:Etc!UTC<CR><LF><CR>OK
<< AT+CFUN?
>> +CFUN: 0<CR>OK
emulator: MemoryReport: uptime: 33024, Res/ResMax/Virt/VirtMax: 506355712 506355712 2117935104 2117939200
emulator: Error while connecting to socket '127.0.0.1:1970': 111 -> Connection refused
emulator: ASC 127.0.0.1:1970: Retrying connection. Connector FD = 18
emulator: MemoryReport: uptime: 34024, Res/ResMax/Virt/VirtMax: 571088896 571088896 2130780160 2130780160
emulator: MemoryReport: uptime: 35024, Res/ResMax/Virt/VirtMax: 615424000 615424000 2130780160 2130780160
Waiting for emulator to start: booting

Based on the logs, it seems related to the camera.

[eighthave]

And I can reproduce this crash on my machine in an account that has KVM access, with Nexus_5X_API_29 created via Android Studio's GUI:

$ /opt/android-sdk/emulator/emulator-headless @Nexus_5X_API_29 -no-accel -no-window -verbose -debug all -camera-front emulated -camera-back emulated
[snip]
emulator: onGuestSendCommand: [0x1270f400] Adb connected, start proxing data
start pairing up wear to phone
successfully opened up connection to port 5037
sending query '0017host:transport:542347d2' to adb
received mesg from adb host:OKAY
sending query '001dshell:getprop ro.product.name' to adb
received mesg from adb host:OKAYFP2

successfully opened up connection to port 5037
sending query '0017host:transport:542347d2' to adb
received mesg from adb host:OKAY
sending query '0032shell:pm list packages com.google.android.wearable' to adb
emulator: MemoryReport: uptime: 40374, Res/ResMax/Virt/VirtMax: 1391083520 1391083520 3381440512 3381440512
received mesg from adb host:OKAY
successfully opened up connection to port 5037
sending query '001chost:transport:emulator-5554' to adb
received mesg from adb host:OKAY
sending query '001dshell:getprop ro.product.name' to adb
emulator: Error while connecting to socket '127.0.0.1:1970': 111 -> Connection refused
emulator: ASC 127.0.0.1:1970: Retrying connection. Connector FD = 27
emulator: _camera_service_connect: Connecting camera client 'Factory'
emulator: qemud_service_connect_client: registered client channel -1 for 'camera' service
emulator: _qemudPipe_sendBuffers: list\x00
emulator: _factory_client_recv Camera factory query 'list'
emulator: _qemud_pipe_send: len=  8 '00000005'
emulator: _qemud_pipe_send: len=  3 'ok:'
emulator: _qemud_pipe_send: len=  2 '\n\x00'
emulator: _qemudPipe_recvBuffers: -> 8 (of 8)
emulator: _qemudPipe_recvBuffers: -> 5 (of 5)
emulator: MemoryReport: uptime: 45374, Res/ResMax/Virt/VirtMax: 1457995776 1457995776 3381440512 3381440512
emulator: Error while connecting to socket '127.0.0.1:1970': 111 -> Connection refused
emulator: ASC 127.0.0.1:1970: Retrying connection. Connector FD = 27
received mesg from adb host:OKAYsdk_phone_x86_64

successfully opened up connection to port 5037
sending query '001chost:transport:emulator-5554' to adb
received mesg from adb host:OKAY
sending query '0032shell:pm list packages com.google.android.wearable' to adb
emulator: Error while connecting to socket '127.0.0.1:1970': 111 -> Connection refused
emulator: ASC 127.0.0.1:1970: Retrying connection. Connector FD = 27
emulator: MemoryReport: uptime: 50374, Res/ResMax/Virt/VirtMax: 1521217536 1521217536 3381440512 3381440512
emulator: Error while connecting to socket '127.0.0.1:1970': 111 -> Connection refused
emulator: ASC 127.0.0.1:1970: Retrying connection. Connector FD = 27
emulator: Error while connecting to socket '127.0.0.1:1970': 111 -> Connection refused
emulator: ASC 127.0.0.1:1970: Retrying connection. Connector FD = 27
Segmentation fault

Removing -camera-front emulated -camera-back emulated seems to make it crash in a different place (this machine has a webcam):

$ /opt/android-sdk/emulator/emulator-headless @Nexus_5X_API_29 -no-accel -no-window -verbose -debug all 
[snip]
sending query '0017host:transport:542347d2' to adb
received mesg from adb host:OKAY
sending query '0032shell:pm list packages com.google.android.wearable' to adb
emulator: MemoryReport: uptime: 45378, Res/ResMax/Virt/VirtMax: 1430560768 1430560768 3367821312 3367821312
emulator: Error while connecting to socket '127.0.0.1:1970': 111 -> Connection refused
emulator: ASC 127.0.0.1:1970: Retrying connection. Connector FD = 27
received mesg from adb host:OKAY
successfully opened up connection to port 5037
sending query '001chost:transport:emulator-5554' to adb
received mesg from adb host:OKAY
sending query '001dshell:getprop ro.product.name' to adb
emulator: Error while connecting to socket '127.0.0.1:1970': 111 -> Connection refused
emulator: ASC 127.0.0.1:1970: Retrying connection. Connector FD = 27
emulator: MemoryReport: uptime: 50378, Res/ResMax/Virt/VirtMax: 1499430912 1499430912 3367821312 3367821312
emulator: Error while connecting to socket '127.0.0.1:1970': 111 -> Connection refused
emulator: ASC 127.0.0.1:1970: Retrying connection. Connector FD = 27
received mesg from adb host:OKAYsdk_phone_x86_64

successfully opened up connection to port 5037
sending query '001chost:transport:emulator-5554' to adb
received mesg from adb host:OKAY
sending query '0032shell:pm list packages com.google.android.wearable' to adb
emulator: Error while connecting to socket '127.0.0.1:1970': 111 -> Connection refused
emulator: ASC 127.0.0.1:1970: Retrying connection. Connector FD = 27
emulator: MemoryReport: uptime: 55378, Res/ResMax/Virt/VirtMax: 1571356672 1571356672 3369922560 3369922560
emulator: _hwFingerprint_connect: connect finger print listen is called

emulator: qemud_service_connect_client: registered client channel -1 for 'fingerprintlisten' service
emulator: _qemudPipe_sendBuffers: 0006
emulator: _qemudPipe_sendBuffers: listen
emulator: got message from guest system fingerprint HAL

Segmentation fault