Repository navigation
omero.LockTimeout exception on "omero admin email" #6394
Description
Hi all,
while sending emails to users using the omero admin email command is working fine in general for us, we seem to be having just too many users on our main OMERO instance.
When running e.g.
omero admin email --everyone "OMERO maintenance completed" < mailbody.txtthe process is taking "a bit" longer and at some point (25s) crashes with an omero.LockTimeout exception like this:
Using session for ehrenfeu@localhost:4064. Idle timeout: 2000 min. Current group: IMCF
Traceback (most recent call last):
File "/opt/omero/server/venv/bin/omero", line 8, in <module>
sys.exit(main())
File "/opt/omero/server/venv/lib/python3.10/site-packages/omero/main.py", line 125, in main
rv = omero.cli.argv()
File "/opt/omero/server/venv/lib/python3.10/site-packages/omero/cli.py", line 1771, in argv
cli.invoke(args[1:])
File "/opt/omero/server/venv/lib/python3.10/site-packages/omero/cli.py", line 1208, in invoke
stop = self.onecmd(line, previous_args)
File "/opt/omero/server/venv/lib/python3.10/site-packages/omero/cli.py", line 1285, in onecmd
self.execute(line, previous_args)
File "/opt/omero/server/venv/lib/python3.10/site-packages/omero/cli.py", line 1367, in execute
args.func(args)
File "<string>", line 1629, in email
File "/opt/omero/server/venv/lib/python3.10/site-packages/omero/clients.py", line 966, in submit
return self.waitOnCmd(
File "/opt/omero/server/venv/lib/python3.10/site-packages/omero/clients.py", line 989, in waitOnCmd
callback.loop(loops, ms) # Throw LockTimeout
File "/opt/omero/server/venv/lib/python3.10/site-packages/omero/callbacks.py", line 258, in loop
raise omero.LockTimeout(
omero.LockTimeout: exception ::omero::LockTimeout
{
serverStackTrace = None
serverExceptionClass = None
message = Command unfinished after 25.0 seconds
backOff = 5000
seconds = 25
}
Unfortunately I failed in finding where to configure this timeout.
➡️ Is there a log / database entries that would tell me who received the email?
All OMERO components are on their latest version, running on Ubuntu 22.04.
Cheers,
Niko
Activity
ehrenfeu commentedon Jun 21, 2024
Forgot to mention,we're currently having about 800 active (=enabled) user accounts on this instance...
will-moore commentedon Jun 24, 2024
@ehrenfeu I'm afraid I don't know the answer to your specific questions, but in general we find it preferable to use the webclient admin tool to send e-mails to users as this doesn't suffer these timeout issues.
Looking at the code at https://github.com/ome/omero-py/blob/8d8108de02104c654741d65f0ecbbf11c85e329d/src/omero/plugins/admin.py#L1599 it looks like you can use a
waitargument which is explained better in the help, but I don't know if that is directly related to the LockTimeout you're seeing.will-moore commentedon Jun 24, 2024
Petr tells me he's used the webclient to e-mail 3000 users at a time and guesses that 800 users would take 40 minutes.
ehrenfeu commentedon Jun 24, 2024
Thanks a lot @will-moore, that's already super helpful. 🚀
We prefer using the CLI for sending emails, as this allows us to store the emails in a local folder on the server and re-use them later on (e.g. to announce maintenance windows, storage downtimes, ...) without having to go for lengthy excursions into one or the other email client's search nightmare.
I'll have a look into the
--waitparameter next time and update this issue then.Unfortunately it's nothing I can easily test under real conditions, as it would mean bothering about 800 people each time 😦
Cheers,
Niko
pwalczysko commentedon Jun 24, 2024
We are having the email texts in g.docs. There must be anyway a fresh editing step prior to each sending (?). I do not suggest that our OMERO.web UI workflow is the best, but possibly the best under the circumstances. The UI workflow in our hands never crashed in the middle of sending emails. Afaik, there is no way to ascertain to which users the email was already sent to. The 40 minutes wait is a pretty safe estimate for 800 users.
ehrenfeu commentedon Jun 25, 2024
By g.docs you're referring to Google docs? Or to some database table?
pwalczysko commentedon Jun 25, 2024
Google docs indeed, sorry for the shorthand.
ehrenfeu commentedon Jul 9, 2025
Just to report back on this, using the
--waitflag prevents the command from ⏳ timing out. ✔️