firebase-queue stop handling new tasks for 20 minutes once in a while

[slootzky]

Hey there ,
using firebase-queue to process tasks , about 60-200 in a minute.
every tasks takes about 150-200ms to process.
for some reason , we see the service freezing after some time and handling about 1 item / 2 minutes.
after restarting it , it resumes handling tasks quickly until it gets stuck again and so on.

this is how we start the service :

const options = {
    'specId': "default_spec",
    'numWorkers': 10

};

const queueRef = await firebaseQueueRef();
const specs = {
    "default_spec": {
        "start_state": null,
        "in_progress_state": "in_progress",
        "finished_state": null,
        "error_state": "error",
        "timeout": 60000,
        "retries": 10
    }
};

what could be the reason for such behaviour? how can we investigate it?
what would be a good workers number for our scenario? is 10 too much or too little?

Really appreciate your help here as it keeps happening about 5 times a day.
Thanks

[katowulf]

We couldn't really take a guess here without a minimal repro that reproduces the behavior. Debug logging may help: firebase.database.enableLogging(true);

[slootzky]

Thanks , I will try that.
how does the message looks like? is there a way to only log warn and above?

[slootzky]

OK , there's too much noise , hard to understand something that way... do I look for something specific?

[slootzky]

This still happens , we've switched to FB functions but since it's in Beta and doesn't have SLA we need to get back to firebase-queue solution.
I can't reproduce since it's totally random.
I did notice that few minutes before it happens , execution time per task starts to raise (normally task would take ~300-1000 ms , before it freezes tasks are taking up to ~18700 ms)
Any ideas would be appreciated.

[farhankhwaja]

Even I am facing the same issue. When I start a worker the tasks are consumed at the same rate as they are produced, i.e., 200 tasks per 30 seconds. As, slootzky said, the workers consumption decreases and thus, the queue size starts increasing. I have tried all possible things, i even tried using a M4.large instance thinking the network might be a bottle neck, but it wasn't. Even with M4.large instance the same behavior is been seen.

[slootzky]

@farhankhwaja we found out that the only cloud platform this service can run on , is google cloud.
we tried both Azure app service and AWS elastic beanstalk.
now it is running in GCP app engine and we didn't face this issue since.
I think it's somehow an issue with firebase that solves itself because of reduced network latency.

[farhankhwaja]

oh ok @slootzky . I am trying to start deploy my app to app engine, but getting some Error. Anyways, thanks :). Will post here, if my situation improves after deploying the app.

[farhankhwaja]

@slootzky got the app engine to work. And yes, you were right, the queue workers is now working at a consistently blazing pace. some black magic it seems

[dflourusso]

I had the same problem, then i've created an alternative for simple queue implementation https://github.com/dflourusso/firebase-simple-queue