Flickering test failures w/ Ember 2.18.2 & 3.4.7

[walter]

This is a follow up on comments in #25 starting here.

Background

In preparation to upgrading from Ember 2.18.2 to 3.4.7, I've been refactoring a fairly complex custom ajax service that my app uses to rely on ember-fetch instead of ic-ajax (ember-ajax has its own service implementation that was in conflict, so opted to move all the way up to fetch based solution).

Everything in development mode seemed to be working fine in 2.18.2, but when I ran my full test suite (modern 3.x testing style already migrated), I began to get a number of failures due to tests not waiting for fetch requests to finish as expected by await calls.

What I have observed

In 2.18.2 with ember-fetch 6.3.1, ember-test-helpers 1.1.0, and ember-qunit 4.2.0:

• if I reran the tests individually, most tests would pass as expected
• adding await settled() calls at a point before the test failure would most times alleviate the test failure when running full suite (after doing this several places, I gave up on this before adding tons of these allover the whole suite)
• ember-fetch registerWaiter seems to be working as designed, but is not enough - I added logging to see if the waiter function was being set up and called as well as the decrementing and they were

It was almost like there was leak somewhere...

A coworker wondered if anything would be different if I tried things under Ember 3.4.7. Unfortunately, the problem was still there. However...

What allows the tests to run as expected in Ember 3.4.7, but not 2.18.2

Under Ember 3.4.7 when I changed ember-cli-build.js to the following, the test suite passed!

  let app = new EmberApp(defaults, {
    // tests: false,
    'ember-fetch': {
      preferNative: true // later added logic here to only be true in test env
    },
   ...

However, when I circled back and tried this with 2.18.2, I got different failures, e.g. Promise rejected during "Help with a ongoing cost": Cannot read property 'error' of undefined, that I assume are related to native promises not being supported in the same way as later Ember 3.x releases.

Since my ultimate goal is to be able to upgrade away from 2.18.2, I'm personally unblocked, but this does seem like an issue with the polyfil either in ember-fetch or possibly upstream.

(@viniciussbs hopefully my workaround helps you, too)

[viniciussbs]

@walter thanks for remember me. I'll let my team know about your workaround. I guess we do not face this problem anymore since we've made another workaround for an issue on ember-graphql/ember-apollo-client#192, though - planning to send a PR to fix that soon.

[arthur5005]

I spent a significant amount of time coming to the same conclusion with our test suites when we upgraded to Ember 3.6 with the use of the latest ember-simple-auth.

Would fail 25% of the time on authenticate, and move on to the next test step without finishing the login one.

Our authenticator made use of ember-fetch to make the authentication request.

I narrowed it down to the use of ember-fetch in the authenticator and the call to .json() on the response object.

Fairly often, the call to.json().then would end this promise chain and move on to the next step.
(In this screenshot, the first breakpoint is hit, the second one is missed, but the test suite continues)

I can confirm

'ember-fetch': {
  preferNative: true
}

is a valid work around for tests.

[fpauser]

We have the same problem in our app (ember-3.9.1). The preferNative workaround helps.

[meirish]

We're seeing this as well, along with preferNative: true "fixing" things. Any way I can help debug / fix this?

[xg-wang]

@meirish The first step to help fix is to construct a minimal flaky test repro