[subnet] improve detection of platform-side inference errors

[camfairchild]

We should do a better job at detecting when the problem is failing due to inference errors on the platform-side.
We can use a heuristic of number of non-halting inference-errors

Should be compatible with early-stopping on error

[statxc]

Hi, @camfairchild Can I work on this? I'm interested in this project, I'd like to contribute.
thank you

[camfairchild]

Hi, @camfairchild Can I work on this? I'm interested in this project, I'd like to contribute.
thank you

Have at it!