lf_interop_zoom.py: harden LANforge/Zoom API error handling and add endpoint status-change monitoring#301
Merged
Conversation
…misses get_resource_data() silently skipped a user_resource that wasn't found in the /resource/all response, leaving device_names/ports_list/user_list shorter than self.real_sta_list. That mismatch surfaces much later as a pandas length-mismatch crash in generate_report(), far from the actual cause. Flatten the resources list into a dict once and look up matches directly instead of a nested scan, and on a miss, log the offending resource plus the full resources dict, then abort immediately so the test fails at the point of the real problem instead of downstream. Signed-off-by: Narayana-CT <narayana.pinapatruni@candelatech.com>
…ailure get_ports_data(), get_interop_data(), and select_real_devices() each used a self.json_get() response without checking for None first, crashing with an unhelpful KeyError/TypeError deep in dict access if LANforge was unreachable or returned no data. Add a None check with a clear log message and exit(1) at each call site instead. get_access_token() raised a bare Exception on both a failed HTTP request and a non-200 response, with no logging at either point. Wrap the request in try/except, log the account_id and failure reason in both cases, and return None instead of raising so callers can handle the failure explicitly rather than relying on an uncaught exception. get_live_data() and get_final_qos_data() now check for a None token before using it, logging and bailing out early instead of proceeding to call get_participants_qos() with an invalid Bearer token. Signed-off-by: Narayana-CT <narayana.pinapatruni@candelatech.com>
change_port_to_ip() wrapped the /port/{shelf}/{resource}/{port} lookup
and the interface/ip key access in a single try/except, so an
unreachable LANforge manager was indistinguishable from "this just
isn't an ethernet port" - both fell through to the same warning and a
silent fallback to the original upstream_port value. Check the raw
response for None first and abort immediately with a clear error in
that case; keep the existing warn-and-fallback behavior only for the
genuine non-ethernet-port case (response present, but missing the
interface/ip keys).
Signed-off-by: Narayana-CT <narayana.pinapatruni@candelatech.com>
The immediate-exit-on-None guards added around /resource/all, /port/all, /adb and the port-to-IP lookup were too trigger-happy for a real testbed: a manager that's momentarily slow to respond would abort the whole test on the very first empty response. Add json_get_with_retry(), which retries a json_get() call every 5 seconds for up to 40 seconds before giving up, and use it at all five call sites instead of a single-shot check. get_resource_data() also gets a None-response guard for the first time here - it never actually had one. Signed-off-by: Narayana-CT <narayana.pinapatruni@candelatech.com>
…onse json_get_with_retry() only guarantees a non-None response - it says nothing about whether that response actually has the keys these functions assume. get_resource_data(), get_ports_data() and get_interop_data() all did direct bracket access into the parsed JSON (resource_values["eid"], port_data["mac"], interop_data["devices"], etc.) with no protection, so a reshaped or partial LANforge/adb response would crash with a bare KeyError deep in the function instead of a clear diagnostic. Wrap each function's parsing logic in try/except: a KeyError branch logs the specific missing key plus the raw response received (pretty- printed via json.dumps), and a generic Exception branch logs with exc_info=True for anything else unanticipated (e.g. the IndexError possible in get_interop_data() from a malformed device identifier string). Both branches exit(1), consistent with the existing fail-fast behavior elsewhere in this file. Also switched get_interop_data()'s dict lookups from .get() to bracket access so a malformed /adb response is actually caught here instead of silently degrading into per-user "not found" placeholders. Signed-off-by: Narayana-CT <narayana.pinapatruni@candelatech.com>
While a test runs, there was no visibility into individual generic endpoint state transitions - only the aggregate check_gen_cx() used to decide when the test could stop. Add monitor_endpoint_status_changes(), called once per iteration of the main test loop in run(): it polls every created endpoint and, only the first time an endpoint's status actually changes, logs it and appends a (timestamp, endpoint_name, status) row to endpoint_status_changes.csv - repeated polls of an unchanged status are silent. If every created endpoint comes back missing in a given poll (e.g. LANforge briefly unreachable), it retries every 5 seconds for up to 40 seconds before giving up; if not even one endpoint has responded by then, it logs and aborts the test, consistent with the fail-fast pattern used elsewhere in this file. Wire the new CSV into all three report-generation paths (generate_report(), generate_report_from_api(), and generate_report_from_data() via _move_report_files()) so it ends up in the report directory alongside the other per-client CSVs regardless of which path a given run takes. Verified CLI: python3 lf_interop_zoom.py --duration 2 --lanforge_ip "192.168.207.75" --signin_email "demo@gmail.com" --signin_passwd 'Demo123' --participants 1 --audio --video --upstream_port 192.168.200.170 --api_stats_collection Signed-off-by: Narayana-CT <narayana.pinapatruni@candelatech.com>
goyalsaurabh06
approved these changes
Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR hardens lf_interop_zoom.py against unreliable LANforge/Zoom API responses, replacing silent failures and deep, unhelpful crashes with fail-fast checks, retries, and clear diagnostic logging. It also adds a new endpoint status-change monitor for per-iteration visibility into test progress.
Fail fast on missing resource data: get_resource_data() now flattens /resource/all into a lookup dict and aborts immediately with the offending resource and full response logged, instead of silently skipping unmatched entries and causing a downstream pandas length-mismatch crash in generate_report().
Guard all remaining API call sites: get_ports_data(), get_interop_data(), and select_real_devices() now check for None responses and exit(1) with a clear message instead of crashing with a bare KeyError/TypeError. get_access_token() no longer raises on request failure or non-200 status — it logs the account ID and reason and returns None, which get_live_data()/get_final_qos_data() now check before proceeding.
Abort on LANforge port lookup failure: change_port_to_ip() now distinguishes an unreachable LANforge manager (raw None response → abort) from a genuine non-ethernet port (response present, missing interface/ip keys → existing warn-and-fallback behavior.
Retry before giving up: New json_get_with_retry() retries a json_get() call every 5 seconds for up to 40 seconds before failing, used at all five LANforge API call sites, so a momentarily slow manager no longer aborts the whole test on the first empty response.
Guard key access on successful responses: get_resource_data(), get_ports_data(), and get_interop_data() now wrap their parsing logic in try/except — a KeyError branch logs the specific missing key plus the raw response, and a generic Exception branch logs with exc_info=True — since a non-None response doesn't guarantee the expected keys are present.
New: endpoint status-change monitoring: monitor_endpoint_status_changes() runs once per iteration of the main test loop, polling every created endpoint and logging/recording (timestamp, endpoint name, status) to endpoint_status_changes.csv only the first time each endpoint's status changes. If every endpoint comes back missing, it retries for up to 40 seconds before aborting. The new CSV is wired into all three report paths (generate_report(), generate_report_from_api(), generate_report_from_data()/_move_report_files()).
Verified CLI: python3 lf_interop_zoom.py --duration 2 --lanforge_ip "192.168.207.75" --signin_email "demo@gmail.com" --signin_passwd 'Demo123' --participants 1 --audio --video --upstream_port 192.168.200.170 --api_stats_collection