Conversation
- Introduced a suite of scripts for comprehensive monitoring of Azure DevOps organization health, including: - `agent-pool-capacity.sh`: Analyzes agent pool capacity and utilization, identifying bottlenecks and reporting issues. - `cross-project-dependencies.sh`: Analyzes cross-project dependencies and shared resource usage, identifying potential issues. - `license-utilization.sh`: Evaluates license usage across the organization, checking for capacity issues and optimization opportunities. - `organization-policies.sh`: Reviews organization-level security policies and compliance settings, identifying configuration issues. - `organization-service-health.sh`: Checks Azure DevOps service health status and verifies organization accessibility. - `platform-issue-investigation.sh`: Performs deep investigations of platform-wide issues, correlating problems across services. - Added a `meta.yaml` file for configuration and documentation of the new code bundle. - Enhanced error handling and reporting across all scripts to improve visibility of issues and next steps for remediation. - Updated documentation to reflect new functionalities and usage guidelines for the monitoring scripts.
- Updated various runbooks to include additional tags for improved categorization, specifically adding 'data:logs-config' and 'data:logs-bulk' to relevant tasks. - Refined the handling of next steps in scripts to ensure clarity and consistency in guidance provided for issues detected. - Improved documentation within the runbooks to reflect changes in task functionalities and enhance user understanding of monitoring processes. - Enhanced error reporting and logging mechanisms across scripts to provide better visibility into health checks and issues encountered.
- Added support for Azure DevOps Personal Access Token (PAT) authentication across multiple scripts, ensuring that the necessary environment variable `AZURE_DEVOPS_PAT` is set before attempting to log in. - Updated runbooks to include the `AZURE_DEVOPS_PAT` variable in suite initialization, improving the handling of authentication methods. - Enhanced error handling to provide clear feedback when the PAT is not set, ensuring better user experience and troubleshooting capabilities.
- Implemented Azure DevOps Personal Access Token (PAT) authentication across multiple scripts, ensuring the `AZURE_DEVOPS_PAT` variable is set for secure access. - Updated runbooks to include the `AZURE_DEVOPS_PAT` variable in suite initialization, improving authentication handling. - Enhanced error handling to provide clear feedback when the PAT is not set, facilitating better user experience and troubleshooting.
- Revised task names in the runbook to emphasize cross-project checks within the Azure DevOps organization, enhancing clarity for users. - Updated expected and actual messages in health checks to reflect the new focus on agent pools and pipelines across projects. - Improved documentation and tagging for various tasks to ensure better categorization and understanding of monitoring processes.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
codebundles/azure-devops-organization-health/agent-pool-capacity.sh
Outdated
Show resolved
Hide resolved
codebundles/azure-devops-organization-health/cross-project-dependencies.sh
Outdated
Show resolved
Hide resolved
codebundles/azure-devops-organization-health/service-incident-check.sh
Outdated
Show resolved
Hide resolved
codebundles/azure-devops-organization-health/service-incident-check.sh
Outdated
Show resolved
Hide resolved
codebundles/azure-devops-organization-health/organization-policies.sh
Outdated
Show resolved
Hide resolved
codebundles/azure-devops-organization-health/organization-policies.sh
Outdated
Show resolved
Hide resolved
… and Clarity - Adjusted severity levels in health checks for agent pools and cross-project dependencies to better reflect the impact of issues detected. - Updated user and project count retrieval methods to ensure accurate reporting of Azure DevOps organization data. - Enhanced output messages for investigations and service incident checks to provide clearer status updates and guidance for users. - Renamed output file for service incident checks to improve clarity on its contents.
codebundles/azure-devops-organization-health/.test/validate-all-tests.sh
Show resolved
Hide resolved
codebundles/azure-devops-organization-health/.test/terraform/scripts/setup-dependencies.sh
Outdated
Show resolved
Hide resolved
codebundles/azure-devops-organization-health/cross-project-dependencies.sh
Show resolved
Hide resolved
codebundles/azure-devops-organization-health/license-utilization.sh
Outdated
Show resolved
Hide resolved
codebundles/azure-devops-organization-health/.test/Taskfile.yaml
Outdated
Show resolved
Hide resolved
…verity Handling - Updated severity levels in agent pool capacity and license utilization checks to better reflect the impact of detected issues. - Improved logic for determining severity based on multiple conditions, ensuring more accurate reporting of health issues. - Enhanced output messages for clarity and consistency across various health monitoring scripts.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 4 potential issues.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable autofix in the Cursor dashboard.
| "organization": "$ORG_URL", | ||
| "setup_date": "$(date -u +%Y-%m-%dT%H:%M:%SZ)", | ||
| "projects": [$(printf '"%s",' "${PROJECTS[@]}" | sed 's/,$//')]", | ||
| "variable_groups": [$(printf '"%s",' "${VARIABLE_GROUPS[@]}" | sed 's/,$//')]", |
There was a problem hiding this comment.
Generated scripts produce invalid JSON with stray quote
Medium Severity
The generated test scripts produce invalid JSON because the array-closing syntax ')]" has a stray double-quote character after the ]. This renders the output as "projects": ["a","b"]", instead of the valid "projects": ["a","b"],. The corresponding Terraform template source files (scripts/setup-dependencies.sh, scripts/validate-security.sh, scripts/run-validation-tests.sh) are correct — they use ')], without the extra quote — so this appears to be a mismatch between the committed generated output and what Terraform would actually produce.
Additional Locations (2)
| "azuredevops_project" | ||
| "azuredevops_agent_pool" | ||
| "azuredevops_user_entitlement" | ||
| ) |
There was a problem hiding this comment.
Validation checks for removed Terraform resource type
Low Severity
The validate_infrastructure() function lists azuredevops_user_entitlement as a required Terraform resource, but main.tf never creates this resource type. The PERMISSIONS.md explicitly states the infrastructure "NO LONGER creates" user entitlements. This validation check will always fail, incrementing FAILED_TESTS and making the overall validation report inaccurate.
| projects_using_pool=$((projects_using_pool + 1)) | ||
| fi | ||
| fi | ||
| done |
There was a problem hiding this comment.
Shared pool detection checks any pipelines, not pool
Medium Severity
The "shared agent pool" detection logic doesn't actually check whether each project uses the specific pool being analyzed. It merely checks whether the project has any pipelines via az pipelines list. This means every self-hosted pool is reported as "shared across N projects" where N is just the number of projects that have at least one pipeline — regardless of which pool those pipelines actually target. This produces false positives and could trigger the severity-3 "Excessive Shared Agent Pools" issue incorrectly.
| - task: test-license-scenarios | ||
| - task: test-security-scenarios | ||
| - task: test-service-scenarios | ||
| - task: validate-results |
There was a problem hiding this comment.
Taskfile references five undefined subtask names
Low Severity
The test-all-scenarios task references five subtasks — test-agent-scenarios, test-license-scenarios, test-security-scenarios, test-service-scenarios, and validate-results — but none of these are defined anywhere in the Taskfile, and there are no includes: directives. Running task test-all-scenarios will immediately fail with a "task does not exist" error.
1 participant
Note
Medium Risk
Mostly additive, but it introduces Terraform and scripts that create/manage Azure/Azure DevOps resources and rely on sensitive credentials/PATs, so misconfiguration could impact real orgs if run outside isolated test environments.
Overview
Introduces a new
azure-devops-organization-healthcodebundle for organization-level Azure DevOps health monitoring, including new RunWhenGenerationRulesand templates to emit anslx,sli, andrunbook(taskset) wired torunbook.robotand configurable thresholds.Adds implementation and packaging assets: new bash check scripts (agent pool capacity, license utilization, cross-project dependency analysis), a
meta.yamland a detailedREADME.mddocumenting monitoring scope, tagging, and auth expectations.Adds a sizable
.testharness to stand up and validate scenarios via Terraform (projects, agent pools, service connections, pipelines, variable groups) plus helper scripts/Taskfile and permission docs for repeatable end-to-end testing and cleanup.Written by Cursor Bugbot for commit f232223. This will update automatically on new commits. Configure here.