Automated setup script for creating custom role-based access control (RBAC) permission groups in Braintrust. This script creates personas for different team roles and assigns appropriate permissions.
⚠️ DISCLAIMER: This is an example script for demonstration purposes and is not production-ready code. It lacks comprehensive error handling, input validation, retry logic, logging, and security hardening required for production use. Review and modify according to your organization's security policies and requirements before deploying.
- Python 3.x
requestslibrary (pip install requests)- Braintrust API key
- Braintrust organization name
pip install requestsSet the following environment variables:
export BRAINTRUST_API_KEY="your-api-key-here"
export BRAINTRUST_ORG_NAME="your-organization-name"python create_rolespy.pyThe script creates six permission groups with predefined roles:
| Persona | Role | Permissions |
|---|---|---|
| Engineering Managers | Viewer | Read-only access to org/projects, experiments, scores, dashboards, logs, traces; alert subscriptions |
| Domain Experts | Viewer | Read-only with future commenting and dataset contribution capabilities (Phase 2) |
| Software Engineers - App Dev | Engineer | Full control within assigned projects; create/edit prompts, datasets, experiments, scores |
| QA Engineers | Engineer | Project-scoped read/write; create/maintain test datasets; run evaluations |
| AI Solutions Engineering | Owner | Full admin access across orgs/projects; manage roles, datasets, experiments, scorers |
| Observability App Teams | Engineer | Project-scoped admin for their application with full traces/logs, dashboards, alert management |
- Automated Group Creation: Creates permission groups for each persona
- Role Assignment: Assigns base roles (Owner, Engineer, Viewer) to groups via ACLs
- Organization Discovery: Automatically resolves organization ID from name
- Error Handling: Comprehensive error handling with clear status messages
- Caching: Role ID caching for improved performance
The script provides detailed progress information:
- ✓ Success indicators for completed operations
- ✗ Error indicators with detailed messages
- Summary report of created groups
- Next steps guidance
After running the script:
- Assign Users to Groups: Add team members to appropriate permission groups via Braintrust UI
- Configure Project-Level Permissions: Set project-specific access for Engineers and QA groups
- Setup Phase 2 Permissions: Enable commenting and dataset contributions for Domain Experts
- Configure Alerts: Setup alert subscriptions for Engineering Managers
- Implement Custom Scorers: Grant SDK access for AI Solutions Engineering team
- Monitoring & Audit: Regularly review access logs and conduct permission audits
This script uses the Braintrust API v1:
/organization- Organization lookup/role- Role management/group- Permission group creation/acl- Access control list management
Main class for managing RBAC operations.
get_organization_id()- Retrieve organization ID from nameget_role_id(role_name)- Get role ID by name with cachingcreate_custom_role(role_name, permissions, member_permissions)- Create custom rolecreate_permission_group(group_name, description, member_users)- Create permission groupassign_group_to_role(group_id, role_id, object_type, object_id, restrict_object_type)- Assign group to role via ACLsetup_persona(persona_name, description, role_name)- Complete persona setup with group and role assignment
Error: BRAINTRUST_API_KEY environment variable not set
- Ensure the API key environment variable is exported in your shell
Error: Organization 'name' not found
- Verify the organization name is correct
- Check API key has access to the organization
Failed to create role
- Some custom roles may need to be created via Braintrust UI first
- Check API permissions and organization access