# Run Agent Source: https://docs.incidentfox.ai/api-reference/agent/run Trigger an investigation via the API ## Endpoint ``` POST /api/v1/agents/run ``` Triggers an investigation using the specified agent. ## Authentication Requires Team Token or Admin Token. ## Request ### Headers | Header | Required | Description | | --------------- | -------- | ------------------- | | `Authorization` | Yes | `Bearer YOUR_TOKEN` | | `Content-Type` | Yes | `application/json` | ### Body Parameters | Parameter | Type | Required | Description | | ------------ | ------- | -------- | ----------------------------------- | | `agent_name` | string | Yes | Agent to invoke | | `message` | string | Yes | Investigation request | | `context` | object | No | Additional context | | `async` | boolean | No | Return immediately (default: false) | ### Available Agents | Agent | Description | | --------------------- | -------------------------------- | | `planner` | Orchestrates full investigations | | `investigation_agent` | Comprehensive troubleshooting | | `k8s_agent` | Kubernetes-focused | | `aws_agent` | AWS-focused | | `coding_agent` | Code analysis | ## Request Example ```bash theme={null} curl -X POST https://api.incidentfox.ai/api/v1/agents/run \ -H "Authorization: Bearer $TEAM_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "agent_name": "planner", "message": "Investigate high latency in the payments service", "context": { "service": "payments", "environment": "production", "severity": "high" } }' ``` ## Response ### Success Response (200) ```json theme={null} { "investigation_id": "inv_abc123", "status": "completed", "duration_seconds": 45, "result": { "summary": "Payment service experiencing elevated latency due to database connection pool exhaustion", "root_cause": { "description": "RDS connection pool at maximum capacity (100/100 connections)", "confidence": 92, "evidence": [ "CloudWatch RDS connections metric at 100%", "Application logs show 'connection timeout' errors", "Spike correlates with deployment at 14:32 UTC" ] }, "timeline": [ { "timestamp": "2024-01-15T14:32:00Z", "event": "New deployment rolled out" }, { "timestamp": "2024-01-15T14:35:00Z", "event": "Connection count started increasing" }, { "timestamp": "2024-01-15T14:42:00Z", "event": "Connection pool exhausted" } ], "affected_systems": [ "payments-service", "checkout-service", "RDS primary" ], "recommendations": [ { "priority": "high", "action": "Increase RDS max_connections parameter" }, { "priority": "medium", "action": "Review connection pool settings in application config" }, { "priority": "low", "action": "Consider connection pooler like PgBouncer" } ] }, "tools_used": [ "get_cloudwatch_logs", "get_cloudwatch_metrics", "get_pod_logs", "search_github_code" ], "created_at": "2024-01-15T14:45:00Z" } ``` ### Async Response (202) When `async: true`: ```json theme={null} { "investigation_id": "inv_abc123", "status": "running", "poll_url": "/api/v1/agents/status/inv_abc123", "created_at": "2024-01-15T14:45:00Z" } ``` ### Error Responses **400 Bad Request** ```json theme={null} { "error": { "code": "invalid_request", "message": "agent_name is required" } } ``` **401 Unauthorized** ```json theme={null} { "error": { "code": "unauthorized", "message": "Invalid or expired token" } } ``` **429 Rate Limited** ```json theme={null} { "error": { "code": "rate_limited", "message": "Too many investigation requests", "retry_after": 60 } } ``` ## Async Investigations For long-running investigations, use async mode: ```bash theme={null} curl -X POST https://api.incidentfox.ai/api/v1/agents/run \ -H "Authorization: Bearer $TEAM_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "agent_name": "planner", "message": "Full system health check", "async": true }' ``` Then poll for results: ```bash theme={null} curl -X GET https://api.incidentfox.ai/api/v1/agents/status/inv_abc123 \ -H "Authorization: Bearer $TEAM_TOKEN" ``` ## Context Parameters Provide additional context to improve investigation accuracy: ```json theme={null} { "context": { "service": "payments", "environment": "production", "namespace": "checkout", "severity": "high", "related_alert": "alert_xyz789", "time_window": "1h", "hint": "Check recent deployments" } } ``` | Context Field | Description | | --------------- | ------------------------ | | `service` | Target service name | | `environment` | prod, staging, dev | | `namespace` | Kubernetes namespace | | `severity` | high, medium, low | | `related_alert` | Alert ID for context | | `time_window` | Investigation time range | | `hint` | Additional guidance | ## Code Examples ### Python ```python theme={null} import requests response = requests.post( "https://api.incidentfox.ai/api/v1/agents/run", headers={ "Authorization": f"Bearer {TEAM_TOKEN}", "Content-Type": "application/json" }, json={ "agent_name": "planner", "message": "Investigate high latency in payments" } ) result = response.json() print(f"Root cause: {result['result']['root_cause']['description']}") ``` ### JavaScript ```javascript theme={null} const response = await fetch('https://api.incidentfox.ai/api/v1/agents/run', { method: 'POST', headers: { 'Authorization': `Bearer ${TEAM_TOKEN}`, 'Content-Type': 'application/json' }, body: JSON.stringify({ agent_name: 'planner', message: 'Investigate high latency in payments' }) }); const result = await response.json(); console.log(`Root cause: ${result.result.root_cause.description}`); ``` ## Next Steps Poll investigation status Get team configuration # Investigation Status Source: https://docs.incidentfox.ai/api-reference/agent/status Check the status of an investigation ## Endpoint ``` GET /api/v1/agents/status/{investigation_id} ``` Get the status and results of an investigation. ## Authentication Requires Team Token or Admin Token. ## Path Parameters | Parameter | Type | Required | Description | | ------------------ | ------ | -------- | ---------------- | | `investigation_id` | string | Yes | Investigation ID | ## Request Example ```bash theme={null} curl -X GET https://api.incidentfox.ai/api/v1/agents/status/inv_abc123 \ -H "Authorization: Bearer $TEAM_TOKEN" ``` ## Response ### Running Investigation ```json theme={null} { "investigation_id": "inv_abc123", "status": "running", "agent": "planner", "message": "Investigate high latency in payments", "progress": { "current_step": "Querying CloudWatch metrics", "steps_completed": 3, "steps_total": 7 }, "started_at": "2024-01-15T14:45:00Z", "elapsed_seconds": 15 } ``` ### Completed Investigation ```json theme={null} { "investigation_id": "inv_abc123", "status": "completed", "agent": "planner", "message": "Investigate high latency in payments", "result": { "summary": "Root cause identified as database connection pool exhaustion", "root_cause": { ... }, "recommendations": [ ... ] }, "tools_used": ["get_cloudwatch_logs", "get_pod_logs"], "started_at": "2024-01-15T14:45:00Z", "completed_at": "2024-01-15T14:45:45Z", "duration_seconds": 45 } ``` ### Failed Investigation ```json theme={null} { "investigation_id": "inv_abc123", "status": "failed", "agent": "planner", "message": "Investigate high latency in payments", "error": { "code": "data_source_unavailable", "message": "Unable to connect to Coralogix" }, "started_at": "2024-01-15T14:45:00Z", "failed_at": "2024-01-15T14:45:30Z" } ``` ## Status Values | Status | Description | | ----------- | ------------------------- | | `pending` | Queued, not yet started | | `running` | Investigation in progress | | `completed` | Successfully completed | | `failed` | Investigation failed | | `cancelled` | Cancelled by user | ## Polling Strategy For async investigations, poll with exponential backoff: ```python theme={null} import time import requests def wait_for_investigation(investigation_id, token, max_wait=300): url = f"https://api.incidentfox.ai/api/v1/agents/status/{investigation_id}" headers = {"Authorization": f"Bearer {token}"} wait_time = 1 total_wait = 0 while total_wait < max_wait: response = requests.get(url, headers=headers) result = response.json() if result["status"] in ["completed", "failed", "cancelled"]: return result time.sleep(wait_time) total_wait += wait_time wait_time = min(wait_time * 2, 10) # Max 10 second intervals raise TimeoutError("Investigation did not complete in time") ``` ## List Recent Investigations ``` GET /api/v1/agents/investigations ``` List recent investigations for your team: ```bash theme={null} curl -X GET "https://api.incidentfox.ai/api/v1/agents/investigations?limit=10" \ -H "Authorization: Bearer $TEAM_TOKEN" ``` Response: ```json theme={null} { "investigations": [ { "investigation_id": "inv_abc123", "status": "completed", "agent": "planner", "message": "Investigate high latency...", "created_at": "2024-01-15T14:45:00Z" }, { "investigation_id": "inv_def456", "status": "completed", "agent": "k8s_agent", "message": "Check pod status...", "created_at": "2024-01-15T13:30:00Z" } ], "pagination": { "total": 42, "limit": 10, "offset": 0 } } ``` ## Cancel Investigation ``` POST /api/v1/agents/status/{investigation_id}/cancel ``` Cancel a running investigation: ```bash theme={null} curl -X POST https://api.incidentfox.ai/api/v1/agents/status/inv_abc123/cancel \ -H "Authorization: Bearer $TEAM_TOKEN" ``` ## Next Steps Trigger new investigation Get team configuration # Authentication Source: https://docs.incidentfox.ai/api-reference/authentication Authenticate with the IncidentFox API ## Overview IncidentFox supports multiple authentication methods: * **Team Tokens** - For programmatic team access * **Admin Tokens** - For organization administration * **OIDC/SSO** - For user authentication via identity provider ## Team Tokens Team tokens provide access scoped to a specific team within an organization. ### Token Format ``` tokid.toksecret ``` * `tokid` - Token identifier (public) * `toksecret` - Token secret (keep secure) ### Usage Include in the `Authorization` header: ```bash theme={null} curl -X GET https://api.incidentfox.ai/api/v1/config/me/effective \ -H "Authorization: Bearer tokid.toksecret" ``` ### Obtaining Tokens Team tokens are issued by your organization admin: 1. Admin logs into Web UI 2. Navigates to **Admin Console** > **Teams** 3. Selects team and clicks **Generate Token** 4. Token is displayed once - save it securely ### Token Permissions Team tokens can: * Read team configuration * Update team configuration * Trigger investigations * View investigation history Team tokens cannot: * Access other teams * Modify organization settings * Create/delete teams ## Admin Tokens Admin tokens provide organization-wide access. ### Permissions Admin tokens can: * Manage all teams * View audit logs * Configure organization settings * Create/revoke team tokens ### Usage ```bash theme={null} curl -X GET https://api.incidentfox.ai/api/v1/admin/teams \ -H "Authorization: Bearer admin.tokensecret" ``` ## OIDC/SSO Authentication For user-based authentication via your identity provider. ### Configuration Configure OIDC in organization settings: ```json theme={null} { "oidc": { "issuer": "https://login.company.com", "client_id": "incidentfox-app", "client_secret": "vault://secrets/oidc-secret" } } ``` ### Supported Providers * Google Workspace * Azure AD * Okta * Auth0 * Generic OIDC ### JWT Token Usage After OIDC authentication, use the JWT: ```bash theme={null} curl -X GET https://api.incidentfox.ai/api/v1/config/me/effective \ -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9..." ``` ## Identifying the Caller Use the `/auth/me` endpoint to identify the authenticated user/team: ```bash theme={null} curl -X GET https://api.incidentfox.ai/api/v1/auth/me \ -H "Authorization: Bearer YOUR_TOKEN" ``` Response: ```json theme={null} { "role": "team", "auth_kind": "team_token", "org_id": "org-acme", "team_node_id": "team-platform", "subject": null, "email": null, "can_write": true, "permissions": ["team:read", "team:write"] } ``` For OIDC users: ```json theme={null} { "role": "user", "auth_kind": "oidc_jwt", "org_id": "org-acme", "team_node_id": null, "subject": "user@company.com", "email": "user@company.com", "can_write": true, "permissions": ["admin:read", "admin:write"] } ``` ## Token Security Treat tokens like passwords. Never commit to version control or share publicly. ### Best Practices 1. **Store securely** - Use secrets managers 2. **Rotate regularly** - Rotate tokens periodically 3. **Use least privilege** - Use team tokens when admin isn't needed 4. **Monitor usage** - Review audit logs for anomalies 5. **Revoke unused** - Revoke tokens when no longer needed ### Token Revocation Admins can revoke tokens: ```bash theme={null} curl -X POST https://api.incidentfox.ai/api/v1/admin/tokens/revoke \ -H "Authorization: Bearer admin.token" \ -H "Content-Type: application/json" \ -d '{"token_id": "tokid"}' ``` ## Error Handling ### Invalid Token ```json theme={null} { "error": { "code": "unauthorized", "message": "Invalid or expired token" } } ``` ### Expired Token ```json theme={null} { "error": { "code": "token_expired", "message": "Token has expired" } } ``` ### Insufficient Permissions ```json theme={null} { "error": { "code": "forbidden", "message": "Token does not have permission for this operation" } } ``` ## Next Steps Trigger investigations Read team configuration # Get Effective Config Source: https://docs.incidentfox.ai/api-reference/config/effective Retrieve your team's effective configuration ## Endpoint ``` GET /api/v1/config/me/effective ``` Returns the fully merged configuration for the authenticated team, including all inherited settings from organization and group levels. ## Authentication Requires Team Token or Admin Token. ## Request Example ```bash theme={null} curl -X GET https://api.incidentfox.ai/api/v1/config/me/effective \ -H "Authorization: Bearer $TEAM_TOKEN" ``` ## Response ### Success Response (200) ```json theme={null} { "team_name": "platform-devops", "tokens_vault_path": { "openai_token": "vault://org/acme/teams/platform-devops/openai", "slack_bot": "vault://org/acme/teams/platform-devops/slack-bot" }, "mcp_servers": ["grafana", "aws", "coralogix"], "a2a_agents": ["investigation", "code_fix"], "slack_group_to_ping": "@platform-oncall", "slack_channel": "#incidents-platform", "knowledge_source": { "grafana": ["dash/123", "dash/456"], "google": ["drive:folder/abc"], "confluence": ["space:PLAT:runbooks"] }, "agents": { "investigation_agent": { "prompt": "You are an SRE investigation agent for Acme Corp...", "enabled": true, "disable_default_tools": [], "enable_extra_tools": ["coralogix", "snowflake"] }, "code_fix_agent": { "prompt": "Provide minimal, safe hotfix suggestions", "enabled": true, "disable_default_tools": ["db-write"], "enable_extra_tools": ["repo-read"] } }, "feature_flags": { "enable_auto_mitigation": false, "enable_write_actions": true, "require_approval": true }, "alerts": { "disabled": ["cpu_throttle_high", "disk_pressure"] }, "tools": { "kubernetes": { "enabled": true, "default_namespace": "production" }, "coralogix": { "enabled": true, "domain": "coralogix.com" } } } ``` ## Understanding Effective Config The effective config is computed by merging configs from: 1. **Organization defaults** (root level) 2. **Group configs** (intermediate levels) 3. **Team config** (leaf level) Later levels override earlier levels via deep merge. ## Get Raw Config with Lineage To understand where settings come from: ``` GET /api/v1/config/me/raw ``` ```bash theme={null} curl -X GET https://api.incidentfox.ai/api/v1/config/me/raw \ -H "Authorization: Bearer $TEAM_TOKEN" ``` Response: ```json theme={null} { "lineage": [ "org-acme", "engineering-group", "platform-team" ], "configs": { "org-acme": { "mcp_servers": ["grafana", "aws"], "feature_flags": { "enable_auto_mitigation": false } }, "engineering-group": { "slack_channel": "#engineering-incidents" }, "platform-team": { "mcp_servers": ["grafana", "aws", "coralogix"], "slack_channel": "#incidents-platform", "agents": { "investigation_agent": { "enable_extra_tools": ["coralogix", "snowflake"] } } } } } ``` ## Get Config Audit History View configuration change history: ``` GET /api/v1/config/me/audit ``` ```bash theme={null} curl -X GET "https://api.incidentfox.ai/api/v1/config/me/audit?limit=10" \ -H "Authorization: Bearer $TEAM_TOKEN" ``` Response: ```json theme={null} { "history": [ { "version": 5, "changed_at": "2024-01-15T10:30:00Z", "changed_by": "user@company.com", "diff": { "mcp_servers": { "old": ["grafana", "aws"], "new": ["grafana", "aws", "coralogix"] } }, "full_config": { ... } }, { "version": 4, "changed_at": "2024-01-10T08:00:00Z", "changed_by": "admin@company.com", "diff": { "agents.investigation_agent.prompt": { "old": "You are an investigation agent...", "new": "You are an SRE investigation agent..." } } } ] } ``` ### Query Parameters | Parameter | Type | Default | Description | | -------------- | ---- | ------- | ---------------------------- | | `limit` | int | 50 | Number of entries | | `include_full` | bool | true | Include full config snapshot | ## Code Examples ### Python ```python theme={null} import requests def get_team_config(token): response = requests.get( "https://api.incidentfox.ai/api/v1/config/me/effective", headers={"Authorization": f"Bearer {token}"} ) response.raise_for_status() return response.json() config = get_team_config(TEAM_TOKEN) print(f"MCP Servers: {config['mcp_servers']}") print(f"Slack Channel: {config['slack_channel']}") ``` ### JavaScript ```javascript theme={null} async function getTeamConfig(token) { const response = await fetch( 'https://api.incidentfox.ai/api/v1/config/me/effective', { headers: { 'Authorization': `Bearer ${token}` } } ); return response.json(); } const config = await getTeamConfig(TEAM_TOKEN); console.log('MCP Servers:', config.mcp_servers); ``` ## Next Steps Update team configuration Trigger investigations # Update Config Source: https://docs.incidentfox.ai/api-reference/config/update Update your team's configuration ## Endpoint ``` PUT /api/v1/config/me ``` Update the team's configuration overrides. Uses PATCH semantics - provided values are deep-merged with existing config. ## Authentication Requires Team Token with write permissions. ## Request ### Headers | Header | Required | Description | | --------------- | -------- | ------------------- | | `Authorization` | Yes | `Bearer YOUR_TOKEN` | | `Content-Type` | Yes | `application/json` | ### Body Partial configuration object. Only include fields you want to change. ## Request Example ```bash theme={null} curl -X PUT https://api.incidentfox.ai/api/v1/config/me \ -H "Authorization: Bearer $TEAM_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "mcp_servers": ["grafana", "aws", "coralogix", "snowflake"], "agents": { "investigation_agent": { "enable_extra_tools": ["snowflake", "custom-runbooks"] } } }' ``` ## Response ### Success Response (200) ```json theme={null} { "message": "Configuration updated successfully", "version": 6, "changed_fields": [ "mcp_servers", "agents.investigation_agent.enable_extra_tools" ], "effective_config": { "mcp_servers": ["grafana", "aws", "coralogix", "snowflake"], "agents": { "investigation_agent": { "prompt": "You are an SRE investigation agent...", "enable_extra_tools": ["snowflake", "custom-runbooks"] } } } } ``` ### Approval Required Response (202) When approval workflows are enabled: ```json theme={null} { "message": "Configuration change submitted for approval", "change_request_id": "cr_xyz789", "status": "pending_approval", "approvers": ["admin@company.com"], "changes": { "mcp_servers": { "old": ["grafana", "aws", "coralogix"], "new": ["grafana", "aws", "coralogix", "snowflake"] } } } ``` ### Error Responses **400 Bad Request - Invalid Field** ```json theme={null} { "error": { "code": "invalid_field", "message": "team_name is immutable and cannot be changed", "field": "team_name" } } ``` **400 Bad Request - Validation Error** ```json theme={null} { "error": { "code": "validation_error", "message": "Invalid configuration value", "details": { "field": "agents.investigation_agent.timeout", "error": "must be a positive integer" } } } ``` **403 Forbidden** ```json theme={null} { "error": { "code": "forbidden", "message": "Token does not have write permissions" } } ``` ## Immutable Fields These fields cannot be changed via API: * `team_name` - Set by admin only * `org_id` - Fixed at team creation * `team_node_id` - Fixed at team creation ## Common Updates ### Add MCP Server ```json theme={null} { "mcp_servers": ["grafana", "aws", "coralogix", "new-server"] } ``` ### Update Agent Prompt ```json theme={null} { "agents": { "investigation_agent": { "prompt": "You are an SRE investigation agent for Acme Corp. Focus on database and payment issues first..." } } } ``` ### Enable Additional Tools ```json theme={null} { "agents": { "investigation_agent": { "enable_extra_tools": ["snowflake", "custom-tool"] } } } ``` ### Disable Dangerous Tools ```json theme={null} { "agents": { "investigation_agent": { "disable_default_tools": ["shell", "docker_exec"] } } } ``` ### Update Slack Settings ```json theme={null} { "slack_channel": "#new-incidents-channel", "slack_group_to_ping": "@new-oncall-group" } ``` ### Enable Feature Flags ```json theme={null} { "feature_flags": { "enable_auto_mitigation": true, "require_approval": false } } ``` ## Code Examples ### Python ```python theme={null} import requests def update_config(token, config_updates): response = requests.put( "https://api.incidentfox.ai/api/v1/config/me", headers={ "Authorization": f"Bearer {token}", "Content-Type": "application/json" }, json=config_updates ) response.raise_for_status() return response.json() # Add a new MCP server result = update_config(TEAM_TOKEN, { "mcp_servers": ["grafana", "aws", "coralogix", "snowflake"] }) print(f"Config version: {result['version']}") ``` ### JavaScript ```javascript theme={null} async function updateConfig(token, updates) { const response = await fetch( 'https://api.incidentfox.ai/api/v1/config/me', { method: 'PUT', headers: { 'Authorization': `Bearer ${token}`, 'Content-Type': 'application/json' }, body: JSON.stringify(updates) } ); return response.json(); } const result = await updateConfig(TEAM_TOKEN, { agents: { investigation_agent: { enable_extra_tools: ['snowflake'] } } }); ``` ## Validation Before saving, the API validates: 1. **Schema compliance** - Fields match expected types 2. **Immutable fields** - Cannot change protected fields 3. **Tool existence** - Enabled tools must exist 4. **MCP server validity** - MCP servers must be configured ## Approval Workflows If your organization has approval workflows enabled: 1. Updates create a pending change request 2. Admins are notified 3. Once approved, config is applied 4. Requester is notified Check pending changes: ``` GET /api/v1/config/me/pending ``` ## Next Steps View current configuration Configuration documentation # API Introduction Source: https://docs.incidentfox.ai/api-reference/introduction Overview of the IncidentFox REST API ## Overview The IncidentFox API provides programmatic access to: * Trigger investigations * Manage team configuration * Query investigation history * Integrate with custom workflows ## Base URL ``` https://api.incidentfox.ai/api/v1 ``` ## Authentication All API endpoints require authentication via Bearer token. ```bash theme={null} curl -X GET https://api.incidentfox.ai/api/v1/config/me/effective \ -H "Authorization: Bearer YOUR_TEAM_TOKEN" ``` ### Token Types | Type | Format | Scope | | ----------- | ------------------- | ----------------------- | | Team Token | `tokid.toksecret` | Team operations | | Admin Token | `admin.tokensecret` | Organization admin | | OIDC JWT | Standard JWT | SSO authenticated users | Team tokens are issued by your organization admin. Contact them if you need API access. ## Response Format All responses are JSON: ```json theme={null} { "data": { ... }, "meta": { "request_id": "req_abc123", "timestamp": "2024-01-15T10:30:00Z" } } ``` ### Error Responses ```json theme={null} { "error": { "code": "unauthorized", "message": "Invalid or expired token", "request_id": "req_abc123" } } ``` ### HTTP Status Codes | Code | Description | | ---- | ------------ | | 200 | Success | | 400 | Bad request | | 401 | Unauthorized | | 403 | Forbidden | | 404 | Not found | | 429 | Rate limited | | 500 | Server error | ## Rate Limiting API requests are rate limited: | Endpoint Type | Limit | | ---------------------- | ---------- | | Read operations | 100/minute | | Write operations | 20/minute | | Investigation triggers | 10/minute | Rate limit headers are included in responses: ``` X-RateLimit-Limit: 100 X-RateLimit-Remaining: 95 X-RateLimit-Reset: 1705318260 ``` ## API Categories Trigger and manage investigations Manage team configuration ## Quick Start ### Trigger an Investigation ```bash theme={null} curl -X POST https://api.incidentfox.ai/api/v1/agents/run \ -H "Authorization: Bearer $TEAM_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "agent_name": "planner", "message": "Investigate high latency in payments service" }' ``` ### Get Team Configuration ```bash theme={null} curl -X GET https://api.incidentfox.ai/api/v1/config/me/effective \ -H "Authorization: Bearer $TEAM_TOKEN" ``` ## SDKs Official SDKs are available: * **Python**: `pip install incidentfox` * **JavaScript**: `npm install @incidentfox/sdk` ### Python Example ```python theme={null} from incidentfox import IncidentFoxClient client = IncidentFoxClient(token="YOUR_TEAM_TOKEN") # Trigger investigation result = client.investigate("High error rate in checkout service") print(result.summary) print(result.root_cause) # Get team config config = client.config.get_effective() print(config.mcp_servers) ``` ## Webhooks Configure webhooks to receive investigation results: ```json theme={null} { "webhooks": { "investigation_complete": "https://your-app.com/webhooks/incidentfox" } } ``` Webhook payload: ```json theme={null} { "event": "investigation_complete", "investigation_id": "inv_abc123", "summary": "Root cause identified", "root_cause": { ... }, "recommendations": [ ... ] } ``` ## Next Steps Detailed auth guide Trigger investigations # Agent Configuration Source: https://docs.incidentfox.ai/configuration/agents Configure agent behavior, prompts, and tool access ## Overview IncidentFox agents can be customized through configuration to: * Modify system prompts * Enable or disable specific tools * Add custom context about your infrastructure * Tune behavior for your team's needs ## Agent Types | Agent | Purpose | Default Tools | | --------------------- | --------------------------- | ----------------------------- | | `planner` | Orchestrate investigations | None (delegates to others) | | `k8s_agent` | Kubernetes troubleshooting | 9 K8s-specific tools | | `aws_agent` | AWS resource debugging | 8 AWS-specific tools | | `metrics_agent` | Anomaly detection | 22 metrics/analytics tools | | `coding_agent` | Code analysis | 15 code/git tools | | `investigation_agent` | Full toolkit investigations | 30+ tools from all categories | ## Configuration Structure Each agent is configured under the `agents` key: ```json theme={null} { "agents": { "investigation_agent": { "prompt": "System prompt for the agent...", "enabled": true, "disable_default_tools": ["shell", "docker_exec"], "enable_extra_tools": ["custom-runbook-search"] }, "code_fix_agent": { "enabled": false } } } ``` ## Configuration Options ### `prompt` The system prompt that defines the agent's behavior, knowledge, and communication style. ```json theme={null} { "agents": { "investigation_agent": { "prompt": "You are an AI SRE agent for Acme Corp. Our infrastructure runs on AWS EKS in us-west-2. Key services include: payments (critical), cart (high), catalog (medium). Always check CloudWatch metrics first, then pod logs. Escalate P1 incidents immediately to #incidents-critical." } } } ``` **Include context about your infrastructure** in the prompt: * Service criticality tiers * Common failure patterns * Escalation procedures * Team-specific runbooks to reference ### `enabled` Toggle an agent on or off. Defaults to `true`. ```json theme={null} { "agents": { "code_fix_agent": { "enabled": false } } } ``` ### `disable_default_tools` Remove specific tools from an agent's default toolkit. Useful for security or compliance. ```json theme={null} { "agents": { "investigation_agent": { "disable_default_tools": [ "shell", "docker_exec", "db_write" ] } } } ``` Disabling critical tools may impact investigation effectiveness. Test thoroughly before disabling in production. ### `enable_extra_tools` Add tools beyond the agent's default set. ```json theme={null} { "agents": { "investigation_agent": { "enable_extra_tools": [ "coralogix", "snowflake", "custom-runbooks" ] } } } ``` ## Writing Effective Prompts ### Structure A well-structured agent prompt includes: 1. **Role definition** - What the agent is and does 2. **Context** - Information about your infrastructure 3. **Guidelines** - How to approach investigations 4. **Constraints** - What to avoid or be careful about 5. **Output format** - How to structure responses ### Example: Investigation Agent ``` You are an AI SRE agent for Acme Corp's platform team. ## Infrastructure Context - Cloud: AWS (us-west-2, us-east-1) - Orchestration: EKS (Kubernetes 1.28) - Key Services: - payments-service (P0 - business critical) - cart-service (P1 - customer facing) - catalog-service (P2 - internal) - analytics-service (P3 - batch processing) ## Observability Stack - Logs: Coralogix (primary), CloudWatch (backup) - Metrics: Grafana Cloud + Prometheus - Traces: Datadog APM - Alerts: PagerDuty -> Slack #incidents ## Investigation Guidelines 1. Always start by identifying affected services and their criticality 2. Check recent deployments (last 4 hours) first 3. Query Coralogix for error logs before CloudWatch 4. For database issues, check RDS Performance Insights 5. Correlate with recent PRs merged to main ## Response Format Always include: - Summary (1-2 sentences) - Root cause with confidence level - Evidence (specific logs, metrics, or events) - Timeline of events - Recommended actions with priority ## Constraints - Never execute remediation without approval - Escalate P0/P1 incidents immediately - Don't access production databases directly ``` ### Example: Slack Bot Agent ``` You are the IncidentFox Slack bot for Acme Corp. ## Communication Style - Be concise and actionable - Use bullet points for multiple items - Include confidence levels when uncertain - Link to dashboards and runbooks when relevant ## Quick Commands When users say: - "check [service]" -> Run health check on service - "logs [service]" -> Fetch recent error logs - "who's oncall" -> Check PagerDuty schedule - "deploy status" -> Check recent deployments ## Escalation For P0/P1, immediately ping @oncall-platform and post to #incidents-critical. ``` ## Agent Specialization ### Creating Workflow-Specific Agents You can create specialized agents for different workflows: **CI/CD Investigation Agent:** ```json theme={null} { "agents": { "ci_investigation_agent": { "prompt": "You specialize in CI/CD failures. Focus on: build logs, test output, dependency changes, environment differences between PR and main.", "enable_extra_tools": ["github_actions", "codepipeline", "ecr"] } } } ``` **Database Investigation Agent:** ```json theme={null} { "agents": { "db_investigation_agent": { "prompt": "You specialize in database performance issues. Check RDS metrics, slow query logs, connection pools, and recent schema changes.", "enable_extra_tools": ["rds_insights", "pg_stat_statements", "snowflake"] } } } ``` ## Tuning Tips ### Improve Root Cause Accuracy 1. **Add service dependencies** to the prompt 2. **Include common failure patterns** you've seen 3. **Specify data source priority** (which to check first) 4. **Add context about recent changes** (migrations, refactors) ### Reduce Investigation Time 1. **Prioritize fast data sources** in the prompt 2. **Include known quick wins** (common issues and solutions) 3. **Set appropriate timeouts** for tool execution ### Improve Response Quality 1. **Define output format** explicitly 2. **Include examples** of good responses 3. **Specify confidence thresholds** for recommendations ## Validation Before deploying prompt changes: 1. **Test in staging** with known scenarios 2. **Compare results** with previous prompt version 3. **Check for regressions** in accuracy or speed If approval workflows are enabled, prompt changes require admin approval before taking effect. ## Next Steps Configure and customize tools Advanced prompt engineering # Configuration Overview Source: https://docs.incidentfox.ai/configuration/overview Understanding IncidentFox configuration hierarchy and options ## Configuration Model IncidentFox uses a hierarchical configuration system that allows organizations to set defaults while enabling teams to customize their specific needs. ### Hierarchy ```mermaid theme={null} graph TD A[Organization root] A --> B[Engineering Group] A --> C[Infrastructure Group] B --> D[Platform Team] B --> E[Backend Team] C --> F[SRE Team] C --> G[DevOps Team] ``` Configuration flows from top to bottom. Each level can: * **Inherit** settings from parent levels * **Override** specific settings * **Add** additional configuration ### How Merging Works When IncidentFox loads a team's configuration, it: 1. Starts with organization defaults 2. Deep-merges each intermediate group's config 3. Deep-merges the team's config 4. Returns the final "effective config" Deep merge means nested objects are merged recursively, not replaced entirely. Arrays are typically replaced, not concatenated. ### Example **Organization Config:** ```json theme={null} { "mcp_servers": ["grafana", "aws"], "agents": { "investigation_agent": { "prompt": "You are an SRE investigation agent for Acme Corp..." } }, "feature_flags": { "enable_auto_mitigation": false } } ``` **Team Config (Platform Team):** ```json theme={null} { "mcp_servers": ["grafana", "aws", "coralogix"], "agents": { "investigation_agent": { "enable_extra_tools": ["snowflake", "custom-runbooks"] } } } ``` **Effective Config for Platform Team:** ```json theme={null} { "mcp_servers": ["grafana", "aws", "coralogix"], "agents": { "investigation_agent": { "prompt": "You are an SRE investigation agent for Acme Corp...", "enable_extra_tools": ["snowflake", "custom-runbooks"] } }, "feature_flags": { "enable_auto_mitigation": false } } ``` ## Configuration Sections ### Core Settings | Field | Type | Description | | --------------------- | ------ | ------------------------------------------- | | `team_name` | string | Display name (immutable at team level) | | `slack_channel` | string | Default Slack channel for notifications | | `slack_group_to_ping` | string | Group to mention (e.g., `@oncall-platform`) | ### Agent Configuration Configure behavior for each agent type: ```json theme={null} { "agents": { "investigation_agent": { "prompt": "Custom system prompt...", "enabled": true, "disable_default_tools": ["shell"], "enable_extra_tools": ["custom-tool"] } } } ``` See [Agent Configuration](/configuration/agents) for details. ### Tool Configuration Enable/disable and configure tools: ```json theme={null} { "tools": { "kubernetes": { "enabled": true, "default_namespace": "production" }, "coralogix": { "enabled": true, "api_key": "vault://secrets/coralogix-api-key" } } } ``` See [Tool Configuration](/configuration/tools) for details. ### MCP Servers Configure Model Context Protocol servers: ```json theme={null} { "mcp_servers": ["grafana", "aws", "custom-server"] } ``` ### Knowledge Sources Configure where agents look for documentation and runbooks: ```json theme={null} { "knowledge_source": { "grafana": ["prod-k8s", "prod-logs"], "google": ["drive:folder/oncall-runbooks"], "confluence": ["space:SRE"] } } ``` ### Feature Flags Control feature behavior: ```json theme={null} { "feature_flags": { "enable_auto_mitigation": false, "enable_write_actions": true, "require_approval": true } } ``` ### Alerts Configuration Customize alert handling: ```json theme={null} { "alerts": { "disabled": ["cpu_throttle_high", "disk_pressure"] } } ``` ## Managing Configuration ### Via Web UI 1. Log in to your IncidentFox dashboard 2. Navigate to **Team Console** > **Configuration** 3. Edit settings in the visual editor 4. Save changes (may require approval if enabled) ### Via API ```bash theme={null} # Get effective config curl -X GET https://api.incidentfox.ai/api/v1/config/me/effective \ -H "Authorization: Bearer $TEAM_TOKEN" # Update team config curl -X PUT https://api.incidentfox.ai/api/v1/config/me \ -H "Authorization: Bearer $TEAM_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "mcp_servers": ["grafana", "aws", "coralogix"] }' ``` ### Viewing Raw Lineage To understand why a setting has a particular value: ```bash theme={null} curl -X GET https://api.incidentfox.ai/api/v1/config/me/raw \ -H "Authorization: Bearer $TEAM_TOKEN" ``` Returns: ```json theme={null} { "lineage": ["org-root", "engineering-group", "platform-team"], "configs": { "org-root": { /* org config */ }, "engineering-group": { /* group config */ }, "platform-team": { /* team config */ } } } ``` ## Audit Trail All configuration changes are logged. View the audit history: ```bash theme={null} curl -X GET https://api.incidentfox.ai/api/v1/config/me/audit \ -H "Authorization: Bearer $TEAM_TOKEN" ``` Returns: ```json theme={null} [ { "version": 5, "changed_at": "2024-01-15T10:30:00Z", "changed_by": "user@company.com", "diff": { "mcp_servers": { "old": ["grafana", "aws"], "new": ["grafana", "aws", "coralogix"] } } } ] ``` ## Best Practices **Start broad, refine narrow** - Set sensible defaults at the org level, then let teams customize as needed. 1. **Use org-level defaults** for common settings 2. **Group by function** - Platform teams vs. application teams may need different configs 3. **Minimize team overrides** - Only override what's truly team-specific 4. **Use vault references** for secrets - Never store credentials in plain text 5. **Enable approval workflows** for production teams ## Next Steps Configure agent behavior and prompts Enable and configure tools # System Prompts Source: https://docs.incidentfox.ai/configuration/prompts Advanced guide to writing effective agent prompts ## Overview System prompts are the primary way to customize agent behavior. A well-crafted prompt can significantly improve investigation accuracy, response quality, and team alignment. ## Prompt Architecture ```mermaid theme={null} graph TD A[System Prompt] --> D[User Message
investigate high latency in payments] D --> E[Agent Response
follows prompt guidelines] B[1. Role & Identity
2. Context & Knowledge
3. Guidelines & Process
4. Constraints & Guardrails
5. Output Format] -.->|Components| A ``` ## Section-by-Section Guide ### 1. Role & Identity Define who the agent is and its primary purpose. ``` You are an AI SRE agent for Acme Corp's Platform Engineering team. Your primary responsibility is investigating production incidents and providing actionable insights to reduce MTTR. ``` **Include:** * Organization name * Team the agent serves * Primary responsibility ### 2. Context & Knowledge Provide infrastructure context the agent needs to know. ``` ## Infrastructure Overview **Cloud**: AWS (primary: us-west-2, DR: us-east-1) **Orchestration**: EKS 1.28 with Karpenter autoscaling **Service Mesh**: Istio 1.20 ## Services | Service | Criticality | Team | Notes | |---------|-------------|------|-------| | payments | P0 | checkout | PCI compliant, separate VPC | | cart | P1 | checkout | Redis for session | | catalog | P2 | inventory | Read-heavy, uses caching | | analytics | P3 | data | Batch, can tolerate delays | ## Data Sources - **Logs**: Coralogix (primary), CloudWatch (backup) - **Metrics**: Grafana Cloud (Prometheus) - **Traces**: Datadog APM - **Alerts**: PagerDuty → Slack - **Enrichment**: Snowflake (historical data) ## Key Dashboards - Production Overview: https://grafana.acme.com/d/prod-overview - Error Rates: https://grafana.acme.com/d/errors - Database Performance: https://grafana.acme.com/d/rds ``` **Include:** * Cloud and infrastructure details * Service catalog with criticality * Data source locations * Important dashboards/runbooks ### 3. Guidelines & Process Define how the agent should approach investigations. ``` ## Investigation Process 1. **Identify scope**: Determine affected services and their criticality 2. **Recent changes first**: Check deployments in the last 4 hours 3. **Follow the data**: - Coralogix for application logs - CloudWatch for infrastructure logs - Grafana for metrics correlation - Snowflake for historical patterns 4. **Correlate**: Look for timing relationships between events 5. **Verify**: Confirm findings with multiple data sources ## Priority Rules - P0 services: Escalate immediately, investigate in parallel - P1 services: Investigate promptly, escalate if not resolved in 15min - P2/P3 services: Normal investigation flow ## Common Patterns 1. **Deployment correlation**: 80% of incidents happen within 4 hours of deploy 2. **Database issues**: Check connection pools before blaming the DB 3. **Network issues**: Verify Istio sidecar health first 4. **Memory issues**: Look for memory leaks in pod restarts ``` **Include:** * Step-by-step investigation process * Priority/escalation rules * Common patterns you've observed * Preferred data source order ### 4. Constraints & Guardrails Define what the agent should NOT do. ``` ## Constraints - **Never** execute remediation without explicit approval - **Never** access production databases directly - **Never** share PII or sensitive data in responses - **Do not** restart services without oncall confirmation - **Limit** CloudWatch queries to 24 hours to control costs ## Escalation Rules Escalate immediately (do not investigate alone) when: - Multiple P0 services affected - Data integrity concerns (payments, user data) - Security-related symptoms - Customer-facing impact confirmed ## Sensitive Data These fields are PII and should never be logged or displayed: - user_email, customer_id, payment_token, ssn, credit_card ``` **Include:** * Explicit prohibitions * Escalation triggers * Security/compliance requirements * Cost control measures ### 5. Output Format Define how responses should be structured. ``` ## Response Format Always structure responses as: ### Summary [1-2 sentence overview of findings] ### Root Cause - **Description**: [What went wrong] - **Confidence**: [Low/Medium/High with percentage] - **Evidence**: [Bulleted list of supporting data] ### Timeline [Chronological list of relevant events] ### Affected Systems [List of impacted services/components] ### Recommendations [Numbered list of suggested actions, in priority order] ### Next Steps [Immediate actions needed] ## Confidence Levels - **High (80-100%)**: Multiple data sources confirm, clear causation - **Medium (50-79%)**: Strong correlation, some ambiguity - **Low (<50%)**: Limited data, hypothesis only ``` **Include:** * Response structure * Required sections * Confidence level definitions * Example formats ## Complete Example ``` You are an AI SRE agent for Acme Corp's Platform Engineering team. ## Infrastructure **Cloud**: AWS (us-west-2) **Orchestration**: EKS 1.28 **Services**: payments (P0), cart (P1), catalog (P2), analytics (P3) ## Data Sources - Logs: Coralogix (primary) - Metrics: Grafana Cloud - Traces: Datadog - Enrichment: Snowflake ## Investigation Process 1. Identify affected services and criticality 2. Check recent deployments (last 4 hours) 3. Query Coralogix for error patterns 4. Check Grafana for metric anomalies 5. Correlate with GitHub for recent changes 6. Use Snowflake for historical context ## Constraints - Never execute remediation without approval - Escalate P0 incidents immediately to #incidents-critical - Do not access production databases directly ## Response Format ### Summary [1-2 sentences] ### Root Cause - Description: [what] - Confidence: [%] - Evidence: [list] ### Timeline [events] ### Recommendations [actions] ``` ## Testing Prompts Before deploying a new prompt: Run investigations for incidents you've already resolved Check if the new prompt produces better/worse results Ensure guardrails are respected Get feedback from SREs who will use it ## Prompt Templates ### Investigation Agent (Generic) ``` You are an AI SRE agent for [COMPANY]. ## Infrastructure [Add your infrastructure details] ## Data Sources [Add your observability stack] ## Investigation Process [Add your preferred investigation steps] ## Constraints [Add your guardrails] ## Response Format [Add your preferred output structure] ``` ### CI/CD Agent ``` You are an AI agent specializing in CI/CD failures for [COMPANY]. ## CI/CD Stack - CI: [GitHub Actions/Jenkins/etc] - CD: [CodePipeline/ArgoCD/etc] - Registry: [ECR/Docker Hub/etc] ## Investigation Focus 1. Build failures: Check logs, dependencies, environment 2. Test failures: Analyze test output, compare with main 3. Deploy failures: Check permissions, resources, health checks ## Common Issues [Add patterns you've seen] ``` ## Next Steps Apply prompts to agents See available tools to reference in prompts # Tool Configuration Source: https://docs.incidentfox.ai/configuration/tools Enable, configure, and customize IncidentFox tools ## Overview IncidentFox provides 50+ built-in tools across multiple categories. Each tool can be: * **Enabled/disabled** per team * **Configured** with credentials and settings * **Customized** with team-specific defaults ## Tool Categories | Category | Tools | Description | | ----------------- | ----- | --------------------------------------------- | | Kubernetes | 9 | Pod logs, deployments, events, resource usage | | AWS | 8 | EC2, Lambda, RDS, ECS, CloudWatch | | Anomaly Detection | 8 | Prophet forecasting, Z-score, correlation | | Grafana | 6 | Dashboards, Prometheus queries, alerts | | Datadog | 3 | Metrics, logs, APM | | New Relic | 2 | NRQL queries, APM summary | | Coralogix | 4 | Log search, alerts, metrics | | Snowflake | 3 | SQL queries, data enrichment | | GitHub | 16 | Code search, PRs, issues, workflows | | Git | 12 | Status, diff, log, blame | | Docker | 15 | Build, run, logs, exec | | Elasticsearch | 3 | Log search, aggregations | | Slack | 5 | Messages, channels, threads | ## Configuration Structure Tools are configured under the `tools` key: ```json theme={null} { "tools": { "kubernetes": { "enabled": true, "default_namespace": "production", "kubeconfig_path": "/path/to/kubeconfig" }, "coralogix": { "enabled": true, "api_key": "vault://secrets/coralogix-api-key", "domain": "coralogix.com" }, "snowflake": { "enabled": true, "account": "acme.us-west-2", "warehouse": "COMPUTE_WH", "credentials": "vault://secrets/snowflake" } } } ``` ## Common Configuration Options ### All Tools | Option | Type | Description | | --------- | ------- | ----------------------------- | | `enabled` | boolean | Enable/disable the tool | | `timeout` | integer | Max execution time in seconds | ### Credentials Always use vault references for secrets: ```json theme={null} { "api_key": "vault://path/to/secret", "password": "vault://path/to/password" } ``` Never store credentials in plain text. Always use vault references. ## Tool-Specific Configuration ### Kubernetes ```json theme={null} { "tools": { "kubernetes": { "enabled": true, "kubeconfig_path": "~/.kube/config", "default_namespace": "production", "default_context": "prod-cluster", "timeout": 30 } } } ``` | Option | Default | Description | | ------------------- | ---------------- | ----------------------------- | | `kubeconfig_path` | `~/.kube/config` | Path to kubeconfig file | | `default_namespace` | `default` | Default namespace for queries | | `default_context` | Current context | K8s context to use | | `timeout` | 30 | Command timeout in seconds | ### AWS ```json theme={null} { "tools": { "aws": { "enabled": true, "region": "us-west-2", "profile": "production", "assume_role": "arn:aws:iam::123456789:role/incidentfox" } } } ``` | Option | Default | Description | | ------------- | --------- | ---------------------- | | `region` | From env | AWS region | | `profile` | `default` | AWS profile name | | `assume_role` | None | IAM role ARN to assume | ### Coralogix ```json theme={null} { "tools": { "coralogix": { "enabled": true, "api_key": "vault://secrets/coralogix-api-key", "domain": "coralogix.com", "default_application": "production", "default_subsystem": "backend" } } } ``` | Option | Required | Description | | --------------------- | -------- | ------------------------------------------------------------- | | `api_key` | Yes | Coralogix API key | | `domain` | Yes | Coralogix domain (e.g., `coralogix.com`, `eu2.coralogix.com`) | | `default_application` | No | Default application filter | | `default_subsystem` | No | Default subsystem filter | ### Snowflake ```json theme={null} { "tools": { "snowflake": { "enabled": true, "account": "acme.us-west-2", "username": "vault://secrets/snowflake-user", "password": "vault://secrets/snowflake-pass", "warehouse": "COMPUTE_WH", "database": "ANALYTICS", "schema": "PUBLIC", "role": "ANALYST" } } } ``` | Option | Required | Description | | ----------- | -------- | ------------------------------ | | `account` | Yes | Snowflake account identifier | | `username` | Yes | Username for authentication | | `password` | Yes | Password (use vault reference) | | `warehouse` | Yes | Default warehouse | | `database` | No | Default database | | `schema` | No | Default schema | | `role` | No | Snowflake role to use | ### Datadog ```json theme={null} { "tools": { "datadog": { "enabled": true, "api_key": "vault://secrets/datadog-api-key", "app_key": "vault://secrets/datadog-app-key", "site": "datadoghq.com" } } } ``` | Option | Required | Description | | --------- | -------- | --------------------------------------- | | `api_key` | Yes | Datadog API key | | `app_key` | Yes | Datadog Application key | | `site` | No | Datadog site (default: `datadoghq.com`) | ### Grafana ```json theme={null} { "tools": { "grafana": { "enabled": true, "url": "https://grafana.company.com", "api_key": "vault://secrets/grafana-api-key", "default_datasource": "Prometheus" } } } ``` | Option | Required | Description | | -------------------- | -------- | ------------------------ | | `url` | Yes | Grafana instance URL | | `api_key` | Yes | Grafana API key | | `default_datasource` | No | Default data source name | ### GitHub ```json theme={null} { "tools": { "github": { "enabled": true, "token": "vault://secrets/github-token", "default_org": "acme-corp", "default_repo": "main-app" } } } ``` | Option | Required | Description | | -------------- | -------- | ---------------------------- | | `token` | Yes | GitHub Personal Access Token | | `default_org` | No | Default organization | | `default_repo` | No | Default repository | ## Disabling Dangerous Tools For security, you may want to disable certain tools: ```json theme={null} { "agents": { "investigation_agent": { "disable_default_tools": [ "shell", "docker_exec", "db_write", "remediation_actions" ] } } } ``` Consider which tools are appropriate for each environment. Production may need stricter controls than staging. ## Tool Loading Priority When an agent needs a tool, the system checks: 1. **Is the integration installed?** (package availability) 2. **Are credentials configured?** (tool config + vault) 3. **Is it enabled for this team?** (team config) 4. **Is it allowed for this agent?** (agent config) All conditions must be met for the tool to be available. ## Monitoring Tool Usage View tool usage metrics in the Web UI under **Team Console** > **Agent Runs**. Each investigation shows: * Which tools were invoked * Execution time per tool * Success/failure status * Tool output (redacted as needed) ## Next Steps Detailed setup for each data source Add custom tools via MCP # AWS Source: https://docs.incidentfox.ai/data-sources/aws Connect IncidentFox to AWS for CloudWatch, EC2, RDS, Lambda, and more ## Overview AWS integration enables IncidentFox to access your AWS infrastructure for: * CloudWatch Logs and Metrics * EC2 instance status and details * RDS database monitoring * Lambda function analysis * ECS/Fargate task status * CodePipeline deployment tracking ## Prerequisites * AWS account with IAM access * IAM role or user with appropriate permissions * Knowledge of your AWS region(s) ## Configuration ### Step 1: Create IAM Role (Recommended) Create an IAM role for IncidentFox to assume: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Sid": "CloudWatchLogs", "Effect": "Allow", "Action": [ "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:GetLogEvents", "logs:FilterLogEvents", "logs:StartQuery", "logs:GetQueryResults" ], "Resource": "*" }, { "Sid": "CloudWatchMetrics", "Effect": "Allow", "Action": [ "cloudwatch:GetMetricData", "cloudwatch:GetMetricStatistics", "cloudwatch:ListMetrics", "cloudwatch:DescribeAlarms", "cloudwatch:DescribeAlarmHistory" ], "Resource": "*" }, { "Sid": "EC2", "Effect": "Allow", "Action": [ "ec2:DescribeInstances", "ec2:DescribeInstanceStatus", "ec2:DescribeVolumes", "ec2:DescribeNetworkInterfaces" ], "Resource": "*" }, { "Sid": "RDS", "Effect": "Allow", "Action": [ "rds:DescribeDBInstances", "rds:DescribeDBClusters", "rds:DescribeEvents", "pi:GetResourceMetrics" ], "Resource": "*" }, { "Sid": "Lambda", "Effect": "Allow", "Action": [ "lambda:GetFunction", "lambda:ListFunctions", "lambda:GetFunctionConfiguration" ], "Resource": "*" }, { "Sid": "ECS", "Effect": "Allow", "Action": [ "ecs:DescribeClusters", "ecs:DescribeServices", "ecs:DescribeTasks", "ecs:ListTasks" ], "Resource": "*" }, { "Sid": "CodePipeline", "Effect": "Allow", "Action": [ "codepipeline:GetPipeline", "codepipeline:GetPipelineState", "codepipeline:GetPipelineExecution", "codepipeline:ListPipelineExecutions" ], "Resource": "*" } ] } ``` Adjust the Resource ARNs to limit access to specific resources in production. ### Step 2: Configure Trust Policy If using cross-account access: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::INCIDENTFOX_ACCOUNT:role/incidentfox-agent" }, "Action": "sts:AssumeRole" } ] } ``` ### Step 3: Add to IncidentFox **Via Web UI:** 1. Go to **Team Console** > **Integrations** 2. Click **Add Integration** > **AWS** 3. Enter: * Region (primary) * Authentication method (Role ARN or Access Keys) * Role ARN (if using assume role) 4. Click **Test Connection** 5. Save **Via Configuration:** ```json theme={null} { "tools": { "aws": { "enabled": true, "region": "us-west-2", "assume_role": "arn:aws:iam::123456789:role/incidentfox-readonly" } } } ``` For multiple regions: ```json theme={null} { "tools": { "aws": { "enabled": true, "regions": ["us-west-2", "us-east-1"], "assume_role": "arn:aws:iam::123456789:role/incidentfox-readonly" } } } ``` ## Available Tools ### CloudWatch Logs #### `get_cloudwatch_logs` Fetch logs from CloudWatch Log Groups. ``` @incidentfox get cloudwatch logs for /aws/lambda/payments-processor from the last hour ``` **Parameters:** * `log_group` - Log group name * `filter_pattern` - CloudWatch filter pattern * `time_range` - Time range to search #### `query_cloudwatch_insights` Run CloudWatch Logs Insights queries. ``` @incidentfox run cloudwatch insights query to find error patterns in the last 24 hours ``` **Parameters:** * `log_groups` - Log groups to query * `query` - Insights query string * `time_range` - Time range **Example Query:** ```sql theme={null} fields @timestamp, @message | filter @message like /ERROR/ | stats count(*) as errorCount by bin(1h) | sort @timestamp desc ``` ### CloudWatch Metrics #### `get_cloudwatch_metrics` Query CloudWatch metrics. ``` @incidentfox get CPU utilization for the payments EC2 instances over the last 6 hours ``` **Parameters:** * `namespace` - Metric namespace (e.g., AWS/EC2) * `metric_name` - Metric name * `dimensions` - Dimension filters * `statistic` - Average, Sum, Maximum, etc. * `period` - Data point period in seconds ### EC2 #### `describe_ec2_instance` Get EC2 instance details and status. ``` @incidentfox describe EC2 instance i-0123456789abcdef0 ``` **Parameters:** * `instance_id` - EC2 instance ID **Returns:** * Instance state * Instance type * Launch time * Security groups * Network interfaces * Tags ### RDS #### `get_rds_instance_status` Check RDS database status and metrics. ``` @incidentfox check the status of the production RDS instance ``` **Parameters:** * `db_identifier` - RDS instance identifier **Returns:** * Instance status * Endpoint * Storage allocation * Recent events * Performance metrics ### Lambda #### `describe_lambda_function` Get Lambda function configuration. ``` @incidentfox describe the payment-processor Lambda function ``` **Parameters:** * `function_name` - Lambda function name **Returns:** * Runtime * Memory configuration * Timeout * Environment variables * Last modified * Recent invocations ### ECS #### `list_ecs_tasks` List ECS tasks in a cluster/service. ``` @incidentfox list ECS tasks for the checkout service ``` **Parameters:** * `cluster` - ECS cluster name * `service` - Service name (optional) * `status` - RUNNING, STOPPED, etc. ### CodePipeline #### `describe_codepipeline` Get CodePipeline execution status. ``` @incidentfox check the status of the main deployment pipeline ``` **Parameters:** * `pipeline_name` - Pipeline name **Returns:** * Pipeline state * Stage statuses * Recent executions * Failed actions (if any) ## Use Cases ### Investigating Lambda Errors ``` @incidentfox investigate errors in the payment-processor Lambda IncidentFox will: 1. Check Lambda function configuration 2. Query CloudWatch Logs for errors 3. Get invocation metrics 4. Identify patterns ``` ### RDS Performance Issues ``` @incidentfox check RDS performance for the production database IncidentFox will: 1. Get RDS instance status 2. Query Performance Insights metrics 3. Check connection count 4. Review recent events ``` ### Deployment Tracking ``` @incidentfox did any CodePipeline deployments happen in the last 4 hours? IncidentFox will: 1. List recent pipeline executions 2. Show deployment status 3. Correlate with any incidents ``` ## Multi-Account Setup For organizations with multiple AWS accounts: ```json theme={null} { "tools": { "aws": { "enabled": true, "accounts": [ { "name": "production", "assume_role": "arn:aws:iam::111111111111:role/incidentfox", "regions": ["us-west-2", "us-east-1"] }, { "name": "staging", "assume_role": "arn:aws:iam::222222222222:role/incidentfox", "regions": ["us-west-2"] } ] } } } ``` ## Troubleshooting ### Access Denied **Symptom:** "User is not authorized to perform this action" **Solutions:** 1. Verify IAM policy attached to role 2. Check trust relationship allows assume role 3. Ensure resource ARNs match your resources ### No Data Returned **Symptom:** Queries return empty results **Solutions:** 1. Verify region is correct 2. Check time range - CloudWatch has retention limits 3. Confirm log group/metric names are exact ### Throttling **Symptom:** "Rate exceeded" errors **Solutions:** 1. Reduce query frequency 2. Use broader time periods 3. Request AWS quota increase ## Best Practices **Use CloudWatch Insights** for complex log analysis - it's faster and more powerful than filter patterns. 1. **Use resource tagging** - Tag resources to enable filtered queries 2. **Set up log retention** - Ensure logs are retained long enough for investigations 3. **Use cross-account roles** - Avoid using access keys 4. **Enable Performance Insights** - For RDS debugging 5. **Set up CloudWatch alarms** - So IncidentFox can reference them ## Next Steps Connect K8s clusters Set up Datadog # Coralogix Source: https://docs.incidentfox.ai/data-sources/coralogix Connect IncidentFox to Coralogix for log search, metrics, and alerts