Log Analysis Tools

​

Overview

​

Tools Available

​

log_get_statistics

@incidentfox show me log statistics for the payments service over the last hour

• Total log volume
• Error rate percentage
• Log level distribution
• Top error messages
• Throughput over time

​

log_sample

@incidentfox sample 100 error logs from the checkout service

• Initial investigation to understand error types
• Pattern discovery before targeted searches
• Representative data for analysis

​

log_search_pattern

@incidentfox search for "timeout after [0-9]+ ms" in the api service logs

• Full regex syntax
• Case-insensitive matching
• Multi-line patterns

​

log_around_timestamp

@incidentfox show me logs 5 minutes before and after the error at 14:32:15 UTC

• Logs from the target service
• Related logs from dependent services
• System events in the timeframe

​

log_correlate_events

@incidentfox correlate logs for trace ID abc-123-def

• Timeline of events across services
• Latency breakdown by service
• Error propagation path

​

log_extract_signatures

@incidentfox extract error signatures from the last 24 hours

1. Clusters similar log messages
2. Extracts common patterns (parameterized)
3. Ranks by frequency and impact

{
  "signatures": [
    {
      "pattern": "Connection refused to {host}:{port}",
      "count": 1234,
      "first_seen": "2024-01-15T10:00:00Z",
      "last_seen": "2024-01-15T14:30:00Z"
    }
  ]
}

​

log_detect_anomalies

@incidentfox detect anomalies in the api service logs

• Unusual log volume spikes/drops
• New error types not seen before
• Abnormal patterns in log messages

​

Configuration

​

Backend Selection

{
  "tools": {
    "log_analysis": {
      "default_backend": "elasticsearch",
      "backends": {
        "elasticsearch": {
          "hosts": ["https://es.your-domain.com:9200"]
        },
        "cloudwatch": {
          "log_group_prefix": "/aws/lambda/"
        }
      }
    }
  }
}

​

Sampling Settings

{
  "tools": {
    "log_analysis": {
      "default_sample_size": 100,
      "max_sample_size": 1000
    }
  }
}

​

Use Cases

​

Error Investigation

1. Start with log_get_statistics to understand volume
2. Use log_sample to see representative errors
3. Apply log_extract_signatures to identify patterns
4. Drill down with log_search_pattern

​

Incident Timeline

1. Identify incident start with log_detect_anomalies
2. Get context with log_around_timestamp
3. Trace across services with log_correlate_events

​

Proactive Monitoring

1. Run log_detect_anomalies to find new issues
2. Extract signatures to track recurring problems
3. Correlate with deployment events

​

Best Practices

​

Time Ranges

• Initial investigation: 1 hour
• Pattern analysis: 24 hours
• Trend analysis: 7 days

​

Filtering

@incidentfox search for errors in the checkout service, excluding health checks

​

Correlation IDs

• Trace ID (OpenTelemetry)
• Request ID
• Session ID

​

Next Steps