How I Built a Multi-Account Deployment Dashboard When Traditional Monitoring Tools Fell Short

Tags: AWS, Observability, GitHub Actions, DynamoDB

If you run many AWS accounts and multiple version streams (app, infra, CI, config), answering “what version runs where?” becomes a fire drill. This article shows you how to turn existing Slack deployment notifications into a near-real-time, single source of truth without agents, heavy monitoring costs, or extra infra.

In modern DevOps, we’ve automated almost everything – deployments, rollbacks, scaling, and monitoring. Yet one fundamental question remains surprisingly difficult to answer: “What version is running where?”

This gets exponentially harder when you’re dealing with multiple versioning schemes. Modern applications often have separate versions for the application code, infrastructure definitions, CI/CD workflows, and configuration schemas. Each component might be deployed independently, creating a matrix of version combinations across environments. When a critical issue occurs, you need to quickly identify not just which application version is running, but which specific combination of components is deployed.

Enterprise monitoring tools focus on metrics and logs – they’ll tell you that latency spiked at 3 PM or that error rates increased after the last deployment. But correlating these issues with specific version combinations across a distributed infrastructure? That’s where they fall short. You end up jumping between GitHub releases, deployment logs, and monitoring dashboards, trying to piece together the actual state of your system.

The Challenge: Tracking Deployments Across 30+ AWS Accounts

I recently faced this challenge in its most extreme form. My team had just finished developing a new product and began deploying it across our infrastructure, which consisted of over 30 AWS accounts spread across multiple environments.
This wasn’t just about scale; it was about complexity multiplied.

Here’s where it got complex:

4 environments: Dev, Test, Staging, and Production
Multiple AWS accounts per environment: Production alone had 16 accounts, Staging had 6, and so on
Dual version tracking: One version for the product itself, another for the GitHub Actions workflows
Result: 60+ different version combinations to track across the infrastructure

The development process involved separate GitHub repositories for each environment, with multiple AWS accounts pointing to the same repository. This architecture, while providing strong isolation and security boundaries, created a tracking nightmare.

Why Datadog Wasn’t the Answer

My first instinct was to use Datadog, the existing monitoring solution.
However, I quickly hit several roadblocks:

Security Constraints

For compliance reasons, production monitoring data couldn’t coexist with non-production data in the same Datadog organization. This meant we’d need multiple Datadog instances, defeating the purpose of centralized monitoring.

Multi-Account Complexity

Managing Datadog agents and configurations across 30+ AWS accounts would require:

Individual agent installations per account
Complex IAM role configurations
Significant cost multiplication (each account would need its own monitoring setup)
No unified view without expensive enterprise features

Cost Considerations

With Datadog’s per-host and per-metric pricing model, monitoring 30+ accounts would have been prohibitively expensive for what was essentially a version tracking requirement.

The Eureka Moment: Leveraging Existing Slack Notifications

While thinking of other solutions, I noticed something we already had: every deployment triggered a Slack notification to a dedicated channel. The GitHub Actions workflows were configured to send notifications at different stages.

Understanding The Notification Strategy

The deployment workflow sends three types of notifications, each serving a critical purpose in tracking deployment state:

Starting Notification: Sent when a deployment workflow begins
- Indicates that a deployment is currently in progress
- Contains the target AWS account and environment information
- Useful for tracking currently running deployments
- Helps identify stuck or long-running deployments
Success Notification: Sent when deployment completes successfully
- Contains the new version numbers for both the product and GitHub Actions
- Confirms that the deployment finished without errors
- Includes a link to the successful GitHub Actions run for audit purposes
Failure Notification: Sent when deployment fails
- Indicates that the deployment attempted but encountered an error
- Crucially, the previous version remains active (auto-rollback)
- Contains error information and a link to the failed GitHub Actions run for debugging

The Key Insight: State Management Logic

Understanding how to handle each notification type was crucial for accurate version tracking:

On “Starting”: We mark the deployment as “in progress” but don’t change any version numbers yet, as we don’t know if it will succeed.
On “Success”: We update both the version numbers AND the status to “success” – this is the only time versions should change.
On “Failure”: We update only the status to “failure” but explicitly preserve the existing version numbers – the previous successful deployment is still running.

This state management ensures that our dashboard always shows the actual running version, not the attempted version. If a deployment fails, we know exactly what version is still active in that environment.

These notifications contained all the information we needed:

AWS account details
Product version
GitHub Actions version
Deployment status (success/failure/starting)
Links to the GitHub Actions runs

Instead of building new infrastructure, why not leverage what we already have?

The Technical Solution

I designed a serverless architecture that would:

Capture Slack messages via webhook
Process and extract deployment information
Store data in DynamoDB
Display everything in a real-time dashboard

Architecture Overview

GitHub Actions → Slack Channel → Slack App → API Gateway → Lambda → DynamoDB
                                                                        ↑
                                                                        │
                                                                  Flask Dashboard

Component 1: Slack Integration

I created a Slack app that listens to the deployment channel and forwards messages to an API Gateway endpoint. The app manifest was simple:

{
  "event_subscriptions": {
    "request_url": "https://api-gateway-url.amazonaws.com/prod/slack-webhook",
    "bot_events": [
      "message.channels",
      "message.groups"
    ]
  }
}

When a message is posted to the channel, Slack sends an HTTP POST request to our API Gateway with a JSON payload:

{
  "type": "event_callback",
  "event": {
    "type": "message",
    "channel": "C076V92HK0A",
    "text": "Deployment notification text",
    "attachments": [
      {
        "text": "*Workflow*: `Deploy`\n*Product Version*: `v0.4.16`\n*GHA Version*: `v1.0.19`\n*AWS Account*: `002678510244_aws-platform-app-test`\n*Status*: `success`\n*Link to Action*: "
      }
    ],
    "blocks": [...]
  }
}

Component 2: Lambda Function

The Lambda function became the core processor, handling multiple Slack message formats and implementing the crucial state management logic.

When GitHub Actions posts to Slack, the messages contain specific fields that we need to extract:

Workflow: The type of deployment (Deploy, Release, etc.)
Product Version: The version of the product being deployed
GHA Version: The version of the GitHub Actions workflows
AWS Account: The target AWS account identifier
Status: Deployment result (success, failure, or starting)
Link to Action: URL to the GitHub Actions run

Here’s the key parsing logic that extracts these fields from Slack’s formatted messages:

def parse_github_actions_attachment(text):
    """Parse deployment info from Slack's markdown-formatted attachments"""

    # The messages arrive with Slack markdown formatting:
    # *Field Name*: `value` or *Field Name*: 

    patterns = {
        'version': r"\*Product Version\*:\s*`([^`]+)`",
        'gha_version': r"\*GHA Version\*:\s*`([^`]+)`",
        'aws_account': r"\*AWS Account\*:\s*`([^`]+)`",
        'status': r"\*Status\*:\s*`([^`]+)`",
        'action_link': r"\*Link to Action\*:\s*(?:<([^>]+)>|`([^`]+)`)"
    }

    extracted_data = {}
    for key, pattern in patterns.items():
        match = re.search(pattern, text)
        if match:
            extracted_data[key] = match.group(1).strip()
    return extracted_data

The function intelligently detects environments based on AWS account names:

def detect_environment(aws_account):
    account_lower = aws_account.lower()
    if "dev" in account_lower:
        return "Dev"
    elif "test" in account_lower:
        return "Test"
    elif "stage" in account_lower:
        return "Stage"
    elif "prod" in account_lower:
        return "Prod"
    else:
        return "Unknown"

Component 3: DynamoDB Schema

I designed a simple but effective schema:

{
    'environment': 'Prod',  # Partition Key
    'awsAccount': 'aws-account-name',  # Sort Key
    'productVersion': 'v1.2.3',
    'ghaVersion': 'v2.1.0',
    'deployTime': '2025-01-26T10:30:00+02:00',
    'deployStatus': 'success',
    'actionLink': 'https://github.com/...'
}

The composite key (environment + awsAccount) ensures we maintain the latest deployment state for each account.

Component 4: Flask Dashboard

The dashboard provides real-time visibility with:

@app.route("/")
def index():
    # Fetch and group deployments by environment
    deployments = table.scan()
    grouped = group_by_environment(deployments)

    return render_template(
        "dashboard.html",
        environments=grouped,
        last_updated=get_last_update_time()
    )

# Background polling for real-time updates
def poll_dynamodb():
    while True:
        update_cache()
        time.sleep(30)  # Poll every 30 seconds

Frontend Features

The dashboard includes:

Environment grouping with visual status indicators
Real-time updates via background polling
Multi-criteria filtering (by status, environment)
Dark mode for those late-night debugging sessions
Collapsible sections for better organization
Direct links to GitHub Actions runs

// Auto-refresh mechanism
setInterval(() => {
    fetch('/api/data')
        .then(response => response.json())
        .then(data => updateDashboard(data));
}, 30000);

Key Implementation Insights

Handling Multiple Message Formats

Slack messages arrive in various formats (attachments, Block Kit, plain text).
The Lambda function tries multiple parsing strategies:

# Try attachment format first
if "attachments" in body["event"]:
    data = parse_github_actions_attachment(attachment["text"])

# Fallback to Block Kit format
elif "blocks" in body["event"]:
    data = parse_slack_blocks(blocks)

# Finally try plain text
elif "text" in body["event"]:
    data = parse_slack_message(text)

Critical State Management for Failed Deployments

The system maintains version integrity by implementing careful state management based on deployment status:

if deploy_status == "success":
    # Update all fields including versions
    table.put_item(Item=complete_item)

elif deploy_status == "failure":
    # Only update status and timestamp, preserve versions
    table.update_item(
        Key={'environment': env, 'awsAccount': account},
        UpdateExpression='SET deployStatus = :s, deployTime = :t',
        ExpressionAttributeValues={':s': status, ':t': timestamp}
    )

elif deploy_status == "starting":
    # Mark as in-progress, don't touch versions
    table.update_item(
        Key={'environment': env, 'awsAccount': account},
        UpdateExpression='SET deployStatus = :s, deployTime = :t',
        ExpressionAttributeValues={':s': 'in-progress', ':t': timestamp}
    )

This logic ensures that:

Failed deployments never overwrite successful version numbers
Starting deployments are tracked without assuming success
The dashboard always shows what’s actually running, not what was attempted

Tracking Latest Versions

I added a special DynamoDB entry to track the latest released versions:

def handle_release_workflow(version, link):
    table.put_item(Item={
        'environment': 'GLOBAL_CONFIG',
        'awsAccount': 'latest_versions',
        'productVersion': version,
        'deployTime': now(),
        'deployStatus': 'success'
    })

Dashboard Hosting Infrastructure

The Flask dashboard runs on a dedicated EC2 instance within our existing AWS infrastructure, providing reliable and secure access for our team.

DNS and Network Access

To ensure secure access, I implemented the following network configuration:

# Route53 private hosted zone configuration

Domain: versions-dashboard.internal

Type: A Record

Target: EC2 instance private IP

Access: Corporate VPN required

This setup ensures that the dashboard is only accessible from within our corporate network, maintaining security while providing easy access to authorized users.

Systemd Service Management

To ensure 24/7 availability, the Flask application runs as a systemd service:

[Unit]
Description=Deployment Dashboard Flask App
After=network.target

[Service]
User=ec2-user
WorkingDirectory=/home/ec2-user/dashboard/
ExecStart=/usr/bin/python3 /home/ec2-user/dashboard/app.py
Restart=always
RestartSec=10
Environment=PYTHONUNBUFFERED=1

[Install]
WantedBy=multi-user.target

This configuration provides:

Automatic startup on system boot
Automatic restart if the application crashes
10-second delay between restart attempts to prevent resource exhaustion
Unbuffered Python output for real-time logging

Background Polling Architecture

The Flask application implements a background thread for continuous DynamoDB polling:

Zero additional infrastructure costs – Leveraged existing Slack notifications
Centralized visibility – Single dashboard for all 30+ accounts
Security compliance – Data remains within our infrastructure
Real-time updates – 30-second polling provides near real-time status
Historical tracking – DynamoDB maintains deployment history
Low maintenance – Serverless architecture requires minimal upkeep

Performance Metrics

Lambda execution time: ~200ms per message
Dashboard load time: <500ms for 30+ accounts
DynamoDB costs: <$5/month for our usage
Total implementation time: 3 days

Lessons Learned

Sometimes the best solution leverages what you already have – We had Slack notifications; we just needed to listen to them differently.
Serverless is perfect for event-driven workflows – Our Lambda function only runs when deployments happen, keeping costs minimal.
Custom solutions can outperform enterprise tools – For specific use cases, a targeted solution beats a general-purpose tool.
UI/UX matters even for internal tools – The dashboard’s filtering, dark mode, and responsive design significantly improved adoption.
State management is critical for deployment tracking – Understanding when to update versions vs. just status prevented countless confusion about what was actually running.

Conclusion

When faced with tracking deployments across 30+ AWS accounts, traditional monitoring tools proved inadequate due to security constraints, complexity, and cost. By creatively leveraging existing Slack notifications and building a serverless processing pipeline, we achieved better visibility at a fraction of the cost and complexity.

The key takeaway? Before reaching for expensive enterprise solutions, consider whether you can repurpose existing data streams. In our case, those Slack notifications we were already receiving became the foundation for a comprehensive deployment tracking system.

The entire solution runs on a single EC2 instance for the Flask app, uses minimal Lambda invocations, and a small DynamoDB table – proving that effective DevOps solutions don’t always require complex or expensive tools.

What’s Next: Future Improvements

While the current solution effectively solves our immediate needs, there are several enhancements I’m planning to implement:

Dockerization and CI/CD

First priority is containerizing the Flask application to enable proper versioning and deployment automation:

Docker container – Package the app with all dependencies for consistent deployments
Version tagging – Track the dashboard app versions alongside
Simple CI/CD pipeline – Automated testing and deployments

This would be a lightweight implementation – just enough automation to ensure reliable dashboard updates without the overhead of complex orchestration.

Real-Time Updates with WebSockets

Currently, the dashboard polls DynamoDB every 30 seconds, which works but isn’t optimal. I’m planning to implement a WebSocket connection between the Flask app and the browser, combined with DynamoDB Streams. This would:

Push updates instantly when deployments occur, rather than waiting for the next poll cycle
Reduce unnecessary DynamoDB read operations, lowering costs
Provide true real-time visibility into deployment progress

# Future implementation concept
from flask_socketio import SocketIO, emit

socketio = SocketIO(app)

def process_dynamodb_stream(record):
    # When DynamoDB changes, emit to all connected clients
    if record['eventName'] in ['INSERT', 'MODIFY']:
        socketio.emit('deployment_update', {
            'environment': record['dynamodb']['Keys']['environment'],
            'data': record['dynamodb']['NewImage']
        })

Historical Deployment Analytics

Adding deployment metrics and trends analysis:

Deployment frequency by environment
Success/failure rates over time
Average deployment duration tracking

Traditional monitoring didn’t solve version correlation across 30+ AWS accounts. By turning existing Slack notifications into a serverless pipeline and a lightweight UI, we got a near real-time, single source of truth at a fraction of the cost and complexity.