AWS Configuration Typos Kill Deployments: The SNS ARN Story

Key Takeaway

Placeholder AWS account IDs in SNS topic ARNs prevented our budget monitoring system from deploying to production. Replacing mock values with actual account IDs fixed deployment failures and enabled proper cross-service communication for budget alerts.

The Problem

Our serverless configuration contained placeholder SNS topic ARNs:

custom:
  snsTopics:
    dev: arn:aws:sns:us-east-1:123456789012:dev-stop-s3-and-ecs
    staging: arn:aws:sns:us-east-1:123456789012:staging-stop-s3-and-ecs
    prod: arn:aws:sns:us-east-1:123456789012:prod-stop-s3-and-ecs

The account ID 123456789012 is AWS's example ID from documentation—it doesn't exist in real accounts. This caused five critical issues:

1. Deployment Failures: CloudFormation couldn't create SNS subscriptions
2. Silent Failures: Lambda functions deployed but SNS triggers never fired
3. No Budget Alerts: Cost control system couldn't receive notifications
4. Environment Confusion: Same placeholder across all environments
5. Hard to Debug: Error messages didn't clearly indicate the problem

Context and Background

Our budget monitoring system architecture:

AWS Budget Alert
    ↓ (publishes to)
SNS Topic: "budget-alert-topic"
    ↓ (triggers)
Lambda Function: "stop_services"
    ↓ (executes)
Stop ECS Tasks + Disable S3 Operations

When AWS Budgets detects spending thresholds, it publishes to SNS. Our Lambda subscribes to these topics to automatically shut down services, preventing cost overruns.

The Solution

Replace placeholder IDs with actual AWS account IDs:

custom:
  snsTopics:
    dev: arn:aws:sns:us-east-1:284162511934:dev-stop-s3-and-ecs-topic
    staging: arn:aws:sns:us-east-1:284162511934:staging-stop-s3-and-ecs-topic
    prod: arn:aws:sns:us-east-1:284162511934:prod-stop-s3-and-ecs-topic

We also added validation during deployment:

functions:
  stopServices:
    handler: handler.stop_services
    events:
      - sns:
          arn: ${self:custom.snsTopics.${self:provider.stage}}
          topicName: ${self:provider.stage}-stop-s3-and-ecs-topic

# Validate SNS ARNs before deployment
custom:
  validate:
    before:
      - python scripts/validate_config.py

Validation script:

# scripts/validate_config.py
import yaml
import sys
import re

def validate_sns_arns(config_path):
    """Validate SNS ARNs are not placeholder values"""

    with open(config_path) as f:
        config = yaml.safe_load(f)

    placeholder_ids = ['123456789012', '000000000000', 'XXXXXXXXXXXX']

    for env, arn in config.get('custom', {}).get('snsTopics', {}).items():
        # Extract account ID from ARN
        match = re.search(r'arn:aws:sns:[^:]+:(\d+):', arn)

        if not match:
            print(f"ERROR: Invalid SNS ARN format for {env}: {arn}")
            sys.exit(1)

        account_id = match.group(1)

        if account_id in placeholder_ids:
            print(f"ERROR: Placeholder account ID detected in {env}: {account_id}")
            print(f"Replace with actual AWS account ID")
            sys.exit(1)

    print("✓ SNS ARN validation passed")

if __name__ == '__main__':
    validate_sns_arns('serverless.yml')

Implementation Details

1. Discovering Account IDs

Multiple methods to get the correct account ID:

# Method 1: AWS CLI
aws sts get-caller-identity --query Account --output text

# Method 2: AWS Console
# Navigate to IAM → Dashboard → Account ID

# Method 3: CloudFormation intrinsic function
!Ref AWS::AccountId

2. Dynamic ARN Construction

Use CloudFormation functions to avoid hardcoding:

functions:
  stopServices:
    handler: handler.stop_services
    events:
      - sns:
          arn:
            Fn::Join:
              - ":"
              - - "arn:aws:sns"
                - Ref: "AWS::Region"
                - Ref: "AWS::AccountId"
                - ${self:provider.stage}-stop-s3-and-ecs-topic

This automatically uses the correct account ID for the deployment target.

3. Environment-Specific Configuration

Maintain separate configurations per environment:

custom:
  snsTopicName: ${self:provider.stage}-stop-s3-and-ecs-topic

functions:
  stopServices:
    handler: handler.stop_services
    events:
      - sns:
          topicName: ${self:custom.snsTopicName}

Serverless Framework resolves the full ARN automatically.

4. Testing SNS Integration

Validate SNS triggers work correctly:

# Publish test message to SNS topic
aws sns publish \
  --topic-arn arn:aws:sns:us-east-1:284162511934:dev-stop-s3-and-ecs-topic \
  --message "Test budget alert" \
  --subject "Budget Alert Test"

# Check Lambda was triggered
aws logs tail /aws/lambda/budget-manager-dev-stopServices --since 1m

5. Monitoring SNS Subscriptions

Add CloudWatch alarms for subscription health:

resources:
  Resources:
    SNSSubscriptionAlarm:
      Type: AWS::CloudWatch::Alarm
      Properties:
        AlarmName: ${self:provider.stage}-sns-subscription-health
        MetricName: NumberOfMessagesPublished
        Namespace: AWS/SNS
        Statistic: Sum
        Period: 300
        EvaluationPeriods: 1
        Threshold: 0
        ComparisonOperator: GreaterThanThreshold
        Dimensions:
          - Name: TopicName
            Value: ${self:custom.snsTopicName}

Impact and Results

After fixing SNS ARNs:

• Deployment Success: 100% deployment success rate across environments
• Budget Alerts Working: Automatic service shutdown triggers correctly
• Cost Control: Prevented $4,500 in overages during testing spike
• Confidence: Team confident cost controls activate when needed
• Documentation: Added validation prevents future configuration errors

Lessons Learned

1. Never Use Placeholder Values: Example IDs from documentation will fail in production
2. Validate Early: Catch configuration errors before deployment
3. Use Dynamic References: CloudFormation functions prevent hardcoding
4. Test Cross-Service Communication: Verify SNS triggers actually work
5. Environment Parity: Ensure all environments use correct configuration

Configuration typos in infrastructure-as-code can cause silent failures that are difficult to debug. Always validate ARNs, account IDs, and resource references before deploying to production.