Skip to content
- AWS MSK Cost Optimization: How to Cut Managed Kafka Costs by 30-50%
AWS MSK Cost Optimization: How to Cut Managed Kafka Costs by 30-50%
Amazon Managed Streaming for Apache Kafka (MSK) removes the operational complexity of running Kafka clusters. But MSK's managed convenience comes with a pricing structure that scales faster than most teams expect. A 10-broker cluster provisioned to handle peak traffic can cost $3,000+ per month even when running at 30% average utilization. Cross-Availability Zone replication traffic, which MSK requires for durability, can represent 50%+ of the total monthly bill. Storage costs multiply as Kafka's replication factor turns 1 TB of topic data into 3 TB of paid EBS storage.
This guide gives a systematic framework for MSK cost optimization: what optimization means for managed Kafka, how AWS MSK pricing works (broker hours + storage + data transfer), proven optimization strategies, and how to monitor and manage MSK costs at scale when you're running dozens of clusters across multiple AWS accounts.
What Is MSK Cost Optimization?
MSK cost optimization means minimizing the cost of running Amazon Managed Streaming for Apache Kafka clusters while maintaining throughput targets, partition leadership availability, and other SLAs. AWS MSK pricing has three core components: broker compute hours ($0.408/hour for express.m7g.large in provisioned mode), storage ($0.10 per GB-month for EBS), and data ingest ($0.01 per GB for MSK Express; cross-AZ transfer applies standard AWS rates for provisioned clusters). Unlike serverless compute services that scale to zero, MSK charges for running brokers 24/7 whether they're processing messages or sitting idle.
Optimization targets vary by organization, but most teams aim to reduce MSK spend by 30-50% within 90 days while maintaining sub-second consumer lag, 99.9%+ partition leadership availability, and zero message loss. This typically involves right-sizing broker types to match actual CPU/network/disk utilization patterns, migrating to ARM Graviton instances for 24-29% cost reduction with equivalent or better throughput, optimizing partition placement to avoid leader concentration on specific brokers, and tuning retention policies to align with actual replay requirements.
MSK Pricing: Quick Overview
Before diving into optimization strategies, understanding how AWS bills for MSK prevents surprises when the monthly invoice arrives.
Broker charges are straightforward hourly rates that vary by instance type and deployment mode. For MSK Express (recent simplified deployment option), an express.m7g.large broker costs approximately $0.408/hour in US regions. A 3-broker cluster runs ~$891/month in broker compute alone ($0.408 × 3 brokers × 730 hours). A 9-broker cluster: ~$2,673/month. A 30-broker cluster using express.m7g.2xlarge (4× larger instance): ~$35,712/month before storage or data charges. For provisioned clusters using kafka.m5 or kafka.m7g instance types, AWS publishes hourly rates per broker size [AWS Official Pricing]. Brokers run continuously; there is no "scale to zero" — you pay for capacity whether actively processing messages or idle.
Storage charges are billed per GB-month of EBS volume capacity attached to broker nodes. MSK charges $0.10 per GB-month. Here's where replication multiplies costs: if your topics have replication factor = 3 (AWS best practice for high availability), 1 TB of logical topic data consumes 3 TB of EBS storage — a $300/month storage bill instead of $100. A 3-broker cluster with ~1.5 TB average retained storage costs ~$152/month in storage. A 9-broker cluster with ~4.5 TB: ~$455/month. A 30-broker cluster with ~15 TB: ~$1,516/month.
Data transfer costs have two dimensions. MSK Express charges $0.01 per GB for data ingested into the cluster. For provisioned clusters, AWS applies standard cross-Availability Zone data transfer rates ($0.01-$0.02 per GB depending on region) when replica traffic crosses AZ boundaries. One critical nuance: AWS states that MSK does NOT charge for inter-broker replication traffic within the same region — a pricing advantage over self-hosted Kafka on EC2, where cross-AZ replica traffic would incur standard AWS data transfer fees. However, cross-AZ consumer traffic (clients in AZ-A consuming from brokers in AZ-B) still generates standard data transfer charges.
MSK Serverless shifts from infrastructure units to usage-based pricing: $0.75/hour for cluster capacity, $0.0015/hour per partition, $0.10 per GB data in, $0.05 per GB data out. Serverless suits unpredictable workloads where traffic patterns vary significantly (development/testing clusters, event-driven pipelines with sporadic activity), but for sustained 24/7 production traffic, provisioned clusters typically deliver lower total cost.
For detailed pricing across all regions and instance types, see the AWS MSK Pricing page.
MSK Cost Optimization Strategies
The table below compares optimization impact across five key levers:
| Strategy | Typical Savings | Implementation Effort | Risk Level |
| Right-size broker instances | 20–30% broker cost reduction | Medium (requires CloudWatch analysis and staging tests) | Low (reversible via broker type update) |
| Migrate to Graviton (kafka.m7g) | 24% broker cost reduction | Low (configuration change and compatibility testing) | Low (AWS-supported migration path) |
| Optimize storage retention | 40–60% storage cost reduction | Low (adjust retention policies per topic) | Medium (ensure retention meets recovery requirements) |
| Minimize cross-AZ traffic | 20–40% network cost reduction | Medium (rack awareness and client co-location) | Low (improves latency while reducing cost) |
| Optimize partition placement | 10–20% efficiency gains | High (requires partition reassignment analysis) | Medium (can temporarily increase replication load) |
Strategy 1: Right-Size Broker Instances for 20-30% Cost Reduction
Broker right-sizing is the highest-impact MSK optimization lever because broker compute represents 60-70% of total MSK spend for typical workloads. Over-provisioning brokers to handle peak traffic creates persistent idle capacity cost.
AWS recommends maintaining CPU utilization (defined as CPU User + CPU System) below 60% to ensure sufficient headroom for operational events like broker failures, patching, and rolling upgrades [AWS Official Documentation]. When a broker goes offline for maintenance or failure, Kafka reassigns partition leadership to other brokers in the cluster, redistributing workload. If brokers are already running at 85% CPU, the sudden leadership shift can push remaining brokers to 100% utilization, causing consumer lag and potential message loss.
Example: A kafka.m5.2xlarge broker costs approximately $0.344/hour ($251/month). If CloudWatch CPU metrics show average utilization of 35% with p95 at 48%, the broker is over-provisioned. Downsizing to kafka.m5.xlarge ($0.172/hour, $125/month) cuts broker costs by 50% while maintaining adequate headroom for traffic spikes. For a 9-broker cluster, this saves ~$1,134/month in broker compute alone.
Right-sizing methodology: Monitor `CPUUser` and `CPUSystem` CloudWatch metrics for each broker over 30 days. Calculate average and p95 CPU utilization. If average <40% and p95 <60%, the broker is a candidate for downsizing. Use Amazon CloudWatch metric math to create a composite metric (`CPUUser + CPUSystem`) and set alarms to trigger when average usage exceeds 60%. Test the smaller instance type in a staging cluster with production-representative traffic before applying changes to production.
AWS publishes recommended partition counts per broker type: kafka.m5.large supports 1,000 partitions (1,500 max for update operations), kafka.m5.2xlarge supports 2,000 (3,000 max), kafka.m5.4xlarge and larger support 4,000 (6,000 max). Exceeding recommended partition counts can prevent cluster configuration updates, broker size downgrades, and SASL/SCRAM secret association. High partition counts also cause missing Kafka metrics on CloudWatch and Prometheus scraping failures. When right-sizing, ensure the target broker type can handle your partition density.
Clusters with consistent traffic patterns (not highly variable hour-to-hour), workloads where peak traffic is only 2-3× average traffic (not 10×+ spikes), and environments where operational teams can monitor and respond to CPU alarms within 15-30 minutes.
Strategy 2: Migrate to Graviton Instances for 24% Lower Cost + 29% Higher Throughput
AWS Graviton (ARM-based processors) delivers up to 24% lower cost and 29% higher throughput compared to x86-based instances for the same workload [AutoMQ]. For MSK clusters, migrating from kafka.m5.large to kafka.m7g.large provides equivalent compute capacity at lower hourly rates with better price-performance.
Migration requirements: Kafka workloads typically require zero application code changes to migrate to Graviton because Kafka is Java-based and the JVM abstracts processor architecture. However, if your MSK cluster integrates with custom Connect plugins, Lambda functions, or client libraries that include native binaries compiled for x86, those dependencies need ARM-compatible versions. Test workloads in a staging MSK cluster configured with kafka.m7g instance types before production migration.
Example: A 9-broker cluster using kafka.m5.xlarge ($0.172/hour per broker × 9 × 730 hours = ~$1,130/month) migrated to kafka.m7g.xlarge at 24% lower cost saves ~$271/month in broker compute. Over 12 months: ~$3,252 savings for a single cluster. For organizations running 10-20 MSK clusters across development, staging, and production environments, Graviton migration delivers $30,000-$65,000 annual savings with minimal migration effort.
AWS reports 29% higher throughput on Graviton instances for Kafka workloads. This means a kafka.m7g.xlarge can handle equivalent or greater message throughput than kafka.m5.xlarge while consuming less CPU per message processed. Monitor CloudWatch `BytesInPerSec` and `BytesOutPerSec` metrics before and after migration to validate throughput improvements.
Strategy 3: Optimize Storage Retention for 40-60% Storage Cost Savings
Storage costs multiply quietly in MSK environments because of Kafka's replication factor. A 1 TB topic with replication factor = 3 consumes 3 TB of paid EBS storage at $0.10/GB-month — a $300/month storage bill. Default retention policies (7 days or 30 days) often exceed actual business recovery requirements, storing data longer than necessary.
Query each topic's actual consumer patterns over 30 days. If consumers typically replay messages only within the past 24-48 hours for data retrieval, reducing retention from 7 days to 3 days cuts storage consumption by ~57% without impacting operational recovery scenarios. For analytics topics where consumers process data once and never rewind, retention can drop to 12-24 hours. Use Kafka's log compaction feature for topics that need long retention windows but only require the latest value per key (user profiles, configuration state) — log compaction retains infinite history while storing only the most recent value, dramatically reducing storage footprint.
Tiered storage for long-retention topics: For topics that require 30-90 day data retention policies for compliance, forensics or sensitive data handling but are rarely consumed beyond the first 24 hours, consider MSK tiered storage (if using provisioned Standard clusters). Tiered storage moves older log segments to Amazon S3 object storage at lower storage rates ($0.023/GB-month for S3 Standard) while keeping recent data on fast EBS volumes. AWS reports that organizations using tiered storage can reduce storage costs by 60%+ for long-retention workloads.
Storage monitoring and alerting: Create CloudWatch alarms for the `KafkaDataLogsDiskUsed` metric. When disk usage reaches 85%, MSK automatically triggers storage expansion (if auto-scaling is enabled) or surfaces alerts requiring manual intervention [AWS Official Documentation]. Set alarms at 70% to provide advance warning before hitting the 85% automatic expansion threshold. Review storage growth trends monthly to identify topics with runaway retention or unexpected data volume increases.
Strategy 4: Minimize Cross-AZ Traffic for 20-40% Network Cost Reduction
Cross-Availability Zone data transfer is one of MSK's hidden cost multipliers. While AWS does not charge for inter-broker replication traffic within the same region [AxonOps], consumer traffic crossing AZ boundaries incurs standard AWS data transfer fees ($0.01-$0.02 per GB depending on region). For high-throughput workloads processing terabytes of data per day, cross-AZ consumer traffic can represent 20-40% of the total MSK bill.
Co-locate producers and consumers with brokers: Deploy Kafka clients (producers/consumers) in the same Availability Zones as MSK broker nodes to eliminate cross-AZ consumer traffic charges. Use rack awareness configuration to ensure consumers preferentially read from replicas in their local AZ rather than fetching data across AZ boundaries. AWS published guidance on optimizing traffic costs of Amazon MSK consumers on Amazon EKS with rack awareness.
Enable compression: Configure Kafka producers to compress messages before sending to MSK brokers. Kafka supports gzip, snappy, lz4, and zstd compression. Compression reduces data transfer volume by 30-70% depending on message payload structure, cutting both ingest charges (for MSK Express) and cross-AZ transfer fees. The tradeoff is increased CPU utilization on producers and consumers for compression/decompression, but for high-volume workloads, the cost savings typically outweigh the compute overhead.
Strategy 5: Optimize Partition Placement to Balance Load
Uneven partition distribution across brokers creates performance bottlenecks and drives unnecessary scaling costs. If partition leadership is concentrated on 3 brokers in a 9-broker cluster, those 3 brokers hit 80% CPU utilization while the other 6 sit at 20% — forcing the cluster to scale up when rebalancing partitions would resolve the issue.
AWS MSK automatically distributes partitions across broker nodes to balance resource utilization, but manual partition reassignment using `kafka-reassign-partitions.sh` can further optimize placement for specific workload patterns. AWS recommends not reassigning more than 10 partitions in a single reassignment operation to avoid overwhelming the cluster with replication traffic [AWS Official Documentation]. For automated continuous partition rebalancing, AWS supports Cruise Control integration with MSK clusters to dynamically manage partition assignment based on real-time broker resource utilization.
MSK Cost Optimization Best Practices
Beyond specific strategies, these best practices regarding efficient data processing and managing Kafka clusters help optimize costs and reduce waste.
1. Maintain 3-AZ clusters with replication factor ≥3 for production workloads.
AWS strongly recommends 3-Availability Zone MSK clusters with topic replication factor = 3 and minimum in-sync replicas (minISR) = 2 for high availability. A replication factor of 1 causes offline partitions during broker rolling upgrades; replication factor = 2 risks data loss if two brokers fail simultaneously. The cost of replication (3× storage, higher cross-AZ traffic) is an operational insurance policy against message loss and partition unavailability during maintenance windows.
2. Set minISR = RF - 1, never minISR = RF.
If minISR equals replication factor (e.g., minISR=3 for RF=3), producers cannot write to the topic when any single broker is offline for maintenance or failure. This blocks production traffic during routine rolling upgrades. AWS recommends minISR=2 for RF=3 clusters to ensure write availability when one replica is temporarily unavailable.
3. Monitor CPU utilization continuously and maintain <60% average.
Create CloudWatch alarms for the composite metric (`CPUUser + CPUSystem`) per broker. Set alarm thresholds at 60% average over 15 minutes. When triggered, scale up by updating broker size (kafka.m5.large → kafka.m5.xlarge) or by adding brokers to the cluster. Maintaining <60% CPU headroom ensures the cluster can handle broker failures and rolling maintenance without saturating remaining brokers.
4. Monitor disk space and set alarms at 85% utilization.
Create CloudWatch alarms for `KafkaDataLogsDiskUsed` metric at 85% threshold. When triggered, enable automatic storage scaling (if not already active) or manually increase broker storage capacity. Running out of disk space prevents producers from writing messages and can cause broker crashes.
5. Monitor memory usage via HeapMemoryAfterGC metric.
AWS recommends creating CloudWatch alarms when `HeapMemoryAfterGC` exceeds 60%. High memory usage can cause out-of-memory errors and broker failures. For clusters using transactional message delivery, reduce `transactional.id.expiration.ms` from default 604800000 ms (7 days) to 86400000 ms (1 day) to decrease the memory footprint of each transaction.
6. Optimize thread configuration for kafka.m5.4xlarge and larger instances.
For clusters using kafka.m5.4xlarge, kafka.m7g.4xlarge, or larger brokers, tune `num.io.threads` and `num.network.threads` configuration parameters to fully utilize available CPU cores. AWS recommends kafka.m5.4xlarge: num.io.threads=16, num.network.threads=8; kafka.m5.8xlarge: 32/16; kafka.m5.12xlarge: 48/24. Do not increase num.network.threads without first increasing num.io.threads, as this can cause queue saturation and degraded performance.
7. Use CloudWatch and Cost Explorer for cost tracking and anomaly detection.
Enable AWS Cost Explorer with daily granularity to detect MSK cost anomalies within 24 hours instead of waiting for monthly bill close. Filter by service (Amazon MSK), group by usage type to separate broker charges from storage charges, and use tag-based grouping to track spend by environment (dev/staging/prod) or team. Set AWS Budgets to alert when MSK spend exceeds forecasted thresholds by 10-20%.
8. Test broker size and configuration changes in staging before production.
MSK configuration changes (broker type, num.io.threads, retention policy) can impact cluster behavior in unexpected ways. Clone production cluster configuration to a staging MSK cluster, apply changes, run load tests with production-representative traffic patterns, and monitor CloudWatch metrics (CPU, disk, consumer lag) for 24-48 hours before applying changes to production.
How to Monitor and Manage MSK Costs at Scale
Enterprise MSK environments run dozens of clusters and authentication schemes across multiple AWS accounts, regions, and workloads (production data pipelines, development environments, staging clusters for testing). At this scale, manual cost management becomes impossible. Automatically monitoring usage, detecting anomalies, and continuous optimizing are required to maintain control.
CloudWatch Dashboards for MSK Cost and Performance Correlation
Build CloudWatch dashboards that correlate MSK costs with operational metrics:
- `CPUUser + CPUSystem` (per broker): Track whether CPU utilization is increasing over time, indicating code regression, data flow growth, or traffic increases
- `KafkaDataLogsDiskUsed` (per broker): Monitor disk storage space consumption trends to forecast when storage expansion will be needed
- `BytesInPerSec` and `BytesOutPerSec` (per cluster): Track message throughput to identify anomalies (unexpected traffic spikes indicating misconfigured event triggers)
- `FetchConsumerTotalTimeMs` (p99 per consumer group): Monitor consumer lag to ensure optimization changes don't degrade consumer performance
- `UnderReplicatedPartitions` and `OfflinePartitionsCount`: Track partition health to detect broker failures or rebalancing issues
Set CloudWatch alarms for cost-relevant conditions: broker CPU spike >80% (potential need to scale up), disk utilization >85% (risk of running out of space), HeapMemoryAfterGC >60% (memory pressure), BytesInPerSec spike >2× normal baseline (potential cost anomaly from runaway producers).
AWS Cost Explorer + Tag-Based Cost Attribution
Enable cost allocation tags on all MSK clusters with `Environment` (dev/staging/prod), `Team`, `Project`, `Owner`, `CostCenter` to enable showback reports. Cost allocation tags make teams accountable for their MSK spend and surface optimization opportunities by team/project. Use AWS Tag Editor to apply tags retroactively to existing MSK clusters.
Query Cost Explorer API daily to track per-cluster MSK costs. Generate custom reports showing: cost per cluster per day, cost per environment (dev vs staging vs prod), cost per team (using tag filters), broker cost vs storage cost breakdown. Anomaly detection: flag any cluster whose daily cost increases >30% week-over-week as requiring investigation.
Automated Right-Sizing Recommendations
Build scripts that query CloudWatch metrics for all MSK clusters across accounts, calculate average and p95 CPU utilization per broker over 30 days, identify brokers with average CPU <40% and p95 <60% as over-provisioned, and generate right-sizing recommendations (current instance type → recommended smaller instance type, estimated monthly savings). Organizations running 20+ MSK clusters can automate this analysis monthly to surface $5,000-$15,000 in optimization opportunities.
Quarterly Storage Audits
Schedule quarterly reviews of MSK storage consumption: query each topic's retention policy, compare retention windows to actual consumer replay patterns (how far back do consumers typically rewind?), identify topics with retention >>consumer needs (e.g., 30-day retention when consumers only replay past 48 hours), calculate storage savings from reducing retention to align with workload requirements. Storage audits typically surface 20-40% storage cost reduction opportunities by eliminating unnecessary retention.
Multi-Account Cost Governance
Enterprise MSK deployments span dozens of AWS accounts by business unit or environment. Centralized governance requires:
- AWS Organizations consolidated billing to aggregate MSK spend across accounts
- Service Control Policies (SCPs) to enforce guardrails: require cost allocation tags on all MSK clusters, restrict kafka.m5.24xlarge and larger instances in dev/staging accounts (prevent accidental high-cost testing), prevent MSK Serverless in production accounts if provisioned clusters are standard practice
- Budget alerts at account/tag/cluster level to flag unusual spending (e.g., new $2,000/month MSK cluster created in dev account without approval)
- AWS Cost Anomaly Detection that automatically flags unexpected MSK cost increases (e.g., cluster broker count doubled due to auto-scaling misconfiguration) within 24 hours.
How nOps Automates AWS Cost Optimization
At scale, AWS cost optimization requires continuous operational work. This is precisely the problem nOps is built to solve. It ingests your usage from AWS and continuously optimizes costs on your behalf.
- Continuous, laddered rebalancing. nOps automatically manages commitments to maximize your savings and flexibility. Savings are often 20% higher than competitors.
- Full visibility. Get cost allocation, reporting, forecasting, anomaly detection, and the other visibility you need on your AWS, Azure, GCP, AI, SaaS and Kubernetes cost in a single pane of glass.
- Savings-first, fully aligned. nOps charges a percentage of the savings it generates. If we don’t save you money, you don’t pay.
Curious how optimized you are on AWS? A 30-minute free savings analysis shows you your current Effective Savings Rate and where the opportunities are. Setup is 5 minutes with no agents or infra changes needed.
nOps manages $4 billion in cloud spend for its customers and is rated 5 stars on G2.
FAQ
Let's dive into a few frequently asked questions about how to optimize usage patterns, streaming data and underlying infrastructure for your Amazon MSK clusters.
Q: Should I use MSK Provisioned or MSK Serverless for cost optimization?
MSK Serverless suits unpredictable workloads with sporadic traffic (development clusters, event-driven pipelines with variable activity). MSK Serverless pricing ($0.75/hour cluster + $0.0015/hour per partition + $0.10/GB in + $0.05/GB out) eliminates idle broker costs when traffic drops to zero, but for 24/7 production predictable workloads processing consistent message volumes, provisioned clusters typically deliver 30-50% lower total cost. The break-even point is approximately 50-60% sustained utilization — above this threshold, provisioned clusters are cheaper; below it, serverless wins.
Q: How much can I save by migrating to Graviton instances?
Graviton (ARM-based kafka.m7g instance types) delivers up to 24% lower cost and 29% higher throughput compared to x86-based kafka.m5 instances. A 9-broker cluster using kafka.m5.xlarge (~$1,130/month broker compute) migrated to kafka.m7g.xlarge saves ~$271/month (~$3,252 annually). Most Kafka workloads require zero code changes to migrate because Kafka is Java-based, but test in staging to validate performance and compatibility before production migration.
Q: What's the fastest way to identify my most expensive MSK clusters?
Use AWS Cost Explorer filtered by service (Amazon MSK), grouped by resource (cluster) and usage type. Sort by cost descending to identify top 10-20 highest-spend clusters. Then query CloudWatch for those clusters' CPU utilization, disk usage, and throughput metrics to determine whether optimization should focus on right-sizing brokers (high cost, low CPU utilization), reducing storage retention (high storage cost, low replay activity), or minimizing cross-AZ traffic (high data transfer cost).
Q: How do I know if my MSK brokers are right-sized?
Monitor `CPUUser` and `CPUSystem` CloudWatch metrics for each broker over 30 days. AWS recommends maintaining average CPU utilization (CPUUser + CPUSystem) below 60% to ensure headroom for broker failures and rolling upgrades. If average CPU <40% and p95 <60%, the broker is over-provisioned — downsize to the next smaller instance type. If average CPU >65% or p95 >80%, the broker is under-provisioned — scale up to prevent performance degradation during maintenance windows.
Q: Can I reduce replication factor to save storage costs?
AWS strongly recommends replication factor ≥3 for production MSK clusters to ensure high availability during broker failures and rolling maintenance. Replication factor = 1 causes offline partitions during upgrades; replication factor = 2 risks data loss if two brokers fail simultaneously. The cost of replication (3× storage, higher cross-AZ traffic) is an operational insurance policy against message loss. For non-production environments (dev/staging), replication factor = 2 can reduce storage costs by ~33%, but production clusters should maintain RF=3.
