What Caused the Network Cost Spike?
Sounds like an easy question right?
If you are responsible for managing cloud spend, AWS NAT Gateways are one of the classic AWS gotchas. What causes a networking spike is not an easy question to answer. NAT gateway cost can run up your AWS bill quickly without you realizing it. The challenge with bandwidth cost is that it is a black box. You can’t solve it if you can’t see it.
We experienced bandwidth-related problems firsthand. We process hundreds of millions of AWS spend. We recently started using Databricks for our data warehouse.
Something interesting happened at the same time we deployed the data warehouse. Our networking cost went up by $1,000 a day. We thought it was related to the new data we were processing.
To our surprise, the increase didn’t come from the Data bricks. Instead, it came from a misconfiguration in the NAT gateway. In this blog post, we’ll walk you through how we used nOps to find NAT misconfiguration.
How to Get Insight into NAT Gateway to Reduce AWS Costs?
The best way to get an understanding of your bandwidth costs is to look at the traffic flow. You can do that by enabling VPC logs. nOps ingests VPC logs, so we can show you the traffic and how much you’re paying for that bandwidth.
When we looked at our bandwidth cost, we saw traffic going from one private IP address to another private IP address. That was already a red flag, because normally you use NAT gateway to route traffic to a public subnet. We quickly found the source of our traffic issues. We came to realize that we are not the only ones who have this problem. Once you shed light on it, you’re able to find the true cost of the bandwidth.
Visualizing Routing Table with nOps VCP Endpoint Service
If you haven’t figured it out yet, a potential way to avoid NAT Gateways (or at least reduce their costs) is to utilize VPC endpoints. Some AWS services support VPC endpoints, and using them would be cheaper than using a NAT gateway. But it becomes incredibly difficult to track different routing tables. nOps automatically creates your routing table diagram so you can easily spot misconfigurations in your environment.
Once we fixed our routing table, our cost dropped by $12,000 monthly. Here is graph showing the drop.
We work with many customers with complex environments. We came to the realization that many of the cost issues are related to misconfiguration. Sign up for nOps today to review your environment.
JT Giri is a father, husband, and entrepreneur. He focuses on building a culture that allows teams to reach their highest potential. He enjoys HIIT workouts, learning, and meditating.