AWS places soft limits on the rate of requests made to the CloudWatch API. Unfortunately these limits don't scale with the number of resources in use, so if you have a large number of instances/resources to monitor, you will most probably run into the limit.
We automatically adapt to the throttling and are able to continue retrieving your data, but it does delay how quickly we can present data to you in our interface.
To ensure seamless operation we recommend you contact your AWS representative or file a support ticket to have your CloudWatch API limits increased (which are set pretty low by them initially). As a rule of thumb, if you don't foresee a significant increase in AWS resources you should ask them to double the limits. If you think your resource use is going to grow considerably, ask them for a 4..5x increase.
Generally it should be sufficient to send a simple request. On some occasions we have heard that an Amazon representative will come back with some questions. We're going to work with our AWS Cloudwatch contacts to simplify the process but in case you receive questions, here are some answers:
We don't record which exact API calls are throttled, however the API calls we make are ListMetrics and GetMetricStatistics. We make far more GetMetricStatistics requests than any other, so its most likely that.
That's a difficult one to answer as it is different for each account and depends on the usage level. Rather than answering with specific requests/sec we recommend asking for a 2x or 3x increase. If they don't accept that, send us an email to firstname.lastname@example.org so that we can check your rate and help you craft a more specific answer.
The requests are staggered randomly amongst all our CloudWatch integration customers.
We do not make any PUT requests.
We use the latest version of the API - in fact we use the AWS Ruby SDK that Amazon provides.
Hopefully these answers will get you what you need. When the limit increase has been completed let us know at email@example.com and we can check the results on our side.