BREAKING NEWS: Contxt and APImetrics are merging to create a solution tailored for API product owners.Read Now >

Test Frequency – Deciding on your test frequency

APImetrics API Governance Monitoring Testing

One of the things that we are clear about is APImetrics is not a load nor stress testing platform.  We’re focused on providing a close indication of the likely quality of the developer experience on REST based APIs.  Consequently, we actually don’t test all that often, our default for our public tests is once every 15 minutes.  One of the reasons for that has been the behavior of some APIs with cached data in app queries.  Testing every 15 minutes in this case actually provides misleading good performance data.

However, this isn’t true for all APIs.  We’re also testing the Intel Cloud Services Platform APIs and finding almost the inverse.  Testing every 15 minutes yields some errors but testing every 6 minutes yields a lot more.  By increasing the test frequency of the testing, we’ve actually seen that the reliability of the API is a full 2% points lower over the course of a typical day, than the 15 minute test interval suggests.

Our proposal for this is to take a mixed approach when you set up the tests.  Set off different trials and see if you can see the impact of caching of results.  This will typically manifest itself on the latency graphs as a ‘saw-tooth’ pattern, or regular increases in latency as the cache clears.  Secondly, if you own the API platform being tests, set up several App accounts with their own ClientID and Secret codes and run the same test with different user credentials and use the test ‘offset’ feature to stagger their run times.

The essence of what we finding is a fascinating range of behavior from standard REST based APIs and a need for providers and developers alike to pay close attention to their test regime and act on the data.

Request A Demo

Find A Slot To Get A Demo Or Speak To One Of Our Support Specialists

Ready To Start Monitoring?

Want to learn more? Check out our technical knowledge base, or our sector by sector data, or even our starters guide to the API economy. So sign up immediately, without a credit card and be running your first API call in minutes.

Related Posts

Join Us Now!

Join the 100s of companies relying on APImetrics.