You may have seen some articles doing the rounds about the 13 key API metrics that teams should track. But going back to the challenge we have with how we describe API consumers, the needs of the ‘team’ in this context are vague. The API metrics performance section lists uptime (or does it?), CPU usage, and Memory usage.
Then they talk about adoption and use metrics – which are important. But why the narrow technical view of API performance? Why ignore the fact that as an API Dependent (a consumer, a solution business owner, a customer success provider, sourcing, and MANY others), if your API performs badly or doesn’t work the way engineering says, you don’t stand a chance with the other metrics?
Some of the API metrics items are spot on – you need to know about adoption, use rates, time to first Hello World (in the real world if you please, and not in the Postman client in the dev environment). But API performance is more than about uptime and uptime and availability is more complex a function than just the rate of which you see failures.
1. API Metrics: Pass Rate
Pass rate is one of the crudest API metrics we have in our toolbox. It’s valued because of things like the dreaded 5 9 API uptime metrics of old (99.999% availability). There’s a handy chart of what that means here, and we use it as one of our API performance metrics.
Basically put, a 99.x% pass rate means you’re down for over 3 days a year, versus 5 minutes with 99.999%
BUT! That’s why we invented CASC scoring. Because you want to know how often something breaks – but you also need to verify that you’re measuring failures in the right way.
APIs can fail – but look like they’re working!
2. API Metrics: Latency
API latency is the “how long is a piece of string” of measurement. You should be measuring and worrying about latency. If your engineering department can’t answer your latency questions, then come to us immediately.
Or if you’re on https://api.expert, you can check for yourself.
- Regional, national, and cloud performance – do you have a problem with a specific cloud? It’s no use only being fastest on the IBM cloud when most of the world is on AWS. Likewise, being perfect in the AWS East US region, but nearly unusable on Azure or a region where most of your users are, is a problem.
- Network problems – for example, is the DNS lookup system your engineering group is using working well? If you can’t see, on-demand, the DNS resolution times from every potential area on every cloud, speak to us about upgrading your API metrics monitoring.
3. API Metrics: Consistency
Is your response consistent? Do you know your percentiles?
Looking at averages often tells you what you want to hear. “Our average is only half a second from anywhere on earth!”
But what if your 95th percentile – the worst 5% of calls – takes longer? Maybe something like 11 seconds! If you have a million calls a day, that’s 50,000 calls – or nearly a WEEK of lost time by users a day.
You do the math.
4. API Metrics: Availability (a systems approach)
Overall, APIs mostly work. But what if the API with problems handles the authentication? In a complex secure framework, this can be critical – if you can’t get or refresh tokens you may as well be down everywhere.
Let’s say you have users authenticate to a bank or healthcare provider through your service, but your secure authentication is failing all or some of the time. You can end up with 100% passing rates on most of your API metrics, and only a small problem showing in your authentication.
We struggle with showing this ourselves, but API metrics for availability should be looked at from a system approach and NOT an API approach.
Monitor the system and understand that if a piece of the critical path is down, everything is down.
5. API Metrics: Quality
There is a sense in the industry that APIs exist as quanta – essentially, little units of on-or-off. They’re not. They’re complex and fuzzy.
You can have an API passing but not returning what you want. An API can be passing but returning an incorrect schema. You can have security failing unsafe or too safe.
Do you have a general sense of the quality of what you offer?
6. API Metrics: Where do you rank?
If you have that general sense, do you know where you rank against providers of similar services? Do you understand why some services have a better reputation than others?
Check out api.expert to see where the industry ranks are, and then compare your performance.
7. Can you make an SLA out of your API metrics data?
Service Level Agreements (SLAs) for APIs are hard to set and hard to measure. But that doesn’t mean that you shouldn’t.
Ensure you have tools measuring systems from the point of view of your end-users and relate that to the points made here. Decide what should and shouldn’t be in your SLA, define the measurement criteria, and be transparent.
All the technical and business API metrics in the world can’t help you if you don’t know how well your services are doing or where the problems and bottlenecks are. Monitoring your system like your users and, for want of a better word, engage in negative monitoring.
Don’t monitor to verify what you’d like to know – monitor to catch the things you’d rather you didn’t.
Conclusion
Measure. Measure not what is easy but what is hard when it comes to your API metrics. Measure from the point of view of your users and consumers and come to terms with the fact that the needs of the API Dependents should not be disregarded over the desires of engineering to measure what they have tools for.
By all means continue to measure at the source, look at your traffic and the data flowing through your servers. But don’t miss out on trying to understand what the world looks like to your users!
The tools and technology exist to make this easy; use them!
Now Available for All APImetrics Accounts
If you are new to APImetrics, sign up for a free trial so see how they could work for you with our free API set.
If you have any questions, please don’t hesitate to reach out.
If you’re new to API monitoring, you can learn about our comprehensive features for testing, monitoring, security and performance and sign up for a free trial account.
