Monitoring the Untestable – A White Paper from APIContext

APImetrics has just published Monitoring the Untestable, a new white paper that focuses on the Catch-22* at the heart of PSD2 and open banking: if you don’t actively monitor your production environment with synthetic transactions, you have no idea whether your bank’s APIs are truly compliant at any given moment. But banks are often reluctant (with good reason) to allow testing in the live environment.
As we’ve said, to manage your APIs, you must monitor them. But if you are only passively monitoring your APIs, you are not monitoring them – which means you are not managing them. So, it’s important to actively monitor both your test environment and your production environment.
For massive, complex, heterogeneous organizations like banks, sandboxes can never fully replicate the behavior of the live system that real users encounter. You need to use an active monitoring system like APImetrics to make transactions with dummy accounts against the API exposed in the production environment.
Active monitoring also naturally supports negative testing and fuzzing. Only properly authorized users should have access to the dummy accounts, so by setting up test calls that don’t have the proper authorization, you know there’s a serious problem if they pass. You simply can’t do that without active monitoring.
With APImetrics, it’s easy to set up an alert if a 200 is returned when it shouldn’t be. Fuzzing is also important as real users will throw all sorts of weird things at your APIs, both accidentally and deliberately, and doing it yourself, you can see exactly how the APIs really respond to everything they get hit by.
So there is a solution to this Catch-22. If you do it properly and carefully, you can test the APIs in your live environment. And you really should.
You can get a copy of the white paper by filling in the form below.  If there is anything we can to help you actively monitor the performance and quality of the APIs you care about – and understand how they are really behaving from the end user perspective – please reach out to us. We’d be more than happy to discuss how you can monitor the untestable.
***
*There’s an interesting question about whether this is technically a Catch-22. In Heller’s original novel, the dilemma is that you have to be sane to be on the crew of a WWII USAAF bomber mission. But you’d have to be insane to participate on the crew of a bomber mission. Therefore the only sane response is to declare to the military authorities that you are insane, thereby by proving that you are sane and thus qualified to be on a bomber crew. A more succinct example might the Equity paradox. In order to get a part as an actor, you must have an Equity card. in order to get an Equity card, you must have a part as an actor. In the case of API testing, banks are required, for operational and regulatory reasons, to test their production APIs, but for security and other operational reasons may be prevented from doing so. This isn’t quite the classic Catch-22, but it is an example of a bureaucratic impasse: something compulsory is forbidden. But Heller in Catch-22 applies the eponymous term more loosely to other examples of Kafkaesque, nightmarish military and administrative logic, so it’s not inappropriate to use it here. See the Wikipedia article on the subject for more on this fascinating subject.