I have been working on a first white paper version of an idea/project that came up based on a proof-of-concept we made recently in order to proxy, crawl, and store certain data flowing through an API (including both requests and responses).
We decided to use Tyk Proxy in order to “intercept” our API. Tyk is open-source and has an excellent middleware “plug-in” system that allows to easily extend its basic features. This proof of concept allowed us to analyze quite complex XML messages and extract data (that we called measures) out of it. The exact data to extract and the way to store it is taken from XML messages using XPath addressing and configured in elements (that we called crawls). Each crawl takes some data from a single request and/or response and stores it in a measure. We made it this simple for the PoC.
The data extracted from the API “traffic” is stored in an InfluxDB instance, a statistical-oriented key-value DB engine very popular for this kind of use.
Last step was to visualize this measures in the most simplistic (less coding) way posible, which leaded us to Grafana, a wonderful tool to create dashboards containing charts pulling data from several popular statistical DB engines at a minimum time/development cost.
Based on this experience we decided to take this as an starting point to create a new approach to make data-analytics from an APIs using what we called an API Analytics Proxy.
If you are interested in the topic take a look at the white paper, and of course feel free to share feedback/ideas/questions.