RFC for Telemetry data collection

[Deepak Kodihalli]

On 07/09/17 8:50 pm, tomjose wrote:
> Hello,
> 
> I am working on the issue 
> (https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openbmc_openbmc_issues_1957&d=DwICaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=LzkOghL3x_3V_EkHiUQJUJ0xrq_s3_wwfssFT35AQXw&m=Gk0KnGVKy2iC82jVgpSqjzR2K_EYRlsFBqs34EjyKys&s=nQguk1LTD_q7dPNEn6dV2p4vsKaMu1vYZs5_Vh0BiNc&e= 
> ) to design a telemetry application for the OpenBMC. I would be 
> explaining a rough idea of how we plan to go about. Please share your 
> thoughts and feedback on this proposal. This issue would depend on the 
> design evolving out of following issues, since this app would utilize 
> the capabilities provided. 
> (https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openbmc_openbmc_issues_1856&d=DwICaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=LzkOghL3x_3V_EkHiUQJUJ0xrq_s3_wwfssFT35AQXw&m=Gk0KnGVKy2iC82jVgpSqjzR2K_EYRlsFBqs34EjyKys&s=2B_nLYU03S0QgMnCrMr8YawangOxRXmXGBqPF9593DY&e= 
> , 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openbmc_openbmc_issues_2102&d=DwICaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=LzkOghL3x_3V_EkHiUQJUJ0xrq_s3_wwfssFT35AQXw&m=Gk0KnGVKy2iC82jVgpSqjzR2K_EYRlsFBqs34EjyKys&s=U6M9vpoDDmNbTJJH5I6M6lPBGFzS1nuqYEEGwXjAviY&e= 
> ).
> 
> Summary of the requirements that we came across relevant to this 
> discussion.
> 
> 
> 1) BMC telemetry data (example VRM rail voltages) where the data is 
> collected at different rates depending on the data and aggregated by the 
> BMC app  (minimum, maximum
>      and average). Based on the collection timing request(frequency) the 
> metrics are logged, so that the user can fetch it for analytics.
> 
> 2)  Users should be able to set thresholds for the temperature limits, 
> and receive alerts. This would allow user to plan the cooling needs.
> 
> 3)  BMC would act as route for the OCC metrics to be send to the user. 
> The OCC would send down telemetric data to the BMC and BMC should figure 
> out a way to
>       alert the user to consume this data.
> 
> 
> We would keep the focus of the discussion on the requirement no 1.
> This proposal presupposes that all the resources( example VRM rail 
> voltages, ambient temperature) that the telemetry app is interested in, 
> should be populated as dbus objects, which can
> be queried to read the instantaneous values. phosphor-hwmon application 
> exposes many of the interested resources.
> 
> The idea is to have a yaml based approach, where the policy of the 
> telemetry app will be expressed. The application would be able to 
> consume the yaml and initiate the telemetry
> data collection. The yaml would express the following:
> 
> a) Dbus Info (object, interface, property) associated with the resource.
> b) Units associated with the value (celsius) and the associated scaling 
> factor).
> c) Granularity - the time between two measures.
> d) Aggregation methods - min,max,avg..etc.
> e) Logging policy - frequency for creating an event and alerting the user.
> 
> The application would operate based on the policy and log the telemetry 
> data. The details of logging would evolve as we progress on the related 
> issue.
> 
> Regards,
> Tom

Hi,

I'd like to bump this topic and add some more details. I'd like to 
discuss design proposals/directions for a couple things :

1) A short/mid term proposal for telemetry requirements specific to IBM 
labs (which need to be implemented in a relatively short span of time, 
so there may not be the bandwidth to write an entirely new application 
not based on D-Bus or the OpenBMC REST API).
2) Industry standard methods for storing and retrieving telemetry data - 
thoughts on how to get here.



1) Telemetry requirements specific to IBM labs
Here are the requirements and a design proposal.

a) Instantaneous readings, such as temperatures, currents, errors, 
events etc. Let's call this Layer 0.

Proposal:
- The D-Bus model is the source for instantaneous readings. This means 
there would be D-Bus objects representing this data, and hence an 
OpenBMC REST API around it.
- These D-Bus objects would not necessarily implement the same D-Bus 
interfaces.
- Interested clients can read these D-Bus objects via the OpenBMC REST API.
- If clients are interested in being notified about "changes" to the 
readings, that's possible via the existing event notification over 
WebSockets mechanism.


b) Instantaneous aggregations - this would mostly apply to, but may not 
be limited to, readings such as temperatures and currents. Let's call 
this Layer 1. This basically is to solve, for eg, "what is the 
min/max/average over the last X seconds?". We have a requirement to do 
such aggregations on the BMC.

Proposal:
- Aggregations are represented as D-Bus objects, created by a telemetry 
app. For eg if we need to know the min/max/avg ambient temp for the last 
5 minutes, and say the the ambient temp is usually at temps/ambient, the 
aggregation could be at temps/aggregations/ambient.
- Implement D-Bus interfaces to denote aggregations, for eg the 
temps/aggregation/ambient object could implement a D-Bus interface 
describing min/max/avg properties.
- Aggregation objects will have the values as described in the D-Bus 
interface (such as min/max/avg), and a timestamp, as properties.
- Enable a config (eg JSON) to let the telemetry app know things like : 
What (supported) aggregations should be performed (min/max/avg)? What 
D-Bus objects should be aggregated? How frequently should they be 
aggregated? What should be the paths of the aggregations? Potentially 
add a REST API to allow changing the (JSON) config at run-time.
- It will be possible to read all aggregation objects, or aggregation 
objects of a specific type via one REST call.


c) Historical aggregations or snapshot. Let's call this Layer 2. This is 
to solve, for eg, "Need a reading corresponding to every X minutes in a 
period of Y hours". This can be a snapshot of Layer 1 or Layer 0 D-Bus 
objects. We have a requirement to store this snapshot on the BMC.

Proposal:
- The snapshot will be represented as a set of D-Bus objects. For eg if 
one needs an hourly reading for a period of 24 hours, the objects could 
be at temps/aggregations/ambient/per-hour/{1..24}.
- Enable a config to let a telemetry app to know things like : What 
D-Bus objects should I keep a history of? What is the duration of the 
snapshot? At what frequency should entries be added into the snapshot? 
Once the snapshot is full, should the entries roll, or should we 
restart? Potentially add a REST API to allow changing the (JSON) config 
at run-time.
- The historical aggregations can be read via one REST call. It should 
be one D-Bus call as well most likely for the REST server, if there's an 
object manager at temps/aggregations/ambient/per-hour for eg.
- These objects in the snapshot will implement the same interfaces as 
Layer 1 objects, so they will have the same properties (eg min/max/avg, 
timestamp).


d) Some notes
- With the proposal above, the API to retrieve the telemetry data is via 
the current OpenBMC REST API, so it may not readily work with telemetry 
tools relying on industry-standard API (see point 2 below), but it seems 
to be the feasible option to rely on to implement IBM's requirements in 
the expected timelines.
- Layer 1 and Layer 2 telemetry apps would be different processes, and 
can function independent of each other.



2) Industry standard methods for storing and retrieving telemetry data

- With the proposal above, the instantaneous readings are D-Bus objects, 
the instantaneous and historical aggregations are D-Bus objects as well. 
The API is the OpenBMC REST API.
- Typically, aggregations may not have to happen on the BMC, in which 
case one can turn off layers 1 and 2.
- This is regarding how the telemetry data is presented, and how we'd 
eventually not use the current OpenBMC REST API in production. I've 
heard (mostly from people on the To: list) of the following 
industry-standard ways to represent/retrieve telemetry data. This would 
mean transforming layer 0 D-Bus objects into these :
	- Via Redfish (events) API
	- Via IPMI events/PEF
	- Via SNMP traps
	- Via an sqlite db, and have something like Logstash parse it
	- Others?
Discussions are already happening regarding Redfish, so telemetry could 
be one aspect to consider as well.
- Aggregations could be done on the BMC with collectd. I need to look at 
this in detail. Aggregations could be stored in an RRD format. Need to 
look at this in detail as well. These are as opposed to a D-Bus model of 
aggregations. Thoughts on this? For eg, would this be much less work 
both for the BMC and the telemetry data users than the proposed D-Bus 
model, but at the same time can address the requirements I've mentioned? 
Do we know what are the commonly used client tools for processing 
telemetry data, and how they expect the data to be presented?



Thanks,
Deepak