Swampd PID-driven Fan Controller

[Patrick Venture]

On Tue, Jun 27, 2017 at 3:34 PM, Brad Bishop
<bradleyb at fuzziesquirrel.com> wrote:
> Hi Patrick
>
>> On Jun 27, 2017, at 12:51 PM, Patrick Venture <venture at google.com> wrote:
>>
>> Per the requirements to have a fan controller for the quanta-q71l that
>> runs within the OpenBMC framework+, I developed the swampd
>> application.  It was designed to leverage sdbusplus and other OpenBMC
>> libraries and coding standards.
>
> Sweet!  First let me say thanks for your willingness to share your
> application.

No problem.
>
>>
>> It supports:
>> * build-time yaml descriptions of the sensors and PID loops
>> * run-time configuration of the above
>> * run-time fan speed set-point control
>> * IPMI controlled manual mode
>> * passively and actively read sensors
>> * sensors that are from the host
>> * any variation on sysfs IO that is configured.
>
> Sounds like a solid feature set.  Can you speak to what level of support
> exists for things like hot plugging and/or degraded mode operation?

Right now it's at ~80*% done, so I'm still working with people here to
design the best practices for what they want in failure modes.  It was
designed with input from a few engineers who have ties to its primary
use cases.  And there is a an expectation that things with this will
behave similarly to how the predecessors behaved.  So that drove a
fair amount, but there are certain failure modes this doesn't yet
handle that it needs to.  If the sensors stop reporting information,
for instance, it goes into a pre-configured fail-safe mode.


*Functionally complete but still has lots of room for improvement.

>
>>
>> The design document is too large to really paste in this email, so
>> I'll summarize.
>>
>> Each sensor is created based on the configuration provided.  Each PID
>> loop is created per zone.  Each zone lives on a separate thread (could
>> be merged into one thread that iterates through zones).  The zone
>> loops every 0.1s to steer the fans closer to their goal while every
>> second it adjusts the goal.
>>
>> Some things I know OpenBMC won't like: it's multi-threaded.  It wasn’t
>
> True, we are trying to avoid multi-threaded applications, but if the
> problem being solved by threads can’t reasonably be solved some other
> way that seems fine.
>
>> built into the current fan-presence package.
>
> I don’t have any issue with this.
>
>>
>> I'm at a crossroads with this project because I'd like to make it
>> generally available.  Should I try to submit it for review on this
>> list, if so, how?  Or should I push it to Google's github projects
>> first?
>
> I think this depends on a couple factors.
>
> As you know, another team is also developing a fan control algorithm.
> If the two algorithms are similar I don’t see any point in hosting two
> applications that do the same thing.  It would just make it more confusing
> for others coming to OpenBMC to pick one or the other for their platform.
>
> Now, if each provides their own unique, fundamentally different fan control
> strategy - each with a different set of pros/cons - that is exactly what we
> had anticipated and bringing it in seems like a great idea.

It does look like there has been some progress on the
phosphor-fan-presence/control.   Although I'd have to hack it anyway
to support Control.FanPwm because I lost that community argument.

There are definitely some differences.  However, some things can be
done in separate daemons -- as has been suggested.  As an alternative
to separate threads.

>
> For the second factor, I’m going to be a little frank and critical, because
> my point is fundamental to the project goals and vision.  Please don’t take
> this personally.
>
> This is what is known in OSS circles as a code dump.  It isn’t OSS development.
> OSS development happens...in the open.  The first three bullet points here touch
> on this:
>
> https://people.redhat.com/rjones/how-to-supply-code-to-open-source-projects/
>
> In an OSS development model, the entire community is given the opportunity
> to collaborate and shape the result of the end product, _as_it_is_developed_
> through feedback and contributions.
>
> I’m no zealot.  I completely understand that many times factors like product
> delivery schedules, etc get in the way of this model, and thats fine.  Thats
> why most products using OSS don’t ship with vanilla versions of the project
> they are based on.
>
> But the other fan control algorithm _is_ being developed in the open.  A lot of
> folks have invested many many hours of peer review in it.
>
> So the question becomes, are you prepared to submit your application to that
> same level of peer review and reciprocate by investing the time required to
> address the feedback and get it to a point where the community feels comfortable
> assuming maintenance responsibility for it?
>

I would have liked to develop this in the open, but as you touched on
it, schedules.  My platform is under a compressed schedule.  Under any
situation, I'll be responsible for maintaining this code base.  And
expanding it to support other platforms.  As implementing it wasn't
done in a vacuum, I feel fairly confident that outside contributions
at least in furthering the design to make it simpler or more modular
would be warranted.

Obviously, it was designed around very specific execution
requirements, which drove my adding an interval to phosphor-hwmon and
quite a few downstream hacks.  That said, those specific execution
requirements do have an impact on the ability to change the design
from input.  If the design suggestions or ideas violate the terms of
service that we're providing the data centers -- it can be forked,
effectively.

> Thx - brad
>
>>
>> +As well as other requirements.
>>
>> Thanks,
>> Patrick