Fan PWM monitoring and control
Matthew Barth
msbarth at linux.ibm.com
Fri Oct 4 06:59:16 AEST 2019
On 10/1/19 3:26 AM, rgrs wrote:
> Hi Matt,
>
> Please can you help me with mapping values in events.yaml?
> (https://github.com/openbmc/phosphor-fan-presence/blob/master/control/example/events.yaml)
>
> How are these values derived?
> How do I make sure, that this is valid for my platform?
> Please can you point to documentation regarding this, i am not able to find from github search.
Unfortunately in this repo's development there was not alot of time
given to get a fully functional and flexible solution in place for
delivery on the system using this as its fan control solution.
Documentation is limited to the example yaml files and the doxygen
located within the source. Enhancing this documentation is on my list of
todo's when I get the time.
>
> # actions:
> # - name: set_floor_from_average_sensor_value
> # map:
> # value:
> # - 27000: 3500
> # - 32000: 4600
> # - 37000: 5200
> # - 40000: 5800
> # type: std::map<int64_t, uint64_t>
> # - name: set_ceiling_from_average_sensor_value
> # map:
> # value:
> # - 25000: 7200
> # - 27000: 10500
> # type: std::map<int64_t, uint64_t>
>
These values are derived by performing thermal characterization. In this
case, it was thermally determined that the values for setting the fan
floor speed (the lowest the fans are allowed) based on the given average
sensor values for the group of sensors defined on the event containing
this action. Similarly, a fan ceiling speed is put in place based on the
same group of sensors' average value so the fans are not allowed to
increase above that speed.
Validating thermal control parameters for any type of fan control needs
characterization done against the system's components(Proc, dimms, pci
cards, etc..) to enabled the correct fan control parameters.
Matt
>
>
>
> Thanks,
> Raj
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Thursday, September 26, 2019 8:19 PM, Matthew Barth <msbarth at linux.ibm.com> wrote:
>
>> On 9/25/19 1:45 AM, rgrs wrote:
>>
>>> Hi Matt,
>>>
>>>> Is the `pwm[#]_enable` sysfs file set to a '1' at any point after the AC cycle (i.e. after a poweron?)?
>>>
>>> No
>>>
>>>> If you think the configs are set correctly, is the hwmon service for the pwm devices running after the AC cycle while at BMC standby?
>>>
>>> I'm not sure, I think phosphor-fan starts after chassis power-on.
>>
>> I was asking if the hwmon service is started at BMC standby. It sounds
>> like it still may be config related but I'm not able to determine where
>> it may be. If you believe that hwmon is not setting the pwm[#]_enable
>> file correctly, please debug and provide a fix if that's the case, it'd
>> be much appreciated!
>>
>> Matt
>>
>>> Thanks,
>>> Raj
>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>> On Tuesday, September 24, 2019 8:25 PM, Matthew Barth msbarth at linux.ibm.com wrote:
>>>
>>>> On 9/23/19 11:25 PM, rgrs wrote:
>>>>
>>>>> Thanks Matt :) It works now.
>>>>> I have one small issue on configuration of ADT7462.
>>>>> After AC power cycle, it defaults to "automatic" fan control.
>>>>> We need to change this "manual" so that phosphor-fan control register writes are not ignored by ADT7462.
>>>>> I came across https://github.com/openbmc/docs/blob/master/sensor-architecture.md
>>>>>
>>>>>> ENABLE
>>>>>> Will write a value to a pwmN_enable file on startup if present.
>>>>>> ENABLE_fan1 = 2 #Write a 2 to pwm1_enable
>>>>>
>>>>> But this doesn't seem to work. We need it to be initialized to '1'.
>>>>> (It always defaults to 2 after AC cycle)
>>>>
>>>> This sounds like the correct hwmon config to use to set it to a '1' upon
>>>> hwmon starting after an AC cycle. First ensure your hwmon config has
>>>> these `ENABLE_fan[#]` entries set to a corresponding `pwm[#]_enable`
>>>> file in sysfs.
>>>> Is the `pwm[#]_enable` sysfs file set to a '1' at any point after the AC
>>>> cycle (i.e. after a poweron?)? If you think the configs are set
>>>> correctly, is the hwmon service for the pwm devices running after the AC
>>>> cycle while at BMC standby?
>>>>
>>>>> Is host-power-on script a good place to "echo 1 > pwmN_enable"?
>>>>> (planning to add a script to initialize before host power on)
>>>>> Thanks,
>>>>> Raj
>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>> On Friday, September 20, 2019 9:49 PM, Matthew Barth msbarth at linux.ibm.com wrote:
>>>>>
>>>>>> On 9/20/19 4:55 AM, rgrs wrote:
>>>>>>
>>>>>>> Hi Matt,
>>>>>>> I'm able to control fan speed via busctl.
>>>>>>> root at saber:/sys/class/hwmon/hwmon0# busctl set-property xyz.openbmc_project.Hwmon-2429843755.Hwmon1 /xyz/openbmc_project/sensors/fan_tach/FAN0 xyz.openbmc_project.Control.FanPwm Target t '250'
>>>>>>> root at saber:/sys/class/hwmon/hwmon0# cat fan*_input
>>>>>>> 12498
>>>>>>> 0
>>>>>>
>>>>>> Great! that's progress.
>>>>>>
>>>>>>> I need a way to verify if the sensor associations are working properly that fans will ramp up when associated temperature sensor's value increases.
>>>>>>
>>>>>> Be sure that your fans.yaml contains the correct sensor name ("FAN0",
>>>>>> etc..) for each fan that contains the Target property on the FanPwm
>>>>>> interface.
>>>>>> You can use busctl to monitor the target pwm sensor changes based on
>>>>>> what you configured within events.yaml for speed increases. This would
>>>>>> show that fan control is requesting updated targets to hwmon. Then hwmon
>>>>>> would write these values out to the pwm sysfs file for your fan
>>>>>> controller device.
>>>>>>
>>>>>>> Thanks,
>>>>>>> Raj
>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>> On Friday, September 20, 2019 2:57 PM, rgrs rgrs at protonmail.com wrote:
>>>>>>>
>>>>>>>> Hi Matt,
>>>>>>>>
>>>>>>>>>> Not sure I follow what you mean by "no pwm writes", is there an
>>>>>>>>
>>>>>>>> application writing to the fan sensor's pwm interface target property?
>>>>>>>> I thought phoshor-fan control writes to pwm to increase fan speed based on temperature sensors mentioned in events.yaml
>>>>>>>> (recipes-phosphor/fans/phosphor-fan-control-events-config/events.yaml)
>>>>>>>> Is it not the case?
>>>>>>>> Thanks,
>>>>>>>> Raj
>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>> On Thursday, September 19, 2019 8:26 PM, Matthew Barth msbarth at linux.ibm.com wrote:
>>>>>>>>
>>>>>>>>> On 9/19/19 5:16 AM, rgrs wrote:
>>>>>>>>>
>>>>>>>>>> Hi Patrick, Lei,
>>>>>>>>>> Thank you, I made some changes according your comments
>>>>>>>>>>
>>>>>>>>>> - added Mutability::Read to fan yaml.
>>>>>>>>>> - I didn't see ipmi-sensor-inventory.yaml of romulus having fan sensors, so removed them from IPMI.
>>>>>>>>>>
>>>>>>>>>> Now,"busctl monitor" displays Fan tach reads, no pwm writes.
>>>>>>>>>
>>>>>>>>> Not sure I follow what you mean by "no pwm writes", is there an
>>>>>>>>> application writing to the fan sensor's pwm interface target property?
>>>>>>>>>
>>>>>>>>>> Is there any way to manually set PWM value and test this?
>>>>>>>>>> How do I verify PWM values are being written to ADT7462?
>>>>>>>>>
>>>>>>>>> If no other application are writing to the PWM sysfs entries, you should
>>>>>>>>> be able to `echo` a value to each fan's PWM sysfs file to alter the fan
>>>>>>>>> speeds.
>>>>>>>>>
>>>>>>>>>> Another observation is Inventory.Item seems to have invalid(stale) entries.
>>>>>>>>>
>>>>>>>>> I believe inventory is persisted by default so that could explain why
>>>>>>>>> these entries are still showing up. These are persisted under
>>>>>>>>> `/var/lib/phosphor-inventory-manager/..`.
>>>>>>>>>
>>>>>>>>>> IPMI sensor names appear in inventory even after removing from IPMI sensor yaml files (exported JSON in attachment).
>>>>>>>>>> Logs:
>>>>>>>>>> busctl monitor | grep -i fan
>>>>>>>>>>
>>>>>>>>>> ========================================================================================================================================================
>>>>>>>>>>
>>>>>>>>>> Monitoring bus message stream.
>>>>>>>>>> Sender=:1.45 Path=/xyz/openbmc_project/sensors/fan_tach/FAN0 Interface=org.freedesktop.DBus.Properties Member=PropertiesChanged
>>>>>>>>>> Sender=:1.45 Path=/xyz/openbmc_project/sensors/fan_tach/FAN1 Interface=org.freedesktop.DBus.Properties Member=PropertiesChanged
>>>>>>>>>> Sender=:1.45 Path=/xyz/openbmc_project/sensors/fan_tach/FAN2 Interface=org.freedesktop.DBus.Properties Member=PropertiesChanged
>>>>>>>>>> [...]
>>>>>>>>>> Sender=:1.45 Path=/xyz/openbmc_project/sensors/fan_tach/FAN1 Interface=org.freedesktop.DBus.Properties Member=PropertiesChanged
>>>>>>>>>> Sender=:1.45 Path=/xyz/openbmc_project/sensors/fan_tach/FAN3 Interface=org.freedesktop.DBus.Properties Member=PropertiesChanged
>>>>>>>>>> ^C
>>>>>>>>>> busctl get-property xyz.openbmc_project.Hwmon-2429843755.Hwmon1 /xyz/openbmc_project/sensors/fan_tach/FAN0 xyz.openbmc_project.Control.FanPwm Target
>>>>>>>>>>
>>>>>>>>>> =============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================
>>>>>>>>>>
>>>>>>>>>> t 0
>>>>>>>>>> busctl set-property xyz.openbmc_project.Hwmon-2429843755.Hwmon1 /xyz/openbmc_project/sensors/fan_tach/FAN0 xyz.openbmc_project.Control.FanPwm Target 8000
>>>>>>>>>>
>>>>>>>>>> ==============================================================================================================================================================
>>>>>>>>>
>>>>>>>>> This is not a valid call to `busctl set-property...`, its missing the
>>>>>>>>> property's signature type (refer to `busctl help`). Also, a target PWM
>>>>>>>>> of 8000 is invalid, PWM is 0 - 255.
>>>>>>>>>
>>>>>>>>>> Failed to create bus message: Invalid argument
>>>>>>>>>> busctl introspect --no-pager xyz.openbmc_project.Hwmon-2429843755.Hwmon1 /xyz/openbmc_project/sensors/fan_tach/FAN1
>>>>>>>>>>
>>>>>>>>>> ===================================================================================================================================================================
>>>>>>>>>>
>>>>>>>>>> NAME TYPE SIGNATURE RESULT/VALUE FLAGS
>>>>>>>>>> org.freedesktop.DBus.Introspectable interface - - -
>>>>>>>>>> .Introspect method - s -
>>>>>>>>>> org.freedesktop.DBus.Peer interface - - -
>>>>>>>>>> .GetMachineId method - s -
>>>>>>>>>> .Ping method - - -
>>>>>>>>>> org.freedesktop.DBus.Properties interface - - -
>>>>>>>>>> .Get method ss v -
>>>>>>>>>> .GetAll method s a{sv} -
>>>>>>>>>> .Set method ssv - -
>>>>>>>>>> .PropertiesChanged signal sa{sv}as - -
>>>>>>>>>> xyz.openbmc_project.Control.FanPwm interface - - -
>>>>>>>>>> .Target property t 0 emits-change writable
>>>>>>>>>> xyz.openbmc_project.Sensor.Value interface - - -
>>>>>>>>>> .MaxValue property x 0 emits-change writable
>>>>>>>>>> .MinValue property x 0 emits-change writable
>>>>>>>>>> .Scale property x 0 emits-change writable
>>>>>>>>>> .Unit property s "xyz.openbmc_project.Sensor.Value.Unit.… emits-change writable
>>>>>>>>>> .Value property x 2983 emits-change writable
>>>>>>>>>> xyz.openbmc_project.State.Decorator.OperationalStatus interface - - -
>>>>>>>>>> .Functional property b true emits-change writable
>>>>>>>>>> Thanks,
>>>>>>>>>> Raj
>>>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>>>> On Wednesday, September 18, 2019 9:35 PM, Patrick Venture venture at google.com wrote:
>>>>>>>>>>
>>>>>>>>>>> On Wed, Sep 18, 2019 at 2:30 AM rgrs rgrs at protonmail.com wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Patrick, Matt,
>>>>>>>>>>>> I tried instructions from this wiki: https://developer.ibm.com/linuxonpower/2019/01/07/how-to-port-openbmc/
>>>>>>>>>>>> There is no GPIO to detect presence of fans, so tach readings is used instead.
>>>>>>>>>>>> Here's what I did while trying to enable Fan control:
>>>>>>>>>>>>
>>>>>>>>>>>> 1. Added Fan tach sensor in hwmon (adt7462.conf) is in the form LABEL=FANx_RPM
>>>>>>>>>>>> 2. "ipmi-sensor-inventory.yaml" has below config:
>>>>>>>>>>>>
>>>>>>>>>>>> 60: &fan_tach
>>>>>>>>>>>> sensorType: 0x04
>>>>>>>>>>>> path: /xyz/openbmc_project/sensors/fan_tach/FAN0_RPM
>>>>>>>>>>>> sensorReadingType: 0x01
>>>>>>>>>>>> multiplierM: 100
>>>>>>>>>>>> offsetB: 0
>>>>>>>>>>>> bExp: 0
>>>>>>>>>>>> unit: xyz.openbmc_project.Sensor.Value.Unit.RPMS
>>>>>>>>>>>> scale: 0
>>>>>>>>>>>> rExp: 0
>>>>>>>>>>>> serviceInterface: org.freedesktop.DBus.Properties
>>>>>>>>>>>> readingType: readingData
>>>>>>>>>>>> interfaces:
>>>>>>>>>>>> xyz.openbmc_project.Sensor.Value:
>>>>>>>>>>>> Value:
>>>>>>>>>>>> Offsets:
>>>>>>>>>>>> 0xFF:
>>>>>>>>>>>> type: int64_t
>>>>>>>>>>>> and FAN PWM as
>>>>>>>>>>>> 89: &fanpwm
>>>>>>>>>>>> sensorType: 0x04
>>>>>>>>>>>> path: /xyz/openbmc_project/sensors/fan_tach/FAN0_RPM
>>>>>>>>>>>> sensorReadingType: 0x01
>>>>>>>>>>>> multiplierM: 1
>>>>>>>>>>>> offsetB: 0
>>>>>>>>>>>> bExp: 0
>>>>>>>>>>>> mutability: Mutability::Write
>>>>>>>>>>>
>>>>>>>>>>> You may need to add ::Read here too, I'm not sure anymore, but you can
>>>>>>>>>>> check the script that builds this:
>>>>>>>>>>> https://github.com/openbmc/phosphor-host-ipmid/blob/master/scripts/writesensor.mako.cpp#L69
>>>>>>>>>>> It'll default to read, but if you've only set write, I don't know.
>>>>>>>>>>>
>>>>>>>>>>>> serviceInterface: org.freedesktop.DBus.Properties
>>>>>>>>>>>> readingType: readingData
>>>>>>>>>>>> interfaces:
>>>>>>>>>>>> xyz.openbmc_project.Control.FanPwm:
>>>>>>>>>>>> Target:
>>>>>>>>>>>> Offsets:
>>>>>>>>>>>> 0xFF:
>>>>>>>>>>>> type: uint64_t
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 3. I copied meta-romulus/recipes-phosphor/fans to my platform and changed FAN names from "fanx" to "FANx_RPM" in yaml files.
>>>>>>>>>>>>
>>>>>>>>>>>> This is what get in busctl tree related to Fans:
>>>>>>>>>>>> (I expected to see FanPwm entries in busctl and I'm not sure why there are multiple entries in inventory).
>>>>>>>>>>>> Service xyz.openbmc_project.Hwmon-2429843755.Hwmon1:
>>>>>>>>>>>> `-/xyz`-/xyz/openbmc_project
>>>>>>>>>>>> `-/xyz/openbmc_project/sensors |-/xyz/openbmc_project/sensors/fan_tach | |-/xyz/openbmc_project/sensors/fan_tach/FAN0_RPM | |-/xyz/openbmc_project/sensors/fan_tach/FAN1_RPM | |-/xyz/openbmc_project/sensors/fan_tach/FAN2_RPM |`-/xyz/openbmc_project/sensors/fan_tach/FAN3_RPM
>>>>>>>>>>>> `-/xyz/openbmc_project/sensors/temperature`-/xyz/openbmc_project/sensors/temperature/BACKPLANE_TEMP
>>>>>>>>>>>
>>>>>>>>>>> You don't get a separate entry for the PWM from phosphor-hwmon.
>>>>>>>>>>> It's a separate interface on the same service and path.
>>>>>>>>>>> I recommend you introspect on the path and see what interfaces are
>>>>>>>>>>> available (as well as adding "| read" above):
>>>>>>>>>>> busctl introspect --no-pager
>>>>>>>>>>> xyz.openbmc_project.Hwmon-2429843755.Hwmon1
>>>>>>>>>>> /xyz/openbmc_project/sensors/fan_tach/FAN0_RPM
>>>>>>>>>>>
>>>>>>>>>>>> Service xyz.openbmc_project.Inventory.Manager:
>>>>>>>>>>>> `-/xyz`-/xyz/openbmc_project
>>>>>>>>>>>> `-/xyz/openbmc_project/inventory`-/xyz/openbmc_project/inventory/system
>>>>>>>>>>>> `-/xyz/openbmc_project/inventory/system/chassis`-/xyz/openbmc_project/inventory/system/chassis/motherboard
>>>>>>>>>>>> |-/xyz/openbmc_project/inventory/system/chassis/motherboard/FAN0
>>>>>>>>>>>> | `-/xyz/openbmc_project/inventory/system/chassis/motherboard/FAN0/FAN0 |-/xyz/openbmc_project/inventory/system/chassis/motherboard/FAN0_RPM |`-/xyz/openbmc_project/inventory/system/chassis/motherboard/FAN0_RPM/FAN0_RPM
>>>>>>>>>>>> |-/xyz/openbmc_project/inventory/system/chassis/motherboard/FAN1
>>>>>>>>>>>> | `-/xyz/openbmc_project/inventory/system/chassis/motherboard/FAN1/FAN1 |-/xyz/openbmc_project/inventory/system/chassis/motherboard/FAN1_RPM |`-/xyz/openbmc_project/inventory/system/chassis/motherboard/FAN1_RPM/FAN1_RPM
>>>>>>>>>>>> |-/xyz/openbmc_project/inventory/system/chassis/motherboard/FAN2
>>>>>>>>>>>> | `-/xyz/openbmc_project/inventory/system/chassis/motherboard/FAN2/FAN2 |-/xyz/openbmc_project/inventory/system/chassis/motherboard/FAN2_RPM |`-/xyz/openbmc_project/inventory/system/chassis/motherboard/FAN2_RPM/FAN2_RPM
>>>>>>>>>>>> |-/xyz/openbmc_project/inventory/system/chassis/motherboard/FAN3
>>>>>>>>>>>> | `-/xyz/openbmc_project/inventory/system/chassis/motherboard/FAN3/FAN3 |-/xyz/openbmc_project/inventory/system/chassis/motherboard/FAN3_RPM |`-/xyz/openbmc_project/inventory/system/chassis/motherboard/FAN3_RPM/FAN3_RPM
>>>>>>>>>>>> |-/xyz/openbmc_project/inventory/system/chassis/motherboard/fan0
>>>>>>>>>>>> |-/xyz/openbmc_project/inventory/system/chassis/motherboard/fan1
>>>>>>>>>>>> |-/xyz/openbmc_project/inventory/system/chassis/motherboard/fan2
>>>>>>>>>>>> `-/xyz/openbmc_project/inventory/system/chassis/motherboard/fan3
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Raj
>>>>>>>>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>>>>>>>>> On Tuesday, September 17, 2019 9:18 PM, Patrick Venture venture at google.com wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Sep 17, 2019 at 8:32 AM Matthew Barth msbarth at linux.ibm.com wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 9/17/19 5:40 AM, rgrs wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>> I need some help with phosphor hwmon related to fan sensors.
>>>>>>>>>>>>>>> I'm trying to control Fans in a server motherboard using ADT7462
>>>>>>>>>>>>>>> controller. Hw mon porting is done and the sysfs interfaces are working
>>>>>>>>>>>>>>> as expected.
>>>>>>>>>>>>>>> While I am able to read RPM, i don't find pwm/target to control the fans
>>>>>>>>>>>>>>> exposed as dbus objects.
>>>>>>>>>>>>>>> Do you have the hwmon config set similar to this to denote the fan
>>>>>>>>>>>>>>> targets are of PWM type?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://github.com/openbmc/openbmc/blob/master/meta-ibm/meta-romulus/recipes-phosphor/sensors/phosphor-hwmon/obmc/hwmon/ahb/apb/pwm-tacho-controller%401e786000.conf
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> busctl --no-page tree xyz.openbmc_project.FanSensor
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ====================================================
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Failed to introspect object / of service xyz.openbmc_project.FanSensor:
>>>>>>>>>>>>>>> The name is not activatable
>>>>>>>>>>>>>>> No objects discovered.
>>>>>>>>>>>>>>> That does not look like a hwmon service name. If you are using
>>>>>>>>>>>>>>> phosphor-objmgr, you can use `# mapper get-service /xyz/openbmc_project/sensors/fan_tach` otherwise you'll need to find the
>>>>>>>>>>>>>>> hwmon service name associated with your fan controller device.
>>>>>>>>>>>>>
>>>>>>>>>>>>> running`busctl tree --no-pager` will just list everything, and you
>>>>>>>>>>>>> should see your fans in there. You can use the mapper too. I was
>>>>>>>>>>>>> curious, are you using entity-manager/dbus-sensors? When you mention
>>>>>>>>>>>>> FanSensor as the service, that's what it sounds like.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I came across this post, I think it is similar to my issue.
>>>>>>>>>>>>>>> (https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-hwmon/+/8353)
>>>>>>>>>>>>>>> Please someone point to latest documentation or reference Fan porting
>>>>>>>>>>>>>>> from openBMC repo?
>>>>>>>>>>>>>>> What are the exact fields in YAML that are used create FanPWM objects.
>>>>>>>>>>>>>>> The Romulus machine should be a good reference for PWM based fan control
>>>>>>>>>>>>>>> using phosphor-hwmon. It uses the
>>>>>>>>>>>>>>> phosphor-fan-presence/[control|monitor] apps for controlling and
>>>>>>>>>>>>>>> monitoring fans
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://github.com/openbmc/openbmc/tree/master/meta-ibm/meta-romulus
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (I'm using Feb release in my sandbox,
>>>>>>>>>>>>>>> https://github.com/openbmc/docs/blob/master/release/release-notes.md#26-feb-4-2019)
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Raj
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Matt
>
>
More information about the openbmc
mailing list