Request for comments: C++ embedded webserver

[Michael.E.Brown at dell.com]

No, we have not used golang in a product. I've been advocated internally for it and believe I have consensus from most of the senior technical people in my area at Dell, it's just a matter of finding the correct place to start using it. (We don't want to re-write something just for the sake of re-writing). I've penciled it in as a perfect language to start a redfish server and have started doing prototypes. I'd like to have a redfish server for openbmc that we can re-use in other areas.

From my experience, ramp up costs for golang are fairly reasonable. I was able to get the basics in less than a week, and was fairly proficient after a month. The language itself is simple, the standard library is well-fleshed-out and it is extensively used in backend rest infrastructure. The syntax is very C like with some logical changes. It has built in threading model and built in concepts that help with avoiding locking (CSP). I think it ticks all of the boxes to make a very promising language for embedded.

I have a yocto recipe for the existing server. I'll post that shortly, though it is relatively simple to cross compile: "GOARCH=arm go build ..."

To address your concern about startup time: I believe that with the right model, we could implement optimizations to reduce startup time. To start with, we can construct the redfish responses and save them to persistent storage. Then those are just read in at startup instead of doing a lot of heavy processing. All of the things needed for fast path can be pre-staged or cached from persistent storage. That way we don't need to construct the entire tree from scratch at startup.

Your question about data model: I personally think this data model is the way to go to get the most extensibility and least amount of coupling. I imagine that most groups that want to ship openbmc as a product will be customizing it in various ways, so the we need to build that in from the beginning to avoid forks for everything.

I'm working on a more comprehensive writeup I hope to send out later today, but the short of it is: one daemon that implements the basic redfish data model in a generic way. It can read in the basic tree, and the basic tree will have metadata for how to get basic values. For more complex values, another daemon can either export an API for the redfish server to use, or it can poke data directly into redfish. The base redfish server has a dbus api and emits signals for all the data changes that would allow OEM processes to modify/override any base behavior they want.

Dell has extensive redfish support in our existing product. I need to have a clean way to implement all of the existing functionality in the openbmc server that we eventually ship without making extensive modifications or forking. I think this model gets us there (with a little/lot of input from you guys, obviously).
--
Michael


-----Original Message-----
From: Tanous, Ed [mailto:ed.tanous at intel.com] 
Sent: Thursday, December 21, 2017 6:33 PM
To: Brown, Michael E <Michael_E_Brown at Dell.com>; benh at kernel.crashing.org; openbmc at lists.ozlabs.org
Subject: RE: Request for comments: C++ embedded webserver

Have you guys used golang in a product before?  Do you think the best practices are defined well enough for the embedded environment?

We have very little to no experience with it in a production environment, so I have some serious reservations about the ramp-up costs of teaching enough developers golang for it to be useful as a whole.  Do you guys have any experience with that?  How close is it to other languages, and what's the ramp-up time?

Other responses inline below.

-Ed

> -----Original Message-----
> From: Michael.E.Brown at dell.com [mailto:Michael.E.Brown at dell.com]
> 
> Yes, my server is posted on my personal github.... however, usual 
> caveats
> apply: this was done on my personal time in order to teach myself 'go' 
> to see if it is worthwhile to pursue. It is nowhere near "production" 
> quality and I am actively working on getting some actual developer 
> time to get it all fleshed out. 
> https://github.com/superchalupa/go-redfish/
AWESOME!  This was ultimately my goal in posting this in the first place, to flush out the other people that have been working this problem space already.
Any chance you have a yocto recipe for building this for target?  My golang-foo appears to be lacking at the moment to get all the dependencies met.

> 
> I used a couple of frameworks to try them out, most of it could be cut 
> out and trimmed down a bit, but it has middleware that provides 
> logging and metrics that have proved very handy.
> 
> Design philosophy (the succinct, non-book version):
> 
> It is based on a standard DDD/CQRS/Event Source design, though it is 
> going to need some modifications because standard Event Sourcing won't 
> really work well in embedded: we'll need a way to coalesce events as 
> we cannot really keep the entire event history forever. If you are 
> familiar with DDD/CQRS, the quick way to explain it is that each Redfish Resource (ie.
> Redfish URI) is an "aggregate". I've defined the  following 
> "commands", and each command has a corresponding event:
> 	CreateRedfishResource
> 	UpdateRedfishResourceProperties
> 	RemoteRedfishResourceProperty
> 	RemoveRedfishResource

Are there OpenBmc use cases that are better served by this type of design versus a normal data model?  I could see a couple interesting use cases where it would make the control flow cleaner (PATCH for example) but the impact of keeping the bookkeeping to keep it up to date with dbus seems like it could be a source of concern.

> 
> The sort of high level philosophy is that we keep the JSON data in 
> memory (as a golang map[string]interface{}) so that most reads are 
> satisfied without ever going out of process. We have an internal event 
> bus where we take external events and publish them on the internal bus 
> for various modules to decide what to do.
I very much suspect this level of caching is going to be a non-starter once built on target.  Among the top 5 priorities for most people in openbmc are system start up time, and persistent (as well as transient) memory usage.  Having a service that starts, and does the very expensive operations to populate the cache on startup, for the sake of saving some cycles per request is likely not going to sit well.  Some of these issues could be resolved with a model that populates on first request, but I still suspect storing all the available data in memory is going to be a non-starter, especially once the system starts to scale.  Ideally we'd like to see this implementation be viable to low end rack manager type duties, and having an event-updated json cache is going to run the BMC out of memory in a hurry.

> 
> Here is what I think is the important part: since it is an event bus 
> and we have the JSON tree structure in memory, you can easily add OEM 
> extensions. The OEM extension simply listens for the creation of 
> objects that it is interested in and can then add new data to the 
> object without conflicting with existing code. This is all very 
> similar in spirit to what you are currently doing with DBUS, and I think these two could very well marry up nicely.

Agreed, I like the idea of being able to add OEM extensions as something represented in the JSON struct.  In the crow implementation, OEM routes are managed by convention and router registrations, which is really unfortunate and could be modeled better.

> 
> The startup process: to create the initial tree of JSON output, we 
> generically read in a redfish dump and do the initial tree creation. 
> This makes for easy customizability because you can simply edit the 
> text JSON files on disk to change them. I'd like to see if there is a 
> way to do the data mappings inside the initial JSON files as well.
This is a really cool feature that I never considered.  Do you guys have a lot of use cases where the available endpoints and json structures need to be modified on the fly?  We've been having a discussion about runtime configurability using json files for other things (sensors, board level config, ect) but I never considered using it to control endpoint availability.


> 
> On the other side of this thread, you have made what I think is a 
> really excellent rundown of the types of data we will need to pull for 
> redfish, so let me move over to that side to address some of those issues.
> 
> And finally: the existing REST interface that generically exposes 
> DBUS: I think this could be implemented in go as well, though that's a 
> completely separate issue. I have security concerns with how we expose 
> DBUS externally and my overall impression is that it would be much 
> nicer to have our UI sit on top of redfish, though I understand that is a fairly big lift.
I really dislike this interface as well, and I've been (sometimes overly) vocal about it on several occasions.  I built the interface to prove that this could be a full replacement for the existing functionality, and could properly host the existing webui until redfish was available.  It is my intention long-term to transition the webui to redfish, at least for the systems I own, and for this interface to be deprecated or removed.



> --
> Michael
> 
> 
> -----Original Message-----
> From: Tanous, Ed [mailto:ed.tanous at intel.com]
> Sent: Thursday, December 21, 2017 12:47 PM
> To: Brown, Michael E <Michael_E_Brown at Dell.com>; 
> benh at kernel.crashing.org; openbmc at lists.ozlabs.org
> Subject: RE: Request for comments: C++ embedded webserver
> 
> Can you post your prototype server somewhere that's accessible externally?
> I'm genuinely curious how it compares.
> 
> I had looked at golang, but there was no precedence for it in the 
> project, nor did I have any experience with it, so I went another 
> direction.  I would love if a language change made all the async 
> handling easier to write than what's there.  My gut tells me to be 
> concerned about the performance of the go garbage collection model on 
> the AST using too much memory for some of the lesser capable systems 
> to handle, but I have zero data on it so I'll withhold judgement until I have some.
> 
> -Ed
> 
> > -----Original Message-----
> > From: Michael.E.Brown at dell.com [mailto:Michael.E.Brown at dell.com]
> > Sent: Thursday, December 21, 2017 9:25 AM
> > To: benh at kernel.crashing.org; Tanous, Ed <ed.tanous at intel.com>; 
> > openbmc at lists.ozlabs.org
> > Subject: RE: Request for comments: C++ embedded webserver
> >
> > Benjamin,
> >
> > I'm new around here, so I didn't want to be the first to bring this 
> > up without doing my research on any older threads, so thank you for 
> > bringing
> this up.
> >
> > My strong opinion is that golang is likely the best candidate for 
> > this specific type of programming environment (ie. OpenBMC).
> > 	- Low runtime memory usage
> > 	- Automatic Memory Management
> > 	- Clean cross compile for ARMv5 and above
> > 	- Compiled
> > 	- strongly typed
> > 	- Fast
> > 	- Cleanly handles threading/async
> > 	- Nice standard library
> >
> > I had prototyped a redfish server in golang enough to get to the 
> > point where I believe that it's the best solution. I have great 
> > things to say internally about the existing openbmc design: I love 
> > the use of DBUS, the more I look at it the more clearly the 
> > advantages seem to me. The nice thing about this is that golang has 
> > a DBUS library and we ought to be able to write different components 
> > in different languages freely, especially if we have the underlying dbus apis correct.
> >
> > --
> > Michael
> >
> > -----Original Message-----
> > From: openbmc [mailto:openbmc-
> > bounces+michael.e.brown=dell.com at lists.ozlabs.org] On Behalf Of
> > Benjamin Herrenschmidt
> > Sent: Wednesday, December 20, 2017 6:34 PM
> > To: Tanous, Ed <ed.tanous at intel.com>; OpenBMC Maillist 
> > <openbmc at lists.ozlabs.org>
> > Subject: Re: Request for comments: C++ embedded webserver
> >
> > On Tue, 2017-12-19 at 04:07 +0000, Tanous, Ed wrote:
> > > I’m looking for comments on a code review that’s been outstanding.
> > > One of the large pushes we’ve made is to attempt to make the web 
> > > server more efficient, and add capabilities that comprehend long 
> > > term needs of OpenBmc.  One key that wasn’t made clear in the 
> > > commit message is that it includes the basic redfish 
> > > implementation that (we
> > > hope) should be extensible to the full redfish specification in 
> > > the short term future.
> >
> > There are plenty of existing HTTP server implementations out there 
> > that are well maintained, any reason why we should invent another 
> > one
> here ?
> >
> > Also, do we really think perpetuating the use of that monstruosity 
> > masquerading as a programming language that is C++ is a good idea ?
> >
> > > https://gerrit.openbmc-project.xyz/#/c/7786/
> > >
> > > We would very much appreciate comments to see if we can move this
> > forward.
> > >
> > > The following description is pulled from the readme.
> > >
> > > # OpenBMC webserver #
> > >
> > > This component attempts to be a "do everything" embedded webserver
> > for openbmc.
> > >
> > > ## Capabilities ##
> > > At this time, the webserver implements a few interfaces:
> > > + Authentication middleware that supports cookie and token based
> > authentication, as well as CSRF prevention backed by linux PAM 
> > authentication credentials.
> > > + An (incomplete) attempt at replicating phosphor-dbus-rest 
> > > + interfaces in
> > C++.  Right now, a few of the endpoint definitions work as expected, 
> > C++but
> > there is still a lot of work to be done.  The portions of the 
> > interface that are functional are designed to work correctly for 
> > phosphor-webui, but may not yet be complete.
> > > + Replication of the rest-dbus backend interfaces to allow bmc 
> > > + debug to
> > logged in users.
> > > + An initial attempt at a read-only redfish interface.  Currently 
> > > + the redfish
> > interface targets ServiceRoot, SessionService, AccountService, 
> > Roles, and ManagersService.  Some functionality here has been 
> > shimmed to make development possible.  For example, there exists only a single user role.
> > > + SSL key generation at runtime.  If an RSA key and cert pair are 
> > > + not
> > available to the server at runtime, keys are generated using the 
> > openssl routines, and written to disk.
> > > + Static file hosting.  Currently, static files are hosted from 
> > > + the fixed location
> > at /usr/share/www.  This is intended to allow loose coupling with 
> > yocto projects, and allow overriding static files at build time.
> > > + Dbus-monitor over websocket.  A generic endpoint that allows UIs 
> > > + to open a websocket and register for notification of events to 
> > > + avoid polling in single page applications.  (this interface may 
> > > + be modified in the future due to security concerns.)
> > >
> > >
> > > Thanks,
> > >
> > > -Ed
> > >