Request for comments: C++ embedded webserver

[Tanous, Ed]

Have you guys used golang in a product before?  Do you think the best practices are defined well enough for the embedded environment?

We have very little to no experience with it in a production environment, so I have some serious reservations about the ramp-up costs of teaching enough developers golang for it to be useful as a whole.  Do you guys have any experience with that?  How close is it to other languages, and what's the ramp-up time?

Other responses inline below.

-Ed

> -----Original Message-----
> From: Michael.E.Brown at dell.com [mailto:Michael.E.Brown at dell.com]
> Sent: Thursday, December 21, 2017 1:41 PM
> To: Tanous, Ed <ed.tanous at intel.com>; benh at kernel.crashing.org;
> openbmc at lists.ozlabs.org
> Subject: RE: Request for comments: C++ embedded webserver
> 
> Yes, my server is posted on my personal github.... however, usual caveats
> apply: this was done on my personal time in order to teach myself 'go' to see
> if it is worthwhile to pursue. It is nowhere near "production" quality and I am
> actively working on getting some actual developer time to get it all fleshed
> out. https://github.com/superchalupa/go-redfish/
AWESOME!  This was ultimately my goal in posting this in the first place, to flush out the other people that have been working this problem space already.
Any chance you have a yocto recipe for building this for target?  My golang-foo appears to be lacking at the moment to get all the dependencies met.

> 
> I used a couple of frameworks to try them out, most of it could be cut out
> and trimmed down a bit, but it has middleware that provides logging and
> metrics that have proved very handy.
> 
> Design philosophy (the succinct, non-book version):
> 
> It is based on a standard DDD/CQRS/Event Source design, though it is going
> to need some modifications because standard Event Sourcing won't really
> work well in embedded: we'll need a way to coalesce events as we cannot
> really keep the entire event history forever. If you are familiar with
> DDD/CQRS, the quick way to explain it is that each Redfish Resource (ie.
> Redfish URI) is an "aggregate". I've defined the  following "commands", and
> each command has a corresponding event:
> 	CreateRedfishResource
> 	UpdateRedfishResourceProperties
> 	RemoteRedfishResourceProperty
> 	RemoveRedfishResource

Are there OpenBmc use cases that are better served by this type of design versus a normal data model?  I could see a couple interesting use cases where it would make the control flow cleaner (PATCH for example) but the impact of keeping the bookkeeping to keep it up to date with dbus seems like it could be a source of concern.

> 
> The sort of high level philosophy is that we keep the JSON data in memory
> (as a golang map[string]interface{}) so that most reads are satisfied without
> ever going out of process. We have an internal event bus where we take
> external events and publish them on the internal bus for various modules to
> decide what to do.
I very much suspect this level of caching is going to be a non-starter once built on target.  Among the top 5 priorities for most people in openbmc are system start up time, and persistent (as well as transient) memory usage.  Having a service that starts, and does the very expensive operations to populate the cache on startup, for the sake of saving some cycles per request is likely not going to sit well.  Some of these issues could be resolved with a model that populates on first request, but I still suspect storing all the available data in memory is going to be a non-starter, especially once the system starts to scale.  Ideally we'd like to see this implementation be viable to low end rack manager type duties, and having an event-updated json cache is going to run the BMC out of memory in a hurry.

> 
> Here is what I think is the important part: since it is an event bus and we have
> the JSON tree structure in memory, you can easily add OEM extensions. The
> OEM extension simply listens for the creation of objects that it is interested
> in and can then add new data to the object without conflicting with existing
> code. This is all very similar in spirit to what you are currently doing with
> DBUS, and I think these two could very well marry up nicely.

Agreed, I like the idea of being able to add OEM extensions as something represented in the JSON struct.  In the crow implementation, OEM routes are managed by convention and router registrations, which is really unfortunate and could be modeled better.

> 
> The startup process: to create the initial tree of JSON output, we generically
> read in a redfish dump and do the initial tree creation. This makes for easy
> customizability because you can simply edit the text JSON files on disk to
> change them. I'd like to see if there is a way to do the data mappings inside
> the initial JSON files as well.
This is a really cool feature that I never considered.  Do you guys have a lot of use cases where the available endpoints and json structures need to be modified on the fly?  We've been having a discussion about runtime configurability using json files for other things (sensors, board level config, ect) but I never considered using it to control endpoint availability.


> 
> On the other side of this thread, you have made what I think is a really
> excellent rundown of the types of data we will need to pull for redfish, so let
> me move over to that side to address some of those issues.
> 
> And finally: the existing REST interface that generically exposes DBUS: I think
> this could be implemented in go as well, though that's a completely separate
> issue. I have security concerns with how we expose DBUS externally and my
> overall impression is that it would be much nicer to have our UI sit on top of
> redfish, though I understand that is a fairly big lift.
I really dislike this interface as well, and I've been (sometimes overly) vocal about it on several occasions.  I built the interface to prove that this could be a full replacement for the existing functionality, and could properly host the existing webui until redfish was available.  It is my intention long-term to transition the webui to redfish, at least for the systems I own, and for this interface to be deprecated or removed.



> --
> Michael
> 
> 
> -----Original Message-----
> From: Tanous, Ed [mailto:ed.tanous at intel.com]
> Sent: Thursday, December 21, 2017 12:47 PM
> To: Brown, Michael E <Michael_E_Brown at Dell.com>;
> benh at kernel.crashing.org; openbmc at lists.ozlabs.org
> Subject: RE: Request for comments: C++ embedded webserver
> 
> Can you post your prototype server somewhere that's accessible externally?
> I'm genuinely curious how it compares.
> 
> I had looked at golang, but there was no precedence for it in the project, nor
> did I have any experience with it, so I went another direction.  I would love if
> a language change made all the async handling easier to write than what's
> there.  My gut tells me to be concerned about the performance of the go
> garbage collection model on the AST using too much memory for some of the
> lesser capable systems to handle, but I have zero data on it so I'll withhold
> judgement until I have some.
> 
> -Ed
> 
> > -----Original Message-----
> > From: Michael.E.Brown at dell.com [mailto:Michael.E.Brown at dell.com]
> > Sent: Thursday, December 21, 2017 9:25 AM
> > To: benh at kernel.crashing.org; Tanous, Ed <ed.tanous at intel.com>;
> > openbmc at lists.ozlabs.org
> > Subject: RE: Request for comments: C++ embedded webserver
> >
> > Benjamin,
> >
> > I'm new around here, so I didn't want to be the first to bring this up
> > without doing my research on any older threads, so thank you for bringing
> this up.
> >
> > My strong opinion is that golang is likely the best candidate for this
> > specific type of programming environment (ie. OpenBMC).
> > 	- Low runtime memory usage
> > 	- Automatic Memory Management
> > 	- Clean cross compile for ARMv5 and above
> > 	- Compiled
> > 	- strongly typed
> > 	- Fast
> > 	- Cleanly handles threading/async
> > 	- Nice standard library
> >
> > I had prototyped a redfish server in golang enough to get to the point
> > where I believe that it's the best solution. I have great things to
> > say internally about the existing openbmc design: I love the use of
> > DBUS, the more I look at it the more clearly the advantages seem to
> > me. The nice thing about this is that golang has a DBUS library and we
> > ought to be able to write different components in different languages
> > freely, especially if we have the underlying dbus apis correct.
> >
> > --
> > Michael
> >
> > -----Original Message-----
> > From: openbmc [mailto:openbmc-
> > bounces+michael.e.brown=dell.com at lists.ozlabs.org] On Behalf Of
> > Benjamin Herrenschmidt
> > Sent: Wednesday, December 20, 2017 6:34 PM
> > To: Tanous, Ed <ed.tanous at intel.com>; OpenBMC Maillist
> > <openbmc at lists.ozlabs.org>
> > Subject: Re: Request for comments: C++ embedded webserver
> >
> > On Tue, 2017-12-19 at 04:07 +0000, Tanous, Ed wrote:
> > > I’m looking for comments on a code review that’s been outstanding.
> > > One of the large pushes we’ve made is to attempt to make the web
> > > server more efficient, and add capabilities that comprehend long
> > > term needs of OpenBmc.  One key that wasn’t made clear in the commit
> > > message is that it includes the basic redfish implementation that
> > > (we
> > > hope) should be extensible to the full redfish specification in the
> > > short term future.
> >
> > There are plenty of existing HTTP server implementations out there
> > that are well maintained, any reason why we should invent another one
> here ?
> >
> > Also, do we really think perpetuating the use of that monstruosity
> > masquerading as a programming language that is C++ is a good idea ?
> >
> > > https://gerrit.openbmc-project.xyz/#/c/7786/
> > >
> > > We would very much appreciate comments to see if we can move this
> > forward.
> > >
> > > The following description is pulled from the readme.
> > >
> > > # OpenBMC webserver #
> > >
> > > This component attempts to be a "do everything" embedded webserver
> > for openbmc.
> > >
> > > ## Capabilities ##
> > > At this time, the webserver implements a few interfaces:
> > > + Authentication middleware that supports cookie and token based
> > authentication, as well as CSRF prevention backed by linux PAM
> > authentication credentials.
> > > + An (incomplete) attempt at replicating phosphor-dbus-rest
> > > + interfaces in
> > C++.  Right now, a few of the endpoint definitions work as expected,
> > C++but
> > there is still a lot of work to be done.  The portions of the
> > interface that are functional are designed to work correctly for
> > phosphor-webui, but may not yet be complete.
> > > + Replication of the rest-dbus backend interfaces to allow bmc debug
> > > + to
> > logged in users.
> > > + An initial attempt at a read-only redfish interface.  Currently
> > > + the redfish
> > interface targets ServiceRoot, SessionService, AccountService, Roles,
> > and ManagersService.  Some functionality here has been shimmed to make
> > development possible.  For example, there exists only a single user role.
> > > + SSL key generation at runtime.  If an RSA key and cert pair are
> > > + not
> > available to the server at runtime, keys are generated using the
> > openssl routines, and written to disk.
> > > + Static file hosting.  Currently, static files are hosted from the
> > > + fixed location
> > at /usr/share/www.  This is intended to allow loose coupling with
> > yocto projects, and allow overriding static files at build time.
> > > + Dbus-monitor over websocket.  A generic endpoint that allows UIs
> > > + to open a websocket and register for notification of events to
> > > + avoid polling in single page applications.  (this interface may be
> > > + modified in the future due to security concerns.)
> > >
> > >
> > > Thanks,
> > >
> > > -Ed
> > >