[libpc] Point buffering protocols - decision point

[Michael P. Gerlek]

I've committed a bunch of code now to demonstrate the framework, and I'm
happy, but I'm now at a good decision point.  Please read on.

Consider this hypothetical lidar workflow (committed in main.cpp), which is
made up of 4 stages:

  - READ: read data from a file -- assume the file only contains four F32
values per point, (X, Y, Z, Time)
  - CROP: if the Z value is <0, remove the point
  - COLORIZE: add three U8 values (R,G,B) to the point which are determined
based on the value of Z
  - WRITE: output just five values to a file for each point, (X,Y,R,G,B)

Note three key things here: we never use the T value, we only use the Z
value for the three of the stages, and we only use the R,G,B values for the
two of the stages.

Recall that the pipeline operates in a "pull" mode in which each stage,
starting at the end with WRITE, hands a buffer to its previous stage and
says "please fill this for me, with whatever data you have".  This buffer
has an associated layout property, which consists of a field for each data
value of interest.  But how does the WRITE stage "know" what fields are of
interest?  That is, how does any given stage know what fields the previous
stage is going to be supplying to it?

Naively, each stage asks its previous stage, recursively down the pipe, and
we compute the set of all possible fields of interest.  In this example,
then, the buffer would always contains points with seven items,
(X,Y,Z,T,R,G,B).

Or, you could be really clever and change the layout of the buffer as you go
through the pipe.  For example, the buffer coming out of READ really needs
only to have three values (X,Y,Z) since T is never used anywhere in the
pipeline.  Also, the buffer coming into WRITE needs only to have five values
(X,Y,R,G,B) since Z is no longer needed at that point.  At no stage do you
ever actually need all seven values in the buffer at the same time.

But changing the layout of the buffer at each stage would, in the worst
cases, require lots of copying values from the old buffer shape to the new
buffer layout, and so this would seem to be not a good idea at all.

Keeping a constant "maximally sized" layout would be far simple, and
potentially faster -- unless you had a degenerate case in which you did
something like a READ stage which reads in 100 different values and the a
series of STAGES and a WRITER which only ever used one of the values.  But I
could live with that.

With me so far?  Good.  Now go read my code, and see if it still makes
sense.

Assuming everyone likes the model I've checked in so far, I'm going with the
latter "maximal" buffering method.  We will undoubtedly evolve things as we
go along, of course, but so far we have a model that seems pretty durned
efficient, with a reasonable degree of expressiveness.  I will need to
implement some more prototype stages, though, to try and expose any lurking
design flaws.

[yes, I'm still using Layout and Field classes; these will eventually be
replaced by Schema and such, but for now just concentrate on the semantics
of the model]

-mpg