[gdal-dev] GDAL Progressive Data Support RFC Discussion

[Frank Warmerdam]

Aron Rubin wrote:
> Sorry to enter the discussion a little late here but I have a couple
> of notes regarding the RFC. The RFC motion forced me to write this
> quickly so I hope I get the points across.
> 
> 1. The RFC is intentionally not callback/event oriented. You can
> derive a non-callback/event version from a callback/event version but
> not the other way around unless the boundaries match exactly.
> This issue is illustrated by a the ability to satisfy the use case of
> a display of a progressive resolution image. I am lacking the time so
> I will abbreviate. In this use case I see two potential choices of
> display - mixed resolution and fixed resolution. In the mixed
> resolution case anything that happens to be in the loading buffer
> (including nodata values) at display iteration time. In the fixed
> resolution case the display is updated at times when the loading
> buffer is in a consistent state, filled with data at a specific
> resolution. A user may choose the fixed resolution because it may less
> visually confusing or because they want to end the reading at a
> specific resolution. If the reading is bounded on I/O time only there
> is no event emitted (callback) when each resolution pass is completed.
> This is fine for the mixed resolution case but not for fixed
> resolution.
> One way to switch to an event basis without callbacks would be to
> specify reading flags to stop at events and then augment the return
> information with the event parameters i.e. which pass was completed.
> This approach is not scalable however. For example if the dataset in
> question was tiled or had subdatasets the event information would need
> to include which of those were loaded at which resolution and whether
> I/O was complete for a specific tile. Is that all supposed to come
> through the GetNextUpdatedRegion arguments? Would the user be expected
> to keep track of time since the I/O epoch started to get the same I/O
> time budget effect?
> If callback style event notification was used it would be possible for
> different drivers to have a capability-matched set of events with an
> appropriate event structure for each, the user would not need to
> manage the I/O budget, and a I/O cancel could be issued if needed.

Aron,

There seem to be a few ideas in here.

With regard to the fixed vs. mixed resolution results, it is true
that the current API provides no mechanism to request particular
levels of completeness, such as passes in an interleaved progressive
display.  I imagine this issue was not addressed in part because
it has no applicability to the JPIP case.  I'm not at all clear on
how important it is.

I still didn't quite grasp why an event/callback approach is better
than GetNextUpdateRegion() style polling to handle distinct levels
of completeness, nor what you mean by mangaging the I/O budget.

> 2. There is no consideration or mention of how this would work with
> user managed I/O if that ever happens. I would like to eventually see
> a GDAL/OGR user managed I/O in both push and pull forms. Asynchronous
> I/O and user managed I/O have implications for each other.

I'm not sure what you mean by user managed I/O.  Do you mean applications
providing their own low level IO handlers - for instance to the VSI*L
layer?  I'm not sure I see the relationship between the progressive data
api and managing IO.

I appreciate your feedback though I am concerned that you are raising
pretty fundamental changes in the implementation.  I'd want pretty
compelling reasoning to start over.

Best regards,
-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent