[gdal-dev] GDAL Progressive Data Support RFC Discussion

[Aron Rubin]

Sorry to enter the discussion a little late here but I have a couple
of notes regarding the RFC. The RFC motion forced me to write this
quickly so I hope I get the points across.

1. The RFC is intentionally not callback/event oriented. You can
derive a non-callback/event version from a callback/event version but
not the other way around unless the boundaries match exactly.
This issue is illustrated by a the ability to satisfy the use case of
a display of a progressive resolution image. I am lacking the time so
I will abbreviate. In this use case I see two potential choices of
display - mixed resolution and fixed resolution. In the mixed
resolution case anything that happens to be in the loading buffer
(including nodata values) at display iteration time. In the fixed
resolution case the display is updated at times when the loading
buffer is in a consistent state, filled with data at a specific
resolution. A user may choose the fixed resolution because it may less
visually confusing or because they want to end the reading at a
specific resolution. If the reading is bounded on I/O time only there
is no event emitted (callback) when each resolution pass is completed.
This is fine for the mixed resolution case but not for fixed
resolution.
One way to switch to an event basis without callbacks would be to
specify reading flags to stop at events and then augment the return
information with the event parameters i.e. which pass was completed.
This approach is not scalable however. For example if the dataset in
question was tiled or had subdatasets the event information would need
to include which of those were loaded at which resolution and whether
I/O was complete for a specific tile. Is that all supposed to come
through the GetNextUpdatedRegion arguments? Would the user be expected
to keep track of time since the I/O epoch started to get the same I/O
time budget effect?
If callback style event notification was used it would be possible for
different drivers to have a capability-matched set of events with an
appropriate event structure for each, the user would not need to
manage the I/O budget, and a I/O cancel could be issued if needed.

2. There is no consideration or mention of how this would work with
user managed I/O if that ever happens. I would like to eventually see
a GDAL/OGR user managed I/O in both push and pull forms. Asynchronous
I/O and user managed I/O have implications for each other.

Thank you for your consideration of these issues.
Aron

-- 
Aron Rubin
Handy Husband & Daddy Jungle Gym & Senior Engineer