[gdal-dev] Call for discussion on "RFC 45: GDAL datasets and raster bands as virtual memory mappings"

[Frank Warmerdam]

On Wed, Dec 18, 2013 at 11:46 AM, Even Rouault <even.rouault at mines-paris.org
> wrote:

> Le mercredi 18 décembre 2013 19:53:37, Frank Warmerdam a écrit :
> > Even,
> >
> > Sorry, I was thinking of mmap() directly to the file, and having
> something
> > like:
> >
> > CPLVirtualMem CPL_DLL* GDALBandGetVirtualMemAuto( GDALRasterBandH hBand,
> >                                          int *pnPixelSpace,
> >                                          GIntBig *pnLineSpace,
> >                                          char **papszOptions );
> >
> > I imagined an available virtual method on the band which could be
> > implemented - primarily by the RawBand class to try and mmap() the data
> and
> > return the layout.  But when that fails, or is unavailable it could use
> > your existing methodology with a layout that seems well tuned to the
> > underlying data organization.
>
> Yes, that should be doable, but with the limitation I raised about the
> memory
> management of file-based mmap() : if you mmap() a file larger than RAM,
> and read
> it entirely, without explicit madvise() to discard regions no longer
> needed,
> it will fill RAM and cause disk swapping. I should retest to confirm.
> Perhaps

there are some OS level tuning to avoid that ?
>

Even,

That was not my experience for readonly mmap() of actual files on disk
"back in the day".

In any event, I'd suggest sticking with what you have, and if I'm keen
perhaps one day I'll try and implement mmap() support.  If I do, I feel
like it needs to go down through the VSI*L system and once a file is
mmapped() the VSI*L IO should also be using the mmaped images.  Once upon a
time this had performance benefits. I'm not sure if that is the case any
more.

Best regards,
Frank

-- 
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam,
warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Software Developer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20131218/f6f7ee46/attachment.html>