<div dir="ltr">Even,<div><br></div><div>Sorry, I was thinking of mmap() directly to the file, and having something like:</div><div><br></div><div>CPLVirtualMem CPL_DLL* GDALBandGetVirtualMemAuto( GDALRasterBandH hBand,</div>
<div> int *pnPixelSpace,</div><div> GIntBig *pnLineSpace,</div><div> char **papszOptions );</div><div>
<br></div><div>I imagined an available virtual method on the band which could be implemented - primarily by the RawBand class to try and mmap() the data and return the layout. But when that fails, or is unavailable it could use your existing methodology with a layout that seems well tuned to the underlying data organization. </div>
<div><br></div><div>Certainly there is no need to hold things up for this. What you are proposing is already wonderfully useful. I'm wondering if there would be ways of making what you propose work with Python Numpy in such a way that a numpy array could be requested which is of this virtual memory. That would also be a nice extension.</div>
<div><br></div><div>Best regards,</div><div>Frank</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Dec 18, 2013 at 2:10 AM, Even Rouault <span dir="ltr"><<a href="mailto:even.rouault@mines-paris.org" target="_blank">even.rouault@mines-paris.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Le mercredi 18 décembre 2013 06:55:50, Frank Warmerdam a écrit :<br>
<div class="im">> Even,<br>
><br>
> Very impressive work, I am supportive.<br>
><br>
> IMHO it would be wonderful if there was also an mmap() based mechanism<br>
> where you could ask for the virtual memory chunk and you get it back (if it<br>
> works) along with stride values to access in it. This could likely be made<br>
> to work for most "raw" based formats and a few others too. It might also<br>
> allow non-mmap() based files to return an organization based more on their<br>
> actual organization for efficiency.<br>
<br>
</div>Hi Frank,<br>
<br>
I'm not completely sure to have understood your idea. Would that be something<br>
like :<br>
<br>
CPLVirtualMem CPL_DLL* GDALDatasetGetVirtualMemAuto( GDALDatasetH hDS,<br>
GDALRWFlag eRWFlag,<br>
int nXOff, int nYOff,<br>
int nXSize, int nYSize,<br>
int nBufXSize, int nBufYSize,<br>
GDALDataType eBufType,<br>
int nBandCount, int* panBandMap,<br>
int *pnPixelSpace,<br>
GIntBig *pnLineSpace,<br>
GIntBig *pnBandSpace,<br>
size_t nCacheSize,<br>
int bSingleThreadUsage,<br>
char **papszOptions );<br>
<br>
Difference with GDALDatasetGetVirtualMem() : the stride values are now output<br>
values and no more nPageSizeHint parameter.<br>
<br>
In your mind, would the spacings be determined in a generic way from the<br>
dataset properties(block size and INTERLEAVED=PIXEL/BAND metadata item), or<br>
would that require some direct cooperation of the driver ?<br>
<br>
Since you mention raw formats, perhaps you are thinking more to a file-based<br>
mmap() rather than a anonymous mmap() combined with RasterIO(), like currently<br>
proposed ? This is something I've mentionned in the "Related thoughts"<br>
paragraph but there are practical annoyance with how Linux manages memory with<br>
file-based mmap(). I'd be happy if someone has successfull experience with that<br>
by the way (and that doesn't require explicit madvise() each time you're done<br>
with a range of memory)<br>
<br>
---------------------------<br>
<br>
Reading again your words, I'm now wondering if you are not thinking to a<br>
Dataset / RasterBand virtual method that could be implemented by drivers ?<br>
<br>
virtual CPLVirtualMem* GetVirtualMem(.......)<br>
<br>
They would directly use the low-level CPLVirtualMem to create the mapping and<br>
provide their own callback to fill pages when page fault occurs. So they could<br>
potentially avoid using the block cache layer and do direct file I/O ?<br>
<br>
Looking at RawRasterBand::IRasterIO(), I can see that it can use (under some<br>
circumstances with a non obvious heuristics) direct file I/O without going to<br>
the block cache. So the current proposed implementation could potentially<br>
already benefit from that. Perhaps we would need a flag to RasterIO to ask it to<br>
avoid block cache when possible. Or just call<br>
CPLSetThreadLocalConfigOption("GDAL_ONE_BIG_READ", "YES") in<br>
GDALVirtualMem::DoIOBandSequential() / DoIOPixelInterleaved()<br>
<div class="HOEnZb"><div class="h5"><br>
Even<br>
<br>
--<br>
Geospatial professional services<br>
<a href="http://even.rouault.free.fr/services.html" target="_blank">http://even.rouault.free.fr/services.html</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>---------------------------------------+--------------------------------------<br>I set the clouds in motion - turn up | Frank Warmerdam, <a href="mailto:warmerdam@pobox.com" target="_blank">warmerdam@pobox.com</a><br>
light and sound - activate the windows | <a href="http://pobox.com/~warmerdam" target="_blank">http://pobox.com/~warmerdam</a><br>and watch the world go round - Rush | Geospatial Software Developer<br>
</div>