[gdal-dev] Support of filemapping for fast rendering in QGIS...

[Even Rouault]

Selon A Huarte <ahuarte47 at yahoo.es>:

> Hi, I want improve the rendering of vector features in QGIS. 
>
> I have opened a new branch of code in the repository of
>
QGIS(https://github.com/qgis/QGIS/pull/927 // https://github.com/ahuarte47/QGIS/tree/Issue_8725-OGR) and
> the drawing tests give me ~3x faster that original code :-)

On which OS ?
Is it the performance gain of QGIS + OGR changes combined ?
What is the speed-up in OGR side only ? You could for example use the test_ogrsf
utility as a potential benchmark. You have to build explicitely  with "make
test_ogrsf" / "nmake /f makefile.vc test_ogrsf.exe", in the apps directory of
GDAL.

>
> I am testing with huge shapefiles (>500.000 records) and I have detect that
> the OGR-shapefile provider would be optimized using filemapping (boost
> <http://www.boost.org/>, or others...)
>
> I wonder if there is any support for FileMapping in gdal or planned.

There is none currently. This is something that has been talked about by a few
people, but never implemented yet. I actually had a local experiment on Linux
but I couldn't observe any real performance gain by using mmap() rather than
fread(). Things are maybe different on the Windows side.

> My
> intention would be to redefine the access to the files in this provider with
> a new function with "SAHooks" using FileMapping. I provide, as explanation,
> an unfinished implementation using boost with which greatly speeds up the
> read of files for critical, massive and repetitive processes such as the
> painting in a map.

A more general approach that could benefit to other drivers than shapefile
(potentially most GDAL and OGR drivers actually) would be to implement a
/vsimemmapped/ virtual file system driver. You have many examples of such
drivers in the port subdirectory. And like it is done currently with /vsicache/,
the default virtual file system in cpl_vsil_unix_stdio_64.cpp or
cpl_vsil_win32.cpp could be instructed to use /vsicache/ if an env. variable is
toggled.

>
>
> On the other hand, this provider may also accelerate with some small changes
> that are pending of review in the ticket
> (http://trac.osgeo.org/gdal/ticket/5272) that I hope will be evaluated.

I had skimmed through that one too :

* Did you evaluate the precision loss of using OGRFastAtof() rather than
CPLAtof() ?

* Are you sure that the performance improvement of replacing memcpy() by *dest =
*src is significant ? A decent compiler (GCC ;-)) should optimize that on most
platforms anyway. The thing is that there are still people using odd platforms
where a non-aligned pointer assingment will cause a SIGBUS, hence the need for
memcpy(). If the performance gain is significant enough we could perhaps have a
MEMCPY_DOUBLE macro that on X86 would evaluate to *dest = *ptr and on other
platforms would default to memcpy().

Best regards,

Even