[gdal-dev] Speed improvement for OGR shapefile provider

[A Huarte]

Hi, I made tests with 'test_ogrsf.exe' and a dbase file.


https://github.com/OSGeo/gdal/pull/24


Regards



>________________________________
> De: Even Rouault <even.rouault at mines-paris.org>
>Para: A Huarte <ahuarte47 at yahoo.es> 
>CC: "gdal-dev at lists.osgeo.org" <gdal-dev at lists.osgeo.org> 
>Enviado: Martes 22 de octubre de 2013 14:17
>Asunto: Re: [gdal-dev]  Support of filemapping for fast rendering in QGIS...
> 
>
>Selon A Huarte <ahuarte47 at yahoo.es>:
>
>> Hi, I want improve the rendering of vector features in QGIS. 
>>
>> I have opened a new branch of code in the repository of
>>
>QGIS(https://github.com/qgis/QGIS/pull/927 // https://github.com/ahuarte47/QGIS/tree/Issue_8725-OGR) and
>> the drawing tests give me ~3x faster that original code :-)
>
>On which OS ?
>Is it the performance gain of QGIS + OGR changes combined ?
>What is the speed-up in OGR side only ? You could for example use the test_ogrsf
>utility as a potential benchmark. You have to build explicitely  with "make
>test_ogrsf" / "nmake /f makefile.vc test_ogrsf.exe", in the apps directory of
>GDAL.
>
>>
>> I am testing with huge shapefiles (>500.000 records) and I have detect that
>> the OGR-shapefile provider would be optimized using filemapping (boost
>> <http://www.boost.org/>, or others...)
>>
>> I wonder if there is any support for FileMapping in gdal or planned.
>
>There is none currently. This is something that has been talked about by a few
>people, but never implemented yet. I actually had a local experiment on Linux
>but I couldn't observe any real performance gain by using mmap() rather than
>fread(). Things are maybe different on the Windows side.
>
>> My
>> intention would be to redefine the access to the files in this provider with
>> a new function with "SAHooks" using FileMapping. I provide, as explanation,
>> an unfinished implementation using boost with which greatly speeds up the
>> read of files for critical, massive and repetitive processes such as the
>> painting in a map.
>
>A more general approach that could benefit to other drivers than shapefile
>(potentially most GDAL and OGR drivers actually) would be to implement a
>/vsimemmapped/ virtual file system driver. You have many examples of such
>drivers in the port subdirectory. And like it is done currently with /vsicache/,
>the default virtual file system in cpl_vsil_unix_stdio_64.cpp or
>cpl_vsil_win32.cpp could be instructed to use /vsicache/ if an env. variable is
>toggled.
>
>>
>>
>> On the other hand, this provider may also accelerate with some small changes
>> that are pending of review in the ticket
>> (http://trac.osgeo.org/gdal/ticket/5272) that I hope will be evaluated.
>
>I had skimmed through that one too :
>
>* Did you evaluate the precision loss of using OGRFastAtof() rather than
>CPLAtof() ?
>
>* Are you sure that the performance improvement of replacing memcpy() by *dest =
>*src is significant ? A decent compiler (GCC ;-)) should optimize that on most
>platforms anyway. The thing is that there are still people using odd platforms
>where a non-aligned pointer assingment will cause a SIGBUS, hence the need for
>memcpy(). If the performance gain is significant enough we could perhaps have a
>MEMCPY_DOUBLE macro that on X86 would evaluate to *dest = *ptr and on other
>platforms would default to memcpy().
>
>Best regards,
>
>Even
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20131106/7664ee1c/attachment.html>