[gdal-dev] vsipreload: enabling VSI Virtual File API for regular I/O

[Etienne Tourigny]

On Tue, May 28, 2013 at 6:57 PM, Etienne Tourigny
<etourigny.dev at gmail.com>wrote:

>
>
> On Sun, May 26, 2013 at 11:46 AM, Even Rouault <
> even.rouault at mines-paris.org> wrote:
>
>> Hi,
>>
>> I've just commited in trunk a new file port/vsipreload.cpp.
>>
>> This file is the source code for a standalone shared library that can be
>> LD_PRELOAD'ed as an overload of libc to enable VSI Virtual FILE API to be
>> used
>> with binaries using regular libc I/O API.
>>
>> WARNING: Linux glibc ONLY. Might work with some adaptations (mainly around
>> 64bit symbols) on other Unix systems
>>
>> Compile:
>> g++ -Wall -fPIC port/vsipreload.cpp -shared -o vsipreload.so -Iport \
>>         -L. -L.libs -lgdal
>>
>> Examples :
>>
>> LD_PRELOAD=vsipreload.so gdalinfo \
>>         /vsicurl/http://download.osgeo.org/gdal/data/ecw/spif83.ecw
>
>
> Even,
>
> I have built vsipreload with a fresh build of svn trunk, but I get the
> following error:
>
> $ g++ -Wall -fPIC port/vsipreload.cpp -shared -o vsipreload.so -Iport -L.
> -L.libs -lgdal
> $ LD_PRELOAD=vsipreload.so gdalinfo         /vsicurl/
> http://download.osgeo.org/gdal/data/ecw/spif83.ecw
> ERROR: ld.so: object 'vsipreload.so' from LD_PRELOAD cannot be preloaded:
> ignored.
> ERROR 4: `/vsicurl/http://download.osgeo.org/gdal/data/ecw/spif83.ecw'
> not recognised as a supported file format.
>
> gdalinfo failed - unable to open '/vsicurl/
> http://download.osgeo.org/gdal/data/ecw/spif83.ecw'.
>
> Is there a way to find out why it cannot be preloaded? Perhaps the
> compilation options should be tweaked?
>

I have managed to get it working buy using LD_PRELOAD=./vsipreload.so

e.g.
LD_PRELOAD=./vsipreload.so gdalinfo /vsicurl/
http://download.osgeo.org/gdal/data/netcdf/utm-big-chunks.nc

It seems that setting LD_PRELOAD outside of the command works also, so this
mechanism would be used transparently.
But does it have a significant overhead and/or chances it might do damage
if always set?

e.g.
export LD_PRELOAD=./vsipreload.so
gdalinfo /vsicurl/
http://download.osgeo.org/gdal/data/netcdf/utm-big-chunks.nc


Is there any way this could be integrated into gdal to transparently
support vsifile for all drivers, without setting LD_PRELOAD?
In this way this feature would be available to all gdal commands or even
other apps that use gdal.

For example, /vsizip/ is used in QGIS, but it does not work with certain
files (e.g. netcdf ) inside archives. But if I use LD_PRELOAD it works for
netcdf files! Neat stuff!
e.g. LD_PRELOAD=./vsipreload.so qgis

great work
Etienne



> Thanks,
> Etienne
>
> My system is Linux Mint 13 ubuntu (quantal), gcc version 4.7.2, 64bits,
> 3.5.0-30-generic kernel
>
>
>>
>> LD_PRELOAD=vsipreload.so gdalinfo \
>>  'HDF4_EOS:EOS_GRID:"/vsicurl/
>> http://download.osgeo.org/gdal/data/hdf4/MOD09Q1G_EVI.A2006233.h07v03.005.2008338190308.hdf
>> ":MODIS_NACP_EVI:MODIS_EVI'
>>
>> LD_PRELOAD=vsipreload.so ogrinfo \
>>          /vsicurl/
>> http://svn.osgeo.org/gdal/trunk/autotest/ogr/data/testavc -ro
>>
>> LD_PRELOAD=vsipreload.so ogrinfo -ro \
>>
>> /vsizip//vsicurl/
>> http://download.osgeo.org/gdal/1.10.0/gdalautotest-1.10.0.zip/gdalautotest-1.10.0/ogr/data/testavc
>>
>> It can work even with non GDAL binaries :
>>
>> LD_PRELOAD=vsipreload.so h5dump -d /x \
>>         /vsicurl/
>> http://download.osgeo.org/gdal/data/netcdf/utm-big-chunks.nc
>>
>> LD_PRELOAD=vsipreload.so sqlite3 \
>>          /vsicurl/http://download.osgeo.org/gdal/data/sqlite3/polygon.db\
>>         "select * from polygon limit 10"
>>
>> This can work with all VSI Large File API filesystems : /vsizip/ ,
>> /vsitar/,
>> /vsisubfile/ , etc...
>>
>> This is still a bit experimental in the sense that only the most common
>> glibc
>> I/O API have been overloaded. If exotic ones are used with /vsi files,
>> crashes
>> are likely.
>>
>> --
>> Geospatial professional services
>> http://even.rouault.free.fr/services.html
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20130528/5c22db1a/attachment-0001.html>