[gdal-dev] HDF5 and NetCDF now have VSI support

[James McClain]

Hello,

Support for use of /vsi paths to HDF5 and NetCDF files has just been merged
into master (please see here https://github.com/OSGeo/gdal/pull/786 ).

We believe that this will be helpful for many people because it is now
possible to work with remote files of those formats on an on-demand basis
rather than being required to download them.  For example, with an
installation built from current master, is now possible to type

gdalinfo
'/vsis3/nasanex/NEX-GDDP/BCSD/rcp85/day/atmos/tasmax/r1i1p1/v1.0/tasmax_day_BCSD_rcp85_r1i1p1_inmcm4_2100.nc'

and receive the appropriate output, or such a path can be used from within
a program.  All of the various VSI drivers should work (e.g. /vsicurl,
/vsitar, et cetera).

There are a few caveats to mention.  First, this capability makes use of
the userfaultfd (user mode page fault handling) capability in recent Linux
kernels so you will need Linux 4.3 or later.

Also, this capability is read-only at this time.

Also, for NetCDF support, you will need libnetcdf 4.5 or later.

Also, because files are mapped into virtual memory, users might want to
impose a (soft) limit on the number of virtual memory pages that can be
consumed by any given file.  That is done by setting the GDAL_UFFD_LIMIT
configuration option to some integer value indicating the number of pages.

One should be careful when using GDAL_UFFD_LIMIT: because it makes use of
custom signal handlers for SIGSEGV and SIGBUS, it is important not to do
anything in another thread in the same process that also changes the
behaviour of those signals while actively using a dataset backed by the
uffd machinery.  A concrete example: you should probably not setup a
/vsimem asset from one thread while another thread is reading a uffd backed
asset (it is, however, perfectly safe to do use /vsimem from one thread and
uffd-backed assets from another as long as the /vsimem setup [when the
changes to signal handling happen] does not overlap in time with the
uffd-backed reads).

I would like to personally thank Even for his patience and accommodation
during this work and for his contributions to it.

Thanks,
James McClain

-- 
"I prayed for freedom for twenty years, but received no answer until I
prayed with my legs."
     -- Frederick Douglass
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20180808/8c42f7c5/attachment.html>