[gdal-dev] Limit on the number of files for gdalbuildvrt

[Jose Gomez-Dans]

Getting through a bit further... It appears that I can get this fixed if I
set GDAL_DISABLE_READDIR_ON_OPEN to YES:
GDAL_DISABLE_READDIR_ON_OPEN=YES gdalbuildvrt -input_file_list flist.txt
outputfile.vrt

In order to use the file, GDAL_DISABLE_READDIR_ON_OPEN needs to be selected
too.

Mystery solved, it would seem.

Thanks!

On Fri, 1 Nov 2019 at 17:21, Jose Gomez-Dans <jgomezdans at gmail.com> wrote:

> Dear Even,
>
> Thanks for your help. After a bit more dredging, I am only able to repeat
> my problem on a particularly mounted filesystem in an HPC facility I use.
> On other systems (and even other mountpoints on that same HPC system)
> things work as expected. I've taken this up with our administrators, but
> just as a warning to others that it can happen and I didn't dream it!
>
> Thanks for your very prompt reply
> Jose
>
> On Wed, 30 Oct 2019 at 16:16, Even Rouault <even.rouault at spatialys.com>
> wrote:
>
>> On mercredi 30 octobre 2019 15:52:54 CET Jose Gomez-Dans wrote:
>> > Hi,
>> >
>> > I'm mosaicking a large number of small tiles using the GDAL python
>> bindings
>> > (python 3.7, gdal 3.0.1, but also tested on 2.4.2). Code is something
>> like
>> >
>> > ```
>> >
>> > *output_fname = "/something/or/other.vrt"*
>> > *fnames = ["file1.tif", ..., "fileN.tif"]*
>> >
>> >
>> >
>> >
>> > *dst_ds = gdal.BuildVRT(            output_fname,            fnames,
>> >     options=gdal.BuildVRTOptions(separate=True),        )*
>> > *dst_ds = None*
>> > ```
>> >
>> > This fails with lots of errors like
>> > ERROR 4: fileXXX.tif: No such file or directory
>> > Warning 1: Can't open fileXXX.tif. Skipping it
>>
>> gdalbuildvrt is normally cautious about non opening too many files at the
>> same
>> time. I can't replicate your issue with the following script
>>
>> ```
>> from osgeo import gdal
>> import shutil
>>
>> fnames = []
>> for i in range(1024):
>>     dstname = 'testbuildvrt/byte%d.tif' % i
>>     shutil.copy('byte.tif', dstname)
>>     fnames.append(dstname)
>>
>> dst_ds = gdal.BuildVRT(
>>             "/tmp/out.vrt",
>>             fnames,
>>             options=gdal.BuildVRTOptions(separate=True),
>>         )
>> dst_ds = None
>> ```
>>
>> > The aim of this is to create a VRT mosaic that can then be converted to
>> > GeoTIFF.
>>
>> If you want a mosaic, you likely want separate=False.
>>
>> Even
>>
>> --
>> Spatialys - Geospatial professional services
>> http://www.spatialys.com
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20191107/e24ff532/attachment.html>