[gdal-dev] gdal_translate .vrt issue

[Even Rouault]

Selon David Fawcett <david.fawcett at gmail.com>:

> I am using gdal_translate and a .vrt to take 115 .jp2 files from the USGS,
> pull them together and then export a lower resolution .jp2.
>
> This is with GDAL 1.10 on OSX.
> gdalinfo --version
> GDAL 1.10.1, released 2013/08/26
>
> When I run gdal_translate, it gets about 30% of the way through the process
> and errors out saying that it can't open one of the .j2w files.  When I run
> gdalinfo on the file, I can open it, and if I run gdal_translate on just
> that file, I can successfully export a reduced resolution copy.
>
> The output of gdalinfo is here:
> https://gist.github.com/fawcett/beead31741e24394b38e
>
> I can also load the .vrt image into QGIS 2.2 and it loads (slowly of
> course).

Even when you zoom out at 10% like your gdal_translate command ?

>
> gdal_translate img_index.vrt mpls34.jp2 -outsize 10% 10% -of JP2OpenJPEG
> -co QUALITY=25 -epo
> Input file size is 45000, 65000
> 0...10...20.ERROR 4:
>
CSLLoad2("/Users/david/projects/mpls_watersheds/data/15TVK830815_201203_0x3000m_CL_1/15TVK830815_201203_0x3000m_CL_1.j2w")
> failed: unable to open file.
> ERROR 4:
>
`/Users/david/projects/mpls_watersheds/data/15TVK830830_201203_0x3000m_CL_1/15TVK830830_201203_0x3000m_CL_1.jp2'
> not recognised as a supported file format.
>
> gdalinfo 15TVK830815_201203_0x3000m_CL_1.jp2
> Driver: JP2OpenJPEG/JPEG-2000 driver based on OpenJPEG library
> Files: 15TVK830815_201203_0x3000m_CL_1.jp2
>        15TVK830815_201203_0x3000m_CL_1.j2w
> Size is 5000, 5000
> Coordinate System is:
> PROJCS["NAD83 / UTM zone 15N",
>     GEOGCS["NAD83",
>         DATUM["North_American_Datum_1983",
>             SPHEROID["GRS 1980",6378137,298.2572221010002,
>                 AUTHORITY["EPSG","7019"]],
>             AUTHORITY["EPSG","6269"]],
>         PRIMEM["Greenwich",0],
>         UNIT["degree",0.0174532925199433],
>         AUTHORITY["EPSG","4269"]],
>     PROJECTION["Transverse_Mercator"],
>     PARAMETER["latitude_of_origin",0],
>     PARAMETER["central_meridian",-93],
>     PARAMETER["scale_factor",0.9996],
>     PARAMETER["false_easting",500000],
>     PARAMETER["false_northing",0],
>     UNIT["metre",1,
>         AUTHORITY["EPSG","9001"]],
>     AUTHORITY["EPSG","26915"]]
> Origin = (483000.000000000000000,4983000.000000000000000)
> Pixel Size = (0.300000000000012,-0.300000000000186)
> Image Structure Metadata:
>   INTERLEAVE=PIXEL
> Corner Coordinates:
> Upper Left  (  483000.000, 4983000.000) ( 93d12'56.50"W, 45d 0' 0.87"N)
> Lower Left  (  483000.000, 4981500.000) ( 93d12'56.32"W, 44d59'12.26"N)
> Upper Right (  484500.000, 4983000.000) ( 93d11'47.99"W, 45d 0' 1.00"N)
> Lower Right (  484500.000, 4981500.000) ( 93d11'47.82"W, 44d59'12.39"N)
> Center      (  483750.000, 4982250.000) ( 93d12'22.16"W, 44d59'36.63"N)
> Band 1 Block=1024x1024 Type=Byte, ColorInterp=Red
>   Overviews: 2500x2500, 1250x1250, 625x625, 312x312, 156x156
>   Overviews: arbitrary
> Band 2 Block=1024x1024 Type=Byte, ColorInterp=Green
>   Overviews: 2500x2500, 1250x1250, 625x625, 312x312, 156x156
>   Overviews: arbitrary
> Band 3 Block=1024x1024 Type=Byte, ColorInterp=Blue
>   Overviews: 2500x2500, 1250x1250, 625x625, 312x312, 156x156
>   Overviews: arbitrary
>
> The .vrt file contents are here:
> https://gist.github.com/fawcett/c8cd5d271b0a89e67779
>
> Does anyone have a suggestion on how to troubleshoot this?  I am wondering
> if I am just running out of memory.

I guess you might rather run out of file descriptors. What is the default limit
on Mac : 1024 as on Linux ?
For each opened JP2 by the OpenJPEG driver, it maintains a file descriptor per
overview level (and for the main level). So in your above case, 6 file
descriptors.
The VRT mechanism shouldn't open simultaneously more than
GDAL_MAX_DATASET_POOL_SIZE=100 datasets
(http://trac.osgeo.org/gdal/browser/trunk/gdal/gcore/gdalproxypool.cpp#L313).
100 * 6 = 600... So if you have JP2 images bigger than the one you sampled, you
might have more overview levels. Or possibly the limit on Mac is lower than
1024.
You might want to try setting the GDAL_MAX_DATASET_POOL_SIZE configuration
option/environmenet variable to a lower value, let's say 10, and see if it works
better.

Even