[gdal-dev] gdal_translate .vrt issue

Tue May 20 10:08:21 PDT 2014

Le mardi 20 mai 2014 18:18:37, David Fawcett a écrit :
> As usual Even, spot on!
> 
> Thank you for precise diagnosis and the quick solution.  My project is back
> on track and the product can be delivered soon.  I am guessing that I could
> bump the pool_size up somewhat.  I plan to play around with that value and
> maybe get a little more performance (on my old Mac laptop...).

The value of the pool size might not be important in some cases. It is highly 
dependant on the read pattern. And GDAL_CACHEMAX will also play a role.

> 
> Thanks again,
> 
> David.
> 
> 
> On Tue, May 20, 2014 at 3:37 AM, Even Rouault
> 
> <even.rouault at mines-paris.org>wrote:
> > Selon David Fawcett <david.fawcett at gmail.com>:
> > > I am using gdal_translate and a .vrt to take 115 .jp2 files from the
> > 
> > USGS,
> > 
> > > pull them together and then export a lower resolution .jp2.
> > > 
> > > This is with GDAL 1.10 on OSX.
> > > gdalinfo --version
> > > GDAL 1.10.1, released 2013/08/26
> > > 
> > > When I run gdal_translate, it gets about 30% of the way through the
> > 
> > process
> > 
> > > and errors out saying that it can't open one of the .j2w files.  When I
> > 
> > run
> > 
> > > gdalinfo on the file, I can open it, and if I run gdal_translate on
> > > just that file, I can successfully export a reduced resolution copy.
> > > 
> > > The output of gdalinfo is here:
> > > https://gist.github.com/fawcett/beead31741e24394b38e
> > > 
> > > I can also load the .vrt image into QGIS 2.2 and it loads (slowly of
> > > course).
> > 
> > Even when you zoom out at 10% like your gdal_translate command ?
> > 
> > > gdal_translate img_index.vrt mpls34.jp2 -outsize 10% 10% -of
> > > JP2OpenJPEG -co QUALITY=25 -epo
> > > Input file size is 45000, 65000
> > 
> > > 0...10...20.ERROR 4:
> > CSLLoad2("/Users/david/projects/mpls_watersheds/data/15TVK830815_201203_0
> > x3000m_CL_1/15TVK830815_201203_0x3000m_CL_1.j2w")
> > 
> > > failed: unable to open file.
> > 
> > > ERROR 4:
> > `/Users/david/projects/mpls_watersheds/data/15TVK830830_201203_0x3000m_CL
> > _1/15TVK830830_201203_0x3000m_CL_1.jp2'
> > 
> > > not recognised as a supported file format.
> > > 
> > > gdalinfo 15TVK830815_201203_0x3000m_CL_1.jp2
> > > Driver: JP2OpenJPEG/JPEG-2000 driver based on OpenJPEG library
> > > Files: 15TVK830815_201203_0x3000m_CL_1.jp2
> > > 
> > >        15TVK830815_201203_0x3000m_CL_1.j2w
> > > 
> > > Size is 5000, 5000
> > > Coordinate System is:
> > > PROJCS["NAD83 / UTM zone 15N",
> > > 
> > >     GEOGCS["NAD83",
> > >     
> > >         DATUM["North_American_Datum_1983",
> > >         
> > >             SPHEROID["GRS 1980",6378137,298.2572221010002,
> > >             
> > >                 AUTHORITY["EPSG","7019"]],
> > >             
> > >             AUTHORITY["EPSG","6269"]],
> > >         
> > >         PRIMEM["Greenwich",0],
> > >         UNIT["degree",0.0174532925199433],
> > >         AUTHORITY["EPSG","4269"]],
> > >     
> > >     PROJECTION["Transverse_Mercator"],
> > >     PARAMETER["latitude_of_origin",0],
> > >     PARAMETER["central_meridian",-93],
> > >     PARAMETER["scale_factor",0.9996],
> > >     PARAMETER["false_easting",500000],
> > >     PARAMETER["false_northing",0],
> > >     UNIT["metre",1,
> > >     
> > >         AUTHORITY["EPSG","9001"]],
> > >     
> > >     AUTHORITY["EPSG","26915"]]
> > > 
> > > Origin = (483000.000000000000000,4983000.000000000000000)
> > > Pixel Size = (0.300000000000012,-0.300000000000186)
> > > 
> > > Image Structure Metadata:
> > >   INTERLEAVE=PIXEL
> > > 
> > > Corner Coordinates:
> > > Upper Left  (  483000.000, 4983000.000) ( 93d12'56.50"W, 45d 0' 0.87"N)
> > > Lower Left  (  483000.000, 4981500.000) ( 93d12'56.32"W, 44d59'12.26"N)
> > > Upper Right (  484500.000, 4983000.000) ( 93d11'47.99"W, 45d 0' 1.00"N)
> > > Lower Right (  484500.000, 4981500.000) ( 93d11'47.82"W, 44d59'12.39"N)
> > > Center      (  483750.000, 4982250.000) ( 93d12'22.16"W, 44d59'36.63"N)
> > > Band 1 Block=1024x1024 Type=Byte, ColorInterp=Red
> > > 
> > >   Overviews: 2500x2500, 1250x1250, 625x625, 312x312, 156x156
> > >   Overviews: arbitrary
> > > 
> > > Band 2 Block=1024x1024 Type=Byte, ColorInterp=Green
> > > 
> > >   Overviews: 2500x2500, 1250x1250, 625x625, 312x312, 156x156
> > >   Overviews: arbitrary
> > > 
> > > Band 3 Block=1024x1024 Type=Byte, ColorInterp=Blue
> > > 
> > >   Overviews: 2500x2500, 1250x1250, 625x625, 312x312, 156x156
> > >   Overviews: arbitrary
> > > 
> > > The .vrt file contents are here:
> > > https://gist.github.com/fawcett/c8cd5d271b0a89e67779
> > > 
> > > Does anyone have a suggestion on how to troubleshoot this?  I am
> > 
> > wondering
> > 
> > > if I am just running out of memory.
> > 
> > I guess you might rather run out of file descriptors. What is the default
> > limit
> > on Mac : 1024 as on Linux ?
> > For each opened JP2 by the OpenJPEG driver, it maintains a file
> > descriptor per
> > overview level (and for the main level). So in your above case, 6 file
> > descriptors.
> > The VRT mechanism shouldn't open simultaneously more than
> > GDAL_MAX_DATASET_POOL_SIZE=100 datasets
> > (
> > http://trac.osgeo.org/gdal/browser/trunk/gdal/gcore/gdalproxypool.cpp#L31
> > 3 ).
> > 100 * 6 = 600... So if you have JP2 images bigger than the one you
> > sampled, you
> > might have more overview levels. Or possibly the limit on Mac is lower
> > than 1024.
> > You might want to try setting the GDAL_MAX_DATASET_POOL_SIZE
> > configuration option/environmenet variable to a lower value, let's say
> > 10, and see if it works
> > better.
> > 
> > Even

-- 
Geospatial professional services
http://even.rouault.free.fr/services.html