[gdal-dev] gdal_merge on large files

[Even Rouault]

Peter,

In fact, tiling the input or output won't help for the issue. The main 
limitation of gdal_merge.py is that it ingests each input file in one gulp. 
So the limit is on the dimensions of the input files. The buffer must hold in 
RAM and be smaller than 2GB (the maximum size of a 32bit integer).

You should be able to generate a huge output file provided that each of your 
input files are reasonnably of modest dimensions. Likely the case for what 
Greg reported.

You might want to try gdalwarp or (gdalbuildvrt + gdal_translate) instead that 
have similar potential to gdal_merge.py but not that limitation.

Best regards,

Even

Le Thursday 28 January 2010 19:17:05 Chaitanya kumar CH, vous avez écrit :
> Peder,
>
> The problem here is the lack of or very large tiles/blocks in the image.
> BandReadAsArray() was asked to read a whole 27989 x 62863 block into
> memory. That's 1.76 Giga pixels. Too big.
>
> Greg Coats' tiff file had tiles of 512 x 512 size.
>
> Creating tiled tiffs is a good place to start. Use the -co TILED=YES option
> in gdal_translate to create tiled tiffs.
>
> Try Tim Sutton's page
> http://linfiniti.com/2009/09/image-mosaicking-with-gdal/ on his experience
> in image mosaicing.
>
> On Thu, Jan 28, 2010 at 1:43 PM, Peder Axensten <peder at axensten.se> wrote:
> > I want to run gdal_merge (and another program I made, based on
> > gdal_merge) on rather large files -- typically 3-7 Gbytes. Not
> > surprisingly I run into problems. I'm not very experienced in Python, but
> > am I right to assume that the error below probably is an indication on a
> > size problem? What is the raster size restrictions in gdal/numpy?
> >
> > If this is a size problem, I thought I'd modify gdal_merge to process big
> > rasters by parts, probably stealing ideas from gdal_calc3 by Chris
> > Yesson.
> >
> > Sorry if I'm not very specific, I'd appreciate any ideas from you
> > experienced guys.
> >
> > Best regards,
> >
> > Peder Axensten
> > Research engineer
> >
> > Swedish University of Agricultural Sciences
> > Forest Resource Management
> > Remote Sensing Laboratory
> >
> >
> >
> > mosaic$ gdal_merge.py -o test.img -n 65565 sverige_VolumeBirch.img
> > sverige_VolumeBeech.img sverige_VolumeOak.img
> > ERROR 2: Invalid dimensions : 27989 x 62863
> > Traceback (most recent call last):
> >  File
> > "/Library/Frameworks/Python.framework/general-site-packages/gdal-tools/gd
> >al_merge.py", line 431, in <module>
> >    fi.copy_into( t_fh, band, band, nodata )
> >  File
> > "/Library/Frameworks/Python.framework/general-site-packages/gdal-tools/gd
> >al_merge.py", line 228, in copy_into
> >    nodata_arg )
> >  File
> > "/Library/Frameworks/Python.framework/general-site-packages/gdal-tools/gd
> >al_merge.py", line 48, in raster_copy
> >    nodata )
> >  File
> > "/Library/Frameworks/Python.framework/general-site-packages/gdal-tools/gd
> >al_merge.py", line 84, in raster_copy_with_nodata
> >    t_xsize, t_ysize )
> >  File
> > "/Library/Frameworks/GDAL.framework/Versions/1.6/Python/site-packages/osg
> >eo/gdal.py", line 835, in ReadAsArray
> >    buf_xsize, buf_ysize, buf_obj )
> >  File
> > "/Library/Frameworks/GDAL.framework/Versions/1.6/Python/site-packages/osg
> >eo/gdal_array.py", line 140, in BandReadAsArray
> >    ar = numpy.reshape(ar, [buf_ysize,buf_xsize])
> >  File
> > "/Library/Frameworks/GDAL.framework/Versions/1.6/Python/site-packages/num
> >py/core/fromnumeric.py", line 152, in reshape
> >    return reshape(newshape, order=order)
> > ValueError: total size of new array must be unchanged
> >
> > _______________________________________________
> > gdal-dev mailing list
> > gdal-dev at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/gdal-dev