[gdal-dev] gdal_merge problems: Negative size passed to PyString_FromStringAndSize

[Andreas Neumann]

Hi Even,

Thank you for your very detailed explanation.

I can now see what the problem is and was able to work around it by
splitting up the larger files first that would exceed the maximum size.

Andreas

On Wed, May 6, 2009 9:36 pm, Even Rouault wrote:
> Le Wednesday 06 May 2009 16:48:06 Andreas Neumann, vous avez écrit :
>> Hi,
>> Filename: a25.tif
>> File Size: 25472x25427x1
>> Pixel Size: 42.431795 x -42.431795
>> UL:(697837651.400476,244274906.596461)
>> LR:(698918474.082743,243195993.344970)
>> Copy 0,0,25472,25427 to 205747,211643,50722,50632.
>
> --> here's the issue. a25.tif is resampled from 25472x25427 to a
> 50722x50632
> raster. But as gdal_merge.py just operates on full buffers, this results
> in
> trying to allocate 50722x50632 pixels = 2 568 156 304 bytes, which is
> above
> the largest 32 bit (signed) integer whose value is 2 147 483 647. So this
> value is interpreated as a negative size by Python...
>
> To make the story short, you didn't do anything wrong. You've just hit a
> known
> limitation of gdal_merge.py.
>
> The issue is that the default pixel size used for the destination merged
> file
> is the pixel size of the first source raster processed, in your case
> 21.309260 x -21.309260 for a01.tif. But a25.tif on which the error occurs
> has
> a pixel size of  42.431795 x -42.431795. So it must be nearly over-sampled
> by
> a factor of 2, so that accounts for his size going from 25472x25427 to a
> 50722x50632.
>
> There are several workarounds :
> - use the -ps flag of gdal_merge.py to increase the pixel size of the
> destination raster. "-ps 30 30" would probably work (provided that you've
> enough memory)
> - or add a preprocessing step that cuts a25.tif into 4 smaller pieces.
> This
> can be done with 4 calls to gdal_translate by using the -srcwin option
> - or use gdalwarp to do the same thing. It has not the limitation of
> gdal_merge.py, but it is known to be much slower.
>
> Patches to improve gdal_merge.py are welcome ;-)
>
>> Traceback (most recent call last):
>>   File "/usr/local/bin/gdal_merge.py", line 427, in <module>
>>     fi.copy_into( t_fh, band, band, nodata )
>>   File "/usr/local/bin/gdal_merge.py", line 227, in copy_into
>>     nodata_arg )
>>   File "/usr/local/bin/gdal_merge.py", line 47, in raster_copy
>>     nodata )
>>   File "/usr/local/bin/gdal_merge.py", line 83, in
>> raster_copy_with_nodata
>>     t_xsize, t_ysize )
>>   File
>> "/usr/lib/python2.5/site-packages/GDAL-1.6.0-py2.5-linux-x86_64.egg/osgeo/g
>>dal.py", line 835, in ReadAsArray
>>     buf_xsize, buf_ysize, buf_obj )
>>   File
>> "/usr/lib/python2.5/site-packages/GDAL-1.6.0-py2.5-linux-x86_64.egg/osgeo/g
>>dal_array.py", line 133, in BandReadAsArray
>>     buf_xsize, buf_ysize, datatype )
>>   File
>> "/usr/lib/python2.5/site-packages/GDAL-1.6.0-py2.5-linux-x86_64.egg/osgeo/g
>>dal.py", line 760, in ReadRaster
>>     return _gdal.Band_ReadRaster(*args, **kwargs)
>> SystemError: Negative size passed to PyString_FromStringAndSize
>> -------------------
>>
>> Is this a problem with my data, georeferencing or projection or is it a
>> problem in gdal_merge?
>>
>> Thanks for any idea what might be wrong in this case!
>>
>> Andreas
>
>
>


-- 
Andreas Neumann
http://www.carto.net/neumann/
http://www.svgopen.org/