[gdal-dev] gdal_merge tries create 377 GB raster

[Rahkonen Jukka]

Hi,

It looks like the Python script gdal_merge.py wants to read all the data into a single array, and that is a big array because the image that covers the whole world with Pixel Size = (0.000800000000000,-0.000800000000000) is big, 450000 by 225000 pixels. Probably gdal_merge is not the right tool for your use case. I would have a try with gdalwarp. Use the big image (or a copy of that) as a target and warp the small one into it. The result may not have the smallest possible file size one because it is not so simple to update a compressed image with new compressed data.

You provided gdalinfo from the vrt but could you provide gdalinfo also from the real files:
       /vsis3/summer-outputs/3_data_ready_for_spectra/coastal-undefended-rcp26-2020.tif
       /vsis3/summer-outputs/5_UK_prod_rasters/coastal-undefended-rcp26-2020.tif

-Jukka Rahkonen-



Lähettäjä: gdal-dev <gdal-dev-bounces at lists.osgeo.org> Puolesta Clive Swan
Lähetetty: maanantai 12. joulukuuta 2022 23.49
Vastaanottaja: gdal-dev at lists.osgeo.org
Aihe: [gdal-dev] gdal_merge tries create 377 GB raster

/data/coastal.tif  should be 6.9GB

I get weird output from gdal_merge??

gdal_merge.py -o /data/coastal.tif -n -9999 -co BIGTIFF=YES -co COMPRESS=LZW -co BLOCKXSIZE=128 -co BLOCKYSIZE=128  -co NUM_THREADS=ALL_CPUS --config CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES  --config GDAL_SWATH_SIZE 1000000  /vsis3/summer/1/coastal.tif /vsis3/summer/2/coastal.tif  & disown -h
[1] 507311
ubuntu at ip-172-31-31-16:~$ 0Traceback (most recent call last):
  File "/usr/bin/gdal_merge.py", line 11, in <module>
    sys.exit(main(sys.argv))
  File "/usr/lib/python3/dist-packages/osgeo_utils/gdal_merge.py", line 530, in main
    fi.copy_into(t_fh, band, band, nodata, verbose)
  File "/usr/lib/python3/dist-packages/osgeo_utils/gdal_merge.py", line 284, in copy_into
    return raster_copy(s_fh, sw_xoff, sw_yoff, sw_xsize, sw_ysize, s_band,
  File "/usr/lib/python3/dist-packages/osgeo_utils/gdal_merge.py", line 56, in raster_copy
    return raster_copy_with_nodata(
  File "/usr/lib/python3/dist-packages/osgeo_utils/gdal_merge.py", line 96, in raster_copy_with_nodata
    data_src = s_band.ReadAsArray(s_xoff, s_yoff, s_xsize, s_ysize,
  File "/usr/lib/python3/dist-packages/osgeo/gdal.py", line 3653, in ReadAsArray
    return gdal_array.BandReadAsArray(self, xoff, yoff,
  File "/usr/lib/python3/dist-packages/osgeo/gdal_array.py", line 422, in BandReadAsArray
    buf_obj = numpy.empty([buf_ysize, buf_xsize], dtype=typecode)
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 377. GiB for an array with shape (225000, 450000) and data type float32


--

 Regards,



Clive Swan

--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221213/82a1df11/attachment.htm>