<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.Shkpostityyli19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 2.0cm 70.85pt 2.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FI" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">It looks like the Python script gdal_merge.py wants to read all the data into a single array, and that is a big array because the image that covers the whole world with
</span><span lang="EN-US">Pixel Size = (0.000800000000000,-0.000800000000000) is big, 450000 by 225000 pixels. Probably gdal_merge is not the right tool for your use case. I would have a try with gdalwarp. Use the big image (or a copy of that) as a target and
warp the small one into it. The result may not have the smallest possible file size one because it is not so simple to update a compressed image with new compressed data.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">You provided gdalinfo from the vrt but could you provide gdalinfo also from the real files:<br>
/vsis3/summer-outputs/3_data_ready_for_spectra/coastal-undefended-rcp26-2020.tif<br>
/vsis3/summer-outputs/5_UK_prod_rasters/coastal-undefended-rcp26-2020.tif<br>
<br>
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">-Jukka Rahkonen-<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>Lähettäjä:</b> gdal-dev <gdal-dev-bounces@lists.osgeo.org>
<b>Puolesta </b>Clive Swan<br>
<b>Lähetetty:</b> maanantai 12. joulukuuta 2022 23.49<br>
<b>Vastaanottaja:</b> gdal-dev@lists.osgeo.org<br>
<b>Aihe:</b> [gdal-dev] gdal_merge tries create 377 GB raster<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">/data/coastal.tif should be 6.9GB<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I get weird output from gdal_merge??<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
gdal_merge.py -o /data/coastal.tif -n -9999 -co BIGTIFF=YES -co COMPRESS=LZW -co BLOCKXSIZE=128 -co BLOCKYSIZE=128 -co NUM_THREADS=ALL_CPUS --config CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES --config GDAL_SWATH_SIZE 1000000 /vsis3/summer/1/coastal.tif
/vsis3/summer/2/coastal.tif & disown -h<br>
[1] 507311<br>
ubuntu@ip-172-31-31-16:~$ 0Traceback (most recent call last):<br>
File "/usr/bin/gdal_merge.py", line 11, in <module><br>
sys.exit(main(sys.argv))<br>
File "/usr/lib/python3/dist-packages/osgeo_utils/gdal_merge.py", line 530, in main<br>
fi.copy_into(t_fh, band, band, nodata, verbose)<br>
File "/usr/lib/python3/dist-packages/osgeo_utils/gdal_merge.py", line 284, in copy_into<br>
return raster_copy(s_fh, sw_xoff, sw_yoff, sw_xsize, sw_ysize, s_band,<br>
File "/usr/lib/python3/dist-packages/osgeo_utils/gdal_merge.py", line 56, in raster_copy<br>
return raster_copy_with_nodata(<br>
File "/usr/lib/python3/dist-packages/osgeo_utils/gdal_merge.py", line 96, in raster_copy_with_nodata<br>
data_src = s_band.ReadAsArray(s_xoff, s_yoff, s_xsize, s_ysize,<br>
File "/usr/lib/python3/dist-packages/osgeo/gdal.py", line 3653, in ReadAsArray<br>
return gdal_array.BandReadAsArray(self, xoff, yoff,<br>
File "/usr/lib/python3/dist-packages/osgeo/gdal_array.py", line 422, in BandReadAsArray<br>
buf_obj = numpy.empty([buf_ysize, buf_xsize], dtype=typecode)<br>
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 377. GiB for an array with shape (225000, 450000) and data type float32<br>
<br>
<br>
-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p style="margin:0cm"><span style="font-size:8.0pt"> Regards,</span><o:p></o:p></p>
<p style="margin:0cm"><o:p> </o:p></p>
<p style="margin:0cm"><span style="font-size:8.0pt">Clive Swan</span><o:p></o:p></p>
<p style="margin:0cm"><span style="font-size:12.0pt">--</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>