<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.xmsonormal, li.xmsonormal, div.xmsonormal
{mso-style-name:x_msonormal;
margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.xgmail-p1, li.xgmail-p1, div.xgmail-p1
{mso-style-name:x_gmail-p1;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.xgmail-s1
{mso-style-name:x_gmail-s1;}
span.Shkpostityyli27
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 2.0cm 70.85pt 2.0cm;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1036269731;
mso-list-type:hybrid;
mso-list-template-ids:1097921174 -2001720808 67829763 67829765 67829761 67829763 67829765 67829761 67829763 67829765;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:2038266334;
mso-list-type:hybrid;
mso-list-template-ids:-314700196 1716556190 67829763 67829765 67829761 67829763 67829765 67829761 67829763 67829765;}
@list l1:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:Calibri;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FI" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Let’s try to get all missing information with one mail.<o:p></o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo2"><span lang="EN-US" style="mso-fareast-language:EN-US">You have two images:
<br>
3_data_ready_for_spectra/coastal-undefended-rcp26-2020.tif and<o:p></o:p></span></li></ul>
<p class="MsoListParagraph"><span lang="EN-US" style="mso-fareast-language:EN-US">5_UK_prod_rasters/coastal-undefended-rcp26-2020.tif.<o:p></o:p></span></p>
<p class="MsoListParagraph"><span lang="EN-US" style="mso-fareast-language:EN-US">Is that right? And the first one covers the world but the second one only UK, right?<o:p></o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l1 level1 lfo1"><span lang="EN-US" style="mso-fareast-language:EN-US">What are the pixel sizes and image sizes (in pixels, width x hight) of the source images?<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l1 level1 lfo1"><span lang="EN-US" style="mso-fareast-language:EN-US">Are they compressed? With what method?<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l1 level1 lfo1"><span lang="EN-US" style="mso-fareast-language:EN-US">What are the block sizes (out of curiosity, not essential knowledge)?<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l1 level1 lfo1"><span lang="EN-US" style="mso-fareast-language:EN-US">Have you considered to create a tiled and compressed output? (you should)<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l1 level1 lfo1"><span lang="EN-US" style="mso-fareast-language:EN-US">Have you considered to use gdalwarp instead? (I would)<o:p></o:p></span></li></ul>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">With the gdalwarp option you could update the world image with the UK contents with “gdalwarp uk_image.tif world_image.tif”. However, your images are in AWS and the same recipes that
work for local images may not work so well in the cloud. Unfortunately that is something that I have no experience yet.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">The logic in “The input files are 6.9 GB + 672 MB
</span><span lang="EN-US">The output should be 7.xx GB” is wrong. If you update part of the pixels of the 6.9 GB file then the output should be about 6.9 GB. But if you convert a 6.9 GB compressed image into an uncompressed image, as I guess that you are doing,
the output will be much bigger. How big it will be can be calculated with a formula that numpy showed (width x height) x (bytes/pixel).
</span><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">-Jukka Rahkonen-<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>Lähettäjä:</b> Clive Swan <cliveswan@gmail.com> <br>
<b>Lähetetty:</b> maanantai 5. joulukuuta 2022 13.47<br>
<b>Vastaanottaja:</b> Rahkonen Jukka <jukka.rahkonen@maanmittauslaitos.fi>; gdal-dev@lists.osgeo.org<br>
<b>Aihe:</b> Re: [gdal-dev] gdal_merge<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The raster <span style="color:black;background:white">one </span>is the world, the second is the UK.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">The UK fits within the first raster...<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="color:#212121"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="color:#212121">We want to update UK data in raster one.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="color:#212121"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="color:#212121">Regards,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="color:#212121"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="color:#212121">Clive<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div id="ms-outlook-mobile-signature">
<p class="MsoNormal">Get <a href="https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C405531fd73e74c5add7208dad6b66773%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638058376154245151%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3iBA3SB0%2BBSUKMk8laLw5Gi7%2BnDzhHsoqTpjnfznUoY%3D&reserved=0">
Outlook for Android</a><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="98%" align="center">
</div>
<div id="divRplyFwdMsg">
<p class="MsoNormal"><b><span style="color:black">From:</span></b><span style="color:black"> Rahkonen Jukka <<a href="mailto:jukka.rahkonen@maanmittauslaitos.fi">jukka.rahkonen@maanmittauslaitos.fi</a>><br>
<b>Sent:</b> Monday, December 5, 2022 11:34:53 AM<br>
<b>To:</b> Clive Swan <<a href="mailto:cliveswan@gmail.com">cliveswan@gmail.com</a>>;
<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a> <<a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a>><br>
<b>Subject:</b> Re: [gdal-dev] gdal_merge</span> <o:p></o:p></p>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="xmsonormal">Hi,<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal"><span lang="EN-US">Maybe your rasters are far apart. Gdal_merge creates a raster with so large extent that all the source images fit within it. Even empty pixels take space.</span><o:p></o:p></p>
<p class="xmsonormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="xmsonormal"><span lang="EN-US">-Jukka Rahkonen-</span><o:p></o:p></p>
<p class="xmsonormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="xmsonormal"><span lang="EN-US"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="xmsonormal"><b>Lähettäjä:</b> gdal-dev <<a href="mailto:gdal-dev-bounces@lists.osgeo.org">gdal-dev-bounces@lists.osgeo.org</a>>
<b>Puolesta </b>Clive Swan<br>
<b>Lähetetty:</b> maanantai 5. joulukuuta 2022 12.24<br>
<b>Vastaanottaja:</b> <a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><br>
<b>Aihe:</b> [gdal-dev] gdal_merge<o:p></o:p></p>
</div>
<p class="xmsonormal"> <o:p></o:p></p>
<div>
<p class="xmsonormal">Hi Even,<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal">Thanks <br>
the –configuration sorted out that issue.<o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:10.0pt;font-family:"Courier New"">CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE is a configuration option /
</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:10.0pt;font-family:"Courier New"">environment variable, so you have to pass it with --config
</span><o:p></o:p></p>
<p class="xmsonormal"><span style="font-size:10.0pt;font-family:"Courier New"">CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES</span><o:p></o:p></p>
<p class="xmsonormal"><br>
<br>
Now I am getting a memory issue?? << The input files are 6.9 GB + 672 MB<br>
The output should be 7.xx GB<o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xgmail-p1" style="mso-margin-top-alt:5.0pt;margin-right:0cm;margin-bottom:0cm;margin-left:21.7pt">
<span class="xgmail-s1"><span style="font-size:9.0pt;font-family:"Menlo",serif;color:purple">6.9 GB + 672MB</span></span><o:p></o:p></p>
<p class="xgmail-p1" style="mso-margin-top-alt:5.0pt;margin-right:0cm;margin-bottom:0cm;margin-left:21.7pt">
<span class="xgmail-s1"><span style="font-size:9.0pt;font-family:"Menlo",serif;color:purple">numpy.core._exceptions._ArrayMemoryError: Unable to allocate 377. GiB for an array with shape (225000, 450000) and data type float32</span></span><o:p></o:p></p>
<p class="xmsonormal"> <o:p></o:p></p>
<p class="xmsonormal" style="margin-bottom:12.0pt">Any suggestions appreciated<o:p></o:p></p>
<p class="xmsonormal">Kind regards,<o:p></o:p></p>
<p class="xmsonormal">Clive<o:p></o:p></p>
<p class="xmsonormal"><br clear="all">
<br>
-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p style="margin:0cm"><span style="font-size:8.0pt"> Regards,</span><o:p></o:p></p>
<p style="margin:0cm"> <o:p></o:p></p>
<p style="margin:0cm"><span style="font-size:8.0pt">Clive Swan</span><o:p></o:p></p>
<p style="margin:0cm"><span style="font-size:12.0pt">--</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>