<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML-esimuotoiltu Char";
margin:0cm;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.HTML-esimuotoiltuChar
{mso-style-name:"HTML-esimuotoiltu Char";
mso-style-priority:99;
mso-style-link:HTML-esimuotoiltu;
font-family:Consolas;
mso-fareast-language:FI;}
span.Shkpostityyli21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 2.0cm 70.85pt 2.0cm;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:735930821;
mso-list-type:hybrid;
mso-list-template-ids:1908582342 1549036014 67829763 67829765 67829761 67829763 67829765 67829761 67829763 67829765;}
@list l0:level1
{mso-level-start-at:64;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l1
{mso-list-id:1170094591;
mso-list-template-ids:1273681086;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FI" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">You send new mails faster than I can read and understand the previous ones but here comes some quick comments.<o:p></o:p></span></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo2"><span lang="EN-US" style="mso-fareast-language:EN-US">The gdalbuildvrt command looks good but it would be kind to tell which file is international and which one is UK. It is not obvious
to me by the paths /5/ and /3/. You want to write the international file first into the VRT and UK after that to suit your workflow.<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo2"><span lang="EN-US" style="mso-fareast-language:EN-US">This does not make sense:
</span><span lang="EN-US">gdal_translate /data/coastal-2020.vrt /data/3/coastal-2020.tif /data/5/coastal-2020.tif. gdal_translate needs one input and one output, so: gdal_translate /data/coastal-2020.vrt output.tif</span><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo2"><span lang="EN-US">I think that gdal_translate does not have option -n</span><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo2"><span lang="EN-US" style="mso-fareast-language:EN-US">NUM_THREADS=ALL_CPUS is not a creation option -co, but an open option -oo, or it can be used as a configuration option
<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo2"><span lang="EN-US" style="mso-fareast-language:EN-US">It may be that -co BLOCKXSIZE=128 -co BLOCKYSIZE=128 can really be used without -co TILED=YES. I have never tried. It is still
essential that the output is tiled.<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo2"><span lang="EN-US" style="mso-fareast-language:EN-US">Without testing I am not sure if the issues above make the input rubbish but it is possible.<o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo2"><span lang="EN-US" style="mso-fareast-language:EN-US">Your second and third trials are doomed to do something else than what you want very soon:
</span><span lang="EN-US">gdalwarp -r near -overwrite. Overwrite switch means that the existing outputfile will be deleted and a new one with the same name will be created. You wanted just update the target file so don’t use -overwrite. And when you use an
existing file as a target the creation options have no effect; the target file is already created with some options.</span><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p></o:p></span></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo2"><span lang="EN-US">Now you understand why The output is: 18376, 17086
<b>not</b> 450000, 225000. (if not I can give a hint: what file did the -overwrite delete?)</span><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p></o:p></span></li></ul>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">I suppose that you start getting frustrated because testing all kind of random commands with huge images takes a long time (“</span><span lang="EN-US">The AWS Instance with over 60 VCPU
ran for over 8 hours”). However, you could have made all your mistakes much faster with much smaller files (1000 by 1000 pixels for example). Make the commands work with small images and when you are satisfied test them with bigger images. That said, I cannot
promise that you will have good performance with updating 450000, 225000 sized LZW compressed GeoTIFF. I have never tried anything like that with such a big image myself.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">-Jukka Rahkonen-<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>Lähettäjä:</b> gdal-dev <gdal-dev-bounces@lists.osgeo.org>
<b>Puolesta </b>Clive Swan<br>
<b>Lähetetty:</b> keskiviikko 14. joulukuuta 2022 18.34<br>
<b>Vastaanottaja:</b> gdal-dev@lists.osgeo.org<br>
<b>Aihe:</b> Re: [gdal-dev] gdalwarp running very slow<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
I want to <b>APPEND</b> the UK data into the international.tif<br>
The updated international size should also be: 450000, 225000<br>
<br>
<b>I first tried </b><br>
gdalbuildvrt -o /data/coastal-2020.vrt /vsis3/summer/3/coastal-2020.tif /vsis3/summer/5/coastal-2020.tif
<br>
<br>
gdal_translate /data/coastal-2020.vrt /data/3/coastal-2020.tif /data/5/coastal-2020.tif -n -9999 -co BIGTIFF=YES -co COMPRESS=LZW -co BLOCKXSIZE=128 -co BLOCKYSIZE=128 -co NUM_THREADS=ALL_CPUS --config CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES --config
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><b>The output was rubbish</b><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The UK image size is: 18376, 17086<o:p></o:p></p>
</div>
<p class="MsoNormal">The international size is: 450000, 225000<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I tried<br>
/data/3/coastal-2020-test.tif = 7GB<br>
/data/5/coastal-2020.tif = 700MB<br>
<br>
gdalwarp -r near -overwrite /data/3/coastal-2020.tif /data/3/coastal-2020-test1.tif -co BIGTIFF=YES -co COMPRESS=LZW -co BLOCKXSIZE=128 -co BLOCKYSIZE=128 -co NUM_THREADS=ALL_CPUS -co PREDICTOR=3 --config CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES & disown
-h<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The AWS Instance with over 60 VCPU ran for over 8 hours<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
</div>
<p class="MsoNormal">I tried:<o:p></o:p></p>
<div>
<p class="MsoNormal">/data/5/coastal-2020.tif = 700MB<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">/data/3/coastal-2020-test.tif = 7GB<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
gdalwarp -r near -overwrite /data/5/coastal-2020.tif /data/3/coastal-2020-test.tif -co BIGTIFF=YES -co COMPRESS=LZW -co BLOCKXSIZE=128 -co BLOCKYSIZE=128 -co NUM_THREADS=ALL_CPUS -co PREDICTOR=3 --config CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES<o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The output is: 18376, 17086 <b>not</b> 450000, 225000<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Any assistance appreciated<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Thanks<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Clive<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">On Wed, 14 Dec 2022 at 09:23, Rahkonen Jukka <<a href="mailto:jukka.rahkonen@maanmittauslaitos.fi" target="_blank">jukka.rahkonen@maanmittauslaitos.fi</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">I don’t mean that you should try this and that blindly but to describe what data you have in your hands and what you are planning to do with it so that the other
GDAL users could consider what reasonable alternatives you could have. I have never done anything that is even close to your use case but due to other experience I can see potential issues in a few places:</span><o:p></o:p></p>
<ul type="disc">
<li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l1 level1 lfo1">
<span lang="EN-US">You try to update image A that has a size 450000 by 225000 pixels with image B that has the same size. The result would be A updated into a full copy of B if all pixels in B are valid.</span><o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l1 level1 lfo1">
<span lang="EN-US">However, image B probably has very much NoData (we do not know because you have not told that) and if GDAL deals with NoData correctly the result would be A updated with valid pixels from B and that is probably what is desired.</span><o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l1 level1 lfo1">
<span lang="EN-US">However, we do not know how effectively GDAL skips the nodata pixels of B. It may be fast or not. If we know that most part of the world is NoData it might be good to crop image B to include just the area where there is data. That’s maybe
UK in your case. If skipping the NoData is fast then cropping won’t give speedup but it is cheap to test.</span><o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l1 level1 lfo1">
<span lang="EN-US">You have compressed images. LZW algorithm is compressing some data more effectively than some other. If you expect that you can replace a chunk of LZW compressed data inside a TIFF file with another chunk of LZW compressed data in place you
are wrong. The new chunk of data may be larger and it just cannot fit into the same space. Assumption that updating a 6 GB image with 600 MB new data would yield a 6 GB image is not correct with compressed data.</span><o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l1 level1 lfo1">
<span lang="EN-US">I can imagine that there could be other technical reasons to write the replacing data at the end of the existing TIFF and update the image directories. If the image size is critical it may require re-writing the updated TIFF into a new TIFF
file. The complete re-write can be done in most optimal way. See this wiki page <a href="https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftrac.osgeo.org%2Fgdal%2Fwiki%2FUserDocs%2FGdalWarp%23GeoTIFFoutput-coCOMPRESSisbroken&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C5b67ce172b5b4c48990108daddf10a33%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638066324581688598%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JyLbYpMyvjON3BsPZ9INgBNAy8DUGMwrJ2TVv6%2BhGHY%3D&reserved=0" target="_blank">
https://trac.osgeo.org/gdal/wiki/UserDocs/GdalWarp#GeoTIFFoutput-coCOMPRESSisbroken</a></span><o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l1 level1 lfo1">
<span lang="EN-US">If the images are in AWS it is possible that the process should be somehow different than with local images. I have no experience about AWS yet.</span><o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l1 level1 lfo1">
<span lang="EN-US">A 450000 by 225000 image is rather big. It is possible that it would be faster to split the image into smaller parts, update the parts that need updating, and combine the parts back into a big image. Or keep the parts and combine them virtually
with gdalbuildvrt into VRT.</span><o:p></o:p></li></ul>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Your use case is not so usual and it is rather heavy but there are certainly several ways to do what you want. What should be avoided it to select an inefficient
method and try to optimize it.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">Good luck with your experiments,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US">-Jukka-</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:18.0pt">
<span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:18.0pt">
<span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:18.0pt">
<span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span lang="EN-US"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0cm 0cm 0cm;border-color:currentcolor currentcolor">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>Lähettäjä:</b> Clive Swan <<a href="mailto:cliveswan@gmail.com" target="_blank">cliveswan@gmail.com</a>>
<br>
<b>Lähetetty:</b> keskiviikko 14. joulukuuta 2022 10.29<br>
<b>Vastaanottaja:</b> Rahkonen Jukka <<a href="mailto:jukka.rahkonen@maanmittauslaitos.fi" target="_blank">jukka.rahkonen@maanmittauslaitos.fi</a>><br>
<b>Aihe:</b> Re: [gdal-dev] gdalwarp running very slow<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi Jukka,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="color:#212121"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="color:#212121">Thanks for that, was really stressed.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="color:#212121">I will export the UK extent, and rerun the script.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="color:#212121"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="color:#212121">Thanks</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="color:#212121">Clive</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div id="m_346335319753308561m_5240173376920134995ms-outlook-mobile-signature">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Sent from
<a href="https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C5b67ce172b5b4c48990108daddf10a33%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638066324581688598%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=VrXwCU6Hs%2FdjSHmxQBVIXJI30tUikSCe36HZaTNpiYA%3D&reserved=0" target="_blank">
Outlook for Android</a><o:p></o:p></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="1" width="98%" align="center">
</div>
<div id="m_346335319753308561m_5240173376920134995divRplyFwdMsg">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="color:black">From:</span></b><span style="color:black"> Rahkonen Jukka <<a href="mailto:jukka.rahkonen@maanmittauslaitos.fi" target="_blank">jukka.rahkonen@maanmittauslaitos.fi</a>><br>
<b>Sent:</b> Wednesday, December 14, 2022 7:18:50 AM<br>
<b>To:</b> Clive Swan <<a href="mailto:cliveswan@gmail.com" target="_blank">cliveswan@gmail.com</a>>;
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a> <<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>><br>
<b>Subject:</b> Re: [gdal-dev] gdalwarp running very slow</span> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
<div>
<div>
<p>Hi,<o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span lang="EN-US">Thank you for the information about the source files. I do not yet understand what you are trying to do and why. The both images have the same size 450000 and 225000 and they cover the same area. Is the “image 5_UK_coastal-2020.tif” just
NoData with pixel value -9999 everywhere outside the UK? The name of the image makes me think so.</span><o:p></o:p></p>
<p><span lang="EN-US"> </span><o:p></o:p></p>
<p><span lang="EN-US">-Jukka Rahkonen-</span><o:p></o:p></p>
<p><span lang="EN-US"> </span><o:p></o:p></p>
<p><span lang="EN-US"> </span><o:p></o:p></p>
<div style="border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0cm 0cm 0cm;border-color:currentcolor currentcolor">
<p><b>Lähettäjä:</b> Clive Swan <<a href="mailto:cliveswan@gmail.com" target="_blank">cliveswan@gmail.com</a>>
<br>
<b>Lähetetty:</b> tiistai 13. joulukuuta 2022 19.22<br>
<b>Vastaanottaja:</b> <a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<b>Kopio:</b> Rahkonen Jukka <<a href="mailto:jukka.rahkonen@maanmittauslaitos.fi" target="_blank">jukka.rahkonen@maanmittauslaitos.fi</a>><br>
<b>Aihe:</b> [gdal-dev] gdalwarp running very slow<o:p></o:p></p>
</div>
<p> <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<p style="margin-bottom:12.0pt"> Greetings,<o:p></o:p></p>
</div>
<p>I am using the same files, I copied them from an AWS Bucket to a local AWS Instance.<o:p></o:p></p>
</div>
<p>I tried gdal_merge << tries to create 300GB file<o:p></o:p></p>
</div>
<p>I tried gdal_translate ran but created 2.5 GB not 6.9 GB file<o:p></o:p></p>
</div>
<div>
<p>Now trying gdalwarp.<o:p></o:p></p>
</div>
<div>
<p> <o:p></o:p></p>
</div>
<div>
<p>the gdalinfo is the same in both datasets:<o:p></o:p></p>
</div>
<div>
<pre>coastal-2020.tif (6.9GB)<o:p></o:p></pre>
</div>
<div>
<p>Driver: GTiff/GeoTIFF<br>
Size is 450000, 225000<br>
Coordinate System is:<br>
GEOGCRS["WGS 84",<br>
DATUM["World Geodetic System 1984",<br>
ELLIPSOID["WGS 84",6378137,298.257223563,<br>
LENGTHUNIT["metre",1]]],<br>
PRIMEM["Greenwich",0,<br>
ANGLEUNIT["degree",0.0174532925199433]],<br>
CS[ellipsoidal,2],<br>
AXIS["geodetic latitude (Lat)",north,<br>
ORDER[1],<br>
ANGLEUNIT["degree",0.0174532925199433]],<br>
AXIS["geodetic longitude (Lon)",east,<br>
ORDER[2],<br>
ANGLEUNIT["degree",0.0174532925199433]],<br>
ID["EPSG",4326]]<br>
Data axis to CRS axis mapping: 2,1<br>
Origin = (-180.000000000000000,90.000000000000000)<br>
Pixel Size = (0.000800000000000,-0.000800000000000)<br>
Metadata:<br>
AREA_OR_POINT=Area<br>
datetime_created=2022-11-14 18:05:14.053301<br>
Image Structure Metadata:<br>
COMPRESSION=LZW<br>
INTERLEAVE=BAND<br>
PREDICTOR=3<br>
Corner Coordinates:<br>
Upper Left (-180.0000000, 90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)<br>
Lower Left (-180.0000000, -90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"S)<br>
Upper Right ( 180.0000000, 90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)<br>
Lower Right ( 180.0000000, -90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"S)<br>
Center ( 0.0000000, 0.0000000) ( 0d 0' 0.01"E, 0d 0' 0.01"N)<br>
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray<br>
Description = score<br>
NoData Value=-9999<br>
Band 2 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = severity_value<br>
NoData Value=-9999<br>
Band 3 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = severity_min<br>
NoData Value=-9999<br>
Band 4 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = severity_max<br>
NoData Value=-9999<br>
Band 5 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = likelihood<br>
NoData Value=-9999<br>
Band 6 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = return_time<br>
NoData Value=-9999<br>
Band 7 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = likelihood_confidence<br>
NoData Value=-9999<br>
Band 8 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = climate_reliability<br>
NoData Value=-9999<br>
Band 9 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = hazard_reliability<br>
NoData Value=-9999<o:p></o:p></p>
</div>
<div>
<div>
<div>
<p> <o:p></o:p></p>
<pre>5_UK_coastal-2020.tif (600MB)<o:p></o:p></pre>
<div>
<div>
<p>Driver: GTiff/GeoTIFF<br>
Size is 450000, 225000<br>
Coordinate System is:<br>
GEOGCRS["WGS 84",<br>
DATUM["World Geodetic System 1984",<br>
ELLIPSOID["WGS 84",6378137,298.257223563,<br>
LENGTHUNIT["metre",1]]],<br>
PRIMEM["Greenwich",0,<br>
ANGLEUNIT["degree",0.0174532925199433]],<br>
CS[ellipsoidal,2],<br>
AXIS["geodetic latitude (Lat)",north,<br>
ORDER[1],<br>
ANGLEUNIT["degree",0.0174532925199433]],<br>
AXIS["geodetic longitude (Lon)",east,<br>
ORDER[2],<br>
ANGLEUNIT["degree",0.0174532925199433]],<br>
ID["EPSG",4326]]<br>
Data axis to CRS axis mapping: 2,1<br>
Origin = (-180.000000000000000,90.000000000000000)<br>
Pixel Size = (0.000800000000000,-0.000800000000000)<br>
Metadata:<br>
AREA_OR_POINT=Area<br>
datetime_created=2022-11-14 18:05:14.053301<br>
hostname=posix.uname_result(sysname='Linux', nodename='ip-172-31-12-125', release='5.15.0-1022-aws', version='#26~20.04.1-Ubuntu SMP Sat Oct 15 03:22:07 UTC 2022', machine='x86_64')<br>
Image Structure Metadata:<br>
COMPRESSION=LZW<br>
INTERLEAVE=BAND<br>
PREDICTOR=3<br>
Corner Coordinates:<br>
Upper Left (-180.0000000, 90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)<br>
Lower Left (-180.0000000, -90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"S)<br>
Upper Right ( 180.0000000, 90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)<br>
Lower Right ( 180.0000000, -90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"S)<br>
Center ( 0.0000000, 0.0000000) ( 0d 0' 0.01"E, 0d 0' 0.01"N)<br>
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray<br>
Description = score<br>
NoData Value=-9999<br>
Band 2 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = severity_value<br>
NoData Value=-9999<br>
Band 3 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = severity_min<br>
NoData Value=-9999<br>
Band 4 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = severity_max<br>
NoData Value=-9999<br>
Band 5 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = likelihood<br>
NoData Value=-9999<br>
Band 6 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = return_time<br>
NoData Value=-9999<br>
Band 7 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = likelihood_confidence<br>
NoData Value=-9999<br>
Band 8 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = climate_reliability<br>
NoData Value=-9999<br>
Band 9 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = hazard_reliability<br>
NoData Value=-9999<br clear="all">
<br>
-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p style="margin:0cm"><span style="font-size:8.0pt"> Regards,</span><o:p></o:p></p>
<p style="margin:0cm"> <o:p></o:p></p>
<p style="margin:0cm"><span style="font-size:8.0pt">Clive Swan</span><o:p></o:p></p>
<p style="margin:0cm"><span style="font-size:12.0pt">--</span><o:p></o:p></p>
<pre>Hi,<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>If you are still struggling with the same old problem could you please finally send the gdalinfo reports of your two input files which are this time:<o:p></o:p></pre>
<pre>coastal-2020.tif<o:p></o:p></pre>
<pre>5_UK_coastal-2020.tif<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>-Jukka Rahkonen-<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>Lähettäjä: gdal-dev <<a href="https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fgdal-dev&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C5b67ce172b5b4c48990108daddf10a33%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638066324581688598%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ksdZuGTu2MsFwMcQTQT1XXt4v8JHhDTDgbVbna0zfoA%3D&reserved=0" target="_blank">gdal-dev-bounces at lists.osgeo.org</a>> Puolesta Clive Swan<o:p></o:p></pre>
<pre>Lähetetty: tiistai 13. joulukuuta 2022 17.23<o:p></o:p></pre>
<pre>Vastaanottaja: <a href="https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fgdal-dev&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C5b67ce172b5b4c48990108daddf10a33%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638066324581688598%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ksdZuGTu2MsFwMcQTQT1XXt4v8JHhDTDgbVbna0zfoA%3D&reserved=0" target="_blank">gdal-dev at lists.osgeo.org</a><o:p></o:p></pre>
<pre>Aihe: [gdal-dev] gdalwarp running very slow<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>Greetings,<o:p></o:p></pre>
<pre>I am running gdalwarp on a 6GB (output) and 600MB (input) tif image, the AWS Instance has approx 60 VCPU<o:p></o:p></pre>
<pre>It has taken over 6 hours so far - still running, is it possible to optimise this and speed it up??<o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>gdalwarp -r near -overwrite coastal-2020.tif 5_UK_coastal-2020.tif -co BIGTIFF=YES -co COMPRESS=LZW -co BLOCKXSIZE=128 -co BLOCKYSIZE=128 -co NUM_THREADS=ALL_CPUS --config CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES<o:p></o:p></pre>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<br>
-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p style="margin:0cm"><span style="font-size:8.0pt"> Regards,</span><o:p></o:p></p>
<p style="margin:0cm"><o:p> </o:p></p>
<p style="margin:0cm"><span style="font-size:8.0pt">Clive Swan</span><o:p></o:p></p>
<p style="margin:0cm"><span style="font-size:12.0pt">--</span><o:p></o:p></p>
<p style="margin:0cm"><o:p> </o:p></p>
<p style="margin:0cm"><span style="font-size:12.0pt;color:gray">M: +44 7766 452665</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>