<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML-esimuotoiltu Char";
margin:0cm;
font-size:10.0pt;
font-family:"Courier New";}
span.HTML-esimuotoiltuChar
{mso-style-name:"HTML-esimuotoiltu Char";
mso-style-priority:99;
mso-style-link:HTML-esimuotoiltu;
font-family:Consolas;
mso-fareast-language:FI;}
span.Shkpostityyli21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 2.0cm 70.85pt 2.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FI" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">Thank you for the information about the source files. I do not yet understand what you are trying to do and why. The both images have the same size
</span><span lang="EN-US">450000 and 225000 and they cover the same area. Is the “image 5_UK_coastal-2020.tif” just NoData with pixel value -9999 everywhere outside the UK? The name of the image makes me think so.</span><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US">-Jukka Rahkonen-<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>Lähettäjä:</b> Clive Swan <cliveswan@gmail.com> <br>
<b>Lähetetty:</b> tiistai 13. joulukuuta 2022 19.22<br>
<b>Vastaanottaja:</b> gdal-dev@lists.osgeo.org<br>
<b>Kopio:</b> Rahkonen Jukka <jukka.rahkonen@maanmittauslaitos.fi><br>
<b>Aihe:</b> [gdal-dev] gdalwarp running very slow<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> Greetings,<o:p></o:p></p>
</div>
<p class="MsoNormal">I am using the same files, I copied them from an AWS Bucket to a local AWS Instance.<o:p></o:p></p>
</div>
<p class="MsoNormal">I tried gdal_merge << tries to create 300GB file<o:p></o:p></p>
</div>
<p class="MsoNormal">I tried gdal_translate ran but created 2.5 GB not 6.9 GB file<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Now trying gdalwarp.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">the gdalinfo is the same in both datasets:<o:p></o:p></p>
</div>
<div>
<pre>coastal-2020.tif (6.9GB)<o:p></o:p></pre>
</div>
<div>
<p class="MsoNormal">Driver: GTiff/GeoTIFF<br>
Size is 450000, 225000<br>
Coordinate System is:<br>
GEOGCRS["WGS 84",<br>
DATUM["World Geodetic System 1984",<br>
ELLIPSOID["WGS 84",6378137,298.257223563,<br>
LENGTHUNIT["metre",1]]],<br>
PRIMEM["Greenwich",0,<br>
ANGLEUNIT["degree",0.0174532925199433]],<br>
CS[ellipsoidal,2],<br>
AXIS["geodetic latitude (Lat)",north,<br>
ORDER[1],<br>
ANGLEUNIT["degree",0.0174532925199433]],<br>
AXIS["geodetic longitude (Lon)",east,<br>
ORDER[2],<br>
ANGLEUNIT["degree",0.0174532925199433]],<br>
ID["EPSG",4326]]<br>
Data axis to CRS axis mapping: 2,1<br>
Origin = (-180.000000000000000,90.000000000000000)<br>
Pixel Size = (0.000800000000000,-0.000800000000000)<br>
Metadata:<br>
AREA_OR_POINT=Area<br>
datetime_created=2022-11-14 18:05:14.053301<br>
Image Structure Metadata:<br>
COMPRESSION=LZW<br>
INTERLEAVE=BAND<br>
PREDICTOR=3<br>
Corner Coordinates:<br>
Upper Left (-180.0000000, 90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)<br>
Lower Left (-180.0000000, -90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"S)<br>
Upper Right ( 180.0000000, 90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)<br>
Lower Right ( 180.0000000, -90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"S)<br>
Center ( 0.0000000, 0.0000000) ( 0d 0' 0.01"E, 0d 0' 0.01"N)<br>
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray<br>
Description = score<br>
NoData Value=-9999<br>
Band 2 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = severity_value<br>
NoData Value=-9999<br>
Band 3 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = severity_min<br>
NoData Value=-9999<br>
Band 4 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = severity_max<br>
NoData Value=-9999<br>
Band 5 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = likelihood<br>
NoData Value=-9999<br>
Band 6 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = return_time<br>
NoData Value=-9999<br>
Band 7 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = likelihood_confidence<br>
NoData Value=-9999<br>
Band 8 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = climate_reliability<br>
NoData Value=-9999<br>
Band 9 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = hazard_reliability<br>
NoData Value=-9999<o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<pre>5_UK_coastal-2020.tif (600MB)<o:p></o:p></pre>
<div>
<div>
<p class="MsoNormal">Driver: GTiff/GeoTIFF<br>
Size is 450000, 225000<br>
Coordinate System is:<br>
GEOGCRS["WGS 84",<br>
DATUM["World Geodetic System 1984",<br>
ELLIPSOID["WGS 84",6378137,298.257223563,<br>
LENGTHUNIT["metre",1]]],<br>
PRIMEM["Greenwich",0,<br>
ANGLEUNIT["degree",0.0174532925199433]],<br>
CS[ellipsoidal,2],<br>
AXIS["geodetic latitude (Lat)",north,<br>
ORDER[1],<br>
ANGLEUNIT["degree",0.0174532925199433]],<br>
AXIS["geodetic longitude (Lon)",east,<br>
ORDER[2],<br>
ANGLEUNIT["degree",0.0174532925199433]],<br>
ID["EPSG",4326]]<br>
Data axis to CRS axis mapping: 2,1<br>
Origin = (-180.000000000000000,90.000000000000000)<br>
Pixel Size = (0.000800000000000,-0.000800000000000)<br>
Metadata:<br>
AREA_OR_POINT=Area<br>
datetime_created=2022-11-14 18:05:14.053301<br>
hostname=posix.uname_result(sysname='Linux', nodename='ip-172-31-12-125', release='5.15.0-1022-aws', version='#26~20.04.1-Ubuntu SMP Sat Oct 15 03:22:07 UTC 2022', machine='x86_64')<br>
Image Structure Metadata:<br>
COMPRESSION=LZW<br>
INTERLEAVE=BAND<br>
PREDICTOR=3<br>
Corner Coordinates:<br>
Upper Left (-180.0000000, 90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)<br>
Lower Left (-180.0000000, -90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"S)<br>
Upper Right ( 180.0000000, 90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)<br>
Lower Right ( 180.0000000, -90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"S)<br>
Center ( 0.0000000, 0.0000000) ( 0d 0' 0.01"E, 0d 0' 0.01"N)<br>
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray<br>
Description = score<br>
NoData Value=-9999<br>
Band 2 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = severity_value<br>
NoData Value=-9999<br>
Band 3 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = severity_min<br>
NoData Value=-9999<br>
Band 4 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = severity_max<br>
NoData Value=-9999<br>
Band 5 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = likelihood<br>
NoData Value=-9999<br>
Band 6 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = return_time<br>
NoData Value=-9999<br>
Band 7 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = likelihood_confidence<br>
NoData Value=-9999<br>
Band 8 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = climate_reliability<br>
NoData Value=-9999<br>
Band 9 Block=128x128 Type=Float32, ColorInterp=Undefined<br>
Description = hazard_reliability<br>
NoData Value=-9999<br clear="all">
<br>
-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p style="margin:0cm"><span style="font-size:8.0pt"> Regards,</span><o:p></o:p></p>
<p style="margin:0cm"><o:p> </o:p></p>
<p style="margin:0cm"><span style="font-size:8.0pt">Clive Swan</span><o:p></o:p></p>
<p style="margin:0cm"><span style="font-size:12.0pt">--</span><o:p></o:p></p>
<pre>Hi,<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>If you are still struggling with the same old problem could you please finally send the gdalinfo reports of your two input files which are this time:<o:p></o:p></pre>
<pre>coastal-2020.tif<o:p></o:p></pre>
<pre>5_UK_coastal-2020.tif<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>-Jukka Rahkonen-<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Lähettäjä: gdal-dev <<a href="https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fgdal-dev&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C5d7db77b38f444c43dcf08dadd2e925b%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638065489336507804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nrK3Bq8p1cpYu%2Ff4qQO4TriuOaupVTK%2FPk47Sv6KYnQ%3D&reserved=0">gdal-dev-bounces at lists.osgeo.org</a>> Puolesta Clive Swan<o:p></o:p></pre>
<pre>Lähetetty: tiistai 13. joulukuuta 2022 17.23<o:p></o:p></pre>
<pre>Vastaanottaja: <a href="https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fgdal-dev&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C5d7db77b38f444c43dcf08dadd2e925b%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638065489336507804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nrK3Bq8p1cpYu%2Ff4qQO4TriuOaupVTK%2FPk47Sv6KYnQ%3D&reserved=0">gdal-dev at lists.osgeo.org</a><o:p></o:p></pre>
<pre>Aihe: [gdal-dev] gdalwarp running very slow<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Greetings,<o:p></o:p></pre>
<pre>I am running gdalwarp on a 6GB (output) and 600MB (input) tif image, the AWS Instance has approx 60 VCPU<o:p></o:p></pre>
<pre>It has taken over 6 hours so far - still running, is it possible to optimise this and speed it up??<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>gdalwarp -r near -overwrite coastal-2020.tif 5_UK_coastal-2020.tif -co BIGTIFF=YES -co COMPRESS=LZW -co BLOCKXSIZE=128 -co BLOCKYSIZE=128 -co NUM_THREADS=ALL_CPUS --config CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES<o:p></o:p></pre>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>