<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.gmail-pl-smi
{mso-style-name:gmail-pl-smi;}
span.gmail-text-bold
{mso-style-name:gmail-text-bold;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Arial",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;mso-fareast-language:EN-US">Given the filesize of the original raster, I suspect it’s mostly empty, and so I’d suggest also specifying:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;mso-fareast-language:EN-US">-co “</span><span style="font-family:"Arial",sans-serif;color:black;background:#FCFCFC">SPARSE_OK=TRUE”</span><span style="font-family:"Arial",sans-serif;background:#FCFCFC"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;background:#FCFCFC"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black;background:#FCFCFC">That instructs GDAL to avoid writing any data at all where the entirety of a tile is nodata, rather than writing a (compressed) block of nodata values. However,
the GDAL documentation does warn that this can cause trouble with external software.</span><span style="font-family:"Arial",sans-serif;background:#FCFCFC"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;background:#FCFCFC"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> gdal-dev <gdal-dev-bounces@lists.osgeo.org>
<b>On Behalf Of </b>Cainã K. Campos<br>
<b>Sent:</b> 13 March 2020 13:08<br>
<b>To:</b> Brian <mulcahy.brianp@gmail.com><br>
<b>Cc:</b> gdal-dev@lists.osgeo.org<br>
<b>Subject:</b> Re: [gdal-dev] Need some help understanding why this raster is so big<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">if you run without the switch it will create the result without compression, so you will be back to a 400Gb file. Instead run everything in one single command.
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">With compression it is likely to take some extra time, as some calculation has to be done to achieve that.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">If you want speed, tile the raster and also there are a few other switches to add so gdalwarp can use more than one core to make this calcs, and you can also too give more memory to it like:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span class="gmail-pl-smi">--config GDAL_SWATH_SIZE 2000000000 --config GDAL_CACHEMAX 2000 -wm 2000 -co "tiled=yes" -co "BLOCKXSIZE=256" -co "BLOCKYSIZE=256" -wo "NUM_THREADS=</span><span class="gmail-text-bold">ALL_CPUS</span><span class="gmail-pl-smi">"
-multi -co "NUM_THREADS=</span><span class="gmail-text-bold">ALL_CPUS</span><span class="gmail-pl-smi">" -co "bigtiff=yes"
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Mar 13, 2020 at 9:43 AM Brian <<a href="mailto:mulcahy.brianp@gmail.com">mulcahy.brianp@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Is it faster to do a gdal_warp with compression then without? Is it safe to assume the drive write speed would be the limiting factor for speed in this case? <o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Mar 13, 2020 at 8:33 AM Cainã K. Campos <<a href="mailto:rupestre.campos@gmail.com" target="_blank">rupestre.campos@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal">Hello Brian,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Try to add the switch -co "COMPRESS=LZW" to the command line to generate a compressed result with lossless compression.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Mar 13, 2020 at 9:07 AM Brian <<a href="mailto:mulcahy.brianp@gmail.com" target="_blank">mulcahy.brianp@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">So compressed this raster is fairly small about 120mb but running gdal_warp produces a raster that is about 416 gb, is this something this list can help with? If so I can upload the file somewhere and let you guys/gals take a look at it.<o:p></o:p></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><o:p></o:p></p>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
<p><strong><span style="color: #f6a125; font-family:arial;">T</span></strong><span style="text-decoration: none; color: #000000; font-family:arial;"> +44 (0) 1756 799919</span><br>
<a style="text-decoration: none; color: #f6a125; font-family:arial;" href="http://www.jbarisk.com">www.jbarisk.com</a></p>
<p><a href="http://www.jbarisk.com"><img src="http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-JBA.png" alt="Visit our website" width="33" height="26"></a> <a href=""><img src="http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-LINKEDIN.png" alt="" height="26"></a>
<a href="https://twitter.com/jbarisk"><img src="http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-TWITTER.png" alt="Follow us on Twitter" width="33" height="26"></a></p>
<o:p></o:p>
<p></p>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:#FF9C00">Our postal address and registered office is JBA Risk Management</span><span style="font-family:"Arial",sans-serif;color:black">
</span><span style="font-family:"Arial",sans-serif;color:#FF9C00">Limited, 1 Broughton Park, Old Lane North, Broughton, Skipton, North Yorkshire BD23 3FD.</span><o:p></o:p></p>
<p><b><span style="background: white; color: black; font-family: "Arial",sans-serif; font-size: 10pt;">Find out more about us here:
<a href="http://www.jbarisk.com/"><font color="#0563c1">www.jbarisk.com</font></a> and
</span></b><b><span style="background: white; color: rgb(68, 68, 68); font-family: "Arial",sans-serif; font-size: 10pt;"><a href="http://twitter.com/JBARisk" target="_blank"><font color="#0563c1">follow us on Twitter @JBARisk</font></a> and
<a href="https://www.linkedin.com/company/2370847?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A2370847%2Cidx%3A2-1-2%2CtarId%3A1447414259786%2Ctas%3AJBA%20RISK%20MANAGEMENT">
<font color="#0563c1">LinkedIn</font></a> </span></b></p>
<p><span style="background: white; color: black; font-family: "Arial",sans-serif; font-size: 8pt;">The JBA Group supports the JBA Trust.</span></p>
<p style="margin: 0cm 0cm 0pt;"><span style="background: white; color: rgb(68, 68, 68); font-family: "Arial",sans-serif; font-size: 8pt;">All JBA Risk Management's email messages contain confidential information and are intended only for the individual(s) named.
If you are not the named addressee you should not disseminate, distribute or copy this e-mail.</span><br>
<span style="color: rgb(68, 68, 68); font-family: "Arial",sans-serif; font-size: 8pt;"><span style="background: white;">Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.</span></span><br>
</p>
<p style="margin: 0cm 0cm 0pt;"><span style="color: rgb(68, 68, 68); font-family: "Arial",sans-serif; font-size: 8pt;"><span style="background: white;">JBA Risk Management Limited is registered in England, company number 07732946, 1 Broughton Park, Old Lane
North, Broughton, Skipton, North Yorkshire, BD23 3FD, </span></span><span style="background: white; color: black; font-family: "Arial",sans-serif; font-size: 8pt;">Telephone: +441756799919</span></p>
<p> </p>
</body>
</html>