[gdal-dev] Need some help understanding why this raster is so big

[Daniel Evans]

Given the filesize of the original raster, I suspect it’s mostly empty, and so I’d suggest also specifying:

-co “SPARSE_OK=TRUE”

That instructs GDAL to avoid writing any data at all where the entirety of a tile is nodata, rather than writing a (compressed) block of nodata values. However, the GDAL documentation does warn that this can cause trouble with external software.


From: gdal-dev <gdal-dev-bounces at lists.osgeo.org> On Behalf Of Cainã K. Campos
Sent: 13 March 2020 13:08
To: Brian <mulcahy.brianp at gmail.com>
Cc: gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] Need some help understanding why this raster is so big

if you run without the switch it will create the result without compression, so you will be back to a 400Gb file. Instead run everything in one single command.
With compression it is likely to take some extra time, as some calculation has to be done to achieve that.
If you want speed, tile the raster and also there are a few other switches to add so gdalwarp can use more than one core to make this calcs, and you can also too give more memory to it like:
--config GDAL_SWATH_SIZE 2000000000 --config GDAL_CACHEMAX 2000 -wm 2000 -co "tiled=yes" -co "BLOCKXSIZE=256" -co "BLOCKYSIZE=256" -wo "NUM_THREADS=ALL_CPUS" -multi -co "NUM_THREADS=ALL_CPUS" -co "bigtiff=yes"


On Fri, Mar 13, 2020 at 9:43 AM Brian <mulcahy.brianp at gmail.com<mailto:mulcahy.brianp at gmail.com>> wrote:
Is it faster to do a gdal_warp with compression then without? Is it safe to assume the drive write speed would be the limiting factor for speed in this case?

On Fri, Mar 13, 2020 at 8:33 AM Cainã K. Campos <rupestre.campos at gmail.com<mailto:rupestre.campos at gmail.com>> wrote:
Hello Brian,

Try to add the switch   -co "COMPRESS=LZW" to the command line to generate a compressed result with lossless compression.

On Fri, Mar 13, 2020 at 9:07 AM Brian <mulcahy.brianp at gmail.com<mailto:mulcahy.brianp at gmail.com>> wrote:
So compressed this raster is fairly small about 120mb but running gdal_warp produces a raster that is about 416 gb, is this something this list can help with? If so I can upload the file somewhere and let you guys/gals take a look at it.
_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/gdal-dev

T +44 (0) 1756 799919
www.jbarisk.com<http://www.jbarisk.com>

[Visit our website]<http://www.jbarisk.com> [http://www.jbagroup.co.uk/imgstore/JBA-Email-Sig-Icons-LINKEDIN.png]  [Follow us on Twitter] <https://twitter.com/jbarisk>
Our postal address and registered office is JBA Risk Management Limited, 1 Broughton Park, Old Lane North, Broughton, Skipton, North Yorkshire BD23 3FD.

Find out more about us here: www.jbarisk.com<http://www.jbarisk.com/> and follow us on Twitter @JBARisk<http://twitter.com/JBARisk> and LinkedIn<https://www.linkedin.com/company/2370847?trk=tyah&trkInfo=clickedVertical%3Acompany%2CclickedEntityId%3A2370847%2Cidx%3A2-1-2%2CtarId%3A1447414259786%2Ctas%3AJBA%20RISK%20MANAGEMENT>

The JBA Group supports the JBA Trust.

All JBA Risk Management's email messages contain confidential information and are intended only for the individual(s) named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system.


JBA Risk Management Limited is registered in England, company number 07732946, 1 Broughton Park, Old Lane North, Broughton, Skipton, North Yorkshire, BD23 3FD, Telephone: +441756799919


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20200313/08f28f9e/attachment-0001.html>