[gdal-dev] gdalwarp running very slow
Clive Swan
cliveswan at gmail.com
Tue Dec 13 09:22:05 PST 2022
Greetings,
I am using the same files, I copied them from an AWS Bucket to a local AWS
Instance.
I tried gdal_merge << tries to create 300GB file
I tried gdal_translate ran but created 2.5 GB not 6.9 GB file
Now trying gdalwarp.
the gdalinfo is the same in both datasets:
coastal-2020.tif (6.9GB)
Driver: GTiff/GeoTIFF
Size is 450000, 225000
Coordinate System is:
GEOGCRS["WGS 84",
DATUM["World Geodetic System 1984",
ELLIPSOID["WGS 84",6378137,298.257223563,
LENGTHUNIT["metre",1]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433]],
CS[ellipsoidal,2],
AXIS["geodetic latitude (Lat)",north,
ORDER[1],
ANGLEUNIT["degree",0.0174532925199433]],
AXIS["geodetic longitude (Lon)",east,
ORDER[2],
ANGLEUNIT["degree",0.0174532925199433]],
ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (0.000800000000000,-0.000800000000000)
Metadata:
AREA_OR_POINT=Area
datetime_created=2022-11-14 18:05:14.053301
Image Structure Metadata:
COMPRESSION=LZW
INTERLEAVE=BAND
PREDICTOR=3
Corner Coordinates:
Upper Left (-180.0000000, 90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)
Lower Left (-180.0000000, -90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"S)
Upper Right ( 180.0000000, 90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"S)
Center ( 0.0000000, 0.0000000) ( 0d 0' 0.01"E, 0d 0' 0.01"N)
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray
Description = score
NoData Value=-9999
Band 2 Block=128x128 Type=Float32, ColorInterp=Undefined
Description = severity_value
NoData Value=-9999
Band 3 Block=128x128 Type=Float32, ColorInterp=Undefined
Description = severity_min
NoData Value=-9999
Band 4 Block=128x128 Type=Float32, ColorInterp=Undefined
Description = severity_max
NoData Value=-9999
Band 5 Block=128x128 Type=Float32, ColorInterp=Undefined
Description = likelihood
NoData Value=-9999
Band 6 Block=128x128 Type=Float32, ColorInterp=Undefined
Description = return_time
NoData Value=-9999
Band 7 Block=128x128 Type=Float32, ColorInterp=Undefined
Description = likelihood_confidence
NoData Value=-9999
Band 8 Block=128x128 Type=Float32, ColorInterp=Undefined
Description = climate_reliability
NoData Value=-9999
Band 9 Block=128x128 Type=Float32, ColorInterp=Undefined
Description = hazard_reliability
NoData Value=-9999
5_UK_coastal-2020.tif (600MB)
Driver: GTiff/GeoTIFF
Size is 450000, 225000
Coordinate System is:
GEOGCRS["WGS 84",
DATUM["World Geodetic System 1984",
ELLIPSOID["WGS 84",6378137,298.257223563,
LENGTHUNIT["metre",1]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433]],
CS[ellipsoidal,2],
AXIS["geodetic latitude (Lat)",north,
ORDER[1],
ANGLEUNIT["degree",0.0174532925199433]],
AXIS["geodetic longitude (Lon)",east,
ORDER[2],
ANGLEUNIT["degree",0.0174532925199433]],
ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (0.000800000000000,-0.000800000000000)
Metadata:
AREA_OR_POINT=Area
datetime_created=2022-11-14 18:05:14.053301
hostname=posix.uname_result(sysname='Linux', nodename='ip-172-31-12-125',
release='5.15.0-1022-aws', version='#26~20.04.1-Ubuntu SMP Sat Oct 15
03:22:07 UTC 2022', machine='x86_64')
Image Structure Metadata:
COMPRESSION=LZW
INTERLEAVE=BAND
PREDICTOR=3
Corner Coordinates:
Upper Left (-180.0000000, 90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)
Lower Left (-180.0000000, -90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"S)
Upper Right ( 180.0000000, 90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"S)
Center ( 0.0000000, 0.0000000) ( 0d 0' 0.01"E, 0d 0' 0.01"N)
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray
Description = score
NoData Value=-9999
Band 2 Block=128x128 Type=Float32, ColorInterp=Undefined
Description = severity_value
NoData Value=-9999
Band 3 Block=128x128 Type=Float32, ColorInterp=Undefined
Description = severity_min
NoData Value=-9999
Band 4 Block=128x128 Type=Float32, ColorInterp=Undefined
Description = severity_max
NoData Value=-9999
Band 5 Block=128x128 Type=Float32, ColorInterp=Undefined
Description = likelihood
NoData Value=-9999
Band 6 Block=128x128 Type=Float32, ColorInterp=Undefined
Description = return_time
NoData Value=-9999
Band 7 Block=128x128 Type=Float32, ColorInterp=Undefined
Description = likelihood_confidence
NoData Value=-9999
Band 8 Block=128x128 Type=Float32, ColorInterp=Undefined
Description = climate_reliability
NoData Value=-9999
Band 9 Block=128x128 Type=Float32, ColorInterp=Undefined
Description = hazard_reliability
NoData Value=-9999
--
Regards,
Clive Swan
--
Hi,
If you are still struggling with the same old problem could you please
finally send the gdalinfo reports of your two input files which are
this time:
coastal-2020.tif
5_UK_coastal-2020.tif
-Jukka Rahkonen-
Lähettäjä: gdal-dev <gdal-dev-bounces at lists.osgeo.org
<https://lists.osgeo.org/mailman/listinfo/gdal-dev>> Puolesta Clive
Swan
Lähetetty: tiistai 13. joulukuuta 2022 17.23
Vastaanottaja: gdal-dev at lists.osgeo.org
<https://lists.osgeo.org/mailman/listinfo/gdal-dev>
Aihe: [gdal-dev] gdalwarp running very slow
Greetings,
I am running gdalwarp on a 6GB (output) and 600MB (input) tif image,
the AWS Instance has approx 60 VCPU
It has taken over 6 hours so far - still running, is it possible to
optimise this and speed it up??
gdalwarp -r near -overwrite coastal-2020.tif 5_UK_coastal-2020.tif
-co BIGTIFF=YES -co COMPRESS=LZW -co BLOCKXSIZE=128 -co BLOCKYSIZE=128
-co NUM_THREADS=ALL_CPUS --config
CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221213/9f56f652/attachment-0001.htm>
More information about the gdal-dev
mailing list