[gdal-dev] gdalwarp running very slow

Rahkonen Jukka jukka.rahkonen at maanmittauslaitos.fi
Tue Dec 13 23:18:50 PST 2022 

Hi,

Thank you for the information about the source files. I do not yet understand what you are trying to do and why. The both images have the same size 450000 and 225000 and they cover the same area. Is the "image 5_UK_coastal-2020.tif" just NoData with pixel value -9999 everywhere outside the UK? The name of the image makes me think so.

-Jukka Rahkonen-


Lähettäjä: Clive Swan <cliveswan at gmail.com>
Lähetetty: tiistai 13. joulukuuta 2022 19.22
Vastaanottaja: gdal-dev at lists.osgeo.org
Kopio: Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi>
Aihe: [gdal-dev] gdalwarp running very slow

 Greetings,
I am using the same files, I copied them from an AWS Bucket to a local AWS Instance.
I tried gdal_merge << tries to create 300GB file
I tried gdal_translate ran but created 2.5 GB not 6.9 GB file
Now trying gdalwarp.

the gdalinfo is the same in both datasets:

coastal-2020.tif (6.9GB)
Driver: GTiff/GeoTIFF
Size is 450000, 225000
Coordinate System is:
GEOGCRS["WGS 84",
    DATUM["World Geodetic System 1984",
        ELLIPSOID["WGS 84",6378137,298.257223563,
            LENGTHUNIT["metre",1]]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433]],
    CS[ellipsoidal,2],
        AXIS["geodetic latitude (Lat)",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433]],
        AXIS["geodetic longitude (Lon)",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433]],
    ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (0.000800000000000,-0.000800000000000)
Metadata:
  AREA_OR_POINT=Area
  datetime_created=2022-11-14 18:05:14.053301
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=BAND
  PREDICTOR=3
Corner Coordinates:
Upper Left  (-180.0000000,  90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)
Lower Left  (-180.0000000, -90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"S)
Upper Right ( 180.0000000,  90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"S)
Center      (   0.0000000,   0.0000000) (  0d 0' 0.01"E,  0d 0' 0.01"N)
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray
  Description = score
  NoData Value=-9999
Band 2 Block=128x128 Type=Float32, ColorInterp=Undefined
  Description = severity_value
  NoData Value=-9999
Band 3 Block=128x128 Type=Float32, ColorInterp=Undefined
  Description = severity_min
  NoData Value=-9999
Band 4 Block=128x128 Type=Float32, ColorInterp=Undefined
  Description = severity_max
  NoData Value=-9999
Band 5 Block=128x128 Type=Float32, ColorInterp=Undefined
  Description = likelihood
  NoData Value=-9999
Band 6 Block=128x128 Type=Float32, ColorInterp=Undefined
  Description = return_time
  NoData Value=-9999
Band 7 Block=128x128 Type=Float32, ColorInterp=Undefined
  Description = likelihood_confidence
  NoData Value=-9999
Band 8 Block=128x128 Type=Float32, ColorInterp=Undefined
  Description = climate_reliability
  NoData Value=-9999
Band 9 Block=128x128 Type=Float32, ColorInterp=Undefined
  Description = hazard_reliability
  NoData Value=-9999


5_UK_coastal-2020.tif (600MB)
Driver: GTiff/GeoTIFF
Size is 450000, 225000
Coordinate System is:
GEOGCRS["WGS 84",
    DATUM["World Geodetic System 1984",
        ELLIPSOID["WGS 84",6378137,298.257223563,
            LENGTHUNIT["metre",1]]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433]],
    CS[ellipsoidal,2],
        AXIS["geodetic latitude (Lat)",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433]],
        AXIS["geodetic longitude (Lon)",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433]],
    ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (0.000800000000000,-0.000800000000000)
Metadata:
  AREA_OR_POINT=Area
  datetime_created=2022-11-14 18:05:14.053301
  hostname=posix.uname_result(sysname='Linux', nodename='ip-172-31-12-125', release='5.15.0-1022-aws', version='#26~20.04.1-Ubuntu SMP Sat Oct 15 03:22:07 UTC 2022', machine='x86_64')
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=BAND
  PREDICTOR=3
Corner Coordinates:
Upper Left  (-180.0000000,  90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)
Lower Left  (-180.0000000, -90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"S)
Upper Right ( 180.0000000,  90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"S)
Center      (   0.0000000,   0.0000000) (  0d 0' 0.01"E,  0d 0' 0.01"N)
Band 1 Block=128x128 Type=Float32, ColorInterp=Gray
  Description = score
  NoData Value=-9999
Band 2 Block=128x128 Type=Float32, ColorInterp=Undefined
  Description = severity_value
  NoData Value=-9999
Band 3 Block=128x128 Type=Float32, ColorInterp=Undefined
  Description = severity_min
  NoData Value=-9999
Band 4 Block=128x128 Type=Float32, ColorInterp=Undefined
  Description = severity_max
  NoData Value=-9999
Band 5 Block=128x128 Type=Float32, ColorInterp=Undefined
  Description = likelihood
  NoData Value=-9999
Band 6 Block=128x128 Type=Float32, ColorInterp=Undefined
  Description = return_time
  NoData Value=-9999
Band 7 Block=128x128 Type=Float32, ColorInterp=Undefined
  Description = likelihood_confidence
  NoData Value=-9999
Band 8 Block=128x128 Type=Float32, ColorInterp=Undefined
  Description = climate_reliability
  NoData Value=-9999
Band 9 Block=128x128 Type=Float32, ColorInterp=Undefined
  Description = hazard_reliability
  NoData Value=-9999

--

 Regards,



Clive Swan

--

Hi,



If you are still struggling with the same old problem could you please finally send the gdalinfo reports of your two input files which are this time:

coastal-2020.tif

5_UK_coastal-2020.tif



-Jukka Rahkonen-





Lähettäjä: gdal-dev <gdal-dev-bounces at lists.osgeo.org<https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fgdal-dev&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C5d7db77b38f444c43dcf08dadd2e925b%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638065489336507804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nrK3Bq8p1cpYu%2Ff4qQO4TriuOaupVTK%2FPk47Sv6KYnQ%3D&reserved=0>> Puolesta Clive Swan

Lähetetty: tiistai 13. joulukuuta 2022 17.23

Vastaanottaja: gdal-dev at lists.osgeo.org<https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fgdal-dev&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C5d7db77b38f444c43dcf08dadd2e925b%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638065489336507804%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=nrK3Bq8p1cpYu%2Ff4qQO4TriuOaupVTK%2FPk47Sv6KYnQ%3D&reserved=0>

Aihe: [gdal-dev] gdalwarp running very slow



Greetings,

I am running gdalwarp on a 6GB (output) and 600MB (input) tif image, the AWS Instance has approx 60 VCPU

It has taken over 6 hours so far - still running, is it possible to optimise this and speed it up??



gdalwarp -r near -overwrite coastal-2020.tif   5_UK_coastal-2020.tif -co BIGTIFF=YES -co COMPRESS=LZW -co BLOCKXSIZE=128 -co BLOCKYSIZE=128  -co NUM_THREADS=ALL_CPUS --config CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221214/c180b143/attachment-0001.htm>

More information about the gdal-dev mailing list