[gdal-dev] gdal_merge

[Rahkonen Jukka]

Hi,

Let's try to get all missing information with one mail.

  *   You have two images:
3_data_ready_for_spectra/coastal-undefended-rcp26-2020.tif and

5_UK_prod_rasters/coastal-undefended-rcp26-2020.tif.

Is that right? And the first one covers the world but the second one only UK, right?

  *   What are the pixel sizes and image sizes (in pixels, width x hight) of the source images?
  *   Are they compressed? With what method?
  *   What are the block sizes (out of curiosity, not essential knowledge)?
  *   Have you considered to create a tiled and compressed output? (you should)
  *   Have you considered to use gdalwarp instead? (I would)

With the gdalwarp option you could update the world image with the UK contents with "gdalwarp uk_image.tif world_image.tif". However, your images are in AWS and the same recipes that work for local images may not work so well in the cloud. Unfortunately that is something that I have no experience yet.

The logic in "The input files are 6.9 GB + 672 MB The output should be 7.xx GB" is wrong. If you update part of the pixels of the 6.9 GB file then the output should be about 6.9 GB. But if you convert a 6.9 GB compressed image into an uncompressed image, as I guess that you are doing, the output will be much bigger. How big it will be can be calculated with a formula that numpy showed (width x height) x (bytes/pixel).

-Jukka Rahkonen-

Lähettäjä: Clive Swan <cliveswan at gmail.com>
Lähetetty: maanantai 5. joulukuuta 2022 13.47
Vastaanottaja: Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi>; gdal-dev at lists.osgeo.org
Aihe: Re: [gdal-dev] gdal_merge

Hi,

The raster one is the world, the second is the UK.
The UK fits within the first raster...

We want to update UK data in raster one.

Regards,

Clive

Get Outlook for Android<https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C405531fd73e74c5add7208dad6b66773%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638058376154245151%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3iBA3SB0%2BBSUKMk8laLw5Gi7%2BnDzhHsoqTpjnfznUoY%3D&reserved=0>
________________________________
From: Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>>
Sent: Monday, December 5, 2022 11:34:53 AM
To: Clive Swan <cliveswan at gmail.com<mailto:cliveswan at gmail.com>>; gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org> <gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>>
Subject: Re: [gdal-dev] gdal_merge


Hi,



Maybe your rasters are far apart. Gdal_merge creates a raster with so large extent that all the source images fit within it. Even empty pixels take space.



-Jukka Rahkonen-





Lähettäjä: gdal-dev <gdal-dev-bounces at lists.osgeo.org<mailto:gdal-dev-bounces at lists.osgeo.org>> Puolesta Clive Swan
Lähetetty: maanantai 5. joulukuuta 2022 12.24
Vastaanottaja: gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Aihe: [gdal-dev] gdal_merge



Hi Even,



Thanks
the -configuration sorted out that issue.

CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE is a configuration option /

environment variable, so you have to pass it with --config

CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES


Now I am getting a memory issue?? << The input files are 6.9 GB + 672 MB
The output should be 7.xx GB



6.9 GB + 672MB

numpy.core._exceptions._ArrayMemoryError: Unable to allocate 377. GiB for an array with shape (225000, 450000) and data type float32



Any suggestions appreciated

Kind regards,

Clive


--

 Regards,



Clive Swan

--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221205/fca487a6/attachment.htm>