[gdal-dev] gdal_merge

Mon Dec 5 05:51:32 PST 2022

Hi,

I asked a few questions and hoped we could solve your problem with one mail if you answered them all but probably we need more. Here is an example about the info I would like to see. You can obtain it with gdalinfo
gdalinfo test.tif
...
Size is 12000, 12000
...
Pixel Size = (0.500000000000000,-0.500000000000000)
...
COMPRESSION=YCbCr JPEG
...
Band 1 Block=256x256 Type=Byte, ColorInterp=Red

The -ps stands for pixel size https://gdal.org/programs/gdal_merge.html#cmdoption-gdal_merge-ps. Thus, how many meters or degrees one pixel is wide/high. Gdal_merge does not have an option for defining the size of the output image in pixels. See my example about half-a-metre pixels above, that is the information that you need. Do not pay attention to negative Y size, use positive number for both directions with gdal_merge. Do not get surprised by the small number that you will probably see, I guess your image in in EPSG:4326 and the units are degrees.

Compression options are format specific, for GeoTIFF read https://gdal.org/drivers/raster/gtiff.html. With gdal_merge read https://gdal.org/programs/gdal_merge.html#cmdoption-gdal_merge-co and add for example "-co COMPRESS=LZW -co TILED=YES" into the command.

-Jukka Rahkonen-

Lähettäjä: Clive Swan <cliveswan at gmail.com>
Lähetetty: maanantai 5. joulukuuta 2022 15.32
Vastaanottaja: Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi>
Kopio: gdal-dev at lists.osgeo.org
Aihe: Re: [gdal-dev] gdal_merge

Greetings,
 I tried
-ps 3600 7200
 -ps 3600,7200
-ps x=3600  y=7200

Just get errors, I don't see any option to select LZW or any compression??

Extent
-180.0000000000000000,-90.0000000000000000 : 180.0000000000000000,90.0000000000000000
Width
7200
Height
3600
Data type
Float32 - Thirty two bit floating point
Compression
LZW

On Mon, 5 Dec 2022 at 12:32, Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>> wrote:
Hi,

Let's try to get all missing information with one mail.

  *   You have two images:
3_data_ready_for_spectra/coastal-undefended-rcp26-2020.tif and

5_UK_prod_rasters/coastal-undefended-rcp26-2020.tif.

Is that right? And the first one covers the world but the second one only UK, right?

  *   What are the pixel sizes and image sizes (in pixels, width x hight) of the source images?
  *   Are they compressed? With what method?
  *   What are the block sizes (out of curiosity, not essential knowledge)?
  *   Have you considered to create a tiled and compressed output? (you should)
  *   Have you considered to use gdalwarp instead? (I would)

With the gdalwarp option you could update the world image with the UK contents with "gdalwarp uk_image.tif world_image.tif". However, your images are in AWS and the same recipes that work for local images may not work so well in the cloud. Unfortunately that is something that I have no experience yet.

The logic in "The input files are 6.9 GB + 672 MB The output should be 7.xx GB" is wrong. If you update part of the pixels of the 6.9 GB file then the output should be about 6.9 GB. But if you convert a 6.9 GB compressed image into an uncompressed image, as I guess that you are doing, the output will be much bigger. How big it will be can be calculated with a formula that numpy showed (width x height) x (bytes/pixel).

-Jukka Rahkonen-

Lähettäjä: Clive Swan <cliveswan at gmail.com<mailto:cliveswan at gmail.com>>
Lähetetty: maanantai 5. joulukuuta 2022 13.47
Vastaanottaja: Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>>; gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Aihe: Re: [gdal-dev] gdal_merge

Hi,

The raster one is the world, the second is the UK.
The UK fits within the first raster...

We want to update UK data in raster one.

Regards,

Clive

Get Outlook for Android<https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2FAAb9ysg&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7Ca53f6e33f709481cefa008dad6c51040%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638058439123548735%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NbhnxmobPMAi%2F%2FrcmPjSwtbKHyRqxAriD5MNJe4CSXA%3D&reserved=0>
________________________________
From: Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>>
Sent: Monday, December 5, 2022 11:34:53 AM
To: Clive Swan <cliveswan at gmail.com<mailto:cliveswan at gmail.com>>; gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org> <gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>>
Subject: Re: [gdal-dev] gdal_merge

Hi,

Maybe your rasters are far apart. Gdal_merge creates a raster with so large extent that all the source images fit within it. Even empty pixels take space.

-Jukka Rahkonen-

Lähettäjä: gdal-dev <gdal-dev-bounces at lists.osgeo.org<mailto:gdal-dev-bounces at lists.osgeo.org>> Puolesta Clive Swan
Lähetetty: maanantai 5. joulukuuta 2022 12.24
Vastaanottaja: gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Aihe: [gdal-dev] gdal_merge

Hi Even,

Thanks
the -configuration sorted out that issue.

CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE is a configuration option /

environment variable, so you have to pass it with --config

CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE YES

Now I am getting a memory issue?? << The input files are 6.9 GB + 672 MB
The output should be 7.xx GB

6.9 GB + 672MB

numpy.core._exceptions._ArrayMemoryError: Unable to allocate 377. GiB for an array with shape (225000, 450000) and data type float32

Any suggestions appreciated

Kind regards,

Clive

--

 Regards,

Clive Swan

--

--

 Regards,

Clive Swan

--

M: +44 7766 452665
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221205/bfebe5e0/attachment-0001.htm>