[gdal-dev] mosaicking is very slow

[Andreas Oxenstierna]

and always remember as stated in the doc-link: "Disks are the slowest 
part of the process."
As fast I/O and file system as technically possible ... e.g. do not use 
your laptop

> Hi,
>
> You can read considerations about memory from
>
> https://trac.osgeo.org/gdal/wiki/UserDocs/GdalWarp#WillincreasingRAMincreasethespeedofgdalwarp
>
> My guess it that increasing CACHEMAX might help a bit in this use case because it feels similar to the example that is used in the user doc "This could occur, for instance, in a scanline oriented input file which is processed in multiple chunks (horizontally) by gdalwarp". But no documentation is as accurate as a good test. Take a stop watch and measure timings with different settings and you will know how your system reacts.
>
> Unfortunately it is easy to test and optimize things which are not relevant, for example to test only different memory settings if the real issue happens to be in the source data that requires unnecessarily much memory. But fortunately it is also easy to run another set of tests with tiled source images and compare the results.
>   
> -Jukka Rahkonen-
>
>
> -----Alkuperäinen viesti-----
> Lähettäjä: Simon <twinbirds at protonmail.com>
> Lähetetty: maanantai 27. tammikuuta 2020 15.21
> Vastaanottaja: Rahkonen Jukka (MML) <jukka.rahkonen at maanmittauslaitos.fi>
> Kopio: gdal-dev at lists.osgeo.org
> Aihe: Re: [gdal-dev] mosaicking is very slow
>
> Hi,
> Okay, thank you.
> By the way, while making copy of gtiffs, is not it good idea to increase GDAL_CACHEMAX?
> Simon
>
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Monday, January 27, 2020 9:04 PM, Rahkonen Jukka (MML) <jukka.rahkonen at maanmittauslaitos.fi> wrote:
>
>> Hi,
>>
>> If your source images are 4 GB then their size is at least 40000 by 40000 pixels, or more if they are compressed. If gdalwarp now needs 256256 pixels worth of source data, from a striped image it must read 25640000 (rowspixels per row), while from tiled source it can read just from 1 to 4 tiles, 256256 each.
>>
>> Pre-processing is additional load but if you pay 10 units extra for tiling and then save 100 units in warping you are still a winner.
>>
>> Make copies as
>> gdal_translate -of GTiff -co tiled=yes -co compress=LZW input1.tif
>> copy1.tif
>>
>> Then create a new VRT file. Absolutely do not remove -co tiled=yes from the gdalwarp command or you'll demand GDAL to write at least 400000 pixels wide stripes.
>>
>> -Jukka Rahkonen-
>>
>> -----Alkuperäinen viesti-----
>> Lähettäjä: Simon twinbirds at protonmail.com
>> Lähetetty: maanantai 27. tammikuuta 2020 13.18
>> Vastaanottaja: Rahkonen Jukka (MML)
>> jukka.rahkonen at maanmittauslaitos.fi
>> Kopio: gdal-dev at lists.osgeo.org
>> Aihe: Re: [gdal-dev] mosaicking is very slow
>>
>> Hi,
>> Thanks for your answer. The source images are not tiled. So, how can I tile them? Is not it additional load? I will try removing these ones:
>> --config GDAL_CACHEMAX 8000
>> -co TILED=YES
>>
>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>> On Monday, January 27, 2020 7:48 PM, jratike80 jukka.rahkonen at maanmittauslaitos.fi wrote:
>>
>>> Hi,
>>> I would suggest to remove --config GDAL_CACHEMAX 8000 first. Then I
>>> would check if the source images are tiled. If they are not it could
>>> be worth an extra step to make tiled copies.
>>> -Jukka Rahkonen-
>>> Simon-4 wrote
>>>
>>>> Hello everyone,
>>>> I am stiching aroung 100 geotiff images each of around 4gb.
>>>> However, translating the vrt file into geotiff is going on for days.
>>>> gtranslate --config GDAL_CACHEMAX 8000 -co TILED=YES -co
>>>> COMPRESS=LZW -co BIGTIFF=YES infile.vrt outfile.tif How can I
>>>> speed up it?
>>>> Thank you.
>>>> Simon
>>>> gdal-dev mailing list
>>>> gdal-dev at .osgeo
>>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>> --
>>> Sent from:
>>> http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
>>> gdal-dev mailing list
>>> gdal-dev at lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev


-- 
Hälsningar

Andreas Oxenstierna
T-Kartor Geospatial AB
Olof Mohlins väg 12 Kristianstad
mobile: +46 733 206831
mailto: ao at t-kartor.se
http://www.t-kartor.com