[gdal-dev] MBTiles creation problems on some iOS devices

[Even Rouault]

On Tuesday 07 June 2016 12:17:58 gunnarblom wrote:
> Hi.
> 
> I'm using GDALTranslate() in C++ to convert GTiffs (5k*5k pixels) to MBTiles
> (resulting size: ~60mb) on iOS.
> 
> It works good on iPhone 6 (1GB ram), iPhone 6S (2GB ram, peaks GDAL usage at
> 160mb) and iPad Mini 3 (1GB ram) and takes around 1-3 minutes. But on e.g.
> iPhone 5S (1GB ram), iPad 3 (1GB ram) and iPad 4 (1GB ram), it takes
> forever (never succeeded with it so far, takes too long).
> 
> What is happening is that on the older devices, GDAL starts to write a lot
> to the disk for some reason. It starts normally and with CPU at 100% for a
> while. Then the writing to disk starts and the CPU goes up and down a lot
> but never at consistent high levels. When I stopped it at 50% progress, it
> had written ~5GB to disk. The iPhone 6S writes  less than 100MB and keeps
> CPU at 100% all the way. All devices have a couple of hundred MB free ram
> during the whole process.
> 
> If I add GDALSetCacheMax(1000 * 1024 * 1024) before converting on iPad 3, it
> succeeds but after a long time and still writes ~1GB of data that slows the
> process a lot (and peaks GDAL memory usage at 250mb, compared to iPhone 6S
> that peaks at 160mb GDAL ram usage).
> 
> 1) What might be wrong here? What situation triggers GDAL to write so much
> to disk even if the RAM seems at good free levels?
> 2) I can set the disk cache with GDALSetCacheMax, any way to set the other
> memory used by this function? (memory for the actual conversion)

Wen creating a MBTiles (or GeoPackage) dataset, there's an heuristics to avoid 
the temporary database (used to handle the fact that GDAL blocks are generally 
not aligned with MBTiles tiles) small enough, and to flush it regularly to the 
main database. One of the condition to trigger  flush is if the remaining free 
space on the device is below 1 GB. You could enable GDAL debugging ( 
CPL_DEBUG=ON as environment variable ) and if that was the case I mentionned 
you would see messages "Free space below 1GB. Flushing part of partial tiles". 
But I wouldn't expect that frequent flushes cause the main DB to grow out of 
control.

Even


-- 
Spatialys - Geospatial professional services
http://www.spatialys.com