[gdal-dev] warp memory limit default too low and does not have a config option

[Even Rouault]

> 
> Is the problematic case you cite rather common or the exceptional?
> Perhaps the -wm option is a good place to set a lower memory limit in
> this case, with the default being higher (instead of the other way
> around)?

Unless we install a spy in gdalwarp that sends reports to a server, nobody 
will know I'm afraid ;-)

> 
> Which is more common, 1) many small images being merged into a larger
> one, 2) warping of large images or 3) merging of large+small images?
> It would be nice as you suggest to have some benchmarks, would the 3
> cases I just mentioned be representative enough? What would be typical
> sizes for each case?

I can imagine, in simple warping scenarios, images of 1000x1000 and 
10000x10000 pixels.
And for mosaicing : merging images of size 1000x1000 in a 50000x50000 RGB 
mosaic.
But this is really arbitrary...

> Perhaps a compromise could be found, that
> minimizes performance penalties on average.
> The mentioned ticket has a proposed patch (by you) -  perhaps it could
> be used in some benchmarks?

Actually, I only saved the patch that was posted in a volatile paste by Adam. 
The rationale behind it is looks reasonable. I think I tested it a bit at that 
time, but was probably too shy to apply it without more testing.

> On a side note, I *think* that most users use default value and do not
> use the -wm option unless they really look into the docs - therefore
> it would be nice to have a better default.

I agree that users will unlikely play with -wm option whose value might need 
some expertise to set it to an appropriate value. So I think that increasing 
the default value for -wm is reasonable if it helps in common scenarios. But I 
think benchmarks will show that we will need to fix the issue raised in the 
ticket first, probably with Adam's patch or something close to that idea.

There are a few cases where choosing 1/4 of the RAM by default might cause 
issues : 

1) If several gdalwarp processes are launched in parallel, they might exhaust 
the RAM and suffer from excessive swapping. But I think it is OK in that 
advanced scenario to tell them : "just set -wm to a lower value so that the 
sum of the -wm values is lower than XX% of the RAM".

2) 1/4 of the RAM isn't appropriate for 32 bit process. You can have a machine 
with 64 GB RAM, but a 32 bit process can only address 2 GB (generally, 
sometimes 3 GB). So we should probably not try to use more than 1 GB (or less)  
to avoid exhausting virtual memory space in that case.

> 
> >> 2) Add a config option so that the default can be set for all
> >> operations by the user (which overrides 1, but can be overridden for
> >> each operation)
> > 
> > Not sure to understand how option 2) if different from the -wm option and
> > how it would be implemented ...
> 
> Simply a config option, like there is for raster i/o cache (GDAL_CACHEMAX).
> This would be a simple fix that allows the user to set the default for
> his/her typical usage (and select another value with -wm when needed).
> There might not be a need to change the default value then.

I'm not convinced that adding a new config option that does the same thing as -
wm will really help. Possibly, it would make things more confusing.

> I have a small patch that implements that, should I create a ticket to
> centralize this discussion and post the patch?
> 
> cheers,
> Etienne
> 
> >> Any comments?
> >> 
> >> regards,
> >> Etienne
> >> _______________________________________________
> >> gdal-dev mailing list
> >> gdal-dev at lists.osgeo.org
> >> http://lists.osgeo.org/mailman/listinfo/gdal-dev