[gdal-dev] warp memory limit default too low and does not have a
even.rouault at mines-paris.org
Thu Feb 9 16:29:05 EST 2012
> Is the problematic case you cite rather common or the exceptional?
> Perhaps the -wm option is a good place to set a lower memory limit in
> this case, with the default being higher (instead of the other way
Unless we install a spy in gdalwarp that sends reports to a server, nobody
will know I'm afraid ;-)
> Which is more common, 1) many small images being merged into a larger
> one, 2) warping of large images or 3) merging of large+small images?
> It would be nice as you suggest to have some benchmarks, would the 3
> cases I just mentioned be representative enough? What would be typical
> sizes for each case?
I can imagine, in simple warping scenarios, images of 1000x1000 and
And for mosaicing : merging images of size 1000x1000 in a 50000x50000 RGB
But this is really arbitrary...
> Perhaps a compromise could be found, that
> minimizes performance penalties on average.
> The mentioned ticket has a proposed patch (by you) - perhaps it could
> be used in some benchmarks?
Actually, I only saved the patch that was posted in a volatile paste by Adam.
The rationale behind it is looks reasonable. I think I tested it a bit at that
time, but was probably too shy to apply it without more testing.
> On a side note, I *think* that most users use default value and do not
> use the -wm option unless they really look into the docs - therefore
> it would be nice to have a better default.
I agree that users will unlikely play with -wm option whose value might need
some expertise to set it to an appropriate value. So I think that increasing
the default value for -wm is reasonable if it helps in common scenarios. But I
think benchmarks will show that we will need to fix the issue raised in the
ticket first, probably with Adam's patch or something close to that idea.
There are a few cases where choosing 1/4 of the RAM by default might cause
1) If several gdalwarp processes are launched in parallel, they might exhaust
the RAM and suffer from excessive swapping. But I think it is OK in that
advanced scenario to tell them : "just set -wm to a lower value so that the
sum of the -wm values is lower than XX% of the RAM".
2) 1/4 of the RAM isn't appropriate for 32 bit process. You can have a machine
with 64 GB RAM, but a 32 bit process can only address 2 GB (generally,
sometimes 3 GB). So we should probably not try to use more than 1 GB (or less)
to avoid exhausting virtual memory space in that case.
> >> 2) Add a config option so that the default can be set for all
> >> operations by the user (which overrides 1, but can be overridden for
> >> each operation)
> > Not sure to understand how option 2) if different from the -wm option and
> > how it would be implemented ...
> Simply a config option, like there is for raster i/o cache (GDAL_CACHEMAX).
> This would be a simple fix that allows the user to set the default for
> his/her typical usage (and select another value with -wm when needed).
> There might not be a need to change the default value then.
I'm not convinced that adding a new config option that does the same thing as -
wm will really help. Possibly, it would make things more confusing.
> I have a small patch that implements that, should I create a ticket to
> centralize this discussion and post the patch?
> >> Any comments?
> >> regards,
> >> Etienne
> >> _______________________________________________
> >> gdal-dev mailing list
> >> gdal-dev at lists.osgeo.org
> >> http://lists.osgeo.org/mailman/listinfo/gdal-dev
More information about the gdal-dev