[gdal-dev] GDAL python bindings memory usage

[Evert Etienne (SITEMARK)]

I actually have no idea what the extra code in your examples is doing, can you explain?

For the behavior: it is the same as psutil (makes sense since memory_profiler uses psutil by default). You can see it reach max cache by logging psutil values in the Translate callback. I also saw the same behavior by looking at the memory usage using top.

Is there another way you know that works that I can try to see if I can replicate the memory usage values?

Thank you

> On 27 Dec 2019, at 21:45, Even Rouault <even.rouault at spatialys.com> wrote:
> 
> I question the reliability of memory_profiler
> 
> See
> 
> Line #    Mem usage    Increment   Line Contents
> ================================================
>    27  112.125 MiB  112.125 MiB   @profile
>    28                             def test2():
>    29  112.676 MiB    0.551 MiB       gdal.SetCacheMax(1000 * 1024 * 1024)
>    30 1101.855 MiB  989.180 MiB       gdal.Translate('/tmp/out.tif', 'byte.tif', options = '-co TILED=YES -outsize 100000 10000')
>    31 1101.855 MiB    0.000 MiB       dummy = 1
> 
> vs
> 
> Line #    Mem usage    Increment   Line Contents
> ================================================
>    28  109.336 MiB  109.336 MiB   @profile
>    29                             def test2():
>    30  109.336 MiB    0.000 MiB       gdal.SetCacheMax(1000 * 1024 * 1024)
>    31 1098.309 MiB  988.973 MiB       gdal.Translate('/tmp/out.tif', 'byte.tif', options = '-co TILED=YES -outsize 100000 10000')
>    32 1098.309 MiB    0.000 MiB       h = ctypes.cdll.LoadLibrary(None)
>    33 1098.309 MiB    0.000 MiB       h.malloc.argtypes = [ctypes.c_size_t]
>    34 1098.309 MiB    0.000 MiB       h.malloc.restype = ctypes.c_void_p
>    35 1098.309 MiB    0.000 MiB       h.free.argtypes = [ctypes.c_void_p]
>    36 1098.309 MiB    0.000 MiB       h.free.restype = None
>    37 1098.309 MiB    0.000 MiB       size = 1024 * 1024
>    38 1098.309 MiB    0.000 MiB       x = h.malloc(size)
>    39  117.422 MiB    0.000 MiB       h.free(x)
>    40  117.422 MiB    0.000 MiB       dummy = 1
> 
> vs
> 
> Line #    Mem usage    Increment   Line Contents
> ================================================
>    28  111.305 MiB  111.305 MiB   @profile
>    29                             def test2():
>    30  111.852 MiB    0.547 MiB       gdal.SetCacheMax(1000 * 1024 * 1024)
>    31  188.504 MiB   76.652 MiB       gdal.Translate('/tmp/out.tif', 'byte.tif', options = '-co TILED=YES -outsize 100000 10000')
>    32  188.504 MiB    0.000 MiB       h = ctypes.cdll.LoadLibrary(None)
>    33  188.504 MiB    0.000 MiB       h.free.argtypes = [ctypes.c_void_p]
>    34  188.504 MiB    0.000 MiB       h.free.restype = None
>    35  188.504 MiB    0.000 MiB       h.free(None)
>    36  188.504 MiB    0.000 MiB       dummy = 1
> 
> vs
> 
> Line #    Mem usage    Increment   Line Contents
> ================================================
>    28  112.164 MiB  112.164 MiB   @profile
>    29                             def test2():
>    30  112.164 MiB    0.000 MiB       gdal.SetCacheMax(1000 * 1024 * 1024)
>    31  112.770 MiB    0.605 MiB       h = ctypes.cdll.LoadLibrary(None)
>    32  112.770 MiB    0.000 MiB       h.free.argtypes = [ctypes.c_void_p]
>    33  112.770 MiB    0.000 MiB       h.free.restype = None
>    34 1101.918 MiB  989.148 MiB       gdal.Translate('/tmp/out.tif', 'byte.tif', options = '-co TILED=YES -outsize 100000 10000')
>    35 1101.918 MiB    0.000 MiB       h.free(None)
>    36 1101.918 MiB    0.000 MiB       dummy = 1
> 
> That doesn't make much sense to me.
> 
> Even
> 
> -- 
> Spatialys - Geospatial professional services
> https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.spatialys.com&data=01%7C01%7Cevert.etienne%40sitemark.com%7C39f1fc8e39b14a21169d08d78b0da8ab%7Cfc89adff07ac47008853b7b7e906068e%7C0&sdata=Mg1ztmCAuQnlKkFq3Ny00bE%2BfmzljlV0cWVRrYyU8PQ%3D&reserved=0