[gdal-dev] GDAL python bindings memory usage

[Even Rouault]

I question the reliability of memory_profiler

See

Line #    Mem usage    Increment   Line Contents
================================================
    27  112.125 MiB  112.125 MiB   @profile
    28                             def test2():
    29  112.676 MiB    0.551 MiB       gdal.SetCacheMax(1000 * 1024 * 1024)
    30 1101.855 MiB  989.180 MiB       gdal.Translate('/tmp/out.tif', 'byte.tif', options = '-co TILED=YES -outsize 100000 10000')
    31 1101.855 MiB    0.000 MiB       dummy = 1

vs

Line #    Mem usage    Increment   Line Contents
================================================
    28  109.336 MiB  109.336 MiB   @profile
    29                             def test2():
    30  109.336 MiB    0.000 MiB       gdal.SetCacheMax(1000 * 1024 * 1024)
    31 1098.309 MiB  988.973 MiB       gdal.Translate('/tmp/out.tif', 'byte.tif', options = '-co TILED=YES -outsize 100000 10000')
    32 1098.309 MiB    0.000 MiB       h = ctypes.cdll.LoadLibrary(None)
    33 1098.309 MiB    0.000 MiB       h.malloc.argtypes = [ctypes.c_size_t]
    34 1098.309 MiB    0.000 MiB       h.malloc.restype = ctypes.c_void_p
    35 1098.309 MiB    0.000 MiB       h.free.argtypes = [ctypes.c_void_p]
    36 1098.309 MiB    0.000 MiB       h.free.restype = None
    37 1098.309 MiB    0.000 MiB       size = 1024 * 1024
    38 1098.309 MiB    0.000 MiB       x = h.malloc(size)
    39  117.422 MiB    0.000 MiB       h.free(x)
    40  117.422 MiB    0.000 MiB       dummy = 1

vs

Line #    Mem usage    Increment   Line Contents
================================================
    28  111.305 MiB  111.305 MiB   @profile
    29                             def test2():
    30  111.852 MiB    0.547 MiB       gdal.SetCacheMax(1000 * 1024 * 1024)
    31  188.504 MiB   76.652 MiB       gdal.Translate('/tmp/out.tif', 'byte.tif', options = '-co TILED=YES -outsize 100000 10000')
    32  188.504 MiB    0.000 MiB       h = ctypes.cdll.LoadLibrary(None)
    33  188.504 MiB    0.000 MiB       h.free.argtypes = [ctypes.c_void_p]
    34  188.504 MiB    0.000 MiB       h.free.restype = None
    35  188.504 MiB    0.000 MiB       h.free(None)
    36  188.504 MiB    0.000 MiB       dummy = 1

vs

Line #    Mem usage    Increment   Line Contents
================================================
    28  112.164 MiB  112.164 MiB   @profile
    29                             def test2():
    30  112.164 MiB    0.000 MiB       gdal.SetCacheMax(1000 * 1024 * 1024)
    31  112.770 MiB    0.605 MiB       h = ctypes.cdll.LoadLibrary(None)
    32  112.770 MiB    0.000 MiB       h.free.argtypes = [ctypes.c_void_p]
    33  112.770 MiB    0.000 MiB       h.free.restype = None
    34 1101.918 MiB  989.148 MiB       gdal.Translate('/tmp/out.tif', 'byte.tif', options = '-co TILED=YES -outsize 100000 10000')
    35 1101.918 MiB    0.000 MiB       h.free(None)
    36 1101.918 MiB    0.000 MiB       dummy = 1

That doesn't make much sense to me.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com