[gdal-dev] Memory allocation issues on Android 11+ and scudo

[Even Rouault]

Hi,

didn't hear about Scudo before, but it seems it is a LLVM side project: 
https://llvm.org/docs/ScudoHardenedAllocator.html

So perhaps you could build and use it on Linux as shown in 
https://llvm.org/docs/ScudoHardenedAllocator.html#library

Besides a potential bug in the allocator, it might be that the S57 
driver has a memory allocation pattern that doesn't please Scudo.

Assuming that the problematic memory allocations are done using GDAL's 
VSIMalloc() (and not C++ new/delete), then have a look at the various 
#define that you can set at the top of port/cpl_vsisimple.cpp and can be 
used to trace memory allocations

// Uncomment to check consistent usage of VSIMalloc(), VSIRealloc(),
// VSICalloc(), VSIFree(), VSIStrdup().
// #define DEBUG_VSIMALLOC

// Uncomment to compute memory usage statistics.
// DEBUG_VSIMALLOC must also be defined.
// #define DEBUG_VSIMALLOC_STATS

// Uncomment to print every memory allocation or deallocation.
// DEBUG_VSIMALLOC must also be defined.
// #define DEBUG_VSIMALLOC_VERBOSE

// Number of bytes of the malloc/calloc/free that triggers a debug trace.
// Can be 0 for all allocs.
#define THRESHOLD_PRINT 10000

Even


Le 28/03/2022 à 14:58, Philippe Lelong a écrit :
>
> Hi,
>
>
> I am searching for this issue for months now, and cannot find any 
> solution.
>
>
> To make a long story short, we are using GDAL to decode OGR/S57 charts 
> for years now. We are facing numerous crashes under Android 11 and up 
> if and only if this Android 11 implementation is using SCUDO as a 
> memory allocator (if jemalloc is used no problems). We face this 
> problem with an old GDAL2.1.3 version, so we updated to GDAL 3.4.1 but 
> the issue is the same.
>
>
> What I can see is that the memory grows exponentially until no more 
> memory is available and crash, even on systems with huge memory 
> available while an Android device without SCUDO and very limited 
> memory (let's say 4Gb) in the same exact conditions, with the same 
> apk, runs perfectly. The logcat command show this:
>
> 03-28 12:40:34.255  4959  5005 W libc    : malloc(264196) failed: 
> returning null pointer
> 03-28 12:40:34.255  4959  5005 W libc    : malloc(264196) failed: 
> returning null pointer
> 03-28 12:40:34.256  4959  5005 W libc    : malloc(264196) failed: 
> returning null pointer
> 03-28 12:40:34.256  4959  5005 W libc    : malloc(264196) failed: 
> returning null pointer
> 03-28 12:40:34.612   630   630 D io_stats: !@ Write_top(KB): 
> kworker/u16:1(32583) 8
> 03-28 12:40:34.820  4959  5041 I scudo   : Scudo ERROR: out of memory 
> trying to allocate 64 bytes
> 03-28 12:40:34.820  4959  5042 I scudo   : Scudo ERROR: out of memory 
> trying to allocate 64 bytes
> 03-28 12:40:34.820  4959  5033 I scudo   : Scudo ERROR: out of memory 
> trying to allocate 64 bytes
> 03-28 12:40:34.820  4959  5031 I scudo   : Scudo ERROR: out of memory 
> trying to allocate 64 bytes
> 03-28 12:40:34.820  4959  5038 I scudo   : Scudo ERROR: out of memory 
> trying to allocate 64 bytes
> 03-28 12:40:34.820  4959  5040 I scudo   : Scudo ERROR: out of memory 
> trying to allocate 64 bytes
>
> and then crash
>
> Any help on how to debug and eventually fix this would be highly 
> appreciated.
>
> Best regards,
> Philippe from qtVlm development team.
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220328/7b5f56e9/attachment.html>