[gdal-dev] There's something strange about two tests that are very similar

Abel Pau a.pau at creaf.uab.cat
Thu Mar 14 10:11:58 PDT 2024 

Thanks Even,
your feedback helped a lot.

De: Even Rouault <even.rouault at spatialys.com>
Enviado el: dijous, 14 de març de 2024 10:13
Para: Abel Pau <a.pau at creaf.uab.cat>; gdal-dev at lists.osgeo.org
Asunto: Re: [gdal-dev] There's something strange about two tests that are very similar


gdb is not going to help for test_ogr_basic_10. As you can see in the test, it "runexternal", that is it forks a child process, so gdb isn't going to debug this child process by default.

The easiest is to run "test_ogrsf -all_drivers" directly / under gdb / under valgrind

Actually, I've just tried that on my system. Nothing under Valgrind, but the execution time was quite slow once it reaches the Miramon driver, and breaking under gdb, I see the following

INFO: MiraMonVector: Creation of /foo/test.pol should have failed.
INFO: MiraMonVector: CreateFeature() at line 780 succeeded but failed to assign FID to feature.
^C--Type <RET> for more, q to quit, c to continue without paging--

Thread 1 "test_ogrsf" received signal SIGINT, Interrupt.
__memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:151
151    ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: Aucun fichier ou dossier de ce type.
(gdb) bt
#0  __memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:151
#1  0x00007ffff5e40ed5 in CPLCalloc (nCount=1, nSize=104857600) at /home/even/gdal/MiraMon-Vector-driver/port/cpl_conv.cpp:164
#2  0x00007ffff6f39a33 in MMMoveFromFileToFile (pSrcFile=0x555555bcd870, pDestFile=0x5555562b3b90, nOffset=0x5555561f3990)
    at /home/even/gdal/MiraMon-Vector-driver/ogr/ogrsf_frmts/miramon/mm_wrlayr.c:2693
#3  0x00007ffff6f388fb in MMCloseArcLayer (hMiraMonLayer=0x5555561e4f90) at /home/even/gdal/MiraMon-Vector-driver/ogr/ogrsf_frmts/miramon/mm_wrlayr.c:2150
#4  0x00007ffff6f38dc5 in MMCloseLayer (hMiraMonLayer=0x5555561e4f90) at /home/even/gdal/MiraMon-Vector-driver/ogr/ogrsf_frmts/miramon/mm_wrlayr.c:2283
#5  0x00007ffff6f2d955 in OGRMiraMonLayer::~OGRMiraMonLayer (this=0x5555561d5aa0, __in_chrg=<optimized out>) at /home/even/gdal/MiraMon-Vector-driver/ogr/ogrsf_frmts/miramon/ogrmiramonlayer.cpp:535
#6  0x00007ffff6f2de22 in OGRMiraMonLayer::~OGRMiraMonLayer (this=0x5555561d5aa0, __in_chrg=<optimized out>) at /home/even/gdal/MiraMon-Vector-driver/ogr/ogrsf_frmts/miramon/ogrmiramonlayer.cpp:619
#7  0x00007ffff6f2af7a in OGRMiraMonDataSource::~OGRMiraMonDataSource (this=0x555556265cb0, __in_chrg=<optimized out>)
    at /home/even/gdal/MiraMon-Vector-driver/ogr/ogrsf_frmts/miramon/ogrmiramondatasource.cpp:50
#8  0x00007ffff6f2b014 in OGRMiraMonDataSource::~OGRMiraMonDataSource (this=0x555556265cb0, __in_chrg=<optimized out>)
    at /home/even/gdal/MiraMon-Vector-driver/ogr/ogrsf_frmts/miramon/ogrmiramondatasource.cpp:57
#9  0x00007ffff73d61a3 in GDALClose (hDS=0x555556265cb0) at /home/even/gdal/MiraMon-Vector-driver/gcore/gdaldataset.cpp:4128
#10 0x000055555555e167 in TestCreateLayer (poDriver=0x55555564e3f0, eGeomType=wkbMultiPoint) at /home/even/gdal/MiraMon-Vector-driver/apps/test_ogrsf.cpp:977
#11 0x000055555555ec61 in TestCreate (poDriver=0x55555564e3f0, bFromAllDrivers=1) at /home/even/gdal/MiraMon-Vector-driver/apps/test_ogrsf.cpp:1094
#12 0x000055555555bd56 in ThreadFunctionInternal (psContext=0x7fffffffdb60) at /home/even/gdal/MiraMon-Vector-driver/apps/test_ogrsf.cpp:325
#13 0x000055555555bc26 in ThreadFunction (user_data=0x7fffffffdb60) at /home/even/gdal/MiraMon-Vector-driver/apps/test_ogrsf.cpp:283
#14 0x000055555555ba62 in main (nArgc=2, papszArgv=0x5555556521b0) at /home/even/gdal/MiraMon-Vector-driver/apps/test_ogrsf.cpp:233

Actually looking at your MMMoveFromFileToFile(), I see it is allocating a working buffer of 100 MB. That's much larger than what you really need. A buffer of 64 KB or so would be equally performant. This function might also leak the buffer at line 2705.

"INFO: MiraMonVector: Creation of /foo/test.pol should have failed." also indicates some logic error in your driver.

and

$ gdb --args ogr2ogr /foo/test.pol autotest/ogr/data/poly.shp

crashes at

ERROR 4: Error pMMPolygonLayer->pF: Cannot open file /foo/test.pol.

Program received signal SIGSEGV, Segmentation fault.
MMAddArcRecordToMMDB (hMiraMonLayer=0x7fffebbe48f0, hMMFeature=0x7fffebc030d0, nElemCount=0, pArcHeader=0x555555af3940) at /home/even/gdal/MiraMon-Vector-driver/ogr/ogrsf_frmts/miramon/mm_wrlayr.c:6602
6602                                     pMMArcLayer->MMAdmDB.pMMBDXP->nRecords + 1))
(gdb) bt
#0  MMAddArcRecordToMMDB (hMiraMonLayer=0x7fffebbe48f0, hMMFeature=0x7fffebc030d0, nElemCount=0, pArcHeader=0x555555af3940)
    at /home/even/gdal/MiraMon-Vector-driver/ogr/ogrsf_frmts/miramon/mm_wrlayr.c:6602
#1  0x00007ffff6f3ccd6 in MMCreateFeaturePolOrArc (hMiraMonLayer=0x7fffebbe48f0, hMMFeature=0x7fffebc030d0) at /home/even/gdal/MiraMon-Vector-driver/ogr/ogrsf_frmts/miramon/mm_wrlayr.c:4112
#2  0x00007ffff6f3de4e in MMAddFeature (hMiraMonLayer=0x7fffebbe48f0, hMiraMonFeature=0x7fffebc030d0) at /home/even/gdal/MiraMon-Vector-driver/ogr/ogrsf_frmts/miramon/mm_wrlayr.c:4598
#3  0x00007ffff6f31787 in OGRMiraMonLayer::MMWriteGeometry (this=0x7fffebbc6010) at /home/even/gdal/MiraMon-Vector-driver/ogr/ogrsf_frmts/miramon/ogrmiramonlayer.cpp:1718
#4  0x00007ffff6f30e01 in OGRMiraMonLayer::MMProcessGeometry (this=0x7fffebbc6010, hGeom=0x555555a30e10, poFeature=0x555555a2d7c0, bcalculateRecord=1 '\001')
    at /home/even/gdal/MiraMon-Vector-driver/ogr/ogrsf_frmts/miramon/ogrmiramonlayer.cpp:1497
#5  0x00007ffff6f30a8c in OGRMiraMonLayer::MMProcessMultiGeometry (this=0x7fffebbc6010, hGeom=0x555555a30e10, poFeature=0x555555a2d7c0)
    at /home/even/gdal/MiraMon-Vector-driver/ogr/ogrsf_frmts/miramon/ogrmiramonlayer.cpp:1389
#6  0x00007ffff6f31027 in OGRMiraMonLayer::ICreateFeature (this=0x7fffebbc6010, poFeature=0x555555a2d7c0) at /home/even/gdal/MiraMon-Vector-driver/ogr/ogrsf_frmts/miramon/ogrmiramonlayer.cpp:1549
#7  0x00007ffff735ec23 in OGRLayer::CreateFeature (this=0x7fffebbc6010, poFeature=0x555555a2d7c0) at /home/even/gdal/MiraMon-Vector-driver/ogr/ogrsf_frmts/generic/ogrlayer.cpp:733
#8  0x00007ffff75c45fc in LayerTranslator::Translate (this=0x7fffffffd9a0, poFeatureIn=0x0, psInfo=0x555555a2b700, nCountLayerFeatures=0, pnReadFeatureCount=0x0, nTotalEventsDone=@0x7fffffffd618: 0,
    pfnProgress=0x0, pProgressArg=0x0, psOptions=0x555555640dc0) at /home/even/gdal/MiraMon-Vector-driver/apps/ogr2ogr_lib.cpp:6292
#9  0x00007ffff75b9d7b in GDALVectorTranslate (pszDest=0x555555640348 "/foo/test.pol", hDstDS=0x0, nSrcCount=1, pahSrcDS=0x7fffffffdaf0, psOptionsIn=0x5555556403c0, pbUsageError=0x7fffffffdadc)
    at /home/even/gdal/MiraMon-Vector-driver/apps/ogr2ogr_lib.cpp:3459
#10 0x000055555555a730 in main (nArgc=3, papszArgv=0x5555556401b0) at /home/even/gdal/MiraMon-Vector-driver/apps/ogr2ogr_bin.cpp:451

By the way, you might want to rebase again against latest master. I've just merged https://github.com/OSGeo/gdal/pull/9454 which re-activate the ASAN job that I had disabled yesterday night. We had some flakiness lately with the ASAN job. hopefully it is more stable now...
Le 14/03/2024 à 09:51, Abel Pau a écrit :
Sorry,
I merged from my phone and I think I did it wrong.

At this time actions are going well except the same that didn’t work at the beginning of this message. The MacOS one.
Merge remote-tracking branch 'upstream/master' into MiraMon-Vector-dr… · AbelPau/gdal at fa6f9e0 (github.com)<https://github.com/AbelPau/gdal/actions/runs/8277097153/job/22646788426#step:6:2463>

Your proposal of rebasing didn’t solved the problem in macOS.

So, I’ll try to get an image of a mac and compile, but when I tried to use a Virtual Box it was extremly slow.

Thanks for any input to try to get some call stack with the error described below.
Using gdb I obtain
ogr/ogr_basic_test.py ✓                                                                                 100% ██████████

Results (7.72s):
       1 passed
[Inferior 1 (process 15792) exited normally]

And the same with Valgrind.
Thanks!

______________________________ test_ogr_basic_10 _______________________________
2464<https://github.com/AbelPau/gdal/actions/runs/8277097153/job/22646788426#step:6:2465>42:
2465<https://github.com/AbelPau/gdal/actions/runs/8277097153/job/22646788426#step:6:2466>42: def test_ogr_basic_10():
2466<https://github.com/AbelPau/gdal/actions/runs/8277097153/job/22646788426#step:6:2467>42:
2467<https://github.com/AbelPau/gdal/actions/runs/8277097153/job/22646788426#step:6:2468>42: import test_cli_utilities
2468<https://github.com/AbelPau/gdal/actions/runs/8277097153/job/22646788426#step:6:2469>42:
2469<https://github.com/AbelPau/gdal/actions/runs/8277097153/job/22646788426#step:6:2470>42: if test_cli_utilities.get_test_ogrsf_path() is None:
2470<https://github.com/AbelPau/gdal/actions/runs/8277097153/job/22646788426#step:6:2471>42: pytest.skip()
2471<https://github.com/AbelPau/gdal/actions/runs/8277097153/job/22646788426#step:6:2472>42:
2472<https://github.com/AbelPau/gdal/actions/runs/8277097153/job/22646788426#step:6:2473>42: ret = gdaltest.runexternal(
2473<https://github.com/AbelPau/gdal/actions/runs/8277097153/job/22646788426#step:6:2474>42: test_cli_utilities.get_test_ogrsf_path() + " -all_drivers"
2474<https://github.com/AbelPau/gdal/actions/runs/8277097153/job/22646788426#step:6:2475>42: )
2475<https://github.com/AbelPau/gdal/actions/runs/8277097153/job/22646788426#step:6:2476>42:
2476<https://github.com/AbelPau/gdal/actions/runs/8277097153/job/22646788426#step:6:2477>42: > assert "INFO" in ret
2477<https://github.com/AbelPau/gdal/actions/runs/8277097153/job/22646788426#step:6:2478>42: E AssertionError: assert 'INFO' in '\nERROR ret code = -4'
2478<https://github.com/AbelPau/gdal/actions/runs/8277097153/job/22646788426#step:6:2479>42:
2479<https://github.com/AbelPau/gdal/actions/runs/8277097153/job/22646788426#step:6:2480>42: /Users/runner/work/gdal/gdal/build/autotest/ogr/ogr_basic_test.py:454: AssertionError


De: gdal-dev <gdal-dev-bounces at lists.osgeo.org><mailto:gdal-dev-bounces at lists.osgeo.org> En nombre de Abel Pau via gdal-dev
Enviado el: dijous, 14 de març de 2024 7:44
Para: Even Rouault <even.rouault at spatialys.com><mailto:even.rouault at spatialys.com>; gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Asunto: Re: [gdal-dev] There's something strange about two tests that are very similar

Hi,
I rebased yesterday, after your message, but same error appeared and some more.
I rebased al 7.00 again and it seems that there are again some errors in linux builds.

I assume it's because of lasts code updates. I'll wait until something stable to rebase again.

Enviat des de l'Outlook per a l'Android<https://aka.ms/AAb9ysg>
________________________________
From: Even Rouault <even.rouault at spatialys.com<mailto:even.rouault at spatialys.com>>
Sent: Thursday, March 14, 2024 12:15:38 AM
To: Abel Pau <a.pau at creaf.uab.cat<mailto:a.pau at creaf.uab.cat>>; gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org> <gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>>
Subject: Re: [gdal-dev] There's something strange about two tests that are very similar


try rebasing on top of latest master. It looks like the errors are only those fixed per https://github.com/OSGeo/gdal/commit/6703d3071de7155d320a39a580f27230428dcaca

--

http://www.spatialys.com

My software is free, but my time generally not.

--

http://www.spatialys.com

My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240314/8ea28379/attachment-0001.htm>

More information about the gdal-dev mailing list