[gdal-dev] There's something strange about two tests that are very similar
Even Rouault
even.rouault at spatialys.com
Wed Mar 13 03:47:04 PDT 2024
Hi,
I've checkout'ed your branch locally and I can reproduce the error when
running the Python tests. Some of the CI checks give an interesting hint
about this being related to a UTF8 issue, and I wondered why
But running "ogrinfo
autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol"
outputs:
"""
ERROR 4: Error pMMPolygonLayer.MMArc.pF: Cannot open file
autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.arc
[QUALITY:LINEAGE:PROCESS1]
nOrganismes=1
history=C:\MiraMon\MM64.exe
date=20230628 16204988+0200
[QUALITY:LINEAGE:PROCESS1:ORGANISME_1]
IndividualName=Abel Pau
PositionName=T�cnic SIG
OrganisationName=Students and educational institutions
[...]
""""
So there are several problems:
1) On failure, you shouldn't output the content of the file in the error
message, to keep it brief the UTF-8 error comes from the fact that the
file content is not UTF-8 encoded)
2) There is an error opening the file.
3) Interestingly something in your code *deletes* file
autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol !
4) You should really invest in running Valgrind. After restoring the
deleted file, I can see:
$ valgrind ogrinfo
autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol
==1737168== Memcheck, a memory error detector
==1737168== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==1737168== Using Valgrind-3.15.0 and LibVEX; rerun with -h for
copyright info
==1737168== Command: ogrinfo
autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol
==1737168==
==1737168== Conditional jump or move depends on uninitialised value(s)
==1737168== at 0x4843C6F: strstr (in
/usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1737168== by 0x635FC44: MMReturnValueFromSectionINIFile
(mm_wrlayr.c:5019)
==1737168== by 0x636219A: MMCheck_REL_FILE (mm_wrlayr.c:5785)
==1737168== by 0x63586B0: MMInitPolygonLayer (mm_wrlayr.c:1665)
==1737168== by 0x6358BC6: MMInitLayerByType (mm_wrlayr.c:1761)
==1737168== by 0x636B818: MMInitLayerToRead (mm_rdlayr.c:142)
==1737168== by 0x634D6A7: OGRMiraMonLayer::OGRMiraMonLayer(char
const*, VSIVirtualHandle*, OGRSpatialReference const*, int, char**,
MiraMonVectMapInfo*) (ogrmiramonlayer.cpp:219)
==1737168== by 0x634C03E: OGRMiraMonDataSource::Open(char const*,
VSIVirtualHandle*, OGRSpatialReference const*, int, char**)
(ogrmiramondatasource.cpp:71)
==1737168== by 0x634C997: OGRMiraMonDriverOpen(GDALOpenInfo*)
(ogrmiramondriver.cpp:87)
==1737168== by 0x67DD68C: GDALDriver::Open(GDALOpenInfo*, bool)
(gdaldriver.cpp:116)
==1737168== by 0x67F61F4: GDALOpenEx (gdaldataset.cpp:3745)
==1737168== by 0x10AC95: GDALDataset::Open(char const*, unsigned int,
char const* const*, char const* const*, char const* const*)
(gdal_priv.h:835)
==1737168==
==1737168== Conditional jump or move depends on uninitialised value(s)
==1737168== at 0x483EF58: strlen (in
/usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1737168== by 0x5262132: CPLStrdup (cpl_conv.cpp:316)
==1737168== by 0x635FC6E: MMReturnValueFromSectionINIFile
(mm_wrlayr.c:5023)
==1737168== by 0x636219A: MMCheck_REL_FILE (mm_wrlayr.c:5785)
==1737168== by 0x63586B0: MMInitPolygonLayer (mm_wrlayr.c:1665)
==1737168== by 0x6358BC6: MMInitLayerByType (mm_wrlayr.c:1761)
==1737168== by 0x636B818: MMInitLayerToRead (mm_rdlayr.c:142)
==1737168== by 0x634D6A7: OGRMiraMonLayer::OGRMiraMonLayer(char
const*, VSIVirtualHandle*, OGRSpatialReference const*, int, char**,
MiraMonVectMapInfo*) (ogrmiramonlayer.cpp:219)
==1737168== by 0x634C03E: OGRMiraMonDataSource::Open(char const*,
VSIVirtualHandle*, OGRSpatialReference const*, int, char**)
(ogrmiramondatasource.cpp:71)
==1737168== by 0x634C997: OGRMiraMonDriverOpen(GDALOpenInfo*)
(ogrmiramondriver.cpp:87)
==1737168== by 0x67DD68C: GDALDriver::Open(GDALOpenInfo*, bool)
(gdaldriver.cpp:116)
==1737168== by 0x67F61F4: GDALOpenEx (gdaldataset.cpp:3745)
==1737168==
[...]
Le 12/03/2024 à 20:43, Abel Pau a écrit :
>
> >>> Are sure all needed files have been committed and pushed for the
> failing test?
>
> Yes, sure. 9 files are needed for oine polygon layer. Absolutely sure.
>
> >>> Perhaps an issue Windows CR-LF vs Unix LF end of line characters ? The
> default settings of git on Windows may mess with binary files. cf
> https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings
>
> I’ll take a look but in binary files the format are bytes of
> information or blanks (there is no line endings or something similar).
> The drivers Writes and reads bytes and it goes to different features
> using offsets of 4 or 8 bytes depending on the version. 4 in this
> case. Here is the format description for more information.
> https://www.miramon.cat/new_note/eng/notes/MiraMon_structured_vectors_file_format.pdf
>
> Taking a look to the log I can see an advice about GDALOpenEx (See below).
>
> Thinking about differences between extension POL and the other two
> (PNT and ARC) is that is the main.
>
> poDriver->SetMetadataItem(GDAL_DMD_EXTENSION, "pol");
>
> poDriver->SetMetadataItem(GDAL_DMD_EXTENSIONS, "pol arc pnt");
>
> I look into multiple examples where ds = gdal.OpenEx() is called with
> one, two or three arguments.
>
> So, could it be related with this issue?
>
> Thanks!!!
>
> THE LOG:
>
> /home/runner/work/gdal/gdal/build-ubuntu_22.04/autotest/ogr/ogr_miramon_vector.py:175:
>
>
> 17281
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17282>_
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _
>
> 17282
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17283>
>
>
> 17283
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17284>args
> = ('data/miramon/Polygons/SimplePolygons/SimplePolFile.pol',), kwargs
> = {}
>
> 17284
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17285>
>
>
> 17285
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17286>
> def OpenEx(*args, **kwargs) -> "GDALDatasetShadow *":
>
> 17286
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17287>
> r"""
>
> 17287
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17288>
> OpenEx(char const * utf8_path, unsigned int nOpenFlags=0, char **
> allowed_drivers=None, char ** open_options=None, char **
> sibling_files=None) -> Dataset
>
> 17288
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17289>
>
>
> 17289
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17290>
>
>
> 17290
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17291>
> Open a raster or vector file as a :py:class:`Dataset`.
>
> 17291
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17292>
> See :cpp:func:`GDALOpenEx`.
>
> 17292
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17293>
>
>
> 17293
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17294>
> Parameters
>
> 17294
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17295>
> ----------
>
> 17295
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17296>
> utf8_path : str
>
> 17296
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17297>
> name of the file to open
>
> 17297
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17298>
> flags : int
>
> 17298
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17299>
> Flags controlling how the Dataset is opened. Multiple ``gdal.OF_XXX``
> flags
>
> 17299
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17300>
> may be combined using the ``|`` operator. See :cpp:func:`GDALOpenEx`.
>
> 17300
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17301>
> allowed_drivers : list, optional
>
> 17301
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17302>
> A list of the names of drivers that may attempt to open the dataset.
>
> 17302
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17303>
> open_options : dict/list, optional
>
> 17303
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17304>
> A dict or list of name=value driver-specific opening options.
>
> 17304
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17305>
> sibling_files: list, optional
>
> 17305
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17306>
> A list of filenames that are auxiliary to the main filename
>
> 17306
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17307>
>
>
> 17307
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17308>
> Returns
>
> 17308
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17309>
> -------
>
> 17309
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17310>
> Dataset, or ``None`` on failure.
>
> 17310
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17311>
>
>
> 17311
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17312>
> See Also
>
> 17312
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17313>
> --------
>
> 17313
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17314>
> :py:func:`Open`
>
> 17314
> <https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17315>
> :py:func:`OpenShared`
>
> Le 12/03/2024 à 17:22, Abel Pau via gdal-dev a écrit :
>
> Hi,
>
> I have in a single test python file with 3 tests inside:
> gdal/autotest/ogr/ogr_miramon_vector.py at master · AbelPau/gdal
> (github.com)
> <https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L134>
>
> In some actions the first and second are Ok but the third is not.
> Example of failure:
>
> Some improvements for empty layers · AbelPau/gdal at 90a289c
> (github.com)
> <https://github.com/AbelPau/gdal/actions/runs/8251854865/job/22569933640#step:12:4863>
>
> Here you have the log (return _gdal.OpenEx(*args, **kwargs)
>
> 4912
> <https://github.com/AbelPau/gdal/actions/runs/8251854865/job/22569933640#step:12:4913>23:
> E RuntimeError: Cannot open file %s.) <--- This happen in the
> driver when open fails.
>
> The first two are OK:
>
> gdal/autotest/ogr/ogr_miramon_vector.py at master · AbelPau/gdal
> (github.com)
> <https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L48>
>
> gdal/autotest/ogr/ogr_miramon_vector.py at master · AbelPau/gdal
> (github.com)
> <https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L99>
>
> But this is not OK:
>
> gdal/autotest/ogr/ogr_miramon_vector.py at master · AbelPau/gdal
> (github.com)
> <https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L134>
>
> And the file are all on the correct place (I am sure about that):
>
> For the second one, for example:
>
> gdal/autotest/ogr/data/miramon/Arcs/SimpleArcs at master ·
> AbelPau/gdal (github.com)
> <https://github.com/AbelPau/gdal/tree/master/autotest/ogr/data/miramon/Arcs/SimpleArcs>
>
> And for the third one:
>
> gdal/autotest/ogr/data/miramon/Polygons/SimplePolygons at master ·
> AbelPau/gdal (github.com)
> <https://github.com/AbelPau/gdal/tree/master/autotest/ogr/data/miramon/Polygons/SimplePolygons>
>
> On local tests are perfectly OK (see below). So, after some days
> of trying to figure out WHY and what is de difference between 2
> and 3th test I have to ask you because it must be obvious but not
> to me.
>
> Any idea? Some permissions stuff?? THANKS!
>
> root at f5b8c1fb00ae:/gdal/build/autotest# python3 -m pytest -vv -p
> no:sugar --color=no ogr/ogr_miramon_vector.py
>
> ================================================= test session
> starts ==================================================
>
> platform linux -- Python 3.10.12, pytest-8.1.1, pluggy-1.4.0 --
> /usr/bin/python3
>
> cachedir: .pytest_cache
>
> benchmark: 4.0.0 (defaults: timer=time.perf_counter
> disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0
> calibration_precision=10 warmup=False warmup_iterations=100000)
>
> GDAL Build Info:
>
> PAM_ENABLED: YES
>
> OGR_ENABLED: YES
>
> CURL_ENABLED: YES
>
> CURL_VERSION: 7.81.0
>
> GEOS_ENABLED: YES
>
> GEOS_VERSION: 3.10.2-CAPI-1.16.0
>
> PROJ_BUILD_VERSION: 8.2.1
>
> PROJ_RUNTIME_VERSION: 8.2.1
>
> COMPILER: GCC 11.4.0
>
> GDAL_DOWNLOAD_TEST_DATA: undefined (tests relying on downloaded
> data may be skipped)
>
> GDAL_RUN_SLOW_TESTS: undefined (tests marked as "slow" will be
> skipped)
>
> rootdir: /gdal/build/autotest
>
> configfile: pytest.ini
>
> plugins: env-1.1.3, benchmark-4.0.0
>
> collected 3 items
>
> ogr/ogr_miramon_vector.py::test_ogr_miramon_simple_point
> PASSED [ 33%]
>
> ogr/ogr_miramon_vector.py::test_ogr_miramon_simple_arc PASSED [ 66%]
>
> ogr/ogr_miramon_vector.py::test_ogr_miramon_simple_polygon PASSED
> [100%]
>
> ================================================== 3 passed in
> 0.11s ===================================================
>
> *Abel Pau Garcia*
>
> *GIS developer*
>
>
>
> https://www.creaf.cat/sites/default/files/creaf-signature.png
>
> *a.pau at creaf.uab.cat* <mailto:a.pau at creaf.uab.cat>
>
> *Let's chat on Teams!*
> <https://teams.microsoft.com/l/chat/0/0?users=a.pau@creaf.uab.cat>
>
> *Tel. +34 934814277*
>
>
>
> https://www.creaf.cat/sites/default/files/so-en-signature.png
>
> https://www.creaf.cat/sites/default/files/twitter-icon-signature.png
> <https://twitter.com/CREAF_ecologia>https://www.creaf.cat/sites/default/files/linkedin-icon-signature.png
> <https://www.linkedin.com/company/1363052?trk=tyah&trkInfo=clickedVertical:company,clickedEntityId:1363052,idx:2-1-2,tarId:1465807877789,tas:creaf>https://www.creaf.cat/sites/default/files/youtube-icon-signature.png
> <https://www.youtube.com/c/creafecologia>https://www.creaf.cat/sites/default/files/instagram-icon-signature.png
> <https://www.instagram.com/CREAF_ecologia/>
>
> *www.creaf.cat* <http://www.creaf.cat>*| **http://blog.creaf.cat*
> <http://blog.creaf.cat>
>
>
>
> https://www.creaf.cat/sites/default/files/uab_logo_signatura.png
>
> CREAF. Campus UAB. Edifici C. 08193 Bellaterra (Barcelona)
>
>
> Before printing this electronic message, think about the environment.
>
> http://www.creaf.uab.cat/_signatura/line.gif
>
>
>
> _______________________________________________
>
> gdal-dev mailing list
>
> gdal-dev at lists.osgeo.org
>
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
--
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240313/fe70d4d4/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 3657 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240313/fe70d4d4/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 2547 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240313/fe70d4d4/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 505 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240313/fe70d4d4/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 446 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240313/fe70d4d4/attachment-0009.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.png
Type: image/png
Size: 553 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240313/fe70d4d4/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.png
Type: image/png
Size: 582 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240313/fe70d4d4/attachment-0011.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image007.jpg
Type: image/jpeg
Size: 2208 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240313/fe70d4d4/attachment-0002.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image008.jpg
Type: image/jpeg
Size: 1111 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240313/fe70d4d4/attachment-0003.jpg>
More information about the gdal-dev
mailing list