[gdal-dev] There's something strange about two tests that are very similar
Abel Pau
a.pau at creaf.uab.cat
Wed Mar 13 07:00:02 PDT 2024
Hi again,
MiraMon files have had INI files containing metadata information for ages.
To read and write sections and key/values from them, we use specific Windows functions (https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-getprivateprofilestring). When I started programming the driver, I had to create a function to do that as I couldn't use the Windows one.
I suspect that these functions are not accurate and are failing.
Is there any function in GDAL that can read INI files (at least to check if a section exists and to get the value of a key in an existing section)? If not, perhaps using the following could be considered: https://github.com/brofield/simpleini
It seems to have an MIT license... If it's possible to use this parser, what would be the procedure? I noticed that Even added a parser for some command line arguments and perhaps something similar can be done with that.
If not, is it possible for me to add some files in the MiraMonVector driver and use them?
It's better to use something existing and multiplatform than to "reinvent the wheel" with some new function that can fail (in fact, is failing now).
Thanks.
De: gdal-dev <gdal-dev-bounces at lists.osgeo.org> En nombre de Abel Pau via gdal-dev
Enviado el: dimecres, 13 de març de 2024 12:38
Para: Even Rouault <even.rouault at spatialys.com>; gdal-dev at lists.osgeo.org
Asunto: Re: [gdal-dev] There's something strange about two tests that are very similar
Hi Even,
thanks for your effort on that.
I’ll spend some time analyzing that but I can imagine where the problem is.
For your information our .REL files are generated in ANSI codification and when this kind of files are read I forgot to translate to UTF-8.
It’s very probable that this is the mistake.
I’ll tell you when I solve that!
About
1) it’s not my intention to output part of the content file (I think that is part of the name which is bad composed because is read from this REL file)
2) Due to codification, certainly
3) It’s odd and I didn’t want to do that, of course! I’ll review
4) I used Valgrind some time, And I’ll do again, thanks
Thanks for the report!
De: Even Rouault <even.rouault at spatialys.com<mailto:even.rouault at spatialys.com>>
Enviado el: dimecres, 13 de març de 2024 11:47
Para: Abel Pau <a.pau at creaf.uab.cat<mailto:a.pau at creaf.uab.cat>>; gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Asunto: Re: [gdal-dev] There's something strange about two tests that are very similar
Hi,
I've checkout'ed your branch locally and I can reproduce the error when running the Python tests. Some of the CI checks give an interesting hint about this being related to a UTF8 issue, and I wondered why
But running "ogrinfo autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol" outputs:
"""
ERROR 4: Error pMMPolygonLayer.MMArc.pF: Cannot open file autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.arc
[QUALITY:LINEAGE:PROCESS1]
nOrganismes=1
history=C:\MiraMon\MM64.exe
date=20230628 16204988+0200
[QUALITY:LINEAGE:PROCESS1:ORGANISME_1]
IndividualName=Abel Pau
PositionName=T�cnic SIG
OrganisationName=Students and educational institutions
[...]
""""
So there are several problems:
1) On failure, you shouldn't output the content of the file in the error message, to keep it brief the UTF-8 error comes from the fact that the file content is not UTF-8 encoded)
2) There is an error opening the file.
3) Interestingly something in your code *deletes* file autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol !
4) You should really invest in running Valgrind. After restoring the deleted file, I can see:
$ valgrind ogrinfo autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol
==1737168== Memcheck, a memory error detector
==1737168== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==1737168== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==1737168== Command: ogrinfo autotest/ogr/data/miramon/Polygons/SimplePolygons/SimplePolFile.pol
==1737168==
==1737168== Conditional jump or move depends on uninitialised value(s)
==1737168== at 0x4843C6F: strstr (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1737168== by 0x635FC44: MMReturnValueFromSectionINIFile (mm_wrlayr.c:5019)
==1737168== by 0x636219A: MMCheck_REL_FILE (mm_wrlayr.c:5785)
==1737168== by 0x63586B0: MMInitPolygonLayer (mm_wrlayr.c:1665)
==1737168== by 0x6358BC6: MMInitLayerByType (mm_wrlayr.c:1761)
==1737168== by 0x636B818: MMInitLayerToRead (mm_rdlayr.c:142)
==1737168== by 0x634D6A7: OGRMiraMonLayer::OGRMiraMonLayer(char const*, VSIVirtualHandle*, OGRSpatialReference const*, int, char**, MiraMonVectMapInfo*) (ogrmiramonlayer.cpp:219)
==1737168== by 0x634C03E: OGRMiraMonDataSource::Open(char const*, VSIVirtualHandle*, OGRSpatialReference const*, int, char**) (ogrmiramondatasource.cpp:71)
==1737168== by 0x634C997: OGRMiraMonDriverOpen(GDALOpenInfo*) (ogrmiramondriver.cpp:87)
==1737168== by 0x67DD68C: GDALDriver::Open(GDALOpenInfo*, bool) (gdaldriver.cpp:116)
==1737168== by 0x67F61F4: GDALOpenEx (gdaldataset.cpp:3745)
==1737168== by 0x10AC95: GDALDataset::Open(char const*, unsigned int, char const* const*, char const* const*, char const* const*) (gdal_priv.h:835)
==1737168==
==1737168== Conditional jump or move depends on uninitialised value(s)
==1737168== at 0x483EF58: strlen (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==1737168== by 0x5262132: CPLStrdup (cpl_conv.cpp:316)
==1737168== by 0x635FC6E: MMReturnValueFromSectionINIFile (mm_wrlayr.c:5023)
==1737168== by 0x636219A: MMCheck_REL_FILE (mm_wrlayr.c:5785)
==1737168== by 0x63586B0: MMInitPolygonLayer (mm_wrlayr.c:1665)
==1737168== by 0x6358BC6: MMInitLayerByType (mm_wrlayr.c:1761)
==1737168== by 0x636B818: MMInitLayerToRead (mm_rdlayr.c:142)
==1737168== by 0x634D6A7: OGRMiraMonLayer::OGRMiraMonLayer(char const*, VSIVirtualHandle*, OGRSpatialReference const*, int, char**, MiraMonVectMapInfo*) (ogrmiramonlayer.cpp:219)
==1737168== by 0x634C03E: OGRMiraMonDataSource::Open(char const*, VSIVirtualHandle*, OGRSpatialReference const*, int, char**) (ogrmiramondatasource.cpp:71)
==1737168== by 0x634C997: OGRMiraMonDriverOpen(GDALOpenInfo*) (ogrmiramondriver.cpp:87)
==1737168== by 0x67DD68C: GDALDriver::Open(GDALOpenInfo*, bool) (gdaldriver.cpp:116)
==1737168== by 0x67F61F4: GDALOpenEx (gdaldataset.cpp:3745)
==1737168==
[...]
Le 12/03/2024 à 20:43, Abel Pau a écrit :
>>> Are sure all needed files have been committed and pushed for the failing test?
Yes, sure. 9 files are needed for oine polygon layer. Absolutely sure.
>>> Perhaps an issue Windows CR-LF vs Unix LF end of line characters ? The default settings of git on Windows may mess with binary files. cf https://docs.github.com/en/get-started/getting-started-with-git/configuring-git-to-handle-line-endings
I’ll take a look but in binary files the format are bytes of information or blanks (there is no line endings or something similar). The drivers Writes and reads bytes and it goes to different features using offsets of 4 or 8 bytes depending on the version. 4 in this case. Here is the format description for more information. https://www.miramon.cat/new_note/eng/notes/MiraMon_structured_vectors_file_format.pdf
Taking a look to the log I can see an advice about GDALOpenEx (See below).
Thinking about differences between extension POL and the other two (PNT and ARC) is that is the main.
poDriver->SetMetadataItem(GDAL_DMD_EXTENSION, "pol");
poDriver->SetMetadataItem(GDAL_DMD_EXTENSIONS, "pol arc pnt");
I look into multiple examples where ds = gdal.OpenEx() is called with one, two or three arguments.
So, could it be related with this issue?
Thanks!!!
THE LOG:
/home/runner/work/gdal/gdal/build-ubuntu_22.04/autotest/ogr/ogr_miramon_vector.py:175:
17281<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17282>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
17282<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17283>
17283<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17284>args = ('data/miramon/Polygons/SimplePolygons/SimplePolFile.pol',), kwargs = {}
17284<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17285>
17285<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17286> def OpenEx(*args, **kwargs) -> "GDALDatasetShadow *":
17286<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17287> r"""
17287<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17288> OpenEx(char const * utf8_path, unsigned int nOpenFlags=0, char ** allowed_drivers=None, char ** open_options=None, char ** sibling_files=None) -> Dataset
17288<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17289>
17289<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17290>
17290<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17291> Open a raster or vector file as a :py:class:`Dataset`.
17291<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17292> See :cpp:func:`GDALOpenEx`.
17292<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17293>
17293<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17294> Parameters
17294<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17295> ----------
17295<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17296> utf8_path : str
17296<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17297> name of the file to open
17297<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17298> flags : int
17298<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17299> Flags controlling how the Dataset is opened. Multiple ``gdal.OF_XXX`` flags
17299<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17300> may be combined using the ``|`` operator. See :cpp:func:`GDALOpenEx`.
17300<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17301> allowed_drivers : list, optional
17301<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17302> A list of the names of drivers that may attempt to open the dataset.
17302<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17303> open_options : dict/list, optional
17303<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17304> A dict or list of name=value driver-specific opening options.
17304<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17305> sibling_files: list, optional
17305<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17306> A list of filenames that are auxiliary to the main filename
17306<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17307>
17307<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17308> Returns
17308<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17309> -------
17309<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17310> Dataset, or ``None`` on failure.
17310<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17311>
17311<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17312> See Also
17312<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17313> --------
17313<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17314> :py:func:`Open`
17314<https://github.com/AbelPau/gdal/actions/runs/8254132320/job/22577551385#step:16:17315> :py:func:`OpenShared`
Le 12/03/2024 à 17:22, Abel Pau via gdal-dev a écrit :
Hi,
I have in a single test python file with 3 tests inside: gdal/autotest/ogr/ogr_miramon_vector.py at master · AbelPau/gdal (github.com)<https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L134>
In some actions the first and second are Ok but the third is not. Example of failure:
Some improvements for empty layers · AbelPau/gdal at 90a289c (github.com)<https://github.com/AbelPau/gdal/actions/runs/8251854865/job/22569933640#step:12:4863>
Here you have the log (return _gdal.OpenEx(*args, **kwargs)
4912<https://github.com/AbelPau/gdal/actions/runs/8251854865/job/22569933640#step:12:4913>23: E RuntimeError: Cannot open file %s.) <--- This happen in the driver when open fails.
The first two are OK:
gdal/autotest/ogr/ogr_miramon_vector.py at master · AbelPau/gdal (github.com)<https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L48>
gdal/autotest/ogr/ogr_miramon_vector.py at master · AbelPau/gdal (github.com)<https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L99>
But this is not OK:
gdal/autotest/ogr/ogr_miramon_vector.py at master · AbelPau/gdal (github.com)<https://github.com/AbelPau/gdal/blob/master/autotest/ogr/ogr_miramon_vector.py#L134>
And the file are all on the correct place (I am sure about that):
For the second one, for example:
gdal/autotest/ogr/data/miramon/Arcs/SimpleArcs at master · AbelPau/gdal (github.com)<https://github.com/AbelPau/gdal/tree/master/autotest/ogr/data/miramon/Arcs/SimpleArcs>
And for the third one:
gdal/autotest/ogr/data/miramon/Polygons/SimplePolygons at master · AbelPau/gdal (github.com)<https://github.com/AbelPau/gdal/tree/master/autotest/ogr/data/miramon/Polygons/SimplePolygons>
On local tests are perfectly OK (see below). So, after some days of trying to figure out WHY and what is de difference between 2 and 3th test I have to ask you because it must be obvious but not to me.
Any idea? Some permissions stuff?? THANKS!
root at f5b8c1fb00ae:/gdal/build/autotest# python3 -m pytest -vv -p no:sugar --color=no ogr/ogr_miramon_vector.py
================================================= test session starts ==================================================
platform linux -- Python 3.10.12, pytest-8.1.1, pluggy-1.4.0 -- /usr/bin/python3
cachedir: .pytest_cache
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
GDAL Build Info:
PAM_ENABLED: YES
OGR_ENABLED: YES
CURL_ENABLED: YES
CURL_VERSION: 7.81.0
GEOS_ENABLED: YES
GEOS_VERSION: 3.10.2-CAPI-1.16.0
PROJ_BUILD_VERSION: 8.2.1
PROJ_RUNTIME_VERSION: 8.2.1
COMPILER: GCC 11.4.0
GDAL_DOWNLOAD_TEST_DATA: undefined (tests relying on downloaded data may be skipped)
GDAL_RUN_SLOW_TESTS: undefined (tests marked as "slow" will be skipped)
rootdir: /gdal/build/autotest
configfile: pytest.ini
plugins: env-1.1.3, benchmark-4.0.0
collected 3 items
ogr/ogr_miramon_vector.py::test_ogr_miramon_simple_point PASSED [ 33%]
ogr/ogr_miramon_vector.py::test_ogr_miramon_simple_arc PASSED [ 66%]
ogr/ogr_miramon_vector.py::test_ogr_miramon_simple_polygon PASSED [100%]
================================================== 3 passed in 0.11s ===================================================
Abel Pau Garcia
GIS developer
[https://www.creaf.cat/sites/default/files/creaf-signature.png]
a.pau at creaf.uab.cat<mailto:a.pau at creaf.uab.cat>
Let's chat on Teams!<https://teams.microsoft.com/l/chat/0/0?users=a.pau@creaf.uab.cat>
Tel. +34 934814277
[https://www.creaf.cat/sites/default/files/so-en-signature.png]
[https://www.creaf.cat/sites/default/files/twitter-icon-signature.png]<https://twitter.com/CREAF_ecologia>[https://www.creaf.cat/sites/default/files/linkedin-icon-signature.png]<https://www.linkedin.com/company/1363052?trk=tyah&trkInfo=clickedVertical:company,clickedEntityId:1363052,idx:2-1-2,tarId:1465807877789,tas:creaf>[https://www.creaf.cat/sites/default/files/youtube-icon-signature.png]<https://www.youtube.com/c/creafecologia>[https://www.creaf.cat/sites/default/files/instagram-icon-signature.png]<https://www.instagram.com/CREAF_ecologia/>
www.creaf.cat<http://www.creaf.cat> | http://blog.creaf.cat
[https://www.creaf.cat/sites/default/files/uab_logo_signatura.png]
CREAF. Campus UAB. Edifici C. 08193 Bellaterra (Barcelona)
Before printing this electronic message, think about the environment.
_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
http://www.spatialys.com
My software is free, but my time generally not.
--
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240313/064fc926/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 3657 bytes
Desc: image001.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240313/064fc926/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 2547 bytes
Desc: image002.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240313/064fc926/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 505 bytes
Desc: image003.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240313/064fc926/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 446 bytes
Desc: image004.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240313/064fc926/attachment-0009.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image005.png
Type: image/png
Size: 553 bytes
Desc: image005.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240313/064fc926/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.png
Type: image/png
Size: 582 bytes
Desc: image006.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240313/064fc926/attachment-0011.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image007.jpg
Type: image/jpeg
Size: 2208 bytes
Desc: image007.jpg
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240313/064fc926/attachment-0002.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image008.jpg
Type: image/jpeg
Size: 1111 bytes
Desc: image008.jpg
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240313/064fc926/attachment-0003.jpg>
More information about the gdal-dev
mailing list