[gdal-dev] Latest GDAL version not transforming lat/lng into geom column

Mon Nov 21 03:16:05 PST 2022

Hi,

Did you ever notice that the result from GDAL 3.5.1 is wrong as well? You csv file has coordinates with a comma as a decimal separator and enclosed between double quotes
my_file.csv
=========
CD_MES,store_id,date_id,ticket_id,hour_id,lat,lon
202112,101,04/12/21,1,11,"19,38358","-99,183735"
202112,101,25/12/21,31,15,"19,38066","-99,183405"

The result that you considered right cuts the decimals
"POINT (-99 19)","202112","101",04/12/21,"1","11"
and I think that it is worse result that not getting coordinates at all.

Editing the csv file to present coordinates as numbers with point as a decimal separator gives correct result
my_file.csv
=========
CD_MES,store_id,date_id,ticket_id,hour_id,lat,lon
202112,101,04/12/21,1,11,19.38358,-99.183735
202112,101,25/12/21,31,15,19.38066,-99.183405
ogrinfo
======
"POINT (-99.183735 19.38358)","202112","101",04/12/21,"1","11"
"POINT (-99.183405 19.38066)","202112","101",25/12/21,"31","15"

-Jukka Rahkonen-

Lähettäjä: gdal-dev <gdal-dev-bounces at lists.osgeo.org> Puolesta Rahkonen Jukka
Lähetetty: maanantai 21. marraskuuta 2022 13.07
Vastaanottaja: Moises Calzado <mcalzado at carto.com>
Kopio: 'gdal-dev at lists.osgeo.org' (gdal-dev at lists.osgeo.org) <gdal-dev at lists.osgeo.org>
Aihe: Re: [gdal-dev] Latest GDAL version not transforming lat/lng into geom column

Hi,

I could reproduce with OSGeo4W version GDAL 3.6.0, released 2022/11/06.  I have also a bit older version GDAL 3.6.0dev from gisinternals.com, probably installed 2022-05-16, and that version finds the geometries. So if it is not because of differences in build systems then something has happened after mid of May 2022.

-Jukka Rahkonen-

Lähettäjä: Moises Calzado <mcalzado at carto.com<mailto:mcalzado at carto.com>>
Lähetetty: maanantai 21. marraskuuta 2022 11.50
Vastaanottaja: Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>>
Aihe: Re: [gdal-dev] Latest GDAL version not transforming lat/lng into geom column

Hi Jukka,

Thanks so much for taking a look at this.

This is the result when running the command in the 3.5.1 version:

GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]
WKT,String,String,String,String,String
geom,CD_MES,store_id,date_id,ticket_id,hour_id
"POINT (-99 19)","202112","101",04/12/21,"1","11"
"POINT (-99 19)","202112","101",25/12/21,"31","15"

And this is what I obtained in the 3.6.0 version:

GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]
WKT,String,String,String,String,String
geom,CD_MES,store_id,date_id,ticket_id,hour_id
,"202112","101",04/12/21,"1","11"
,"202112","101",25/12/21,"31","15"

That's the file that I've used for testing: https://storage.googleapis.com/import-testing-stream/my_file.csv<https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstorage.googleapis.com%2Fimport-testing-stream%2Fmy_file.csv&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C7597a8d011e749c72a0a08dacbb0844e%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638046256280278841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XU3u4Oy0HUhuyLS%2FBuDD6XSWoNzVqNYuv57zudr29yU%3D&reserved=0>

El lun, 21 nov 2022 a las 10:28, Rahkonen Jukka (<jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>>) escribió:
Hi,

Please add a one line test file "my_file.csv" so we can try to reproduce. Confirm first that it fails with your GDAL version.

-Jukka Rahkonen-

Lähettäjä: gdal-dev <gdal-dev-bounces at lists.osgeo.org<mailto:gdal-dev-bounces at lists.osgeo.org>> Puolesta Moises Calzado via gdal-dev
Lähetetty: maanantai 21. marraskuuta 2022 11.21
Vastaanottaja: gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Aihe: [gdal-dev] Latest GDAL version not transforming lat/lng into geom column

Hello everyone,

We've just updated to the latest GDAL version (v3.6.0) and it seems that something is not working correctly when trying to obtain a geom column from a CSV containing latitudes and longitudes.

This is the command that is being used:

ogr2ogr -f CSV -skipfailures -makevalid /vsistdout/ CSV:my_file.csv -simplify 0.00001 -dim XY -t_srs EPSG:4326 -lco GEOMETRY=AS_WKT -lco GEOMETRY_NAME=geom -lco CREATE_CSVT=YES -s_srs EPSG:4326 -oo KEEP_GEOM_COLUMNS=NO -oo X_POSSIBLE_NAMES=point_longitude,longitude,longitud,lon,Lon,Longitude,longitudedecimal,decimallongitude,decimallong,lng,long,Lng -oo Y_POSSIBLE_NAMES=latitude,latitud,lati,lat,Latitude,decimallat,decimallatitude,latitudedecimal,point_latitude -oo GEOM_POSSIBLE_NAMES=geom,Geom,geometry,the_geom,wkt,wkb,wkt_geometry,wkb_geometry -oo EMPTY_STRING_AS_NULL=YES -oo QUOTED_FIELDS_AS_STRING=NO -lco PRECISION=NO -oo AUTODETECT_SIZE_LIMIT=500000

The file that is being processed contains a column called lat and another one called lon, and when I execute the same process on a docker container running the version 3.5.1 of GDAL it works like a charm. We've also tried to execute this process on the 3.5.3 version, and it also fails. Is that expected?

Looking forward to your response,
Regards!

--
Moises Calzado

Support Engineer

(US) +1 917 463 3232 | (ES) +34 911 165 823 | mcalzado at carto.com<mailto:mcalzado at carto.com>
[Lähettäjä poisti kuvan.]<https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.carto.com%2F&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C7597a8d011e749c72a0a08dacbb0844e%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638046256280278841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=1A8HxLvT6%2FV0p9wIm6AReJJebpOPu3t%2Br6WRm6mRhN8%3D&reserved=0>

--
Moises Calzado

Support Engineer

(US) +1 917 463 3232 | (ES) +34 911 165 823 | mcalzado at carto.com<mailto:mcalzado at carto.com>
[Lähettäjä poisti kuvan.]<https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.carto.com%2F&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7C7597a8d011e749c72a0a08dacbb0844e%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C0%7C638046256280278841%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=1A8HxLvT6%2FV0p9wIm6AReJJebpOPu3t%2Br6WRm6mRhN8%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221121/e569f225/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 823 bytes
Desc: image001.jpg
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20221121/e569f225/attachment-0001.jpg>