<div dir="ltr">Can something such as<div>head -n -2</div><div>Be part of the pipeline?</div><div><br></div><div>The 3 text files are being combined into 1 stream.</div><div><ul><li>Line 1 CRS/SRID from the .prj</li><li>Line 2 Types from the .cvst</li><li>Line 3 to the end from the .csv</li></ul></div><div>Which is great in some ways as the SRID does not go missing and header info is at the head.</div><div><br></div><div>It is just that I found from line 3 to the end were well formed with the renamed geometry column but I am testing on Windows 10 with 3.6.</div><div><br></div><div>I do not know if /vsizip/ as output is allowed or works i.e. all three text files as one streamed zip file then extract just the CSV file later in the process.</div><div><br></div><div>Moving to a one file spatial format as mentioned above might help. It is just that a GeoCSV dataset is a combination of three files.</div><div><br></div><div>Maybe a many-to-one-back-to-many-scenario might help. </div><div><br></div><div>There are several multi-file spatial formats that would need to be zipped so that you could stream just one thing.</div><div><br></div><div>I hope that makes sense.</div><div><br></div><div><br></div><div>. </div><div><br><div><br></div></div></div><div dir="ltr"></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 5, 2023 at 2:58 AM Rahkonen Jukka <<a href="mailto:jukka.rahkonen@maanmittauslaitos.fi" target="_blank">jukka.rahkonen@maanmittauslaitos.fi</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div lang="FI">
<div>
<p class="MsoNormal"><span>Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Have you considered to output GeoJSONseq
<a href="https://gdal.org/drivers/vector/geojsonseq.html" target="_blank">https://gdal.org/drivers/vector/geojsonseq.html</a> instead of CSV, that for my mind is a workaround as a geodata format. Maybe JSON could handle your newlines by the same.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">-Jukka Rahkonen-<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b>Lähettäjä:</b> gdal-dev <<a href="mailto:gdal-dev-bounces@lists.osgeo.org" target="_blank">gdal-dev-bounces@lists.osgeo.org</a>>
<b>Puolesta </b>Moises Calzado via gdal-dev<br>
<b>Lähetetty:</b> perjantai 5. toukokuuta 2023 12.32<br>
<b>Vastaanottaja:</b> <a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<b>Aihe:</b> Re: [gdal-dev] Ogr2ogr CSV driver not handling correctly line breaks inside columns<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hi Even! <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I've just created the two issues:<u></u><u></u></p>
<div>
<p class="MsoNormal">- <a href="https://github.com/OSGeo/gdal/issues/7699" target="_blank">https://github.com/OSGeo/gdal/issues/7699</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">- <a href="https://github.com/OSGeo/gdal/issues/7700" target="_blank">https://github.com/OSGeo/gdal/issues/7700</a><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Robert, as I explained before, we need the `/vsistdout/` driver as we're processing the file in streaming mode, so we can't save the result to the storage.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Unforteunately, the problem arises when using that driver.<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">El jue, 4 may 2023 a las 15:39, Even Rouault (<<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>>) escribió:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p>Moises,<u></u><u></u></p>
<p>please fild 2 issues in the github issue tracker:<u></u><u></u></p>
<p>- one about /vsistdout/ where .csvt and .prj content shouldn't be emitted<u></u><u></u></p>
<p>- one about decoupling the layer GEOMETRY_NAME creation option with CREATE_CSVT=YES<u></u><u></u></p>
<p>Even<u></u><u></u></p>
<div>
<p class="MsoNormal">Le 04/05/2023 à 13:58, Moises Calzado via gdal-dev a écrit :<u></u><u></u></p>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<p class="MsoNormal">Hi Robert! <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I think that we're losing a bit the main issue that we reported, as in fact the problem is related with line breaks in the output generated while using /vsistdout and the CREATE_CSVT=YES option.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Even pointed out that avoiding that flag it works as expected, but when it's used the generated output is not okay as the "Fields with embedded line breaks must be quoted" rule is not followed.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">IMHO although the generated output is not a CSV itself, we should be able to delete the first two lines (projection info and types) and deal with the rest of the content as a CSV.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">What we're doing is streaming the output of the /vsistdout driver to another process that perform some steps with the resultant CSV. In all cases it works correctly, as the output of the ogr2ogr execution is a valid CSV when deleting the
first two lines, but in the case reported in my first email it's not.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">The CREATE_CSVT=YES option is mandatory for us as for the moment, it's requires to use the GEOMETRY_NAME=<b>geom
</b>one, so we don't have any workaround.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Just wanted to confirm if that's expected for you (generating an output that it's not a valid CSV in the end)!<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">El mié, 3 may 2023 a las 21:05, Robert Hewlett (<<a href="mailto:rob.hewy@gmail.com" target="_blank">rob.hewy@gmail.com</a>>) escribió:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal">Hi,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I just tested with : GDAL 3.6.4, released 2023/04/17<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Using the ogr2ogr as follows:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">ogr2ogr -f CSV poi_out.csv poi.shp -lco CREATE_CSVT=YES<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I get three files but no geometry<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">ogr2ogr -f CSV poi_out.csv poi.shp -lco CREATE_CSVT=YES -lco GEOMETRY=AS_WKT<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I get three file with the geometry as WKT with the column name WKT<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><b>WKT</b>,id,poi_name,poi_types<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">"POINT (508878.602179846 5433913.2763688)","1",crescent,"4"<br>
"POINT (517836.918121302 5447702.01715829)","2",Tynehead Regional Park,"1"<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">ogr2ogr -f CSV poi_out.csv poi.shp -lco CREATE_CSVT=YES -lco GEOMETRY=AS_WKT -lco GEOMETRY_NAME=<b>geom</b><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I get three file with the geometry as WKT but the column called
<b>geom</b><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><b>geom</b>,id,poi_name,poi_types<br>
"POINT (508878.602179846 5433913.2763688)","1",crescent,"4"<br>
"POINT (517836.918121302 5447702.01715829)","2",Tynehead Regional Park,"1"<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">What does<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><b>ogr2ogr --version </b><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">report back<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, May 3, 2023 at 9:38 AM Robert Hewlett <<a href="mailto:rob.hewy@gmail.com" target="_blank">rob.hewy@gmail.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">Hi, <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Not to start a controversy but it feels like the standard hints at three files. Did the standard change?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">If it is three files which works for me in QGIS and geopandas i.e. data lands where it is suppose to, then more layer creations options are needed to handle the SRID/CRS<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">CREATE_PRJ=YES/NO<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">or -t_srs and/or -s_srs triggers the dot-prj file being created.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Just saying <span style="font-family:"Segoe UI Emoji",sans-serif">
😊</span>.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">In the meantime would a short python script help parse the one file into three?<u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, May 3, 2023 at 9:16 AM Moises Calzado via gdal-dev <<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">Hi Robert, <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Yes, we're getting one with all the info!<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">El mié, 3 may 2023 a las 18:14, Robert Hewlett (<<a href="mailto:rob.hewy@gmail.com" target="_blank">rob.hewy@gmail.com</a>>) escribió:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt">Just to clarify, instead of getting three files you are getting one with all the info: types, projection, data?<u></u><u></u></p>
</div>
<p class="MsoNormal"><a href="https://giswiki.hsr.ch/GeoCSV" target="_blank">https://giswiki.hsr.ch/GeoCSV</a><u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, May 3, 2023 at 8:57 AM Moises Calzado via gdal-dev <<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">We're also specifying the GEOM_POSSIBLE_NAMES, so it would be great if with that option we could use the GEOMETRY_NAME without using the CREATE_CSVT=YES option.
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Regarding emitting the .prj and .csvt in /vsistdout mode, that's why I'm saying that there is an issue while generating the resultant CSV.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">The way we see it is that when using the /vsistdout mode, the result is a CSV file with the .prj information in the first line, and the .csvt in the second line. We're dealing with the result deleting the first two lines and using the rest
of the content as a CSV, which should be equal to the result obtained when using ogr2ogr without the CREATE_CSVT=YES option.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Probably we're losing something, but as we see it, the generated CSV should be a valid one. Does that make sense?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks so much for your help!<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">El mié, 3 may 2023 a las 15:10, Robert Hewlett (<<a href="mailto:rob.hewy@gmail.com" target="_blank">rob.hewy@gmail.com</a>>) escribió:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">The .CSVT and .PRJ help to make a proper geocsv dataset. Helps with QGIS And geopandas. The column name that I use in the CSV is usually geom and WKT shows up in the CSVT file which seems to be a one line file that hints at the data types
in the CSV file. <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I hope that makes sense.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">CSVT<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Integer, Integer,WKT<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">CSV<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">line_id,point_id,geom<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">1,1,"POINT(1000 1000)"<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">PRJ<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">EPSG:26910<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, May 3, 2023, 05:23 Moises Calzado via gdal-dev <<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p class="MsoNormal">Hi Even, <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks so much for taking a look into that one!<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I have one doubt regarding the CSVT content, as we're not really using it, but it's required when using the GEOMETRY_NAME layer creation option, as can be checked in the CSV driver documentation:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<p style="margin-right:0cm;margin-bottom:0cm;margin-left:54pt;line-height:18pt;box-sizing:border-box">
<u></u><span style="font-size:10pt;font-family:Symbol"><span>·<span style="font:7pt "Times New Roman"">
</span></span></span><u></u><b>GEOMETRY_NAME</b>=name (Starting with GDAL 2.1): Name of geometry column. Only used if GEOMETRY=AS_WKT and CREATE_CSVT=YES. Defaults to WKT<u></u><u></u></p>
</blockquote>
<div>
<p class="MsoNormal">We really need this flag as we are processing files that contain geometries with different column names, and we always want the same geometry name in the generated output. Are we losing something when using that flag to avoid this problem?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">In my humble opinion, generating an invalid CSV when using the -lco CREATE_CSVT=YES looks like a bug for me, as I can't see the reason why strings containing line breaks can't be quoted.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Could you please shed some light on this?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Looking forward to your reply,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Regards.<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">El mié, 3 may 2023 a las 14:00, Even Rouault (<<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>>) escribió:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p>you didn't post to the list<u></u><u></u></p>
<div>
<p class="MsoNormal">Le 03/05/2023 à 13:49, Moises Calzado a écrit :<u></u><u></u></p>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<p class="MsoNormal">Hi Even, <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks so much for taking a look into that one!<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I have one doubt regarding the CSVT content, as we're not really using it, but it's required when using the GEOMETRY_NAME layer creation option, as can be checked in the CSV driver documentation:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<p style="margin-right:0cm;margin-bottom:0cm;margin-left:54pt;line-height:18pt;box-sizing:border-box">
<u></u><span style="font-size:10pt;font-family:Symbol"><span>·<span style="font:7pt "Times New Roman"">
</span></span></span><u></u><b>GEOMETRY_NAME</b>=name (Starting with GDAL 2.1): Name of geometry column. Only used if GEOMETRY=AS_WKT and CREATE_CSVT=YES. Defaults to WKT<u></u><u></u></p>
</blockquote>
<div>
<p class="MsoNormal">We really need this flag as we are processing files that contain geometries with different column names, and we always want the same geometry name in the generated output. Are we losing something when using that flag to avoid this problem?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">In my humble opinion, generating an invalid CSV when using the -lco CREATE_CSVT=YES looks like a bug for me, as I can't see the reason why strings containing line breaks can't be quoted.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Could you please shed some light on this?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Looking forward to your reply,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Regards.<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">El sáb, 29 abr 2023 a las 15:44, Even Rouault (<<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>>) escribió:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<p>Moises,<u></u><u></u></p>
<p>as far as I can see with your example, the CSV driver behaves "properly" in reading and writing of field values with line breaks.<u></u><u></u></p>
<p>It follows the "Fields with embedded line breaks must be quoted" rule of <a href="https://en.wikipedia.org/wiki/Comma-separated_values" target="_blank">
https://en.wikipedia.org/wiki/Comma-separated_values</a><u></u><u></u></p>
<p>$ ogr2ogr out.csv /vsizip/dataframe.zip<u></u><u></u></p>
<p>$ cat out.csv<br>
id,descriptio<br>
"1",This is my third row<br>
"2","this is<br>
my string<br>
"<br>
"3",This is my third row<u></u><u></u></p>
<p style="margin-bottom:12pt">$ ogrinfo out.csv -al<br>
INFO: Open of `out.csv'<br>
using driver `CSV' successful.<br>
<br>
Layer name: out<br>
Geometry: None<br>
Feature Count: 3<br>
Layer SRS WKT:<br>
(unknown)<br>
id: String (0.0)<br>
descriptio: String (0.0)<br>
OGRFeature(out):1<br>
id (String) = 1<br>
descriptio (String) = This is my third row<br>
<br>
OGRFeature(out):2<br>
id (String) = 2<br>
descriptio (String) = this is<br>
my string<br>
<br>
<br>
OGRFeature(out):3<br>
id (String) = 3<br>
descriptio (String) = This is my third row<u></u><u></u></p>
<p>But in your example using /vsistdout/ and -lco CREATE_CSVT=YES is going to result in an invalid CSV file which will mix both the .csvt and .csv content<u></u><u></u></p>
<p>Even<u></u><u></u></p>
<div>
<p class="MsoNormal">Le 24/04/2023 à 13:34, Moises Calzado via gdal-dev a écrit :<u></u><u></u></p>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<p class="MsoNormal">Hello! <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We're trying to convert a Shapefile into a CSV using ogr2ogr and we're having some issues while dealing with some columns that contain line breaks inside their values. If we have a line with the following string, ogr2ogr detects that the
line break is a new line and it returns two lines.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">"this is my \n value"<u></u><u></u></p>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">That's the command that we're executing:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">ogr2ogr -f CSV -skipfailures -makevalid /vsistdout/ /vsizip/shapefile.zip -simplify 0.00001 -dim XY -t_srs EPSG:4326 -lco GEOMETRY=AS_WKT -lco GEOMETRY_NAME=geom -lco CREATE_CSVT=YES > result.csv<u></u><u></u></p>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Is this an expected behaviour, or is there any way to avoid this?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Sharing an example Shapefile so that you can try to reproduce that behaviour:
<a href="https://drive.google.com/file/d/1gFqfTP02KTFoavJyyO-Ix05YwZB2tS24/view?usp=sharing" target="_blank">
https://drive.google.com/file/d/1gFqfTP02KTFoavJyyO-Ix05YwZB2tS24/view?usp=sharing</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks so much in advance,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Regards.<br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<p class="MsoNormal"><strong><span style="font-family:Calibri,sans-serif">Moises Calzado</span></strong>
<u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:3pt;margin-left:0cm">
Support Engineer <u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:15pt;margin-left:0cm">
+34671264286 | <a href="mailto:mcalzado@carto.com" target="_blank">mcalzado@carto.com</a> |
<a href="https://www.carto.com/" target="_blank">CARTO</a> <u></u><u></u></p>
<p class="MsoNormal"><a href="https://spatial-data-science-conference.com/2023/london/" target="_blank"><span style="text-decoration:none"><img border="0" width="517" height="50" style="width: 5.3819in; height: 0.5208in;" id="m_-1083489909395425235m_973077475011377906_x0000_i1031" src="https://storage.googleapis.com/carto-it-files/signature/SDSC-2023-LND_Signature-mail.jpg"></span></a><u></u><u></u></p>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<pre>_______________________________________________<u></u><u></u></pre>
<pre>gdal-dev mailing list<u></u><u></u></pre>
<pre><a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><u></u><u></u></pre>
<pre><a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><u></u><u></u></pre>
</blockquote>
<pre>-- <u></u><u></u></pre>
<pre><a href="http://www.spatialys.com/" target="_blank">http://www.spatialys.com</a><u></u><u></u></pre>
<pre>My software is free, but my time generally not.<u></u><u></u></pre>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<p class="MsoNormal"><strong><span style="font-family:Calibri,sans-serif">Moises Calzado</span></strong>
<u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:3pt;margin-left:0cm">
Support Engineer <u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:15pt;margin-left:0cm">
+34671264286 | <a href="mailto:mcalzado@carto.com" target="_blank">mcalzado@carto.com</a> |
<a href="https://www.carto.com/" target="_blank">CARTO</a> <u></u><u></u></p>
<p class="MsoNormal"><a href="https://spatial-data-science-conference.com/2023/london/" target="_blank"><span style="text-decoration:none"><img border="0" width="517" height="50" style="width: 5.3819in; height: 0.5208in;" id="m_-1083489909395425235m_973077475011377906_x0000_i1030" src="https://storage.googleapis.com/carto-it-files/signature/SDSC-2023-LND_Signature-mail.jpg"></span></a><u></u><u></u></p>
</div>
</blockquote>
<pre>-- <u></u><u></u></pre>
<pre><a href="http://www.spatialys.com/" target="_blank">http://www.spatialys.com</a><u></u><u></u></pre>
<pre>My software is free, but my time generally not.<u></u><u></u></pre>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<p class="MsoNormal"><strong><span style="font-family:Calibri,sans-serif">Moises Calzado</span></strong>
<u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:3pt;margin-left:0cm">
Support Engineer <u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:15pt;margin-left:0cm">
+34671264286 | <a href="mailto:mcalzado@carto.com" target="_blank">mcalzado@carto.com</a> |
<a href="https://www.carto.com/" target="_blank">CARTO</a> <u></u><u></u></p>
<p class="MsoNormal"><a href="https://spatial-data-science-conference.com/2023/london/" target="_blank"><span style="text-decoration:none"><img border="0" width="517" height="50" style="width: 5.3819in; height: 0.5208in;" id="m_-1083489909395425235m_973077475011377906_x0000_i1029" src="https://storage.googleapis.com/carto-it-files/signature/SDSC-2023-LND_Signature-mail.jpg"></span></a><u></u><u></u></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal">_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<p class="MsoNormal"><strong><span style="font-family:Calibri,sans-serif">Moises Calzado</span></strong>
<u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:3pt;margin-left:0cm">
Support Engineer <u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:15pt;margin-left:0cm">
+34671264286 | <a href="mailto:mcalzado@carto.com" target="_blank">mcalzado@carto.com</a> |
<a href="https://www.carto.com/" target="_blank">CARTO</a> <u></u><u></u></p>
<p class="MsoNormal"><a href="https://spatial-data-science-conference.com/2023/london/" target="_blank"><span style="text-decoration:none"><img border="0" width="517" height="50" style="width: 5.3819in; height: 0.5208in;" id="m_-1083489909395425235m_973077475011377906_x0000_i1028" src="https://storage.googleapis.com/carto-it-files/signature/SDSC-2023-LND_Signature-mail.jpg"></span></a><u></u><u></u></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal">_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<p class="MsoNormal"><strong><span style="font-family:Calibri,sans-serif">Moises Calzado</span></strong>
<u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:3pt;margin-left:0cm">
Support Engineer <u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:15pt;margin-left:0cm">
+34671264286 | <a href="mailto:mcalzado@carto.com" target="_blank">mcalzado@carto.com</a> |
<a href="https://www.carto.com/" target="_blank">CARTO</a> <u></u><u></u></p>
<p class="MsoNormal"><a href="https://spatial-data-science-conference.com/2023/london/" target="_blank"><span style="text-decoration:none"><img border="0" width="517" height="50" style="width: 5.3819in; height: 0.5208in;" id="m_-1083489909395425235m_973077475011377906_x0000_i1027" src="https://storage.googleapis.com/carto-it-files/signature/SDSC-2023-LND_Signature-mail.jpg"></span></a><u></u><u></u></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><u></u><u></u></p>
</blockquote>
</div>
</blockquote>
</div>
<p class="MsoNormal">_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<p class="MsoNormal"><strong><span style="font-family:Calibri,sans-serif">Moises Calzado</span></strong>
<u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:3pt;margin-left:0cm">
Support Engineer <u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:15pt;margin-left:0cm">
+34671264286 | <a href="mailto:mcalzado@carto.com" target="_blank">mcalzado@carto.com</a> |
<a href="https://www.carto.com/" target="_blank">CARTO</a> <u></u><u></u></p>
<p class="MsoNormal"><a href="https://spatial-data-science-conference.com/2023/london/" target="_blank"><span style="text-decoration:none"><img border="0" width="517" height="50" style="width: 5.3819in; height: 0.5208in;" id="m_-1083489909395425235m_973077475011377906_x0000_i1026" src="https://storage.googleapis.com/carto-it-files/signature/SDSC-2023-LND_Signature-mail.jpg"></span></a><u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<pre>_______________________________________________<u></u><u></u></pre>
<pre>gdal-dev mailing list<u></u><u></u></pre>
<pre><a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><u></u><u></u></pre>
<pre><a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><u></u><u></u></pre>
</blockquote>
<pre>-- <u></u><u></u></pre>
<pre><a href="http://www.spatialys.com/" target="_blank">http://www.spatialys.com</a><u></u><u></u></pre>
<pre>My software is free, but my time generally not.<u></u><u></u></pre>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><span>-- </span><u></u><u></u></p>
<div>
<p class="MsoNormal"><strong><span style="font-family:Calibri,sans-serif">Moises Calzado</span></strong>
<u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:3pt;margin-left:0cm">
Support Engineer <u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:15pt;margin-left:0cm">
+34671264286 | <a href="mailto:mcalzado@carto.com" target="_blank">mcalzado@carto.com</a> |
<a href="https://www.carto.com/" target="_blank">CARTO</a> <u></u><u></u></p>
<p class="MsoNormal"><a href="https://spatial-data-science-conference.com/2023/london/" target="_blank"><span style="text-decoration:none"><img border="0" width="517" height="50" style="width: 5.3819in; height: 0.5208in;" id="m_-1083489909395425235m_973077475011377906_x0000_i1025" src="https://storage.googleapis.com/carto-it-files/signature/SDSC-2023-LND_Signature-mail.jpg"></span></a><u></u><u></u></p>
</div>
</div>
</div>
_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br>
</div></blockquote></div>