[Qgis-developer] Delimited text issues and ideas

[Régis Haubourg]

Hi Chris, thanks for enlarging the discussion here, that's always better. I
really like the idea to interpret CSVT, but users will need a gui to create
it. That remains a good patch for 2.0 if possible. 
We could also add cvst / vrt writing in a standalone function (vector menu)
.. not urgent to me, these are workaround of a good import assistant. 

We kept on digging all import problems, and have no simple solution. MM QGIS
gives some functions, but needs to write at least two intermediate shapes
(01 for import + XY, 2 to convert text fields to real when needed) . 

Additionnal problem, almost no common tool is exporting csv correctly
according to csv specs, we then have to build good import tools, dealing
with all existing variations of txt and csv files (including BOM problems)..
Excel does good when dealing with decimal delimiters, but skips text
delimiters, so GDAL will interpret all numeric-like identifier as
identifier. My "09" is read as 9 when imported.. No possibility for me to
reinvent french administrative codes.
Libre Office is hard to use when dealing with commas/point as text
delimiters

What I have in mind to let my user work: 
*1- Short term proposal for actual users :*
QGIS 1.8: 
Me (funding or coding) : developp a quick plugin to add in vector menu:
Create point / geometry from text  into a memory layer. I will advice users
to import with a clean xls 2003 format when dealing with such data sources.
GDAL works Ok (better than csv, with no csvt needed)
CSV is OK with your import tool, but will put numeric fields in string. Use
of MM qgis or add field + update in shp is mandatory

*2 -QGIS 2.0 :* well, feature freeze is her, so having the create point
plugin compatible with new API. 

3- QGIS 2.1 : 
 fund (me certainly, if a consensus is reached) what have been previously
discussed here [0] with Nathan mockups of unified data access dialog, and
migrate your importer transparently there. 
      1: Unify data access : previsualization of data / character enconding
choice / text delimiter / decimal delimiter / data type chooser assistant /
advanced options to write csvt and/or vrt for power users
      2: port to C++ create point or geometry plugin
      3: add basic ETL functions to Core : rename, reorder, change type,
change names, save through OGR or mem Layer 
  

[0] 
http://osgeo-org.1560.x6.nabble.com/Clarifying-access-to-non-geographical-datas-td4986567.html#a4987225

Thoughts?



--
View this message in context: http://osgeo-org.1560.x6.nabble.com/Delimited-text-issues-and-ideas-tp5055027p5055180.html
Sent from the Quantum GIS - Developer mailing list archive at Nabble.com.