[Qgis-developer] morph delimited-text-provider to generic table-provider?

Fri Jul 15 12:01:12 PDT 2016

Hi,
I have had that in mind for quite a while now:
- non spatial datasources that can be spatialized with XY / WKT  or adress
geocoding.
- Those non spatial datasources are most of the time untyped datasources.
Some try to autotype like Excel, some not like CSV.

Delimited text provider handles the text file sources, is very mature now,
except on casting some string to numeric types.
We also have OGR access to all text / excel / ods files. It works quite
well, especially with xlsx, and other formats storing some information on
data type. But we have no UI to help user choosing field format like in all
other softwares (excel/ calc / access...).
That situation leads to some cumbersome troubles like virtual layers trying
to access delimited text as OGR datasource, when sometimes it comes from
delimited text provider plugin. Obviously, this lead to a failing query.
Users also get confused too by those two datasource, each one stll with
drawbacks

What I had in mind was funding the unification of vector/spreadsheet
datasource GUI, and merge delimited text provider as a frontend to ogr csvt
/ vrt.  Nathan's old mockups were very interesting because we had an
overview of data and a UI to help user choose types / delimiters /
encoding, XY - WKT field.

Dragging a csv to QGIS woould then open that window to help importation.

Adding some memory of import schemas by file name could also gain a lot a
clicks also so that importing csv with same column names could assume using
last user choices and not default typing.  Anyway, not asking user for a
schema like excel does is a total mistake we must not reproduce.

We could maybe use the geometry generator, but if we do that, I'm afraid
that virtual layers seeing datasource through OGR will not see spatial
columns and lead to cumbersome failures. Many algorithms may also be
entroubled there.

I still see a drawback in ogr vrt / csvt approach, the same as Mapinfo back
in the old times, is that when user has readonly rights on the datasource,
vrt/csvt must be stored elsewhere that along the source file.  If we want a
user to be able to share the qgs project, I see no other places than the
qgs itself, and a mecanism to have OGR and virtual layers work seamlessly
with that.

My vote is then for a pure core way, not another plugin, with close
discussion with OGR-GDAL so that any tool sees datasources the same way in
QGIS. QGIS will do his part with a super easy import-preview window, pretty
close from current delimited text provider.

Unfortunately, I won't be able to fund that  since I'm leaving my job to go
working with Oslandia team ! If we agree on a plan, I'll try to convince
some funders :))

Régis

2016-07-09 11:44 GMT+02:00 Matthias Kuhn <matthias at opengis.ch>:

> Hi,
>
> I think there are already 2 possibilities to show x,y /wkt or other
> geometries for any kind of layers
>
>  * Geometry generators
>
>  * Virtual layers (no expert, but I guess that's possible)
> So at least read-only this should be straightforward, read-write maybe
> with virtual layers (?).
>
> Matthias
>
>
> On 07/09/2016 11:34 AM, Raymond Nijssen wrote:
>
> Great idea! ;)
>
> Question is where to implement it in the GUI. It overlaps the
> functionality in the delimited text reader. Like pointing to the x and y
> columns or choosing a column containing wkt.
>
> Maybe it could be a 2 steps process? Step 1 is importing any table and
> step 2 is creating a spatial layer from it. But we would loose the quick
> "Add delimited text layer" functionality.
>
> Any ideas?
>
> @Paolo, which plugin can do this? XyTools?
>
> Regards,
> Raymond
>
>
>
>
> On 09-07-16 11:14, Paolo Cavallini wrote:
>
> Agreed. There is a plugin doing this, but of course having it as a
> standard function is desirable.
> Thanks for raising this.
>
> Il 9 luglio 2016 12:00:38 EEST, Richard Duivenvoorde<rdmailings at duif.net> <rdmailings at duif.net> ha scritto:
>
>     Hi Devs,
>
>     Raymond Nijssen had an (I think) intriguing idea:
>
>     QGIS can open different types of table-data: dbf, excel etc etc
>
>     But if the excel contains an x,y,wkt column, you have to save it to
>     csv/tsv first, to be able to use the delimited-text-provider to open it
>     as a layer.
>
>     His idea: can't we use the delimited-text-provider for all tables that
>     are open in QGIS immediately?
>
>     Had a look into the code
>
>     https://github.com/qgis/QGIS/blob/master/src/providers/delimitedtext/qgsdelimitedtextfile.cpp#L583
>
>     which 'just' looks like a QTextStream:
>
>     https://github.com/qgis/QGIS/blob/master/src/providers/delimitedtext/qgsdelimitedtextfile.h#L352
>
>     And
>     according to this:
>     https://forum.qt.io/topic/63310/how-to-save-a-treeview-as-csv-file-in-qt
>
>     It is easy to create a textstream from QModel (in which we have the data
>     I think???)...
>
>     So Question:
>     - (to devs): would this indeed be doable/easy?
>     - (to UI's): should the Delimited Text File dialog be updated then (or
>     only with a 'open excisting table button'?
>     - should we rename the provider to 'table' provider.
>
>     Comments/Opinions?
>
>     Regards,
>
>     Richard Duivenvoorde
>     ------------------------------------------------------------------------
>
>     Qgis-developer mailing list
>     Qgis-developer at lists.osgeo.org
>     List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>     Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>
>
> --
> Paolo Cavalliniwww.faunalia.eu <http://www.faunalia.eu> <http://www.faunalia.eu>
>
>
> _______________________________________________
> Qgis-developer mailing listQgis-developer at lists.osgeo.org
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>
> _______________________________________________
> Qgis-developer mailing listQgis-developer at lists.osgeo.org
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>
>
> --
> Matthias Kuhn
> OPENGIS.ch - https://www.opengis.ch
> Spatial • (Q)GIS • PostGIS • Open Source
>
>
> _______________________________________________
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>

-- 
Régis Haubourg

Attention, changement d'adresse mail!
Mon adresse principale devient désormais regis.haubourg at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20160715/17495f47/attachment.html>