[Qgis-developer] import CSV to an existing layer

[Nathan Woodrow]

>> But obviously that makes it a much bigger task :-)

I'm not really sure it is any different.  This is almost all the code you
would need (minus the field mapping stuff and error checking)

for feature in sourcelayer.getFeatures():
    destlayer.dataProvider().addFeatures([feature])

By making it high level like this means you don't need to care what the
provider is only that you can write to it. That means I can append MS SQL
-> MS SQL, MS SQL -> PostGIS, Shape -> PostGIS, KML -> PostGIS, all without
caring.

This is a append/import/merge dialog that I have used in another
application while working in local gov and I found it pretty good
http://www.assetic.com/files/myData-software-interfaces.jpg.  Ignore the
Primary Key bit at the top as that doesn't really matter in our case.  The
important part is the ability to say Field A == Field C.

- Nathan


On Mon, Sep 30, 2013 at 5:44 PM, Denis Rouzaud <denis.rouzaud at gmail.com>wrote:

> Hi Crhis,
>
> Thanks a lot for your comments.
>
>
> On 09/27/2013 10:08 PM, Chris Crook wrote:
>
>> Hi Denis
>>
>> I'm not sure if you are suggesting adding this to the existing delimited
>> text layer function (bad idea!), or adding a new function using components
>> of the existing delimited text layer.  I say bad idea for a couple of
>> reasons.  FIrstly the UI is already cluttered enough, and adding some
>> additional controls to offer appending to an existing layer would I think
>> make it even more confusing.  Secondly I think an "import data" function is
>> very different from a "create layer" function, so they don't belong in the
>> same place in the menu - it would be too hard to find.
>>
> Indeed, my first intention was to use the same dialog. I agree it's not
> the right place.
>
>
>> On the other hand I can see that there is a very valid case for an import
>> data type of function (and as Nathan points out, it isn't specific to CSV
>> files).  In your case that is what you want, but I can easily imagine users
>> wanting to import data from other data sources (eg databases, web services,
>> etc).  But obviously that makes it a much bigger task :-)
>>
> Then, there is two ways to achieve import:
>
> - as Nathan suggested, copy features from a layer to antoher
> - unify the add layer dialogs and add an option in them that specify
> "append to a layer". And this could possibly open the "copy features
> dialog" from the first option.
>
> We could use an attribute table with editable header to do the field
> mapping.
>
> Then, would you consider this a good solution:
> - add a "copy features to another layer" option
> - add an option in the "add layer" dialogs for appending instead of
> creating a new layer
>
>
>  You were asking if anyone is working on this.  I don't know for sure (!).
>>  But things that may impact on it that I do know about:
>>
>> 1) Régis Haubourg was suggesting that we should merge the delimited text
>> layer with the OGR CSV provider, so that in effect the UI just becomes a
>> front end for setting up the OGR driver.  This would mean updating the OGR
>> driver to support the functions in the delimited text driver first (such a
>> regular expression - messy because there are many different implementations
>> and currently we are using the Qt version).
>>
> Thanks for the info. Although, I am not sure it would change something to
> the workflow.
>
>  2) The dialog (and provider) needs to be improved to support explicitly
>> selecting field types (slightly messy because the field types can come from
>> a CSVT file).  Automatically inferring field types is generally useful, but
>> has created issues with linking data (eg when text field is interpreted as
>> numeric)
>>
>> 3) There is a suggestion that it could have a capability to support
>> importing multiple files at once.
>>
>> Anyway just thought this could be useful background if you are working in
>> this area...
>>
>> Cheers
>> Chris
>>
>> ______________________________**__________
>> From: Denis Rouzaud [denis.rouzaud at gmail.com]
>> Sent: 28 September 2013 00:08
>> To: Nathan Woodrow
>> Cc: qgis-dev
>> Subject: Re: [Qgis-developer] import CSV to an existing layer
>>
>> Hi Nathan,
>>
>> I understand your point.
>>
>> But this would require more steps: add the layer to the project, copy the
>> features (with field mapping) and remove the layer.
>> I see a main usage of appending to an existing layer in the import of GPS
>> points for example. So mainly csv files.
>> It's true that there could be a great interest for shape files.
>>
>> The point is that I can use the existing header of the table in the
>> demilited text UI to do the field mapping. So, it should not be a great
>> effort to do so.
>>
>> It's clear that there is no a great cohesion with all these add layers
>> dialogs and I don't want to add more confusion to this.
>> But is there any plan to improve this?
>>
>> Denis
>>
>>
>>
>>
>>
>> On 09/27/2013 01:43 PM, Nathan Woodrow wrote:
>> I think a general higher level Append Features to Layer feature would be
>> much better.  This way it would work with any layer opened in QGIS.
>>
>> Being able to supply a layer to layer mapping would be even better in
>> case the fields don't match.
>>
>> - Nathan
>>
>>
>> On Fri, Sep 27, 2013 at 9:34 PM, Denis Rouzaud <denis.rouzaud at gmail.com<*
>> *mailto:denis.rouzaud at gmail.com**>> wrote:
>> Hi all,
>>
>> I have the intention to add the possibility of importing features from a
>> CSV into an existing layer.
>> My idea is to use the existing dialog "Create a layer from a delimited
>> text file" and to add the choice "create a layer" or "append to an existing
>> layer".
>>
>> First of all, is anyone already doing something similar?
>>
>> Second, if anyone already thought about this and has some ideas to share,
>> they're welcome!
>>
>> Cheers,
>>
>> Denis
>>
>>
>> ______________________________**_________________
>> Qgis-developer mailing list
>> Qgis-developer at lists.osgeo.org**<mailto:Qgis-developer at lists.**osgeo.org<Qgis-developer at lists.osgeo.org>
>> >
>> http://lists.osgeo.org/**mailman/listinfo/qgis-**developer<http://lists.osgeo.org/mailman/listinfo/qgis-developer>
>>
>>
>>
>> This message contains information, which is confidential and may be
>> subject to legal privilege. If you are not the intended recipient, you must
>> not peruse, use, disseminate, distribute or copy this message. If you have
>> received this message in error, please notify us immediately (Phone 0800
>> 665 463 or info at linz.govt.nz) and destroy the original message. LINZ
>> accepts no responsibility for changes to this email, or for any
>> attachments, after its transmission from LINZ. Thank You.
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20130930/1dda7fa3/attachment-0001.html>