[Qgis-developer] Report 5 - QGIS Symbology Sharing Tools
Vincent Picavet (ml)
vincent.ml at oslandia.com
Mon Jun 27 09:28:34 PDT 2016
Hello,
On 27/06/2016 12:41, Matthias Kuhn wrote:
> On 06/27/2016 11:32 AM, Akbar Gumbira wrote:
>> Thanks for the feedback. Unfortunately raw URL is only provided for a
>> single file, not a directory.
>
> Couldn't all the files be downloaded individually?
> This will result in a couple of additional requests, but I'm not sure
> this is something to worry about.
> The main question would then be, where the client could retrieve the
> file list (paths relative). Generating this list could be part of the
> upload script.
I may be late to the party, but I really do not get the point of using
git here for data retrieval.
And furthermore, I wouldn't want QGIS to have a hard dependency on git,
which is a developer tool users are not supposed to have installed by
default.
If you want to focus on the client part of the feature and not implement
a resource server, maybe a simpler approach would be better for a start.
Why not having a metadata file (JSON or XML) listing the resources
provided by the repository, and then the raw files at specified urls.
All of this served by a bare HTTP server, no more no less.
If some want to manage versioning of the resources and use github for
that, they can use the site generation tool (github pages) to do so.
Easy and lightweight. Or if we do not want to depend on a proprietary
platform, just generate the metadata+data files on any HTTP hosting
platform.
And we can later write a full resource server in our language of choice
to dynamically serve resources and metadata.
My 2 cents, sorry if this has been discussed before.
Vincent
>
>> As one repository could have many
>> collections (collection is a directory containing resources), with git
>> protocol, there are 2 ways downloading a single collection from a
>> repository: git sparse checkout and git archive. Sparse checkout is not
>> supported in Dulwich nor in Pygit2 (a Python wrapper using libgit2). And
>> with the 2nd option, Github doesn't allow git archive.
>>
>> One other alternative (which I am exploring now) is to clone a
>> repository the first time user downloads a collection from that
>> repository, and next time user downloads other collections from that
>> repository, we just need to update and pull that repository. This is
>> more efficient compared to downloading the whole repository (using zip
>> url) every time user wants to download a collection.
>
> Is the plan to mainly have users create their own repositories (and then
> manually enter the path to them in the style manager - or have a
> repository of repositories :) )?
> Or to have one main repository with many collections (like currently
> with the plugins)?
>
> If it's the first, I wouldn't care too much about the couple of extra
> bytes downloaded.
> If it's the second, that may result in some huge download and disk space
> consumption caused by one or a few collections.
>
> Cheers
> Matthias
>
>>
>> We came to the decision that one repository could have many collections
>> in the first place as it doesn't make sense (we thought) for users to
>> create one repository only for one collection.
>>
>> Cheers
>>
>> On Sun, Jun 26, 2016 at 7:50 PM, Matthias Kuhn <matthias at opengis.ch
>> <mailto:matthias at opengis.ch>> wrote:
>>
>> Hi Akbar,
>>
>> thanks a lot for the update, I especially liked the video, this makes it
>> a lot easier for the audience (at least me :) ) to get an impression of
>> the status quo. Good job so far!
>>
>> Concerning dulwich, did you checkout the #dulwich IRC channel and the
>> mailing list which are mentioned in the project's readme [1] ?
>>
>> What exactly are the efficiency problems you are referring to? Maybe
>> it's also worth looking into downloading individual files over http
>> instead of the whole .zip file (raw.githubusercontent.com
>> <http://raw.githubusercontent.com>), that also
>> allows doing partial downloads if you don't need a complete repository
>> and you know which files you want.
>>
>> Looking forward to hearing more of this project, keep up the good work!
>>
>> Matthias
>>
>> [1] https://github.com/jelmer/dulwich#help
>>
>> On 06/26/2016 07:10 PM, Akbar Gumbira wrote:
>> > Hi,
>> >
>> > I made a video of the progress so
>> > far: https://www.youtube.com/watch?v=OmJ2Vh3a63U
>> >
>> >
>> > *What did you get done this week?*
>> >
>> > * Saving the metadata of the collections to local as a cache (after
>> > some operations like adding/removing/deleting repository) so that
>> > when user doesn't have internet, (s)he will still be able to browse
>> > collection.
>> > * Filtering collections with custom QSortFilterProxyModel to allow
>> > filtering based on author, name, description, etc.
>> > * Fix unicode problem (The problem is when parsing the metadatafile.
>> > Using ConfigParser it was read as str. Changed it to
>> > SafeConfigParser with codecs module)
>> > * Implemented reload repositories. Also when fetching metadata,
>> never
>> > use cache
>> > * Implemented Ale's suggestion to have a directory file in official
>> > repository and containing available repositories (and by default all
>> > those repositories will be registered in the plugin)
>> > * Still thinking to use git as it will be more efficient rather than
>> > using zip url and tried Dulwich (Python pure git implementation),
>> > but it's freezing the UI (see the video above)
>> >
>> > *What do you plan on doing next week?*
>> >
>> > * My code is getting harder to read. Need to refactor some and
>> make tests
>> > * Tried to load a repository that contains 1000 collections. Right
>> > now, I just saved the metadata's cache (dict) to the qsettings. The
>> > QGIS config file is getting too big. WIll move this to a separate
>> > pickled file.
>> > * Explored Dulwich more
>> >
>> >
>> > Are you blocked on anything?
>> > I have problem with Dulwich. It's freezing the UI. I want to show the
>> > progress of cloning/pulling repository realtime but I haven't figured
>> > out a way yet.
>> >
>> > --
>> >
>> > *-------------------*
>> >
>> > *Akbar Gumbira *
>> > *www.akbargumbira.com <http://www.akbargumbira.com>
>> <http://www.akbargumbira.com>*
>> >
>> >
>> > _______________________________________________
>> > Qgis-developer mailing list
>> > Qgis-developer at lists.osgeo.org <mailto:Qgis-developer at lists.osgeo.org>
>> > List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>> > Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>> >
>> _______________________________________________
>> Qgis-developer mailing list
>> Qgis-developer at lists.osgeo.org <mailto:Qgis-developer at lists.osgeo.org>
>> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>>
>>
>>
>> --
>>
>> *-------------------*
>>
>> *Akbar Gumbira *
>> *www.akbargumbira.com <http://www.akbargumbira.com>*
> _______________________________________________
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>
More information about the Qgis-developer
mailing list