[Qgis-developer] Report 5 - QGIS Symbology Sharing Tools

[Matthias Kuhn]

Hi Akbar

On 06/27/2016 11:32 AM, Akbar Gumbira wrote:
> Hi Matthias, 
> 
> Thanks for the feedback. Unfortunately raw URL is only provided for a
> single file, not a directory.

Couldn't all the files be downloaded individually?
This will result in a couple of additional requests, but I'm not sure
this is something to worry about.
The main question would then be, where the client could retrieve the
file list (paths relative). Generating this list could be part of the
upload script.

> As one repository could have many
> collections (collection is a directory containing resources), with git
> protocol, there are 2 ways downloading a single collection from a
> repository: git sparse checkout and git archive. Sparse checkout is not
> supported in Dulwich nor in Pygit2 (a Python wrapper using libgit2). And
> with the 2nd option, Github doesn't allow git archive.
> 
> One other alternative (which I am exploring now) is to clone a
> repository the first time user downloads a collection from that
> repository, and next time user downloads other collections from that
> repository, we just need to update and pull that repository. This is
> more efficient compared to downloading the whole repository (using zip
> url) every time user wants to download a collection.

Is the plan to mainly have users create their own repositories (and then
manually enter the path to them in the style manager - or have a
repository of repositories :) )?
Or to have one main repository with many collections (like currently
with the plugins)?

If it's the first, I wouldn't care too much about the couple of extra
bytes downloaded.
If it's the second, that may result in some huge download and disk space
consumption caused by one or a few collections.

Cheers
Matthias

> 
> We came to the decision that one repository could have many collections
> in the first place as it doesn't make sense  (we thought) for users to
> create one repository only for one collection.
> 
> Cheers
> 
> On Sun, Jun 26, 2016 at 7:50 PM, Matthias Kuhn <matthias at opengis.ch
> <mailto:matthias at opengis.ch>> wrote:
> 
>     Hi Akbar,
> 
>     thanks a lot for the update, I especially liked the video, this makes it
>     a lot easier for the audience (at least me :) ) to get an impression of
>     the status quo. Good job so far!
> 
>     Concerning dulwich, did you checkout the #dulwich IRC channel and the
>     mailing list which are mentioned in the project's readme [1] ?
> 
>     What exactly are the efficiency problems you are referring to? Maybe
>     it's also worth looking into downloading individual files over http
>     instead of the whole .zip file (raw.githubusercontent.com
>     <http://raw.githubusercontent.com>), that also
>     allows doing partial downloads if you don't need a complete repository
>     and you know which files you want.
> 
>     Looking forward to hearing more of this project, keep up the good work!
> 
>     Matthias
> 
>     [1] https://github.com/jelmer/dulwich#help
> 
>     On 06/26/2016 07:10 PM, Akbar Gumbira wrote:
>     > Hi,
>     >
>     > I made a video of the progress so
>     > far: https://www.youtube.com/watch?v=OmJ2Vh3a63U
>     >
>     >
>     > *What did you get done this week?*
>     >
>     >   * Saving the metadata of the collections to local as a cache (after
>     >     some operations like adding/removing/deleting repository) so that
>     >     when user doesn't have internet, (s)he will still be able to browse
>     >     collection.
>     >   * Filtering collections with custom QSortFilterProxyModel to allow
>     >     filtering based on author, name, description, etc.
>     >   * Fix unicode problem (The problem is when parsing the metadatafile.
>     >     Using ConfigParser it was read as str. Changed it to
>     >     SafeConfigParser with codecs module)
>     >   * Implemented reload repositories. Also when fetching metadata,
>     never
>     >     use cache
>     >   * Implemented Ale's suggestion to have a directory file in official
>     >     repository and containing available repositories (and by default all
>     >     those repositories will be registered in the plugin)
>     >   * Still thinking to use git as it will be more efficient rather than
>     >     using zip url and tried Dulwich (Python pure git implementation),
>     >     but it's freezing the UI (see the video above)
>     >
>     > *What do you plan on doing next week?*
>     >
>     >   * My code is getting harder to read. Need to refactor some and
>     make tests
>     >   * Tried to load a repository that contains 1000 collections. Right
>     >     now, I just saved the metadata's cache (dict) to the qsettings. The
>     >     QGIS config file is getting too big. WIll move this to a separate
>     >     pickled file.
>     >   * Explored Dulwich more
>     >
>     >
>     > Are you blocked on anything?
>     > I have problem with Dulwich. It's freezing the UI. I want to show the
>     > progress of cloning/pulling repository realtime but I haven't figured
>     > out a way yet.
>     >
>     > --
>     >
>     > *-------------------*
>     >
>     > *Akbar Gumbira *
>     > *www.akbargumbira.com <http://www.akbargumbira.com>
>     <http://www.akbargumbira.com>*
>     >
>     >
>     > _______________________________________________
>     > Qgis-developer mailing list
>     > Qgis-developer at lists.osgeo.org <mailto:Qgis-developer at lists.osgeo.org>
>     > List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>     > Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>     >
>     _______________________________________________
>     Qgis-developer mailing list
>     Qgis-developer at lists.osgeo.org <mailto:Qgis-developer at lists.osgeo.org>
>     List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
>     Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> 
> 
> 
> 
> -- 
> 
> *-------------------*
> 
> *Akbar Gumbira *
> *www.akbargumbira.com <http://www.akbargumbira.com>*