[Qgis-developer] Report 5 - QGIS Symbology Sharing Tools

[Akbar Gumbira]

Hi Matthias,

Thanks for the feedback. Unfortunately raw URL is only provided for a
single file, not a directory. As one repository could have many collections
(collection is a directory containing resources), with git protocol, there
are 2 ways downloading a single collection from a repository: git sparse
checkout and git archive. Sparse checkout is not supported in Dulwich nor
in Pygit2 (a Python wrapper using libgit2). And with the 2nd option, Github
doesn't allow git archive.

One other alternative (which I am exploring now) is to clone a repository
the first time user downloads a collection from that repository, and next
time user downloads other collections from that repository, we just need to
update and pull that repository. This is more efficient compared to
downloading the whole repository (using zip url) every time user wants to
download a collection.

We came to the decision that one repository could have many collections in
the first place as it doesn't make sense  (we thought) for users to create
one repository only for one collection.

Cheers

On Sun, Jun 26, 2016 at 7:50 PM, Matthias Kuhn <matthias at opengis.ch> wrote:

> Hi Akbar,
>
> thanks a lot for the update, I especially liked the video, this makes it
> a lot easier for the audience (at least me :) ) to get an impression of
> the status quo. Good job so far!
>
> Concerning dulwich, did you checkout the #dulwich IRC channel and the
> mailing list which are mentioned in the project's readme [1] ?
>
> What exactly are the efficiency problems you are referring to? Maybe
> it's also worth looking into downloading individual files over http
> instead of the whole .zip file (raw.githubusercontent.com), that also
> allows doing partial downloads if you don't need a complete repository
> and you know which files you want.
>
> Looking forward to hearing more of this project, keep up the good work!
>
> Matthias
>
> [1] https://github.com/jelmer/dulwich#help
>
> On 06/26/2016 07:10 PM, Akbar Gumbira wrote:
> > Hi,
> >
> > I made a video of the progress so
> > far: https://www.youtube.com/watch?v=OmJ2Vh3a63U
> >
> >
> > *What did you get done this week?*
> >
> >   * Saving the metadata of the collections to local as a cache (after
> >     some operations like adding/removing/deleting repository) so that
> >     when user doesn't have internet, (s)he will still be able to browse
> >     collection.
> >   * Filtering collections with custom QSortFilterProxyModel to allow
> >     filtering based on author, name, description, etc.
> >   * Fix unicode problem (The problem is when parsing the metadatafile.
> >     Using ConfigParser it was read as str. Changed it to
> >     SafeConfigParser with codecs module)
> >   * Implemented reload repositories. Also when fetching metadata, never
> >     use cache
> >   * Implemented Ale's suggestion to have a directory file in official
> >     repository and containing available repositories (and by default all
> >     those repositories will be registered in the plugin)
> >   * Still thinking to use git as it will be more efficient rather than
> >     using zip url and tried Dulwich (Python pure git implementation),
> >     but it's freezing the UI (see the video above)
> >
> > *What do you plan on doing next week?*
> >
> >   * My code is getting harder to read. Need to refactor some and make
> tests
> >   * Tried to load a repository that contains 1000 collections. Right
> >     now, I just saved the metadata's cache (dict) to the qsettings. The
> >     QGIS config file is getting too big. WIll move this to a separate
> >     pickled file.
> >   * Explored Dulwich more
> >
> >
> > Are you blocked on anything?
> > I have problem with Dulwich. It's freezing the UI. I want to show the
> > progress of cloning/pulling repository realtime but I haven't figured
> > out a way yet.
> >
> > --
> >
> > *-------------------*
> >
> > *Akbar Gumbira *
> > *www.akbargumbira.com <http://www.akbargumbira.com>*
> >
> >
> > _______________________________________________
> > Qgis-developer mailing list
> > Qgis-developer at lists.osgeo.org
> > List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> > Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> >
> _______________________________________________
> Qgis-developer mailing list
> Qgis-developer at lists.osgeo.org
> List info: http://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: http://lists.osgeo.org/mailman/listinfo/qgis-developer




-- 

*-------------------*
*Akbar Gumbira *
*www.akbargumbira.com <http://www.akbargumbira.com>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20160627/66fca53e/attachment-0001.html>