[SoC] [Qgis-developer] Report 2 - QGIS Symbology Sharing Tools

[Akbar Gumbira]

Hi Ale,

I just tried Dulwich. There are three ways that I see to checkout a single
file: git sparse checkout, git archive, or shallow clone (still cloning all
files though).

With git sparse checkout, Dulwich does not provide the porcelain-level API.
The author says that it's possible though (
https://github.com/jelmer/dulwich/issues/405#issuecomment-224579678). I am
not sure how much work to implement it dealing with the git object, tree,
and blob itself at the plumbing level.

With git archive, unfortunately Github doesn't allow us to do this (
https://twitter.com/GitHubHelp/status/322818593748303873). WIth shallow
clone left, I feel that I would over-engineer it forcing to use git just to
be able to do this. I think it's more simple if for now we use the raw
endpoint for each hosts that we know (Github, Bitbucket, etc), and we add
more supported hosts as users put the collection in other hosts as long as
we know the URL to the raw metadata file.

What do you think?

Cheers

On Wed, Jun 8, 2016 at 9:51 AM, Akbar Gumbira <akbargumbira at gmail.com>
wrote:

> Wasn't that a requisite? If we are going to use git as a storage we need a
>> git client (pure python seems to exist, could be used as a fallback in case
>> git is not installed in the system) https://github.com/jelmer/dulwich .
>>
>
>
>> BTW: if you do not want to deal with git in this first task, you could
>> used the zip endpoint, and assume that the metadata will be available at a
>> known location in the remote http repo and just wget it.
>
>
> Ok. I was just trying to find solution that could work out well without
> having git (as from the last discussion with Martin, it's better to avoid
> using git for now). I will try pygit2 and dulwich and I'll report you back
> what I thought using those two.
>
> On Wed, Jun 8, 2016 at 9:41 AM, Alessandro Pasotti <apasotti at gmail.com>
> wrote:
>
>> 2016-06-08 9:21 GMT+02:00 Akbar Gumbira <akbargumbira at gmail.com>:
>>
>>> Yes, I have tried it. We can do it with sparse checkout, but, it
>>> requires git on the client.
>>>
>>
>>
>> Wasn't that a requisite? If we are going to use git as a storage we need
>> a git client (pure python seems to exist, could be used as a fallback in
>> case git is not installed in the system)
>> https://github.com/jelmer/dulwich .
>>
>> BTW: if you do not want to deal with git in this first task, you could
>> used the zip endpoint, and assume that the metadata will be available at a
>> known location in the remote http repo and just wget it.
>>
>>
>>
>>>
>>> On Wed, Jun 8, 2016 at 9:18 AM, Alessandro Pasotti <apasotti at gmail.com>
>>> wrote:
>>>
>>>>
>>>> 2016-06-08 9:10 GMT+02:00 Akbar Gumbira <akbargumbira at gmail.com>:
>>>>
>>>>> Hi Akbar,
>>>>>> The most flexible installation tool that I know is probably python
>>>>>> pip.
>>>>>> pip can install software from a zip file, from git and from other
>>>>>> sources too.
>>>>>> I'd suggest you to have a look to pip implementation of the install
>>>>>> functionality, maybe there is some interesting for you.
>>>>>
>>>>> I just read pip code base. I think I can pick something from there for
>>>>> downloading the resources. But the problem I have right now is to get only
>>>>> the metadata file from the repository. Or are you suggesting that when
>>>>> users add a repository connection, it also downloads the repository
>>>>> directly?
>>>>>
>>>>
>>>>
>>>> If you are working on the git repos you can probably fetch just one
>>>> file without cloning the whole repo, I did not test it but here are
>>>> probably some pointers:
>>>>
>>>>
>>>> http://stackoverflow.com/questions/2466735/how-to-checkout-only-one-file-from-git-repository
>>>>
>>>> http://stackoverflow.com/questions/1125476/retrieve-a-single-file-from-a-repository
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> On Mon, Jun 6, 2016 at 8:39 AM, Alessandro Pasotti <apasotti at gmail.com
>>>>> > wrote:
>>>>>
>>>>>> 2016-06-05 10:13 GMT+02:00 Richard Duivenvoorde <rdmailings at duif.net>
>>>>>> :
>>>>>>
>>>>>>> On 05-06-16 09:02, Akbar Gumbira wrote:
>>>>>>>
>>>>>>> > *Are you blocked on anything?*
>>>>>>> > ... In Github or Bitbucket they provide a direct link to
>>>>>>> > the raw file. But I think I should look at more general approach
>>>>>>> without
>>>>>>> > manipulating the URL depending on the host. If you have some
>>>>>>> input, I
>>>>>>> > would be happy to assess it.
>>>>>>>
>>>>>>> Thanks Akbar,
>>>>>>>
>>>>>>> I did some googling:
>>>>>>>
>>>>>>> http://stackoverflow.com/questions/14405782/git-fetch-single-file-from-remote-repository-programatically
>>>>>>> If you really want to keep it git, it looks like a shallow
>>>>>>> clone/copy is
>>>>>>> the only way? That post also talks about some undocumented feature,
>>>>>>> but
>>>>>>> I would not depend on that?
>>>>>>>
>>>>>>> Personally I would be ok when both Github and Gitlab/Gog would work
>>>>>>> (as
>>>>>>> both a closed source and open source member of the git-web-world)...
>>>>>>>
>>>>>>> Or: a script running somewhere on our server, (shallow) cloning all
>>>>>>> registred repositories periodically, and making just the metadata.txt
>>>>>>> files available via http/webserver? (maybe giving us some time to
>>>>>>> check
>>>>>>> the repo's on structure and (malicious?) content?
>>>>>>>
>>>>>>> Or else: a django app for the metadata...
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Richard
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> Hi Akbar,
>>>>>>
>>>>>> The most flexible installation tool that I know is probably python
>>>>>> pip.
>>>>>>
>>>>>> pip can install software from a zip file, from git and from other
>>>>>> sources too.
>>>>>>
>>>>>> I'd suggest you to have a look to pip implementation of the install
>>>>>> functionality, maybe there is some interesting for you.
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Alessandro Pasotti
>>>>>> w3:   www.itopen.it
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> *-------------------*
>>>>> *Akbar Gumbira *
>>>>> *www.akbargumbira.com <http://www.akbargumbira.com>*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Alessandro Pasotti
>>>> w3:   www.itopen.it
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> *-------------------*
>>> *Akbar Gumbira *
>>> *www.akbargumbira.com <http://www.akbargumbira.com>*
>>>
>>
>>
>>
>> --
>> Alessandro Pasotti
>> w3:   www.itopen.it
>>
>
>
>
> --
>
> *-------------------*
> *Akbar Gumbira *
> *www.akbargumbira.com <http://www.akbargumbira.com>*
>



-- 

*-------------------*
*Akbar Gumbira *
*www.akbargumbira.com <http://www.akbargumbira.com>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/soc/attachments/20160609/cd71fd41/attachment.html>