<p dir="ltr">Using the raw end point sounds fine to me for now. Easy to change later if needed.</p>
<br><div class="gmail_quote"><div dir="ltr">On Thu, 9 Jun 2016 8:14 am Akbar Gumbira <<a href="mailto:akbargumbira@gmail.com">akbargumbira@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Ale,<div><br></div><div>I just tried Dulwich. There are three ways that I see to checkout a single file: git sparse checkout, git archive, or shallow clone (still cloning all files though). </div><div><br></div><div>With git sparse checkout, Dulwich does not provide the porcelain-level API. The author says that it's possible though (<a href="https://github.com/jelmer/dulwich/issues/405#issuecomment-224579678" target="_blank">https://github.com/jelmer/dulwich/issues/405#issuecomment-224579678</a>). I am not sure how much work to implement it dealing with the git object, tree, and blob itself at the plumbing level. </div><div><br></div><div>With git archive, unfortunately Github doesn't allow us to do this (<a href="https://twitter.com/GitHubHelp/status/322818593748303873" target="_blank">https://twitter.com/GitHubHelp/status/322818593748303873</a>). WIth shallow clone left, I feel that I would over-engineer it forcing to use git just to be able to do this. I think it's more simple if for now we use the raw endpoint for each hosts that we know (Github, Bitbucket, etc), and we add more supported hosts as users put the collection in other hosts as long as we know the URL to the raw metadata file.</div><div><br>What do you think?<br><br>Cheers</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 8, 2016 at 9:51 AM, Akbar Gumbira <span dir="ltr"><<a href="mailto:akbargumbira@gmail.com" target="_blank">akbargumbira@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span><blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" class="gmail_quote">Wasn't that a requisite? If we are going to use git as a storage we need a git client (pure python seems to exist, could be used as a fallback in case git is not installed in the system) <a href="https://github.com/jelmer/dulwich" target="_blank">https://github.com/jelmer/dulwich</a> .<br></blockquote><div> </div><blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" class="gmail_quote">BTW: if you do not want to deal with git in this first task, you could used the zip endpoint, and assume that the metadata will be available at a known location in the remote http repo and just wget it.</blockquote><div><br></div></span><div>Ok. I was just trying to find solution that could work out well without having git (as from the last discussion with Martin, it's better to avoid using git for now). I will try pygit2 and dulwich and I'll report you back what I thought using those two.</div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 8, 2016 at 9:41 AM, Alessandro Pasotti <span dir="ltr"><<a href="mailto:apasotti@gmail.com" target="_blank">apasotti@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>2016-06-08 9:21 GMT+02:00 Akbar Gumbira <span dir="ltr"><<a href="mailto:akbargumbira@gmail.com" target="_blank">akbargumbira@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Yes, I have tried it. We can do it with sparse checkout, but, it requires git on the client.</div></blockquote><div><br><br></div></span><div>Wasn't that a requisite? If we are going to use git as a storage we need a git client (pure python seems to exist, could be used as a fallback in case git is not installed in the system) <a href="https://github.com/jelmer/dulwich" target="_blank">https://github.com/jelmer/dulwich</a> .<br><br></div><div>BTW: if you do not want to deal with git in this first task, you could used the zip endpoint, and assume that the metadata will be available at a known location in the remote http repo and just wget it.<br><br></div><div><div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 8, 2016 at 9:18 AM, Alessandro Pasotti <span dir="ltr"><<a href="mailto:apasotti@gmail.com" target="_blank">apasotti@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><span>2016-06-08 9:10 GMT+02:00 Akbar Gumbira <span dir="ltr"><<a href="mailto:akbargumbira@gmail.com" target="_blank">akbargumbira@gmail.com</a>></span>:<br></span><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">Hi Akbar,<br>The most flexible installation tool that I know is probably python pip.<br>pip can install software from a zip file, from git and from other sources too.<br>I'd suggest you to have a look to pip implementation of the install functionality, maybe there is some interesting for you.</blockquote></span><div>I just read pip code base. I think I can pick something from there for downloading the resources. But the problem I have right now is to get only the metadata file from the repository. Or are you suggesting that when users add a repository connection, it also downloads the repository directly? </div></div></blockquote><div><br><br></div></span><div>If you are working on the git repos you can probably fetch just one file without cloning the whole repo, I did not test it but here are probably some pointers: <br><br><a href="http://stackoverflow.com/questions/2466735/how-to-checkout-only-one-file-from-git-repository" target="_blank">http://stackoverflow.com/questions/2466735/how-to-checkout-only-one-file-from-git-repository</a><br><a href="http://stackoverflow.com/questions/1125476/retrieve-a-single-file-from-a-repository" target="_blank">http://stackoverflow.com/questions/1125476/retrieve-a-single-file-from-a-repository</a><br><br><br></div><div><div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_extra"><div><div><br><div class="gmail_quote">On Mon, Jun 6, 2016 at 8:39 AM, Alessandro Pasotti <span dir="ltr"><<a href="mailto:apasotti@gmail.com" target="_blank">apasotti@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div>2016-06-05 10:13 GMT+02:00 Richard Duivenvoorde <span dir="ltr"><<a href="mailto:rdmailings@duif.net" target="_blank">rdmailings@duif.net</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On <a href="tel:05-06-16%2009" value="+3905061609" target="_blank">05-06-16 09</a>:02, Akbar Gumbira wrote:<br>
<br>
> *Are you blocked on anything?*<br>
> ... In Github or Bitbucket they provide a direct link to<br>
<span>> the raw file. But I think I should look at more general approach without<br>
> manipulating the URL depending on the host. If you have some input, I<br>
> would be happy to assess it.<br>
<br>
</span>Thanks Akbar,<br>
<br>
I did some googling:<br>
<a href="http://stackoverflow.com/questions/14405782/git-fetch-single-file-from-remote-repository-programatically" rel="noreferrer" target="_blank">http://stackoverflow.com/questions/14405782/git-fetch-single-file-from-remote-repository-programatically</a><br>
If you really want to keep it git, it looks like a shallow clone/copy is<br>
the only way? That post also talks about some undocumented feature, but<br>
I would not depend on that?<br>
<br>
Personally I would be ok when both Github and Gitlab/Gog would work (as<br>
both a closed source and open source member of the git-web-world)...<br>
<br>
Or: a script running somewhere on our server, (shallow) cloning all<br>
registred repositories periodically, and making just the metadata.txt<br>
files available via http/webserver? (maybe giving us some time to check<br>
the repo's on structure and (malicious?) content?<br>
<br>
Or else: a django app for the metadata...<br>
<br>
Regards,<br>
<br>
Richard<br>
<br>
<br></blockquote><div><br><br></div></div></div><div>Hi Akbar,<br><br></div><div>The most flexible installation tool that I know is probably python pip.<br><br></div><div>pip can install software from a zip file, from git and from other sources too.<br><br></div><div>I'd suggest you to have a look to pip implementation of the install functionality, maybe there is some interesting for you.<span><font color="#888888"><br></font></span></div></div><span><font color="#888888"><br><br clear="all"><br>-- <br><div data-smartmail="gmail_signature">Alessandro Pasotti<br>w3: <a href="http://www.itopen.it" target="_blank">www.itopen.it</a></div>
</font></span></div></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><p><b><span style="font-size:11pt;font-family:"Calibri","sans-serif";color:rgb(31,73,125)">-------------------</span></b></p><b>Akbar Gumbira </b></div><div><b style="font-size:12.8px"><a href="http://www.akbargumbira.com" target="_blank">www.akbargumbira.com</a></b></div></div></div></div></div>
</span></div>
</blockquote></div></div></div><div><div><br><br clear="all"><br>-- <br><div data-smartmail="gmail_signature">Alessandro Pasotti<br>w3: <a href="http://www.itopen.it" target="_blank">www.itopen.it</a></div>
</div></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><p><b><span style="font-size:11pt;font-family:"Calibri","sans-serif";color:rgb(31,73,125)">-------------------</span></b></p><b>Akbar Gumbira </b></div><div><b style="font-size:12.8px"><a href="http://www.akbargumbira.com" target="_blank">www.akbargumbira.com</a></b></div></div></div></div></div>
</div>
</div></div></blockquote></div></div></div><div><div><br><br clear="all"><br>-- <br><div data-smartmail="gmail_signature">Alessandro Pasotti<br>w3: <a href="http://www.itopen.it" target="_blank">www.itopen.it</a></div>
</div></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><p><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">-------------------</span></b></p><b>Akbar Gumbira </b></div><div><b style="font-size:12.8px"><a href="http://www.akbargumbira.com" target="_blank">www.akbargumbira.com</a></b></div></div></div></div></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><p><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">-------------------</span></b></p><b>Akbar Gumbira </b></div><div><b style="font-size:12.8px"><a href="http://www.akbargumbira.com" target="_blank">www.akbargumbira.com</a></b></div></div></div></div></div>
</div>
_______________________________________________<br>
Qgis-developer mailing list<br>
<a href="mailto:Qgis-developer@lists.osgeo.org" target="_blank">Qgis-developer@lists.osgeo.org</a><br>
List info: <a href="http://lists.osgeo.org/mailman/listinfo/qgis-developer" rel="noreferrer" target="_blank">http://lists.osgeo.org/mailman/listinfo/qgis-developer</a><br>
Unsubscribe: <a href="http://lists.osgeo.org/mailman/listinfo/qgis-developer" rel="noreferrer" target="_blank">http://lists.osgeo.org/mailman/listinfo/qgis-developer</a></blockquote></div>