[QGIS-Developer] [Qgis-user] Python Implementation of QgsNetworkContentFetcher

[C Hamilton]

For some reason I cannot seem to get QgsBlockingNetworkRequest to work.
Also I think there is an error in the documentation where it says, "After
completion of a request, the reply content should be retrieved by calling
getReplyContent()." It appears to me to be reply().

This is basically my code. Am I doing something wrong?
request = QgsBlockingNetworkRequest()
request.setAuthCfg(authcfg)
error_code = request.get(QNetworkRequest(QUrl(url)), forceRefresh=True)

The error that is returned is a ServerExceptionError. Is
QgsBlockingNetworkRequest really handling redirects?

I've had luck with QgsFileDownloader, but using a method you say I
shouldn't. This works.

tempfile = QgsProcessingUtils.generateTempFilename('data.json')
loop = QEventLoop()
dl = QgsFileDownloader(QUrl(url), tempfile, authcfg, delayStart=True)
dl.downloadExited.connect(loop.quit)
dl.startDownload()
loop.exec_()

In this case the data was correctly written into the tempfile.



On Wed, Nov 27, 2019 at 12:35 AM Nyall Dawson <nyall.dawson at gmail.com>
wrote:

> On Wed, 27 Nov 2019 at 15:26, Denis Rouzaud <denis.rouzaud at gmail.com>
> wrote:
> >
> > Thanks Nyall for the precisions.
> > I wasn't aware of QgsBlockingNetworkRequest.
> >
> > So basically, if I don't require authcf, then starting from 3.6 there is
> no reason to use my python nam, I should use both QgsNetworkContentFetcher
> and QgsBlockingNetworkRequest. And from 3.10 if require authcfg in
> QgsNetworkContentFetcher.
> > Is this correct?
>
> QgsBlockingNetworkRequest has always supported auth, so you should use
> that in 3.6 and above in all situations WHEN you need a blocking
> request.
>
> QgsNetworkContentFetcher is async, so definitely prefer to use that
> whenever you don't need a blocking request.
>
> I'd say the matrix is something like
>
> QGIS 3.4:
>
>  Blocking:
>    With auth: python nam (but be aware that it CAN and DOES crash
> crashes if used on the main thread)
>    Without auth: python nam (but be aware that it CAN and DOES crash
> crashes if used on the main thread)
>
>  Non-blocking:
>    With auth:      nothing available off-the-shelf -- roll your own
> using raw the qt and QGIS auth api?
>    Without auth: QgsNetworkContentFetcher
>
> QGIS 3.6 - 3.8:
>
>  Blocking:
>    With auth: QgsBlockingNetworkRequest
>    Without auth: QgsBlockingNetworkRequest
>
>  Non-blocking:
>    With auth:      nothing available off-the-shelf -- roll your own
> using raw the qt and QGIS auth api?
>    Without auth: QgsNetworkContentFetcher
>
> QGIS 3.10:
>
>  Blocking:
>    With auth: QgsBlockingNetworkRequest
>    Without auth: QgsBlockingNetworkRequest
>
>  Non-blocking:
>    With auth:      QgsNetworkContentFetcher
>    Without auth: QgsNetworkContentFetcher
>
>
> Nyall
>
>
>
> >
> > Cheers,
> > Denis
> >
> > Le mer. 27 nov. 2019 à 00:27, Nyall Dawson <nyall.dawson at gmail.com> a
> écrit :
> >>
> >> On Wed, 27 Nov 2019 at 00:44, Denis Rouzaud <denis.rouzaud at gmail.com>
> wrote:
> >> >
> >> > Hi,
> >> >
> >> > Jumping late but we use the work from NAM Alessandro did in a couple
> of locator filters.
> >> >
> https://github.com/opengisch/qgis-swiss-locator/blob/master/swiss_locator/core/network_access_manager.py
> >> >
> >> > I believe I have done very very few adaptations/fixes to the original
> work and it works like a charm.
> >>
> >> Not quite. It works in some circumstances, but it uses an event loop.
> >> So if you run it on the main thread, you're potentially doing an
> >> effective call to QApplication.processEvents, which can lead to all
> >> sorts of crashes. That said, it's generally a safe approach to use for
> >> QGIS <= 3.4... just don't use it in any code which relates to
> >> rendering!
> >>
> >> >
> >> > I had in mind to bring this as is within QGIS core for some time. The
> main point being for us the blocking request rather than authcfg.
> >>
> >> That's EXACTLY what QgsBlockingNetworkRequest is -- please don't
> >> re-re-implement this!
> >>
> >> Nyall
> >>
> >>
> >> >
> >> > Denis
> >> >
> >> > Le mar. 26 nov. 2019 à 15:36, C Hamilton <adenaculture at gmail.com> a
> écrit :
> >> >>
> >> >> Thanks everyone. This gives me some good ideas. These are the
> possible solutions:
> >> >>
> >> >> 1. Write a python implementation of QgsNetworkContentFetcher or
> QgsBlockingNetworkRequest. With this I could look at the QGIS version and
> if it is 3.10 or greater import the core method; otherwise, import the
> python implementation.
> >> >> 2. Use the old Boundless implementation if it still works in QGIS 3.
> >> >> 3. Use QgsFileDownloader and save it as a temporary file before
> processing.
> >> >>
> >> >> Calvin
> >> >>
> >> >> On Tue, Nov 26, 2019 at 2:29 AM Alessandro Pasotti <
> apasotti at gmail.com> wrote:
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Tue, Nov 26, 2019 at 8:22 AM Richard Duivenvoorde <
> rdmailings at duif.net> wrote:
> >> >>>>
> >> >>>> On 26/11/2019 00.06, Nyall Dawson wrote:
> >> >>>>
> >> >>>> > There's not. But there's the ex-boundless "networkaccessmanager"
> >> >>>> > implementation that supports authcfg, which you can find at
> >> >>>> >
> https://github.com/north-road/qgis-redistricting-plugin/blob/master/redistrict/linz/networkaccessmanager.py
> >> >>>> > (and other projects -- as far as I'm aware there's no real
> upstream
> >> >>>> > for this... Alessandro?).
> >> >>>>
> >> >>>> I used this Boundless thingie for several plugins now (original
> source
> >> >>>> is now pointing/redirecting to planetlabs [0]).
> >> >>>
> >> >>>
> >> >>> Yes, that's the "upstream" but I don't know if that one is
> maintained anymore.
> >> >>>
> >> >>> Btw, there is also a specialized class for sync/async downloads
> that supports authcfg:
> https://qgis.org/api/classQgsFileDownloader.html#details
> >> >>>
> >> >>>
> >> >>>>
> >> >>>>
> >> >>>> I think we already discussed several times that it would be good to
> >> >>>> incorporate this into a commons lib for QGIS. But if I'm correct
> the
> >> >>>> actual interface was not exactly what some wanted, and if I recall
> >> >>>> correct Nyall tried to pull this into c++ but hit some
> multithreading
> >> >>>> issues.
> >> >>>>
> >> >>>> But as Nyall says: I think this is a great lib to use in a plugin
> (also
> >> >>>> because it used the QgisNetwork-stuff, so you can receive the
> signals
> >> >>>> from it or log the precise urls, like we do in the
> QGISnetworklogger
> >> >>>> plugin [1]
> >> >>>>
> >> >>>> Regards,
> >> >>>>
> >> >>>> Richard Duivenvoorde
> >> >>>>
> >> >>>> [0]
> >> >>>>
> https://github.com/planetfederal/lib-qgis-commons/blob/master/qgiscommons2/network/networkaccessmanager.py
> >> >>>> [1] https://github.com/rduivenvoorde/qgisnetworklogger
> >> >>>> _______________________________________________
> >> >>>> Qgis-user mailing list
> >> >>>> Qgis-user at lists.osgeo.org
> >> >>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> >> >>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
> >> >>>
> >> >>>
> >> >>>
> >> >>> --
> >> >>> Alessandro Pasotti
> >> >>> w3:   www.itopen.it
> >> >>
> >> >> _______________________________________________
> >> >> QGIS-Developer mailing list
> >> >> QGIS-Developer at lists.osgeo.org
> >> >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >> >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >> >
> >> > _______________________________________________
> >> > QGIS-Developer mailing list
> >> > QGIS-Developer at lists.osgeo.org
> >> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20191127/60083043/attachment.html>