[QGIS-Developer] [Qgis-user] Python Implementation of QgsNetworkContentFetcher

[Denis Rouzaud]

There is a redirect policy
https://doc.qt.io/qt-5/qnetworkrequest.html#RedirectPolicy-enum

Le mer. 27 nov. 2019 à 17:07, C Hamilton <adenaculture at gmail.com> a écrit :

> For some reason I cannot seem to get QgsBlockingNetworkRequest to work.
> Also I think there is an error in the documentation where it says, "After
> completion of a request, the reply content should be retrieved by calling
> getReplyContent()." It appears to me to be reply().
>
> This is basically my code. Am I doing something wrong?
> request = QgsBlockingNetworkRequest()
> request.setAuthCfg(authcfg)
> error_code = request.get(QNetworkRequest(QUrl(url)), forceRefresh=True)
>
> The error that is returned is a ServerExceptionError. Is
> QgsBlockingNetworkRequest really handling redirects?
>
> I've had luck with QgsFileDownloader, but using a method you say I
> shouldn't. This works.
>
> tempfile = QgsProcessingUtils.generateTempFilename('data.json')
> loop = QEventLoop()
> dl = QgsFileDownloader(QUrl(url), tempfile, authcfg, delayStart=True)
> dl.downloadExited.connect(loop.quit)
> dl.startDownload()
> loop.exec_()
>
> In this case the data was correctly written into the tempfile.
>
>
>
> On Wed, Nov 27, 2019 at 12:35 AM Nyall Dawson <nyall.dawson at gmail.com>
> wrote:
>
>> On Wed, 27 Nov 2019 at 15:26, Denis Rouzaud <denis.rouzaud at gmail.com>
>> wrote:
>> >
>> > Thanks Nyall for the precisions.
>> > I wasn't aware of QgsBlockingNetworkRequest.
>> >
>> > So basically, if I don't require authcf, then starting from 3.6 there
>> is no reason to use my python nam, I should use both
>> QgsNetworkContentFetcher and QgsBlockingNetworkRequest. And from 3.10 if
>> require authcfg in QgsNetworkContentFetcher.
>> > Is this correct?
>>
>> QgsBlockingNetworkRequest has always supported auth, so you should use
>> that in 3.6 and above in all situations WHEN you need a blocking
>> request.
>>
>> QgsNetworkContentFetcher is async, so definitely prefer to use that
>> whenever you don't need a blocking request.
>>
>> I'd say the matrix is something like
>>
>> QGIS 3.4:
>>
>>  Blocking:
>>    With auth: python nam (but be aware that it CAN and DOES crash
>> crashes if used on the main thread)
>>    Without auth: python nam (but be aware that it CAN and DOES crash
>> crashes if used on the main thread)
>>
>>  Non-blocking:
>>    With auth:      nothing available off-the-shelf -- roll your own
>> using raw the qt and QGIS auth api?
>>    Without auth: QgsNetworkContentFetcher
>>
>> QGIS 3.6 - 3.8:
>>
>>  Blocking:
>>    With auth: QgsBlockingNetworkRequest
>>    Without auth: QgsBlockingNetworkRequest
>>
>>  Non-blocking:
>>    With auth:      nothing available off-the-shelf -- roll your own
>> using raw the qt and QGIS auth api?
>>    Without auth: QgsNetworkContentFetcher
>>
>> QGIS 3.10:
>>
>>  Blocking:
>>    With auth: QgsBlockingNetworkRequest
>>    Without auth: QgsBlockingNetworkRequest
>>
>>  Non-blocking:
>>    With auth:      QgsNetworkContentFetcher
>>    Without auth: QgsNetworkContentFetcher
>>
>>
>> Nyall
>>
>>
>>
>> >
>> > Cheers,
>> > Denis
>> >
>> > Le mer. 27 nov. 2019 à 00:27, Nyall Dawson <nyall.dawson at gmail.com> a
>> écrit :
>> >>
>> >> On Wed, 27 Nov 2019 at 00:44, Denis Rouzaud <denis.rouzaud at gmail.com>
>> wrote:
>> >> >
>> >> > Hi,
>> >> >
>> >> > Jumping late but we use the work from NAM Alessandro did in a couple
>> of locator filters.
>> >> >
>> https://github.com/opengisch/qgis-swiss-locator/blob/master/swiss_locator/core/network_access_manager.py
>> >> >
>> >> > I believe I have done very very few adaptations/fixes to the
>> original work and it works like a charm.
>> >>
>> >> Not quite. It works in some circumstances, but it uses an event loop.
>> >> So if you run it on the main thread, you're potentially doing an
>> >> effective call to QApplication.processEvents, which can lead to all
>> >> sorts of crashes. That said, it's generally a safe approach to use for
>> >> QGIS <= 3.4... just don't use it in any code which relates to
>> >> rendering!
>> >>
>> >> >
>> >> > I had in mind to bring this as is within QGIS core for some time.
>> The main point being for us the blocking request rather than authcfg.
>> >>
>> >> That's EXACTLY what QgsBlockingNetworkRequest is -- please don't
>> >> re-re-implement this!
>> >>
>> >> Nyall
>> >>
>> >>
>> >> >
>> >> > Denis
>> >> >
>> >> > Le mar. 26 nov. 2019 à 15:36, C Hamilton <adenaculture at gmail.com> a
>> écrit :
>> >> >>
>> >> >> Thanks everyone. This gives me some good ideas. These are the
>> possible solutions:
>> >> >>
>> >> >> 1. Write a python implementation of QgsNetworkContentFetcher or
>> QgsBlockingNetworkRequest. With this I could look at the QGIS version and
>> if it is 3.10 or greater import the core method; otherwise, import the
>> python implementation.
>> >> >> 2. Use the old Boundless implementation if it still works in QGIS 3.
>> >> >> 3. Use QgsFileDownloader and save it as a temporary file before
>> processing.
>> >> >>
>> >> >> Calvin
>> >> >>
>> >> >> On Tue, Nov 26, 2019 at 2:29 AM Alessandro Pasotti <
>> apasotti at gmail.com> wrote:
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> On Tue, Nov 26, 2019 at 8:22 AM Richard Duivenvoorde <
>> rdmailings at duif.net> wrote:
>> >> >>>>
>> >> >>>> On 26/11/2019 00.06, Nyall Dawson wrote:
>> >> >>>>
>> >> >>>> > There's not. But there's the ex-boundless "networkaccessmanager"
>> >> >>>> > implementation that supports authcfg, which you can find at
>> >> >>>> >
>> https://github.com/north-road/qgis-redistricting-plugin/blob/master/redistrict/linz/networkaccessmanager.py
>> >> >>>> > (and other projects -- as far as I'm aware there's no real
>> upstream
>> >> >>>> > for this... Alessandro?).
>> >> >>>>
>> >> >>>> I used this Boundless thingie for several plugins now (original
>> source
>> >> >>>> is now pointing/redirecting to planetlabs [0]).
>> >> >>>
>> >> >>>
>> >> >>> Yes, that's the "upstream" but I don't know if that one is
>> maintained anymore.
>> >> >>>
>> >> >>> Btw, there is also a specialized class for sync/async downloads
>> that supports authcfg:
>> https://qgis.org/api/classQgsFileDownloader.html#details
>> >> >>>
>> >> >>>
>> >> >>>>
>> >> >>>>
>> >> >>>> I think we already discussed several times that it would be good
>> to
>> >> >>>> incorporate this into a commons lib for QGIS. But if I'm correct
>> the
>> >> >>>> actual interface was not exactly what some wanted, and if I recall
>> >> >>>> correct Nyall tried to pull this into c++ but hit some
>> multithreading
>> >> >>>> issues.
>> >> >>>>
>> >> >>>> But as Nyall says: I think this is a great lib to use in a plugin
>> (also
>> >> >>>> because it used the QgisNetwork-stuff, so you can receive the
>> signals
>> >> >>>> from it or log the precise urls, like we do in the
>> QGISnetworklogger
>> >> >>>> plugin [1]
>> >> >>>>
>> >> >>>> Regards,
>> >> >>>>
>> >> >>>> Richard Duivenvoorde
>> >> >>>>
>> >> >>>> [0]
>> >> >>>>
>> https://github.com/planetfederal/lib-qgis-commons/blob/master/qgiscommons2/network/networkaccessmanager.py
>> >> >>>> [1] https://github.com/rduivenvoorde/qgisnetworklogger
>> >> >>>> _______________________________________________
>> >> >>>> Qgis-user mailing list
>> >> >>>> Qgis-user at lists.osgeo.org
>> >> >>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> >> >>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> --
>> >> >>> Alessandro Pasotti
>> >> >>> w3:   www.itopen.it
>> >> >>
>> >> >> _______________________________________________
>> >> >> QGIS-Developer mailing list
>> >> >> QGIS-Developer at lists.osgeo.org
>> >> >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> >> >> Unsubscribe:
>> https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> >> >
>> >> > _______________________________________________
>> >> > QGIS-Developer mailing list
>> >> > QGIS-Developer at lists.osgeo.org
>> >> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>> >> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20191127/b43a411f/attachment-0001.html>