[QGIS-Developer] Custom processing script instabilities (Nzl_T)

Nyall Dawson nyall.dawson at gmail.com
Fri Jun 25 23:47:18 PDT 2021 

On Fri, 25 Jun 2021 at 20:14, Nazli Turini [BBA]
<nazli.turini at broadband-academy.de> wrote:
>
> Hi Evereybody,
>
> I am facing the same issue.
>
> I am creating a custom processing algorithm in which it loads 1, the google satellite 2. get my study_region.shp file 3. loop through the features in study_region.shp 4. and for each feature create the mbtiles using processing.run('qgis:tilesxyzmbtiles'...)5. Save the result on the predefined folder.
>
> The script runs when the ZOOM_MAX and ZOOM_MIN are 12. When I change the ZOOM_MAX to 19, the processing window vanished and qgis crashed. Maybe it gets some hint to @ Alexis

Your script uses a lot of thread-unsafe calls. I've marked them below.
You should find alternatives for these, or make sure your algorithm
returns the FlagNoThreading flag from an override of the
QgsProcessingAlgorithm.flags() method.

>
>         #if Projektnummer.find(SEPARATOR) > 0:
>         output_path = os.path.join(out_folder, Projektnummer)
>         #Projektnummer = path.basename(Projektnummer)
>             # Create folder if not present
>         if not os.path.exists(output_path):
>             makedirs(output_path)
>         #Load the google xyz data
>         urlWithParams = 'type=xyz&url=https://mt1.google.com/vt/lyrs%3Ds%26x%3D%7Bx%7D%26y%3D%7By%7D%26z%3D%7Bz%7D&zmax=19&zmin=0'
>         rlayer = QgsRasterLayer(urlWithParams, 'Google', 'wms')
>         QgsProject.instance().addMapLayer(rlayer, False)

not thread safe -- you CANNOT interact with QgsProject.instance() from
a background thread.


>         layerTree = iface.layerTreeCanvasBridge().rootGroup()

NOT thread safe -- iface methods should only ever be accessed from the
main (gui) thread.

>         layerTree.insertChildNode(0, QgsLayerTreeLayer(rlayer))

not thread safe -- the QgsLayerTree classes must be accessed only from
the thread which the parent QgsProject belongs to.

>         if feedback.isCanceled():
>             return {}
>
>         input_featuresource = self.parameterAsSource(parameters, 'Ausbaugebiet', context)
>
>         for feature in input_featuresource.getFeatures():
>             attrs = feature.attributes()
>             name=attrs[1]
>             bbox = feature.geometry().boundingBox()
>             bbox_extent = '%f,%f,%f,%f' % (bbox.xMinimum(), bbox.xMaximum(), bbox.yMinimum(), bbox.yMaximum())
>             alg_params = {
>             'BACKGROUND_COLOR': QColor(0, 0, 0, 0),
>             'DPI': 300,
>             'EXTENT': bbox_extent,
>             'METATILESIZE': 6,
>             'QUALITY': 75,
>             'TILE_FORMAT': 0,
>             'ZOOM_MAX': 12,
>             'ZOOM_MIN': 12,
>             'OUTPUT_FILE': output_path+'/googlestallite_%s.mbtiles' %(name)
>             }
>             google=processing.run('qgis:tilesxyzmbtiles', alg_params,is_child_algorithm=True,context=context, feedback=feedback)['OUTPUT_FILE']
>                     # Check for cancelation
>             if feedback.isCanceled():
>                 return {}
>             #Remove the layer
>         QgsProject.instance().removeMapLayers([rlayer.id()])

Again, not thread safe.

Hope that helps!
Nyall

>         del rlayer
>         return result
>
> Thank you so much and best regards,
> Nazli
>
> -----Ursprüngliche Nachricht-----
> Von: QGIS-Developer <qgis-developer-bounces at lists.osgeo.org> Im Auftrag von qgis-developer-request at lists.osgeo.org
> Gesendet: Freitag, 25. Juni 2021 09:15
> An: qgis-developer at lists.osgeo.org
> Betreff: QGIS-Developer Digest, Vol 188, Issue 36
>
> Send QGIS-Developer mailing list submissions to
>         qgis-developer at lists.osgeo.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.osgeo.org/mailman/listinfo/qgis-developer
> or, via email, send a message with subject or body 'help' to
>         qgis-developer-request at lists.osgeo.org
>
> You can reach the person managing the list at
>         qgis-developer-owner at lists.osgeo.org
>
> When replying, please edit your Subject line so it is more specific than "Re: Contents of QGIS-Developer digest..."
>
>
> Today's Topics:
>
>    1. Custom processing script instabilities (Alexis R.L.)
>    2. Re: Custom processing script instabilities (Luigi Pirelli)
>    3. Re: Custom processing script instabilities (Alexis R.L.)
>    4. Re: Custom processing script instabilities (Luigi Pirelli)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 24 Jun 2021 15:50:54 -0400
> From: "Alexis R.L." <alroyliz0 at gmail.com>
> To: qgis-developer <qgis-developer at lists.osgeo.org>
> Subject: [QGIS-Developer] Custom processing script instabilities
> Message-ID:
>         <CAOM0XWDiy-WUXQrL4fEFXKxUyzBb-K9Mix3kBTRwK4nnRAhHww at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Greetings,
>
> I have one custom script that uses PIL to get the exif tag, and when I execute the script for the first time, the processing window vanishes and qgis stall and then crashes.
>
> To get it to run properly I had to set the return before opening the files with PIL to run a neutered version and then subsequent runs were flawless during that qgis instance.
>
> I ended up using a standalone exif reader, though slower , but there are no more crashes.
>
> Has anyone else had similar issues and is anyone aware of a way to prevent such crashes or handle them more gracefully, as no trace is left currently.
>
> Thanks and have a nice day,
>
> Alex
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20210624/d49ff6b4/attachment-0001.html>
>
> ------------------------------
>
> Message: 2
> Date: Thu, 24 Jun 2021 22:34:01 +0200
> From: Luigi Pirelli <luipir at gmail.com>
> To: "Alexis R.L." <alroyliz0 at gmail.com>
> Cc: qgis-developer <qgis-developer at lists.osgeo.org>
> Subject: Re: [QGIS-Developer] Custom processing script instabilities
> Message-ID:
>         <CAFO80_qOK0pBn1KajrPN0paAF=+SrsObeLs=QGqoY_vs=Oq3XQ at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> hard to say something without reading the code... btw generally you do not have to interact with main GUi from a processing alg that by default run in a separated thread => this can cause crash if you do not use standard processing libs (e.g. feedback) to send message to the user.
> No idea if this could be the source of the problem., IMHO sharing the code would help and also help to verify if it can be replicable.
>
> regards
>
> Luigi Pirelli
>
> **************************************************************************************************
> * LinkedIn: https://www.linkedin.com/in/luigipirelli
> * Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
> * GitHub: https://github.com/luipir
> * Book: Mastering QGIS3 - 3rd Edition
> <https://www.packtpub.com/eu/application-development/mastering-geospatial-development-qgis-3x-third-edition>
> * Hire a team: http://www.qcooperative.net
> **************************************************************************************************
>
>
> On Thu, 24 Jun 2021 at 21:51, Alexis R.L. <alroyliz0 at gmail.com> wrote:
>
> > Greetings,
> >
> > I have one custom script that uses PIL to get the exif tag, and when I
> > execute the script for the first time, the processing window vanishes
> > and qgis stall and then crashes.
> >
> > To get it to run properly I had to set the return before opening the
> > files with PIL to run a neutered version and then subsequent runs were
> > flawless during that qgis instance.
> >
> > I ended up using a standalone exif reader, though slower , but there
> > are no more crashes.
> >
> > Has anyone else had similar issues and is anyone aware of a way to
> > prevent such crashes or handle them more gracefully, as no trace is left currently.
> >
> > Thanks and have a nice day,
> >
> > Alex
> > _______________________________________________
> > QGIS-Developer mailing list
> > QGIS-Developer at lists.osgeo.org
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20210624/084a8256/attachment-0001.html>
>
> ------------------------------
>
> Message: 3
> Date: Thu, 24 Jun 2021 17:05:09 -0400
> From: "Alexis R.L." <alroyliz0 at gmail.com>
> To: Luigi Pirelli <luipir at gmail.com>
> Cc: qgis-developer <qgis-developer at lists.osgeo.org>
> Subject: Re: [QGIS-Developer] Custom processing script instabilities
> Message-ID:
>         <CAOM0XWB22aCNvEZ8O04yi-VATkjotqN_rHJ26rBpGC87Asv7BA at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Greetings Luigi,
>
> There were no interaction, I used glob to gather all the images in a folder, used Image.Open and then used _getexif(). But returning before this part was the only way to get a stable first run.
>
> Sadly I can share the code (work and IP).
>
> The rest of it looped through datapoints and correlated the images to the points using the exif timestamp and the geolocalisation.
>
> Alex
>
> Le jeu. 24 juin 2021 ? 16:34, Luigi Pirelli <luipir at gmail.com> a ?crit :
>
> > hard to say something without reading the code... btw generally you do
> > not have to interact with main GUi from a processing alg that by
> > default run in a separated thread => this can cause crash if you do
> > not use standard processing libs (e.g. feedback) to send message to the user.
> > No idea if this could be the source of the problem., IMHO sharing the
> > code would help and also help to verify if it can be replicable.
> >
> > regards
> >
> > Luigi Pirelli
> >
> >
> > **********************************************************************
> > ****************************
> > * LinkedIn: https://www.linkedin.com/in/luigipirelli
> > * Stackexchange:
> > http://gis.stackexchange.com/users/19667/luigi-pirelli
> > * GitHub: https://github.com/luipir
> > * Book: Mastering QGIS3 - 3rd Edition
> > <https://www.packtpub.com/eu/application-development/mastering-geospat
> > ial-development-qgis-3x-third-edition>
> > * Hire a team: http://www.qcooperative.net
> >
> > **********************************************************************
> > ****************************
> >
> >
> > On Thu, 24 Jun 2021 at 21:51, Alexis R.L. <alroyliz0 at gmail.com> wrote:
> >
> >> Greetings,
> >>
> >> I have one custom script that uses PIL to get the exif tag, and when
> >> I execute the script for the first time, the processing window
> >> vanishes and qgis stall and then crashes.
> >>
> >> To get it to run properly I had to set the return before opening the
> >> files with PIL to run a neutered version and then subsequent runs
> >> were flawless during that qgis instance.
> >>
> >> I ended up using a standalone exif reader, though slower , but there
> >> are no more crashes.
> >>
> >> Has anyone else had similar issues and is anyone aware of a way to
> >> prevent such crashes or handle them more gracefully, as no trace is
> >> left currently.
> >>
> >> Thanks and have a nice day,
> >>
> >> Alex
> >> _______________________________________________
> >> QGIS-Developer mailing list
> >> QGIS-Developer at lists.osgeo.org
> >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >>
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20210624/415ff84b/attachment-0001.html>
>
> ------------------------------
>
> Message: 4
> Date: Fri, 25 Jun 2021 09:14:43 +0200
> From: Luigi Pirelli <luipir at gmail.com>
> To: "Alexis R.L." <alroyliz0 at gmail.com>
> Cc: qgis-developer <qgis-developer at lists.osgeo.org>
> Subject: Re: [QGIS-Developer] Custom processing script instabilities
> Message-ID:
>         <CAFO80_oh3NPcZE9oOaoB+0+FfhGLrKU74t71+PsYGXCw=736dw at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> strange that a library call exit, so I would inspect if the lib faces a raise condition. I would check the code outside qgis checking if it terminates for some reason.
>
> Luigi Pirelli
>
> **************************************************************************************************
> * LinkedIn: https://www.linkedin.com/in/luigipirelli
> * Stackexchange: http://gis.stackexchange.com/users/19667/luigi-pirelli
> * GitHub: https://github.com/luipir
> * Book: Mastering QGIS3 - 3rd Edition
> <https://www.packtpub.com/eu/application-development/mastering-geospatial-development-qgis-3x-third-edition>
> * Hire a team: http://www.qcooperative.net
> **************************************************************************************************
>
>
> On Thu, 24 Jun 2021 at 23:05, Alexis R.L. <alroyliz0 at gmail.com> wrote:
>
> > Greetings Luigi,
> >
> > There were no interaction, I used glob to gather all the images in a
> > folder, used Image.Open and then used _getexif(). But returning before
> > this part was the only way to get a stable first run.
> >
> > Sadly I can share the code (work and IP).
> >
> > The rest of it looped through datapoints and correlated the images to
> > the points using the exif timestamp and the geolocalisation.
> >
> > Alex
> >
> > Le jeu. 24 juin 2021 ? 16:34, Luigi Pirelli <luipir at gmail.com> a ?crit :
> >
> >> hard to say something without reading the code... btw generally you
> >> do not have to interact with main GUi from a processing alg that by
> >> default run in a separated thread => this can cause crash if you do
> >> not use standard processing libs (e.g. feedback) to send message to the user.
> >> No idea if this could be the source of the problem., IMHO sharing the
> >> code would help and also help to verify if it can be replicable.
> >>
> >> regards
> >>
> >> Luigi Pirelli
> >>
> >>
> >> *********************************************************************
> >> *****************************
> >> * LinkedIn: https://www.linkedin.com/in/luigipirelli
> >> * Stackexchange:
> >> http://gis.stackexchange.com/users/19667/luigi-pirelli
> >> * GitHub: https://github.com/luipir
> >> * Book: Mastering QGIS3 - 3rd Edition
> >> <https://www.packtpub.com/eu/application-development/mastering-geospa
> >> tial-development-qgis-3x-third-edition>
> >> * Hire a team: http://www.qcooperative.net
> >>
> >> *********************************************************************
> >> *****************************
> >>
> >>
> >> On Thu, 24 Jun 2021 at 21:51, Alexis R.L. <alroyliz0 at gmail.com> wrote:
> >>
> >>> Greetings,
> >>>
> >>> I have one custom script that uses PIL to get the exif tag, and when
> >>> I execute the script for the first time, the processing window
> >>> vanishes and qgis stall and then crashes.
> >>>
> >>> To get it to run properly I had to set the return before opening the
> >>> files with PIL to run a neutered version and then subsequent runs
> >>> were flawless during that qgis instance.
> >>>
> >>> I ended up using a standalone exif reader, though slower , but there
> >>> are no more crashes.
> >>>
> >>> Has anyone else had similar issues and is anyone aware of a way to
> >>> prevent such crashes or handle them more gracefully, as no trace is
> >>> left currently.
> >>>
> >>> Thanks and have a nice day,
> >>>
> >>> Alex
> >>> _______________________________________________
> >>> QGIS-Developer mailing list
> >>> QGIS-Developer at lists.osgeo.org
> >>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >>>
> >>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20210625/fcd0affd/attachment.html>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
>
> ------------------------------
>
> End of QGIS-Developer Digest, Vol 188, Issue 36
> ***********************************************
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer

More information about the QGIS-Developer mailing list