[GRASS-user] Crashing during long raster processing loops

[Moody, Alex]

Hi Micah,

Thanks and good catch. I guess I assumed that the grass_session.close() function would clear the path variable, but like you said, that doesn't appear to be the case. The number of entries increases by 5 each time in this case.

This doesn't need addressing, but before I reformatted this to use grass_session I would crash GRASS when running a similar script within the GUI python console. Unfortunately I didn't save the error from that, but maybe it could have been due to similar reasons? I remember it being a memory error.

Alex

From: Micha Silver [mailto:tsvibar at gmail.com]
Sent: Thursday, August 22, 2019 12:07 AM
To: Moody, Alex <Alex.Moody at idwr.idaho.gov>; grass-user at lists.osgeo.org
Subject: Re: [GRASS-user] Crashing during long raster processing loops

Just a guess...
You are repeatedly adding the path to GRASS to your environment variable "PATH" until it becomes longer than allowed.
I would say, open *one* session (in main()), then run your processing loop.

On 22/08/2019 01:49, Moody, Alex wrote:
Hi all,

I consistently have an issue where I am running long loops for raster processing that crash after 100 iterations or so. Typically the error is a ValueError where "ValueError: the environment variable is longer than 32767 bytes"

This has happened when running scripts in the GUI and when using the grass_session package. A sketch of my loop is as follows:


def doSomething(x):
     user = Session()
     user.open(gisdb=mygisdb, location=mylocation, mapset=mymapset,
                   create_opts='')

     v.surf_idw(....)
     r.out_gdal(...)

     user.close()

def main():
     monthlist = list of dates
     for month in monthlist:
           doSomething(month)

if __name__ == "__main__":
    sys.exit(main())

So I'm opening and closing a session in each loop, which  I hoped would take care of memory Issues I may run into.

I'm using 7.6.1 on a 64-Bit Windows 7. The full error is

Traceback (most recent call last):
  File "D:\TreasureValley\waterbudget\riparian\et_interpolate_refet.py", line 82, in <module>
    sys.exit(main())
  File "D:\TreasureValley\waterbudget\riparian\et_interpolate_refet.py", line 79, in main
    makeETrSurface(ts)
  File "D:\TreasureValley\waterbudget\riparian\et_interpolate_refet.py", line 30, in doSomething
    user = Session()
  File "D:\OSGEO4~1\apps\Python27\lib\site-packages\grass_session\session.py", line 244, in __init__
    self.env = set_grass_path_env(gisbase=self.gisbase, env=self.env)
  File "D:\OSGEO4~1\apps\Python27\lib\site-packages\grass_session\session.py", line 102, in set_grass_path_env
    env['PATH'] += os.pathsep + os.path.join(gisbase, 'extrabin')
  File "D:\OSGEO4~1\apps\Python27\lib\os.py", line 422, in __setitem__
    putenv(key, item)
ValueError: the environment variable is longer than 32767 bytes

Any ideas of how to prevent the ValueError? Perhaps some arguments when calling Session?

Thanks as always,
Alex



_______________________________________________

grass-user mailing list

grass-user at lists.osgeo.org<mailto:grass-user at lists.osgeo.org>

https://lists.osgeo.org/mailman/listinfo/grass-user

--

Micha Silver

Ben Gurion Univ.

Sde Boker, Remote Sensing Lab

+972-523-665918
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/grass-user/attachments/20190822/82b093ac/attachment.html>