[QGIS-Developer] Using memory layer in Processing with SAGA and GRASS

[Benjamin Ducke]

Hi,

On 26/04/18 08:53, Nyall Dawson wrote:
> On 26 April 2018 at 16:42, matteo <matteo.ghetta at gmail.com> wrote:
>> Hi Nyall,
>>
>>> No - neither SAGA nor GRASS have any concept of QGIS memory layers.
>>> You need to give them a disk-based file path to save their outputs to.

That's not quite accurate. Even if they did have
such a concept, it would be of no use in this
context.

SAGA and GRASS commands run as separate processes,
in their own address spaces. Any reasonably secure
operating system (let's generously include Windows
and macOS in that category), will make sure that
processes cannot simply manipulate each other's
memory.

So QGIS needs to create temporary files to pass
both input and output data to/from GRASS/SAGA.

An alternative would be to refactor all
GRASS/SAGA modules as library objects so that they
could run in the same address space as QGIS.
That would probably also involve creating a new
interface API for use by QGIS (or other "host GIS").

Note that we are talking about C/C++ binaries here.
So one would also need to inspect every single GRASS/SAGA
binary and library for memory leaks (which can currently
be ignored, because memory is freed by the OS as soon as
the GRASS/SAGA commands' process ends).

>>
>> ok, clear now why this was not working
>>
>>
>>> If you use a memory layer as an input to these algs it will also have
>>> to save it out to a disk based format, but this is done automatically
>>> by saving to a file in the temp folder. We could potentially do the
>>> same thing to allow SAGA/GRASS algs to output to a memory layer by
>>> first saving the output to a shp in the temp folder, then loading that
>>> file and copying the features to a memory layer. But it would be
>>> rather inefficient and counter productive - you're still being forced
>>> to loop over the disk-based format, so you're just adding extra
>>> processing for little gain.
>>
>> so within a script (or a Processing plugin) both input layer and output
>> one have to come from a disk saved file.
> 
> Not quite - the input layer doesn't *have* to, because processing will
> automatically save it to a disk file if required.

In the case of SAGA, all this auto-creating files
is not too bad, because SAGA uses simple shapefiles
for vector data.

In the case of GRASS, this is a more complicated matter.
GRASS uses a fully topological vector format and all
input data will go through rigorous topology testing
by default. The more intermediate steps, the higher
the risk that data will get crippled in the process.
Note that GRASS now has an alternative way of linking
to external data instead of importing it:

https://grass.osgeo.org/grass74/manuals/v.external.html

So perhaps it would be a good idea for the QGIS GRASS
provider to offer v.external as an alternative to
v.in.ogr for piping QGIS vector layers into the GRASS
backend (however, read "NOTES" on the page cited above!).

Best,

Ben

> 
> Nyall
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> 



-- 
Dr. Benjamin Ducke
Deutsches Archäologisches Institut (DAI)
Zentrale Berlin, IT-Referat
* Projekt "Stunde Null" *