[QGIS-Developer] Processing: questions on temporary files

[Andreas Neumann]

Hi, 

I got information from Matthias, that the new "Temporary Output Folder
Path" is only used by algorithms around GDAL, SAGA and ExecuteSQL, and
not for vector based QGIS algorithms. I don't know about GRASS -
probably this also runs through files? 

I think we need to address this issue with priority, ideally also with
at backport to our LT version 3.10. Running large vector files always
through memory sounds dangerous to me. It can easily bring a system to
halt. 

The question is, if the switch between memory only operations, or disk
based operations can be determined by software or if it requires user
intervention. Often, it is hard to guess in advance, whether an
algorithm runs out of memory or not. Certainly our average users would
have no idea about this. 

I wonder, how other systems do it? As far as I know, FME always does
things disk based, to stay on the safe side - or am I wrong? 

Greetings, 

Andreas 

On 2019-12-05 08:18, Richard Duivenvoorde wrote:

> On 05/12/2019 01.01, Nyall Dawson wrote: 
> 
>> What's needed/missing is a method for the memory data provider to
>> detect when a layer is growing too large, and start writing this out
>> to disk. (Important note: we CAN'T use any of the standard spatial
>> formats for this (as they are lossy when compared to the memory
>> provider support), so we'd need to directly serialize the memory layer
>> features to a custom binary format, with indexing support for rapid
>> retrieval of features.
> 
> If I understand correct, FME is also working with (indexed) temp files?
> I'm not an FME user, so I can bu wrong, but I always wondered what kind
> of format that is...
> 
> IF we (as FOSS community) could come up with some (new?) indexed file
> format, it (besides being a in-between product for transformations) also
> a potential data exchange format (maybe some zipped version of file+index?)?
> 
> Anybody into this?
> 
> Regards,
> 
> Richard Duivenvoorde
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20191205/64419263/attachment.html>