[QGIS-Developer] Processing: questions on temporary files

[Nyall Dawson]

On Wed, 4 Dec 2019 at 21:08, Andreas Neumann <a.neumann at carto.net> wrote:

> I'd like to better understand how QGIS processing works. What happens in such a case if the data is really, really big? Would it crash the machine, or switch from memory layer to a real file "automagically"? If yes, how would this be decided? Or is it the responsibility of the OS to swap such really large temporary layers into files, without the user noticing?

Yes -- a memory layer is currently purely "memory only", and it's up
to the os to decide what to do if the available ram runs out. (On
linux this usually results in the app crashing, on Windows it'll start
smashing the hard disk and grind to a halt).

What's needed/missing is a method for the memory data provider to
detect when a layer is growing too large, and start writing this out
to disk. (Important note: we CAN'T use any of the standard spatial
formats for this (as they are lossy when compared to the memory
provider support), so we'd need to directly serialize the memory layer
features to a custom binary format, with indexing support for rapid
retrieval of features.

Nyall