[QGIS-Developer] Processing: questions on temporary files

[Nyall Dawson]

On Mon, 9 Dec 2019 at 18:07, Matthias Kuhn <matthias at opengis.ch> wrote:
>
> On 12/8/19 11:35 PM, Nyall Dawson wrote:
> > On Fri, 6 Dec 2019 at 23:19, Matthias Kuhn <matthias at opengis.ch> wrote:
> >> Other proposals are very welcome as well. I don't insist on GeoPackage.
> >> All I do is being a bit skeptic that rolling our own format will
> >> magically solve problems that one hundred other formats did not solve :-)
> >>
> > The issue is that the memory provider, by design, MUST be a lossless
> > reflection of all the capabilities offered in QGIS vector layers. That
> > means support for every geometry type, and field type handled by QGIS.
> > Geopackage might be capable of that now... but it won't always be, due
> > to the inherent limitations of that format. (No multi-geometry
> > support, no mixed CRS geometry support, no support for advanced field
> > types like range/interval fields, etc).
>
> Having our own binary format makes sense from a full QGIS data type
> support perspective.
>
> As mentioned before, I don't insist on GeoPackage (but now that you say
> it, does it not have multi-geometry support? According to the specs it
> should. And just like we can binary dump all our field formats into our
> own binary file, we could dump them into any file, like e.g. GeoPackage).

Nope: https://www.geopackage.org/guidance/modeling.html

"Allowing multiple geometries per feature table would compromise
GeoPackage's position in the GIS application ecosystem"

(FWIW, that page is ridiculous, arrogant, and completely neglects some
very valid use cases)

> > I don't think it's useful to even think of rolling our own data format
> > -- that's NOT what's required here. All that we need is a way of
> > paging the currently in-memory storage of features used by the memory
> > provider to disk when required. This disk based paging would be
> > totally temporary (always deleted when the layer is removed), have no
> > requirement for a stable binary format (we could change the paging
> > binary structure version by version if needed), and absolutely no need
> > for ANY other programs to be able to parse or interpret. It's not a
> > data format at all -- it's just a disk based reflection of what we
> > currently store in RAM, along with some smart indexing to optimize
> > access.
>
> Implementing indexing and paged loading of requested data resolved by an
> index sound like fun, do you think the effort (or the risks involved) is
> smaller than tacking support for the missing bits to another format?

Honestly, yes. I suspect it's more work upfront, but less long-term
pain of applying hacks on hacks to try to duct tape functionality into
a standard format.

>
> I am counting the days until someone wants to recover from one of these
> files ;)

As soon as anyone tries that, I'm closing the bug report/feature
request as "hell-no-wont-fix" ;)

Nyall


>
> Matthias
>