[QGIS-Developer] Processing: questions on temporary files

[Matthias Kuhn]

On 12/9/19 10:09 AM, Nyall Dawson wrote:
> On Mon, 9 Dec 2019 at 18:07, Matthias Kuhn <matthias at opengis.ch> wrote:
>> On 12/8/19 11:35 PM, Nyall Dawson wrote:
>>> On Fri, 6 Dec 2019 at 23:19, Matthias Kuhn <matthias at opengis.ch> wrote:
>>>> Other proposals are very welcome as well. I don't insist on GeoPackage.
>>>> All I do is being a bit skeptic that rolling our own format will
>>>> magically solve problems that one hundred other formats did not solve :-)
>>>>
>>> The issue is that the memory provider, by design, MUST be a lossless
>>> reflection of all the capabilities offered in QGIS vector layers. That
>>> means support for every geometry type, and field type handled by QGIS.
>>> Geopackage might be capable of that now... but it won't always be, due
>>> to the inherent limitations of that format. (No multi-geometry
>>> support, no mixed CRS geometry support, no support for advanced field
>>> types like range/interval fields, etc).
>> Having our own binary format makes sense from a full QGIS data type
>> support perspective.
>>
>> As mentioned before, I don't insist on GeoPackage (but now that you say
>> it, does it not have multi-geometry support? According to the specs it
>> should. And just like we can binary dump all our field formats into our
>> own binary file, we could dump them into any file, like e.g. GeoPackage).
> Nope: https://www.geopackage.org/guidance/modeling.html
>
> "Allowing multiple geometries per feature table would compromise
> GeoPackage's position in the GIS application ecosystem"
>
> (FWIW, that page is ridiculous, arrogant, and completely neglects some
> very valid use cases)

The geopackage specs say otherwise 
(http://www.geopackage.org/spec/#geometry_types) and so does our very 
own dialog to create geopackage.

If someone wrote modelling guidelines, that was probably meant as best 
practice (on which one can agree or not).

>
>>> I don't think it's useful to even think of rolling our own data format
>>> -- that's NOT what's required here. All that we need is a way of
>>> paging the currently in-memory storage of features used by the memory
>>> provider to disk when required. This disk based paging would be
>>> totally temporary (always deleted when the layer is removed), have no
>>> requirement for a stable binary format (we could change the paging
>>> binary structure version by version if needed), and absolutely no need
>>> for ANY other programs to be able to parse or interpret. It's not a
>>> data format at all -- it's just a disk based reflection of what we
>>> currently store in RAM, along with some smart indexing to optimize
>>> access.
>> Implementing indexing and paged loading of requested data resolved by an
>> index sound like fun, do you think the effort (or the risks involved) is
>> smaller than tacking support for the missing bits to another format?
> Honestly, yes. I suspect it's more work upfront, but less long-term
> pain of applying hacks on hacks to try to duct tape functionality into
> a standard format.

Wasn't it just thought as a band aid until flatgeobuf comes (riding on a 
unicorn) to solve all our issues?

>
>> I am counting the days until someone wants to recover from one of these
>> files ;)
> As soon as anyone tries that, I'm closing the bug report/feature
> request as "hell-no-wont-fix" ;)

That will be a good move if we follow this path. But I still think 
accessible formats have their merits.

Matthias