[QGIS-Developer] Processing: questions on temporary files

[Nyall Dawson]

On Mon, 9 Dec 2019 at 20:10, Matthias Kuhn <matthias at opengis.ch> wrote:
>
> Thanks for the clarification Andreas, indeed there was a misunderstanding on my end.
>
> That's admittedly a pain point with GPKG, I hope this can be resolved in future versions of the spec.
>
> But let's be honest, in QGIS we are not much better yet, integration of additional geometry columns is not very polished. We have a "main" geometry column to work with and "secondary" geometry columns which can be accessed to some degree. A concept which we can tack onto GPKG (or any format with no field content limitation --  guess who just dropped out of the candidate formats) as well.

Yes, this is something I think we need to work on. Secondary geometry
fields should be exposed as geometry values, not wkt or wkb or another
encoded format.

I've submitted various PRs over the years to try to implement this,
but none of them have been good enough to merge :P (yet!)

Nyall


>
> Matthias
>
> On 12/9/19 10:29 AM, Andreas Neumann wrote:
>
> If I may chime in: I think Matthias and Nyall are talking about two different things:
>
> Matthias: Multigeometry data types, like MultiPolygon, MultiLinestring, etc. - that is well supported by Geopackage
>
> Nyall: mulitple geometry columns - that works in Postgis, Interlis and some other formats, but not Geopackage. Yes, there are some valid use cases, where the same object should allow different representations, e.g. for different scales, or as point and polygon.
>
> Greetings,
> Andreas
>
> On 2019-12-09 10:22, Matthias Kuhn wrote:
>
>
> On 12/9/19 10:09 AM, Nyall Dawson wrote:
>
> On Mon, 9 Dec 2019 at 18:07, Matthias Kuhn <matthias at opengis.ch> wrote:
>
> On 12/8/19 11:35 PM, Nyall Dawson wrote:
>
> On Fri, 6 Dec 2019 at 23:19, Matthias Kuhn <matthias at opengis.ch> wrote:
>
> Other proposals are very welcome as well. I don't insist on GeoPackage.
> All I do is being a bit skeptic that rolling our own format will
> magically solve problems that one hundred other formats did not solve :-)
>
> The issue is that the memory provider, by design, MUST be a lossless
> reflection of all the capabilities offered in QGIS vector layers. That
> means support for every geometry type, and field type handled by QGIS.
> Geopackage might be capable of that now... but it won't always be, due
> to the inherent limitations of that format. (No multi-geometry
> support, no mixed CRS geometry support, no support for advanced field
> types like range/interval fields, etc).
>
> Having our own binary format makes sense from a full QGIS data type
> support perspective.
>
> As mentioned before, I don't insist on GeoPackage (but now that you say
> it, does it not have multi-geometry support? According to the specs it
> should. And just like we can binary dump all our field formats into our
> own binary file, we could dump them into any file, like e.g. GeoPackage).
>
> Nope: https://www.geopackage.org/guidance/modeling.html
>
> "Allowing multiple geometries per feature table would compromise
> GeoPackage's position in the GIS application ecosystem"
>
> (FWIW, that page is ridiculous, arrogant, and completely neglects some
> very valid use cases)
>
>
> The geopackage specs say otherwise (http://www.geopackage.org/spec/#geometry_types) and so does our very own dialog to create geopackage.
>
> If someone wrote modelling guidelines, that was probably meant as best practice (on which one can agree or not).
>
>
> I don't think it's useful to even think of rolling our own data format
> -- that's NOT what's required here. All that we need is a way of
> paging the currently in-memory storage of features used by the memory
> provider to disk when required. This disk based paging would be
> totally temporary (always deleted when the layer is removed), have no
> requirement for a stable binary format (we could change the paging
> binary structure version by version if needed), and absolutely no need
> for ANY other programs to be able to parse or interpret. It's not a
> data format at all -- it's just a disk based reflection of what we
> currently store in RAM, along with some smart indexing to optimize
> access.
>
> Implementing indexing and paged loading of requested data resolved by an
> index sound like fun, do you think the effort (or the risks involved) is
> smaller than tacking support for the missing bits to another format?
>
> Honestly, yes. I suspect it's more work upfront, but less long-term
> pain of applying hacks on hacks to try to duct tape functionality into
> a standard format.
>
>
> Wasn't it just thought as a band aid until flatgeobuf comes (riding on a unicorn) to solve all our issues?
>
>
> I am counting the days until someone wants to recover from one of these
> files ;)
>
> As soon as anyone tries that, I'm closing the bug report/feature
> request as "hell-no-wont-fix" ;)
>
>
> That will be a good move if we follow this path. But I still think accessible formats have their merits.
>
> Matthias
>
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
>