[QGIS-Developer] Questions wrt. problems with empty layers (SOLVED - somewhat)
Bo Victor Thomsen
bo.victor.thomsen at gmail.com
Sat May 11 10:44:51 PDT 2019
Hi all -
(for the impatient, read the last part of this mail)
I've been digging a little bit further into my own setup - essentially
doing the same operations as Richard
I examined the 4 layers i GeoServer
* groenreg:bykerne: *SqlServer* based layer *with* rows.
Geometry column defined as "geometry", but defined in
"geometry_columns" as "MULTIPOLYGON", dimensions = 2 and srid = 25832
* groenreg:bykerne_pg: *Postgres* based layer*with* rows
Geometry column defined as "geometry (MULTIPOLYON,25832) and defined
in "geometry_columns" as "MULTIPOLYGON", dimensions = 2 and srid =
25832
* groenreg:bykerne_empty:*Sql-server* based layer *without* rows
Geometry column defined as "geometry", but defined in
"geometry_columns" as "MULTIPOLYGON", dimensions = 2 and srid = 25832
* groenreg:bykerne_pg_empty: *Postgres* based layer *without* rows .
Geometry column defined as "geometry (MULTIPOLYON,25832) and defined
in "geometry_columns" as "MULTIPOLYGON", dimensions = 2 and srid =
25832
The datastore in Geoserver for the layers in SQLServer is set up so the
location of the geometry_columns is known to GeoServer
The documentation for GeoServer clearly states, that if there is a
geometry_columns meta table present and is known to GeoServer, GeoServer
will use information from this table to ascertain the geometry type for
the layers:
https://docs.geoserver.org/stable/en/user/data/database/sqlserver.html#using-the-geometry-metadata-table
When I make a WFS describefeature - request for each of the 4 layers I
get the following result:
* groenreg:bykerne: ...<xsd:element maxOccurs="1" minOccurs="0"
name="ogr_geometry" nillable="true"
type="*gml:GeometryPropertyType*"/>...
* groenreg:bykerne_pg: ... <xsd:element maxOccurs="1" minOccurs="0"
name="wkb_geometry" nillable="true"
type="*gml:MultiSurfacePropertyType*"/>...
* groenreg:bykerne_empty: ...<xsd:element maxOccurs="1" minOccurs="0"
name="ogr_geometry" nillable="true"
type="*gml:GeometryPropertyType*"/>...
* groenreg:bykerne_pg_empty: ... <xsd:element maxOccurs="1"
minOccurs="0" name="wkb_geometry" nillable="true"
type="*gml:MultiSurfacePropertyType*"/>...
I.e. The layers based on ms-sqlserver (with or without rows) always is
seen as type="*gml:GeometryPropertyType*" (general geometry, no subtype)
and layers based on Postgres is seen as
type="*gml:MultiSurfacePropertyType (*i.e. the correct subtype for the
specific layer)
My conclusion:
GeoServer is *not* using the information stored in the geometry_columns
meta table to define the real geometry subtype for a MS-SQLServer based
layer. Hence QGIS is forced to detect the geometry-type by fetching one
row and determine the type from this row. If there is no rows, QGIS is
unable the determine the type.
On the other hand Postgres based layers from GeoServer shows the right
geometry subtype in the WFS describefeature request. These layers in
QGIS gets the right type with or without rows in the source database table.
Why GeoServer isn't using the information in the geometry columns meta
table is unknown to me. It can be a faulty setup by me or a bug in the
GeoServer code.
*NB!!! STOP PRESS -**
*
I've just investigated the structure of geometry_columns in my sqlserver
database and compared it to the geometry_columns structure that
GeoServer prescribes:
There *is* a bloody difference: The *geometry_type* column in my
sqlserver geometry_columns table has the "wrong" name. It should be
named simply *type*
A simple renaming of column *geometry_type* to *type *in table
geometry_columns solves the issue with the zero-row sqlserver tables as
layers
(Just turned away from computer and stomping head into wall for about 3
minutes...
.... Back again with a slight headache...)
Retrying the WFS describefeature request on the sql server layers now
gives the equivalent result as with Postgres:
* ...<xsd:element maxOccurs="1" minOccurs="0"
name="ogr_geometry" nillable="true"
type="*gml:MultiSurfacePropertyType*"/>...
There seems to be a discrepancy in how different spatial software
interprets how geometry_columns should be structured
* GDAL uses "geometry_type" (hence SQLServer uses this too, because it
was created by ogc2ogc command-line tool in sqlserver)
* PostGIS view use "type"
* Spatialite uses "geometry_type"
* GeoServer uses "type"
and so on ....
How do we ever reach "world peace" or solve the climate crisis if we
even can't agree on the name of friggin' meta table column ....
Regards
Bo Victor Thomsen
GIS & Database specialist
AestasGIS
Denmark
Den 10-05-2019 kl. 17:10 skrev Alessandro Pasotti:
>
>
> On Fri, May 10, 2019 at 5:04 PM Richard Duivenvoorde
> <rdmailings at duif.net <mailto:rdmailings at duif.net>> wrote:
>
> Hi,
>
> Just tried with a Geoserver (2.15) I have running here.
>
> Moved a shp file with 1 feature in a postgis db, created a wfs service
> for it and loaded in QGIS, styled it etc: fine.
>
> Removed the (one) feature from the table, retarted Geoserver, and
> had a
> look into the requests QGIS is firing [0].
>
> QGIS first does a GetCapabilities request:
> /geoserver/wfs?SERVICE=WFS&REQUEST=GetCapabilities&ACCEPTVERSIONS=2.0.0,1.1.0,1.0.0
>
> Then a describefeature request (to determine the type):
>
> geoserver/wfs?SERVICE=WFS&REQUEST=DescribeFeatureType&VERSION=2.0.0&TYPENAME=test:wfspointPostgis
>
> That one responses with
>
> <xsd:element maxOccurs="1" minOccurs="0" name="geom" nillable="true"
> type="gml:PointPropertyType"/>
>
> So Geoserver (still?) knows about the geometry type of the table.
> Not sure if that is cached? But it is available in the
> 'geometry_columns'-view of Postgis (stating type=POINT).
>
> And QGIS keeps the styling, even after several restarts/reloads.
>
> One thing I can think of: is that the postgis table does not have
> a type
> defined, with someting like:
> geom geometry
> instead of
> geom geometry(Point,28992),
> what I see when the table created via QGIS...
>
> Could it be that
> - geoserver cannot determine the type?
>
> To test I created a minimal table:
> CREATE TABLE public.testgeoms
> (
> id integer,
> geom geometry,
> fid integer NOT NULL DEFAULT nextval('testgeoms_fid_seq'::regclass),
> CONSTRAINT foo PRIMARY KEY (fid)
> )
>
> At that moment the geometry_columns view shows type=GEOMETRY
> So (I think) without data, geoserver can never determine the type,
> hence
> shows:
> <xsd:element maxOccurs="1" minOccurs="0" name="geom" nillable="true"
> type="gml:GeometryPropertyType"/>
> in the DescribeFeatureType response...
>
> And indeed if I then empty the table, and reopen the project the
> WFS-point layer shows an 'img', as it cannot style the zero features.
> AND it does not have type info?
>
> So (confirmed) having that layer with zero (of unknown type) features,
> QGIS forgets the style...
>
> Is that a bug?
>
>
> QGIS styles are bound to geometry types, it the type is unknown there
> is not much we can do .... something similar happens for bad/broken
> layers.
>
> For that implementation I'm storing the original XML layer style that
> can be re-applied in case the "bad layer" is fixed, maybe we can
> re-use that mechanism for this use case.
>
>
> IF you create the layers with a feature TYPE (geom
> geometry(Point,28992)) then the type will show up in the
> 'geometry_columns'-view and then in the describefeature response.
>
> Without that knowledge QGIS retrieves NO type information (via the
> describefeature) upon loading of the project/layer and (apparently)
> forgets the type in the layer???
>
> Does it work with you if you put the type in the table definitions in
> Postgis (so they show up in the 'geometry_columns'-view)?
> With me it does, but I just did a quick test.
>
> Else I would create an issue for it or let somebody dive into this...
>
> Regards,
>
> Richard Duivenvoorde
>
>
> [0]
> https://github.com/rduivenvoorde/qgisnetworklogger/tree/model_rewrite
>
>
> On 09/05/2019 22.22, Bo Victor Thomsen wrote:
> > Hi Lars and other members of this list..
> >
> > I feel somewhat obligated to help Lars since it's a problem he has
> > inherited from me. (Sorry about that...)
> >
> > To sum it up:
> >
> > If QGIS connects to a WFS based layer from Geoserver using an
> existing
> > project with styled layers and one or more layers containing /zero/
> > objects, QGIS can't determine the object type (point, line-string,
> > polygon...) for those layers. QGIS will not show legends and
> promptly
> > forget any layer information stored in the project related to the
> > object-type, ex. styling information.
> >
> > And what is worse - this is a permanent situation: If you save the
> > project in the above-mentioned state and reopen it at a later time -
> > where the layers /now/ contains objects - QGIS has forgotten the
> styling
> > and other information, so the layers stays invisible and without the
> > former styling.
> >
> > (long posting...)
> >
> > I've done some tests to determine the root cause of this problem:
> >
> > * Setup of a Postgres database /and /a SQL Server database.
> >
> > * Loaded the same MapInfo-dataset twice into each database, called
> > them tableA and tableB, and populated the meta-table
> > "dbo.geometry_columns" with the correct information for
> tableA and
> > tableB in SQLServer.
> >
> > * Installed a fresh Geoserver 2.15 and MapServer TinyOWS WFS-T
> server
> >
> > * Setup a Workspace in Geoserver with 4 layers: tableA and -B from
> > SQLServer and the same tables from Postgres using WFS.
> >
> > * In Geoserver, the datastore definition for the SQLServer
> tables has
> > a pointer to the "dbo.geometry_columns" meta-table.
> >
> > * Setup of TinyOWS with tableA and -B from Postgres
> >
> > * And lastly: Made a project in QGIS 3.4.7 /Windows 64bit with
> the 6
> > WFS layers:
> >
> > o tableA is shown 3 times:
> >
> > 1. From SQLServer through GeoServer;
> > 2. From Postgres through GeoServer
> > 3. From Postgres through Tiny-WFS
> >
> > o tableB is shown 3 times using the same setup as tableA
> >
> > All 6 layers in QGIS was shown correctly and the object type for all
> > layers was recognized.
> >
> > After this I /deleted/ all rows from tableB in /both/
> MS-SQLServer and
> > in Postgres and reopened the same QGIS project:
> >
> > * The layer showing (the zero objects) tableB from SQLServer
> through
> > GeoServer was not in order.
> > * The 5 other layers was shown correctly. Legends was correct and
> > styling was not deleted
> >
> > I'm drawing the following conclusion:
> >
> > * GeoServer WFS connections (maybe in conjunction with QGIS)
> doesn't
> > work properly when the dataprovider is MS-SQLServer and the
> layers
> > contains zero objects. More precisely: GeoServer's SQLServer
> > dataprovider can't or don't use geometry_columns meta-table
> defined
> > in the datastore as described in:
> >
> https://docs.geoserver.org/stable/en/user/data/database/sqlserver.html#using-the-geometry-metadata-table
> > .
> >
> > After this test I've tried to change the placement and existence
> of the
> > "geometry_columns" metadata table:
> >
> > * No value
> > * dbo.geometry_column (ordinary placement)
> > * data.geometry_column (I made a copy of the geometry_columns
> table
> > and placed it in the same schema as the test tables)
> > * geometry_columns (without schema definition)
> >
> > Nothing of the above worked.
> >
> > @Lars:
> > You might change your database system to Postgres and perhaps
> replace
> > GeoServer with TinyOWS for WFS purposes (ducks and and run....) .
> > Or getting the issue investigated and fixed using some GeoServer
> core
> > developer, perhaps in cooperation with a QGIS core developer?
> > Apparently the problem is fixable, since neither TinyOWS or
> > GeoServer/Postgres have the same issues.
> >
> > Obviously, the above is a far cry from a really rigorous
> investigation
> > of the problem, but it points at some of the "pain" points
> >
> > Regards
> >
> > Bo Victor Thomsen
> >
> > AestasGIS
> > Denmark
> >
> > Den 09-05-2019 kl. 13:36 skrev Lars I. Nielsen, LIFA A/S:
> >>
> >> Hi developers.
> >>
> >>
> >>
> >> I asked this question on the user’s list, so far without any
> answers,
> >> but have been advised to ask here as well.
> >>
> >>
> >>
> >>
> >>
> >> I’m working with a number of projects, which statically open a
> fixed
> >> number of layers. All layers are fetched using WFS / WFS-T.
> >>
> >>
> >>
> >> Each layer has an extensive rule based styling, and an extensive
> >> field/attribute dialog setup. This is saved in the project files.
> >>
> >>
> >>
> >> Our plugin changes the data provider filter for many layers,
> when the
> >> user works with the data. These filter expressions are also
> saved in
> >> the project, of course.
> >>
> >>
> >>
> >> Our problem is, that if the user accidently chooses a work area
> (= new
> >> data provider filter), which contain no data in any layer, those
> >> layers lose all styling etc. And it’s apparently impossible to
> recover
> >> these styles etc. by changing the filter back.
> >>
> >>
> >>
> >> So basically the user’s project is unrecoverably trashed at
> this point
> >> in time.
> >>
> >>
> >>
> >> A - Is this a correct description of how QGIS handles “no-data”
> layers ?
> >>
> >>
> >>
> >> B - Are there any standard ways to handle this in QGIS (using
> metadata
> >> or otherwise) in an automated fashion (i.e. not manually) ?
> >>
> >>
> >>
> >> C - If the answers to A and B are yes and no resp., are there any
> >> plans to remedy this unnecessarily serious problem ?
> >>
> >>
> >>
> >>
> >>
> >> Might it not be solved simply by adding an (optional?) layer
> metadata
> >> key, that stores the last used topology, and revert to this if
> no data
> >> are found on the layer ?
> >>
> >>
> >>
> >>
> >>
> >> Using 3.4.5 x64 (Danish)
> >>
> >>
> >>
> >>
> >>
> >> Med venlig hilsen
> >>
> >> Lars I. Nielsen, LIFA A/S
> >> GIS-konsulent, FME Certified Professional
> >>
> >> Beskrivelse: http://website.lifa.dk/lsp.gif
> >>
> >> *T*
> >>
> >>
> >>
> >> 6313 6800
> >>
> >>
> >>
> >> *@*
> >>
> >>
> >>
> >> lin at lifa.dk <mailto:lin at lifa.dk>
> >>
> >> *D*
> >>
> >>
> >>
> >> 6313 6849
> >>
> >>
> >>
> >> *W*
> >>
> >>
> >>
> >> www.lifa.dk <http://www.lifa.dk> <http://www.lifa.dk>
> >>
> >> *M*
> >>
> >>
> >>
> >> 2492 4866
> >>
> >>
> >>
> >> *CVR*
> >>
> >>
> >>
> >> 20937289
> >>
> >>
> >> Beskrivelse:
> >>
> C:\Users\lin\AppData\Roaming\Microsoft\signatures\21x21-Images-Get-L749-l8.png
> >> <https://www.linkedin.com/company/lifa-a-s/>
> >> Følg os på LinkedIn og læs de seneste nyheder fra LIFA A/S
> >>
> >>
> >>
> >> _______________________________________________
> >> QGIS-Developer mailing list
> >> QGIS-Developer at lists.osgeo.org
> <mailto:QGIS-Developer at lists.osgeo.org>
> >> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >> Unsubscribe:
> https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >
> >
> > _______________________________________________
> > QGIS-Developer mailing list
> > QGIS-Developer at lists.osgeo.org
> <mailto:QGIS-Developer at lists.osgeo.org>
> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> >
>
> _______________________________________________
> QGIS-Developer mailing list
> QGIS-Developer at lists.osgeo.org <mailto:QGIS-Developer at lists.osgeo.org>
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-developer
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-developer
>
>
>
> --
> Alessandro Pasotti
> w3: www.itopen.it <http://www.itopen.it>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20190511/af0a7661/attachment-0001.html>
More information about the QGIS-Developer
mailing list