[mapserver-users] WFS layer - XML/JSON safe chars
Rahkonen Jukka (MML)
jukka.rahkonen at maanmittauslaitos.fi
Fri Apr 22 08:47:37 PDT 2016
Found a way for controlling gml:id, it is in the documentation http://mapserver.org/ogc/wfs_server.html
wfs_featureid
(Required for MapServer 4.10) Field to be used for the ID of the feature in the output GML. gml_featureid or ows_featureid can be specified instead.
The default behavior is not documented but I guess that if wfs_featureid or ows_featureid are not defined then primary key is used instead, and if there is no primary key then heaven knows which field. I think that we should somehow emphasize the importance of wfs_featureid and recommend strongly to use an unique integer field.
What you need to do is to make sure that you have an integer field with unique values in your table and add this to your LAYER METADATA. WFS probably works even if field is not unique but for example GetFeatureById needs unique id:s.
"wfs_featureid" "my_unique_integer_column"
-Jukka-
-----Alkuperäinen viesti-----
Lähettäjä: mapserver-users [mailto:mapserver-users-bounces at lists.osgeo.org] Puolesta Rahkonen Jukka (MML)
Lähetetty: 22. huhtikuuta 2016 18:32
Vastaanottaja: Ian Walberg; mapserver-users at lists.osgeo.org
Aihe: Re: [mapserver-users] WFS layer - XML/JSON safe chars
Hi,
You do not need to understand GML or WFS, we must just get the configuration right.
How have you created your table? Spatialite-gui has a tool in the right click menu "Show CREATE statement" or you can use SQL command PRAGMA table_info('name_of_your_table'). Send the output or check at least what you have as Primary Key (pk).
I believe that if you creat the table to have primary key column like PK_UID here your problem with gml:id disappears.
CREATE TABLE "states" (
"PK_UID" INTEGER PRIMARY KEY AUTOINCREMENT, "STATE_NAME" TEXT, "STATE_FIPS" TEXT,
Ogr2ogr makes such PK by default and so does the import tools of Spatialite-gui.
-Jukka-
-----Alkuperäinen viesti-----
Lähettäjä: Ian Walberg [mailto:ian.walberg at airborne.aero]
Lähetetty: 22. huhtikuuta 2016 18:19
Vastaanottaja: Rahkonen Jukka (MML); mapserver-users at lists.osgeo.org
Aihe: RE: [mapserver-users] WFS layer - XML/JSON safe chars
Jukka,
Thanks for replying and what I would hope to get is "Rio&Paulo" as you say.
However I do not understand your explanation.
Regards
Ian
-----Original Message-----
From: Rahkonen Jukka (MML) [mailto:jukka.rahkonen at maanmittauslaitos.fi]
Sent: Friday, April 22, 2016 8:14 AM
To: Ian Walberg <ian.walberg at airborne.aero>; mapserver-users at lists.osgeo.org
Subject: Re: [mapserver-users] WFS layer - XML/JSON safe chars
Hi Ian,
I can see that those names are used as a part of gml:id:
<ms:DATA_TEST gml:id="DATA_TEST.Rio and Paulo"
and I suppose that you would like to see something like <ms:DATA_TEST gml:id="DATA_TEST.Rio&Paulo"
but I believe that it will never happen because gml:id http://www.datypic.com/sc/niem20/a-gml_id.html is of type xsd:ID http://www.datypic.com/sc/xsd/t-xsd_ID.html which must be NCName, which in turn means that "it must start with a letter or underscore, and can only contain letters, digits, underscores, hyphens, and periods." Et-character is denied there both as raw as escaped.
I am not sure how you can control how Mapserver is building the gml:id. However, I guess that if you use an integer field as primary key in your Spatialite database then it would be utilized and gml:id would look like <ms:DATA_TEST gml:id="DATA_TEST.1"
-Jukka Rahkonen-
-----Alkuperäinen viesti-----
Lähettäjä: mapserver-users [mailto:mapserver-users-bounces at lists.osgeo.org] Puolesta Ian Walberg
Lähetetty: 22. huhtikuuta 2016 17:10
Vastaanottaja: mapserver-users at lists.osgeo.org
Aihe: Re: [mapserver-users] WFS layer - XML/JSON safe chars
Folks,
I think we may be taking about different things, the wfs requests below show the problem we are seeing.
There are 2 cities in each of the sqlite databases and the only difference is on has the text 'and' and the other has the symbol '&'
Layer DATA_TEST - INSERT INTO "test" VALUES('-14.235004','-51.92528','Rio and Paulo');
http://wfs-dev.airborne.aero/cgi-bin/mapserv?map=/data/wfs/WFS_TEST.map&SERVICE=WFS&VERSION=1.1.0&REQUEST=GetFeature&TYPENAME=DATA_TEST
Layer DATA_TEST2 - INSERT INTO "test" VALUES('-14.235004','-51.92528','Rio & Paulo');
http://wfs-dev.airborne.aero/cgi-bin/mapserv?map=/data/wfs/WFS_TEST.map&SERVICE=WFS&VERSION=1.1.0&REQUEST=GetFeature&TYPENAME=DATA_TEST2
So our question is do the strings in database need to be xml safe or does mapserver check that the strings are safe before serving.
Thanks
Ian
-----Original Message-----
From: Even Rouault [mailto:even.rouault at spatialys.com]
Sent: Friday, April 15, 2016 9:15 AM
To: Ian Walberg <ian.walberg at airborne.aero>
Cc: mapserver-users at lists.osgeo.org
Subject: Re: [mapserver-users] WFS layer - XML/JSON safe chars
Le vendredi 15 avril 2016 17:32:36, Ian Walberg a écrit :
> If I understand this we would not get the issue if we use a WFS
> version
> 1.0.0 request.
>
> We have just tested that and it appears that changing the version does
> not change the behavior.
>
> This is with mapserver version 7.0.1.
We are probably talking about different things.
In
https://github.com/mapserver/mapserver/commit/105877468590e99401d234e1bbac13d7645b77d0
, I have just fixed issues in GetCapabilities response generation, but your issues might be different.
I've also just tested reading from a CSV (the input datastore type shouldn't
matter) file a field with & and it is properly escaped as & in the GML output of WFS 1.0, 1.1 and 2.0
Without a mapfile + data + request you issue, it is difficult to be sure we're talking about the same thing.
>
> Thanks
>
> Ian
>
> -----Original Message-----
> From: mapserver-users [mailto:mapserver-users-bounces at lists.osgeo.org]
> On Behalf Of Even Rouault Sent: Friday, April 15, 2016 7:50 AM
> To: mapserver-users at lists.osgeo.org
> Subject: Re: [mapserver-users] WFS layer - XML/JSON safe chars
>
> Le vendredi 15 avril 2016 16:41:28, Richard Watson a écrit :
> > I am using Map Server 6.4.1. I have a map project file which has
> > the
> >
> > following title:
> > WEB
> >
> > METADATA
> >
> > [ text omitted]
> > "wms_title" "(*#(@#*$_)@#&\\\///"
> >
> > END # METADATA
> >
> > END # WEB
> >
> > When a GetCapabilties is done on the project then this is the source
> > XML which is returned:
> >
> > <Service>
> > <Name>WMS</Name>
> > <Title>(*#(@#*$_)@#&\\///</Title>
> > It seems to be escaping as required.
>
> WMS and WFS 1.0 use MapServer manual XML composition with its own XML
> escaping routines. Some other services (such as WFS 1.1 & 2.0) use
> libxml (sometimes not in an appropriate way).
>
> > On Fri, Apr 15, 2016 at 9:21 AM, Rahkonen Jukka (MML) <
> >
> > jukka.rahkonen at maanmittauslaitos.fi> wrote:
> > > Hi,
> > >
> > >
> > >
> > > I was thinking that the output of GetCapabilities is XML and &
> > > must be escaped in XML. So if I write & into the abstract and
> > > Mapserver works correctly it should write my test abstract “test ä
> > > &” escaped perhaps as
> > >
> > > test ä &
> > >
> > > That is the output from the online tool at
> > > http://www.freeformatter.com/xml-escape.html#ad-output
> > >
> > >
> > >
> > > So my conclusion was that because Mapserver does not do it right
> > > here it won’t do it right elsewhere.
> > >
> > >
> > >
> > > But now I made another test by editing the Geoserver test file
> > > “states.shp” in Spatialite as
> > >
> > > UPDATE states set STATE_NAME=’Colo&rado’ WHERE
> > > STATE_NAME=’Colorado’;
> > >
> > >
> > >
> > > Output of GetFeature from WFS 1.1.0 from Mapserver 7.0.0 includes
> > >
> > > <ms:STATE_NAME>Colo&rado</ms:STATE_NAME>
> > >
> > > So my current conclusion is that my Mapserver does not escape “&”
> > > correctly when it is used inside a mapfile and when I do
> > > GetCapabilities request. However, when data come from Spatialite
> > > database and they are consumed by GetFeature then escaping happens
> > > correctly.
> > >
> > >
> > >
> > > We are still comparing different things because you have PostGIS
> > > and I have Spatialite.
> > >
> > >
> > >
> > > -Jukka Rahkonen-
> > >
> > >
> > >
> > >
> > >
> > > Ian Walberg wrote:
> > >
> > >
> > >
> > > Jukka,
> > >
> > >
> > >
> > > I am not sure I understand the test you did.
> > >
> > >
> > >
> > > The data we are referring to is from the database field for the
> > > individual features name.
> > >
> > >
> > >
> > > Regards
> > >
> > >
> > >
> > > Ian
> > >
> > >
> > >
> > > *From:* Rahkonen Jukka (MML)
> > > [mailto:jukka.rahkonen at maanmittauslaitos.fi
> > > <jukka.rahkonen at maanmittauslaitos.fi>]
> > > *Sent:* Thursday, April 14, 2016 10:19 PM
> > > *To:* Ian Walberg <ian.walberg at airborne.aero>; MapserverList OSGEO
> > > < mapserver-users at lists.osgeo.org>
> > > *Subject:* Re: WFS layer - XML/JSON safe chars
> > >
> > >
> > >
> > > Hi Ian,
> > >
> > >
> > >
> > > Thanks for the clarification.
> > >
> > > I made a simple test by adding an abstract into layer metadata.
> > > This shows “test ä” as an abstract
> > >
> > > "wfs_abstract" "test ä”
> > >
> > >
> > >
> > > This one shows an empty abstract
> > >
> > > "wfs_abstract" "test ä &"
> > >
> > >
> > >
> > > So it looks like Mapserver doesn’t know to convert “&” into “&”
> > > for XML.
> > >
> > >
> > >
> > > -Jukka Rahkonen-
> > >
> > >
> > >
> > > Ian Walberg wrote:
> > >
> > >
> > >
> > > Jukka,
> > >
> > >
> > >
> > > Our question was for the data/text read from the database and then
> > > published as XML by the WFS service.
> > >
> > >
> > >
> > > So a pub with the name ‘Dog & bone’ created illegal xml as the &
> > > is not converted to &
> > >
> > >
> > >
> > > Regards
> > >
> > >
> > >
> > > Ian
> > >
> > >
> > >
> > > *From:* Rahkonen Jukka (MML)
> > > [mailto:jukka.rahkonen at maanmittauslaitos.fi
> > > <jukka.rahkonen at maanmittauslaitos.fi>]
> > > *Sent:* Thursday, April 14, 2016 3:10 PM
> > > *To:* Ian Walberg <ian.walberg at airborne.aero>; MapserverList OSGEO
> > > < mapserver-users at lists.osgeo.org>
> > > *Subject:* Re: WFS layer - XML/JSON safe chars
> > >
> > >
> > >
> > > Hi,
> > >
> > >
> > >
> > > You can’t have & in feature type names or attribute names in WFS
> > > because they are defined in the XML schema to be “QName”. The
> > > prefix and LocalPart are of type NCName
> > > https://en.wikipedia.org/wiki/QName
> > > and from
> > > http://stackoverflow.com/questions/1631396/what-is-an-xsncname-typ
> > > e-
> > > and-w
> > > hen-should-it-be-used
> > >
> > >
> > >
> > > The practical restrictions of NCName are that it cannot contain
> > > several symbol characters like :, @, $, %, &, /, +, ,, ;,
> > > whitespace characters or different parenthesis. Furthermore an
> > > NCName cannot begin with a number, dot or minus character although
> > > they can appear later in an NCName.
> > >
> > >
> > >
> > > -Jukka Rahkonen-
> > >
> > >
> > >
> > > *Lähettäjä:* mapserver-users [
> > > mailto:mapserver-users-bounces at lists.osgeo.org
> > > <mapserver-users-bounces at lists.osgeo.org>] *Puolesta *Ian Walberg
> > > *Lähetetty:* 14. huhtikuuta 2016 19:23
> > > *Vastaanottaja:* MapserverList OSGEO
> > > *Aihe:* [mapserver-users] WFS layer - XML/JSON safe chars
> > >
> > >
> > >
> > > Folks,
> > >
> > >
> > >
> > > Can mapserver produce XML or JSON safe characters for WFS served
> > > data or do we need to ensure the source data has the escaped values.
> > >
> > >
> > >
> > > We got tripped up today as there were some ‘&’ in our names which
> > > we had not considered.
> > >
> > >
> > >
> > > Thanks
> > >
> > >
> > >
> > > Ian
> > >
> > > _______________________________________________
> > > mapserver-users mailing list
> > > mapserver-users at lists.osgeo.org
> > > http://lists.osgeo.org/mailman/listinfo/mapserver-users
>
> --
> Spatialys - Geospatial professional services http://www.spatialys.com
> _______________________________________________ mapserver-users
> mailing list mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
--
Spatialys - Geospatial professional services http://www.spatialys.com _______________________________________________
mapserver-users mailing list
mapserver-users at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users
_______________________________________________
mapserver-users mailing list
mapserver-users at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users
More information about the mapserver-users
mailing list