[mapserver-users] WFS layer - XML/JSON safe chars
Even Rouault
even.rouault at spatialys.com
Fri Apr 22 08:28:39 PDT 2016
Le vendredi 22 avril 2016 17:18:36, Ian Walberg a écrit :
> Jukka,
>
> Thanks for replying and what I would hope to get is "Rio&Paulo" as you
> say.
>
> However I do not understand your explanation.
Jukka meant that per the GML specification, it is illegal to have "funky"
characters like &, XML encoded or not, as the gml:id. You should use a integer
primary key for the feature_id.
MapServer GML output indeed doesn't do any attempt of XML encoding the content
of the feature_id field, and it has no real strong reason of attempting to do
that. Except perhaps making well formed XML output, but even when doing so,
that wouldn't validate the GML schemas.
>
> Regards
>
> Ian
>
> -----Original Message-----
> From: Rahkonen Jukka (MML) [mailto:jukka.rahkonen at maanmittauslaitos.fi]
> Sent: Friday, April 22, 2016 8:14 AM
> To: Ian Walberg <ian.walberg at airborne.aero>;
> mapserver-users at lists.osgeo.org Subject: Re: [mapserver-users] WFS layer -
> XML/JSON safe chars
>
> Hi Ian,
>
> I can see that those names are used as a part of gml:id:
> <ms:DATA_TEST gml:id="DATA_TEST.Rio and Paulo"
>
> and I suppose that you would like to see something like <ms:DATA_TEST
> gml:id="DATA_TEST.Rio&Paulo"
>
> but I believe that it will never happen because gml:id
> http://www.datypic.com/sc/niem20/a-gml_id.html is of type xsd:ID
> http://www.datypic.com/sc/xsd/t-xsd_ID.html which must be NCName, which in
> turn means that "it must start with a letter or underscore, and can only
> contain letters, digits, underscores, hyphens, and periods." Et-character
> is denied there both as raw as escaped.
>
> I am not sure how you can control how Mapserver is building the gml:id.
> However, I guess that if you use an integer field as primary key in your
> Spatialite database then it would be utilized and gml:id would look like
> <ms:DATA_TEST gml:id="DATA_TEST.1"
>
> -Jukka Rahkonen-
>
>
> -----Alkuperäinen viesti-----
> Lähettäjä: mapserver-users [mailto:mapserver-users-bounces at lists.osgeo.org]
> Puolesta Ian Walberg Lähetetty: 22. huhtikuuta 2016 17:10
> Vastaanottaja: mapserver-users at lists.osgeo.org
> Aihe: Re: [mapserver-users] WFS layer - XML/JSON safe chars
>
> Folks,
>
> I think we may be taking about different things, the wfs requests below
> show the problem we are seeing.
>
> There are 2 cities in each of the sqlite databases and the only difference
> is on has the text 'and' and the other has the symbol '&'
>
> Layer DATA_TEST - INSERT INTO "test" VALUES('-14.235004','-51.92528','Rio
> and Paulo');
>
> http://wfs-dev.airborne.aero/cgi-bin/mapserv?map=/data/wfs/WFS_TEST.map&SER
> VICE=WFS&VERSION=1.1.0&REQUEST=GetFeature&TYPENAME=DATA_TEST
>
>
> Layer DATA_TEST2 - INSERT INTO "test" VALUES('-14.235004','-51.92528','Rio
> & Paulo');
>
> http://wfs-dev.airborne.aero/cgi-bin/mapserv?map=/data/wfs/WFS_TEST.map&SER
> VICE=WFS&VERSION=1.1.0&REQUEST=GetFeature&TYPENAME=DATA_TEST2
>
>
> So our question is do the strings in database need to be xml safe or does
> mapserver check that the strings are safe before serving.
>
> Thanks
>
> Ian
>
>
>
> -----Original Message-----
> From: Even Rouault [mailto:even.rouault at spatialys.com]
> Sent: Friday, April 15, 2016 9:15 AM
> To: Ian Walberg <ian.walberg at airborne.aero>
> Cc: mapserver-users at lists.osgeo.org
> Subject: Re: [mapserver-users] WFS layer - XML/JSON safe chars
>
> Le vendredi 15 avril 2016 17:32:36, Ian Walberg a écrit :
> > If I understand this we would not get the issue if we use a WFS
> > version
> > 1.0.0 request.
> >
> > We have just tested that and it appears that changing the version does
> > not change the behavior.
> >
> > This is with mapserver version 7.0.1.
>
> We are probably talking about different things.
>
> In
> https://github.com/mapserver/mapserver/commit/105877468590e99401d234e1bbac1
> 3d7645b77d0 , I have just fixed issues in GetCapabilities response
> generation, but your issues might be different.
>
> I've also just tested reading from a CSV (the input datastore type
> shouldn't matter) file a field with & and it is properly escaped as &
> in the GML output of WFS 1.0, 1.1 and 2.0
>
> Without a mapfile + data + request you issue, it is difficult to be sure
> we're talking about the same thing.
>
> > Thanks
> >
> > Ian
> >
> > -----Original Message-----
> > From: mapserver-users [mailto:mapserver-users-bounces at lists.osgeo.org]
> > On Behalf Of Even Rouault Sent: Friday, April 15, 2016 7:50 AM
> > To: mapserver-users at lists.osgeo.org
> > Subject: Re: [mapserver-users] WFS layer - XML/JSON safe chars
> >
> > Le vendredi 15 avril 2016 16:41:28, Richard Watson a écrit :
> > > I am using Map Server 6.4.1. I have a map project file which has
> > > the
> > >
> > > following title:
> > > WEB
> > >
> > > METADATA
> > >
> > > [ text omitted]
> > > "wms_title" "(*#(@#*$_)@#&\\\///"
> > >
> > > END # METADATA
> > >
> > > END # WEB
> > >
> > > When a GetCapabilties is done on the project then this is the source
> > > XML which is returned:
> > >
> > > <Service>
> > > <Name>WMS</Name>
> > > <Title>(*#(@#*$_)@#&\\///</Title>
> > > It seems to be escaping as required.
> >
> > WMS and WFS 1.0 use MapServer manual XML composition with its own XML
> > escaping routines. Some other services (such as WFS 1.1 & 2.0) use
> > libxml (sometimes not in an appropriate way).
> >
> > > On Fri, Apr 15, 2016 at 9:21 AM, Rahkonen Jukka (MML) <
> > >
> > > jukka.rahkonen at maanmittauslaitos.fi> wrote:
> > > > Hi,
> > > >
> > > >
> > > >
> > > > I was thinking that the output of GetCapabilities is XML and &
> > > > must be escaped in XML. So if I write & into the abstract and
> > > > Mapserver works correctly it should write my test abstract “test ä
> > > > &” escaped perhaps as
> > > >
> > > > test ä &
> > > >
> > > > That is the output from the online tool at
> > > > http://www.freeformatter.com/xml-escape.html#ad-output
> > > >
> > > >
> > > >
> > > > So my conclusion was that because Mapserver does not do it right
> > > > here it won’t do it right elsewhere.
> > > >
> > > >
> > > >
> > > > But now I made another test by editing the Geoserver test file
> > > > “states.shp” in Spatialite as
> > > >
> > > > UPDATE states set STATE_NAME=’Colo&rado’ WHERE
> > > > STATE_NAME=’Colorado’;
> > > >
> > > >
> > > >
> > > > Output of GetFeature from WFS 1.1.0 from Mapserver 7.0.0 includes
> > > >
> > > > <ms:STATE_NAME>Colo&rado</ms:STATE_NAME>
> > > >
> > > > So my current conclusion is that my Mapserver does not escape “&”
> > > > correctly when it is used inside a mapfile and when I do
> > > > GetCapabilities request. However, when data come from Spatialite
> > > > database and they are consumed by GetFeature then escaping happens
> > > > correctly.
> > > >
> > > >
> > > >
> > > > We are still comparing different things because you have PostGIS
> > > > and I have Spatialite.
> > > >
> > > >
> > > >
> > > > -Jukka Rahkonen-
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Ian Walberg wrote:
> > > >
> > > >
> > > >
> > > > Jukka,
> > > >
> > > >
> > > >
> > > > I am not sure I understand the test you did.
> > > >
> > > >
> > > >
> > > > The data we are referring to is from the database field for the
> > > > individual features name.
> > > >
> > > >
> > > >
> > > > Regards
> > > >
> > > >
> > > >
> > > > Ian
> > > >
> > > >
> > > >
> > > > *From:* Rahkonen Jukka (MML)
> > > > [mailto:jukka.rahkonen at maanmittauslaitos.fi
> > > > <jukka.rahkonen at maanmittauslaitos.fi>]
> > > > *Sent:* Thursday, April 14, 2016 10:19 PM
> > > > *To:* Ian Walberg <ian.walberg at airborne.aero>; MapserverList OSGEO
> > > > < mapserver-users at lists.osgeo.org>
> > > > *Subject:* Re: WFS layer - XML/JSON safe chars
> > > >
> > > >
> > > >
> > > > Hi Ian,
> > > >
> > > >
> > > >
> > > > Thanks for the clarification.
> > > >
> > > > I made a simple test by adding an abstract into layer metadata.
> > > > This shows “test ä” as an abstract
> > > >
> > > > "wfs_abstract" "test ä”
> > > >
> > > >
> > > >
> > > > This one shows an empty abstract
> > > >
> > > > "wfs_abstract" "test ä &"
> > > >
> > > >
> > > >
> > > > So it looks like Mapserver doesn’t know to convert “&” into “&”
> > > > for XML.
> > > >
> > > >
> > > >
> > > > -Jukka Rahkonen-
> > > >
> > > >
> > > >
> > > > Ian Walberg wrote:
> > > >
> > > >
> > > >
> > > > Jukka,
> > > >
> > > >
> > > >
> > > > Our question was for the data/text read from the database and then
> > > > published as XML by the WFS service.
> > > >
> > > >
> > > >
> > > > So a pub with the name ‘Dog & bone’ created illegal xml as the &
> > > > is not converted to &
> > > >
> > > >
> > > >
> > > > Regards
> > > >
> > > >
> > > >
> > > > Ian
> > > >
> > > >
> > > >
> > > > *From:* Rahkonen Jukka (MML)
> > > > [mailto:jukka.rahkonen at maanmittauslaitos.fi
> > > > <jukka.rahkonen at maanmittauslaitos.fi>]
> > > > *Sent:* Thursday, April 14, 2016 3:10 PM
> > > > *To:* Ian Walberg <ian.walberg at airborne.aero>; MapserverList OSGEO
> > > > < mapserver-users at lists.osgeo.org>
> > > > *Subject:* Re: WFS layer - XML/JSON safe chars
> > > >
> > > >
> > > >
> > > > Hi,
> > > >
> > > >
> > > >
> > > > You can’t have & in feature type names or attribute names in WFS
> > > > because they are defined in the XML schema to be “QName”. The
> > > > prefix and LocalPart are of type NCName
> > > > https://en.wikipedia.org/wiki/QName
> > > > and from
> > > > http://stackoverflow.com/questions/1631396/what-is-an-xsncname-typ
> > > > e-
> > > > and-w
> > > > hen-should-it-be-used
> > > >
> > > >
> > > >
> > > > The practical restrictions of NCName are that it cannot contain
> > > > several symbol characters like :, @, $, %, &, /, +, ,, ;,
> > > > whitespace characters or different parenthesis. Furthermore an
> > > > NCName cannot begin with a number, dot or minus character although
> > > > they can appear later in an NCName.
> > > >
> > > >
> > > >
> > > > -Jukka Rahkonen-
> > > >
> > > >
> > > >
> > > > *Lähettäjä:* mapserver-users [
> > > > mailto:mapserver-users-bounces at lists.osgeo.org
> > > > <mapserver-users-bounces at lists.osgeo.org>] *Puolesta *Ian Walberg
> > > > *Lähetetty:* 14. huhtikuuta 2016 19:23
> > > > *Vastaanottaja:* MapserverList OSGEO
> > > > *Aihe:* [mapserver-users] WFS layer - XML/JSON safe chars
> > > >
> > > >
> > > >
> > > > Folks,
> > > >
> > > >
> > > >
> > > > Can mapserver produce XML or JSON safe characters for WFS served
> > > > data or do we need to ensure the source data has the escaped values.
> > > >
> > > >
> > > >
> > > > We got tripped up today as there were some ‘&’ in our names which
> > > > we had not considered.
> > > >
> > > >
> > > >
> > > > Thanks
> > > >
> > > >
> > > >
> > > > Ian
> > > >
> > > > _______________________________________________
> > > > mapserver-users mailing list
> > > > mapserver-users at lists.osgeo.org
> > > > http://lists.osgeo.org/mailman/listinfo/mapserver-users
> >
> > --
> > Spatialys - Geospatial professional services http://www.spatialys.com
> > _______________________________________________ mapserver-users
> > mailing list mapserver-users at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/mapserver-users
>
> --
> Spatialys - Geospatial professional services http://www.spatialys.com
> _______________________________________________ mapserver-users mailing
> list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
--
Spatialys - Geospatial professional services
http://www.spatialys.com
More information about the mapserver-users
mailing list