[mapserver-dev] Questions on MapServer architecture and attribute conversions

Sat Mar 16 03:57:59 PDT 2019

I wish we could make it less GMLish and more generic. For example, it feels odd in MVT code to be relying on what feels like format-specific metadata. The various service specs can use a generic “ows” or service specific metadata namespaces. Could do the same for output... “out” as the generic namespace with “gml”, “mvt”, “ogr”, “kml” as specific namespaces.

gmlItemObj and gmlItemListObj could also be renamed more generically and the latter could become iteminfo as part of a layer. A cast function for type conversations would also be helpful for more standard use.

Just thinking out loud - I know that’s way more than you want to get into,

—Steve

________________________________
From: mapserver-dev <mapserver-dev-bounces at lists.osgeo.org> on behalf of Seth G <sethg at geographika.co.uk>
Sent: Friday, March 15, 2019 6:58:41 PM
To: mapserver-dev at lists.osgeo.org
Subject: Re: [mapserver-dev] Questions on MapServer architecture and attribute conversions

Hi Steve, Even,

Thanks for your replies. Thinking back I  think I've run into the issue of strings being used for int fields from a database, and CLASS expressions.

I may make an attempt at the following based on both of your comments:

- add gmlItemObj (and also gmlItemListObj) as classes to MapScript
- allow msGMLGetItems to be called on a layerObj to populate this list
- allow this list to be associated/passed to a method on a shapeObj, and use the GML types to set the output attribute types
- if there are no field definitions supplied then attribute names will simply be indexes, and all values strings

Seth
--
web:http://geographika.co.uk
twitter: @geographika

On Thu, Mar 14, 2019, at 10:14 PM, Lime, Steve D (MNIT) wrote:
> Couple of questions here...
>
> On your pull request. I don't think there's a need for a RFC given the
> scope of the change. it's not breaking anything and seems like a nice
> addition - I certainly defer to your expertise as the Python MapScript
> maintainer. I guess it's never a bad idea to alert the -dev list and
> take your email as that notice - so +1 from me.
>
> On your assumptions, both are correct.
>
> On item types, it's funny this hasn't come up more over the years. Type
> conversions seem to happen either for output using the metadata
> referenced or in the expression handling based on context. You're
> correct the that MapScript, most of the time you'd get the column/item
> names from layer but not always since you can work with shapefiles
> directly and independently of a layerObj or mapObj.
>
> Each layer has an attribute has something called iteminfo and it holds
> the index value associated with corresponding value in the items array.
> So you'd look up the item someone wanted for labeling purposes (or
> whatever) and would know what value to grab from a shapes values array.
> Those item index values are stored all over the place (e.g. labelitem
> and labelitemindex). I think with a scheme like Even describes
> items/iteminto could be replaced by one property (items) that would
> equate to an array of field definitions and then specific things like a
> labelitem becomes a struct consisting of a string and a pointer to one
> of the items. For some reason I thought Dan made a run at that at one
> point but I might be mistaken.
>
> -----Original Message-----
> From: mapserver-dev [mailto:mapserver-dev-bounces at lists.osgeo.org] On
> Behalf Of Even Rouault
> Sent: Thursday, March 14, 2019 11:39 AM
> To: mapserver-dev at lists.osgeo.org
> Subject: Re: [mapserver-dev] Questions on MapServer architecture and
> attribute conversions
>
> Seth,
>
> >
> > I've created a pull request to add a __geo_interface__ to the Python
> > MapScript PointObj, LineObj, and ShapeObjs. This allows Python scripts to
> > easily share MapScript objects with other Python geospatial libraries such
> > as Shapely, QGIS, and ArcPy. It also completes a nice cycle as Sean Gillies
> > was a core developer of the MapScript Python bindings and created
> > __geo_interface__.  This only affects the Python MapScript bindings - do I
> > need to create a RFC or ask for a vote to merge this for a 7.4 release?
> >
> > I also have a few questions on the internal architecture of MapServer
> > classes. Could other devs confirm (or correct) the following assumptions?
> >
> > - the shapeObj has no link back to a layerObj - they are completely
> > independent with no way to get a reference to a layer from a shape - all
> > shapeObj attribute values are stored internally as strings
>
> Yes that's my understanding too.
>
> >
> > To get attribute names for a shapeObj I had to get retrieve them  from the
> > relevant layerObj and store these in a new property on the shapeObj in
> > Python MapScript. I can then output something like the following:
> >
> >     "properties": {
> >         "guid": "954BADBF-2891-48ED-A132-AED39C60E4C9",
> >         "featureid": "203529",
> >         "classid": "12"
> >     }
> >
> > It would be nice to be able to convert the property values to their relevant
> > types e.g. integers and floats. Looking through the MapServer source they
> > seem to only be converted to types using METADATA blocks and
> > "gml_FIELD_NAME_type" "datatype" and in certain cases e.g. in
> > mapogroutput.c Are these conversions all handled differently for different
> > data inputs? Is there an easy way to reuse these conversions in MapScript?
>
> Yes this lack of proper typing on layer attributes is a bit inconvenient and
> causes lot of adhoc coding in various areas of the code base: ogroutput, OGC
> filtering, WFS, GML output, MVT output, etc.
> The fundamental issue is that layerObj::items is just an array of field names.
>
> For brainstorming... A potential solution could take inspiration from what is
> done in GDAL where:
> OGRLayer (~ equivalent of layerObj) -> contains a OGRFeatureDefn
> OGRFeature (~ equivalent of shapeObj) -> points to a OGRFeatureDefn, and has
> an array of field values
>
> A OGRFeatureDefn contains (among other things) an array of OGRFieldDefn
> A OGRFieldDefn is a set of attribute: field name, field type, etc...
>
> Or if not doing a majorhaul, have indeed a pointer from shapeObj back to
> layerObj (which interesting lifetime consideration, but I'm not sure MapScript
> really shines on maintaining object ownership), and use information returned
> by msGMLGetItems().
>
>
> Even
>
>
> --
> Spatialys - Geospatial professional services
> https://gcc01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.spatialys.com&data=02%7C01%7Csteve.lime%40state.mn.us%7C8af807e4be164b6cb28608d6a9a229ad%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636882911288492978&sdata=gq%2FwxERwIC7hIsUULQuBdz5NuWGPSjEEejynXTrU%2BWc%3D&reserved=0
> _______________________________________________
> mapserver-dev mailing list
> mapserver-dev at lists.osgeo.org
> https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fmapserver-dev&data=02%7C01%7Csteve.lime%40state.mn.us%7C8af807e4be164b6cb28608d6a9a229ad%7Ceb14b04624c445198f26b89c2159828c%7C0%7C0%7C636882911288492978&sdata=K6js4DgRQJ4giCNpYqJ3oROAb7HQg3W6xHJaAbXgUxI%3D&reserved=0
> _______________________________________________
> mapserver-dev mailing list
> mapserver-dev at lists.osgeo.org
> https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fmapserver-dev&data=02%7C01%7Csteve.lime%40state.mn.us%7C8af807e4be164b6cb28608d6a9a229ad%7Ceb14b04624c445198f26b89c2159828c%7C0%7C1%7C636882911288502991&sdata=48CiR%2B8XMq030GchIzqmpAdse4sgV1b9Fvjk8Jzt%2Fq8%3D&reserved=0
_______________________________________________
mapserver-dev mailing list
mapserver-dev at lists.osgeo.org
https://gcc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fmapserver-dev&data=02%7C01%7Csteve.lime%40state.mn.us%7C8af807e4be164b6cb28608d6a9a229ad%7Ceb14b04624c445198f26b89c2159828c%7C0%7C1%7C636882911288502991&sdata=48CiR%2B8XMq030GchIzqmpAdse4sgV1b9Fvjk8Jzt%2Fq8%3D&reserved=0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20190316/8d61598e/attachment-0001.html>