[gdal-dev] ogr2ogr problem converting UK Ordnance Survey MasterMap
Peter J Halls
P.Halls at york.ac.uk
Thu Jul 1 10:55:35 EDT 2010
Jez, Even,
there are actually several issues relating to using GDAL/OGR to read
Ordnance Survey of Great Britain (OSGB) GML files distributed as their Mastermap
product. One of these I reported as bug #1604 - I now find that this was
against GDAL-1.4.0 - which concerns handling 'duplicate' tokens: OGR ignores
Namespace and so treats <osgb:point> as the starter and fouls on the following
<gml:point> which contains the geometry. There is a similar problem with
polygon objects.
The data Jez describes below is simpler than much of the data in the file -
perhaps this next point is not an issue for him. Several of the tokens are
described in the schema snippet as 'unbounded': this means that there can be
several instances
<osgb:changeHistory>
<osgb:changeDate>2004-12-19</osgb:changeDate>
<osgb:reasonForChange>Revised</osgb:reasonForChange>
<osgb:changeDate>2002-09-07</osgb:changeDate>
<osgb:reasonForChange>Revised</osgb:reasonForChange>
<osgb:changeDate>2001-03-12</osgb:changeDate>
<osgb:reasonForChange>New</osgb:reasonForChange>
</osgb:changeHistory>
Here it is not only GDAL/OGR that has a problem! Currently, I know of no
importer that can handle this construct, other than the tool (from Snowflake)
used by OSGB to generate it - and there is also the question of onwards storage.
OSGB use Oracle internally and so I presume that these data are stored as
VARRAYs or 'Nested Tables' - complex objects that have no equivalent in many
other data structures. GDAL/OGR has support in the OCI driver for VARRAYs, but
only in the context of the Geometry column; GDAL/OGR has a data structure for a
string array, but this is not supported in the OCI driver. Whilst many of us
may be happy to ignore the change history and descriptive term data in the
example Jez provides, according to the OSGB Mastermap User Guide (Reference
Section) OSGB use unbounded multiple instances for:
theme )
changeHistory )
descriptiveGroup ) present in all Mastermap layers
descriptiveTerm )
referenceToFeature CartographicSymbol layer
networkMember )
referenceToTopgraphicArea )
referenceToNetwork )
dateTimeQualifier ) ITN layer; the qualifiers contain critical data
vehicleQualifier )
environmentQualifier )
subsectionDistance )
subsectionPoint )
Also in the ITN layer, directedNode is a two-part structure.
Even notes that XML permits the specification of a hierarchical dataset,
whilst GDAL/OGR's approach is linear. Surely to comply with the OGC
specification, a gml reader must be able to handle an hierarchical structure?
GML appears to be defined in that way (see Lake, et al, 2004, Geography Markup
Language, Chichester: John Wiley & Sons). I do not know whether the GDAL/OGR
GML driver was designed primarily for writing gml rather than for reading: maybe.
Where does this leave us? As I mentioned, there are also problems with most
other gml readers: this is not solely an issue with GDAL/OGR. I have an
immediate need for the ITN data and have written my own parser to extract the
information from the gml source: so far, so good. However, as I mentioned,
there is now the problem of how to store these data: shapefiles use the dBaseIV
format and have no structure for handling these multiple attributes. In a
sample dataset, I have a record with 49 changeHistory records, for example; some
other multiple constructs have several thousand entries. I happen to have
access to Oracle, although to use GDAL/OGR to write to it requires that I do
some significant work on the oci driver: I've been trying to understand the code
of that to assess what I can reasonably do. Alternatively, I could use oci
directly and bypass GDAL/OGR entirely. All this, however, is non-trivial and
holding me back from doing what I am supposed to be doing ... but does seem to
be the only way forward, having exhausted FME, etc.
Please do not take this as a rant or criticism: that is far from my
intention. Frank and team do an extremely good job but, with the best will in
the world, they cannot be expected to do everything. What I am trying to do is
report the problem that I raised a while back (but had not then realised how big
the task was) and which has now been raised by Jez. If GDAL/OGR is to be
extended to handle these OSGB data, then it is going to require a group of us to
work together to design and deliver: OSGB must be a low priority for the core
team as it can only be of national interest.
Best wishes,
Peter
(sorry that's so long, but there is a lot of information there ...)
Jez Walters wrote:
> Even,
>
>
> This is precisely the fix I came up with yesterday, but I was wondering if there was a more "official" solution instead of editing the GML?
>
>
> Warm regards,
>
>
> Jez
>
>
> -----Original Message-----
> From: Even Rouault [mailto:even.rouault at mines-paris.org]
> Sent: Wednesday 30 June 2010 19:54
> To: gdal-dev at lists.osgeo.org
> Cc: Jez Walters; 'Frank Warmerdam'
> Subject: Re: [gdal-dev] ogr2ogr problem converting UK Ordnance Survey MasterMap
>
> Jez,
>
> the reason for the orientation field to be recognize in the CartographicSymbol
> layer and not in the CartographicText layer lies in the structure of the GML
> file and limitations of the OGR GML driver and/or OGR data model.
>
> Namely OGR only supports a "flat" organization of fields whereas GML (being
> XML based) supports a tree-like organization. In the CartographicSymbol
> layer, the orientation element is at the first level, whereas in the
> CartographicText, it is included inside the <textRendering> element, and then
> is ignored (see below examples)
>
> <osgb:CartographicSymbol fid='osgb1000000729439973'>
> <osgb:featureCode>10165</osgb:featureCode>
> <osgb:version>1</osgb:version>
> <osgb:versionDate>2001-11-11</osgb:versionDate>
> <osgb:theme>Rail</osgb:theme>
> <osgb:changeHistory>
> <osgb:changeDate>2001-03-12</osgb:changeDate>
> <osgb:reasonForChange>New</osgb:reasonForChange>
> </osgb:changeHistory>
> <osgb:descriptiveGroup>Rail</osgb:descriptiveGroup>
> <osgb:descriptiveTerm>Switch</osgb:descriptiveTerm>
> <osgb:orientation>3009</osgb:orientation>
> <osgb:physicalLevel>50</osgb:physicalLevel>
> <osgb:point>
> <gml:Point srsName='osgb:BNG'>
> <gml:coordinates>291781.470,92943.570</gml:coordinates>
> </gml:Point>
> </osgb:point>
> </osgb:CartographicSymbol>
>
> <osgb:CartographicText fid='osgb1000000729439996'>
> <osgb:featureCode>10026</osgb:featureCode>
> <osgb:version>1</osgb:version>
> <osgb:versionDate>2001-11-11</osgb:versionDate>
> <osgb:theme>Buildings</osgb:theme>
> <osgb:anchorPoint>
> <gml:Point srsName='osgb:BNG'>
> <gml:coordinates>291636.850,92668.150</gml:coordinates>
> </gml:Point>
> </osgb:anchorPoint>
> <osgb:changeHistory>
> <osgb:changeDate>1987-12-01</osgb:changeDate>
> <osgb:reasonForChange>New</osgb:reasonForChange>
> </osgb:changeHistory>
> <osgb:descriptiveGroup>Buildings Or Structure</osgb:descriptiveGroup>
> <osgb:make>Manmade</osgb:make>
> <osgb:physicalLevel>50</osgb:physicalLevel>
> <osgb:textRendering>
> <osgb:anchorPosition>3</osgb:anchorPosition>
> <osgb:font>2</osgb:font>
> <osgb:height>1.500</osgb:height>
> <osgb:orientation>545</osgb:orientation>
> </osgb:textRendering>
> <osgb:textString>44</osgb:textString>
> </osgb:CartographicText>
>
> We could imagine an enhancement of the GML driver to try to "flatten" the
> attributes. For example to return textRendering_anchorPosition,
> textRendering_font, .... But it doesn't look trivial to add.
>
> To workaround this, you can open the GML with your favorite text editor (or
> sed ;-)) and replace <osgb:textRendering> and </osgb:textRendering> by
> nothing
>
> This works nicely ;-)
>
> Best regards,
>
> Even
>
> Le Wednesday 30 June 2010 17:17:12 Jez Walters, vous avez écrit :
>> Frank,
>>
>>
>> Thanks for looking in to this! I've just repeated the problem with the
>> Ordnance Survey sample MasterMap, which is available from here:
>>
>> http://www.ordnancesurvey.co.uk/oswebsite/products/innovations/sampledata/O
>> SMasterMap_Topo/58116-SX9192-2c1.gz
>>
>> I tried converting the above GML to ESRI shapefile using the following
>> command:
>>
>> ogr2ogr SX9192 58116-SX9192-2c1
>>
>> One thing I've noticed is that the schema referenced in the GML
>> (http://www.ordnancesurvey.co.uk/xml/schema/v7/OSDNFFeatures.xsd) defines a
>> number of elements called "complexType". The "complexType" element with a
>> name of "CartographicTextType" is defined like this:
>>
>> <complexType name="CartographicTextType">
>> <complexContent>
>> <extension base="osgb:AbstractFeatureType">
>> <sequence>
>> <element name="anchorPoint" type="gml:PointPropertyType"/>
>> <element name="changeHistory" type="osgb:changeHistoryType"
>> maxOccurs="unbounded"/> <element name="descriptiveGroup"
>> type="osgb:descriptiveGroupType" minOccurs="0" maxOccurs="unbounded"/>
>> <element name="descriptiveTerm" type="string" minOccurs="0"
>> maxOccurs="unbounded"/> <element name="make" type="osgb:makeType"
>> minOccurs="0"/>
>> <element name="physicalLevel" type="osgb:physicalLevelType"/>
>> <element name="physicalPresence" type="osgb:physicalPresenceType"
>> minOccurs="0"/> <element name="textRendering"
>> type="osgb:textRenderingType"/>
>> <element name="textString" type="string"/>
>> </sequence>
>> </extension>
>> </complexContent>
>> </complexType>
>>
>> >From which I conclude that the orientation is actually part of the
>>> "textRendering" element. I'm sure I'm on to something with this, but I'm
>>> not sure how to proceed from here!
>> Warm regards,
>>
>>
>> Jez
>>
>>
>> -----Original Message-----
>> From: fwarmerdam at gmail.com [mailto:fwarmerdam at gmail.com] On Behalf Of Frank
>> Warmerdam Sent: Wednesday 30 June 2010 15:38
>> To: Jez Walters
>> Cc: gdal-dev at lists.osgeo.org
>> Subject: Re: [gdal-dev] ogr2ogr problem converting UK Ordnance Survey
>> MasterMap
>>
>> Jez,
>>
>> Can you provide a mastermap file I can test with?
>>
>> Best regards,
>>
>> On Wed, Jun 30, 2010 at 4:31 PM, Jez Walters <jez.walters at ipl.com> wrote:
>>> Not quite sure if I've got the right forum, but I'm having trouble using
>>> the 'ogr2ogr' utility (part of GDAL 1.7.2) to convert Ordnance Survey GML
>>> into ESRI shapefiles.
>>>
>>> The 'orientation' field is being ignored when I convert the
>>> CartographicText layer, although this field gets successfully translated
>>> for the
>>> CartographicSymbol layer. All the other CartographicText layer fields
>>> appear just fine in the generated '.dbf' file.
>>>
>>> Is there a way of forcing ogr2ogr to recognise the orientation field?
>>> Can anyone suggest an alternative Open Source conversion utility? I'm
>>> fairly new to GIS systems, so I might be making an obvious mistake, but
>>> any advice is very gratefully received! :-)
>>> --
>>> View this message in context:
>>> http://osgeo-org.1803224.n2.nabble.com/ogr2ogr-problem-converting-UK-Ordn
>>> ance-Survey-MasterMap-tp5237796p5237796.html Sent from the GDAL - Dev
>>> mailing list archive at Nabble.com.
>>> _______________________________________________
>>> gdal-dev mailing list
>>> gdal-dev at lists.osgeo.org
>>> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
> The information transmitted is intended only for the person
> or entity to which it is addressed and may contain
> confidential and/or privileged material. If you are not the
> addressee, any disclosure, reproduction, copying,
> distribution, or other dissemination or use of this
> communication is strictly prohibited. If you have received
> this transmission in error please notify the sender
> immediately and then delete this email.
>
> Any representations or commitments expressed in this email
> are subject to contract.
>
> This message has been scanned for viruses and dangerous
> content. However, it is essential that the recipient also
> checks this message using commercially available mail
> scanning and anti-virus software. IPL Information Processing
> Limited accepts no liability for any loss or damage resulting
> from any virus or other dangerous content in this message.
>
> IPL Information Processing Limited is registered in England
> and Wales under company registration number 1418818.
> Registration took place at Cardiff on 10 May 1979. IPL
> Information Processing Limited's registered office and
> normal place of business is Eveleigh House, Grove Street,
> Bath, BA1 5LR, United Kingdom. IPL is also registered for
> Value Added Tax (VAT) under registration number GB 601 2931 83.
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
--------------------------------------------------------------------------------
Peter J Halls, GIS Advisor, University of York
Telephone: 01904 433806 Fax: 01904 433740
Snail mail: Computing Service, University of York, Heslington, York YO10 5DD
This message has the status of a private and personal communication
--------------------------------------------------------------------------------
More information about the gdal-dev
mailing list