[gdal-dev] Handling Ordnance Survey Change Only Updates

[Peter Halls]

Jez,

On Aug 17 2010, Jez Walters wrote:

>Peter,
>
>
> I can't see how to use ogr2ogr with Change Only Updates (COU) either, but 
> I was hoping for a more definitive response (no offense intended)! :-)
>
> I'm receiving full and COU data in 'chunks' for the whole of Great 
> Britain, and features can move from chunk to chunk between updates. For 
> example, a bridge feature might be first described in the full data file 
> '123456-000012-10.gz' but amendments to this feature might be issued 
> three months later in the COU data file '123456-000011-10.gz'.
>
> Because of the data volumes involved, I can't process each full data 
> shapefile in turn to check its features against ALL of the changed 
> features for the WHOLE of Great Britain. I need an algorithm that 
> processes COU data files one at a time.
>
> I see the first step as establishing which full data shapefile contains a 
> given feature that has changed. Should I maintain my own feature index 
> for this, or can the OGR library work this out for me?
>
> Once I know which shapefile to edit, is it possible (however risky) to 
> add, update and delete features from it - and if so, how?
>

Echoing, perhaps, what Frank and Even have said, so long as your data 
repository has to be as shapefiles, I think you are going to have a lot of 
work to do. I'm fortunate in having a corporate Oracle licence on which to 
build, but PostGIS is a perfectly suitable alternative - shapefiles have so 
many built-in restrictions, consequent on their origin fifteen or so years 
ago. Indeed, if you need ESRI compatibility I gather PostGIS is supported 
from ArcGIS release 10. Managing the data in PostGIS should be as 
straightforward as in Oracle: you can easily extract shapefile segments as 
required but would no longer have the headache of creating and managing an 
index system to determine which feature is in which shapefile.

The OSGB FID can be either alphabetic or numeric - the first four 
characters are always OSGB and the rest is a sixteen digit integer (64 bit 
integer, then, rather than 32). I've been toying with the idea of using the 
integer form as the unique record ID / index column, but I think GDAL 
currently requires this column to be a 32bit integer; I do not yet know 
whether SDE 10 would handle a 64bit integer but I will need to interwork 
with this, so it may be a constraint for me but not for you. As I am 
working primarily with ITN-RRI, rather than the Topographic Layer, the way 
I handle the ID is critical ...

Even mentioned that there are some differences in the GML handling for GML2 
/ GML3. The OSGB GML claims to be GML2 but includes some GML3 constructs - 
I recall someone describing it as GML2.5 but cannot remember who or where. 
I do not know whether this is liable to cause any problems for you.

Best wishes,

Peter

>
>Warm regards,
>
>
>Jez
>
>
>-----Original Message-----
>From: pjh1 at york.ac.uk [mailto:pjh1 at york.ac.uk] On Behalf Of Peter Halls
>Sent: Tuesday 17 August 2010 11:40
>To: Jez Walters
>Cc: 'gdal-dev at lists.osgeo.org'
>Subject: RE: [gdal-dev] Handling Ordnance Survey Change Only Updates
>
>Jez,
>
>   I rather doubt you could use ogr2ogr to effect a MasterMap update, I'm 
> afraid. I do not believe it to be designed to take two inputs and, 
> effectively, merge them to produce the output.
>
>   Yes, it is true that you can edit shapefiles to the extent that 
> records can be marked for deletion and new records added. However, you 
> need logic to identify the records to be updated - which, so far as I am 
> aware, does not exist within ogr2ogr, which is a data conversion tool. 
> There have been other postings regarding updating bounds and indeces, 
> which can all be wored around.
>
>    Because I am paranoid, I would not wish to do an in situ update of my 
> base reference data, such as MasterMap, without preserving the previous 
> version. As I do not believe the logic to exist to update a Mastermap 
> dataset from the Changes GML using GDAL, you will need to create this. 
> Thus the most sensible approach is to take the two inputs, existing 
> shapefile and GML updates, apply the updates and write out a new, updated 
> shapefile. You can, of course, delete or archive the old one once your QA 
> processes have been run and you are confident with the result.
>
>Best wishes,
>
>Peter
>
>On Aug 17 2010, Jez Walters wrote:
>
>>Peter,
>>
>>
>> I may be misinterpreting it, but the GDAL documentation 
>> (http://www.gdal.org/ogr/drv_shapefile.html) suggests that shapefiles 
>> CAN be edited:
>>
>> "The OGR shapefile driver supports rewriting existing shapes in a 
>> shapefile as well as deleting shapes. Deleted shapes are marked for 
>> deletion in the .dbf file, and then ignored by OGR. To actually remove 
>> them permanently (resulting in renumbering of FIDs) invoke the SQL 
>> 'REPACK <tablename>' via the datasource ExecuteSQL() method."
>>
>> I was rather hoping this would be implemented via ogr2ogr - can you 
>> confirm I'm right that ogr2ogr does NOT support this?
>> 
>>
>>Warm regards,
>>
>>
>>Jez
>>
>>
>>-----Original Message-----
>>From: pjh1 at york.ac.uk [mailto:pjh1 at york.ac.uk] On Behalf Of Peter Halls
>>Sent: Tuesday 17 August 2010 08:36
>>To: Jez Walters
>>Cc: gdal-dev at lists.osgeo.org
>>Subject: Re: [gdal-dev] Handling Ordnance Survey Change Only Updates
>>
>>Jez,
>>
>>   I would be surprised if GDAL can handle OSGB MasterMap change-only 
>> updates automatically. The process defined by OS is to locate and 
>> replace the records concerned. Because the update may change the 
>> geometry of a record, I do not believe this can be done on a shapefile 
>> in situ. Instead the procedure would be to input the shapefile 
>> sequentially until a record to be changed is liocated, outputing the 
>> input to a new shapefile and substituting the change record for the 
>> input as appropriate, gaining a new shapefile. On completion, the new 
>> shapefile will have the correct spatial index and bounds. Indeed, it is 
>> probably most sensible to adopt this approach to such updates regardless 
>> of the output format, as this is 'safe' against update problems - which, 
>> in theory, could result from a problematic GML file and permits easy 
>> reversion.
>>
>>Best wishes,
>>
>>Peter
>>
>>On Aug 16 2010, Jez Walters wrote:
>>
>>>
>>>Hello fellow GDAL users!
>>>
>>>
>>>I need to handle Change Only Updates (COU) for the UK Ordnance Survey
>>>MasterMap Topography Layer, and I was wondering how to approach this.
>>>
>>> Both the initial and COU data are supplied as GML, and I am currently 
>>> converting the initial data to ESRI shapefiles using ogr2ogr. However, 
>>> the ogr2ogr "-update" option won't update the shapefiles using the COU 
>>> data - unless I'm missing something?
>>>
>>>I currently see two ways of overcoming this:
>>>
>>> 1. Use the COU data to update the initial GML data, and then re-convert 
>>> the merged data to ESRI shapefiles 2. Use the COU data to directly 
>>> update the ESRI shapefiles
>>>
>>> Obviously the second approach is more efficient, but how difficult is 
>>> it to create, update and delete features using the GDAL python library?
>>>
>>>A few pointers for how to achieve this would be most welcome!  :-)
>>>
>>>
>>>Warm regards,
>>>
>>>
>>>Jez
>>>
>>
>>
>
>

-- 
--------------------------------------------------------------------------
Peter Halls, University of York GIS Advisor
Snail Mail: Computing Service, Heslington, York YO10 5DD, England
Telephone: 01904 433806   FAX: 01904 433740

This message has the status of a private & personal communication
--------------------------------------------------------------------------