[gdal-dev] Handling Ordnance Survey Change Only Updates

[Jez Walters]

Peter,


I can't see how to use ogr2ogr with Change Only Updates (COU) either, but I was hoping for a more definitive response (no offense intended)!  :-)

I'm receiving full and COU data in 'chunks' for the whole of Great Britain, and features can move from chunk to chunk between updates.  For example, a bridge feature might be first described in the full data file '123456-000012-10.gz' but amendments to this feature might be issued three months later in the COU data file '123456-000011-10.gz'.

Because of the data volumes involved, I can't process each full data shapefile in turn to check its features against ALL of the changed features for the WHOLE of Great Britain.  I need an algorithm that processes COU data files one at a time.

I see the first step as establishing which full data shapefile contains a given feature that has changed.  Should I maintain my own feature index for this, or can the OGR library work this out for me?

Once I know which shapefile to edit, is it possible (however risky) to add, update and delete features from it - and if so, how?


Warm regards,


Jez


-----Original Message-----
From: pjh1 at york.ac.uk [mailto:pjh1 at york.ac.uk] On Behalf Of Peter Halls
Sent: Tuesday 17 August 2010 11:40
To: Jez Walters
Cc: 'gdal-dev at lists.osgeo.org'
Subject: RE: [gdal-dev] Handling Ordnance Survey Change Only Updates

Jez,

   I rather doubt you could use ogr2ogr to effect a MasterMap update, I'm afraid. I do not believe it to be designed to take two inputs and, 
effectively, merge them to produce the output.

   Yes, it is true that you can edit shapefiles to the extent that records can be marked for deletion and new records added. However, you need logic 
to identify the records to be updated - which, so far as I am aware, does not exist within ogr2ogr, which is a data conversion tool. There have been 
other postings regarding updating bounds and indeces, which can all be wored around.

    Because I am paranoid, I would not wish to do an in situ update of my base reference data, such as MasterMap, without preserving the previous 
version. As I do not believe the logic to exist to update a Mastermap dataset from the Changes GML using GDAL, you will need to create this. Thus 
the most sensible approach is to take the two inputs, existing shapefile and GML updates, apply the updates and write out a new, updated shapefile. 
You can, of course, delete or archive the old one once your QA processes have been run and you are confident with the result.

Best wishes,

Peter

On Aug 17 2010, Jez Walters wrote:

>Peter,
>
>
> I may be misinterpreting it, but the GDAL documentation 
> (http://www.gdal.org/ogr/drv_shapefile.html) suggests that shapefiles CAN 
> be edited:
>
> "The OGR shapefile driver supports rewriting existing shapes in a 
> shapefile as well as deleting shapes. Deleted shapes are marked for 
> deletion in the .dbf file, and then ignored by OGR. To actually remove 
> them permanently (resulting in renumbering of FIDs) invoke the SQL 
> 'REPACK <tablename>' via the datasource ExecuteSQL() method."
>
> I was rather hoping this would be implemented via ogr2ogr - can you 
> confirm I'm right that ogr2ogr does NOT support this?
> 
>
>Warm regards,
>
>
>Jez
>
>
>-----Original Message-----
>From: pjh1 at york.ac.uk [mailto:pjh1 at york.ac.uk] On Behalf Of Peter Halls
>Sent: Tuesday 17 August 2010 08:36
>To: Jez Walters
>Cc: gdal-dev at lists.osgeo.org
>Subject: Re: [gdal-dev] Handling Ordnance Survey Change Only Updates
>
>Jez,
>
>   I would be surprised if GDAL can handle OSGB MasterMap change-only 
> updates automatically. The process defined by OS is to locate and replace 
> the records concerned. Because the update may change the geometry of a 
> record, I do not believe this can be done on a shapefile in situ. Instead 
> the procedure would be to input the shapefile sequentially until a record 
> to be changed is liocated, outputing the input to a new shapefile and 
> substituting the change record for the input as appropriate, gaining a 
> new shapefile. On completion, the new shapefile will have the correct 
> spatial index and bounds. Indeed, it is probably most sensible to adopt 
> this approach to such updates regardless of the output format, as this is 
> 'safe' against update problems - which, in theory, could result from a 
> problematic GML file and permits easy reversion.
>
>Best wishes,
>
>Peter
>
>On Aug 16 2010, Jez Walters wrote:
>
>>
>>Hello fellow GDAL users!
>>
>>
>>I need to handle Change Only Updates (COU) for the UK Ordnance Survey
>>MasterMap Topography Layer, and I was wondering how to approach this.
>>
>> Both the initial and COU data are supplied as GML, and I am currently 
>> converting the initial data to ESRI shapefiles using ogr2ogr. However, 
>> the ogr2ogr "-update" option won't update the shapefiles using the COU 
>> data - unless I'm missing something?
>>
>>I currently see two ways of overcoming this:
>>
>> 1. Use the COU data to update the initial GML data, and then re-convert 
>> the merged data to ESRI shapefiles 2. Use the COU data to directly 
>> update the ESRI shapefiles
>>
>> Obviously the second approach is more efficient, but how difficult is it 
>> to create, update and delete features using the GDAL python library?
>>
>>A few pointers for how to achieve this would be most welcome!  :-)
>>
>>
>>Warm regards,
>>
>>
>>Jez
>>
>
>

-- 
--------------------------------------------------------------------------
Peter Halls, University of York GIS Advisor
Snail Mail: Computing Service, Heslington, York YO10 5DD, England
Telephone: 01904 433806   FAX: 01904 433740

This message has the status of a private & personal communication
--------------------------------------------------------------------------





The information transmitted is intended only for the person
or entity to which it is addressed and may contain
confidential and/or privileged material. If you are not the
addressee, any disclosure, reproduction, copying,
distribution, or other dissemination or use of this
communication is strictly prohibited. If you have received
this transmission in error please notify the sender
immediately and then delete this email.

Any representations or commitments expressed in this email
are subject to contract.

This message has been scanned for viruses and dangerous
content. However, it is essential that the recipient also
checks this message using commercially available mail
scanning and anti-virus software. IPL Information Processing
Limited accepts no liability for any loss or damage resulting
from any virus or other dangerous content in this message.

IPL Information Processing Limited is registered in England
and Wales under company registration number 1418818.
Registration took place at Cardiff on 10 May 1979. IPL
Information Processing Limited's registered office and
normal place of business is Eveleigh House, Grove Street,
Bath, BA1 5LR, United Kingdom. IPL is also registered for
Value Added Tax (VAT) under registration number GB 601 2931 83.