[OSGeo-UK] Astun Technology OSMM Loader In Practice

[Tim Martin]

Hi All

 

Matt Walker asked if I would document my use of the OSMM Loader, so here
goes

 

I tested both the initial (direct PostGIS loading using INSERT) and the
significantly quicker PGDump version on national coverage OSMM.

 

I used the non-geo chunks, as I didn't want to have to post process the
database tables to remove identical features. Matt has this down as a
future improvement and I would second that along with COU as being
extremely useful. 

 

Details as follows

 

1) Step 1 was to use the Astun loader to create the SQL Dump/COPY files.
For this I used a High-CPU Windows 64 Bit with Python 2.5 and GDAL 1.8
installed via the OSGeo4W installer.

 

Using Non-Geo chunks of OSMM there are 1297 files equating to 35.9GB, I
used 6 threads (the computer had 8 cores) to process 216 files/6Gb each
and this took 15 hours and outputted 215Gb of SQL files, one for each
Gzip file

 

 

 

2) Step 2 Data Loading

 

I used the inbuilt PSQL loader and run 2 threads on a Large Windows
Amazon Instance to load the data in PostGIS. Only 2 threads could be
used as the Large instances only have 2 cores - although it was only
using 30-40% of total CPU, so could potentially up this to 4 threads.

 

The load took 13-14 hours

 

3) Running the spatial indexes

 

I amended the config of PostGIS and upped the maintenance memory to 1Gb.
This helped created the spatial indexes in less than 10 hours compared
to the original 17 hours with default PostGIS

 

So SQL creation + data load + indexes = 40 hours.

 

I guess for many people it will be the fear of using a commandline tool
(people love a good GUI) and the fact they do not have access to these
powerful computers?

I have been very impressed though and would like to thank Matt for his
support

 

Tim

 

 

 

 

Tim Martin Msc Bsc(Hons)

Web Services Consultant and INSPIRE Technical Business Development
Manager

Ordnance Survey 

Mobile: +44 (0) 7917 550 225

www.ordnancesurvey.co.uk | Tim.Martin at ordnancesurvey.co.uk
<mailto:Tim.Martin at ordnancesurvey.co.uk>  

 


This email is only intended for the person to whom it is addressed and may contain confidential information. If you have received this email in error, please notify the sender and delete this email which must not be copied, distributed or disclosed to any other person.

Unless stated otherwise, the contents of this email are personal to the writer and do not represent the official view of Ordnance Survey. Nor can any contract be formed on Ordnance Survey's behalf via email. We reserve the right to monitor emails and attachments without prior notice.

Thank you for your cooperation.

Ordnance Survey
Adanac Drive
Southampton SO16 0AS
Tel: 08456 050505
http://www.ordnancesurvey.co.uk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/uk/attachments/20110826/3949cca3/attachment.html