[OSGeo-UK] Astun Technology OSMM Loader In Practice

[Mike Saunt]

Tim

Thanks for the summary - would be good for some joint PR - Astun/OS/OSGeo if
we could.

FYI - I loaded Vector Map Local using the same software for the whole of
Cambridgeshire and it ran without a hitch - very happy.  We're building some
MapServer .map files with this styled which we'll make available at some
stage.  We already have a few variants for MasterMap - if anyone wants them
let us know and we'll publish.

COU - I'm pretty confident with this as we already have a couple of SQL
statements to derive new tables of only the latest TOIDs (for MM) so in
effect the COU files can be loaded / inserted into the same POSTGIS tables
and we'd derive another as a post process.  This way we can have a full
history if required.  Some of our customers have simply commented that they
will probably just take a full update each year and rebuild from scratch.

Re User Interface - we'll be trying to simplify things somewhat and will be
providing 'official' (we already do for quite a few customers) Enterprise
Support, Training and Custom Development for this and other OSGeo products
in the very near future

Many thanks
Mike

PS - my goal would be to have this whole process for national coverage being
achievable in under 24 hours - some tweaks and PostGIS optimizations will
help but I guess we've also got Moore's Law on our side

*Mike Saunt*
Astun Technology Ltd, The Coach House, 17 West Street, Epsom, Surrey, KT18
7RL, UK
t:+44 1372 744 009 m:+44 788 7777 680
iShare - Data integration and publishing
platform<http://www.astuntechnology.com>
See the new Helpdesk and Customer portal<http://support.astuntechnology.com>
*****************************************
Over 35% of the 4 star council websites use iShare - SOCITM Better Connected
2010

See our customer commendations and
awards<http://www.isharemaps.com/customers/awards-and-commendations/>

 Company registration no. 5410695. Registered in England and Wales.
Registered office: 120 Manor Green Road, Epsom, Surrey, KT19 8LN VAT no.
864201149.



On 26 August 2011 12:52, Tim Martin <Tim.Martin at ordnancesurvey.co.uk> wrote:

> ** **
>
> Hi All****
>
> ** **
>
> Matt Walker asked if I would document my use of the OSMM Loader, so here
> goes****
>
> ** **
>
> I tested both the initial (direct PostGIS loading using INSERT) and the
> significantly quicker PGDump version on national coverage OSMM.****
>
> ** **
>
> I used the non-geo chunks, as I didn’t want to have to post process the
> database tables to remove identical features. Matt has this down as a future
> improvement and I would second that along with COU as being extremely
> useful. ****
>
>  ****
>
> Details as follows****
>
>  ****
>
> 1) Step 1 was to use the Astun loader to create the SQL Dump/COPY files.
> For this I used a High-CPU Windows 64 Bit with Python 2.5 and GDAL 1.8
> installed via the OSGeo4W installer.****
>
>  ****
>
> Using Non-Geo chunks of OSMM there are 1297 files equating to 35.9GB, I
> used 6 threads (the computer had 8 cores) to process 216 files/6Gb each and
> this took 15 hours and outputted 215Gb of SQL files, one for each Gzip file
> ****
>
>  ****
>
>  ****
>
>  ****
>
> 2) Step 2 Data Loading****
>
>  ****
>
> I used the inbuilt PSQL loader and run 2 threads on a Large Windows Amazon
> Instance to load the data in PostGIS. Only 2 threads could be used as the
> Large instances only have 2 cores – although it was only using 30-40% of
> total CPU, so could potentially up this to 4 threads.****
>
>  ****
>
> The load took 13-14 hours****
>
> ** **
>
> 3) Running the spatial indexes****
>
>  ****
>
> I amended the config of PostGIS and upped the maintenance memory to 1Gb.
> This helped created the spatial indexes in less than 10 hours compared to
> the original 17 hours with default PostGIS****
>
>  ****
>
> So SQL creation + data load + indexes = 40 hours.****
>
> ** **
>
> I guess for many people it will be the fear of using a commandline tool
> (people love a good GUI) and the fact they do not have access to these
> powerful computers?****
>
> I have been very impressed though and would like to thank Matt for his
> support****
>
> ** **
>
> Tim****
>
> ** **
>
> ** **
>
>  ****
>
> ** **
>
> *Tim Martin Msc Bsc(Hons)*****
>
> *Web Services Consultant and INSPIRE Technical Business Development
> Manager*****
>
> *Ordnance Survey* ****
>
> ****Mobile****: +44 (0) 7917 550 225
> ****
>
> www.ordnancesurvey.co.uk | Tim.Martin at ordnancesurvey.co.uk ****
>
> ** **
>
> This email is only intended for the person to whom it is addressed and may contain confidential information. If you have received this email in error, please notify the sender and delete this email which must not be copied, distributed or disclosed to any other person.
>
> Unless stated otherwise, the contents of this email are personal to the writer and do not represent the official view of Ordnance Survey. Nor can any contract be formed on Ordnance Survey's behalf via email. We reserve the right to monitor emails and attachments without prior notice.
>
> Thank you for your cooperation.
>
> Ordnance Survey
> Adanac Drive
> Southampton SO16 0AS
> Tel: 08456 050505http://www.ordnancesurvey.co.uk
>
>
> _______________________________________________
> UK mailing list
> UK at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/uk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/uk/attachments/20110826/608de592/attachment-0001.html