[OSGeo-UK] Astun Technology OSMM Loader In Practice

Fri Aug 26 09:55:16 EDT 2011

Hi matt

Just given this a go by changing the osmm.config to not point to a gfs
file an debug to True

Pointed it to one VML gz and I get this error

c:\Astun\python>python osmmloader.py osmmloader.config

Config: {'tmp_dir': 'C:\\Astun\\temp', 'out_dir': 'C:\\Astun\\ouput',
'ogr_cmd':

 "ogr2ogr -append -skipfailures -f PostgreSQL PG:'dbname=vml
active_schema=publi

c host=localhost user=postgres password=so164gu' $file_path",
'gfs_file': '', 'd

ebug': 'True', 'src_dir': 'C:\\Astun\\data', 'prep_cmd': 'python
preposmm4ogr.py

 $file_path'}

No valid gfs file found, output schema and geometry types will be
determed dynam

ically by OGR

Processing: C:\Astun\data\su00se.gz

Prepared file: C:\Astun\temp\osmmloader_20110826_144945\su00se

Prep command: python preposmm4ogr.py C:\Astun\data\su00se.gz

Loading: C:\Astun\data\su00se.gz

OGR command: ogr2ogr -append -skipfailures -f PostgreSQL PG:dbname=vml
active_sc

hema=public host=localhost user=###### password=#####
C:\Astun\temp\osmmload

er_20110826_144945\su00se

Traceback (most recent call last):

  File "osmmloader.py", line 185, in <module>

    main()

  File "osmmloader.py", line 174, in main

    loader.run(config)

  File "osmmloader.py", line 67, in run

    self.load()

  File "osmmloader.py", line 151, in load

    rtn = subprocess.call(ogr_args)

  File "C:\Python25\lib\subprocess.py", line 444, in call

    return Popen(*popenargs, **kwargs).wait()

  File "C:\Python25\lib\subprocess.py", line 594, in __init__

    errread, errwrite)

  File "C:\Python25\lib\subprocess.py", line 822, in _execute_child

    startupinfo)

WindowsError: [Error 2] The system cannot find the file specified

Tried both the original and the PGDump option

In the temp directly it seems to create a GML file but no .gfs file

Any ideas

Thanks

Tim

Tim Martin Msc Bsc(Hons)

Web Services Consultant and INSPIRE Technical Business Development
Manager

Ordnance Survey 

Mobile: +44 (0) 7917 550 225

www.ordnancesurvey.co.uk | Tim.Martin at ordnancesurvey.co.uk
<mailto:Tim.Martin at ordnancesurvey.co.uk>  

________________________________

From: Matt Walker [mailto:walkermatt at longwayaround.org.uk] 
Sent: 26 August 2011 13:39
To: Tim Martin
Cc: uk at lists.osgeo.org
Subject: Re: [OSGeo-UK] Astun Technology OSMM Loader In Practice

Hi Tim,

Thanks for the write-up.

The easiest way to create a suitable .gfs file is to run the loader
against a single .gz in debug mode with the gfs_file option blank
(gfs_file=). This will cause OGR to create a .gfs for the input dataset
in your temp directory which you can then open in a text editor and
tweak as you see fit.

In related news, a few weeks back I merged some work that Pete Wells at
Faunalia did to provide enhanced output tailored to ESRI Shape and use
in QGIS which he is using with the QIS plugin
<http://www.faunalia.co.uk/content/ordnance-survey-translator-updated>
that's built on top of the OSMM Loader.

As ever the latest OSMM Loader can be found here:
https://github.com/AstunTechnology/osmmloader and the QGIS Plugin here:
http://www.faunalia.it/qgis/plugins.xml

Cheers,

Matt.
-- 
Matt Walker
Astun Technology
@_walkermatt

On 26 August 2011 13:16, Tim Martin <Tim.Martin at ordnancesurvey.co.uk>
wrote:

Hi Mike

For VML would I just need the new .gfs file? If you have one can you
send it over and will try a national load as I have the data already on
an amazon box that I could kick off.

Best person for the PR would be Matt White, will give him an update and
hopefully you guys could do something, especially with AGI not far away.

If you can get the load time down to 24 hours, or close to ;), then COU
almost becomes redundant as storage is becoming such a commodity

Cheers

Tim

Tim Martin Msc Bsc(Hons)

Web Services Consultant and INSPIRE Technical Business Development
Manager

Ordnance Survey 

Mobile: +44 (0) 7917 550 225 <tel:%2B44%20%280%29%C2%A07917%20550%20225>

www.ordnancesurvey.co.uk | Tim.Martin at ordnancesurvey.co.uk 

________________________________

From: Mike Saunt [mailto:mikesaunt at astuntechnology.com] 
Sent: 26 August 2011 13:03
To: Tim Martin
Cc: uk at lists.osgeo.org; William Allbrook
Subject: Re: [OSGeo-UK] Astun Technology OSMM Loader In Practice

Tim

Thanks for the summary - would be good for some joint PR -
Astun/OS/OSGeo if we could.

FYI - I loaded Vector Map Local using the same software for the whole of
Cambridgeshire and it ran without a hitch - very happy.  We're building
some MapServer .map files with this styled which we'll make available at
some stage.  We already have a few variants for MasterMap - if anyone
wants them let us know and we'll publish.

COU - I'm pretty confident with this as we already have a couple of SQL
statements to derive new tables of only the latest TOIDs (for MM) so in
effect the COU files can be loaded / inserted into the same POSTGIS
tables and we'd derive another as a post process.  This way we can have
a full history if required.  Some of our customers have simply commented
that they will probably just take a full update each year and rebuild
from scratch.

Re User Interface - we'll be trying to simplify things somewhat and will
be providing 'official' (we already do for quite a few customers)
Enterprise Support, Training and Custom Development for this and other
OSGeo products in the very near future

Many thanks

Mike

PS - my goal would be to have this whole process for national coverage
being achievable in under 24 hours - some tweaks and PostGIS
optimizations will help but I guess we've also got Moore's Law on our
side

Mike Saunt
Astun Technology Ltd, The Coach House, 17 West Street, Epsom, Surrey,
KT18 7RL, UK 
t:+44 1372 744 009 <tel:%2B44%201372%20744%20009>  m:+44 788 7777 680
<tel:%2B44%20788%207777%20680>  
iShare - Data integration and publishing platform
<http://www.astuntechnology.com> 
See the new Helpdesk and Customer portal
<http://support.astuntechnology.com> 
***************************************** 
Over 35% of the 4 star council websites use iShare - SOCITM Better
Connected 2010

See our customer commendations and awards
<http://www.isharemaps.com/customers/awards-and-commendations/> 

Company registration no. 5410695. Registered in England and Wales.
Registered office: 120 Manor Green Road, Epsom, Surrey, KT19 8LN VAT no.
864201149.

On 26 August 2011 12:52, Tim Martin <Tim.Martin at ordnancesurvey.co.uk>
wrote:

Hi All

Matt Walker asked if I would document my use of the OSMM Loader, so here
goes

I tested both the initial (direct PostGIS loading using INSERT) and the
significantly quicker PGDump version on national coverage OSMM.

I used the non-geo chunks, as I didn't want to have to post process the
database tables to remove identical features. Matt has this down as a
future improvement and I would second that along with COU as being
extremely useful. 

Details as follows

1) Step 1 was to use the Astun loader to create the SQL Dump/COPY files.
For this I used a High-CPU Windows 64 Bit with Python 2.5 and GDAL 1.8
installed via the OSGeo4W installer.

Using Non-Geo chunks of OSMM there are 1297 files equating to 35.9GB, I
used 6 threads (the computer had 8 cores) to process 216 files/6Gb each
and this took 15 hours and outputted 215Gb of SQL files, one for each
Gzip file

2) Step 2 Data Loading

I used the inbuilt PSQL loader and run 2 threads on a Large Windows
Amazon Instance to load the data in PostGIS. Only 2 threads could be
used as the Large instances only have 2 cores - although it was only
using 30-40% of total CPU, so could potentially up this to 4 threads.

The load took 13-14 hours

3) Running the spatial indexes

I amended the config of PostGIS and upped the maintenance memory to 1Gb.
This helped created the spatial indexes in less than 10 hours compared
to the original 17 hours with default PostGIS

So SQL creation + data load + indexes = 40 hours.

I guess for many people it will be the fear of using a commandline tool
(people love a good GUI) and the fact they do not have access to these
powerful computers?

I have been very impressed though and would like to thank Matt for his
support

Tim

Tim Martin Msc Bsc(Hons)

Web Services Consultant and INSPIRE Technical Business Development
Manager

Ordnance Survey 

Mobile: +44 (0) 7917 550 225 <tel:%2B44%20%280%29%C2%A07917%20550%20225>

www.ordnancesurvey.co.uk | Tim.Martin at ordnancesurvey.co.uk 

This email is only intended for the person to whom it is addressed and
may contain confidential information. If you have received this email in
error, please notify the sender and delete this email which must not be
copied, distributed or disclosed to any other person.

Unless stated otherwise, the contents of this email are personal to the
writer and do not represent the official view of Ordnance Survey. Nor
can any contract be formed on Ordnance Survey's behalf via email. We
reserve the right to monitor emails and attachments without prior
notice.

Thank you for your cooperation.

Ordnance Survey
Adanac Drive
Southampton SO16 0AS
Tel: 08456 050505
http://www.ordnancesurvey.co.uk

_______________________________________________
UK mailing list
UK at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/uk

This email is only intended for the person to whom it is addressed and
may contain confidential information. If you have received this email in
error, please notify the sender and delete this email which must not be
copied, distributed or disclosed to any other person.

Unless stated otherwise, the contents of this email are personal to the
writer and do not represent the official view of Ordnance Survey. Nor
can any contract be formed on Ordnance Survey's behalf via email. We
reserve the right to monitor emails and attachments without prior
notice.

Thank you for your cooperation.

Ordnance Survey
Adanac Drive
Southampton SO16 0AS
Tel: 08456 050505
http://www.ordnancesurvey.co.uk

_______________________________________________
UK mailing list
UK at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/uk

This email is only intended for the person to whom it is addressed and may contain confidential information. If you have received this email in error, please notify the sender and delete this email which must not be copied, distributed or disclosed to any other person.

Unless stated otherwise, the contents of this email are personal to the writer and do not represent the official view of Ordnance Survey. Nor can any contract be formed on Ordnance Survey's behalf via email. We reserve the right to monitor emails and attachments without prior notice.

Thank you for your cooperation.

Ordnance Survey
Adanac Drive
Southampton SO16 0AS
Tel: 08456 050505
http://www.ordnancesurvey.co.uk

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.osgeo.org/pipermail/uk/attachments/20110826/375a589e/attachment-0001.html