Hi Tim,<br><font style="font-family: arial,helvetica,sans-serif;" size="2"><br>It looks like ogr2ogr can't be found, is ogr2ogr available on your path? Can you also check to make sure the </font><font face="Times New Roman" size="3"><span style="font-size:12.0pt"><font style="font-family: arial,helvetica,sans-serif;" size="2">C:\Astun\temp\osmmloader_</font><font style="font-family: arial,helvetica,sans-serif;" size="2">20110826_144945\su00se looks like a GML document by opening it in a text edit.<br>
<br>Thanks,<br><br>Matt.</font><br></span></font><br><div class="gmail_quote">On 26 August 2011 14:55, Tim Martin <span dir="ltr"><<a href="mailto:Tim.Martin@ordnancesurvey.co.uk">Tim.Martin@ordnancesurvey.co.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<u></u>
<u></u>
<u></u>
<u></u>
<u></u>
<u></u>
<div link="blue" vlink="blue" lang="EN-GB">
<div>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy">Hi matt<u></u><u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"><u></u> <u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy">Just given this a go by changing the osmm.config
to not point to a gfs file an debug to True<u></u><u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"><u></u> <u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy">Pointed it to one VML gz and I get this error<u></u><u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"><u></u> <u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">c:\Astun\python>python osmmloader.py osmmloader.config<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">Config: {'tmp_dir': 'C:\\Astun\\temp', 'out_dir': 'C:\\Astun\\ouput',
'ogr_cmd':<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> "ogr2ogr -append -skipfailures -f PostgreSQL PG:'dbname=vml
active_schema=publi<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">c host=localhost user=postgres password=so164gu' $file_path",
'gfs_file': '', 'd<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">ebug': 'True', 'src_dir': 'C:\\Astun\\data', 'prep_cmd': 'python
preposmm4ogr.py<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> $file_path'}<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">No valid gfs file found, output schema and geometry types will be
determed dynam<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">ically by OGR<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">Processing: C:\Astun\data\su00se.gz<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">Prepared file: C:\Astun\temp\osmmloader_20110826_144945\su00se<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">Prep command: python preposmm4ogr.py C:\Astun\data\su00se.gz<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">Loading: C:\Astun\data\su00se.gz<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">OGR command: ogr2ogr -append -skipfailures -f PostgreSQL PG:dbname=vml
active_sc<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">hema=public host=localhost user=###### password=#####
C:\Astun\temp\osmmload<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">er_20110826_144945\su00se<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">Traceback (most recent call last):<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> File "osmmloader.py", line 185, in <module><u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> main()<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> File "osmmloader.py", line 174, in main<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> loader.run(config)<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> File "osmmloader.py", line 67, in run<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> self.load()<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> File "osmmloader.py", line 151, in load<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> rtn = subprocess.call(ogr_args)<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> File "C:\Python25\lib\subprocess.py", line 444, in
call<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> return Popen(*popenargs, **kwargs).wait()<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> File "C:\Python25\lib\subprocess.py", line 594, in
__init__<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> errread, errwrite)<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> File "C:\Python25\lib\subprocess.py", line 822, in
_execute_child<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> startupinfo)<u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">WindowsError: [Error 2] The system cannot find the file specified<u></u><u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"><u></u> <u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"><u></u> <u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy">Tried both the original and the PGDump option<u></u><u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"><u></u> <u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy">In the temp directly it seems to create a GML
file but no .gfs file<u></u><u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"><u></u> <u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy">Any ideas<u></u><u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"><u></u> <u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy">Thanks<u></u><u></u></span></font></p><div class="im">
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"><br>
Tim<u></u><u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"><u></u> <u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"><u></u> <u></u></span></font></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"><u></u> <u></u></span></font></p>
<div>
<div>
<div style="margin-bottom:9.0pt">
<p class="MsoNormal" style="line-height:12.0pt"><b><b><font color="#4d4d4d" face="Tahoma" size="2"><span style="font-size:10.5pt;font-family:Tahoma;color:#4D4D4D">Tim Martin Msc Bsc(Hons)</span></font></b></b><font color="#4d4d4d" face="Tahoma" size="2"><span style="font-size:10.5pt;font-family:Tahoma;color:#4D4D4D"><u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal" style="line-height:12.0pt"><b><b><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray">Web Services Consultant and INSPIRE Technical Business
Development Manager</span></font></b></b><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray"><u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal" style="line-height:12.0pt"><span><b><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray;font-weight:bold">Ordnance Survey</span></font></b></span><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray"> <u></u><u></u></span></font></p>
</div>
<div><span title="work">
<p class="MsoNormal" style="line-height:12.0pt"><u></u><u></u><span><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray">Mobile</span></font></span><u></u><u></u><span><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray">: </span></font></span><span><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray"><a href="tel:%2B44%20%280%29%C2%A07917%20550%20225" value="+447917550225" target="_blank">+44 (0) 7917 550 225</a></span></font></span></p>
</span><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray"><u></u><u></u></span></font>
</div>
<div>
<p class="MsoNormal" style="line-height:12.0pt"><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray"><a href="http://www.ordnancesurvey.co.uk" target="_blank">www.ordnancesurvey.co.uk</a> | <a href="mailto:Tim.Martin@ordnancesurvey.co.uk" target="_blank">Tim.Martin<span>@ordnancesurvey.co.uk</span></a>
</span></font><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray"><u></u><u></u></span></font></p>
</div>
</div>
</div>
</div><div>
<div class="MsoNormal" style="text-align:center" align="center"><font face="Times New Roman" size="3"><span style="font-size:12.0pt" lang="EN-US">
<hr align="center" size="2" width="100%">
</span></font></div>
<p class="MsoNormal"><b><font face="Tahoma" size="2"><span style="font-size:10.0pt;font-family:Tahoma;font-weight:bold" lang="EN-US">From:</span></font></b><font face="Tahoma" size="2"><span style="font-size:10.0pt;font-family:Tahoma" lang="EN-US">
Matt Walker [mailto:<a href="mailto:walkermatt@longwayaround.org.uk" target="_blank">walkermatt@longwayaround.org.uk</a>] <br>
<b><span style="font-weight:bold">Sent:</span></b> 26 August 2011 13:39</span></font></p><div class="im"><font face="Tahoma" size="2"><br>
<b><span style="font-weight:bold">To:</span></b> Tim Martin<br>
<b><span style="font-weight:bold">Cc:</span></b> <a href="mailto:uk@lists.osgeo.org" target="_blank">uk@lists.osgeo.org</a><br>
</font></div><div><div><font face="Tahoma" size="2"></font></div><div class="h5"><font face="Tahoma" size="2"><b><span style="font-weight:bold">Subject:</span></b> Re: [OSGeo-UK] Astun
Technology OSMM Loader In Practice</font></div></div><span lang="EN-US"><u></u><u></u></span><p></p>
</div><div><div></div><div class="h5">
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"><u></u> <u></u></span></font></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">Hi Tim,<br>
<br>
Thanks for the write-up.<br>
<br>
The easiest way to create a suitable .gfs file is to run the loader against a
single .gz in debug mode with the gfs_file option blank (gfs_file=). This will
cause OGR to create a .gfs for the input dataset in your temp directory which
you can then open in a text editor and tweak as you see fit.<br>
<br>
In related news, a few weeks back I merged some work that Pete Wells at
Faunalia did to provide enhanced output tailored to ESRI Shape and use in QGIS
which he is using with the <a href="http://www.faunalia.co.uk/content/ordnance-survey-translator-updated" target="_blank">QIS
plugin</a> that's built on top of the OSMM Loader.<br>
<br>
As ever the latest OSMM Loader can be found here: <a href="https://github.com/AstunTechnology/osmmloader" target="_blank">https://github.com/AstunTechnology/osmmloader</a>
and the QGIS Plugin here: <a href="http://www.faunalia.it/qgis/plugins.xml" target="_blank">http://www.faunalia.it/qgis/plugins.xml</a><br>
<br>
Cheers,<br>
<br>
Matt.<br>
-- <br>
Matt Walker<br>
Astun Technology<br>
@_walkermatt<br>
<br>
<u></u><u></u></span></font></p>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">On 26 August 2011 13:16, Tim Martin <<a href="mailto:Tim.Martin@ordnancesurvey.co.uk" target="_blank">Tim.Martin@ordnancesurvey.co.uk</a>>
wrote:<u></u><u></u></span></font></p>
<div link="blue" vlink="blue">
<div>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy">Hi Mike</span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy">For VML would I just need the new .gfs file? If you have one can
you send it over and will try a national load as I have the data already on an
amazon box that I could kick off.</span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy">Best person for the PR would be Matt White, will give him an update
and hopefully you guys could do something, especially with AGI not far away.</span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy">If you can get the load time down to 24 hours, or close to ;), then
COU almost becomes redundant as storage is becoming such a commodity</span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy">Cheers</span></font><u></u><u></u></p>
<div>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy">Tim</span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font color="navy" face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial;color:navy"> </span></font><u></u><u></u></p>
<div>
<div>
<div style="margin-bottom:9.0pt">
<p class="MsoNormal" style="line-height:12.0pt"><b><font color="#4d4d4d" face="Tahoma" size="2"><span style="font-size:10.5pt;font-family:Tahoma;color:#4D4D4D;font-weight:bold">Tim
Martin Msc Bsc(Hons)</span></font></b><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="line-height:12.0pt"><b><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray;font-weight:bold">Web
Services Consultant and INSPIRE Technical Business Development Manager</span></font></b><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="line-height:12.0pt"><b><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray;font-weight:bold">Ordnance
Survey</span></font></b><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray"> </span></font><u></u><u></u></p>
</div>
<div><span title="work">
<p class="MsoNormal" style="line-height:12.0pt"><u></u><u></u><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray">Mobile</span></font><u></u><u></u><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray">: <a href="tel:%2B44%20%280%29%C2%A07917%20550%20225" value="+447917550225" target="_blank">+44 (0) 7917 550 225</a></span></font><u></u><u></u></p>
</span></div>
<div>
<p class="MsoNormal" style="line-height:12.0pt"><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray"><a href="http://www.ordnancesurvey.co.uk" target="_blank">www.ordnancesurvey.co.uk</a> | <a href="mailto:Tim.Martin@ordnancesurvey.co.uk" target="_blank">Tim.Martin@ordnancesurvey.co.uk</a>
</span></font><u></u><u></u></p>
</div>
</div>
</div>
</div>
<div>
<div class="MsoNormal" style="text-align:center" align="center"><font face="Times New Roman" size="3"><span style="font-size:12.0pt" lang="EN-US">
<hr align="center" size="2" width="100%">
</span></font></div>
<p class="MsoNormal"><b><font face="Tahoma" size="2"><span style="font-size:10.0pt;font-family:Tahoma;font-weight:bold" lang="EN-US">From:</span></font></b><font face="Tahoma" size="2"><span style="font-size:10.0pt;font-family:Tahoma" lang="EN-US"> Mike Saunt [mailto:<a href="mailto:mikesaunt@astuntechnology.com" target="_blank">mikesaunt@astuntechnology.com</a>]
<br>
<b><span style="font-weight:bold">Sent:</span></b> 26 August 2011 13:03<br>
<b><span style="font-weight:bold">To:</span></b> Tim Martin<br>
<b><span style="font-weight:bold">Cc:</span></b> <a href="mailto:uk@lists.osgeo.org" target="_blank">uk@lists.osgeo.org</a>;
William Allbrook<br>
<b><span style="font-weight:bold">Subject:</span></b> Re: [OSGeo-UK] Astun
Technology OSMM Loader In Practice</span></font><u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> <u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">Tim<u></u><u></u></span></font></p>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> <u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">Thanks for the
summary - would be good for some joint PR - Astun/OS/OSGeo if we could.<u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> <u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">FYI - I loaded
Vector Map Local using the same software for the whole of Cambridgeshire and it
ran without a hitch - very happy. We're building some MapServer
.map files with this styled which we'll make available at some stage.
We already have a few variants for MasterMap - if anyone wants them let
us know and we'll publish.<u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> <u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">COU - I'm pretty
confident with this as we already have a couple of SQL statements to derive new
tables of only the latest TOIDs (for MM) so in effect the COU files can be
loaded / inserted into the same POSTGIS tables and we'd derive another as a
post process. This way we can have a full history if required. Some
of our customers have simply commented that they will probably just take a full
update each year and rebuild from scratch.<u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> <u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">Re User Interface
- we'll be trying to simplify things somewhat and will be providing 'official'
(we already do for quite a few customers) Enterprise Support, Training and
Custom Development for this and other OSGeo products in the very near future<u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> <u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">Many thanks<u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">Mike<br clear="all">
<u></u><u></u></span></font></p>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> <u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">PS - my goal would
be to have this whole process for national coverage being achievable in under
24 hours - some tweaks and PostGIS optimizations will help but I guess we've
also got <u></u><u></u>Moore<u></u><u></u>'s
Law on our side<u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> <u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><b><font face="Times New Roman" size="3"><span style="font-size:12.0pt;font-weight:bold">Mike
Saunt</span></font></b><br>
Astun Technology Ltd, The Coach House, <u></u><u></u>17 West Street<u></u><u></u>, Epsom, <u></u><u></u>Surrey<u></u>, <u></u>KT18 7RL<u></u>,
<u></u>UK<u></u><u></u> <br>
t:<a href="tel:%2B44%201372%20744%20009" value="+441372744009" target="_blank">+44
1372 744 009</a> m:<a href="tel:%2B44%20788%207777%20680" value="+447887777680" target="_blank">+44 788 7777 680</a> <br>
<a href="http://www.astuntechnology.com" target="_blank">iShare - Data
integration and publishing platform</a><br>
<a href="http://support.astuntechnology.com" target="_blank">See the new
Helpdesk and Customer portal</a><br>
***************************************** <br>
<font color="red"><span style="color:red">Over 35% of the 4 star council websites
use iShare - SOCITM Better Connected 2010</span></font><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><font color="red" face="Times New Roman" size="3"><span style="font-size:12.0pt;color:red"><br>
</span></font><a href="http://www.isharemaps.com/customers/awards-and-commendations/" target="_blank">See our customer commendations and awards</a><br>
<br>
Company registration no. 5410695. Registered in <u></u>England<u></u>
and <u></u><u></u>Wales<u></u><u></u>.
Registered office: <u></u><u></u>120 Manor Green
Road<u></u><u></u>, Epsom, <u></u><u></u>Surrey<u></u>,
<u></u>KT19 8LN<u></u><u></u> VAT no.
864201149.<u></u><u></u></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"><u></u> <u></u></span></font></p>
<div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt">On 26 August 2011
12:52, Tim Martin <<a href="mailto:Tim.Martin@ordnancesurvey.co.uk" target="_blank">Tim.Martin@ordnancesurvey.co.uk</a>> wrote:<u></u><u></u></span></font></p>
<div link="blue" vlink="purple">
<div>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial">Hi All</span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial">Matt Walker
asked if I would document my use of the OSMM Loader, so here goes</span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial">I tested both
the initial (direct PostGIS loading using INSERT) and the significantly quicker
PGDump version on national coverage OSMM.</span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial">I used the
non-geo chunks, as I didn’t want to have to post process the database tables to
remove identical features. Matt has this down as a future improvement and I
would second that along with COU as being extremely useful. </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial">Details as
follows</span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial">1) Step 1
was to use the Astun loader to create the SQL Dump/COPY files. For this I used
a High-CPU Windows 64 Bit with Python 2.5 and GDAL 1.8 installed via the
OSGeo4W installer.</span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial">Using
Non-Geo chunks of OSMM there are 1297 files equating to 35.9GB, I used 6
threads (the computer had 8 cores) to process 216 files/6Gb each and this took
15 hours and outputted 215Gb of SQL files, one for each Gzip file</span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial">2) Step 2
Data Loading</span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial">I used the
inbuilt PSQL loader and run 2 threads on a Large Windows Amazon Instance to
load the data in PostGIS. Only 2 threads could be used as the Large instances only
have 2 cores – although it was only using 30-40% of total CPU, so could
potentially up this to 4 threads.</span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial">The load
took 13-14 hours</span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial">3) Running
the spatial indexes</span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial">I amended
the config of PostGIS and upped the maintenance memory to 1Gb. This helped
created the spatial indexes in less than 10 hours compared to the original 17
hours with default PostGIS</span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial">So SQL
creation + data load + indexes = 40 hours.</span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial">I guess for
many people it will be the fear of using a commandline tool (people love a good
GUI) and the fact they do not have access to these powerful computers?</span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial">I have been
very impressed though and would like to thank Matt for his support</span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial">Tim</span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> <u></u><u></u></span></font></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<p class="MsoNormal"><font face="Arial" size="2"><span style="font-size:10.0pt;font-family:Arial"> </span></font><u></u><u></u></p>
<div>
<div style="margin-bottom:9.0pt">
<p class="MsoNormal" style="line-height:12.0pt"><b><font color="#4d4d4d" face="Tahoma" size="2"><span style="font-size:10.5pt;font-family:Tahoma;color:#4D4D4D;font-weight:bold">Tim
Martin Msc Bsc(Hons)</span></font></b><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="line-height:12.0pt"><b><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray;font-weight:bold">Web
Services Consultant and INSPIRE Technical Business Development Manager</span></font></b><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="line-height:12.0pt"><b><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray;font-weight:bold">Ordnance
Survey</span></font></b><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray"> </span></font><u></u><u></u></p>
</div>
<div><span title="work">
<p class="MsoNormal" style="line-height:12.0pt"><u></u><u></u><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray">Mobile</span></font><u></u><u></u><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray">: <a href="tel:%2B44%20%280%29%C2%A07917%20550%20225" value="+447917550225" target="_blank">+44 (0) 7917 550 225</a></span></font><u></u><u></u></p>
</span></div>
<div>
<p class="MsoNormal" style="line-height:12.0pt"><font color="gray" face="Tahoma" size="1"><span style="font-size:9.0pt;font-family:Tahoma;color:gray"></span><a href="http://www.ordnancesurvey.co.uk" target="_blank">www.ordnancesurvey.co.uk</a> | <a href="mailto:Tim.Martin@ordnancesurvey.co.uk" target="_blank">Tim.Martin@ordnancesurvey.co.uk</a>
</font><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> <u></u><u></u></span></font></p>
</div>
</div>
<pre><font face="Courier New" size="2"><span style="font-size:10.0pt">This email is only intended for the person to whom it is addressed and may contain confidential information. If you have received this email in error, please notify the sender and delete this email which must not be copied, distributed or disclosed to any other person.<u></u><u></u></span></font></pre>
<pre><font face="Courier New" size="2"><span style="font-size:10.0pt"> <u></u><u></u></span></font></pre><pre><font face="Courier New" size="2"><span style="font-size:10.0pt">Unless stated otherwise, the contents of this email are personal to the writer and do not represent the official view of Ordnance Survey. Nor can any contract be formed on Ordnance Survey's behalf via email. We reserve the right to monitor emails and attachments without prior notice.<u></u><u></u></span></font></pre>
<pre><font face="Courier New" size="2"><span style="font-size:10.0pt"> <u></u><u></u></span></font></pre><pre><font face="Courier New" size="2"><span style="font-size:10.0pt">Thank you for your cooperation.<u></u><u></u></span></font></pre>
<pre><font face="Courier New" size="2"><span style="font-size:10.0pt"> <u></u><u></u></span></font></pre><pre><font face="Courier New" size="2"><span style="font-size:10.0pt">Ordnance Survey<u></u><u></u></span></font></pre>
<pre><u></u><u></u><font face="Courier New" size="2"><span style="font-size:10.0pt">Adanac Drive</span></font><u></u><u></u><u></u><u></u></pre><pre><u></u><font face="Courier New" size="2"><span style="font-size:10.0pt">Southampton</span></font><u></u> SO16 0AS<u></u><u></u></pre>
<pre><font face="Courier New" size="2"><span style="font-size:10.0pt">Tel: 08456 050505<u></u><u></u></span></font></pre><pre><font face="Courier New" size="2"><span style="font-size:10.0pt"><a href="http://www.ordnancesurvey.co.uk" target="_blank">http://www.ordnancesurvey.co.uk</a><u></u><u></u></span></font></pre>
<pre><font face="Courier New" size="2"><span style="font-size:10.0pt"> <u></u><u></u></span></font></pre>
<p class="MsoNormal" style="margin-bottom:12.0pt"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"><br>
_______________________________________________<br>
<u></u><u></u>UK<u></u><u></u>
mailing list<br>
<a href="mailto:UK@lists.osgeo.org" target="_blank">UK@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/mailman/listinfo/uk" target="_blank">http://lists.osgeo.org/mailman/listinfo/uk</a><u></u><u></u></span></font></p>
</div>
<p class="MsoNormal"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"> <u></u><u></u></span></font></p>
</div>
</div>
</div>
</div>
</div>
<div>
<div><pre><font face="Courier New" size="2"><span style="font-size:10.0pt">This email is only intended for the person to whom it is addressed and may contain confidential information. If you have received this email in error, please notify the sender and delete this email which must not be copied, distributed or disclosed to any other person.<u></u><u></u></span></font></pre>
<pre><font face="Courier New" size="2"><span style="font-size:10.0pt"><u></u> <u></u></span></font></pre><pre><font face="Courier New" size="2"><span style="font-size:10.0pt">Unless stated otherwise, the contents of this email are personal to the writer and do not represent the official view of Ordnance Survey. Nor can any contract be formed on Ordnance Survey's behalf via email. We reserve the right to monitor emails and attachments without prior notice.<u></u><u></u></span></font></pre>
<pre><font face="Courier New" size="2"><span style="font-size:10.0pt"><u></u> <u></u></span></font></pre><pre><font face="Courier New" size="2"><span style="font-size:10.0pt">Thank you for your cooperation.<u></u><u></u></span></font></pre>
<pre><font face="Courier New" size="2"><span style="font-size:10.0pt"><u></u> <u></u></span></font></pre><pre><font face="Courier New" size="2"><span style="font-size:10.0pt">Ordnance Survey<u></u><u></u></span></font></pre>
<pre><u></u><u></u><font face="Courier New" size="2"><span style="font-size:10.0pt">Adanac Drive</span></font><u></u><u></u><u></u><u></u></pre><pre><u></u><font face="Courier New" size="2"><span style="font-size:10.0pt">Southampton</span></font><u></u> SO16 0AS<u></u><u></u></pre>
<pre><font face="Courier New" size="2"><span style="font-size:10.0pt">Tel: 08456 050505<u></u><u></u></span></font></pre><pre><font face="Courier New" size="2"><span style="font-size:10.0pt"><a href="http://www.ordnancesurvey.co.uk" target="_blank">http://www.ordnancesurvey.co.uk</a><u></u><u></u></span></font></pre>
<pre><font face="Courier New" size="2"><span style="font-size:10.0pt"><u></u> <u></u></span></font></pre></div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"><br>
_______________________________________________<br>
<u></u><u></u>UK<u></u><u></u>
mailing list<br>
<a href="mailto:UK@lists.osgeo.org" target="_blank">UK@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/mailman/listinfo/uk" target="_blank">http://lists.osgeo.org/mailman/listinfo/uk</a><u></u><u></u></span></font></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><font face="Times New Roman" size="3"><span style="font-size:12.0pt"><br>
<br>
<u></u><u></u></span></font></p>
</div></div></div>
</div><div><div></div><div class="h5">
<pre>This email is only intended for the person to whom it is addressed and may contain confidential information. If you have received this email in error, please notify the sender and delete this email which must not be copied, distributed or disclosed to any other person.
Unless stated otherwise, the contents of this email are personal to the writer and do not represent the official view of Ordnance Survey. Nor can any contract be formed on Ordnance Survey's behalf via email. We reserve the right to monitor emails and attachments without prior notice.
Thank you for your cooperation.
Ordnance Survey
Adanac Drive
Southampton SO16 0AS
Tel: 08456 050505
<a href="http://www.ordnancesurvey.co.uk" target="_blank">http://www.ordnancesurvey.co.uk</a>
</pre></div></div></blockquote></div><br><br clear="all"><br>-- <br>Matt Walker<br>@_walkermatt<br><br>