<div dir="ltr">I'm writing file using class ShapeConverter(osmium.SimpleHandler) and ogr (not fiona)<br>But, do not know whether transaction are all written instantaneously and simultaneously or not...<br></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Nov 1, 2018 at 8:34 PM koji higuchi <<a href="mailto:koji51guchi@gmail.com">koji51guchi@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Andreas<br>I'm writing to a file into my PC disk.<br></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Nov 1, 2018 at 8:32 PM Andreas Neumann <<a href="mailto:a.neumann@carto.net" target="_blank">a.neumann@carto.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hi,</p>
<p>Just a quick stupid question: are you writing to a local file
system or to a network filesystem. The latter can be substantially
slower than the former, esp. with Geopackage.</p>
<p>Andreas<br>
</p>
<div class="m_5044177143251796378m_7867423460946871683moz-cite-prefix">Am 01.11.18 um 12:10 schrieb koji
higuchi:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi Even
<div>I managed to write gpkg without ogr without fiona using <span style="color:rgb(0,0,255);background-color:rgb(217,234,211)">class
ShapeConverter(osmium.</span><span style="color:rgb(0,0,255);background-color:rgb(217,234,211)">SimpleHandler).</span><br>
</div>
<div>still, speed seems slow.<br>
</div>
<div>Any idea to increase it up.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Thu, Nov 1, 2018 at 7:05 PM koji higuchi <<a href="mailto:koji51guchi@gmail.com" target="_blank">koji51guchi@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Even
<div><br>
</div>
<div>I am trying to replace using fiona by pure ogr.</div>
<div>but having difficult to collapse the code with class
and objects.</div>
<div>If i could extract geom and tags in a ogr pythonic way,
then it be better</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Thu, Nov 1, 2018 at 6:45 PM Even Rouault
<<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">You are
perhaps using a too old version of Fiona. Recent versions
insert <br>
features within OGR transactions. Should be fine with
latest Fiona 1.8.0<br>
See <a href="https://github.com/Toblerity/Fiona/issues/476" rel="noreferrer" target="_blank">https://github.com/Toblerity/Fiona/issues/476</a><br>
<br>
> Hi, following is the code I used:<br>
> <br>
> Import os, osmium, fiona<br>
> <br>
> fi = 'europe-latest.osm.pbf'<br>
> fo = 'europe-latest.dpkg'<br>
> <br>
> drv = 'DPKG'<br>
> <br>
> crs = {'no_defs': True, 'ellps': 'WGS84', 'datum':
'WGS84', 'proj':<br>
> 'longlat'}<br>
> <br>
> schema = {'geometry': 'LineString',<br>
> 'properties': {'id': 'float',
'name' : 'str', 'kind' :<br>
> 'str'}}<br>
> <br>
> outfile = fiona.open(fo, 'w', driver=drv, crs=crs,
schema=schema)<br>
> <br>
> geomfab = osmium.geom.GeoJSONFactory()<br>
> <br>
> class ShapeConverter(osmium.SimpleHandler):<br>
> def way(self, w):<br>
> if 'place' in w.tags:<br>
> rec = {'geometry' :
eval(geomfab.create_linestring(w)),<br>
> 'properties' : {'id' : float(<a href="http://w.id" rel="noreferrer" target="_blank">w.id</a>),<br>
> 'name' :
w.tags.get('name'),<br>
> 'kind' :
w.tags['place']}}<br>
> outfile.write(rec)<br>
> <br>
> ShapeConverter().apply_file(fi, locations=True)<br>
> <br>
> On Thu, Nov 1, 2018 at 4:10 PM jratike80 <<br>
> <br>
> <a href="mailto:jukka.rahkonen@maanmittauslaitos.fi" target="_blank">jukka.rahkonen@maanmittauslaitos.fi</a>>
wrote:<br>
> > koji higuchi wrote<br>
> > <br>
> > > Hi<br>
> > > I am extracting .osm.pbf file into .gpkg;
but the writing rate is very<br>
> > > slow.<br>
> > > When I write into .shp, rate is faster but
its limit is 4gb.<br>
> > > So, what other format is better for larger
than 4gb requirement?<br>
> > > Thanks for your ideas.<br>
> > > Koji<br>
> > <br>
> > Hi,<br>
> > <br>
> > Show the exact command that you are using so we
can see if it could be<br>
> > made<br>
> > faster. For example if you happen to use
-skipfailures for handling the<br>
> > invalid geometries of the OSM data then you will
make GDAL to do a new<br>
> > transaction for each row and that certainly is
slow. But let's have a look<br>
> > at your command first.<br>
> > <br>
> > -Jukka Rahkonen-<br>
> > <br>
> > <br>
> > <br>
> > --<br>
> > Sent from: <a href="http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html" rel="noreferrer" target="_blank">http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html</a><br>
> > _______________________________________________<br>
> > gdal-dev mailing list<br>
> > <a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
> > <a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br>
<br>
<br>
-- <br>
Spatialys - Geospatial professional services<br>
<a href="http://www.spatialys.com" rel="noreferrer" target="_blank">http://www.spatialys.com</a><br>
</blockquote>
</div>
</blockquote>
</div>
<br>
<fieldset class="m_5044177143251796378m_7867423460946871683mimeAttachmentHeader"></fieldset>
<pre class="m_5044177143251796378m_7867423460946871683moz-quote-pre">_______________________________________________
gdal-dev mailing list
<a class="m_5044177143251796378m_7867423460946871683moz-txt-link-abbreviated" href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>
<a class="m_5044177143251796378m_7867423460946871683moz-txt-link-freetext" href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a></pre>
</blockquote>
</div>
_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a></blockquote></div>
</blockquote></div>