[gdal-dev] Alternative to GPKG format

[Even Rouault]

You are perhaps using a too old version of Fiona. Recent versions insert 
features within OGR transactions. Should be fine with latest Fiona 1.8.0
See https://github.com/Toblerity/Fiona/issues/476

> Hi, following is the code I used:
> 
> Import os, osmium, fiona
> 
> fi = 'europe-latest.osm.pbf'
> fo = 'europe-latest.dpkg'
> 
> drv = 'DPKG'
> 
> crs = {'no_defs': True, 'ellps': 'WGS84', 'datum': 'WGS84', 'proj':
> 'longlat'}
> 
>  schema = {'geometry': 'LineString',
>                    'properties': {'id': 'float', 'name' : 'str', 'kind' :
> 'str'}}
> 
> outfile = fiona.open(fo, 'w', driver=drv, crs=crs, schema=schema)
> 
> geomfab = osmium.geom.GeoJSONFactory()
> 
> class ShapeConverter(osmium.SimpleHandler):
>       def way(self, w):
>            if 'place' in w.tags:
>                rec = {'geometry' : eval(geomfab.create_linestring(w)),
>                       'properties' : {'id' : float(w.id),
>                                      'name' : w.tags.get('name'),
>                                        'kind' : w.tags['place']}}
>                outfile.write(rec)
> 
> ShapeConverter().apply_file(fi, locations=True)
> 
> On Thu, Nov 1, 2018 at 4:10 PM jratike80 <
> 
> jukka.rahkonen at maanmittauslaitos.fi> wrote:
> > koji higuchi wrote
> > 
> > > Hi
> > > I am extracting .osm.pbf file into .gpkg; but the writing rate is very
> > > slow.
> > > When I write into .shp, rate is faster but its limit is 4gb.
> > > So, what other format is better for larger than 4gb requirement?
> > > Thanks for your ideas.
> > > Koji
> > 
> > Hi,
> > 
> > Show the exact command that you are using so we can see if it could be
> > made
> > faster. For example if you happen to use -skipfailures for handling the
> > invalid geometries of the OSM data then you will make GDAL to do a new
> > transaction for each row and that certainly is slow. But let's have a look
> > at your command first.
> > 
> > -Jukka Rahkonen-
> > 
> > 
> > 
> > --
> > Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
> > _______________________________________________
> > gdal-dev mailing list
> > gdal-dev at lists.osgeo.org
> > https://lists.osgeo.org/mailman/listinfo/gdal-dev


-- 
Spatialys - Geospatial professional services
http://www.spatialys.com