[gdal-dev] Alternative to GPKG format

[Sean Gillies]

Hi all,

Upgrading to Fiona 1.8.0 won't help the code below because every
outfile.write() call entails a transaction of size 1. The user needs to
pass a sequence or iterator of many records to outfile.writerecords() if
they want to benefit from larger transactions.

On Thu, Nov 1, 2018 at 3:46 AM Even Rouault <even.rouault at spatialys.com>
wrote:

> You are perhaps using a too old version of Fiona. Recent versions insert
> features within OGR transactions. Should be fine with latest Fiona 1.8.0
> See https://github.com/Toblerity/Fiona/issues/476
>
> > Hi, following is the code I used:
> >
> > Import os, osmium, fiona
> >
> > fi = 'europe-latest.osm.pbf'
> > fo = 'europe-latest.dpkg'
> >
> > drv = 'DPKG'
> >
> > crs = {'no_defs': True, 'ellps': 'WGS84', 'datum': 'WGS84', 'proj':
> > 'longlat'}
> >
> >  schema = {'geometry': 'LineString',
> >                    'properties': {'id': 'float', 'name' : 'str', 'kind' :
> > 'str'}}
> >
> > outfile = fiona.open(fo, 'w', driver=drv, crs=crs, schema=schema)
> >
> > geomfab = osmium.geom.GeoJSONFactory()
> >
> > class ShapeConverter(osmium.SimpleHandler):
> >       def way(self, w):
> >            if 'place' in w.tags:
> >                rec = {'geometry' : eval(geomfab.create_linestring(w)),
> >                       'properties' : {'id' : float(w.id),
> >                                      'name' : w.tags.get('name'),
> >                                        'kind' : w.tags['place']}}
> >                outfile.write(rec)
> >
> > ShapeConverter().apply_file(fi, locations=True)
> >
> > On Thu, Nov 1, 2018 at 4:10 PM jratike80 <
> >
> > jukka.rahkonen at maanmittauslaitos.fi> wrote:
> > > koji higuchi wrote
> > >
> > > > Hi
> > > > I am extracting .osm.pbf file into .gpkg; but the writing rate is
> very
> > > > slow.
> > > > When I write into .shp, rate is faster but its limit is 4gb.
> > > > So, what other format is better for larger than 4gb requirement?
> > > > Thanks for your ideas.
> > > > Koji
> > >
> > > Hi,
> > >
> > > Show the exact command that you are using so we can see if it could be
> > > made
> > > faster. For example if you happen to use -skipfailures for handling the
> > > invalid geometries of the OSM data then you will make GDAL to do a new
> > > transaction for each row and that certainly is slow. But let's have a
> look
> > > at your command first.
> > >
> > > -Jukka Rahkonen-
> > >
> > >
> > >
> > > --
> > > Sent from: http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
> > > _______________________________________________
> > > gdal-dev mailing list
> > > gdal-dev at lists.osgeo.org
> > > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev



-- 
Sean Gillies
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181101/17b6b980/attachment.html>