[gdal-dev] Alternative to GPKG format

[Andreas Neumann]

Hi,

Just a quick stupid question: are you writing to a local file system or 
to a network filesystem. The latter can be substantially slower than the 
former, esp. with Geopackage.

Andreas

Am 01.11.18 um 12:10 schrieb koji higuchi:
> Hi Even
> I managed to write gpkg without ogr without fiona using class 
> ShapeConverter(osmium.SimpleHandler).
> still, speed seems slow.
> Any idea to increase it up.
>
> On Thu, Nov 1, 2018 at 7:05 PM koji higuchi <koji51guchi at gmail.com 
> <mailto:koji51guchi at gmail.com>> wrote:
>
>     Hi Even
>
>     I am trying to replace using fiona by pure ogr.
>     but having difficult to collapse the code with class and objects.
>     If i could extract geom and tags in a ogr pythonic way, then it be
>     better
>
>
>     On Thu, Nov 1, 2018 at 6:45 PM Even Rouault
>     <even.rouault at spatialys.com <mailto:even.rouault at spatialys.com>>
>     wrote:
>
>         You are perhaps using a too old version of Fiona. Recent
>         versions insert
>         features within OGR transactions. Should be fine with latest
>         Fiona 1.8.0
>         See https://github.com/Toblerity/Fiona/issues/476
>
>         > Hi, following is the code I used:
>         >
>         > Import os, osmium, fiona
>         >
>         > fi = 'europe-latest.osm.pbf'
>         > fo = 'europe-latest.dpkg'
>         >
>         > drv = 'DPKG'
>         >
>         > crs = {'no_defs': True, 'ellps': 'WGS84', 'datum': 'WGS84',
>         'proj':
>         > 'longlat'}
>         >
>         >  schema = {'geometry': 'LineString',
>         >                    'properties': {'id': 'float', 'name' :
>         'str', 'kind' :
>         > 'str'}}
>         >
>         > outfile = fiona.open(fo, 'w', driver=drv, crs=crs,
>         schema=schema)
>         >
>         > geomfab = osmium.geom.GeoJSONFactory()
>         >
>         > class ShapeConverter(osmium.SimpleHandler):
>         >       def way(self, w):
>         >            if 'place' in w.tags:
>         >                rec = {'geometry' :
>         eval(geomfab.create_linestring(w)),
>         >                       'properties' : {'id' : float(w.id
>         <http://w.id>),
>         >                                      'name' :
>         w.tags.get('name'),
>         >                                        'kind' :
>         w.tags['place']}}
>         >                outfile.write(rec)
>         >
>         > ShapeConverter().apply_file(fi, locations=True)
>         >
>         > On Thu, Nov 1, 2018 at 4:10 PM jratike80 <
>         >
>         > jukka.rahkonen at maanmittauslaitos.fi
>         <mailto:jukka.rahkonen at maanmittauslaitos.fi>> wrote:
>         > > koji higuchi wrote
>         > >
>         > > > Hi
>         > > > I am extracting .osm.pbf file into .gpkg; but the
>         writing rate is very
>         > > > slow.
>         > > > When I write into .shp, rate is faster but its limit is 4gb.
>         > > > So, what other format is better for larger than 4gb
>         requirement?
>         > > > Thanks for your ideas.
>         > > > Koji
>         > >
>         > > Hi,
>         > >
>         > > Show the exact command that you are using so we can see if
>         it could be
>         > > made
>         > > faster. For example if you happen to use -skipfailures for
>         handling the
>         > > invalid geometries of the OSM data then you will make GDAL
>         to do a new
>         > > transaction for each row and that certainly is slow. But
>         let's have a look
>         > > at your command first.
>         > >
>         > > -Jukka Rahkonen-
>         > >
>         > >
>         > >
>         > > --
>         > > Sent from:
>         http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
>         > > _______________________________________________
>         > > gdal-dev mailing list
>         > > gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>
>         > > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
>         -- 
>         Spatialys - Geospatial professional services
>         http://www.spatialys.com
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181101/ea8f8a5b/attachment-0001.html>