[gdal-dev] Alternative to GPKG format

[koji higuchi]

Hi  Andreas
I'm writing to a file into my PC disk.

On Thu, Nov 1, 2018 at 8:32 PM Andreas Neumann <a.neumann at carto.net> wrote:

> Hi,
>
> Just a quick stupid question: are you writing to a local file system or to
> a network filesystem. The latter can be substantially slower than the
> former, esp. with Geopackage.
>
> Andreas
> Am 01.11.18 um 12:10 schrieb koji higuchi:
>
> Hi Even
> I managed to write gpkg without ogr without fiona using class
> ShapeConverter(osmium.SimpleHandler).
> still, speed seems slow.
> Any idea to increase it up.
>
> On Thu, Nov 1, 2018 at 7:05 PM koji higuchi <koji51guchi at gmail.com> wrote:
>
>> Hi Even
>>
>> I am trying to replace using fiona by pure ogr.
>> but having difficult to collapse the code with class and objects.
>> If i could extract geom and tags in a ogr pythonic way, then it be better
>>
>>
>> On Thu, Nov 1, 2018 at 6:45 PM Even Rouault <even.rouault at spatialys.com>
>> wrote:
>>
>>> You are perhaps using a too old version of Fiona. Recent versions insert
>>> features within OGR transactions. Should be fine with latest Fiona 1.8.0
>>> See https://github.com/Toblerity/Fiona/issues/476
>>>
>>> > Hi, following is the code I used:
>>> >
>>> > Import os, osmium, fiona
>>> >
>>> > fi = 'europe-latest.osm.pbf'
>>> > fo = 'europe-latest.dpkg'
>>> >
>>> > drv = 'DPKG'
>>> >
>>> > crs = {'no_defs': True, 'ellps': 'WGS84', 'datum': 'WGS84', 'proj':
>>> > 'longlat'}
>>> >
>>> >  schema = {'geometry': 'LineString',
>>> >                    'properties': {'id': 'float', 'name' : 'str',
>>> 'kind' :
>>> > 'str'}}
>>> >
>>> > outfile = fiona.open(fo, 'w', driver=drv, crs=crs, schema=schema)
>>> >
>>> > geomfab = osmium.geom.GeoJSONFactory()
>>> >
>>> > class ShapeConverter(osmium.SimpleHandler):
>>> >       def way(self, w):
>>> >            if 'place' in w.tags:
>>> >                rec = {'geometry' : eval(geomfab.create_linestring(w)),
>>> >                       'properties' : {'id' : float(w.id),
>>> >                                      'name' : w.tags.get('name'),
>>> >                                        'kind' : w.tags['place']}}
>>> >                outfile.write(rec)
>>> >
>>> > ShapeConverter().apply_file(fi, locations=True)
>>> >
>>> > On Thu, Nov 1, 2018 at 4:10 PM jratike80 <
>>> >
>>> > jukka.rahkonen at maanmittauslaitos.fi> wrote:
>>> > > koji higuchi wrote
>>> > >
>>> > > > Hi
>>> > > > I am extracting .osm.pbf file into .gpkg; but the writing rate is
>>> very
>>> > > > slow.
>>> > > > When I write into .shp, rate is faster but its limit is 4gb.
>>> > > > So, what other format is better for larger than 4gb requirement?
>>> > > > Thanks for your ideas.
>>> > > > Koji
>>> > >
>>> > > Hi,
>>> > >
>>> > > Show the exact command that you are using so we can see if it could
>>> be
>>> > > made
>>> > > faster. For example if you happen to use -skipfailures for handling
>>> the
>>> > > invalid geometries of the OSM data then you will make GDAL to do a
>>> new
>>> > > transaction for each row and that certainly is slow. But let's have
>>> a look
>>> > > at your command first.
>>> > >
>>> > > -Jukka Rahkonen-
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Sent from:
>>> http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
>>> > > _______________________________________________
>>> > > gdal-dev mailing list
>>> > > gdal-dev at lists.osgeo.org
>>> > > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>
>>>
>>> --
>>> Spatialys - Geospatial professional services
>>> http://www.spatialys.com
>>>
>>
> _______________________________________________
> gdal-dev mailing listgdal-dev at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181101/c14603d9/attachment.html>