[gdal-dev] Alternative to GPKG format

[koji higuchi]

I'm writing file using class ShapeConverter(osmium.SimpleHandler)  and ogr
(not fiona)
But, do not know whether transaction are all written instantaneously and
simultaneously or not...

On Thu, Nov 1, 2018 at 8:34 PM koji higuchi <koji51guchi at gmail.com> wrote:

> Hi  Andreas
> I'm writing to a file into my PC disk.
>
> On Thu, Nov 1, 2018 at 8:32 PM Andreas Neumann <a.neumann at carto.net>
> wrote:
>
>> Hi,
>>
>> Just a quick stupid question: are you writing to a local file system or
>> to a network filesystem. The latter can be substantially slower than the
>> former, esp. with Geopackage.
>>
>> Andreas
>> Am 01.11.18 um 12:10 schrieb koji higuchi:
>>
>> Hi Even
>> I managed to write gpkg without ogr without fiona using class
>> ShapeConverter(osmium.SimpleHandler).
>> still, speed seems slow.
>> Any idea to increase it up.
>>
>> On Thu, Nov 1, 2018 at 7:05 PM koji higuchi <koji51guchi at gmail.com>
>> wrote:
>>
>>> Hi Even
>>>
>>> I am trying to replace using fiona by pure ogr.
>>> but having difficult to collapse the code with class and objects.
>>> If i could extract geom and tags in a ogr pythonic way, then it be better
>>>
>>>
>>> On Thu, Nov 1, 2018 at 6:45 PM Even Rouault <even.rouault at spatialys.com>
>>> wrote:
>>>
>>>> You are perhaps using a too old version of Fiona. Recent versions
>>>> insert
>>>> features within OGR transactions. Should be fine with latest Fiona 1.8.0
>>>> See https://github.com/Toblerity/Fiona/issues/476
>>>>
>>>> > Hi, following is the code I used:
>>>> >
>>>> > Import os, osmium, fiona
>>>> >
>>>> > fi = 'europe-latest.osm.pbf'
>>>> > fo = 'europe-latest.dpkg'
>>>> >
>>>> > drv = 'DPKG'
>>>> >
>>>> > crs = {'no_defs': True, 'ellps': 'WGS84', 'datum': 'WGS84', 'proj':
>>>> > 'longlat'}
>>>> >
>>>> >  schema = {'geometry': 'LineString',
>>>> >                    'properties': {'id': 'float', 'name' : 'str',
>>>> 'kind' :
>>>> > 'str'}}
>>>> >
>>>> > outfile = fiona.open(fo, 'w', driver=drv, crs=crs, schema=schema)
>>>> >
>>>> > geomfab = osmium.geom.GeoJSONFactory()
>>>> >
>>>> > class ShapeConverter(osmium.SimpleHandler):
>>>> >       def way(self, w):
>>>> >            if 'place' in w.tags:
>>>> >                rec = {'geometry' : eval(geomfab.create_linestring(w)),
>>>> >                       'properties' : {'id' : float(w.id),
>>>> >                                      'name' : w.tags.get('name'),
>>>> >                                        'kind' : w.tags['place']}}
>>>> >                outfile.write(rec)
>>>> >
>>>> > ShapeConverter().apply_file(fi, locations=True)
>>>> >
>>>> > On Thu, Nov 1, 2018 at 4:10 PM jratike80 <
>>>> >
>>>> > jukka.rahkonen at maanmittauslaitos.fi> wrote:
>>>> > > koji higuchi wrote
>>>> > >
>>>> > > > Hi
>>>> > > > I am extracting .osm.pbf file into .gpkg; but the writing rate is
>>>> very
>>>> > > > slow.
>>>> > > > When I write into .shp, rate is faster but its limit is 4gb.
>>>> > > > So, what other format is better for larger than 4gb requirement?
>>>> > > > Thanks for your ideas.
>>>> > > > Koji
>>>> > >
>>>> > > Hi,
>>>> > >
>>>> > > Show the exact command that you are using so we can see if it could
>>>> be
>>>> > > made
>>>> > > faster. For example if you happen to use -skipfailures for handling
>>>> the
>>>> > > invalid geometries of the OSM data then you will make GDAL to do a
>>>> new
>>>> > > transaction for each row and that certainly is slow. But let's have
>>>> a look
>>>> > > at your command first.
>>>> > >
>>>> > > -Jukka Rahkonen-
>>>> > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > Sent from:
>>>> http://osgeo-org.1560.x6.nabble.com/GDAL-Dev-f3742093.html
>>>> > > _______________________________________________
>>>> > > gdal-dev mailing list
>>>> > > gdal-dev at lists.osgeo.org
>>>> > > https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>>>
>>>>
>>>> --
>>>> Spatialys - Geospatial professional services
>>>> http://www.spatialys.com
>>>>
>>>
>> _______________________________________________
>> gdal-dev mailing listgdal-dev at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20181101/ff15c71e/attachment-0001.html>