<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.Shkpostityyli19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.pl-k
{mso-style-name:pl-k;}
span.pl-c1
{mso-style-name:pl-c1;}
span.pl-c
{mso-style-name:pl-c;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FI" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-US">The error comes from <a href="https://github.com/OSGeo/gdal/blob/master/ogr/ogrsf_frmts/osm/ogrosmdatasource.cpp#L2067">
https://github.com/OSGeo/gdal/blob/master/ogr/ogrsf_frmts/osm/ogrosmdatasource.cpp#L2067</a> and it happens before your SQL, when GDAL is reading the data in from the huge planet.pbf file.<o:p></o:p></span></p>
<p class="MsoNormal"><span class="pl-k"><span lang="EN-US"><o:p> </o:p></span></span></p>
<p class="MsoNormal"><span class="pl-k">if</span>( nNextKeyIndex >= <span class="pl-c1">
32768</span> ) <span class="pl-c">/* somewhat arbitrary */<o:p></o:p></span></p>
<p class="MsoNormal"><span class="pl-c"><o:p> </o:p></span></p>
<p class="MsoNormal"><span class="pl-c"><span lang="EN-US">The error means that there are more than 32768 keys in the planet file. Maybe that hard coded limit could be enlarged but if you need for example 1 by 1 degree area I believe that there are much better
tools than GDAL for splitting a subset. </span></span><span lang="EN-US">I would recommend to
</span><span lang="EN-US">try for example osmosis <a href="https://wiki.openstreetmap.org/wiki/Osmosis/Examples#Breaking_OSM_file_into_several_bounding_boxes">
https://wiki.openstreetmap.org/wiki/Osmosis/Examples#Breaking_OSM_file_into_several_bounding_boxes</a> or osmconvert
<a href="https://wiki.openstreetmap.org/wiki/Osmconvert#Clipping_based_on_a_Polygon">
https://wiki.openstreetmap.org/wiki/Osmconvert#Clipping_based_on_a_Polygon</a>. The cropped .pbf file probably has less than 32768 distinct keys and GDAL can handle it. You would also save very much time.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">-Jukka Rahkonen-<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="mso-fareast-language:FI">Lähettäjä:</span></b><span lang="EN-US" style="mso-fareast-language:FI"> gdal-dev <gdal-dev-bounces@lists.osgeo.org>
<b>Puolesta </b>Schmetzer, Tobias<br>
<b>Lähetetty:</b> perjantai 13. toukokuuta 2022 10.47<br>
<b>Vastaanottaja:</b> gdal-dev@lists.osgeo.org<br>
<b>Aihe:</b> [gdal-dev] OSM extract: Too many different keys in file<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Dear GDAL dev team, <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I am not sure if I am following a wrong approach, if there is an issue with the osm driver, the distributed OSM file or if the error message is just ambiguous and could be improved.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I used ogr2ogr to select 12 keys to be extracted as polygons along with something around 40 conditions. The algorithm had worked well on a tiny OSM file with the city of Munich so tested it I on a small sample area of
1°x1° on the global planet OSM file:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">ogr2ogr -spat 10 45 11 46 -f gpkg c:\daten\osm_planet\1x1.gpkg c:\daten\osm_planet\planet-220502.osm.pbf multipolygons -select "name,aeroway,amenity,building,historic,landuse,leisure,military,office,tourism,shop,landuse
" -where @ogr2ogr_condition.txt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">The first 70% were reached after one hour but then the process slowed down and after 19 hours I got an error message:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">0...10...20...30...40...50...60...70...80...90.ERROR 1: Too many different keys in file<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">If this is because one or more features exceed the maximum amount of doable keys, is the officially by OSM distributed file wrong or too large to be processed by ogr2ogr or what's the matter? I tried to read the relevant
source code file where the error message occurs but it's too cryptic to me.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Content of ogr2ogr_condition.txt for the sake of completeness:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">historic is null and <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> office is not null or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> building='hotel' or <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> building='hospital' or <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> building='apartments' or <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> building='barracks' or <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> building='dormitory' or <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> building='warehouse' or <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> building='monastery' or <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> building='public' or <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> building='hangar' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> tourism='guest_house' or <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> tourism='apartment' or <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> tourism='hostel' or <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> tourism='museum' or <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> tourism='gallery' or <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> tourism='motel' or <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> tourism='hotel' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> amenity='university' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> amenity='research_institute' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> amenity='social_facility' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> amenity='school' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> amenity='kindergarten' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> amenity='kindergarden' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> amenity='exhibition centre' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> amenity='student_accommodation' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> amenity='library' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> amenity='clinic' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> amenity='hospital' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> </span><span lang="DE">amenity='public_building' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE"> </span><span lang="EN-US">amenity='concert_hall' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> amenity='prison' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> amenity='theatre' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> amenity='courthouse' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> aeroway='terminal' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> shop='mall' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> military='base' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> military='barracks' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> military='office' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> landuse='education' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> landuse='commercial' or<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> landuse='industrial'<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">I’d be grateful for any hints and glad to contribute to any error message improval if indicated.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">Kind regards, Tobias Schmetzer<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">ZAE Bayern<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">Tobias Schmetzer, Dipl. Ing.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">Wissenschaftlicher Mitarbeiter Systementwicklung
<span style="color:black">| </span>Scientific Staff Member Systems Engineering<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">Bereich Energiespeicherung| Division Energy Storage<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">Walther-Meißner-Str.
</span><span lang="EN-US" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">85748 Garching<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">Tel.: +49 89 329442-65<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">Fax: +49 89 329442-12<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE"><a href="mailto:tobias.schmetzer@zae-bayern.de"><span style="color:blue">tobias.schmetzer@zae-bayern.de</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE"><a href="https://eur06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.zae-bayern.de%2F&data=05%7C01%7Cjukka.rahkonen%40maanmittauslaitos.fi%7Cb03bc6c9f5b542ed51ff08da34b64dd1%7Cc4f8a63255804a1c92371d5a571b71fa%7C0%7C1%7C637880254860879658%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000%7C%7C%7C&sdata=xkCacr5vK0eKSDXGyhCzWrhN5ckc%2BYNEVWPcChPcs2Y%3D&reserved=0"><span style="color:blue">http://www.zae-bayern.de</span></a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">ZAE Bayern - Bayerisches Zentrum für Angewandte Energieforschung e. V.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">Vorstand/Board:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">Prof. Dr. Hartmut Spliethoff (Vorsitzender/Chairman),<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">Prof. Dr. Vladimir Dyakonov<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">Sitz/Registered Office: Würzburg<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">Registergericht/Register Court: Amtsgericht Würzburg<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">Registernummer/Register Number: VR 1386<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">Sämtliche Willenserklärungen, z. B. Angebote, Aufträge, Anträge und Verträge, sind für das ZAE Bayern nur in schriftlicher und ordnungsgemäß
unterschriebener Form rechtsverbindlich. Diese E-Mail ist ausschließlich zur Nutzung durch den/die vorgenannten Empfänger bestimmt. Jegliche unbefugte Offenbarung, Nutzung oder Verbreitung, sei es insgesamt oder teilweise, ist untersagt. Sollten Sie diese
E-Mail irrtümlich erhalten haben, benachrichtigen Sie bitte unverzüglich den Absender und löschen Sie diese E-Mail.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="DE" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:8.0pt;font-family:"Arial",sans-serif;mso-fareast-language:DE">Any declarations of intent, such as quotations, orders, applications and contracts, are legally binding for ZAE Bayern only if expressed in
a written and duly signed form. This e-mail is intended solely for use by the recipient(s) named above. Any unauthorised disclosure, use or dissemination, whether in whole or in part, is prohibited. If you have received this e-mail in error, please notify
the sender immediately and delete this e-mail.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</body>
</html>