[gdal-dev] Ogr2ogr: disk I/O errors on external hard drive

[Schmetzer, Tobias]

Hello,

I get a lot of disk I/O errors using exec commands like:
"C:\OSGeo4W\bin\ogr2ogr -f gpkg E:\OSM\vektor_gpkg\Kontinente\europe-latest.osm.noAuthor_-44-46.gpkg E:\OSM\Kontinente\europe-latest.osm.noAuthor_-44-46.pbf multipolygons -select "name,aeroway,amenity,building,historic,landuse,office,tourism,shop,landuse" -where @ogr2ogr_condition.txt"

The error message is very generic. The external hard drive is of good quality and brand new formatted with NTFS so I wouldn't think of a damaged HDD.
Sometimes I did simultaneous processing with ogr2ogr but never with the same source or output files.

I had set up a temporary directory on a large external hard drive that is connected via USB3 using "set temp=E:\OSM\temp" to cover for very large temporary files during ogr2ogr osm processing.
Might the reason be that during processing the algorithm outputs data faster than the external HDD is able to write it so I shouldn't set the external HDD as a temp directory?

These are the errors that occurred:

ERROR 1: failed to execute insert : disk I/O error
ERROR 1: Unable to write feature 617369107 from layer multipolygons.
ERROR 1: Terminating translation prematurely after failed
translation of layer multipolygons (use -skipfailures to skip errors)
ERROR 1: sqlite3_exec(CREATE TABLE gpkg_extensions (table_name TEXT,column_name TEXT,extension_name TEXT NOT NULL,definition TEXT NOT NULL,scope TEXT NOT NULL,CONSTRAINT ge_tce UNIQUE (table_name, column_name, extension_name))) failed: disk I/O error
ERROR 1: sqlite3_exec(UPDATE gpkg_contents SET min_x = -31.2625834, min_y = 32.6335129, max_x = 46.66876959999999, max_y = 45.99999699999999 WHERE lower(table_name) = lower('multipolygons') AND Lower(data_type) = 'features') failed: disk I/O error
ERROR 1: sqlite3_exec(UPDATE gpkg_contents SET last_change = strftime('%Y-%m-%dT%H:%M:%fZ','now') WHERE lower(table_name) = lower('multipolygons')) failed: disk I/O error
ERROR 1: sqlite3_exec(ROLLBACK) failed: cannot rollback - no transaction is active
...

ERROR 1: Cannot read node 5100255009
ERROR 1: Cannot read node 5100255010
ERROR 1: Cannot read node 5100255011
ERROR 1: Cannot read node 5100255020
ERROR 1: Cannot read node 5100255044
ERROR 1: Cannot read node 5100255045
...

0...10...20...30...40...50...60.ERROR 1: Failed inserting way 11064599: disk I/O error
ERROR 1: Failed inserting way 11064601: disk I/O error
ERROR 1: Failed inserting way 11064602: disk I/O error
ERROR 1: Failed inserting way 11064603: disk I/O error
ERROR 1: Failed inserting way 11064605: disk I/O error
...


Any ideas?



Regards, Tobias Schmetzer

ZAE Bayern
Tobias Schmetzer, Dipl. Ing.
Wissenschaftlicher Mitarbeiter Systementwicklung | Scientific Staff Member Systems Engineering
Bereich Energiespeicherung| Division Energy Storage

Walther-Meißner-Str. 6
85748 Garching

Tel.: +49 89 329442-65
Fax: +49 89 329442-12
tobias.schmetzer at zae-bayern.de<mailto:tobias.schmetzer at zae-bayern.de>
http://www.zae-bayern.de<http://www.zae-bayern.de/>


ZAE Bayern - Bayerisches Zentrum für Angewandte Energieforschung e. V.
Vorstand | Board: Dr. Andreas Hauer (Vorsitzender | Chairman), Dr. Hans-Peter Ebert
Sitz/Registered Office: Würzburg
Registergericht/Register Court: Amtsgericht Würzburg
Registernummer/Register Number: VR 1386

Sämtliche Willenserklärungen, z. B. Angebote, Aufträge, Anträge und Verträge, sind für das ZAE Bayern nur in schriftlicher und ordnungsgemäß unterschriebener Form rechtsverbindlich. Diese E-Mail ist ausschließlich zur Nutzung durch den/die vorgenannten Empfänger bestimmt. Jegliche unbefugte Offenbarung, Nutzung oder Verbreitung, sei es insgesamt oder teilweise, ist untersagt. Sollten Sie diese E-Mail irrtümlich erhalten haben, benachrichtigen Sie bitte unverzüglich den Absender und löschen Sie diese E-Mail.

Any declarations of intent, such as quotations, orders, applications and contracts, are legally binding for ZAE Bayern only if expressed in a written and duly signed form. This e-mail is intended solely for use by the recipient(s) named above. Any unauthorised disclosure, use or dissemination, whether in whole or in part, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete this e-mail.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220520/9f141700/attachment.htm>