[gdal-dev] Ogrinfo , ogr2ogr file size limits
Even Rouault
even.rouault at spatialys.com
Tue May 17 03:11:19 PDT 2022
Ah of course. And the ILI2 driver does ingest the whole file into memory
ili2c is not used by the driver. It is just a preparation step to create
the .imd file. So that wouldn't help here. But your alternatives are
good hints. Could be worth to extend
https://gdal.org/drivers/vector/ili.html to mention them, especially if
they can process files of arbitrary files with modest RAM usage
Even
Le 17/05/2022 à 11:58, Andreas Neumann a écrit :
>
> Hi,
>
> Most likely, Frando is talking about the "Interlis" format - a format
> common in Switzerland (and also Colombia, by export from Switzerland).
>
> see https://gdal.org/drivers/vector/ili.html
>
> Since this involves Java ili2c, you should set the corresponding XmX
> memory setting for the Java compiler, which is memory limited by default.
>
> There is also an alternative XTF import export tool available, called
> il2db (ili2pg, ili2ora, ili2gpkg, ili2fgdb). See
> https://github.com/claeis/ili2db/blob/master/docs/ili2db.rst
>
> I generally recommend to use ili2db versions rather than the ogr
> interlis drive. The ili2db is way more tested and flexible with loads
> of options.
>
> Greetings,
>
> Andreas
>
> On 2022-05-17 11:53, Joaquim Manuel Freire Luís wrote:
>
>> xtf is the extension by a format of sidescan sonar data.
>>
>> This one was picked randomly from a google search
>>
>> https://www.usna.edu/Users/oceano/pguth/md_help/html/sidescan.htm
>>
>> *From:*gdal-dev <gdal-dev-bounces at lists.osgeo.org> *On Behalf Of
>> *Even Rouault
>> *Sent:* Tuesday, May 17, 2022 10:43 AM
>> *To:* Gonzalez Velasquez, Frando Alexis
>> <fagonzalezv at indracompany.com>; gdal-dev at lists.osgeo.org
>> *Cc:* Serrano Granados, Yency <yserrano at indracompany.com>
>> *Subject:* Re: [gdal-dev] Ogrinfo , ogr2ogr file size limits
>>
>> Frando,
>>
>> The .xtf extension doens't ring a bell to me. Which OGR driver
>> recognizes such file ? (remove the -q option and look at the
>> beginning of the report of ogrinfo), and what is your GDAL version ?
>>
>> Most OGR drivers work in streaming mode and can handle arbitrary big
>> files, but a few ones have no other choice than ingesting the whole
>> file, in which case there's no workaround apart splitting the file
>> with other tools or buy more RAM
>>
>> Even
>>
>> Le 17/05/2022 à 04:21, Gonzalez Velasquez, Frando Alexis a écrit :
>>
>> Hi dev team!
>>
>> I am writing to inquire about big files problem,
>>
>> We are using the ogrinfo and ogr2ogr executables over small files
>> with success (1 gb max) but with 10 gb files we are experiencing
>> memory problems and and it does not work, is there a file size
>> limit? And if there is a size limit, is there any strategy to
>> handle this problema and process big files with success ?
>>
>> For example:
>>
>> */ogrinfo -q -so C:\\tmp\\file.xtf /*
>>
>> When file is 1Gb success ; but when the file is 10gb size the
>> process fails.
>>
>> Any advice would be appreciated, thanks.
>>
>> minsait-indra
>>
>> FRANDO GONZÁLEZ
>>
>> Senior Systems Engineer
>>
>> fagonzalezv at indracompany.com <mailto:fagonzalezv at indracompany.com>
>>
>> Calle 93 16 25
>>
>> Bogotá, Colombia
>>
>> T +57 1 646 36 00
>>
>> M +57 3006604908
>>
>> minsait.com
>> <https://eur04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.minsait.com%2F&data=04%7C01%7Cfagonzalezv%40indracompany.com%7C6c419c97a7134841c0da08d9f7ce4282%7C7808e00514894374954bd3b08f193920%7C0%7C0%7C637813287514711623%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=0Jr%2BkI3f3Uh3g%2FG5C5yElQX9pHauhEi9ds6w%2FPOxLtM%3D&reserved=0>
>>
>> indracompany.com
>> <https://eur04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.indracompany.com%2F&data=04%7C01%7Cfagonzalezv%40indracompany.com%7C6c419c97a7134841c0da08d9f7ce4282%7C7808e00514894374954bd3b08f193920%7C0%7C0%7C637813287514711623%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=W2xsw4RMlZ8TUymKMTE5zqosvKN2LV5QhwvGmAxgZM0%3D&reserved=0>
>>
>> ------------------------------------------------------------------------
>>
>>
>> Este correo electrónico y, en su caso, cualquier fichero anexo al
>> mismo, contiene información de carácter confidencial
>> exclusivamente dirigida a su destinatario o destinatarios. Si no
>> es vd. el destinatario indicado, queda notificado que la lectura,
>> utilización, divulgación y/o copia sin autorización está
>> prohibida en virtud de la legislación vigente. En el caso de
>> haber recibido este correo electrónico por error, se ruega
>> notificar inmediatamente esta circunstancia mediante reenvío a la
>> dirección electrónica del remitente.
>> Evite imprimir este mensaje si no es estrictamente necesario.
>>
>> This email and any file attached to it (when applicable)
>> contain(s) confidential information that is exclusively addressed
>> to its recipient(s). If you are not the indicated recipient, you
>> are informed that reading, using, disseminating and/or copying it
>> without authorisation is forbidden in accordance with the
>> legislation in effect. If you have received this email by
>> mistake, please immediately notify the sender of the situation by
>> resending it to their email address.
>> Avoid printing this message if it is not absolutely necessary.
>>
>>
>> _______________________________________________
>>
>> gdal-dev mailing list
>>
>> gdal-dev at lists.osgeo.org
>>
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>> --
>> http://www.spatialys.com
>> My software is free, but my time generally not.
>>
>> _______________________________________________
>> gdal-dev mailing list
>> gdal-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
--
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220517/2c863fc5/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 1476 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220517/2c863fc5/attachment-0001.png>
More information about the gdal-dev
mailing list