[gdal-dev] Ogr2ogr issue with big integers

[Even Rouault]

Moises,

the reason is that autodetection by default only considers the content 
of the first megabyte of the file, and in the first megabyte, only 
integer values that fit on 32 bit integers are present, hence this data 
type is selected. But later in the file, there are values that require a 
64 bit integer.

You can add -oo AUTODETECT_SIZE_LIMIT=0 to mean unlimited size limit. 
Unfortunately I just noticed this got broken in 3.5.1, where 
AUTODETECT_SIZE_LIMIT=0 would disable any autodetection. This will be 
fixed per https://github.com/OSGeo/gdal/pull/6087. So if you've 3.5.1, 
use AUTODETECT_SIZE_LIMIT=-1 instead. Or a large value in bytes not 
greater than 2 billion. In the previously mentioned fix, I've also added 
a warning if integer values that don't fit on 32 bit are set.

Even

Le 18/07/2022 à 15:54, Moises Calzado via gdal-dev a écrit :
> test.csv 
> <https://drive.google.com/file/d/11moLc5Shd78iTCKzsHxCKL33xSFHaVva/view?usp=drive_web>
> Hello everyone!
>
> I'm dealing with some issues with ogr2ogr trying to convert my data to 
> a CSV file guessing the data types. I'm using the following command:
>
> |ogr2ogr -f CSV -skipfailures -makevalid /vsistdout/ CSV:test.csv -oo 
> AUTODETECT_TYPE=YES -lcoCREATE_CSVT=YES > test.csv|
> When I add the AUTODETECT_TYPE=YES flag, ogr2ogr is applying some 
> weird transformations on the "uprn" column. If you check the generated 
> output and the original file, you'll notice that some 
> integers have changed into a negative number. Do you know if I'm doing 
> something wrong, or if it's a bug?
>
> Thanks so much for your help,
> Regards!
>
>
>
> -- 
> *Moises Calzado*
>
> Support Engineer
>
> (US) +1 917 463 3232 | (ES) +34 911 165 823 | mcalzado at carto.com
>
> <https://www.carto.com/>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220718/18f3e865/attachment.htm>