[Qgis-user] Import of Large CSV File

[Nicolas Cadieux]

Hi,

That can work up well to a million lines.  That does work well for smaller files. If the file come from Excel, it will have a tendency of adding extra empty columns in cells that have been erased. Notepad++ will get you farther and GVim even more but these are simple texte editor.  If the error something like a empty row or commas in a texte field that add row, you will most likely never find it.  That’s how I got into Python.  The simple ideas in Python is to count the row of each line in a comma separated or space sep file and to make sure each x and y fields are actually floats and to print out the first few problem line.That will find most errors In a LiDAR file anyway.

Nicolas

> Le 16 avr. 2019 à 08:43, <b.j.kobben at utwente.nl> <b.j.kobben at utwente.nl> a écrit :
> 
> simple test is also to check if the file can be imported in Excel or OpenOffice spreadsheet...
>  
>  
> --
> Barend Köbben
>  
>  
> On 16/04/2019, 14:11, "Qgis-user on behalf of Nicolas Cadieux" <qgis-user-bounces at lists.osgeo.org on behalf of nicolas.cadieux at archeotec.ca> wrote:
>  
> Hi,
>  
> If Even’s solution does not work, the file is probably problematic. You probably have missing or corrupted line.  I have seen crashes were comma separated lines are mixed with space separated lines, or with files or different row length. Did the file work in the past?
>  
> Solution would would be to look at each line and count the field with a python script.  I  could help if the file is really important.  If it was downloaded, try downloading the file again.  What are the symptoms? Crashes?  Can’t see all the line?  These are points I imagine?
>  
> Nicolas
>  
> 
> Le 16 avr. 2019 à 04:06, Grant Boxer <boxerg at iinet.net.au> a écrit :
> 
> I have a 2.5 GB csv file which contains 9 fields and 17,167,595 records. QGIS does not seem to be able to import this using the “Delimited Text” option. Any suggestions?
>  
> Regards Grant
>  
> Grant Boxer
> Perth, Western Australia
>  
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20190416/b13a8649/attachment.html>