[Qgis-user] WG: Convert huge shapefile to small size

Fri Jul 29 03:46:34 PDT 2022

Hi,

maybe I'm wrong, but isn't the maximum file size for the dbf format 2 GB? Obviously it is possible to put more data in it (with ESRI software ?), but this may lead to problems if the file is used with other software.

Cheers,
Michael

Michael Reetz

*****************************************
Michael Reetz
Datenbanken und Anwendungsentwicklung
Nationalpark-Verwaltung Niedersächsisches Wattenmeer Virchowstr. 1 | 26382 Wilhelmshaven
Tel.: +49 (0) 4421 911 291
e-mail: michael.reetz at nlpvw.niedersachsen.de
Web: https://www.nationalpark-wattenmeer.de 

-----Ursprüngliche Nachricht-----
Von: Qgis-user <qgis-user-bounces at lists.osgeo.org> Im Auftrag von Nyall Dawson via Qgis-user
Gesendet: Freitag, 29. Juli 2022 03:33
An: Madry, Scott <madrys at email.unc.edu>
Cc: Qgis Users List <qgis-user at lists.osgeo.org>; Tim Sutton <tim at linfiniti.com>
Betreff: Re: [Qgis-user] Convert huge shapefile to small size

ACHTUNG!! Diese E-Mail erreicht Sie von einem Absender außerhalb der niedersächsischen Landesverwaltungs-Infrastruktur mit TLS-Verschlüsselung. Bitte klicken Sie auf keine Links oder öffnen Sie keine E-Mail-Anhänge, falls Sie den Absender nicht kennen und nicht wissen, ob der Inhalt sicher ist.

On Fri, 29 Jul 2022 at 06:22, Madry, Scott via Qgis-user <qgis-user at lists.osgeo.org> wrote:
>
> Thanks Chris, but I do think that this is a topic is worth considering by the QGIS community, beyond the limitations of the archaeic shapefile format. We are moving into an era of very big data, both raster and vector, and QGIS should be able to meet this challenge. This will require some foresight and planning, thus my raising the issue. Or not...

In truth QGIS is already well prepared for handling massive datasets.
(E.g. you'll see in recent versions how well it handles mammoth point cloud formats!). Sure, there's further improvements which can be made, but they're evolutionary rather than revolutionary!

Nyall

>
> Best,
>
> Scott
>
> On Jul 28, 2022, at 4:05 PM, chris hermansen <clhermansen at gmail.com> wrote:
>
> Scott and list,
>
> With respect I think this conversation is moving off the point, which 
> was related to a 5gb shapefile
>
> https://en.m.wikipedia.org/wiki/Shapefile
>
> Which is beyond the design limits of the Shapefile standards and perhaps therefore the source of the problem that the OP has.
>
> Notably not a limitation of QGIS
>
> On Thu, Jul 28, 2022, 10:45 Madry, Scott via Qgis-user <qgis-user at lists.osgeo.org> wrote:
>>
>> Dear colleagues,
>>
>> As we move more into the world of big data, IOE, data mining, and data analytics, dealing with larger and larger data sets will become required, and, eventually, the norm.
>>
>> GRASS GIS has made very good progress in this regard, optimizing the code to run on massively parallel computer architectures, with very good effect:
>>
>> https://grasswiki.osgeo.org/wiki/Supercomputing
>>
>> Perhaps it is time for QGIS to consider this in its future development plans? We will certainly be needing this in the future.
>>
>> And thanks to all the team who keep the QGIS train running, I am constantly amazed at the energy, effort, and availability of people to the broader community.
>>
>> Scott
>>
>> Scott Madry, Ph.D.
>> Research Associate Professor of Archaeology The University of North 
>> Carolina at Chapel Hill
>>
>> Tel         1-919-448-4493
>> Email:    madrys at email.unc.edu
>> https://scottmadry.web.unc.edu
>> Skype:   scott madry
>>
>>
>>
>> On Jul 28, 2022, at 10:14 AM, jhubbslist--- via Qgis-user <qgis-user at lists.osgeo.org> wrote:
>>
>> Past a point, trying to run a GIS app with very large data structures just doesn't work well. If you really need all the information you're carrying around for your analysis or whatever, you may need to reach for different tools or use the tools you have differently.
>>
>> I assume it's not so much the >5GiB of disk space that's the issue and that you've maxxed out the CPU, graphics, and disk I/O rate as much as is practical so mostly it's a matter of how long it takes maps etc. to paint onscreen. It may help you to move the heavy-lift onto PostgreSQL/PostGIS where you can make use of spatial indexing. Or, you can craft your operations the way you want them in QGIS but do the actual work with e.g. GDAL calls in Python. I spoke with someone a couple weeks ago whose particular GIS process worked better in GRASS than in QGIS, so that's something you might look into as well.
>>
>> On 7/28/22 7:52 AM, krishna Ayyala via Qgis-user wrote:
>>
>> dbf file itself is 5.1GB. Rest all other files are less than 500MB. It is the number of records which is huge. It has about 117,2100 points.
>>
>> On Thu, Jul 28, 2022 at 12:02 AM Bernd Vogelgesang via Qgis-user <qgis-user at lists.osgeo.org> wrote:
>>>
>>> how big is the dbf file of that shape? Maybe you can also drop some 
>>> attributes.
>>>
>>> Am 28.07.22 um 03:46 schrieb krishna Ayyala via Qgis-user:
>>> > Hello,
>>> > I have a shapefile of 5GB in size. Is it possible to convert this 
>>> > shapefile to a smaller size file? It can be any format, not 
>>> > necessarily a shapefile. But, preferably a vector format. I tried 
>>> > to convert it into tiles, but that didn't work as it was losing 
>>> > the resolution. I am looking to convert this 5GB size file to about 500MB.
>>> >
>>> > Regards.
>>> >
>>> > _______________________________________________
>>> > Qgis-user mailing list
>>> > Qgis-user at lists.osgeo.org
>>> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>> _______________________________________________
>>> Qgis-user mailing list
>>> Qgis-user at lists.osgeo.org
>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>
>>
>> _______________________________________________
>> Qgis-user mailing list
>> Qgis-user at lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>
>>
>> _______________________________________________
>> Qgis-user mailing list
>> Qgis-user at lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>
>>
>> _______________________________________________
>> Qgis-user mailing list
>> Qgis-user at lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
_______________________________________________
Qgis-user mailing list
Qgis-user at lists.osgeo.org
List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user