[Qgis-user] Convert huge shapefile to small size

[chris hermansen]

Scott and list,

With respect I think this conversation is moving off the point, which was
related to a 5gb shapefile

https://en.m.wikipedia.org/wiki/Shapefile

Which is beyond the design limits of the Shapefile standards and perhaps
therefore the source of the problem that the OP has.

Notably not a limitation of QGIS

On Thu, Jul 28, 2022, 10:45 Madry, Scott via Qgis-user <
qgis-user at lists.osgeo.org> wrote:

> Dear colleagues,
>
> As we move more into the world of big data, IOE, data mining, and data
> analytics, dealing with larger and larger data sets will become required,
> and, eventually, the norm.
>
> GRASS GIS has made very good progress in this regard, optimizing the code
> to run on massively parallel computer architectures, with very good effect:
>
> https://grasswiki.osgeo.org/wiki/Supercomputing
>
> Perhaps it is time for QGIS to consider this in its future development
> plans? We will certainly be needing this in the future.
>
> And thanks to all the team who keep the QGIS train running, I am
> constantly amazed at the energy, effort, and availability of people to the
> broader community.
>
> Scott
>
> Scott Madry, Ph.D.
> Research Associate Professor of Archaeology
> The University of North Carolina at Chapel Hill
>
> Tel         1-919-448-4493
> Email:    madrys at email.unc.edu <madrys at email.unc.edu>
> https://scottmadry.web.unc.edu
> Skype:   scott madry
>
>
>
> On Jul 28, 2022, at 10:14 AM, jhubbslist--- via Qgis-user <
> qgis-user at lists.osgeo.org> wrote:
>
> Past a point, trying to run a GIS app with very large data structures just
> doesn't work well. If you really need all the information you're carrying
> around for your analysis or whatever, you may need to reach for different
> tools or use the tools you have differently.
>
> I assume it's not so much the >5GiB of disk space that's the issue and
> that you've maxxed out the CPU, graphics, and disk I/O rate as much as is
> practical so mostly it's a matter of how long it takes maps etc. to paint
> onscreen. It may help you to move the heavy-lift onto PostgreSQL/PostGIS
> where you can make use of spatial indexing. Or, you can craft your
> operations the way you want them in QGIS but do the actual work with e.g.
> GDAL calls in Python. I spoke with someone a couple weeks ago whose
> particular GIS process worked better in GRASS than in QGIS, so that's
> something you might look into as well.
>
> On 7/28/22 7:52 AM, krishna Ayyala via Qgis-user wrote:
>
> dbf file itself is 5.1GB. Rest all other files are less than 500MB. It is
> the number of records which is huge. It has about 117,2100 points.
>
> On Thu, Jul 28, 2022 at 12:02 AM Bernd Vogelgesang via Qgis-user <
> qgis-user at lists.osgeo.org> wrote:
>
>> how big is the dbf file of that shape? Maybe you can also drop some
>> attributes.
>>
>> Am 28.07.22 um 03:46 schrieb krishna Ayyala via Qgis-user:
>> > Hello,
>> > I have a shapefile of 5GB in size. Is it possible to convert this
>> > shapefile to a smaller size file? It can be any format, not
>> > necessarily a shapefile. But, preferably a vector format. I tried to
>> > convert it into tiles, but that didn't work as it was losing the
>> > resolution. I am looking to convert this 5GB size file to about 500MB.
>> >
>> > Regards.
>> >
>> > _______________________________________________
>> > Qgis-user mailing list
>> > Qgis-user at lists.osgeo.org
>> > List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> > Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> _______________________________________________
>> Qgis-user mailing list
>> Qgis-user at lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>
>
> _______________________________________________
> Qgis-user mailing listQgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
>
> _______________________________________________
> Qgis-user mailing list
> Qgis-user at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20220728/fec72ec8/attachment.htm>