<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Thanks Chris, but I do think that this is a topic is worth considering by the QGIS community, beyond the limitations of the archaeic shapefile format. We are moving into an era of
<b class="">very</b> big data, both raster and vector, and QGIS should be able to meet this challenge. This will require some foresight and planning, thus my raising the issue. Or not...
<div class=""><br class="">
</div>
<div class="">Best,<br class="">
<div class=""><br class="">
</div>
<div class="">Scott<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jul 28, 2022, at 4:05 PM, chris hermansen <<a href="mailto:clhermansen@gmail.com" class="">clhermansen@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="auto" class="">Scott and list,
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">With respect I think this conversation is moving off the point, which was related to a 5gb shapefile</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class=""><a href="https://en.m.wikipedia.org/wiki/Shapefile" class="">https://en.m.wikipedia.org/wiki/Shapefile</a></div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">Which is beyond the design limits of the Shapefile standards and perhaps therefore the source of the problem that the OP has.</div>
<div dir="auto" class=""><br class="">
</div>
<div dir="auto" class="">Notably not a limitation of QGIS</div>
</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Jul 28, 2022, 10:45 Madry, Scott via Qgis-user <<a href="mailto:qgis-user@lists.osgeo.org" class="">qgis-user@lists.osgeo.org</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;line-break:after-white-space" class="">Dear colleagues,
<div class=""><br class="">
</div>
<div class="">As we move more into the world of big data, IOE, data mining, and data analytics, dealing with larger and larger data sets will become required, and, eventually, the norm.</div>
<div class=""><br class="">
</div>
<div class="">GRASS GIS has made very good progress in this regard, optimizing the code to run on massively parallel computer architectures, with very good effect:</div>
<div class=""><br class="">
</div>
<div class=""><a href="https://grasswiki.osgeo.org/wiki/Supercomputing" target="_blank" rel="noreferrer" class="">https://grasswiki.osgeo.org/wiki/Supercomputing</a></div>
<div class=""><br class="">
</div>
<div class="">Perhaps it is time for QGIS to consider this in its future development plans? We will certainly be needing this in the future.</div>
<div class=""><br class="">
</div>
<div class="">And thanks to all the team who keep the QGIS train running, I am constantly amazed at the energy, effort, and availability of people to the broader community.</div>
<div class=""><br class="">
</div>
<div class="">Scott</div>
<div class=""><br class="">
</div>
<div class=""><span style="border-collapse:separate;font-variant-ligatures:normal;font-variant-numeric:normal;font-variant-alternates:normal;font-variant-east-asian:normal;line-height:normal;border-spacing:0px" class="">
<div style="word-wrap:break-word;line-break:after-white-space" class="">
<div class="">
<div class=""><font size="3" class=""><span style="font-size:12px" class="">Scott Madry, Ph.D.</span></font></div>
<div class=""><font size="3" class=""><span style="font-size:12px" class="">Research Associate Professor of Archaeology</span></font></div>
<div class=""><font size="3" class=""><span style="font-size:12px" class="">The University of North Carolina at Chapel Hill</span></font></div>
<div class=""><br class="">
</div>
<div class=""><font size="3" class=""><span style="font-size:12px" class="">Tel 1-919-448-4493</span></font></div>
<div class=""><font size="3" class=""><span style="font-size:12px" class=""><a href="mailto:madrys@email.unc.edu" target="_blank" rel="noreferrer" class="">Email: madrys@email.unc.edu</a></span></font></div>
<div class=""><span style="font-size:12px" class=""><a href="https://scottmadry.web.unc.edu/" target="_blank" rel="noreferrer" class="">https://scottmadry.web.unc.edu</a></span></div>
<div class=""><font size="3" class=""><span style="font-size:12px" class="">Skype: scott madry</span></font></div>
</div>
<div class=""><br class="">
</div>
</div>
</span><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jul 28, 2022, at 10:14 AM, jhubbslist--- via Qgis-user <<a href="mailto:qgis-user@lists.osgeo.org" target="_blank" rel="noreferrer" class="">qgis-user@lists.osgeo.org</a>> wrote:</div>
<br class="">
<div class="">
<div class="">
<div class="">Past a point, trying to run a GIS app with very large data structures just doesn't work well. If you really need all the information you're carrying around for your analysis or whatever, you may need to reach for different tools or use the tools
you have differently. <br class="">
</div>
<div class=""><br class="">
</div>
<div class="">I assume it's not so much the >5GiB of disk space that's the issue and that you've maxxed out the CPU, graphics, and disk I/O rate as much as is practical so mostly it's a matter of how long it takes maps etc. to paint onscreen. It may help you
to move the heavy-lift onto PostgreSQL/PostGIS where you can make use of spatial indexing. Or, you can craft your operations the way you want them in QGIS but do the actual work with e.g. GDAL calls in Python. I spoke with someone a couple weeks ago whose
particular GIS process worked better in GRASS than in QGIS, so that's something you might look into as well.<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">On 7/28/22 7:52 AM, krishna Ayyala via Qgis-user wrote:<br class="">
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">dbf file itself is 5.1GB. Rest all other files are less than 500MB. It is the number of records which is huge. It has about 117,2100 points.</div>
<br class="">
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Jul 28, 2022 at 12:02 AM Bernd Vogelgesang via Qgis-user <<a href="mailto:qgis-user@lists.osgeo.org" target="_blank" rel="noreferrer" class="">qgis-user@lists.osgeo.org</a>> wrote:<br class="">
</div>
<blockquote class="gmail_quote">how big is the dbf file of that shape? Maybe you can also drop some<br class="">
attributes.<br class="">
<br class="">
Am 28.07.22 um 03:46 schrieb krishna Ayyala via Qgis-user:<br class="">
> Hello,<br class="">
> I have a shapefile of 5GB in size. Is it possible to convert this<br class="">
> shapefile to a smaller size file? It can be any format, not<br class="">
> necessarily a shapefile. But, preferably a vector format. I tried to<br class="">
> convert it into tiles, but that didn't work as it was losing the<br class="">
> resolution. I am looking to convert this 5GB size file to about 500MB.<br class="">
><br class="">
> Regards.<br class="">
><br class="">
> _______________________________________________<br class="">
> Qgis-user mailing list<br class="">
> <a href="mailto:Qgis-user@lists.osgeo.org" target="_blank" rel="noreferrer" class="">
Qgis-user@lists.osgeo.org</a><br class="">
> List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer noreferrer" target="_blank" class="">
https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br class="">
> Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer noreferrer" target="_blank" class="">
https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br class="">
_______________________________________________<br class="">
Qgis-user mailing list<br class="">
<a href="mailto:Qgis-user@lists.osgeo.org" target="_blank" rel="noreferrer" class="">Qgis-user@lists.osgeo.org</a><br class="">
List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer noreferrer" target="_blank" class="">
https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br class="">
Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer noreferrer" target="_blank" class="">
https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br class="">
</blockquote>
</div>
<br class="">
<fieldset class=""></fieldset>
<pre class="">_______________________________________________
Qgis-user mailing list
<a href="mailto:Qgis-user@lists.osgeo.org" target="_blank" rel="noreferrer" class="">Qgis-user@lists.osgeo.org</a>
List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank" rel="noreferrer" class="">https://lists.osgeo.org/mailman/listinfo/qgis-user</a>
Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank" rel="noreferrer" class="">https://lists.osgeo.org/mailman/listinfo/qgis-user</a>
</pre>
</blockquote>
<p class=""><br class="">
</p>
</div>
_______________________________________________<br class="">
Qgis-user mailing list<br class="">
<a href="mailto:Qgis-user@lists.osgeo.org" target="_blank" rel="noreferrer" class="">Qgis-user@lists.osgeo.org</a><br class="">
List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank" rel="noreferrer" class="">
https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br class="">
Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" target="_blank" rel="noreferrer" class="">
https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br class="">
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
_______________________________________________<br class="">
Qgis-user mailing list<br class="">
<a href="mailto:Qgis-user@lists.osgeo.org" target="_blank" rel="noreferrer" class="">Qgis-user@lists.osgeo.org</a><br class="">
List info: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer noreferrer" target="_blank" class="">
https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br class="">
Unsubscribe: <a href="https://lists.osgeo.org/mailman/listinfo/qgis-user" rel="noreferrer noreferrer" target="_blank" class="">
https://lists.osgeo.org/mailman/listinfo/qgis-user</a><br class="">
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</body>
</html>