<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi,</p>
<p>See below.<br>
</p>
<div class="moz-cite-prefix">On 2020-12-10 5:36 p.m., Francesco
Pelullo wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAH6xWviNKnSg+RaQT=uZrd5GSfwLBmRFfYxX93K1XgkNbVmZ0w@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="auto">
<div><br>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Il gio 10 dic 2020, 16:50
Nicolas Cadieux <<a
href="mailto:njacadieux.gitlab@gmail.com"
target="_blank" rel="noreferrer" moz-do-not-send="true">njacadieux.gitlab@gmail.com</a>>
ha scritto:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">Hi,
<div>I also used txt files to load LiDAR points in QGIS
because .shp was not good. This was before .gpkg came
along, before I learned Python or CloudCompare. I am
just starting to familiarize myself with the
geopackage format so i am no .gpkg expert but this is
what I would do or ask myself:</div>
<div><br>
</div>
<div>How are the files stored? Server, USB stick,
hard-drive, SSD? </div>
</div>
</blockquote>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">They are usually stored in a SMB server,
accessible via VPN because this Company has multiple offices.
However, due to the really slowly data access, projects and
data are moved in local SSD disk before access/editing.</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div><br>
</div>
<div>Are the files in the same CRS as the project? If
not, every thing may be read in cache and reprojected
even before you start...</div>
</div>
</blockquote>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Yes, projects and data layers are created with
same CRS.</div>
<div dir="auto"><br>
</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div><br>
</div>
<div>What happens when the files are simplified? Try
more files and less layer, try more points in the same
files.</div>
</div>
</blockquote>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">This could be easy to do, i will check on next
job.</div>
<div dir="auto"><br>
</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div> Try a fresh project an open the layers one by
one. Is one more problematic than others? </div>
</div>
</blockquote>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">No, It seems to me that geopackages becames slow
after some days. E.g., if i create a new project and new
geopackages, qgis has no problems to open them in a decent
time (one or two minutes). But after some hours, if i close
qgis and restart it with the same project/data, access becames
really slow (15 minutes and more).</div>
</div>
</blockquote>
<p>This is probably what we need to google for then. <br>
</p>
<p>I Found this ???<br>
</p>
<p><a class="moz-txt-link-freetext" href="https://gis.stackexchange.com/questions/338478/reading-geopackage-with-large-feature-classes-is-very-slow">https://gis.stackexchange.com/questions/338478/reading-geopackage-with-large-feature-classes-is-very-slow</a><br>
</p>
<p>Perhaps you should get on the QGIS dev mailing list and ask why
geopackage get progressively slower. <br>
</p>
<p>Cheers!<br>
</p>
<blockquote type="cite"
cite="mid:CAH6xWviNKnSg+RaQT=uZrd5GSfwLBmRFfYxX93K1XgkNbVmZ0w@mail.gmail.com">
<div dir="auto">
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div> Enough memory?</div>
</div>
</blockquote>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Win10, i5, 16GB RAM, SSD 1TB, dual screen.</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div>Disable all plugins.</div>
</div>
</blockquote>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Ok. </div>
<div dir="auto"><br>
</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div><br>
</div>
<div>Caching features: in your QGIS options, you can
change the amount of featured that are cashed when you
open a lector layer. Try cashing more feature or much
less features (like 1). What happens? If you have a
max of 16000 features per layer, chances are you are
caching everything so everything is being loaded to
memory.</div>
</div>
</blockquote>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Ok i will try.</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div><br>
</div>
<div>File creation: how are the files created? Look at
the options in Gdal if the files are created in QGIS.
<a href="https://gdal.orgdrivers/vector/gpkg.html"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true">https://gdal.orgdrivers/vector/gpkg.html</a> Try
creating the files with Gdal translate (vector menu)
instead of just “export as” or save as in QGIS, force
extra option to force index creation.</div>
</div>
</blockquote>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">They are imported in qgis as CSV (are point
geometries with a single attribute as REAL) and then exported
in gpkg format.</div>
<div dir="auto">In export options, spatial index creation is
enabled by default.</div>
<div dir="auto"><br>
</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div><br>
</div>
<div>Do you have a spatial index build in? In theory
.gpkg comes with a spatial index but perhaps you can
rebuild one? Or make sure you build on from the start
when you create the file. (See above).</div>
</div>
</blockquote>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Sometime i run Vacuum from QGIS db manager. Also
sometime i force a new spatial index creation from layer
properties, but with no appreciable differences.</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div><br>
</div>
<div>I remember that when I created a .gpkg in python,
and used it in ArcGis, the file would take a long time
to open because it was reading every point of the file
in order just to get the files extent. Maybe that is
going on? I think you can specify the files extent in
the file metadata. (I would need to check my code but
I don’t see my solution here. I normally post them
whenI find them...).<a
href="https://gis.stackexchange.com/questions/374408/using-geopandas-generated-gpkg-in-arcmap"
rel="noreferrer noreferrer" target="_blank"
moz-do-not-send="true"> https://gis.stackexchange.com/questions/374408/using-geopandas-generated-gpkg-in-arcmap</a> </div>
<div><br>
</div>
<div>Just a bunch of ideas...</div>
</div>
</blockquote>
</div>
</div>
<div dir="auto"><br>
</div>
<div dir="auto">Thank you for your suggestions.</div>
<div dir="auto"><br>
</div>
<div dir="auto"><br>
</div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Nicolas Cadieux
<a class="moz-txt-link-freetext" href="https://gitlab.com/njacadieux">https://gitlab.com/njacadieux</a></pre>
</body>
</html>