[Qgis-user] [EXTERNAL] Re: Geopackage slow on NAS if not read-only

Árni Geirsson arni at alta.is
Thu Mar 16 05:49:31 PDT 2023


Thank you for the various responses.
Let me be clear that what I am griping about is not the locking aspect that
would allow multiple users to edit without conflict. I fully understand
that the geopackage is not safe for that, nor is the shapefile and perhaps
also the file geodatabase. In my case, the likelihood of two people wanting
to edit the same geopackage is negligible, so I take my chances. My only
gripe is the rendering speed of larger geopackages on a shared network, if
they are not set specifically to read-only. That is not a problem with the
shapefile and I think probably not with a file geodatabase. I am a long
time user of geopackages in a networked environment and I will continue to
use them - they are great - there is just this problem of rendering speed
over a network.

Árni


On Thu, 16 Mar 2023 at 12:33, Sebastian Gutwein <bas at rdgland.com> wrote:

> This is all beyond my expertise but I have also struggled with the
> effective use of gpkg’s. In my case over google drive.
> Here is what SQLite (the underlying software for gpkgs) says about using a
> SQLite database on a network:
> https://www.sqlite.org/useovernet.html
>
> I hope that someday someone implements a system that allows this to work
> efficiently as gpkgs are better in many respects than shapefiles. Perhaps
> Geodiff could be part of that solution.
> https://github.com/MerginMaps/geodiff
>
> On Thu, Mar 16, 2023 at 8:04 AM Árni Geirsson via QGIS-User <
> qgis-user at lists.osgeo.org> wrote:
>
>> Hi Jorge
>> I understand what you are pointing at and I use databases such as
>> Postgres/PostGIS also with good results. The thing is that working with
>> data in files sometimes has advantages and that is certainly how shapefiles
>> have been used. The geopackage has been suggested as a replacement for the
>> shapefile in the context of regular QGIS usage without any caveat saying
>> that geopackages only replace the storage and transfer role of shapefiles
>> and that geopackages should not be used for editing in a shared
>> environment, as is perfectly possible with shapefiles and widely practiced.
>> Not all users have access to database systems such as Postgres. I'm just
>> looking for some clarity on best practices for the common scenario of
>> working with QGIS using file based data in a network environment.
>>
>> Árni
>>
>>
>> On Thu, 16 Mar 2023 at 11:47, Jorge Gustavo Rocha via QGIS-User <
>> qgis-user at lists.osgeo.org> wrote:
>>
>>> Hi,
>>>
>>> For data storage and manipulation databases are suitable. Files are not.
>>>
>>> Geopackages are wonderful to transfer data between systems or to archive
>>> an entire project (snapshot of data, styles and the project itself).
>>>
>>> Regards,
>>>
>>> Jorge
>>> On 16/03/23 11:30, Árni Geirsson via QGIS-User wrote:
>>>
>>> Thank you Jarosław. Isn't it strange that this was discussed 5 years ago
>>> and SMB file sharing is very common? Would a linux based NAS be able to use
>>> another protocol? What are my options for file based data sharing in QGIS?
>>> Abandoning geopackages is not a realistic option for me, but I could get a
>>> different kind of NAS unit, if that helps, what kind then? What amazes me
>>> is how little I see this discussed. There was a message in this thread this
>>> morning from Thomas Struller, but I am not sure it is about the same root
>>> problem, maybe Thomas can elaborate.
>>> Should this perhaps be discussed in another forum, closer to the
>>> development of geopackage/sqlite?
>>>
>>> Árni Geirsson
>>>
>>> On Thu, 16 Mar 2023 at 09:52, Sadowski Jarosław <
>>> jaroslaw.sadowski at cpk.pl> wrote:
>>>
>>>> Long story short: gpkg is bad idea for network drives as SMB/NAS etc
>>>>
>>>>
>>>>
>>>> Sources:
>>>> Write-Ahead Logging (sqlite.org)
>>>> <https://www.sqlite.org/wal.html#advantages>
>>>>
>>>> writing gpkg and sqlite on samba shares fails · Issue #628 ·
>>>> r-spatial/sf · GitHub <https://github.com/r-spatial/sf/issues/628>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *_________________________________*
>>>>
>>>> *Jarosław Sadowski*
>>>> Kierownik Zespołu ds. Ochrony Środowiska | *Biuro Strategii i
>>>> Planowania, Projektowania i Inżynierii Podprogramu Kolejowego*
>>>> *Environmental Protection Team Leader **| Railway Subprogramme
>>>> Strategy & Planning, Design & Engineering Department*
>>>> e: jaroslaw.sadowski at cpk.pl
>>>> m: +48 532 720 230
>>>>
>>>> Centralny Port Komunikacyjny Sp. z o.o. z siedzibą w Warszawie, Aleje
>>>> Jerozolimskie 142B, 02-305 Warszawa
>>>> <https://www.google.com/maps/search/Aleje%0D%0A++++++++++++++++++++Jerozolimskie+142B,+02-305+Warszawa?entry=gmail&source=g>;
>>>> nr KRS 0000759991, Sąd Rejonowy dla m.st. Warszawy, XII Wydział
>>>> Gospodarczy Krajowego Rejestru Sądowego; NIP 701-08-94-497; REGON
>>>> 381918620; kapitał zakładowy 1.277.500.000,00 zł
>>>> Administratorem danych osobowych przekazanych przez Panią/Pana m.in. w
>>>> korespondencji mailowej jest Centralny Port Komunikacyjny Sp. z o.o. z
>>>> siedzibą w Warszawie. Przetwarzamy dane osobowe zgodnie z przepisami
>>>> ogólnego rozporządzenia o ochronie danych (RODO), więcej informacji na ten
>>>> temat znajduje się w zakładce Polityka Prywatności oraz w Klauzuli
>>>> informacyjnej na stronie internetowej www.cpk.pl.
>>>> Treści zawarte w niniejszej wiadomości i załącznikach do niej stanowią
>>>> Tajemnicę Przedsiębiorstwa w rozumieniu ustawy z dnia 16 kwietnia 1993 r. o
>>>> zwalczaniu nieuczciwej konkurencji. Jeśli otrzymałeś tę wiadomość przez
>>>> pomyłkę, bezzwłocznie skontaktuj się z nadawcą wiadomości oraz usuń jej
>>>> treść.
>>>> Centralny Port Komunikacyjny Sp. z o.o. with headquarters in Warsaw,
>>>> Aleje Jerozolimskie 142B, 02-305 Warsaw
>>>> <https://www.google.com/maps/search/Warsaw,+Aleje%0D%0A++++++++++++++++++++Jerozolimskie+142B,+02-305+Warsaw?entry=gmail&source=g>;
>>>> KRS No. 0000759991, District Court for the Capital City of Warsaw, 12th
>>>> Commercial Department of the National Court Register; NIP 701-08-94-497;
>>>> REGON 381918620; share capital of PLN 1.277.500.000,00.
>>>> The personal data controller of the personal data provided by you,
>>>> among others, in the e-mail correspondence, is Centralny Port Komunikacyjny
>>>> Sp. z o. o. based in Warsaw. The personal data is processed by us in
>>>> accordance with the provisions of the General Data Protection Regulation
>>>> (GDPR), for further information please read the Privacy Policy tab and the
>>>> Information Clause on the www.cpk.pl website.
>>>> The content of this message and its attachments constitute the Business
>>>> Secret within the meaning of the Act of 16 April 1993 on combating unfair
>>>> competition. If you received this message by mistake, contact the sender of
>>>> the message immediately and delete its content
>>>>
>>>> *From:* QGIS-User <qgis-user-bounces at lists.osgeo.org> *On Behalf Of *Árni
>>>> Geirsson via QGIS-User
>>>> *Sent:* Thursday, March 16, 2023 10:18 AM
>>>> *To:* jhubbslist at att.net
>>>> *Cc:* qgis-user at lists.osgeo.org
>>>> *Subject:* [EXTERNAL] Re: [Qgis-user] Geopackage slow on NAS if not
>>>> read-only
>>>>
>>>>
>>>>
>>>> UWAGA: Ta wiadomość pochodzi spoza CPK Sp. z o.o. Proszę pomóż zadbać o
>>>> bezpieczeństwo naszej organizacji. Zastanów się, zanim otworzysz link
>>>> lub załącznik.
>>>>
>>>>
>>>>
>>>> Yes, as far as I know, the NAS unit uses SMB as the file sharing
>>>> protocol. I used another NAS from QNAP before this one, also using SMB and
>>>> with the same problem. I thought pretty much all of the Linux based NAS
>>>> units were using SMB and if that is the problem, it should be widespread,
>>>> but I don't see any signs of that. Is SMB a problem for geopackage?
>>>>
>>>> I did a quick test in QGIS: A dataset of 178.000 line features is
>>>> rendered in about 1 second from a read only geopackage. When I remove the
>>>> read only flag, it is rendered in about 5 seconds.
>>>>
>>>> If I store the geopackage on the local hard drive, the problem
>>>> disappears, read-write or read-only does not matter.
>>>>
>>>> I have had suspicions about SMB being part of the problem but I don't
>>>> know enough about file access deep down in the operating system to
>>>> understand it.
>>>>
>>>>
>>>>
>>>> Árni Geirsson
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, 16 Mar 2023 at 01:45, jhubbslist--- via QGIS-User <
>>>> qgis-user at lists.osgeo.org> wrote:
>>>>
>>>> Árni -
>>>>
>>>>
>>>>
>>>> Are you using SMB/CIFS to access this NAS, and are you using wifi or
>>>> Ethernet to connect to it?
>>>>
>>>>
>>>>
>>>> - Jeff
>>>>
>>>>
>>>>
>>>> On 3/15/23 3:30 PM, Árni Geirsson via QGIS-User wrote:
>>>>
>>>> Hello all QGIS and geopackage users.
>>>>
>>>> I store my geopackages on a Synology RackStation NAS unit, like all
>>>> other documents that are kept on a shared drive in the office. For larger
>>>> datasets, the rendering is very slow, unless I open the properties dialog
>>>> for the file in Windows and check the read only box. After that, the
>>>> features are rendered blazingly fast. Nothing else is changed to see the
>>>> dramatic difference in the rendering speed. Luckily, I don't need to edit
>>>> many of the larger datasets, such as road networks and elevation contours
>>>> and the geopackage can be kept read only. Shapefiles are not affected.
>>>>
>>>> What explains this and does anyone know how to solve the problem?
>>>>
>>>> Do other users experience this?
>>>>
>>>>
>>>>
>>>> Árni Geirsson
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>>
>>>> QGIS-User mailing list
>>>>
>>>> QGIS-User at lists.osgeo.org
>>>>
>>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user <https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fqgis-user&data=05%7C01%7Cjaroslaw.sadowski%40cpk.pl%7C11269f199d644f13364608db25ff69c5%7Cfa798250ca0b4a1bb47381cff4a1752b%7C1%7C0%7C638145551176051605%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4NfOOZrB2k%2B7Q3V0QHNXIvEOTYHFWp4D10C1MzYTTQQ%3D&reserved=0>
>>>>
>>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user <https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fqgis-user&data=05%7C01%7Cjaroslaw.sadowski%40cpk.pl%7C11269f199d644f13364608db25ff69c5%7Cfa798250ca0b4a1bb47381cff4a1752b%7C1%7C0%7C638145551176051605%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4NfOOZrB2k%2B7Q3V0QHNXIvEOTYHFWp4D10C1MzYTTQQ%3D&reserved=0>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> QGIS-User mailing list
>>>> QGIS-User at lists.osgeo.org
>>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>>> <https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fqgis-user&data=05%7C01%7Cjaroslaw.sadowski%40cpk.pl%7C11269f199d644f13364608db25ff69c5%7Cfa798250ca0b4a1bb47381cff4a1752b%7C1%7C0%7C638145551176051605%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4NfOOZrB2k%2B7Q3V0QHNXIvEOTYHFWp4D10C1MzYTTQQ%3D&reserved=0>
>>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>>> <https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fqgis-user&data=05%7C01%7Cjaroslaw.sadowski%40cpk.pl%7C11269f199d644f13364608db25ff69c5%7Cfa798250ca0b4a1bb47381cff4a1752b%7C1%7C0%7C638145551176051605%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4NfOOZrB2k%2B7Q3V0QHNXIvEOTYHFWp4D10C1MzYTTQQ%3D&reserved=0>
>>>>
>>>>
>>> _______________________________________________
>>> QGIS-User mailing listQGIS-User at lists.osgeo.org
>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>>
>>> --
>>> Jorge Gustavo Rocha
>>> Geomaster, LDA
>>> "Take the open source journey with us"
>>>
>>> VAT/NIF: PT 510 906 109
>>> Phone: +351 253 257 173 (landline)
>>> Mobile: +351 910 333 888 (cellular)
>>> Rua Afonso Palmeira, 31, 7 <https://www.google.com/maps/search/Rua+Afonso+Palmeira,+31,+7?entry=gmail&source=g> D
>>> 4715-278 Braga
>>> PORTUGAL
>>> GPS 41.54627, -8.40432
>>>
>>>
>>> _______________________________________________
>>> QGIS-User mailing list
>>> QGIS-User at lists.osgeo.org
>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>>
>> _______________________________________________
>> QGIS-User mailing list
>> QGIS-User at lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>
> --
> Sent from a phone. Amazing! But possibly garbled. Sebastian Gutwein
> Regenerative Design Group 1 Chevalier Ave Greenfield MA Bas at rdgland.com
> (631) 241-1018
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20230316/2e8c3777/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: transparency_ce4012c8-87e7-4198-a48f-d42a38680681.png
Type: image/png
Size: 2458 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20230316/2e8c3777/attachment-0001.png>


More information about the QGIS-User mailing list