[Qgis-user] [EXTERNAL] Re: Geopackage slow on NAS if not read-only

Thu Mar 16 05:18:36 PDT 2023

Hi,

Apologies for pulling off further in the tangent, but I just want to check
your underlying assumption:

> that geopackages should not be used for editing in a shared environment,
as is perfectly possible with shapefiles and widely practiced.

Is this actually the case with Shapefiles? From a quick search online,
neither Shapefiles nor FileGDBs appear to be safe in the situation of
simultaneous network writes (=editing in a shared environment). It works by
chance if only one user modifies the file at once, but is not intrinsically
safe.

Regards,
Daniel

On Thu, 16 Mar 2023 at 12:04, Árni Geirsson via QGIS-User <
qgis-user at lists.osgeo.org> wrote:

> Hi Jorge
> I understand what you are pointing at and I use databases such as
> Postgres/PostGIS also with good results. The thing is that working with
> data in files sometimes has advantages and that is certainly how shapefiles
> have been used. The geopackage has been suggested as a replacement for the
> shapefile in the context of regular QGIS usage without any caveat saying
> that geopackages only replace the storage and transfer role of shapefiles
> and that geopackages should not be used for editing in a shared
> environment, as is perfectly possible with shapefiles and widely practiced.
> Not all users have access to database systems such as Postgres. I'm just
> looking for some clarity on best practices for the common scenario of
> working with QGIS using file based data in a network environment.
>
> Árni
>
>
> On Thu, 16 Mar 2023 at 11:47, Jorge Gustavo Rocha via QGIS-User <
> qgis-user at lists.osgeo.org> wrote:
>
>> Hi,
>>
>> For data storage and manipulation databases are suitable. Files are not.
>>
>> Geopackages are wonderful to transfer data between systems or to archive
>> an entire project (snapshot of data, styles and the project itself).
>>
>> Regards,
>>
>> Jorge
>> On 16/03/23 11:30, Árni Geirsson via QGIS-User wrote:
>>
>> Thank you Jarosław. Isn't it strange that this was discussed 5 years ago
>> and SMB file sharing is very common? Would a linux based NAS be able to use
>> another protocol? What are my options for file based data sharing in QGIS?
>> Abandoning geopackages is not a realistic option for me, but I could get a
>> different kind of NAS unit, if that helps, what kind then? What amazes me
>> is how little I see this discussed. There was a message in this thread this
>> morning from Thomas Struller, but I am not sure it is about the same root
>> problem, maybe Thomas can elaborate.
>> Should this perhaps be discussed in another forum, closer to the
>> development of geopackage/sqlite?
>>
>> Árni Geirsson
>>
>> On Thu, 16 Mar 2023 at 09:52, Sadowski Jarosław <jaroslaw.sadowski at cpk.pl>
>> wrote:
>>
>>> Long story short: gpkg is bad idea for network drives as SMB/NAS etc
>>>
>>>
>>>
>>> Sources:
>>> Write-Ahead Logging (sqlite.org)
>>> <https://www.sqlite.org/wal.html#advantages>
>>>
>>> writing gpkg and sqlite on samba shares fails · Issue #628 ·
>>> r-spatial/sf · GitHub <https://github.com/r-spatial/sf/issues/628>
>>>
>>>
>>>
>>>
>>>
>>> *_________________________________*
>>>
>>> *Jarosław Sadowski*
>>> Kierownik Zespołu ds. Ochrony Środowiska | *Biuro Strategii i
>>> Planowania, Projektowania i Inżynierii Podprogramu Kolejowego*
>>> *Environmental Protection Team Leader **| Railway Subprogramme Strategy
>>> & Planning, Design & Engineering Department*
>>> e: jaroslaw.sadowski at cpk.pl
>>> m: +48 532 720 230
>>>
>>> Centralny Port Komunikacyjny Sp. z o.o. z siedzibą w Warszawie, Aleje
>>> Jerozolimskie 142B, 02-305 Warszawa; nr KRS 0000759991, Sąd Rejonowy dla
>>> m.st. Warszawy, XII Wydział Gospodarczy Krajowego Rejestru Sądowego;
>>> NIP 701-08-94-497; REGON 381918620; kapitał zakładowy 1.277.500.000,00
>>>  zł
>>> Administratorem danych osobowych przekazanych przez Panią/Pana m.in. w
>>> korespondencji mailowej jest Centralny Port Komunikacyjny Sp. z o.o. z
>>> siedzibą w Warszawie. Przetwarzamy dane osobowe zgodnie z przepisami
>>> ogólnego rozporządzenia o ochronie danych (RODO), więcej informacji na ten
>>> temat znajduje się w zakładce Polityka Prywatności oraz w Klauzuli
>>> informacyjnej na stronie internetowej www.cpk.pl.
>>> Treści zawarte w niniejszej wiadomości i załącznikach do niej stanowią
>>> Tajemnicę Przedsiębiorstwa w rozumieniu ustawy z dnia 16 kwietnia 1993 r. o
>>> zwalczaniu nieuczciwej konkurencji. Jeśli otrzymałeś tę wiadomość przez
>>> pomyłkę, bezzwłocznie skontaktuj się z nadawcą wiadomości oraz usuń jej
>>> treść.
>>> Centralny Port Komunikacyjny Sp. z o.o. with headquarters in Warsaw,
>>> Aleje Jerozolimskie 142B, 02-305 Warsaw; KRS No. 0000759991, District Court
>>> for the Capital City of Warsaw, 12th Commercial Department of the National
>>> Court Register; NIP 701-08-94-497; REGON 381918620; share capital of PLN
>>> 1.277.500.000,00.
>>> The personal data controller of the personal data provided by you, among
>>> others, in the e-mail correspondence, is Centralny Port Komunikacyjny Sp. z
>>> o. o. based in Warsaw. The personal data is processed by us in accordance
>>> with the provisions of the General Data Protection Regulation (GDPR), for
>>> further information please read the Privacy Policy tab and the Information
>>> Clause on the www.cpk.pl website.
>>> The content of this message and its attachments constitute the Business
>>> Secret within the meaning of the Act of 16 April 1993 on combating unfair
>>> competition. If you received this message by mistake, contact the sender of
>>> the message immediately and delete its content
>>>
>>> *From:* QGIS-User <qgis-user-bounces at lists.osgeo.org> *On Behalf Of *Árni
>>> Geirsson via QGIS-User
>>> *Sent:* Thursday, March 16, 2023 10:18 AM
>>> *To:* jhubbslist at att.net
>>> *Cc:* qgis-user at lists.osgeo.org
>>> *Subject:* [EXTERNAL] Re: [Qgis-user] Geopackage slow on NAS if not
>>> read-only
>>>
>>>
>>>
>>> UWAGA: Ta wiadomość pochodzi spoza CPK Sp. z o.o. Proszę pomóż zadbać o
>>> bezpieczeństwo naszej organizacji. Zastanów się, zanim otworzysz link
>>> lub załącznik.
>>>
>>>
>>>
>>> Yes, as far as I know, the NAS unit uses SMB as the file sharing
>>> protocol. I used another NAS from QNAP before this one, also using SMB and
>>> with the same problem. I thought pretty much all of the Linux based NAS
>>> units were using SMB and if that is the problem, it should be widespread,
>>> but I don't see any signs of that. Is SMB a problem for geopackage?
>>>
>>> I did a quick test in QGIS: A dataset of 178.000 line features is
>>> rendered in about 1 second from a read only geopackage. When I remove the
>>> read only flag, it is rendered in about 5 seconds.
>>>
>>> If I store the geopackage on the local hard drive, the problem
>>> disappears, read-write or read-only does not matter.
>>>
>>> I have had suspicions about SMB being part of the problem but I don't
>>> know enough about file access deep down in the operating system to
>>> understand it.
>>>
>>>
>>>
>>> Árni Geirsson
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, 16 Mar 2023 at 01:45, jhubbslist--- via QGIS-User <
>>> qgis-user at lists.osgeo.org> wrote:
>>>
>>> Árni -
>>>
>>>
>>>
>>> Are you using SMB/CIFS to access this NAS, and are you using wifi or
>>> Ethernet to connect to it?
>>>
>>>
>>>
>>> - Jeff
>>>
>>>
>>>
>>> On 3/15/23 3:30 PM, Árni Geirsson via QGIS-User wrote:
>>>
>>> Hello all QGIS and geopackage users.
>>>
>>> I store my geopackages on a Synology RackStation NAS unit, like all
>>> other documents that are kept on a shared drive in the office. For larger
>>> datasets, the rendering is very slow, unless I open the properties dialog
>>> for the file in Windows and check the read only box. After that, the
>>> features are rendered blazingly fast. Nothing else is changed to see the
>>> dramatic difference in the rendering speed. Luckily, I don't need to edit
>>> many of the larger datasets, such as road networks and elevation contours
>>> and the geopackage can be kept read only. Shapefiles are not affected.
>>>
>>> What explains this and does anyone know how to solve the problem?
>>>
>>> Do other users experience this?
>>>
>>>
>>>
>>> Árni Geirsson
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> QGIS-User mailing list
>>>
>>> QGIS-User at lists.osgeo.org
>>>
>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user <https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fqgis-user&data=05%7C01%7Cjaroslaw.sadowski%40cpk.pl%7C11269f199d644f13364608db25ff69c5%7Cfa798250ca0b4a1bb47381cff4a1752b%7C1%7C0%7C638145551176051605%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4NfOOZrB2k%2B7Q3V0QHNXIvEOTYHFWp4D10C1MzYTTQQ%3D&reserved=0>
>>>
>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user <https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fqgis-user&data=05%7C01%7Cjaroslaw.sadowski%40cpk.pl%7C11269f199d644f13364608db25ff69c5%7Cfa798250ca0b4a1bb47381cff4a1752b%7C1%7C0%7C638145551176051605%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4NfOOZrB2k%2B7Q3V0QHNXIvEOTYHFWp4D10C1MzYTTQQ%3D&reserved=0>
>>>
>>>
>>>
>>> _______________________________________________
>>> QGIS-User mailing list
>>> QGIS-User at lists.osgeo.org
>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>> <https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fqgis-user&data=05%7C01%7Cjaroslaw.sadowski%40cpk.pl%7C11269f199d644f13364608db25ff69c5%7Cfa798250ca0b4a1bb47381cff4a1752b%7C1%7C0%7C638145551176051605%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4NfOOZrB2k%2B7Q3V0QHNXIvEOTYHFWp4D10C1MzYTTQQ%3D&reserved=0>
>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>> <https://eur06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fqgis-user&data=05%7C01%7Cjaroslaw.sadowski%40cpk.pl%7C11269f199d644f13364608db25ff69c5%7Cfa798250ca0b4a1bb47381cff4a1752b%7C1%7C0%7C638145551176051605%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=4NfOOZrB2k%2B7Q3V0QHNXIvEOTYHFWp4D10C1MzYTTQQ%3D&reserved=0>
>>>
>>>
>> _______________________________________________
>> QGIS-User mailing listQGIS-User at lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>
>> --
>> Jorge Gustavo Rocha
>> Geomaster, LDA
>> "Take the open source journey with us"
>>
>> VAT/NIF: PT 510 906 109
>> Phone: +351 253 257 173 (landline)
>> Mobile: +351 910 333 888 (cellular)
>>
>> Rua Afonso Palmeira, 31, 7 D
>> 4715-278 Braga
>> PORTUGAL
>> GPS 41.54627, -8.40432
>>
>>
>> _______________________________________________
>> QGIS-User mailing list
>> QGIS-User at lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>
> _______________________________________________
> QGIS-User mailing list
> QGIS-User at lists.osgeo.org
> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20230316/9a43bcb4/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: transparency_ce4012c8-87e7-4198-a48f-d42a38680681.png
Type: image/png
Size: 2458 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/qgis-user/attachments/20230316/9a43bcb4/attachment.png>