[QGIS-Developer] [Qgis-user] Multi users environment: From geopackage to a database?

Wed May 6 05:45:07 PDT 2020

Hi Stefan,

Thanks for the forward.

We are definitely aware of these issues and you can expect further 
discussion on the QGIS and GDAL developer lists in the near future. 
Several key QGIS users already identified this as a major hurdle when 
migrating data from Shp to gpkg.

Hopefully a good solution can be found.

Andreas

Am 06.05.20 um 14:18 schrieb Stefan Keller:
> QGIS-Devs
>
> I'm forwarding this thread from QGIS-users mailinglist to this list.
>
> This is about a crash report (aka severe bug but also a show stopper)
> regarding GeoPackage - even read-only.
>
> Goal should IMHO be that 1. QGIS does not crash, 2. QGIS supports
> read-only multi-user access, and 3. QGIS prevents users to open
> GeoPackage as write access unless it's sure the resource is locked.
>
> I did not find any open issue on this topic except [1] and [2] which
> are from 2017 and 2018. But perhaps I'm missing something.
>
> => Is there any dev working on this long known and outstanding issue?
>
> :Stefan
>
> [1] https://github.com/qgis/QGIS/issues/23991
> [2] https://github.com/qgis/QGIS/issues/27899
>
> Am Mi., 6. Mai 2020 um 12:22 Uhr schrieb Robert Nuske
> <rsn.mailinglists at gmail.com>:
>> Hi Bruno,
>>
>> another list of known issues with SQLite/GeoPackage and SAMBA in particular
>> https://trac.osgeo.org/gdal/wiki/UserDocs/SQLite
>>
>>
>> regards
>>    robert
>>
>> Am 06.05.20 um 12:00 schrieb Stefan Keller:
>>> Hi,
>>>
>>> This should be IMHO an issue of highest priority for QGIS.
>>>
>>> Jésahel  wrote
>>>> I've created a script to deactivate wal on samba shares, because samba is the main issue.
>>>> It contains this only line : QSettings().setValue("/qgis/walForSqlite3", False)
>>> That's interesting. Are you sure it's Samba?
>>> SQLite FAQ [2] confirms that NFS has problems.
>>> Did not find issues on Samba, but file locking with fcntl is mentioned here [3].
>>>
>>> Goal should IMHO be that 1. No crash, 2. read-only multi-user access
>>> should be possible and 3. writing users must be warned if QGIS isn't
>>> able to exclusively open the file.
>>> And Jésahel  script suggests, that at least point 1 and 2 are possible.
>>>
>>> Andreas wrote:
>>>> Unfortunately it is a known issue that Geopackages and QGIS are bad at handling simultaneous write and even read requests
>>>> and developers are working on finding solutions (hopefully soon).
>>> The only issue I find on QGIS tracker is [1]. Am I missing something?
>>>
>>> :Stefan
>>>
>>> [1] https://github.com/qgis/QGIS/issues/23991
>>> [2] https://www.sqlite.org/faq.html#q5
>>> [3] https://www.samba.org/samba/docs/old/Samba3-HOWTO/locking.html
>>>
>>> Am Mi., 6. Mai 2020 um 10:42 Uhr schrieb Jésahel Benoist <djes1975 at gmail.com>:
>>>> Hi Bruno,
>>>>
>>>> I've worked a lot on this issue for two years now. I've started with sqlite (that was faster and better) but finally I decided to migrate to geopackage. It was painful at first but now it's OK. But it needs some rules.
>>>>
>>>> First, I've broken the original big geopackage files into smaller ones. The users are advised to work only one by one on a specific thematic/file (read access is possible).
>>>> I've created a script to deactivate wal on samba shares, because samba is the main issue. It contains this only line : QSettings().setValue("/qgis/walForSqlite3", False)
>>>> You should download some tools like Spatialite_GUI and SQLiteDatabaseBrowser so that you can really work on the db and have better control. QGIS's dbmanager is a bit limited and sometimes buggy for sqlite/geopackage, especially with special chars.
>>>> Try to work with QGIS internal explorer, it's a tool that has good export/import functions.
>>>> Backup files at least one time a day !
>>>>
>>>> Now with the covid I have created my own online postgis server so that we can work at home without "help" of IT department and it's GREAT. Multi-user, faster, smaller, stronger (lol). And some nice capabilities like spatial filtering ! I highly recommend this.
>>>>
>>>> Questions for all : I've recently discovered this https://gdal.org/user/virtual_file_systems.html#virtual-file-systems
>>>> Wouldn't it be possible to avoid samba's multi-users bug with vsi ?
>>>>
>>>> Jésahel
>>>>
>>>>
>>>>
>>>> Le mer. 6 mai 2020 à 09:12, Andreas Neumann <a.neumann at carto.net> a écrit :
>>>>> Hi Bruno,
>>>>>
>>>>> Unfortunately it is a known issue that Geopackages and QGIS are bad at handling simultaneous write and even read requests and developers are working on finding solutions (hopefully soon). In single user scenarios Geopackages are fine. But Multiuser must be avoided, even for reading only, unfortunately.
>>>>>
>>>>> As to PostgreSQL vs. Postgis: Postgis is the spatial extension of PostgreSQL. If you need geometries (which I assume) than you will need Postgis.
>>>>>
>>>>> Andreas
>>>>>
>>>>> Am 06.05.20 um 08:58 schrieb brunos at mailbox.org:
>>>>>
>>>>> Hi
>>>>>
>>>>> In my company we are five people who sometimes work with GIS, and we are moving from ArcGIS to QGIS (how cool is QGIS and SLYR!) and in this context from ESRI file geodatabase to geopackage. Today we run into serious issues when two people worked with the same geopackage. I was aware, that it is dangerous and unwise to edit the same geopackage from two different computers at the same time. But I did not expect both QGIS applications to crash immediately and the geopackage to get corrupted. I hoped to get a warning or simply not be able to enter edit mode (We later found out, that the corrupted geopackage could be recoveredby by adding a new table, this somehow fixed it. Whew!).
>>>>>
>>>>> Worse, I found that the same thing happens, when I open a QGIS project with a layer from a geopackage for display only (no edit mode), which is being edited by an other user on an other computer. Again, both QGIS application crashed immediately with data loss. Again, the non-responsive QGIS tasks could not be killed by windows taskmanager, only by restarting of the computers.
>>>>>
>>>>> We are using QGIS 3.12.2 on Windows 10. The Geopackages are located on a samba network share.
>>>>>
>>>>> My questions:
>>>>>
>>>>> 1. Is this setup that bad? Do you experience the same fatal consequences if someone loads a layer from a geopackage which is being edited by someone else? Is there something I do wrong or I that can do to improve the situation?
>>>>>
>>>>> 2. I realize, that we should probably move from geopackage to a database. Do we need PostGIS or can we use PostgreSQL? We don't need more functinality than a geopackage offers (exept multi user read/edit possibility and stability).
>>>>>
>>>>> Many thanks for some hints that put me in the right direction!
>>>>>
>>>>> Bruno
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Qgis-user mailing list
>>>>> Qgis-user at lists.osgeo.org
>>>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>>>>
>>>>> _______________________________________________
>>>>> Qgis-user mailing list
>>>>> Qgis-user at lists.osgeo.org
>>>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>>> _______________________________________________
>>>> Qgis-user mailing list
>>>> Qgis-user at lists.osgeo.org
>>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>> _______________________________________________
>>> Qgis-user mailing list
>>> Qgis-user at lists.osgeo.org
>>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> _______________________________________________
>> Qgis-user mailing list
>> Qgis-user at lists.osgeo.org
>> List info: https://lists.osgeo.org/mailman/listinfo/qgis-user
>> Unsubscribe: https://lists.osgeo.org/mailman/listinfo/qgis-user