[gdal-dev] Writing concurrent indexes to a FileGDB

Mon Feb 6 04:06:50 PST 2023

Hi Even

Many thanks for your response and its great to get confirmation that I wasn't doing something wrong or missing something in the docs.

I can definitely run multiple processes that write an index on different tables in a single FileGDB at the same time and it seems "ok" - ie I haven't seen any errors.

But my tests definitely show you cannot write multiple indexes concurrently on the same table which seems to fit with the issue of the .gdbindexes file you mention.

Thanks again

Tim

From: Even Rouault <even.rouault at spatialys.com>
Sent: 06 February 2023 12:01
To: Tim Martin <Tim.Martin at os.uk>; gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] Writing concurrent indexes to a FileGDB

External Email: Take care with attachments & links.

Tim,

Currently there is no guarantee that concurrent updates to the same FileGDB database, either from the same process or from different process, are safe, as no locks are set, and the driver might cache in memory things assuming that nothing else changes them behind its back.

That said, I *believe* (but definitely doesn't guarantee) that you could create indexes on different layers/tables of a FileGDB from concurrent ogrinfo instances. It is probably not safe however to create indexes on different columns of the same table concurrently, because the .gdbindexes file might be written concurrently

Even
Le 06/02/2023 à 12:03, Tim Martin via gdal-dev a écrit :
Hi GDAL Devs

I posted this question on GIS Stack Exchange and someone recommended I contact this group (https://gis.stackexchange.com/questions/451331/adding-multiple-attribute-indexes-to-filegdb-using-ogrinfo/451336#451336<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgis.stackexchange.com%2Fquestions%2F451331%2Fadding-multiple-attribute-indexes-to-filegdb-using-ogrinfo%2F451336%23451336&data=05%7C01%7CTim.Martin%40os.uk%7C4ab8822f4294480dbbbf08db0839dd47%7C7988742dc5434b9a87a910a7b354d289%7C0%7C0%7C638112816847475257%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QQPsbRNKS9SnnDeMJXYvq07Uy%2Fe2h4IGt%2Fzd53hRApQ%3D&reserved=0>)

I have the unfortunate task of creating some very large FileGDB and need to add multiple indexes to several attributes on each Feature Type.

I saw that the OpenFileGDB driver under Dev is now able to Write, so am using that version, which by my testing is 3 or 4 times faster than the original FielGDB driver - which is amazing!!!

Now I need to index these columns and it has taken 26hrs to add 5 attribute indexes due to the sheer number of features (200mill or so).

So am wondering if there is a way to use ogrinfo and write indexes concurrently?

With thanks Tim

This email and any attachments are intended only for the intended recipient and may contain sensitive information. If you are not the intended recipient, please immediately delete this email and inform the sender.

OS email communications may be monitored to ensure the secure and effective operation of our systems and for other lawful purposes. Subject to contract: No rights are to be derived from any proposal contained in this email until a written agreement containing all necessary terms is executed between the relevant parties.

Thank you for your cooperation.

Ordnance Survey Limited (Company Registration number 09121572)
Registered Office: Explorer House
Adanac Drive
Southampton SO16 0AS
Tel: 03456 050505
http://www.os.uk<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.os.uk%2F&data=05%7C01%7CTim.Martin%40os.uk%7C4ab8822f4294480dbbbf08db0839dd47%7C7988742dc5434b9a87a910a7b354d289%7C0%7C0%7C638112816847475257%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ep6unLvZ8F7RJ8uIq2rSLXt4X0WlcOdZnKsR259FngQ%3D&reserved=0>

_______________________________________________

gdal-dev mailing list

gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>

https://lists.osgeo.org/mailman/listinfo/gdal-dev<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.osgeo.org%2Fmailman%2Flistinfo%2Fgdal-dev&data=05%7C01%7CTim.Martin%40os.uk%7C4ab8822f4294480dbbbf08db0839dd47%7C7988742dc5434b9a87a910a7b354d289%7C0%7C0%7C638112816847631480%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=slNtsriM%2Bd%2FIuMOOZlROtgj4LOiELXh3QHiAsWdNBU4%3D&reserved=0>

--

http://www.spatialys.com<https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.spatialys.com%2F&data=05%7C01%7CTim.Martin%40os.uk%7C4ab8822f4294480dbbbf08db0839dd47%7C7988742dc5434b9a87a910a7b354d289%7C0%7C0%7C638112816847631480%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=AmRBIbuSyjqnkz3pyZ3V0qwtZFxjp2SV1xZcWJ4RJtY%3D&reserved=0>

My software is free, but my time generally not.

This email and any attachments are intended only for the intended recipient and may contain sensitive information. If you are not the intended recipient, please immediately delete this email and inform the sender.

OS email communications may be monitored to ensure the secure and effective operation of our systems and for other lawful purposes. Subject to contract: No rights are to be derived from any proposal contained in this email until a written agreement containing all necessary terms is executed between the relevant parties.

Thank you for your cooperation.

Ordnance Survey Limited (Company Registration number 09121572)
Registered Office: Explorer House
Adanac Drive
Southampton SO16 0AS
Tel: 03456 050505
http://www.os.uk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20230206/d903f759/attachment-0001.htm>