<html><body><style> .apply-styles pre{background:transparent}.apply-styles pre.-wm-western{font-family:'Liberation Mono',monospace;font-size:10pt}.apply-styles pre.-wm-cjk{font-family:'Noto Sans Mono CJK SC',monospace;font-size:10pt}.apply-styles pre.-wm-ctl{font-family:'Liberation Mono',monospace;font-size:10pt}.apply-styles p{line-height:115%;margin-bottom:0.1in;background:transparent}.apply-styles strong{font-weight:bold}.apply-styles code.-wm-western{font-family:'Liberation Mono',monospace}.apply-styles code.-wm-cjk{font-family:'Noto Sans Mono CJK SC',monospace}.apply-styles code.-wm-ctl{font-family:'Liberation Mono',monospace}</style><p>Hi
Even,</p><p><br></p>
<p>I was a bit confused about FIDs in the MVT driver, so thanks a lot for your earlier explanations. (Sorry I didn’t reply directly to your email — I can see
it in the GDAL archives but not in my inbox, not sure what happened.)</p><p><br></p>
<p>When implementing <code class="-wm-western">CreateFeature</code> and
<code class="-wm-western">DeleteFeature</code>, I initially assumed that
the <code class="-wm-western">idx</code> identifier in the temporary
SQLite table (created in <code class="-wm-western">ICreateFeature</code>)
could be relied on as a kind of “global FID”. So I implemented
DeleteFeature with idx parameter as input. :-) I now understand that
this assumption was wrong, and that there are actually three distinct
identifiers involved:</p>
<ol><li><p><code class="-wm-western">idx/nSerial (ID in temp SQLITE db)</code><br>
This identifier is
created in <code class="-wm-western">ICreateFeature</code> and simply
represents the order in which features are written into the
temporary table - has no direct relationship to the FIDs exposed
later through the GDAL API.</p>
</li><li><p><code>localID (feature ID in tile)</code><br>
This ID exists only when reading a
specific tile. It corresponds to
<code class="-wm-western">poUnderlyingFeature->GetFID()</code> and
uniquely identifies a feature fragment within a single tile.</p>
</li><li><p>dataset-level (global) FID<br>
This
is the FID returned by <code class="-wm-western">GetFID()</code> when
reading an MVT dataset. It is computed from a tile-specific <code class="-wm-western">FIDBase</code>
combined with the <code class="-wm-western">localFID</code>, as you
described in your email.</p>
</li></ol>
<p>In addition, I implemented the small extension you suggested: when
the <code class="-wm-western">OGR_MVT_ADD_TILE_FIELDS</code> option is
enabled, I can now retrieve the tile coordinates (<code class="-wm-western">z</code>,
<code class="-wm-western">x</code>, <code class="-wm-western">y</code>) for
each feature while reading the dataset. I’ve pushed it as a PR https://github.com/OSGeo/gdal/pull/13596, as it’s a useful improvement that may be helpful when
reading the dataset anyway. Could you please take a look at it?</p><p><br></p>
<p>With this in place, I am currently able to build a table like the
following when reading an MVT dataset (example shown for the first
three parcel features):</p>
<pre class="-wm-western"><code class="-wm-western">globalFID fidBase z x y parcel_id localFID</code>
<code class="-wm-western">5926 5926 7 46 38 76343521010 0</code>
<code class="-wm-western">22310 5926 7 46 38 73527254010 1</code>
<code class="-wm-western">38694 5926 7 46 38 73527253010 2</code></pre><p>
So now I know which <code class="-wm-western">globalFID</code>s
correspond to each parcel, including its <code class="-wm-western">fidBase</code>
and the tile coordinates (<code class="-wm-western">z</code>, <code class="-wm-western">x</code>,
<code class="-wm-western">y</code>). Nice :-) </p><p>And internally I also have
the full update pipeline implemented - when the dataset is opened in
<code class="-wm-western">GDAL_OF_UPDATE</code> mode, an instance of
<code class="-wm-western">OGRMVTWriterDataset</code> is created, which
allows me to call <code class="-wm-western">DeleteFeature</code> (D) and
<code class="-wm-western">CreateFeature</code> (I), or a combination of
both when an existing feature needs to be updated (U). During these
operations, I collect the affected tiles, and at the end of the
update process the pipeline calls the <code class="-wm-western">UpdateOutput</code>
method to regenerate only the <code class="-wm-western">.pbf</code> files or MBTILES rows that were impacted.</p><p><br></p>
<p>So the remaining challenge is that I don’t see a clear way to
modify the driver so that a feature can be reliably deleted from the
intermediate SQLite database. I initially considered whether it might
be possible to store the <code class="-wm-western">globalFID</code> (or
at least the <code class="-wm-western">localFID</code>) back into the
temporary SQLite table during the <code class="-wm-western">CreateOutput</code>
step, after encoding each individual tile. However, at that stage the
driver is not in a dataset-reading context, and I do not see a
reliable way to access or derive the <code class="-wm-western">localFID</code>
during writing (perhaps such a mechanism could be implemented, but
even if it exists, it does not seem like an ideal approach).</p><p><br></p>
<p>Given this, I started thinking about an alternative solution based
on a stable, domain-level identifier such as <code class="-wm-western">domain_id</code>
(in practice, e.g., <code class="-wm-western">parcel_id</code>), which is
already known at write time. This would require storing such an
identifier explicitly in the intermediate SQLite table (in addition
to the internal <code class="-wm-western">idx</code>, geometry, and tile
coordinates). <code class="-wm-western">DeleteFeature</code> could then
be implemented by removing all rows associated with that domain-level
identifier (that would be an input parameter).</p><p><br></p>
<p>What do you think, Even? Does this approach make architectural
sense to you?</p><p><br></p>
<p>Thanks again for your help, and I hope you have a great
Christmas!<br>
Linda Karlovská</p>
<p style="line-height:100%;margin-bottom:0in"><br>
</p>
<br><aside>---------- Původní e-mail ----------<br>Od: Linda Kladivová via gdal-dev <gdal-dev@lists.osgeo.org><br>Komu: gdal-dev@lists.osgeo.org<br>Datum: 23. 11. 2025 13:07:27<br>Předmět: [gdal-dev] Decoding feature blobs and tracking FIDs in MVT datasets</aside><br><blockquote data-email="gdal-dev@lists.osgeo.org"><p>Hi,</p><p><br></p>
<p>I’d like to ask about one specific aspect of the MVT driver. In my
workflow, I need to track the FIDs of features inside an MVT dataset
together with their corresponding internal codes (actual parcel IDs).</p><p><br></p>
<p>My use case involves an MVT dataset of about 20 million features that
is frequently updated. For this purpose, I have implemented an
extension of the MVT driver that can open an MVT dataset in update mode
and create/delete features based on changes detected by a preceding ETL
process. Updates typically affect only a few individual features every
hour.</p><p><br></p>
<p>After the initial loading of the full dataset using <code>ogr2ogr</code>
I need a way to iterate through the dataset and build a separate table
mapping each internal FID to the parcel’s real ID. I have tried opening
and inspecting the <code>temp.db</code> file (which I do not delete):</p><p><br></p><p>GDALDataset* poTempDb =<br> (GDALDataset*) GDALOpenEx(dstTempDb.c_str(),<br> GDAL_OF_VECTOR, nullptr, nullptr, nullptr);<br><br>OGRLayer* poTempLayer = poTempDb->GetLayerByName("temp");<br>poTempLayer->ResetReading();<br><br>OGRFeature* poFeat = nullptr;<br><br>while ((poFeat = poTempLayer->GetNextFeature()) != nullptr)<br>{<br> GIntBig fid = poFeat->GetFID();<br> std::cout << "FID=" << fid << " — ";<br> int idx = poFeat->GetFieldIndex("feature");<br> int blobSize = 0;<br> const GByte* pBlob = poFeat->GetFieldAsBinary(idx, &blobSize);<br> std::cout << " Blob size=" << blobSize << std::endl;<br>}<br><br></p><p>I can successfully access the <code>feature</code>
blob, but I need to decode it in order to extract my custom parcel ID
attribute and store those values externally. This would allow me to keep
track of which parcel corresponds to which internal FID.</p><p><br>This is a one-time (potentially slow) process. Afterward, during incremental updates, I will be calling <code>CreateFeature</code> and <code>DeleteFeature</code> (and updating my external mapping table accordingly). For edits, I currently use a combination of <code>DeleteFeature</code> and <code>CreateFeature</code> (I haven’t implemented <code>SetFeature</code> yet).</p><p><br></p>
<p>My question is:<br>
<span>Is there a way to decode this blob using existing GDAL/MVT functionality, or would this require implementing a new function?</span></p><p>Thank you very much for your help.</p><p><br></p><p>Linda Karlovská</p>_______________________________________________
<br>gdal-dev mailing list
<br>gdal-dev@lists.osgeo.org
<br>https://lists.osgeo.org/mailman/listinfo/gdal-dev
<br></blockquote></body></html>