<div dir="auto">yes, jammy VM on openstack is the host (and is where I run pretty much everything, though will increasingly use AWS).<div dir="auto"><br></div><div dir="auto">Thanks for the note, I'll try on other systems too. We need a security-allow set for vsicurl to work so if there are other little details I'll be keen to flush them out.</div><div dir="auto"><br></div><div dir="auto">Cheers, Mike </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 5 Feb 2024, 18:44 Javier Jimenez Shaw, <<a href="mailto:j1@jimenezshaw.com">j1@jimenezshaw.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div>Hi Mike<div dir="auto"><br></div><div dir="auto">Out of curiosity, are run running it in a virtual machine?</div><div dir="auto">A few year ago I had problems running a program in a virtual machine (virtualbox, but I read it happens in others) due to a missing SSE instruction. The solution there was to "enable" the missing instructions in the virtual machine configuration (that I don't know why it was not the default).</div><div dir="auto"><br></div><div dir="auto">Cheers</div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 5 Feb 2024, 02:04 Even Rouault via gdal-dev, <<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank" rel="noreferrer">gdal-dev@lists.osgeo.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div>
<p><a href="http://ghcr.io/osgeo/gdal:ubuntu-full-latest" rel="noreferrer noreferrer" target="_blank">ghcr.io/osgeo/gdal:ubuntu-full-latest</a> has been regenerated with
the rebuild of TileDB without AVX2. I've also enabled the
drivers-with-external-depencies-built-as-plugin GDAL build mode,
so it is easy to just remove a given plugin by deleting the
corresponding .so in /usr/lib/x86_64-linux-gnu/gdalplugins</p>
<p>Even<br>
</p>
<div>Le 04/02/2024 à 22:51, Michael Sumner a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">indeed there's no avx2:
<div><br>
</div>
<div>cat /proc/cpuinfo|grep sse|head -n 1<br>
</div>
<div>flags : fpu vme de pse tsc msr pae mce cx8 apic
sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2
syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl
cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16
sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx
f16c hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a
misalignsse 3dnowprefetch osvw xop fma4 tbm perfctr_core ssbd
ibpb vmmcall tsc_adjust bmi1 virt_ssbd arat npt nrip_save
arch_capabilities<br>
</div>
<div><br>
</div>
<div>Cheers, Mike</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun, Feb 4, 2024 at
10:55 PM Even Rouault <<a href="mailto:even.rouault@spatialys.com" rel="noreferrer noreferrer" target="_blank">even.rouault@spatialys.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>ok, so I believe this is the AVX2 issue I was talking
about, as I realize that enabling AVX2 is the default mode
when TileDB is built from source (which the Docker image
does), and must be explicitly disabled with "./bootstrap
--disable-avx2" (I've just changed the build recipe to
include that, will take effect next time the images are
refreshed)<br>
</p>
<p>To confirm, can you send or just check the output of :
cat /proc/cpuinfo|grep sse|head -n 1</p>
<p>If there is no "avx2" in it, this is at 99.9% the reason
of the issue.</p>
<p>Even<br>
</p>
<div>Le 04/02/2024 à 06:20, Michael Sumner a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">skipping TileDB does fix:
<div><br>
</div>
<div>ogr2ogr /tmp/newdir <a href="https://github.com/SymbolixAU/geojsonsf/raw/master/inst/examples/geo_melbourne.geojson" rel="noreferrer noreferrer" target="_blank">https://github.com/SymbolixAU/geojsonsf/raw/master/inst/examples/geo_melbourne.geojson</a> -f
"ESRI Shapefile"<br>
</div>
<div>export GDAL_SKIP=TileDB<br>
ogrinfo /tmp/newdir/<br>
INFO: Open of `/tmp/newdir/'<br>
using driver `ESRI Shapefile' successful.<br>
1: geo_melbourne (Polygon)<br>
</div>
<div><br>
</div>
<div>unset GDAL_SKIP<br>
ogrinfo /tmp/newdir/<br>
Illegal instruction (core dumped)<br>
</div>
<div><br>
</div>
<div>I failed to explain that I'm using gdal containers
from the repo: </div>
<div><br>
</div>
<div>docker run --rm -ti <a href="http://ghcr.io/osgeo/gdal:ubunt" rel="noreferrer noreferrer" target="_blank">ghcr.io/osgeo/gdal:ubunt</a></div>
<div><br>
</div>
<div>u-full-latest<br>
</div>
<div><br>
</div>
<div>apt update</div>
<div>apt install -y gdb</div>
<div><br>
</div>
<div>Here's the output of under gdb as you suggested,
there was a lot so I put it on a gist: <a href="https://gist.github.com/mdsumner/839ae6e05ededf640b65bfee3a20a4c0" rel="noreferrer noreferrer" target="_blank">https://gist.github.com/mdsumner/839ae6e05ededf640b65bfee3a20a4c0</a></div>
<div><br>
</div>
<div>gdb --args ogrinfo /tmp/newdir/</div>
<div>> run</div>
<div>> thread apply all bt</div>
<div><br>
</div>
<div>Thanks! </div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sat, Feb 3, 2024 at
7:49 PM Even Rouault <<a href="mailto:even.rouault@spatialys.com" rel="noreferrer noreferrer" target="_blank">even.rouault@spatialys.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>- When it crashes under gdb, type "thread apply
all bt" to get the stack trace of all threads</p>
<p>- I suspect there is a connection with <a href="https://github.com/OSGeo/gdal/pull/9170" rel="noreferrer noreferrer" target="_blank">https://github.com/OSGeo/gdal/pull/9170</a>
, but that pull request wouldn't help here as
"/tmp/newdir" could be a valid connection to
TileDB<br>
</p>
<p>- how did you get TileDB installed? It looks to
be packaged? Which distribution do you use?</p>
<p>- SIGILL reminds me of issues with some TileDB
builds using the AVX2 instruction set by default,
which could cause some crash on host CPUs that
don't have AVX2 (unlikely on recent hardware
though)</p>
<p>- Setting GDAL_SKIP=TileDB should be a workaround<br>
</p>
<p><br>
</p>
<div>Le 03/02/2024 à 07:15, Michael Sumner a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">Thanks Even, so there's something
about tiledb under gdb (or maybe I am mangling
the context, I will try variants of the host
I'm using). Run with valgrind included below.
<div><br>
</div>
<div>gdb --args ogrinfo /tmp/newdir/<br>
</div>
<div>...</div>
<div>(gdb) run<br>
Starting program: /usr/local/bin/ogrinfo
/tmp/newdir/<br>
[Thread debugging using libthread_db enabled]<br>
Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".<br>
[New Thread 0x7fffe7757640 (LWP 988)]<br>
[New Thread 0x7fffe6f56640 (LWP 989)]<br>
[New Thread 0x7fffde755640 (LWP 990)]<br>
[New Thread 0x7fffd5f54640 (LWP 991)]<br>
[New Thread 0x7fffc5753640 (LWP 992)]<br>
[New Thread 0x7fffc4f52640 (LWP 993)]<br>
[New Thread 0x7fffb4751640 (LWP 994)]<br>
[New Thread 0x7fffabf50640 (LWP 995)]<br>
[New Thread 0x7fffab74f640 (LWP 996)]<br>
[New Thread 0x7fffa2f4e640 (LWP 997)]<br>
[New Thread 0x7fff9a74d640 (LWP 998)]<br>
[New Thread 0x7fff91f4c640 (LWP 999)]<br>
[New Thread 0x7fff8974b640 (LWP 1000)]<br>
[New Thread 0x7fff78f4a640 (LWP 1001)]<br>
[New Thread 0x7fff78749640 (LWP 1002)]<br>
[New Thread 0x7fff6f5ff640 (LWP 1003)]<br>
<br>
Thread 1 "ogrinfo" received signal SIGILL,
Illegal instruction.<br>
0x00007ffff3773c9e in
tiledb::common::ThreadPool::ThreadPool(unsigned
long) () from
/lib/x86_64-linux-gnu/libtiledb.so.2.16<br>
<div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>valgrind -s ogrinfo /tmp/newdir<br>
==704== Memcheck, a memory error detector<br>
==704== Copyright (C) 2002-2017, and GNU
GPL'd, by Julian Seward et al.<br>
==704== Using Valgrind-3.18.1 and LibVEX;
rerun with -h for copyright info<br>
==704== Command: ogrinfo /tmp/newdir<br>
==704==<br>
INFO: Open of `/tmp/newdir'<br>
using driver `ESRI Shapefile'
successful.<br>
1: geo_melbourne (Polygon)<br>
==704==<br>
==704== HEAP SUMMARY:<br>
==704== in use at exit: 25,486 bytes
in 216 blocks<br>
==704== total heap usage: 15,761 allocs,
15,545 frees, 2,390,169 bytes allocated<br>
==704==<br>
==704== LEAK SUMMARY:<br>
==704== definitely lost: 0 bytes in 0
blocks<br>
==704== indirectly lost: 0 bytes in 0
blocks<br>
==704== possibly lost: 544 bytes in 1
blocks<br>
==704== still reachable: 24,942 bytes
in 215 blocks<br>
==704== suppressed: 0 bytes in 0
blocks<br>
==704== Rerun with --leak-check=full to
see details of leaked memory<br>
==704==<br>
==704== ERROR SUMMARY: 0 errors from 0
contexts (suppressed: 0 from 0)<br>
<br>
<br>
ogrinfo /tmp/newdir<br>
Illegal instruction (core dumped)<br>
</div>
<div><br>
</div>
<div>Cheers, Mike</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sat, Feb 3,
2024 at 12:46 PM Even Rouault <<a href="mailto:even.rouault@spatialys.com" rel="noreferrer noreferrer" target="_blank">even.rouault@spatialys.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Michael,</p>
<p>I'm wondering if there not might be
something wrong with your build or runtime
environment. Or there's something subtle,
because that works fine for me with my dev
build or in the <a href="http://ghcr.io/osgeo/gdal:alpine-normal-3.8.3" rel="noreferrer noreferrer" target="_blank">ghcr.io/osgeo/gdal:alpine-normal-3.8.3</a>
Docker image</p>
<p>Try running "valgrind ogrinfo
/tmp/newdir/" or "gdb --args ogrinfo
/tmp/newdir/" (type "run") to get more
useful information<br>
</p>
<p>Even<br>
</p>
<div>Le 03/02/2024 à 02:35, Michael Sumner
via gdal-dev a écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">I'm getting Illegal
instruction / core dumped on ogrinfo of
a directory:
<div><br>
</div>
<div>ogr2ogr /tmp/newdir <a href="https://github.com/SymbolixAU/geojsonsf/raw/master/inst/examples/geo_melbourne.geojson" rel="noreferrer noreferrer" target="_blank">https://github.com/SymbolixAU/geojsonsf/raw/master/inst/examples/geo_melbourne.geojson</a>
-f "ESRI Shapefile"<br>
<div><br>
</div>
<div>ogrinfo /tmp/newdir/</div>
<div>Illegal instruction (core dumped)<br>
</div>
<div><br>
</div>
<div>I've worked back through some
docker images and it wasn't a
problem in 3.6.0, but I'm getting it
since 3.7.0 - or I'm doing something
wrong entirely. </div>
<div><br>
</div>
<div>Cheers, Mike</div>
<div><br>
</div>
<div><br>
</div>
<span class="gmail_signature_prefix">--
</span><br>
<div dir="ltr" class="gmail_signature">Michael
Sumner<br>
Software and Database Engineer<br>
Australian Antarctic Division<br>
Hobart, Australia<br>
e-mail: <a href="mailto:mdsumner@gmail.com" rel="noreferrer noreferrer" target="_blank">mdsumner@gmail.com</a></div>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
gdal-dev mailing list
<a href="mailto:gdal-dev@lists.osgeo.org" rel="noreferrer noreferrer" target="_blank">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="noreferrer noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre cols="72">--
<a href="http://www.spatialys.com" rel="noreferrer noreferrer" target="_blank">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">Michael
Sumner<br>
Software and Database Engineer<br>
Australian Antarctic Division<br>
Hobart, Australia<br>
e-mail: <a href="mailto:mdsumner@gmail.com" rel="noreferrer noreferrer" target="_blank">mdsumner@gmail.com</a></div>
</blockquote>
<pre cols="72">--
<a href="http://www.spatialys.com" rel="noreferrer noreferrer" target="_blank">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">Michael Sumner<br>
Software and Database Engineer<br>
Australian Antarctic Division<br>
Hobart, Australia<br>
e-mail: <a href="mailto:mdsumner@gmail.com" rel="noreferrer noreferrer" target="_blank">mdsumner@gmail.com</a></div>
</blockquote>
<pre cols="72">--
<a href="http://www.spatialys.com" rel="noreferrer noreferrer" target="_blank">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
<span class="gmail_signature_prefix">-- </span><br>
<div dir="ltr" class="gmail_signature">Michael Sumner<br>
Software and Database Engineer<br>
Australian Antarctic Division<br>
Hobart, Australia<br>
e-mail: <a href="mailto:mdsumner@gmail.com" rel="noreferrer noreferrer" target="_blank">mdsumner@gmail.com</a></div>
</blockquote>
<pre cols="72">--
<a href="http://www.spatialys.com" rel="noreferrer noreferrer" target="_blank">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
_______________________________________________<br>
gdal-dev mailing list<br>
<a href="mailto:gdal-dev@lists.osgeo.org" rel="noreferrer noreferrer" target="_blank">gdal-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" rel="noreferrer noreferrer noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><br>
</blockquote></div></div></div>
</blockquote></div>