[QGIS-Developer] Floating an idea: ban AI based contributed from non-core developers?

[Julien Cabieces]

Hi all,

>> One key point here. It is really **hard** to decrypt the quality of
>> LLM generated code, because it is trained to be plausible.
>>
> I triple on that. I've always found that (in depth) reviewing was a
> task consuming a lot of mental energy, often more than coding, but in
> pre-LLM times, you knew that the contributor had already invested a
> lot of energy into creating their submission, so that made your own
> energy investment acceptable. Now, the reviewer is mostly the only one
> that sweats.  When you make comments and get a reply, you never know
> if the reply comes from the human, the machine or a mix of them. This
> is really really really unpleasant.  In our highly decentralized
> community, the lack of direct interaction pre-LLM could also be a
> challenge (mechanisms behind the working of human psychology have been
> forged by natural selection over ~ 3 million years of face-to-face
> interactions, add tens of extra millions years inherited from our
> closest cousins in the animal reign, compared to hardly 50 years of
> digital communication), but at least you knew that at @someone was a
> peer.  Now you don't know anymore. You're becoming effectively the
> slave of an agent.
>
> Among the things I dislike is the wall'o'text LLM generated PR
> descriptions. I believe we should ban them and require contributors to
> come with their own succinct and to-the-point descriptions.

+1

I'm in favor of banning the use of AI in PR descriptions and
comments. We ask for the human-in-the-loop to fully understand every
line of code possibily generated with AI and, so, like the quote said
"If you can't explain it simply, you don't understand it well enough."

So, is this too much to ask to write a succint PR description, and to
answer comments without using AI?


> Or if they
> can't come up with one, just none. That makes at least less stuff to
> review!  But then the "human in the loop" policy no longer works.
>
> Maybe we should charge a symbolic fee to new contributors for each PR
> they submit, until one is accepted? I bet this would make them think
> twice. Unfortunately that's not a github feature.
>

I'm wondering if all contributors are aware about the fact that reviewing PR is a lot of work
and somebody should pay for it.

Shall we modify our template to invite new contributors to get in touch
with core developers (or maybe QGIS.org) to fund the review of their
PRs ? especially if it is AI-assisted because it requires more thorough
review.

I was wondering also if we should not add a label "Funded review" to
publicly show that some PRs have a designated and funded reviewer
(exchanging PR reviews between core developer is a kind of funded review).


>>  Knowing this, a friendly contributor, willing to getting involved
>> in the long term, and who will use AI as a helper for its first
>> contributions, should know that it will put a very high charge on
>> reviewers. So this is like a poisoned gift.
>>
> One difficulty is that it is likely really hard for someone not having
> already experienced themselves reviewing LLM generated PRs to know how
> it feels to deal with them.
>
> Writing all the above with https://github.com/qgis/QGIS/pull/65778 in
> mind. Took me maybe one hour of digging (yes not fast but CMake is
> full of subtelties) to realize that there is (likely) a much simpler
> fix than the one suggested (I may be wrong, let's see what the bot
> answers). I believe that 6 months ago I would have approved the exact
> same PR content without much thinking. So one could argue that the
> extra skepticism when confronting to LLM generated PR is a good thing,
> but frankly it is exhausting. I believe there's a new job lurking:
> "mental coach/psychologist for developers dealing with LLM
> contributions"


Regards,
Julien


-- 

Julien Cabieces
Senior Developer at Oslandia
julien.cabieces at oslandia.com