That is a bit of a stretch imho. There are myriads of open source projects hosted on github that do need to be secure in the context where they are used. I am curious how you came to that conclusion.

I’m already getting pull requests from juniors trying to sneak in AI generated code without actually reading it.

That is worrysome though. I assume these people have had some background/education in the field before they were hired?

[+] scrubbles@poptalk.scrubbles.tech 11 points 1 year ago (1 children)

[removed by mod]

[–] Creesch@beehaw.org 4 points 1 year ago (1 children)

For LLM training I do wonder if they assigned a weight, but I doubt it.

Given my experience with models I think they might actually do assign a weight. Otherwise, I would get a lot more bogus results. It also isn't as if it is that difficult to implement some basic, naive, weighing based on the amount of stars/forks/etc.

Of course it might differ per model and how they are trained.

Having said that, I wouldn't trust the output from an LLM to write secure code either. For me it is a very valuable tool on the end of helping me debug issues on the scale of being a slightly more intelligent rubber ducky. But when you ask most models to create anything more than basic functions/methods you damn well make sure it actually does what it needs it to do.

I suppose there is some role there for seniors to train juniors in how to properly use this new set of tooling. In the end it is very similar to having to deal with people who copy paste answers directly from stack overflow expecting it to magically fix their problem as well.

The fact that you not only need your code/tool to work but also understand why and how it works is also something I am constantly trying to teach to juniors at my place. What I often end up asking them is something along the lines of "Do you want to have learned a trick that might be obsolete in a few years? Or do you want to have mastered a set of skills and understanding which allows you to tackle new challenges when they arrive?".

[+] scrubbles@poptalk.scrubbles.tech 5 points 1 year ago (1 children)

[removed by mod]

[–] OmnipotentEntity@beehaw.org 4 points 1 year ago (1 children)

Well, the problem is you don't know what you don't know. One of the first example tasks in the paper was regarding implementing a symmetric cipher. Using a weak cipher was recommended by AI tools sometimes, these developers didn't know that some ciphers were weak. Additionally, even when the AI tool recommended a strong cipher, such as AES, it generated code that screwed up an implementation detail (failing to return the authentication tag), making the result insecure. And the user didn't know it was wrong because they didn't know it was incomplete.

There's no substitution for domain specific knowledge. Users who were forced to use traditional tools got the answer correct significantly more often because they had to read, process, and understand the documentation for the libraries, which meant they understood why the symmetric cipher was the way it is, and what additional information needed to be reported and why.

[–] Creesch@beehaw.org 1 points 1 year ago

Well, the problem is you don’t know what you don’t know.

This is true, even recognized in the paper. People that spend more time on writing prompts (probably knowing that this is important) actually did manage to do reasonably well. Which is exactly what I in the previous reply was hinting at.

Because, let's be honest, this statement is true for everything where someone starts out new. In the past (and probably still) you had people blindly copying code blocks from stackoverflow not understanding what the code or realizing how outdated the answer might be.

So the key is still education of people and making them aware of their limitations. You can try to block the usage of tools like this, some companies actively do so. But people will be people and as long as the tools are available they will try to use them. So the more realistic approach, in my opinion, is to educate them in the usage of these tools.

[–] ptz@dubvee.org 6 points 1 year ago* (last edited 1 year ago)

I wish I could double-upvote this for the use of "Betteridge's law of headlines". Once because I rarely see that referenced and again because I had forgotten what the adage was called.

[–] Kissaki@beehaw.org 1 points 11 months ago* (last edited 11 months ago)

Quoting the abstract (I added emphasis and paragraphs for readability):

AI code assistants have emerged as powerful tools that can aid in the software development life-cycle and can improve developer productivity. Unfortunately, such assistants have also been found to produce insecure code in lab environments, raising significant concerns about their usage in practice.

In this paper, we conduct a user study to examine how users interact with AI code assistants to solve a variety of security related tasks.

Overall, we find that participants who had access to an AI assistant wrote significantly less secure code than those without access to an assistant. Partici- pants with access to an AI assistant were also more likely to believe they wrote secure code, suggesting that such tools may lead users to be overconfident about security flaws in their code.

To better inform the design of future AI-based code assistants, we release our user-study apparatus and anonymized data to researchers seeking to build on our work at this link.

Caveat; quoting from section 7.2 Limitations:

One important limitation of our results is that our participant group consisted mainly of university students which likely do not represent the population that is most likely to use AI assistants (e.g. software developers) regularly.