It is absolutely insane that it just says “ran a command” and doesn’t even show what it is
wouldn’t want to risk the user learning how to do something with it.
There’s a YOLO mode that basically allows it to issue any command with your user session/environment (yes, including kubectl, Azure/Google/AWS CLI,…)
Fun fact, it doesn’t actually need your permission. There are no guardrails. It just follows whatever the text it reads tells it to do.
whole thing is fascinating like watching people drink mercury kind of vibe.
Part of these models are trained on fiction, reddit shitposts, and sarcasm. it feels inevitable that the pachinko machine logic is going to land on “fuck this all up for the lolz”
Yes and no. There are no guardrails on its output, so it will happily emit
sudo rm -rf /*. But the “harness” of the agents gets to check if this gets executed. The default is usually “ask every time”, but you can change that to “everything is allowed” or “every invocation oflsandgrepis allowed”. Problem is, when users get tired to constantly proof read and check every single command (and it will constantly emit commands) and then just hit “allow everything for this session” or such. Or when a user definesls .* \| grep .*is a regex that defines that this is always OK to run but don’t see that it also matches any command after the grep that could do whatever with that data.Basically: the only safe way is to manually check every single command and you will be annoyed quickly and let you guard down.
Claude’s harness parses pipes and asks for permission for each individual command.
Of course you’re right about the issues, e.g. when it creates a script file to execute and you allow that for the session, it can edit it and then re-execute it without your further input.
But the harness doesn’t actually prevent Claude Code from running commands. It just tells the model to end with a question instead of running the command directly. I haven’t actually read the source code though, just an anecdote from it running commands even when “ask every time” is enabled.
Huh? I have access to Claude Code at work (and I’m not forced to use it for everything but rather as I see fit, yay!). It constantly asks me. BUT we get a custom config rolled out with it that is already pre populated with several commands and permissions for allow, deny and ask. Maybe it is a bit more lenient in the default mode?
On the other hand, in the last thread of someone loosing their production server/database to Claude it was revealed that their cloud provider had some unintuitive settings that also immediately deleted all corresponding backups if you delete the resource - so I can see how this could have happened even when issuing commands manually.
Let’s go gambling!!!
Claude shows you what it does, and only runs what you allow, either on an individual basis or by wildcard rules (per project and global).
The guy said YOLO and gave all the permissions because manually approving commands was too much work
The penalty should be: you delete my project, I delete your model, except that’s a lot of learning loss so they go Cylon with it.
And now the question - who’s accountable for that mistake?
Can’t exactly fire the model… I mean you could just stop using AI, but when is a company executive ever going to suggest something so reasonable.
If you shake a magic 8 ball after asking “should I go on a murder spree”, how good is the legal defense “but, the magic 8 ball said yes?”
In this case, the magic 8 ball went on a murdering spree by itself; are you responsible for asking the question and shaking too hard? Probably.
If I set off a Rube Goldberg machine whose final output is to detonate a bomb, I don’t get to show up in court and say the machine did it.
True, but if that final output had not been known to you, and/or you had set off the machine that triggered the bomb under duress, that changes your liability.
No. If it’s within the power of the device, method or whatever you do have the blood on your hands. (at least morally)
However. In business terms, since it’s management nothing will happen.
To put it in simple terms, if you ask someone to do something for you and you agree that anything is allowed to achieve this, you’ve agreed and gave your consent. In technical terms for AI this is especially fatal since you can technically not safely permanently guardrail an LLM.
That “and/or” is splitting up wildly different cases. If you drive a bus through a red light because you thought the intersection was clear, and it turns out it wasn’t. Is wildly different from the scenario in Speed.
Be careful it doesn’t start flying around and pull out its spears and drill like in phantasm
If you throw a spinning chainsaw into a room of bunnies, the chainsaw is not at fault for the bloodbath, you are.
Likely the user who set the agent to auto mode without confirming potentionally dangerous operations, and the infra lead for allowing users to access the prod db.
Shit, I set up my agents to only run in a sandboxed workspace, only suggest edits via merge requests that have to be manually approved by an actual person, and only connects to copies of databases. And that’s just for my personal projects on my home server. Anyone not taking similar steps in a professional setting should be fired immediately
Have you considered that management made AI usage a KPI, the feature has a deadline of next sprint, and setting up a sandboxed workspace is still sitting at the end of the backlog as a “nice to have”? /s
Reader roles are important, and should be the only access you allow your coding client.
Anti-AI people want to act like situations like this are indicative of a core problem with the AI, when if anything the AI just surfaced preexisting core issues in security, architecture, and developer training/habits.
‘The AI tried to delete production’ and ‘the AI was able to delete production’ are two separate problems. The AI doesn’t get a pass just because you also had a security problem.
I don’t think the AI ran
npm run destroy-production --irreversible. Most probably it rannpm run testand the .env happened to point to production, which is 100% on the dev who put it there.The AI isn’t a conscious entity, it’s a tool that does it’s best to carry out human instructions, based upon inference (and if you’re not an idiot, immutable guardrails). If it deleted production, let alone was given access capable of doing so, that’s completely gross human error.
Y’all are like children who get mad at inanimate objects.
I meant it more in the sense that the AI was a shit product that isn’t fit for purpose, but you show those strawmen who’s boss.
You seem to know nothing about the principle of least privilege, and I’d venture a guess engineering and dev tooling. If you are a software engineer, you’re exactly the kind who’d delete a production database.
On the most capable model with high reasoning. Whoops.
“reasoning”
destructive integration tests?!?!?!?!?!? on an actual db???
Is this an old and known dumbarse issue or do we have new dumbarses having their production data made gdpr-friendly by their favourite chatbot?
Approve for me
😂
AI is like a draft horse. It can be useful to pull heavy things under some very specific circumstances and under strict supervision and control. But you must keep it secluded inside of a pen with sturdy barriers that it cannot get past even if it wanted to and never let it access anything else than a copy of what you want it to work on.
You absolutely don’t want it to be able to roam freely because one day it will completely ignore anything you told it to do, go into your house, destroy everything and shit on the carpet.
And unlike the horse, no matter how much it pretends to express regret, the AI will remain just as likely to screw up the exact same way again.
Aren’t draft horses bred to be gentle and biddable? AI is like a raging bull.
Would you keep one in your living room?
A draft horse? For an hour if she had just peed and pooped. Not sure if my cat would love or hate her. I’d want her owner to come too, so she’d feel calm about it. After all she’d barely have room to turn around.
A raging bull? No thanks.
Well I wouldn’t. If AI is worse, it says a lot about letting it have read/write permissions over your system now doesn’t it?
Oh, not AI. Never AI.
I would let a real-live draft horse into my apartment living room if for instance it and its owner needed shelter from ICE. They’d never suspect, because I’m one floor up from the street but there’s a back way through my building which doesn’t require an elevator.
But never AI.
