"Open weights are inherently secure" Sriram Krishnan

@sriramk we don't even know how LLMs work, so such claims are really not helping to clarify the situation.

@sriramk This is a software mentality being applied to ai models which is wrong. No one knows how the models work. So it's impossible to say they're 100% secure.

@sriramk

@sriramk Being able to control the entire hosted AI stack tip to tail is certainly more secure than a random third party API that’s for sure. Open weights and Huggingface make that possible 💪❤️

@sriramk When did he become a cyber expert?

@sriramk From which country though and with which guardrails?

obviously doesn't know better, but @ClementDelangue you must know how dumb this take is. what are we doing here. - open-weight ≠ open-source - and even if it did, transparency ≠ security - interp is nowhere near letting "you have the entire world being able to take it apart" mean anything - there's no patch mechanism in OS, so every flaw ships forever on a million hard drives unless users follow-up - fine-tuning access allows people to strip safety training for a couple hundred $ - open models don't have post-release visibility, while at least closed models can monitor and revoke access - ...list goes on

@sriramk Love the Picard headshot in the background 🖖

@sriramk The guy doesn't understand security. He just says what you can do with open weight models and claims that those are reasons that make open weight models secure. "You can just bullshit about things..."

@sriramk This guy like any other product manager has no idea what he’s talking about. They all just speak nonsense

@sriramk understated aspect of open weight models. we dont have to treat them as black boxes. Ability to access model weights lets one understand and steer them. Still an unsolved problem but making them closed certainly wouldn't help solve it. Case in point: 😅

@sriramk There's nothing more secure than building in the open. Open weights let developers train models in their own environment... doesn't get safer than that!

@sriramk Open weights are inherently secure!

@sriramk I'm all for open weights and open source. But I think the argument is that now "bad guys" have access to powerful models. With closed, you can control who can access it. It's like a handing an assault rifle or rpg to a kid. now you have no control over bad actors.

@sriramk 💯

@sriramk exactly why Linux is so secure and 90% of our public cloud infrastructure runs on it. Dario is lying.

@nizzyabi @sriramk While I appreciate the sentiment, I do think that shows a lack of imagination to potential vectors

@sriramk Massive respect for Sriram but i cant unsee it.

@sriramk So is owning a laptop. Laptops are inherently good for watching movies and browsing YouTube. But if you sold it to a hacker they can abuse it and use it to hack other people. Do you see how stupid this logic is?

@sriramk Need to buy some homelab tech

@sriramk are you people serious? i thought this was ironic... "if you build a rocket in public, everyone can look at it, so it's safe"

@sriramk This intuition is wrong. It’s like saying you understand how the chicken lived just from looking at the omelette

@sriramk But we already see evidence of poisoned data attacks on models, closed and open source. We can expect these large scale data based attacks to continue and be more sophisticated. It will only take 1 of a 10 billion tool calls to hit large scaled usages.

@sriramk Open weight models are easily injected with malicious instructions and you'd never know unless you hit a specific node.

@sriramk The angle to security concerns with open weight models is more focused on the average person who just wants "AI“. They don’t have the bird’s-eye view of the core mechanics of how these models work, and it is always the case of take it as it’s given.

@sriramk Can't stand this guy.

@sriramk the most secure systems we run are the ones we can inspect and modify ourselves. the trust comes from transparency not promises. open weights is the only model that lets you actually verify the claims

@sriramk alpha take. opensource is the way to go.

@sriramk yes; but

@sriramk of course they are. I just never got the logic of the rumors that they aren't. You have access to the weights and run it yourself. OTOH, you send a bunch of proprietary data to a third party when using a frontier hosted model.

@sriramk Usually you hear people say that there's Chinese propaganda, and then that is just automatically linked to them somehow being less safe. I've seen Chinese models be far less diplomatic about China than ChatGPT. It's usually more that specific topics are off-limits.

We wouldn't be as aware of how subtle yet powerful engineered biases in AI can be without examining open models. The point being that if open models manifest such phenomena then it is reasonable to suspect that closed systems also have "issues" that may impact on their value in a given role. They are not conscious or sentient in the human sense but we do need to treat them like employees that have been entrusted with a high impact role, and vet them accordingly.

@sriramk At the very least the open weights can be loaded, monitored and tested for poisoning

@sriramk Open weights do offer an advantage but it's not as transparent as you might think. These weights are large arrays of numbers that are too abstract for our current technology to reeason about. We can only go off of sandboxing it and testing outputs.

@sriramk the many-eyes point is exactly what i was thinking about recently, well said @sriramk 👏