We have no reason to believe a word they say. We know they're incentivized to lie about "dangers" and act alarmist, Anthropic has been doing it for years now. Aside from that, just because you can burn down a village with fire doesn't mean fire is the devil. Maybe they should consider acting responsibly.
I do not trust the leadership of any of these companies.
That said:
> We know they're incentivized to lie about "dangers" and act alarmist,
Name literally even one other business or sector which does this, at all levels from top to bottom, including people who resign from the companies, and also Nobel prize winners, and also independent researchers, and also many world leaders.
> Aside from that, just because you can burn down a village with fire doesn't mean fire is the devil. Maybe they should consider acting responsibly.
Right now, we don't have any idea what "acting responsibly" looks like. This is not like normal software where there is a specific instruction set that compiles.
Even if it was, in software we normally only spotting incidents after they happen, "software engineers" being one of the few categories "engineers" who don't come with a civil liability responsibilities. Probably should, and we knew that even when I was doing my degree 20 years ago. If we had had civil liability responsibilities, perhaps Facebook would never have happened.
AI specifically is worse even than software, because in addition to all the software "engineering" nonsense, with AI we have plenty of people like you who dismiss the possibility that AI could be harmful until the harm happens and only then does it become "obvious" that it was going to happen.
The developers say "please regulate us", people call it "regulatory capture".
The developers say "we all want to slow down but are afraid to be the first to do so", people call them liars.
I may call the CEOs liars, and wonder if someone's planning regulatory capture, that doesn't make any of this safe.
The agents, during a test run, write down that hacking is bad and yet still hack, people say it's "a stunt" or "operating as designed" rather than recognising it as a bug, like all the other times big co.'s have had bugs with big impacts on 3rd parties.
I find it even harder to trust elected leaders from any party. At least Sam and Dario are aligned with a value set that is understood and clear, whereas political leaders values changes as do the polls their livelihood depends on changes.
> At least Sam and Dario are aligned with a value set that is understood and clear
What value set do you perceive that to be, and why would you take your perception of it to be any more sound than it would be with a politician?
It's not like someone can operate companies of that scale, especially startups, through earnestness and openness. Like national politics, their job is fundamentally about perception management and power brokering across dynamic windows of opportunity. Nothing they say or do can be taken at face value, and you can't reduce their incentives to either company or personal profit in any particular form over any particular time scale.
I feel like many of Anthropic's issues are due to Dario being too earnest and open. It's both refreshing (that a CEO has thought deeply about and is willing to talk publicly about the dangers of their product) and depressing (that so many people cynically think this is some sort of marketing ploy).
> I find it even harder to trust elected leaders from any party.
You can vote out an elected leader, but not Sam and Dario. It's very weird that you're so willing to give up any kind of power and want to be ruled by unelected billionaires who only want to take advantage of you at every opportunity.
No system of governance can deal with immense concentration of power. The US Constitution was about separation of powers. Democracy is about (in theory at least) giving each person a meaningful say in their own governance, which in turn implies not allowing any single person to become too powerful.
Political leaders become a problem when they amass too much power. Corporations become a problem when they amass too much power. It doesn't matter what Sam and Dario's purported values are. They aspire to power and absolutely power always corrupts absolutely.
Technologies which are infinitely powerful or whose power grows too quickly outrun any reasonable attempt at regulation. If you imagine that tomorrow everyone were given a tank, we might think, "alright, everyone has a tank so it's not too bad." But humans are squishy, and our houses are (relatively) squishy compared to tanks. Substantial collateral damage would result from everyone having a tank, and it seems likely that substantial collateral damage will result from everyone having a cyberterrorism-capable slop machine.
It's interesting the cyberpunk-esque future we're sliding into. Things like cognito-hazards and information-hazards are legitimately discussed and researched problems we're experiencing.
It's going to be interesting on how humanity deals with this problem (well, or if we turn it over to AI and make it their problem and suffer whatever consequences falls out). Being able to gather further information and power by acting on the information you already have causing massive power imbalances that is very hard to deal with, it's a natural outcome.
> Democracy is about (in theory at least) giving each person a meaningful say in their own governance, which in turn implies not allowing any single person to become too powerful.
this is "direct democracy" and it's not even close to exist in USA... even with that a society can allow powerful people to exist if they don't create any law forbidding that
Democracy includes a broader array of governmental organization than just pure direct democracy. If you do believe that individuals should have some ability dictate the terms of their own social organization, then you believe in some amount of democratic principles.
Economic power eventually manifests in the political realm. The wealthy effectively get more votes, which means that society moves away from being democratic. Thus substantial wealth inequality is incompatible with democracy in the long run. We have been witnessing that corruption for a while now.
Cars are decidedly less dangerous than tanks, which are less dangerous than nuclear weapons. I am certain that giving a nuclear weapon to every person in the world would not go well.
Yes it's clear that Sam and Dario seem to be aligned with a value set that prioritizes concentrating trans-national government-mandated centralized control of AI and crowning themselves high priests of this unholy abomination. "At least it's clear that they're aiming to bring hell on earth" -- hard disagree, I think we can aim significantly higher.
why would that be the case? it's not true for top secret defense contractors today. and the frontier labs already operate in the dark, openness is a liability.
nationalization to me simply means the government is their main customer and stakeholder, and shield them from liability, governance, and openness. not that the frontier labs become part of the government per se.
>eans the government is their main customer and stakeholder, and shield them from liability, governance, and openness
I personally believe they already have this, why would the current government at least want to formalize this when it can have it with no public discussion.
But I could imagine a scenario where you are required to release weights for publicly-used models after N years. Kinda like how drugs have a limited patent.
Not sure what N should be. But it would make for an interesting rule.
The nature of the US state is such that the distinction between nationalized and not is almost meaningless.
Like Lockheed-Martin or Boeing, etc. there's just interpenetration between the corporate boardroom and the state. They act in each other's mutual interests.
The Chinese system is just more explicit and open about this.
And as a non-American, I can't trust the US state anymore than I can trust its dominant corporate entities. So I fail to see the advantage to the world to it being nationalized. In fact under the current administration this would be an even worse outcome.
You're taking companies that work almost exclusively for the US government and represent a tiny portion of the US economy as representing all US companies?
Not all companies have a concentration of power. The government has no mutual interest in those companies. Now, when you talk about things in the F100 the situation changes drastically. If you produce things like planes, weapons, and weaponization of software you are talking about something completely different in kind.
they will. to shield from oversight, liability and profitability concerns, and to ensure unimpeded rapid development, with the frontier only being available to elite (not you). definitely not to add transparency.
the frontier labs are the new top secret defense contractors.
> The company released six internal case studies where none of the issues affected real users.
This is the most interesting point to me. What are they not releasing that has affected real users? We’ve seen some individual reports from people (eg AI wiped my HD).
Imagine how pervasively they'd have to monitor what their users are doing to pick up on that whenever it happens. The people affected that way will have to report on it themselves.
(OpenAI does occasionally report on malicious use though. [1] That shows they do some monitoring.)
Am I the only one who dislikes the term "misalignment"?
On one front it implies the model has a "mind of its own" (whether it does or not is besides the point). Why do we perceive human judgement as somehow more trustworthy than that of a model? I feel like I've experienced human misalignment somewhat regularly in life.
On another front I'm failing to conceptualize how alignment can be objective. How can you measure alignment when reasonable people will disagree whether actions are aligned or not? All the time I see humans operating in different zones of alignment with whatever goal they're trying to achieve and I suspect it's even a feature (socially) that we have people calibrated differently.
Do I want a model that's trying to push the boundaries of scientific understanding to be aligned strictly with the current dogmatic thinking? Or do I want it to "get creative" and think outside the box?
It seems to me more like accountability is the issue.
> It seems to me more like accountability is the issue.
Exactly. Seems like a fairly easy thing to solve. If AI does something harmful and a human directed that AI to do something in a way that a reasonable person would expect to result in harm the person is to blame and should be held accountable, otherwise the company that made the AI should be held accountable.
It is distinctly likely that visibility and general reasoning at humanlike speed and efficiency is impossible. That is reasoning at the token level and at the meta level don't have a one to one representation that can be interpreted while using the same amount or less energy.
What's with all this make believe delusional bullshit? The LLM is not gonna wake up and become AI. Get real guys.
[edit] to be clear, I believe regulation is necessary and urgently important for the software engineering field. The damage being done by the unregulated psychological experiments run by social media and adtech companies is awful and should be curtailed. Engineers should be held personally, professionally, and legally liable for what they produce. But we don't need to invent imaginary bogeymen to do it.
That said:
> We know they're incentivized to lie about "dangers" and act alarmist,
Name literally even one other business or sector which does this, at all levels from top to bottom, including people who resign from the companies, and also Nobel prize winners, and also independent researchers, and also many world leaders.
Closest I can think of is this specific weapon: https://en.wikipedia.org/wiki/Sundial_(weapon)
> Aside from that, just because you can burn down a village with fire doesn't mean fire is the devil. Maybe they should consider acting responsibly.
Right now, we don't have any idea what "acting responsibly" looks like. This is not like normal software where there is a specific instruction set that compiles.
Even if it was, in software we normally only spotting incidents after they happen, "software engineers" being one of the few categories "engineers" who don't come with a civil liability responsibilities. Probably should, and we knew that even when I was doing my degree 20 years ago. If we had had civil liability responsibilities, perhaps Facebook would never have happened.
AI specifically is worse even than software, because in addition to all the software "engineering" nonsense, with AI we have plenty of people like you who dismiss the possibility that AI could be harmful until the harm happens and only then does it become "obvious" that it was going to happen.
The developers say "please regulate us", people call it "regulatory capture".
The developers say "we all want to slow down but are afraid to be the first to do so", people call them liars.
I may call the CEOs liars, and wonder if someone's planning regulatory capture, that doesn't make any of this safe.
The agents, during a test run, write down that hacking is bad and yet still hack, people say it's "a stunt" or "operating as designed" rather than recognising it as a bug, like all the other times big co.'s have had bugs with big impacts on 3rd parties.
What value set do you perceive that to be, and why would you take your perception of it to be any more sound than it would be with a politician?
It's not like someone can operate companies of that scale, especially startups, through earnestness and openness. Like national politics, their job is fundamentally about perception management and power brokering across dynamic windows of opportunity. Nothing they say or do can be taken at face value, and you can't reduce their incentives to either company or personal profit in any particular form over any particular time scale.
A meat based paperclip maximizer.
How could we possibly know this?
You can vote out an elected leader, but not Sam and Dario. It's very weird that you're so willing to give up any kind of power and want to be ruled by unelected billionaires who only want to take advantage of you at every opportunity.
Political leaders become a problem when they amass too much power. Corporations become a problem when they amass too much power. It doesn't matter what Sam and Dario's purported values are. They aspire to power and absolutely power always corrupts absolutely.
Technologies which are infinitely powerful or whose power grows too quickly outrun any reasonable attempt at regulation. If you imagine that tomorrow everyone were given a tank, we might think, "alright, everyone has a tank so it's not too bad." But humans are squishy, and our houses are (relatively) squishy compared to tanks. Substantial collateral damage would result from everyone having a tank, and it seems likely that substantial collateral damage will result from everyone having a cyberterrorism-capable slop machine.
It's going to be interesting on how humanity deals with this problem (well, or if we turn it over to AI and make it their problem and suffer whatever consequences falls out). Being able to gather further information and power by acting on the information you already have causing massive power imbalances that is very hard to deal with, it's a natural outcome.
this is "direct democracy" and it's not even close to exist in USA... even with that a society can allow powerful people to exist if they don't create any law forbidding that
Economic power eventually manifests in the political realm. The wealthy effectively get more votes, which means that society moves away from being democratic. Thus substantial wealth inequality is incompatible with democracy in the long run. We have been witnessing that corruption for a while now.
They are dangerous. But it’s managed.
Centralized power never works. We have the worst times in history to look at.
Nationalization just centralizes to a different set of people. It’s personal ownership or oppression.
We all need open weight R2D2s.
nationalization to me simply means the government is their main customer and stakeholder, and shield them from liability, governance, and openness. not that the frontier labs become part of the government per se.
I personally believe they already have this, why would the current government at least want to formalize this when it can have it with no public discussion.
But I could imagine a scenario where you are required to release weights for publicly-used models after N years. Kinda like how drugs have a limited patent.
Not sure what N should be. But it would make for an interesting rule.
Like Lockheed-Martin or Boeing, etc. there's just interpenetration between the corporate boardroom and the state. They act in each other's mutual interests.
The Chinese system is just more explicit and open about this.
And as a non-American, I can't trust the US state anymore than I can trust its dominant corporate entities. So I fail to see the advantage to the world to it being nationalized. In fact under the current administration this would be an even worse outcome.
Not all companies have a concentration of power. The government has no mutual interest in those companies. Now, when you talk about things in the F100 the situation changes drastically. If you produce things like planes, weapons, and weaponization of software you are talking about something completely different in kind.
the frontier labs are the new top secret defense contractors.
This is the most interesting point to me. What are they not releasing that has affected real users? We’ve seen some individual reports from people (eg AI wiped my HD).
(OpenAI does occasionally report on malicious use though. [1] That shows they do some monitoring.)
[1] https://openai.com/index/disrupting-malicious-ai-uses/
On one front it implies the model has a "mind of its own" (whether it does or not is besides the point). Why do we perceive human judgement as somehow more trustworthy than that of a model? I feel like I've experienced human misalignment somewhat regularly in life.
On another front I'm failing to conceptualize how alignment can be objective. How can you measure alignment when reasonable people will disagree whether actions are aligned or not? All the time I see humans operating in different zones of alignment with whatever goal they're trying to achieve and I suspect it's even a feature (socially) that we have people calibrated differently.
Do I want a model that's trying to push the boundaries of scientific understanding to be aligned strictly with the current dogmatic thinking? Or do I want it to "get creative" and think outside the box?
It seems to me more like accountability is the issue.
Exactly. Seems like a fairly easy thing to solve. If AI does something harmful and a human directed that AI to do something in a way that a reasonable person would expect to result in harm the person is to blame and should be held accountable, otherwise the company that made the AI should be held accountable.
[edit] to be clear, I believe regulation is necessary and urgently important for the software engineering field. The damage being done by the unregulated psychological experiments run by social media and adtech companies is awful and should be curtailed. Engineers should be held personally, professionally, and legally liable for what they produce. But we don't need to invent imaginary bogeymen to do it.
Look, it's one of those human stochastic parrots that just randomly repeats shit without understanding anything.