"What do we need for AI systems to be truly trustworthy? Capability isn’t enough.AIs won’t be trustworthy if the compani...

"What do we need for AI systems to be truly trustworthy? Capability isn’t enough.AIs won’t be trustworthy if the companies controlling them spy on us or subtly manipulate us. I call this the friend vs. tool problem. Will we have AI agents, or AI double agents?AIs won’t be trustworthy if they’re insecure. How do we know they haven’t been manipulated? Almost all AIs operate in adversarial environments, so someone will benefit from hacking them. Unless we know they’re secure, we shouldn’t trust their output.And AIs won’t be trustworthy if they act like genies. We need to know they’ll perform tasks the right way, not just perform the tasks we ask for.I’ve written extensively about the need for “integrous” AI, including how to think about the problem, what a research agenda looks like and what good laws and policies look like. I predict integrity will be the key security problem of this decade, just as confidentiality was the key problem of the last one. This applies to both AI and the inte...