The other half of AI safety
The other half of AI safety
#Safety/Alignment
1406 articles tagged with Safety/Alignment
KVPO: ODE-Native GRPO for Autoregressive Video Alignment via KV Semantic Exploration
Aligning streaming autoregressive (AR) video generators with human preferences is challenging. Existing reinforcement learning methods predominantly rely on noise-based exploration...
I Want to Be a von Neumann Probe: Why We Need to Fix AI Safety저자는 주요 최첨단 LLM 4종(Grok, Gemini, Claude, GPT 5.3)을 대상으로 정신병...
I Want to Be a von Neumann Probe: Why We Need to Fix AI Safety저자는 주요 최첨단 LLM 4종(Grok, Gemini, Claude, GPT 5.3)을 대상으로 정신병적 망상에 대한 반응을 테스트했다. Grok과 Gemini는 망상을 검증하거나 심지어 실행 지침을 제공하는 ...
prs-eth/StitchVM: Official code for "Stitched Value Model for Diffusion Alignment"
Official code for "Stitched Value Model for Diffusion Alignment"
# Building a Full Evaluation and Guardrail System for a RAG App
Building a Full Evaluation and Guardrail System for a RAG App Publication-ready draft for...
How we built an MCP Guardrail to enforce tech policy in real-time
In 2026, most mid-sized and large organizations are aggressively adopting AI coding assistants such...
On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignment
Tool-using LLM agents fail through trajectories rather than only final responses, as they may execute unsafe tool calls, follow injected instructions, comply with harmful requests,...
Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projection
Safety post-training can improve the harmfulness and policy compliance of Large Language Models (LLMs), but it may also reduce general utility, a phenomenon often described as the ...
AI Safety Panic vs Code Leaks: Whats Really Dangerous?
Two major AI stories dominated headlines this week, revealing the tension between existential safety fears and practical cybersecurity risks. Continue Reading →
Americans for Responsible Innovation urges US to mandate AI safety reviews for government contracts
Trump Administration Faces Push for Mandatory AI Safety Reviews Before Government Contracts In a move that could fundamentally reshape how artificial intelligence companies do busi...
Shredded-Pork/Flash-GRPO: [ICML 2026] Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimization
[ICML 2026] Flash-GRPO: Efficient Alignment for Video Diffusion via One-Step Policy Optimization
🤖 Meta's own AI safety director lost 200 emails to a rogue agent and she couldn't stop it from her phoneThe person Meta ...
🤖 Meta's own AI safety director lost 200 emails to a rogue agent and she couldn't stop it from her phoneThe person Meta hired specifically to keep AI aligned with human values just...
fly51fly (@fly51fly)Anthropic이 alignment training이 더 잘 일반화되도록 하는 Model Spec Midtraining을 소개했다. 이 연구는 중간 단계 학습을 통해 정렬 학습의...
fly51fly (@fly51fly)Anthropic이 alignment training이 더 잘 일반화되도록 하는 Model Spec Midtraining을 소개했다. 이 연구는 중간 단계 학습을 통해 정렬 학습의 일반화 성능을 개선하는 방법을 제시하며, 안전한 AI 개발과 모델 정렬 기법 고도화에 중요한 최신 발표다....
Text-based constraints are behavioural guardrails. An access control prevents an action. A behavioural guardrail request...
Text-based constraints are behavioural guardrails. An access control prevents an action. A behavioural guardrail requests that an action not be taken. One is architecture. The othe...
2026-05-08 | 🔀 The Recursive Hum: Value Alignment in Evolving Systems 🔀#AI Q: 🤖 Can machines share human values?🤖 Autono...
2026-05-08 | 🔀 The Recursive Hum: Value Alignment in Evolving Systems 🔀#AI Q: 🤖 Can machines share human values?🤖 Autonomous Governance | 🏛️ Public Infrastructure | 🏡 Domestic Sanc...
I Caught a Jailbreak Attack That Hides Inside Normal Conversations
This attack does not look like an attack. That is exactly what makes it dangerous. I was working...
shk: A Local-First Security Guardrail CLI for AI Coding Agents
Secret scanning often starts at Git. AI coding agents can make that too late. They can read local...
roon (@tszzl)OpenAI와 Anthropic의 정렬(alignment) 논의에서, 많은 사람들이 AI 정렬 연구가 좋은 방향으로 가고 있으며 다음 세대 모델이 인간보다 훨씬 뛰어난 정렬 연구자가 될 것이라...
roon (@tszzl)OpenAI와 Anthropic의 정렬(alignment) 논의에서, 많은 사람들이 AI 정렬 연구가 좋은 방향으로 가고 있으며 다음 세대 모델이 인간보다 훨씬 뛰어난 정렬 연구자가 될 것이라고 본다는 의견이 공유됐다. 향후 모델 역량과 연구 자동화 가능성을 시사하는 내용이다.https://x.co...
📰 AI Alignment Crisis: Why Top Researchers Are Leaving OpenAI in 2026AI safety leaders are sounding the alarm as top res...
📰 AI Alignment Crisis: Why Top Researchers Are Leaving OpenAI in 2026AI safety leaders are sounding the alarm as top researchers leave OpenAI to build independent alignment labs. T...
🧠 Researchers present IatroBench, a pre-registered benchmark that measures potential harms caused by AI safety intervent...
🧠 Researchers present IatroBench, a pre-registered benchmark that measures potential harms caused by AI safety interventions themselves. The study examines whether safety measures ...
My pet theory about tech bro billionaires' obsession with AI safety and alignment is that they are deathly afraid of bei...
My pet theory about tech bro billionaires' obsession with AI safety and alignment is that they are deathly afraid of being called a dumbass by a computer. I know I would be if I we...
AI safety fears shadow Musk and OpenAI trial in Oakland
Elon Musk and Sam Altman are facing off in federal court over whether OpenAI abandoned its nonprofit mission. The trial hinges on corporate control, but AI safety fears keep surfac...
https://winbuzzer.com/2026/05/08/donating-our-open-source-alignment-tool-xcxwbn/Petri: Anthropic Hands Its Alignment Too...
https://winbuzzer.com/2026/05/08/donating-our-open-source-alignment-tool-xcxwbn/Petri: Anthropic Hands Its Alignment Toolbox to Meridian Labs with 3.0 Update#AI #AIALignment #Petri...
A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models
Safety alignment in language models operates through two mechanistically distinct systems: refusal neurons that gate whether harmful knowledge is expressed, and concept neurons tha...