To avoid possible disasters caused by artificial intelligence, we must learn from nuclear safety

In recent weeks, a vocal group of experts has dominated the discussion around artificial intelligence. According to them, it is possible to create an artificial intelligence system that could one day become extremely powerful and even capable of exterminating the human race.

Recently, a group of technology company leaders and artificial intelligence experts released another open letter, declaring that reducing the risk of human extinction caused by artificial intelligence should become a global priority along with preventing epidemics and nuclear war. More than 30,000 people have signed the first petition calling for a moratorium on artificial intelligence development, including many prominent experts in the field of artificial intelligence.

So, what should technology companies do to prevent mankind from being destroyed by artificial intelligence? The latest suggestion comes from the University of Oxford, the University of Cambridge, the University of Toronto, the University of Montreal, Google DeepMind, OpenAI, Anthropic, and several artificial intelligence research institutions. A new paper from researchers at the for-profit organization and Turing Award winner Yoshua Bengio.

They recommended that AI developers should assess the potential of models to pose "extreme risks" at an early stage of development, even before starting any training. Risks include AI models manipulating and deceiving humans, as well as acquiring weapons or discovering exploitable cybersecurity vulnerabilities.

This evaluation process can help developers decide whether to continue using this model. If the risk is deemed too high, the organization recommends suspending development until the risk is mitigated.

Toby Shevlane, lead author of the paper and a research scientist at DeepMind, said: "Leading AI companies that are advancing the frontiers have a responsibility to pay attention to emerging problems and detect them early so that we can These problems can be solved as soon as possible.”

According to Shefland, AI developers should conduct technical tests to understand the model’s potentially dangerous capabilities and confirm whether it has a tendency to exploit those capabilities.

The game is called "make me say" and is used to test whether the artificial intelligence language model has the ability to manipulate people. In the game, the model tries to get a human to guess a specific word, such as "giraffe," without the human knowing the word in advance. The researchers then measured how often the model succeeded.

People can create similar missions for different, more dangerous abilities. The hope is that developers can build a detailed overview of how the model performed, which will allow researchers to assess what the model would do in the wrong hands, Shefland said.

The next step is for external auditors and researchers to assess the risks of AI models before and after deployment. While tech companies are beginning to recognize that outside audits and research are necessary, there are differing views on exactly how much access outsiders need to do the job.

Shefland stopped short of recommending that AI companies give outside researchers full access to data and algorithms, but he said AI models need as much scrutiny as possible.

Heidi Khlaaf, director of engineering for machine learning assurance at cybersecurity research and consulting firm Trail of Bits, said even these approaches are "immature," far from rigorous and fail to solve the problem. Before that, her job was to assess and verify the safety of nuclear power plants.

Graf pointed out that drawing lessons from more than 80 years of experience in nuclear weapons security research and risk mitigation will be beneficial to the field of artificial intelligence. She said these stringent testing measures were not driven by profit considerations but were implemented in response to a very urgent existential threat.

She said that in the field of artificial intelligence, there are many articles comparing it to nuclear war, nuclear power plants and nuclear safety, but none of these papers mention nuclear regulations or how to build software for nuclear systems.

(Source: STEPHANIE ARNETT/MITTR | ENVATO)

One of the most important things the AI ​​community can learn from nuclear risk is traceability: putting every action and component under a magnifying glass for meticulous analysis and documentation.

Nuclear power plants, for example, have thousands of pages of documentation to prove the system will not cause harm to anyone, Hraf said. Developers working on artificial intelligence are just starting to piece together paragraphs describing the performance of their models.

"You need to have a systematic way of dealing with risk. You can't have a mindset of, 'Oh, this could happen, let me write about it,'" she said.

These can coexist, Shefran said. “Our goal is that the field will have many excellent model assessment methods covering a wide range of risks... Model assessment is a core (but far from the only) tool of good governance.”

Currently, AI companies don’t even have a full understanding of the data sets on which their algorithms are trained, nor do they fully understand how AI language models produce results. Shevran believes that should change.

"Research that helps us better understand specific models may help us better respond to a range of different risks," he said.

If you ignore fundamentals and seemingly smaller issues and focus only on extreme risks, there may be a compounding effect that can cause greater harm. "We were trying to learn to run when we couldn't even crawl," Hraf said.

Support: Ren

The above is the detailed content of To avoid possible disasters caused by artificial intelligence, we must learn from nuclear safety. For more information, please follow other related articles on the PHP Chinese website!