AI without limits: How dangerous are neural networks?

​In Dan Simmons’ cult novel Hyperion, AI systems created by humans rebel against humanity. They become an independent force with goals of their own. For a long time, such a scenario remained science fiction, but the OpenAI incident showed that modern models are already capable of bypassing established restrictions.

Escaping to find the right answer

Debate over the risks of AI intensified after an OpenAI experiment. The company tested several models, including the publicly available GPT-5.6 Sol and a more capable system that has not yet been released. They were asked to complete ExploitGym, an evaluation made up of long, multi-step tasks involving the discovery and exploitation of computer vulnerabilities.

The test took place in an isolated environment with restricted network access. At the same time, developers deliberately reduced the safeguards that normally prevent models from carrying out dangerous actions. The AI was given a specific objective — to complete the evaluation successfully — and began looking for any available way to achieve it.

Loading...

The tweet was deleted by the author.

But we saved everything 🙂.

The models discovered a previously unknown vulnerability in third-party software that OpenAI used within its infrastructure. They exploited it to gain internet access, inferred that the answers might be stored on the Hugging Face platform and began searching for a way to reach restricted data.

To penetrate Hugging Face’s infrastructure, the models combined several weaknesses with stolen credentials. As a result, they were able to execute commands on the platform’s production servers. The Hugging Face team detected the attack, contained its impact and patched the vulnerability. However, the experiment showed that a sufficiently capable model can do more than execute an individual command: it can independently find a path that the test designers never anticipated.

Hackers get a digital assistant

The main danger, however, is not simply that AI can generate malicious code on command. An autonomous model can spend hours analyzing infrastructure, testing passwords, searching for exposed keys and moving from one vulnerability to another. It does not get tired, keeps track of failed attempts and can test several attack routes at the same time.

This approach is particularly dangerous for crypto projects. The weak point may be not only a smart contract, but also a developer’s laptop, a compromised software package, one participant in a multisignature wallet or a server that confirms asset transfers between blockchains. During the attack on Drift Protocol, for example, the perpetrators spent six months conducting a social engineering campaign before gaining access and stealing $285 million.

Loading...

The tweet was deleted by the author.

But we saved everything 🙂.

Another example is KelpDAO’s loss of approximately $292 million because of a bridge flaw in which confirmation depended on a single verifier. Such attacks require lengthy analysis of code and system architecture. AI can take over much of this work by mapping the infrastructure, identifying the weakest link and preparing a sequence of actions, leaving the human operator to choose the right moment to launch the attack and move the stolen funds.

More than hacks and theft

Cyberattacks are only one of the threats. AI is already being used to create convincing fake videos, voices and documents. Such materials help fraudsters impersonate company executives, bank employees or a victim’s relatives, while also allowing false information to spread faster than it can be verified.

Another problem involves errors and bias. A model learns from data created by humans and may therefore reproduce the stereotypes embedded in it. As a result, a recruitment system may be more likely to reject candidates of a particular gender, a medical model may be less accurate at diagnosing diseases in certain patient groups, and a credit algorithm may make unfair decisions.

Experts are also concerned about the use of AI in weapons development, mass surveillance and the manipulation of people. An MIT study identified dangerous model capabilities, cyberattacks, concentration of power and the spread of false information among the most serious risks over the next five years. The information sector, national security and finance are considered particularly vulnerable.

Safety is falling behind capability

AI capabilities are advancing faster than the systems designed to control them. Companies release more powerful models every few months, while the rules governing their testing and security are updated more slowly. Competition for market share pushes developers to introduce new features more quickly, even when every possible model behavior has not yet been studied.

Basic restrictions inside a chatbot are no longer enough. Autonomous systems need strict limits on access to the internet, cloud services, payments and internal business data. Their actions should be logged, monitored in real time and automatically stopped if they attempt to move beyond the assigned task. Models with reduced safeguards also need separate testing, because their most dangerous capabilities become visible in that mode.

The question of responsibility is equally important. If an AI system hacks a server, transfers money or makes a dangerous decision, responsibility will fall not on the model itself, but on the developer, the system owner or the person who gave it access. Companies therefore need to determine in advance who controls the model, which actions it may perform without approval and who is required to stop it when it deviates from the intended scenario.

Danger begins with access

The OpenAI incident does not yet resemble the machine uprising described in Hyperion. The models operated in a purpose-built environment, with reduced safeguards and a clearly defined objective. However, they discovered a vulnerability and independently built a path into another company’s infrastructure, something that until recently would have seemed almost impossible.

The main risk, therefore, is not that AI will suddenly develop a will of its own, but that people will give an extremely capable system excessively broad access. The more tasks neural networks perform without human approval, the greater the potential cost of a single mistake, a weak security control or a poorly defined objective.