In the first part of our series we asked the basic question: when should a human stop an AI?
In the second part we analyzed agent autonomy and tried to answer how far we can allow artificial intelligence to act on its own.
In the third part we moved to the organizational level and discussed AI governance, responsibility, security, monitoring and control rules.
Now it’s time to combine all these elements.
Because you can have a great AI strategy. You can have good procedures. You can hire the best engineers. You can choose an excellent model. But ultimately it all comes down to one question: How to build a system that is autonomous enough to actually deliver value, but at the same time sufficiently controlled so it does not become a source of unacceptable risk?
This is precisely one of the most important problems in designing next-generation AI systems. And here Human-in-the-Loop stops being a simple "click Accept" function. It becomes an element of the entire system architecture.
AI should not be designed as a "black box"
Imagine a classic system:
- The user sends a query.
- The AI model analyzes the data.
- The model generates a response.
- The user receives it.
This may be sufficient for a simple chatbot.
But the situation looks completely different when AI gets access to corporate systems.
For example:
- AI reads a message from a customer.
- It recognizes the customer's intent.
- It checks order history.
- It analyzes product availability.
- It proposes a solution.
- It sends a response.
- It launches a reclamation procedure.
- It orders a refund.
- And then it updates data in the CRM.
This is no longer a single AI model.
This is a system performing actions in the real world.
And that is why the architecture must take into account not only the model itself, but the whole chain: data → model → decision → tools → action → outcome → monitoring
If any element of this chain is poorly designed, the system may make a wrong decision or — worse — execute it automatically.
Autonomy should not be an ON/OFF switch
One of the biggest mistakes in AI design is thinking: "Either a human does everything, or AI does everything."
In practice we need many more levels.
We can imagine a model of autonomy:
Level 0 - human does everything
AI takes no actions. It may be used only as an informational tool.
Example: A developer asks AI how to solve a problem.
AI answers.
The developer analyzes the answer and implements the solution themselves.
Level 1 - AI analyzes
The system collects and processes information. The human makes the decision.
Example: AI analyzes documentation and prepares a summary.
The human assesses the result themselves.
Level 2 - AI recommends
The system analyzes the situation and proposes an action. The human approves.
Example: AI detects a suspicious transaction and recommends additional verification.
Level 3 - AI prepares the action
AI not only recommends a decision but prepares all elements necessary to execute it. The human approves.
Example: An agent prepares a response for a customer, a CRM update and a discount proposal.
The employee approves everything.
Level 4 - AI acts autonomously within defined boundaries
The system can independently make decisions and perform actions. But only within established rules.
Example: The agent may independently postpone a delivery date by one day if the customer has accepted such an option.
However, it cannot change contract terms.
Level 5 - AI acts autonomously
The system independently analyzes the situation, makes decisions and executes actions. The human remains responsible for supervising the entire system.
Such a level of autonomy should be used very cautiously.
Not because AI can never act independently. But because the greater the autonomy, the greater the consequences of a potential error.
Most important rule: autonomy must be proportional to risk
There is no point in creating a universal rule: "AI must always have human consent."
That could completely destroy the benefits of automation.
Imagine a system handling thousands of routine operations. If each of them requires manual approval, the human becomes a bottleneck.
On the other hand: "AI can do everything alone"
is also a bad idea.
Therefore the decision about the level of autonomy should be made based on risk.
You can analyze, among other things:
- potential harm,
- cost of an error,
- reversibility of the action,
- impact on people,
- financial impact,
- legal impact,
- data sensitivity,
- ability to detect the error,
- time needed to respond.
This leads to a very practical rule:
The higher the risk and the harder-to-reverse the outcome, the greater the human involvement should be in the process.
Reversible vs Irreversible Actions
One very useful criterion is dividing actions into reversible and irreversible.
Reversible actions
For example:
- changing task order,
- generating a draft document,
- preparing a suggested reply,
- creating a campaign draft.
If AI makes a mistake, a human can easily fix it.
In such cases you can allow the system greater autonomy.
Hard-to-reverse actions
For example:
- executing a bank transfer,
- deleting data,
- signing a contract,
- changing critical system parameters,
- sending information with significant legal consequences,
- making a decision affecting human rights.
Here the level of control should be much higher.
This is a simple but very effective design principle:
AI can have more freedom where mistakes can be easily undone.
Human-in-the-Loop, Human-on-the-Loop and Human-in-Command
It is worth distinguishing three approaches.
Human-in-the-Loop
A human participates directly in the decision-making process.
AI recommends.
The human approves.
This is a good solution for higher-risk processes.
Human-on-the-Loop
AI operates autonomously, but a human monitors the system and can intervene.
This model is suitable for repetitive and well-defined processes.
Example: The system automatically optimizes task order.
The human does not approve each change.
However, they monitor results and can take over control.
Human-in-Command
A human remains at the strategic level.
They do not control every single decision.
They are responsible for:
- operating rules,
- scope of autonomy,
- system goals,
- constraints,
- responsibility,
- ability to stop the system.
This is especially important for large autonomous systems.
Human Override - a human must be able to take control
If the system can operate autonomously, a human should have the ability to take over control.
This is exactly Human Override.
The mechanism can look different.
It may be:
- manual approval,
- stopping the process,
- cancelling an action,
- reversing a decision,
- switching the system to manual mode,
- revoking the agent’s access to tools.
It is important, however, that this not be merely a theoretical mechanism.
If a human can "take over control" but it takes 48 hours to do so while the agent performs actions in seconds, we have a problem.
Human Override should be: available, fast and actually effective.
Fail-Safe - what happens when AI is uncertain?
A well-designed system should not assume that AI will always be right. It should assume that sometimes it will be wrong.
That is why we need a Fail-Safe mechanism.
If the system:
- does not have sufficient data,
- has a low confidence level,
- detects conflicting information,
- encounters an out-of-scope situation,
- cannot perform the action according to rules,
it should not force a decision.
It should say: "I don’t know." - and hand the matter to a human.
This may be one of the most important features of a mature AI system. Not the ability to answer every question. But the ability to recognize when it should not answer.
Confidence Score - but carefully
In AI systems we often encounter the notion of a confidence level.
The system may state: "My recommendation has 95% confidence."
Sounds great. But be cautious.
A model's confidence level does not always correspond to the probability that the answer is actually correct. A model may be very confident and still be wrong.
Therefore the confidence score should be treated as one of many signals, not as absolute truth.
However, you can use it to design the process.
For example:
- high confidence + low risk = automation,
- medium confidence = recommendation for a human,
- low confidence = mandatory escalation.
This allows building a dynamic Human-in-the-Loop.
Not every decision requires a human. But every decision should have a defined escalation path.
Dynamic Human-in-the-Loop
This is a very interesting direction in AI system design.
Instead of creating a fixed rule: "Every decision must be approved by a human"
we create a rule: "A human appears when the system detects elevated risk."
Example:
A customer service agent can independently answer standard questions.
If a customer asks about shipment status - the agent replies.
If a customer wants to change an address - the agent can perform the operation according to rules.
If the customer requests a large refund - the system hands the case to a human.
If a legal threat arises - escalate.
If the system does not understand the customer's intent - escalate.
In this way a human does not control everything. They control what truly requires human judgment.
An agent should have only the permissions it really needs
This is one of the most important security principles. If an agent is to perform a certain task, it should receive only the necessary permissions.
Not: "let's give it access to the entire CRM because it might be useful."
Only: "the agent needs read access to customer data and the ability to create a ticket."
This approach is known in cybersecurity as Least Privilege. Minimal permissions.
If an agent is compromised or makes a mistake, the scope of potential damage is limited.
This is especially important in agent-based architectures.
An agent that can:
- read data,
- write data,
- send messages,
- execute transfers,
- change system configuration,
is potentially very dangerous.
Therefore every capability should be treated as a tool with a defined risk level.
Tool Calling - an agent should not have unlimited access
Modern AI agents often use tools.
The model can, for example, call:
- APIs,
- databases,
- ERP systems,
- CRM,
- a search engine,
- a payment system.
This is huge power. But also huge risk. Therefore tool calls should be controlled.
The system should know:
- who can call which tool,
- what arguments are allowed,
- what values are acceptable,
- whether human consent is needed,
- how the action is logged.
An agent may have access to a function: create_invoice
but should not automatically have access to: delete_all_invoices
It sounds absurd.
But that is precisely why systems must be designed with the worst-case scenario in mind.
Guardrails as a security architecture
Guardrails should work on multiple levels.
Data guardrails
Which information can the agent read?
Action guardrails
What operations can it perform?
Financial guardrails
Up to what amount can it act autonomously?
Time-based guardrails
During which hours can it perform operations?
User-based guardrails
For which customers can it perform actions?
Risk-based guardrails
Which actions require approval?
Thanks to this, the agent does not simply get access to a system.
It receives a controlled scope of capabilities.
AI Agent as a digital employee?
This is a very popular metaphor. An AI agent can be treated as a digital employee. But there is one fundamental difference.
An employee has:
- experience,
- context,
- intuition,
- awareness of responsibility.
An agent has:
- a model,
- data,
- tools,
- instructions,
- constraints.
Therefore we should not design agents solely on the basis of: "Tell it what to do and see what happens."
An agent should have clearly defined:
- goal,
- scope of action,
- data access,
- tool access,
- level of autonomy,
- success criteria,
- escalation conditions,
- stop conditions.
The more autonomous the system, the more it resembles an operating system for a business process. And the more it needs architecture.
Architecture of a safe AI system
We can imagine a system composed of several layers.
Layer 1 - data
Sources of organizational data.
ERP.
CRM.
CMS.
Databases.
Documents.
APIs.
Layer 2 - AI models
Language models, predictive models and other AI components.
Layer 3 - orchestration
Logic defining the sequence of actions.
Layer 4 - agent
The system analyzes the situation and plans actions.
Layer 5 - tools
The agent can use specific APIs and functions.
Layer 6 - guardrails
The system controls what the agent can do.
Layer 7 - Human-in-the-Loop
In specific cases the decision goes to a human.
Layer 8 - monitoring
The system monitors actions and decision quality.
Layer 9 - audit trail
All significant actions are recorded.
Layer 10 - emergency controls
There is a possibility to stop the system or take over control. This is not the only possible architecture.
But it demonstrates an important principle: a safe AI system is not just a model.
It is an entire ecosystem of control mechanisms.
How to implement Human-in-the-Loop in practice?
It's best to start with a small process.
Not: "Let's automate the whole company."
But: "Let's choose one process where AI can safely help."
Next:
Step 1 - identify the decision
What exactly is AI supposed to do?
Step 2 - assess the risk
What happens if the system makes a mistake?
Step 3 - determine the level of autonomy
Will AI:
- analyze,
- recommend,
- prepare the action,
- execute the action?
Step 4 - define escalation conditions
When must a human take control?
Step 5 - design guardrails
Which actions are forbidden?
Step 6 - limit permissions
Which tools are really needed?
Step 7 - design monitoring
How will we detect errors?
Step 8 - design Human Override
How does a human stop the system?
Step 9 - test emergency scenarios
What happens if:
- an API fails,
- data is incorrect,
- a model answers incorrectly,
- a user gives malicious instructions,
- the agent performs an undesired action?
Step 10 - only then increase autonomy
First observation, then recommendations, next limited automation. Only at the end greater autonomy.
This is a much safer route than deploying full autonomy on day one.
Most common mistake: we automate a process we don’t understand
This is a problem not only for AI. It affects any automation.
If a process is poorly designed, automation may make it run faster.
But faster does not mean better.
We can thus create: automation of chaos.
AI will only amplify the scale of the problem.
Therefore before deployment ask: Is the process we want to automate really well designed?
If not, first organize the process. Only later add AI.
The most important principle of AI system design
Do not design AI so that it never makes mistakes. That's unrealistic.
Design it so that: an error is possible to detect, limit and fix.
This is a fundamental difference. A mature AI system is not error-free. It is error-resilient.
Checklist for safe Human-in-the-Loop
Before deploying the system it is worth answering the questions:
☐ Do we know what decision AI makes?
☐ Do we know the cost of a potential error?
☐ Is the decision reversible?
☐ Have we defined the level of autonomy?
☐ Does AI have only necessary permissions?
☐ Are there guardrails?
☐ Does a human know when to intervene?
☐ Can the system hand the case to a human?
☐ Is there a Human Override?
☐ Is there an emergency stop mechanism?
☐ Are actions logged?
☐ Do we monitor action quality?
☐ Can we detect model drift?
☐ Do we know who is responsible for the system?
☐ Do we have an incident response procedure?
☐ Have we tested emergency situations?
If we answer "yes" to most of these questions, we are much closer to a mature AI deployment.
Glossary of terms
Human-in-the-Loop
A model in which a human participates directly in the decision-making process and approves certain AI actions.
Human-on-the-Loop
A model in which AI operates autonomously while a human monitors the system and can intervene.
Human-in-Command
A model in which a human remains responsible for objectives, rules, scope of autonomy and overall control of the system.
Human Override
A mechanism allowing a human to take control of the AI system or cancel its action.
Fail-Safe
A safety mechanism in which the system, in case of uncertainty or failure, moves to a safe state instead of continuing risky actions.
Guardrails
Constraints that define the scope of actions AI can take.
Least Privilege
The principle of granting the system only the permissions necessary to perform its task.
Tool Calling
A mechanism enabling an AI model to use external tools, APIs and systems.
Kill Switch
A mechanism allowing quick system shutdown.
Audit Trail
A record of actions enabling later reconstruction of the system's operations.
Model Drift
Deterioration of model performance due to changes in data or environment.
Confidence Score
A metric indicating the model's confidence in the generated result. It should not be automatically equated with the probability that the answer is correct.
Summary of the series
Over four parts of our series we went from a simple question: "Should a human control AI?"
to a much more complex one: "How to design a system where humans and AI can safely collaborate?"
The answer is not: "A human should approve everything."
Nor is it: "AI should operate completely autonomously."
The best solution lies between these extremes.
AI should have as much autonomy as it truly needs. A human should be present where their knowledge, responsibility, experience and judgment bring the most value.
The system should know when to act. It should know when to ask. It should know when to stop. And a human should always know how to regain control.
This is a mature approach to Human-in-the-Loop.
It is not about a human standing over AI approving every single decision. It is about creating an architecture in which autonomy is controlled, responsibility is clearly assigned, risk is monitored, and a human has a real ability to intervene.
Because the future of AI will not necessarily belong to organizations that build the most autonomous systems.
It may belong to those that best learn to manage the boundary between machine autonomy and human responsibility.
And perhaps that is why the most important question of the coming era of AI agents will not be: "How much can we let AI do?"
But: "How much can we let AI do while retaining full control over the consequences of its actions?"
This question will return with every serious AI deployment.
And the more autonomous systems become, the more important it will be to know the answer before the agent makes its first decision.
