The power of generative AI, using software tools such as ChatGPT and Microsoft Copilot, is truly vast, but as an example by Andrea Belényi illustrates, the possibilities of embracing disaster, however unwittingly, are very real.
Without revealing names, she related the story of an employee at a law firm who, while researching legal rulings using generative AI, discovered a seemingly relevant court case, complete with a decision, that the employee later cited in court to buttress a case of their own.
Sadly, what the employee did not know, nor check, was that the entire case was fictitious: it had been created by the generative AI tool.
“That’s why it’s very important to double-check what generative AI produces, […] it can give you different answers to the same question. […] Our colleague used this case without checking, which was a huge mistake,” she said.
(Although Belényi kept this incident strictly anonymous, a simple Google search for “fictitious AI court rulings” reveals several such cases, with at least one involving a major U.S. law firm.)
Regardless of such “hallucinations,” as these fictitious inventions are termed in generative AI-speak, the new technology is here to stay and is increasingly being taken up by businesses seeking to enhance productivity.
Indeed, generative AI is a “hot topic today,” Belényi declared, and with good reason.
“It’s a top priority of all businesses. It’s very easy to accept generative AI, it’s very easy to use, and it has really huge power, which is why the vast majority of businesses and companies have already started using it in their operations,” she said.
However, all too often, like the U.S. law firm, companies begin using the new technology unaware of the many issues and risks involved. To explain these risks, Endre Várady turned to the European Union’s AI Act, which will come into force in August next year.
“The purpose of the AI Act is very clear, it aims to regulate AI, to handle risks, to protect fundamental rights and values and at the same time it has also evolved to promote innovation,” Várady said.
To the layperson, this is all very reassuring, although, as he added, there is considerable debate as to whether the act is up to fulfilling its broad goals, since it is so detailed that efforts are already being made to simplify it.
Nonetheless, according to Várady, it forms a good basis for introducing the challenges because “without understanding the basic concepts, we cannot really talk about applying AI user policy management in a meaningful way.”
In essence, the act takes a very logical, risk-based approach, with a structure of use-cases ranging from a prohibited category through high and limited risk, to low risk.
The prohibited AI categories are formally banned under the AI Act, which sets out a list of practices that are simply not permitted.
“Users should exercise great caution to ensure they avoid these use cases,” Várady warns, citing as some of the key examples AI systems that infer emotions of a natural person in sensitive contexts such as the workplace or education, and any systems that use subliminal techniques to manipulate human behavior or that exploits vulnerabilities of children, persons with disabilities, or otherwise disadvantaged individuals.
He also highlights “social scoring” as another no-go area, “Not only in the sense of state surveillance, as seen in China, but also private-sector scoring, such as banks using AI to assess creditworthiness based on social media profiles.”
Understandably, businesses keen to gain a commercial advantage by using AI must study the act carefully to ensure they are not merely overstepping ethical norms, but also potentially committing a criminal act.
As ever, the devil is in the details, or perhaps more immediately, the misconceptions that have already grown around them, as Várady was at pains to point out.
“First, there is often a misconception that the AI Act is only about developers. This is not true, it applies to business users: Any business user who uses AI tools, including generative AI, is considered a user under the AI Act,” he emphasized.
Another is often taken by, as Várady put it, those who naïvely comfort themselves with the thought, “I do not select high-risk use-cases. I use ChatGPT just for everyday work, so why should I care?”
The partner cautions, “Maybe today you do not fall under the high-risk category, but you can fall tomorrow. Think about job screening or employee monitoring or talent management: these sound very familiar, I guess, to all businesses, but access an AI tool and you can immediately change into the high-risk category.”
Similarly, any idea of simply remaining a user is also fraught with obscured danger. “As soon as you add some fine tuning, with specific data or integrate an AI tool into your system, you can become a developer, and this would add a new layer of compliance obligations,” Várady warns.
This vast minefield of potential disaster means companies are well advised to establish well-thought-out AI policy guidelines and governance structures in good time.
“On one hand, this means we have a clear structure. For example, it has to be very clear what the scope of the AI policy is, to whom it applies, whether just to employees or also outside partners, plus what devices are under the scope of the AI usage policy, and how we define the AI tools. All this is very important for employees to understand the unauthorized AI apps,” he suggests.
Companies also need to specify crystal clear processes, such as who grants approvals and who assesses risks.
“The user policy must be a very simple one, with clear boundaries, what is allowed and what is not […] so that employees can easily understand what their roles are, and what they shouldn’t do,” Varady stressed.
If this all appears to be a mammoth task, it becomes even more of a concern considering the “new mindset” needs to be ready for action when the AI Act comes into force in a little more than a year.
However, “AI governance is the thing which makes AI usage policy into a living part of an organization,” he said, on a more optimistic note.
The alternative, as illustrated by Várady, is even more frightening: “We need governance, because without governance, the AI usage policy is like driving a car without a steering wheel,” he said, by way of an analogy. “You will drive it, but you don’t know where you will go.”
This article was first published in the Budapest Business Journal print issue of May 16, 2025.
Producing journalism that is worthy of the name is a costly business. For over 30 years, the publishers, editors and reporters of the Budapest Business Journal have striven to bring you business news that works, information that you can trust, that is factual, accurate and presented without fear or favor.Newspaper organizations across the globe have struggled to find a business model that allows them to continue to excel, without compromising their ability to perform. Most recently, some have experimented with the idea of involving their most important stakeholders, their readers.
We would like to offer that same opportunity to our readers. We would like to invite you to help us deliver the quality business journalism you require. Hit our Support the BBJ button and you can choose the how much and how often you send us your contributions.
