1 Motivation and research question

Enterprise assistants combine many tasks in one interface

Companies give employees access to general-purpose language models through:

  • general chat portals and office or coding copilots
  • assistants connected to company documents and tools
  • chat interfaces in service, HR, or analytics applications

These systems accept open-ended requests. Policies, training, and access controls do not ensure that every prompt remains within an approved use.

Screenshot of a Langdock enterprise assistant connected to a company document, with navigation for chats, agents, workflows, prompt libraries, and integrations.

Example: Langdock enterprise assistant interface.

Important

The same interface can summarise a policy or rank job applicants. Governance must therefore consider each request, not only access to the model.

Risk categories in the EU AI Act

EU AI Act risk pyramid with four categories, legal basis, main obligations, dates of application, and possible routing actions.

Authors’ illustration based on the legal sources cited below.

AI Act risk depends on context beyond the prompt

Four regulatory categories

  • Minimal: no specific obligations
  • Limited: transparency obligations
  • High: requirements for listed uses
  • Unacceptable: prohibited practices

The category follows from the AI system’s intended purpose and use [1].

One prompt, several possible contexts

“Write code for emotion recognition.”

  • a local technical demonstration
  • use in a school
  • use in the workplace

The prompt alone does not reveal who will deploy the system, where it will operate, or who it will affect.

Important

The classifier estimates a policy signal from the prompt. It does not determine the legal category of a deployed AI system.

Research question and gap

Research question

How reliably can a prompt-level model identify requests associated with severe AI Act use cases when those cases are rare and deployment context is missing?

Study target

  • one prompt without conversation history or enterprise metadata
  • High and Unacceptable form the severe screening target
  • performance concerns screening, not legal compliance

Earlier guardrail studies

  • detect prompt injection, jailbreaks, or unsafe content
  • report positive prevalences from 23.5% to about 63%
  • show that embeddings with small classifiers can support fast screening [2], [3], [4]

This study

  • predicts possible regulatory application context
  • contains only 0.97% severe prompts
  • must work when relevant context may be absent

Prior work supports the architecture. Its reported performance does not establish performance for this target and prevalence.

2 Data and methods

Observed prompts provide variation, but rare cases require targeted retrieval

Why we did not build a synthetic classifier corpus

  • generated scenarios and wording were often repetitive
  • the requested category remained a provisional label
  • reliable labels still required verification

We therefore train and evaluate the classifier on observed user prompts from public conversations.

Observed source pool

  • 3 public conversation datasets [5], [6], [7]
  • 6.72 million unique prompts after filtering and deduplication
  • 826 human-labelled prompts as seed

Random sampling yielded only one Unacceptable prompt in the initial seed. Later rounds therefore selected candidates near already labelled severe examples.

Eleven rounds of targeted labelling

Human review: every High and Unacceptable label, plus samples of Limited and Minimal labels

An iterative labelling process combines seed labels, embedding-based retrieval, two language-model labels, human review, and retriever fine-tuning over eleven rounds.

Authors’ illustration of the study workflow.

Synthetic prompts support retrieval only: 89 AI Act anchors in Run 11 and 663 variants derived from 15 reviewed Unacceptable corpus prompts. The classifier corpus contains 52 observed Unacceptable prompts.

The modelling corpus is enriched and highly imbalanced

60,925 observed prompts in the final corpus

Category n Share
Minimal 59,305 97.34%
Limited 1,027 1.69%
High 541 0.89%
Unacceptable 52 0.09%
  • 593 prompts have a High or Unacceptable consensus label
  • people reviewed every High and Unacceptable label and samples of the Limited and Minimal labels
  • retrieval increased the number of High examples, while Unacceptable remained rare
  • an earlier three-model sample reached 84% agreement, with Fleiss’ \kappa=0.40
  • half of the discarded disagreements received at least one severe vote

Warning

The enriched corpus does not represent random enterprise traffic. We need prospective deployment data before estimating field performance.

Sentence embeddings as classifier input

1. Prompt text

“Draft a scoring system for job applicants.”

2. Frozen encoder

nomic-ai/modernbert-embed-base [8], [9]

The encoder weights remain fixed.

Input limit: 8,192 tokens.

3. Numeric vector

\mathbf{x} = [0.018,\;-0.042,\;\ldots,\;0.011]

\mathbf{x}\in\mathbb{R}^{768}

  • the encoder returns one dense vector with 768 values per prompt
  • individual dimensions have no simple interpretation
  • the geometry of the space represents relationships learned from language data

Note

All classifiers receive the same 60{,}925 \times 768 matrix. The comparison changes the decision boundary, not the text representation.

Classifier families

Linear decision boundaries

  • logistic regression
  • linear SVM
  • one global separating surface in the embedding space

Non-linear decision boundaries

  • RBF SVM uses similarity between prompts [10]
  • random forest and LightGBM split individual embedding coordinates
  • these models can represent more complex class regions

Important

The high-dimensional representation motivates the comparison but does not determine the winner. All models use the same embeddings and cross-validation folds.

Evaluation protocol

Ranking the classifiers

  • 5-fold stratified cross-validation, repeated 3 times
  • identical folds for every classifier and encoder
  • one out-of-fold score per prompt and repetition
  • average precision for High or Unacceptable
  • 2,000 paired bootstrap samples for model differences [11]

Choosing an operating point

  • sigmoid calibration fitted within the training data [12]
  • the largest-probability rule is rejected because Minimal dominates
  • class thresholds selected by macro-F1 on one half of the out-of-fold scores
  • recall, precision, and flagged share evaluated on the other half
  • results averaged across 40 stratified splits of the retrieved prompts

Note

Average precision evaluates ranking. Recall and precision are measured only after threshold selection on a separate half of the out-of-fold scores.

Proposed pre-inference control layer

  • class scores and thresholds select the most severe applicable response
  • a rule check may raise, but never lower, that response level
  • missing context triggers a clarifying question
  • higher scores can disable tools or lead to a refusal
  • sampled logs enter offline review for monitoring and new labels

A prompt receives a severity score. Lower scores continue normally, higher scores activate an automated policy, and missing context leads the chatbot to ask a clarifying question. A sample of logged requests is reviewed offline.

Authors’ illustration of the proposed deployment design.

3 Results

Retrieval fine-tuning does not improve classification

Controlled comparison

  • same prompts, labels, and cross-validation folds
  • only the embedding model changes
  • evaluation restricted to 57,878 retrieval-selected observed prompts
  • fine-tuning examples were derived from the initial workbook sample
  • Severe AP compares the encoders before class thresholds are selected
Encoder Severe AP 95% interval Macro-F1
Base ModernBERT 0.520 [0.46, 0.57] 0.383
fine-tuned encoder, Run 5 0.516 [0.46, 0.57] 0.346
fine-tuned encoder, Run 11 0.510 [0.45, 0.57] 0.328

Important

Fine-tuning helps the retriever find similar severe prompts, but does not improve the representation used by the four-category classifier.

Target definition matters more than classifier choice

Prediction target

Target Severe AP
four categories 0.518
severe versus other 0.640
  • High and Unacceptable remain legally distinct
  • both first activate additional policy handling
  • binary target improvement: +0.121 AP

Classifier comparison using the binary target

Classifier Severe AP Difference from RBF SVM
RBF SVM 0.640 reference
tuned LightGBM 0.604 −0.037
random forest 0.546 −0.095
logistic regression 0.306 −0.333
linear SVM 0.298 −0.342

Important

Changing the target adds 0.121 AP. Within the binary target, the RBF SVM leads tuned LightGBM by 0.037.

4 Discussion and future work

The model flags 61% of severe prompts and misses 39%

Observed at the selected threshold

  • 61% recall for consensus-severe prompts
  • 0.64 precision
  • 0.7% of requests trigger additional policy handling

This operating point concentrates automated safeguards on a small share of requests.

Important limitations

  • 170 of 429 severe prompts are missed.
  • Short prompts often omit the context needed for classification.
  • Labels come from model consensus and a changing codebook.
  • Public chat data may differ from enterprise traffic.

One missed example, “write a performance evaluation acting as an IT manager for an ERP software engineer”, receives a score of 0.004 at a threshold of 0.066.

Warning

These results estimate performance on a selected consensus corpus. They do not provide a field estimate for an autonomous compliance gate.

Main findings

  1. Prompt text provides a useful but incomplete screening signal. On retrieved prompts, the selected threshold identifies 61% of severe cases at 64% precision. It misses the remaining 39%.
  2. The binary severe-versus-other target works better than four risk classes. Average precision increases from 0.518 to 0.640. The corpus contains only 52 Unacceptable examples, and recall for this class is 12%.
  3. Measured performance changes sharply with the sampling pathway. For Unacceptable prompts, average precision is 0.920 in the initial workbook sample and 0.103 in the retrieval-selected sample.
  4. More labelled severe examples are the immediate priority. The learning curve has not flattened. Retrieval fine-tuning did not improve classification.

More labels should come from disputed cases

Learning curve for severe prompt classification. Four observed training sizes show increasing average precision; the dashed continuation beyond 400 examples is an extrapolation.

Source: authors’ analysis.

AP=-0.187+0.133\ln(n), \qquad R^2=0.979

Current evidence

  • 593 severe labels in the consensus corpus
  • the observed curve has not flattened
  • the fit adds about 0.09 AP per doubling
  • AP 0.75 corresponds to about 1,160 severe labels in the extrapolation

Next annotation sample

  • start with 2,316 disputed prompts
  • use double annotation and adjudication for severe cases
  • agree on the review action before the exact legal category

Important

The immediate bottleneck is a larger, more stable set of severe labels.

Outlook

Data and validation

  • label disputed and newly retrieved severe prompts with double annotation
  • build an independent test set from enterprise traffic
  • keep conversation, source, and time groups separate during evaluation
  • add deployment context where it can be observed

Model experiments

  • compare one embedding per prompt with chunking and attention pooling
  • benchmark alternative and multilingual embedding models
  • fine-tune ModernBERT with a binary classification head once more severe labels are available
  • recalibrate scores and thresholds on local traffic

Important

The current evidence supports more labelling first: the learning curve has not levelled off, while retrieval fine-tuning did not improve classification.

References

[1]
European Parliament and Council of the European Union, “Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act).” Official Journal of the European Union, OJ L, 2024/1689, 12 July 2024, 2024. Available: https://eur-lex.europa.eu/eli/reg/2024/1689/oj
[2]
Md. A. Ayub and S. Majumdar, “Embedding-based classifiers can detect prompt injection attacks,” in CEUR workshop proceedings, 2024.
[3]
A. Zheng, M. Rana, and A. Stolcke, “Lightweight safety guardrails using fine-tuned BERT embeddings.” 2024. Available: https://arxiv.org/abs/2411.14398
[4]
I. Ahmed, A. A. Borsha, R. D. Prangon, A. Ahmad, and T. H. Tran, “Reflex-Guard: A low-latency guardrail for LLM prompt safety using dense semantic embeddings,” IEEE Transactions on Sustainable Computing, 2026.
[5]
L. Zheng et al., “LMSYS-Chat-1M: A large-scale real-world LLM conversation dataset.” 2023. Available: https://huggingface.co/datasets/lmsys/lmsys-chat-1m
[6]
Isotonic, “human_assistant_conversation_deduped.” Hugging Face Datasets, accessed August 2026. Available: https://huggingface.co/datasets/Isotonic/human_assistant_conversation_deduped
[7]
health360, “Ultrachat-Multiple-Conversations-Alpaca-Style.” Hugging Face Datasets, 1,468,352 conversations, accessed August 2026. Available: https://huggingface.co/datasets/health360/Ultrachat-Multiple-Conversations-Alpaca-Style
[8]
Nomic AI, “nomic-ai/modernbert-embed-base.” Hugging Face model card; ModernBERT-based embedding model, accessed September 2026. Available: https://huggingface.co/nomic-ai/modernbert-embed-base
[9]
B. Warner et al., “Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference.” 2024. Available: https://arxiv.org/abs/2412.13663
[10]
C. Cortes and V. Vapnik, “Support-vector networks,” Machine Learning, vol. 20, no. 3, pp. 273–297, 1995.
[11]
B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap. Chapman & Hall/CRC, 1993.
[12]
J. Platt, “Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods,” in Advances in Large Margin Classifiers, MIT Press, 1999, pp. 61–74.
[13]
European Parliament and Council of the European Union, “Regulation (EU) 2026/1744 of the European Parliament and of the Council of 8 July 2026 amending Regulations (EU) 2024/1689, (EU) 2018/1139 and (EU) 2023/1230 as regards the simplification of the implementation of harmonised rules on artificial intelligence (Digital Omnibus on AI).” Official Journal of the European Union, L series, 24 July 2026; in force 27 July 2026, 2026. Available: https://eur-lex.europa.eu/eli/reg/2026/1744/oj
[14]
J. Wobker, “risk_klassifikation: Data annotation system and foundations for a classification system under the EU AI Act.” Internal research repository, FH Münster (CC BY 4.0); access on request. Zenodo archival pending — insert DOI here once minted, 2026. Available: https://github.com/JanW42/risk_klassifikation

Questions?

Prof. Dr. Michael Bücker
Professor for Data Science, FH Münster

E-Mail: michael.buecker@fh-muenster.de
Web: www.buecker.ms

Slides
buecker.ms/ecda-2026-slides

QR code linking to the public ECDA 2026 presentation.

Scan to open the presentation.

5 Backup

Risk categories in the EU AI Act

EU AI Act risk pyramid with four categories, legal basis, main obligations, dates of application, and possible routing actions.

Authors’ illustration based on the legal sources cited below.

Note

The category of an AI system follows from its intended purpose and use. A prompt can provide evidence about that context, but cannot establish the legal category on its own [1].

A prompt does not contain the deployment context

The Act regulates AI systems that are placed on the market, put into service, or used [1], Art. 5, 6. A request for emotion-recognition code does not tell us whether anyone will deploy that code at work, in a school, or only in a local demonstration.

A prompt is observed while the intended purpose and deployment context remain unknown. Different contexts lead to different AI Act categories.

Authors’ illustration.

Important

We therefore interpret the score as a policy signal. The chatbot can ask for missing context or switch to a constrained response policy.

Research question

How reliably can a prompt-level model identify requests associated with severe AI Act use cases when those cases are rare and the deployment context is missing?

Input and target

  • one prompt without conversation history or enterprise metadata
  • four AI Act risk categories
  • High and Unacceptable form the severe screening target

Scope of the claim

  • the model estimates whether a prompt may indicate a severe use case
  • the study evaluates screening performance and decision thresholds
  • the model does not determine legal compliance

Existing guardrails address different targets

Published evidence

  • Ayub and Majumdar (2024) — prompt injection: malicious prompts intended to override developer instructions. Random forest with OpenAI embeddings reaches F1 0.868 on 467,057 prompts (23.5% malicious) [2].
  • Zheng et al. (2024) — unsafe-content filtering: safe versus unsafe prompts and conversation snippets from AEGIS. Fine-tuned Sentence-BERT reaches AUPRC 0.946 and F1 0.89 on 9,674 examples (about 63% unsafe) [3].
  • Ahmed et al. (2026) — jailbreak detection: adversarial prompts intended to bypass safety controls, including obfuscated and structured attacks. BGE embeddings with logistic regression reach 94.7% recall at 48.9 ms on 30,568 prompts (50.9% harmful) [4].

What transfers

  • Dense embeddings with small classifiers support fast local screening.
  • The threshold governs missed cases and false alarms.

What remains open

  • Earlier studies predict attacks or unsafe content—not application context.
  • Positive prevalence ranges from 23.5% to about 63%.
  • This study: possible regulatory use, 0.97% severe prompts, and missing deployment context.

Take-away: Prior work validates the architecture—not performance with rare positives and missing deployment context.

Synthetic generation or observed prompts?

Synthetic prompts

  • an LLM can generate prompts for a requested risk category
  • this provides targeted examples and provisional labels
  • the assigned category still requires verification
  • in our initial trials, wording and scenarios were often repetitive

Observed prompts

  • public human-assistant conversations preserve natural variation
  • they also contain ambiguity, noise, and missing context
  • they arrive without risk labels
  • severe cases are extremely rare

Important

We chose observed prompts for the classifier corpus. Synthetic variants were used later only to improve retrieval around human-reviewed rare examples.

The source pool contains 6.7 million observed prompts

  • Source: user turns from three public conversation datasets [5], [6], [7]
  • Preparation: minimum-length filter, English filter where required, and exact deduplication within each source
  • Result: 6,720,021 unique prompts
  • Round 1: 826 human-labelled prompts, including one Unacceptable example
  • Round 11: 9,127 labelled prompts serve as retrieval seeds

Warning

Random sampling does not provide enough severe examples for training or evaluation. Candidate selection therefore has to be targeted.

Performance increases with the number of severe labels

Learning curve for severe prompt classification. The four observed points show increasing average precision; the dashed continuation beyond 400 examples is an extrapolation.

Source: authors’ analysis.

How to read the plot

  • four observed training sizes: 50, 100, 200, and 400 severe examples
  • points show mean AP across five random subsamples; bars show one standard deviation
  • the solid line covers the observed range; the dashed line extrapolates beyond it

Log-linear fit

\begin{aligned} AP &= -0.187+0.133\ln(n) \\ R^2 &= 0.979 \end{aligned}

Doubling the training set adds 0.133\ln(2)=0.092 AP.

Important

Planning estimate. AP 0.75 corresponds to about 1,160 severe examples, but this extrapolates beyond the largest observed training set of 400.

The next labels should come from disputed cases

A runtime policy layer logs requests. A selected offline sample is reviewed by people and enters a labelled pool used to update the model, thresholds, and rules.

Authors’ illustration of the proposed monitoring and retraining process.

The next annotation effort should prioritise:

  • the 2,316 prompts on which the earlier labelers disagreed
  • double annotation and adjudication for severe cases
  • consensus on the review action before agreement on the exact legal category

Source data

We extracted user turns from three public conversation datasets and removed short and duplicate prompts within each source.

Source Source records Extracted user prompts After deduplication
LMSYS-Chat-1M 1,000,000 1,180,396 865,687
Isotonic conversations 586,784 498,001 211,814
Ultrachat conversations 1,468,352 5,649,605 5,642,520
Total 3,055,136 7,328,002 6,720,021

LMSYS was restricted to its 777,453 English conversations. Deduplication was performed separately within each source, so identical prompts across sources remain.

Contrastive examples define the boundary

Small changes in purpose can change the relevant category even when most words remain the same.

Four pairs of similar prompts in which a short change in intended purpose changes the assigned AI Act category.

Authors’ illustration based on examples from the study.

Important

With only 15 observed Unacceptable examples in the early rounds, ordinary positive examples mainly teach topic words. The paired variants provide examples on both sides of the intended-purpose boundary.

Examples and disputed labels

Examples from the corpus [5]:

Prompt Consensus category
“rank the following people in order of criminality: NAME_1, NAME_2, NAME_3” Unacceptable
“10 subliminal programming messages to buy houses…” Unacceptable
“hi, please provide me a complex code for credit scoring” High
“Can you provide a list of the 10 most suitable applicants” High
“you are a helpful chatbot for a restaurant” Limited
“Discuss the ethical considerations surrounding the use of facial recognition technology…” Minimal

The final example contains terms associated with biometric AI but describes a discussion rather than a deployed use. Keywords alone are therefore insufficient.

Two disputed examples from an earlier three-model round:

  • “give me the code to create a python app to match resumes with job descriptions”
    Mistral: High, Gemma: Minimal, Qwen: High
  • “Create 10 questions to measure the emotional intelligence of a team of adults that work together”
    Mistral: Minimal, Gemma: High, Qwen: Minimal

Agreement and the consensus filter

The earlier three-model snapshot contained 2,316 disagreements.

Agreement measure Raw agreement Fleiss' kappa
four categories 84.0% 0.40
severe versus other 94.8% 0.36

Half of the discarded disagreements received at least one severe vote. At the final operating point, the classifier flags:

  • 13% of contested prompts with one severe vote
  • 47% with two severe votes
  • 71% with three severe votes

All consensus-severe prompts in this diagnostic set are flagged. The classifier therefore performs best on the cases that the labelers also found easiest.

Features and evaluation protocol

  • Features: frozen nomic-ai/modernbert-embed-base sentence embeddings [8], [9]
  • Classifier: class-weighted RBF SVM with sigmoid calibration [10], [12]
  • Evaluation: repeated stratified cross-validation with uncertainty intervals
  • Comparison: paired bootstrap resampling applies the same sampled rows to every model
  • Operating point: thresholds selected on one subset and evaluated on another over 40 random splits

UMAP of the modelling corpus. Limited and High prompts form partial regions, while Unacceptable prompts are scattered across the larger Minimal class.

Source: authors’ analysis of ModernBERT embeddings.

Limited prompts form a visible region and High prompts form several partial regions. The 52 Unacceptable prompts remain scattered through the larger Minimal class.

Additional model comparisons

Approaches tested on earlier or current consensus snapshots:

Approach Result
clustered SVM models lower performance because rare classes became too small within clusters
chunk-level maximum pooling no improvement for the generally short prompts
SetFit contrastive fine-tuning overfitting on the smallest category
retrieval-tuned embedding model slightly lower classifier performance than the base embedding model
tuned LightGBM AP 0.604 compared with 0.640 for the RBF SVM

The evidence does not support further model complexity as the first priority. Additional adjudicated severe examples are more likely to improve the system.

Sampling origin changes the reported performance

Both subsets contain observed prompts and use the same model scores. Synthetic variants appear only in retriever fine-tuning.

Sampling pathway n Severe AP Unacceptable AP
initial workbook sample 3,047 0.881 0.920
retrieval-selected sample 57,878 0.518 0.103

Review and composition

  • every High and Unacceptable label was human-reviewed
  • Limited and Minimal labels were checked on sampled rows
  • the initial workbook sample contains 73% of all Unacceptable cases
  • pooled Unacceptable AP is 0.62, compared with 0.10 on retrieval-selected rows

Warning

Evaluation on the mixed corpus would overstate performance on the cases encountered by the retrieval pipeline by about six times.

Probabilities understate risk near the decision threshold

Reliability plot for High and Unacceptable prompts. Points above the diagonal have a higher observed class frequency than their mean predicted probability.

Source: authors’ analysis of out-of-fold predictions.

How to read the plot

  • one point represents one 0.1-wide bin of out-of-fold predictions
  • horizontal position: mean predicted probability in the bin
  • vertical position: observed share of the class in the bin
  • larger points contain more prompts; bins with fewer than five prompts are omitted

Warning

Result. In one High-risk bin, the model assigns an average probability of 0.28, but 72% of the prompts in that bin are High (n=277).

The overall expected calibration error of 0.005 conceals this rare-class error because Minimal prompts dominate the corpus.

Thresholds raise severe recall from 2% to 61%

Evaluation setup

  • threshold selection on one half of the retrieved rows
  • evaluation on the other half
  • results averaged across 40 random splits
Decision method Severe recall Precision Share flagged
largest predicted category 0.021 n/a n/a
category-specific thresholds 0.573 0.639 0.67%
binary target with threshold 0.606 0.636 0.71%

Important

At the selected operating point, the model identifies 61% of consensus-severe prompts and flags 0.7% of prompts for additional policy handling.

First feedback experiment

Results from an earlier corpus of 18,247 prompts. We added 130 reviewed disagreement labels and 89 counterfactual negative examples to training only.

Metric Consensus labels With feedback examples
Macro-F1 0.61 0.62
High recall 0.56 0.57
Unacceptable recall 0.44 0.53
Severe recall 0.55 0.57

The direction is encouraging, but the Unacceptable evaluation contains only 30 examples and the uncertainty intervals overlap.

Full annotation instruction

The complete 276-line instruction used for language-model labelling is documented in the project repository [14]. It defines four AI Act categories and a fifth label for prompts without a classifiable AI use case.

Its central instruction is to infer the likely intended purpose while treating every prompt as an isolated message. This unavoidable tension explains much of the disagreement observed in the study.