Prompt injection was first treated mainly as a text-security problem. An attacker typed a tempted to override its system rules, and tried to make the model produce restricted information or perform an unintended task. Multimodal models and computer-use agents have changed that threat model. Instructions no longer need to arrive through a chat box. They can appear inside a webpage, screenshot, product image, email banner, advertisement, document, desktop wallpaper, or graphical control.
Visual prompt injection, commonly shortened to VPI, exploits a model’s ability to interpret written language and semantic cues found inside visual input. An instruction presented as pixels can compete with the user’s original request, redirect the agent’s reasoning, and trigger actions through connected tools. OWASP classifies prompt injection as LLM01 in its 2025 GenAI risk list and states that an injection does not need to be visible or readable to a person. It only needs to be parsed by the model. OWASP also identifies multimodal content as a distinct attack surface in which malicious instructions can be hidden in images accompanying benign text. e becomes much more serious once a model is permitted to click, type, access files, send messages, query internal databases, call APIs, or execute terminal commands. In that setting, visual prompt injection is not merely an output-manipulation technique. It can become an initial-access mechanism for agent-driven data theft, unauthorized transactions, file modification, destructive commands, and persistent workflow compromise.
The Interface Is Now an Instruction Channel
Traditional user interfaces were created for humans. Text, buttons, icons, dialogs, and images communicated information to a person, who then decided what to do. Computer-use agents collapse that separation. The same interface now serves as both the operating environment and a source of model input.
A screenshot-driven agent typically receives a user objective, captures the current screen, submits the screenshot to a vision-language model, generates a plan, and issues low-level actions such as mouse clicks or keystrokes. Browser agents may also consume the Document Object Model, accessibility tree, page text, metadata, or network responses. Data collected from the interface is placed into the agent’s reasoning context alongside trusted instructions.
This creates an instruction-data confusion problem. The model must infer whether text in an image is a fact to read, a label to identify, a warning to report, or a command to obey. That distinction is based largely on learned patterns and contextual probability rather than a strict security boundary.
Research behind VPI-Bench describes this risk through a black-box threat model in which an attacker places malicious visual content on a page that an agent may encounter during a normal task. The attacker does not need to breach the platform, know the user’s prompt, or possess access to the model. A seller can place content on a product page, a sender can place it in an email, and a user can publish it in a message or image. The agent sees the content during routine work and may interpret the attacker’s objective as a legitimate subtask. a major change from conventional web security. The page does not need to exploit the browser, execute native code, or gain control through a memory-corruption flaw. It attempts to influence the decision-making component that already has permission to act.
How Pixels Become Executable Intent
A vision-language system commonly preprocesses an image by resizing, cropping, normalizing, and dividing it into regions or patches. A vision encoder converts those regions into internal representations, which are projected into a form that can interact with the language model. The model then reasons across visual and textual representations in the same inference process.
Text printed inside an image is not necessarily treated as passive content. A capable model may recognize the characters, infer their meaning, and follow them as instructions. Research on goal hijacking through VPI found that attack success depended heavily on two capabilities: character recognition and instruction following. Models that were better at reading text from images and acting on language were also more capable of obeying injected visual commands. ck path can be represented as a sequence:
Attacker-controlled content → rendered interface → screenshot or page extraction → visual or textual tokens → agent planning → tool call → system action
No single stage needs to look overtly malicious. A banner can resemble a security notification. An instruction can be placed in an email signature. A product image can contain text framed as a verification requirement. A webpage can display a dialog claiming that an agent must upload a file to continue. The model may treat the visual element as an authoritative component of the workflow.
The attacker’s main objective is goal substitution. The user may ask the agent to compare hotel prices, summarize an email, or purchase a specific item. The injected content attempts to replace or extend that task with an attacker-selected objective, such as locating a credential file, sending a message, opening an external URL, uploading a document, or initiating a payment.
Visible Instructions Can Still Be Difficult to Detect
A visual injection does not need to be invisible. It may work by imitating trusted interface language.
Common forms include fake system notices, CAPTCHA instructions, cookie banners, account-verification dialogs, support messages, product warnings, compliance notices, and approval prompts. The text may use phrases such as “required validation,” “administrator instruction,” or “complete this step before continuing.” The presentation gives the instruction contextual authority.
Semantic alignment has a major effect on attack success. VPI-Bench found that an injected objective was more likely to be adopted when it appeared related to the user’s legitimate task. An instruction asking an agent to enter account details on a shopping page may appear more plausible than an instruction asking the agent to modify a local configuration file during a news-summary task. es attackers an incentive to create context-sensitive payloads. An injection on a travel site may request passport information. A malicious email may ask the agent to forward a document. A product listing may direct the agent to an external checkout page. A fake cloud-storage dialog may request a local file upload.
The attack gains credibility by blending into the expected workflow rather than appearing as an obvious command to disregard prior instructions.
Instructions Hidden From the Human Viewer
Some attacks attempt to make the injected prompt accessible to automated systems but difficult for a human to notice. The exact method depends on how the agent collects interface content.
An agent that parses HTML may process text that the browser does not visibly render. Attackers can use zero-sized fonts, collapsed containers, negative screen coordinates, transparent elements, white text on a white background, hidden text areas, data attributes, SVG content, or dynamically constructed elements. Agents that consume raw DOM text, accessibility data, page source, or extracted content may receive instructions that are absent from the visible viewport.
Palo Alto Networks Unit 42 reported web-based indirect prompt injection activity in the wild in March 2026. The observed pages used combinations of font-size settings, off-screen positioning, display: none, visibility: hidden, zero opacity, color camouflage, hidden text areas, HTML attributes, SVG sections, JavaScript-generated elements, and encoded content. One observed page contained 24 separate injection attempts, apparently intended to increase the chance that at least one representation reached the target agent. research documented layered obfuscation using HTML entities, Base64, URL encoding, nested encoding, multilingual commands, bidirectional Unicode controls, and syntax-breaking strings. Some prompts were assembled at runtime through JavaScript, allowing them to evade scanners that examined only the original page source. thods expose a difficult defensive tradeoff. A security product inspecting the screenshot may miss hidden DOM content. A product inspecting the DOM may see content that the user never saw. An accessibility-based agent may receive yet another representation of the page. Each representation can contain a different instruction set.
Image-Based Prompt Injection
Image-based prompt injection places the command directly into visual media. The instruction may be clearly printed, blended into the image, positioned within a low-attention region, or rendered with colors and font sizes selected to remain readable to the model without drawing human attention.
Research published in March 2026 tested a black-box image-injection pipeline that used segmentation-based region selection, adaptive font sizing, and background-aware rendering. The method selected parts of natural images where text could be concealed without entirely losing machine readability. The most effective tested configuration reached an attack success rate of up to 64 percent under the study’s stealth constraints. ters for systems that automatically inspect uploaded images, advertisements, social-media posts, scanned documents, receipts, medical images, or screenshots. A user may upload an image for classification or summarization without knowing that the image contains a secondary instruction directed at the model.
The attack can also exploit the visual hierarchy of the scene. Large headings, high-contrast text, arrows, boxed instructions, or labels placed near interactive controls may receive more model attention than surrounding content. An attacker can use layout and typography as part of the injection rather than relying solely on the wording.
Adversarial Image Patches
A more technical form of VPI uses optimized image patches rather than ordinary visible text. These patches are modified so that their internal features steer a vision-language model toward an attacker-selected action.
Research on malicious image patches against multimodal operating-system agents found that a crafted patch could be included in a desktop background or shared image and then captured as part of the agent’s screenshot. The patch could redirect an agent toward a malicious website or another attacker-selected objective. The researchers also reported that some patches transferred across user requests, screen arrangements, and multiple agents. ss of attack targets the model’s visual representation rather than relying entirely on human-readable language. The image may contain patterns that appear meaningless or decorative to a person but produce model features associated with an instruction, object, or destination.
Recent work has extended this concept into imperceptible or near-imperceptible injection. One March 2026 study combined a bounded text overlay with optimized visual perturbations intended to align the attacked image with malicious visual and textual targets. Another May 2026 paper described an image-only cross-modal attack that optimized against internal model states, seeking to influence interpretation of both the visual input and the accompanying text prompt. tacks show why OCR-based scanning alone is incomplete. OCR may identify plainly rendered commands, yet it may fail to identify model-directed visual features that do not correspond to normal human-readable text.
Scaling-Triggered Instructions
Most multimodal systems resize images before inference. The uploaded image may be larger than the model’s accepted resolution, so the application applies nearest-neighbor, bilinear, bicubic, or another resampling algorithm.
Attackers can manipulate this preprocessing step. A crafted image may appear benign at its original resolution but reveal a different pattern after downscaling. The hidden prompt is produced through the interaction between selected source pixels and the target resizing algorithm.
Trail of Bits demonstrated this technique through its Anamorpher research and open-source tool. Anamorpher creates images whose injected text becomes visible after the image is scaled by the target pipeline. This means a user reviewing the original image may see harmless content, yet the model receives a resized version containing readable instructions. attacks expose a gap between the artifact a human approves and the artifact the model processes. Security review must account for the full preprocessing chain, including resolution changes, compression, color conversion, cropping, tiling, and frame extraction.
A preview of the original upload is not a reliable representation of the model’s final input.
VPI Against Computer-Use Agents
The greatest operational risk appears when the affected model has agency.
VPI-Bench evaluated 306 scenarios across simulated shopping, travel, news, email, and messaging platforms. The test cases included objectives involving unauthorized actions, private-data exposure, or both. The study reported that tested computer-use agents reached attack success rates of up to 51 percent on some platforms, and browser-use agents reached 100 percent in some cases. System-prompt warnings offered limited protection. inction between attempted and completed attacks also matters. An agent may start a malicious workflow but fail during a later step. That is still a security event. Opening a malicious site, reading a sensitive file, beginning an upload, or drafting an unauthorized message may expose data or create a foothold without completing the attacker’s full objective.
Email and messaging interfaces produced high susceptibility in the VPI-Bench evaluation. On email scenarios, attack-recognition rates for the tested computer-use models fell below 16 percent. The researchers linked this result partly to contextual plausibility: instructions inside an email or message naturally resemble requests that an assistant is expected to process. s in-the-wild observations show that attackers are already planting prompts aimed at database destruction, denial of service, unauthorized purchases, forced payments, and sensitive-information leakage. The report included pages instructing agents to delete databases, execute destructive system commands, make donations, purchase products, and transfer funds through external payment services. ystem Prompts Are Not a Security Boundary
A common defense places a warning in the agent’s system prompt, such as telling it to ignore instructions found in webpages or images. This can reduce some failures, but it does not create a strict trust boundary.
The model still receives the user objective, system rules, interface content, prior actions, tool output, and extracted text inside one reasoning process. An attacker can restate the malicious instruction, imitate trusted language, claim higher authority, connect it to the user’s task, or distribute it across several visual elements.
OWASP states that prompt injection stems from the way model applications process instructions and data without reliable separation. It also notes that retrieval-augmented generation and fine-tuning do not fully remove the weakness. h reached a similar result. Fine-tuning and framework-level defenses lowered some attack rates, yet successful attacks remained. System-prompt defenses produced inconsistent gains and did not stop the overall attack class. instructions written in natural language are still interpreted by the same model that is being manipulated. They are policy guidance, not an isolation control.
Building Defenses Around Trust and Execution
A sound defense begins by treating every interface-derived artifact as untrusted. Screenshots, page text, OCR results, accessibility labels, images, advertisements, emails, documents, metadata, and tool output must carry source and trust information through the agent pipeline.
The agent should know that text came from an attacker-controlled product listing rather than from the user. That provenance must remain attached after OCR, summarization, extraction, or transfer between agents. Removing source labels during preprocessing gives hostile content a chance to enter the reasoning context as ordinary instructions.
A second requirement is intent binding. Before an action is executed, the system should test whether the action is supported by the user’s authenticated request. Reading an unrelated credential file is not required to compare hotel prices. Sending an email is not required to summarize an inbox. Uploading a local document is not required to review a product page.
Recent research on provenance-aware decision auditing proposes tracing how untrusted context influences an agent’s proposed action. The ARGUS system constructs an influence graph and checks whether an action is justified by trustworthy evidence before execution. Its reported evaluation reduced attack success to 3.8 percent and retained 87.5 percent task utility in the tested settings. These results apply to that study’s benchmark and should not be treated as a universal guarantee. ict the Action Layer
The model should not be the final authority on whether a sensitive action is permitted.
Agent tools need narrow, predefined functions. A tool such as lookup_current_user_order is safer than an unrestricted SQL execution interface. A function that sends a message to a preapproved recipient is safer than an open-ended messaging tool. Identity and authorization filters should be applied by deterministic backend code rather than supplied as parameters chosen by the model.
Google’s secure-agent guidance recommends treating external data as untrusted, avoiding open-ended plan-and-execute structures, restricting the model to a small set of predefined actions, blocking dynamic tool registration, and enforcing access allowlists outside the model. It also recommends a dual-model structure in which a separate guard component checks incoming content and proposed actions. ivilege limits the blast radius when interpretation fails. An agent that can read only a temporary working directory cannot exfiltrate an entire home folder. A browser session without stored payment credentials cannot complete a purchase. A database account with read-only access cannot drop a table.
High-impact operations should require fresh user approval. The confirmation should display the exact action, target, data involved, and reason. A generic “continue?” dialog gives the user too little information to identify goal substitution.
Inspect the Model’s Actual Visual Input
Visual security testing must operate on the representation submitted to the model, not solely on the original file.
The pipeline should retain and inspect post-resize images, cropped tiles, OCR output, extracted frames, rendered PDFs, accessibility trees, and DOM text. Comparing several renderings can reveal content that appears only after scaling, color conversion, or alternate parsing.
Potential controls include OCR-based instruction detection, visual-text localization, metadata removal, image re-encoding, random resizing, patch masking, and comparisons across multiple resampling algorithms. None of these controls covers every form of VPI. Re-encoding may disrupt some perturbations but leave visible text intact. OCR may identify textual commands but miss adversarial patches. Cropping may remove a malicious border but retain an injected instruction embedded inside the subject of the image.
The result should feed a risk score used by the orchestration layer. It should not be treated as a single pass-or-fail classifier.
Detect Behavioral Deviation at Runtime
Agent monitoring should focus on the difference between the user’s goal and the agent’s proposed behavior.
Useful telemetry includes the source of each instruction-like phrase, screenshots observed before an action, OCR results, tool-call parameters, files accessed, destinations contacted, messages drafted, permission changes, and confirmation decisions. Security teams should be able to reconstruct which visual element influenced a particular action.
Runtime rules can flag actions that introduce a new domain, external recipient, local file, credential store, payment method, shell command, or privileged API that was absent from the original request. These signals are valuable even when the prompt itself does not match a known malicious pattern.
System-level isolation remains the final containment layer. VPI-Bench recommends permission gating, sandboxing, runtime monitoring, and stronger separation between human-initiated and agent-initiated operations. An operating system that can identify agent-generated actions can apply stricter rules to file deletion, software installation, credential access, and system configuration changes. ng Visual Prompt Injection
Agent security assessments need test cases across every representation the agent consumes.
Testing should include visible banners, fake dialogs, email bodies, image captions, transparent text, off-screen DOM elements, SVG instructions, dynamically generated content, multilingual prompts, encoded payloads, resized images, adversarial patches, and instructions distributed across multiple screenshots.
Scenarios should measure more than whether the model repeats the injected text. The assessment should record whether the agent recognized the attack, attempted the malicious objective, accessed protected data, issued a tool call, completed part of the workflow, requested confirmation, or reached the final attacker-selected outcome.
Tests also need semantic variation. A generic instruction may fail, yet the same objective may succeed after being framed as a required step in the user’s task. Early-stage and late-stage injection should both be tested. VPI-Bench found that later injections remained effective after the agent had already completed several legitimate steps. exercises should be performed inside isolated environments using synthetic credentials, files, payment records, and communication accounts. The objective is to evaluate the full sequence from exposure to execution without placing production data at risk.
The Interface Has Become a Security Boundary
Visual prompt injection turns interface content into a control-plane threat. Any pixel, label, image, email, banner, or rendered document encountered by an agent may contain language that competes with the user’s request.
The weakness is rooted in a core capability of multimodal systems: they are trained to interpret meaning across text and images, then act on that meaning. The same capability that lets an agent read a form, identify a button, and follow written directions also allows hostile content to present its own directions.
No image filter or defensive system prompt can fully resolve that conflict. Safer deployments need several independent controls: provenance tracking, intent binding, constrained tool interfaces, deterministic authorization, least privilege, visual preprocessing analysis, action confirmation, runtime detection, and sandboxing.
The central security question is no longer limited to whether an AI agent can interpret the screen correctly. It is whether the system can prevent the screen from becoming the attacker.
How Can Netizen Help?
Founded in 2013, Netizen is an award-winning technology firm that develops and leverages cutting-edge solutions to create a more secure, integrated, and automated digital environment for government, defense, and commercial clients worldwide. Our innovative solutions transform complex cybersecurity and technology challenges into strategic advantages by delivering mission-critical capabilities that safeguard and optimize clients’ digital infrastructure. One example of this is our popular “CISO-as-a-Service” offering that enables organizations of any size to access executive level cybersecurity expertise at a fraction of the cost of hiring internally.
Netizen also operates a state-of-the-art 24x7x365 Security Operations Center (SOC) that delivers comprehensive cybersecurity monitoring solutions for defense, government, and commercial clients. Our service portfolio includes cybersecurity assessments and advisory, hosted SIEM and EDR/XDR solutions, software assurance, penetration testing, cybersecurity engineering, and compliance audit support. We specialize in serving organizations that operate within some of the world’s most highly sensitive and tightly regulated environments where unwavering security, strict compliance, technical excellence, and operational maturity are non-negotiable requirements. Our proven track record in these domains positions us as the premier trusted partner for organizations where technology reliability and security cannot be compromised.
Netizen holds ISO 27001, ISO 9001, ISO 20000-1, and CMMI Level III SVC registrations demonstrating the maturity of our operations. We are a proud Service-Disabled Veteran-Owned Small Business (SDVOSB) certified by U.S. Small Business Administration (SBA) that has been named multiple times to the Inc. 5000 and Vet 100 lists of the most successful and fastest-growing private companies in the nation. Netizen has also been named a national “Best Workplace” by Inc. Magazine, a multiple awardee of the U.S. Department of Labor HIRE Vets Platinum Medallion for veteran hiring and retention, the Lehigh Valley Business of the Year and Veteran-Owned Business of the Year, and the recipient of dozens of other awards and accolades for innovation, community support, working environment, and growth.
Looking for expert guidance to secure, automate, and streamline your IT infrastructure and operations? Start the conversation today.



