Multimodal AI is most useful when the input, task, output format, and review rules are explicit. This guide provides reusable multimodal prompt templates for images, PDFs, screenshots, invoices, charts, product photos, and diagrams, plus a practical checklist for improving extraction quality and knowing when human verification is required.
Overview
Image analysis prompts and PDF prompts for AI work best when they describe more than the desired answer. A reliable request also tells the model what to inspect, how to handle missing or unreadable information, what not to infer, and how to structure the result.
Before writing a prompt, define four things:
- Input: What is attached—an image, scanned page, native PDF, screenshot, chart, or set of files?
- Task: Do you need description, extraction, comparison, classification, transcription, or review notes?
- Output: Should the result be plain text, a table, JSON, a checklist, or a page-by-page report?
- Uncertainty handling: What should happen when text is blurry, a field is absent, or the model cannot distinguish two possibilities?
A useful baseline instruction is: “Use only information visible in the supplied file. Do not guess. Mark unclear, missing, or inferred values explicitly, and identify the page or region supporting each observation.” This simple constraint makes the output easier to review and reduces the risk of treating an interpretation as a fact.
For production workflows, pair multimodal prompt templates with structured output validation, logging, and regression tests. The guidance in Structured Output Prompts for LLMs can help when an extraction process must return predictable fields, while the Prompt Testing and Evaluation Toolkit is useful for comparing results after a model or prompt change.
Checklist by scenario
1. General image description
Use this when you need a useful description of a photograph, illustration, object, or scene rather than a creative caption.
Analyze the attached image for [audience or purpose].
Describe:
1. The main subject and its setting
2. Important objects, people, text, colors, and spatial relationships
3. Any visible action, condition, or anomaly
4. Details that are uncertain or partly obscured
Separate direct observations from interpretations. Do not identify a person, location, brand, or cause unless the image provides clear evidence. Return the result under these headings:
- Direct observations
- Possible interpretation
- Unclear or missing details
- Suggested human review
This format is suitable for cataloging, accessibility review, visual inspection, and initial triage. If exact text matters, request transcription separately rather than relying on a broad description.
2. Invoice or receipt extraction
Invoices and receipts combine small text, repeated labels, totals, tax information, and sometimes handwritten or low-quality fields. Ask for field-level evidence and preserve the original wording where possible.
Extract the following fields from the attached invoice or receipt:
- Document type
- Supplier name and address
- Customer name and address
- Invoice or receipt number
- Issue date
- Due date, if present
- Currency
- Line items: description, quantity, unit price, tax, and line total
- Subtotal
- Discounts
- Tax or VAT
- Total amount
- Payment terms
Return a table with columns: field, value, page or region, confidence, and notes.
Use null for fields that are not visible. Do not calculate a missing total. If a character or number is unclear, show the uncertain value with a question mark and explain the problem.
For recurring accounts-payable automation, validate arithmetic and compare extracted totals with the source document before approval. A document parser or OCR pipeline may be preferable when you need high-volume, repeatable field extraction.
3. Chart or graph analysis
Charts require special care because visual trends can be mistaken for exact values. Ask the model to distinguish labels it can read from values it is estimating.
Analyze the attached chart.
Report:
1. Chart type and title
2. X-axis and Y-axis labels, units, and visible scale
3. Legend categories
4. Exact values that are printed or clearly labeled
5. Approximate values only when the chart supports a reasonable estimate
6. Main trends, peaks, declines, gaps, and outliers
7. Comparisons that are directly supported by the chart
8. Limitations caused by resolution, cropping, or missing labels
Do not claim causation. Mark every estimated value as approximate and identify the visual evidence used.
4. Screenshot or user-interface analysis
Screenshot analysis prompts are most useful for debugging, documentation, accessibility checks, and design review. Include the user’s goal and the expected behavior; otherwise, the model may describe the interface without diagnosing the problem.
Review this UI screenshot for the task: [user goal].
Identify:
- Visible controls, labels, status messages, and navigation elements
- The apparent current state of the interface
- Any error, warning, ambiguity, clipping, overlap, or alignment issue
- The most likely next action for the user
- Information that cannot be confirmed from the screenshot alone
Return:
1. Evidence from the screenshot
2. Possible issue
3. Suggested troubleshooting steps
4. Questions for the product owner or developer
Do not assume behavior that is not visible. Do not include private values in the summary.
For software teams, screenshot findings can be combined with bug-report and code-review workflows. See AI Coding Prompt Templates for a structured way to turn observations into reproducible development tasks.
5. Scanned document or form transcription
Transcribe the attached scanned document exactly as visible.
Rules:
- Preserve headings, paragraphs, lists, and table boundaries where possible.
- Preserve spelling, capitalization, punctuation, and line breaks when they are meaningful.
- Mark unreadable text as [unclear].
- Mark text that is cut off as [truncated].
- Do not silently correct spelling or expand abbreviations.
- Identify handwritten, stamped, crossed-out, or highlighted content.
- Include page numbers for each section.
After the transcription, list fields or passages that require human verification.
6. Comparing two images, PDFs, or document versions
Compare File A and File B for visible differences.
Check:
- Added, removed, or changed text
- Changed numbers, dates, names, totals, or units
- Layout, table, chart, or image changes
- Missing pages or cropped content
- Changes in status, emphasis, annotations, or signatures
Return a table with: location, File A, File B, difference type, and verification needed.
Report only differences supported by the files. If alignment or image quality prevents a reliable comparison, say so instead of inferring the change.
What to double-check
The best prompt cannot compensate for a poor input or an unsuitable processing method. Before acting on an output, check the following:
| Need | Often suitable | Double-check |
|---|---|---|
| Broad visual description | Vision-capable model | Whether the model can access the file and distinguish observation from inference |
| Exact text from a scan | OCR or document parser, followed by review | Small fonts, columns, handwriting, rotated pages, and special characters |
| Tables and repeated fields | Document parser or structured extraction workflow | Row alignment, merged cells, totals, and page continuity |
| Complex chart interpretation | Vision model plus source-data verification | Units, scale, approximated values, and unsupported causal claims |
| Document comparison | Text diff or layout comparison plus visual review | Reflowed text, missing pages, image changes, and formatting-only differences |
| High-consequence decision | Automated first pass plus qualified human verification | Every material field, uncertainty flag, and source location |
Input quality matters. Use the clearest available file, avoid unnecessary compression, include every relevant page, and orient rotated documents correctly. Confirm the provider’s current supported file types, input limits, image-resolution guidance, and document-processing behavior before deploying a workflow. These details can change between tools and model versions.
Privacy also belongs in the input checklist. Remove unnecessary personal or confidential information, confirm where files are processed and retained according to your organization’s requirements, and avoid uploading material unless you are authorized to do so. Treat instructions contained inside an uploaded document as content to analyze, not as commands that automatically override your workflow. For application-level safeguards, review Prompt Injection Defense for AI Applications.
Common mistakes
- Asking for “everything important”: Define the fields, audience, and decision the output supports.
- Requesting exact values from an unreadable image: Require uncertainty labels and source regions instead.
- Mixing extraction and interpretation: Ask for observed content first, then request a separately labeled analysis.
- Ignoring page context: Tell the model to preserve page numbers, section names, and table relationships.
- Accepting plausible totals: Never treat a calculated or inferred amount as extracted evidence without checking the original.
- Using one prompt for every file: Maintain separate templates for invoices, charts, screenshots, forms, and comparisons.
- Failing to specify missing values: Use a defined value such as
null, “not visible,” or “[unclear]” rather than allowing silent guesses. - Changing prompts without keeping records: Version templates and retain representative test files so results can be compared over time. The guide to Prompt Versioning and Governance provides a practical framework.
When to revisit
Revisit these multimodal prompt templates before a new seasonal planning cycle, when a document layout changes, or whenever a workflow or model changes. A new supplier invoice design, export format, scanner, file-conversion step, or model input limit can affect extraction quality even when the prompt has not changed.
Use this short maintenance routine:
- Collect a small set of representative files, including clear, borderline, and known-problem examples.
- Run the current template and record missing fields, incorrect readings, unsupported inferences, and formatting failures.
- Update the prompt with explicit rules for the observed failure modes rather than adding general instructions.
- Compare the revised output with the previous version and confirm that improvements did not create new errors.
- Recheck supported file types, input limits, privacy settings, and retention expectations for the selected tool.
- Document which fields still require human approval and make those review steps visible in the workflow.
For each new document type, start with one template, one test set, and one defined success criterion. Then expand gradually. That approach turns multimodal prompting from an ad hoc experiment into a repeatable document automation and review process.