Multimodal GraphRAG: Connecting Text, Images, and Audio

A technician photographs a cracked housing on pump 4471. A customer service call about the same pump is recorded and transcribed. A warranty claim references the part number. A maintenance log records a vibration reading from the week before. Four artifacts, four modalities, one physical object.

Today’s multimodal models can interpret every one of those individually, and do it well. Show a model the photograph and it will describe the crack. Give it the transcript and it will summarize the complaint. What no model does on its own is tell you that these four things are about the same pump — because that fact is not in the image, not in the audio, and not learnable from either.

Connecting them is a data problem, and it is the one that decides whether multimodal AI produces a party trick or an answer.

What multimodal models solved, and what they didn’t

The last few years genuinely changed what is possible at the perception layer. A single model can now take an image, a document, and an audio clip and produce coherent output about all three. That is a real capability and it removes an entire category of pipeline work — the bespoke OCR, the separate speech-to-text service, the image classifier trained on 400 labeled examples.

What it did not do is give the enterprise a shared notion of what things are.

Perception tells you the image contains a cracked casting. It does not tell you that the casting belongs to pump 4471, that pump 4471 is installed at the Rotterdam facility, that it is covered under a warranty expiring in November, or that three similar failures were reported on the same production lot. Each of those is a relationship to something outside the artifact. No amount of looking at the picture recovers them.

This is the same distinction we drew between retrieval and reasoning in GraphRAG vs. Vector RAG, showing up one layer earlier. Similarity search finds artifacts that resemble your query. It does not establish that two artifacts concern the same entity, and resemblance is a poor proxy for identity — two photographs of the same pump taken from different angles may embed further apart than photographs of two different pumps taken from the same angle.

The unit of connection is the entity, not the embedding

The instinctive architecture for multimodal retrieval is a shared embedding space: encode text, images, and audio into the same vector space so that a query in one modality retrieves artifacts in another. This works, and it is genuinely useful for the task it was designed for — finding an image that matches a description, or a clip that matches a phrase.

It is the wrong primitive for enterprise questions, for three reasons that show up immediately in production.

It retrieves resemblance, not membership. “Show me everything we have on pump 4471” is a request for a complete, correct set. A vector search returns the top-k most similar artifacts, which is neither complete nor guaranteed to be about the right pump. If the fifth-ranked result is the one that matters, and you retrieved four, the answer is wrong and nothing in the system indicates it.

It cannot express relationships between artifacts. The photograph was taken during the site visit that generated the report that triggered the claim. That sequence is what makes the story legible, and it is not a similarity relationship. It is a set of typed edges.

It has no place to put governance. The audio recording may be subject to consent restrictions the photograph is not. The claim document may be privileged. Rights, retention, and access differ per artifact and often per region, and an undifferentiated vector index has nowhere to record that, let alone enforce it.

The alternative is to make the entity the anchor. Each artifact — image, transcript, document, sensor reading — becomes a node in the graph, described by its modality and its provenance, and linked to the entities it depicts or mentions. Retrieval then starts from the entity and traverses outward, returning every artifact connected to it, across every modality, with the relationships intact and access policy applied per fact.

The query stops being “find things that look like this” and becomes “return what we know about this pump, and how we came to know it.”

How multimodal GraphRAG works in practice

The pipeline has four stages, and the interesting engineering is concentrated in the second.

Perceive. Models process each artifact in its native modality and produce structured output: entities and relationships extracted from a document, objects and conditions identified in an image, speakers and topics and named entities from an audio transcript. Multimodal models made this stage straightforward, and you should use them aggressively here. This is the neural half of the neural-symbolic loop doing what it is best at — reading input nobody could specify in advance — and the stages below are the symbolic half holding it to account.

Anchor. Every extracted mention is resolved to a canonical entity in the graph — or explicitly flagged as unresolvable. “The pump,” “4471,” “unit 4471-B,” and a serial number visible in the photograph all have to land on the same node, or they land on four, and every downstream answer fragments. Entity resolution applied to model output is where multimodal projects succeed or quietly fail. A mention the system cannot resolve confidently should be quarantined for review rather than attached to a best guess, because a wrongly anchored artifact is worse than an unanchored one — it contaminates an entity’s history with another entity’s facts.

Link. Artifacts are connected to each other as well as to entities: this transcript is about the incident that this photograph documents, both of which concern this asset, under this claim. Each link carries provenance — which system produced it, when, at what confidence, and whether a model or a person asserted it. This is the step that turns a pile of anchored files into something traversable.

Traverse and answer. At query time, retrieval starts from the resolved entity and walks the graph, gathering the connected artifacts and facts that the asking identity is permitted to see. The model receives pre-connected context — the pump, its installation, its failure history, the photograph, the transcript summary, the warranty terms — rather than a ranked list of loosely similar files, and generates an answer it can cite.

That last row matters more than it looks. Enterprises keep acquiring modalities — video walkthroughs, CAD files, telemetry, scanned signatures. In an entity-anchored design, a new modality is new node types and new edges attached to entities that already exist. In an embedding-first design, it is a new encoder, a new index, and an open question about how its vectors relate to the existing ones.

Our position on the perceive stage: deterministic first, models when earned

We should be concrete about how the perception stage ought to be built, because the default assumption — send every page to a vision model and let it figure things out — is both the most expensive option and, on a lot of documents, not the most accurate one.

Fluree Doc Parse is our open implementation of the alternative. It reads PDF, HTML, Markdown, DOCX, PPTX, and raster images through a three-tier escalation pipeline rather than a single model pass:

• Tier 1 is pure deterministic extraction — Rust, no models, no GPU — using the document’s own geometry and text layer. On the 200-document opendataloader-bench corpus it scores 0.8896 overall at roughly 8 milliseconds per document.
• Tier 2 adds model-assisted layout arbitration for headings and table regions where the deterministic read is genuinely disputed, reaching 0.8967 at about 0.2 seconds.
• Tier 3 is pixel-level deep reading for content the earlier tiers could not recover — scanned pages, figures, glyphs with no usable text layer — reaching 0.9297 at about 1.5 seconds. On that benchmark only 87 of 200 documents escalate that far.

Two design principles do the work, and both are stated in the repository. The first: “Models arbitrate structure; the page tier owns its page.” A deterministic reading stands unless a model’s proposed shape meaningfully disagrees with it, and where two independent readers agree against the deeper model, they win. The second: “Escalation is earned, not configured.” There is no threshold to tune — each tier’s own signals decide whether the next tier runs, per document and per region.

The result worth noting is that the deterministic engine alone ranks third among seventeen systems on that public benchmark. Models are not optional because they are unnecessary; they are reserved because most pages do not need them, and the ones that do are identifiable.

This matters for multimodal retrieval for a reason beyond cost. Deterministic extraction is reproducible — the same PDF yields the same structure every run — which is exactly the property that makes committed structure an asset rather than a recurring computation. Where a model does arbitrate, that fact is recorded rather than blended in.

The output shape reinforces the anchoring argument above. Documents emit as DoCO JSON-LD graphs — Discourse Components — preserving hierarchical containment down to character offsets. Every element carries bounding-box provenance tying it to its page coordinates, and an IRI stamped via --doc-iri, so a re-run retracts and replaces precisely what changed. They are, in the repository’s phrasing, insertable into a Fluree ledger as-is.

That is the “handoff must be typed data” principle applied to the messiest input an enterprise has. The parser does not hand the graph a summary of the document; it hands it addressable, provenance-carrying statements about identified elements — the raw material anchoring needs.

Three patterns where this pays off

The value concentrates wherever the evidence for a decision is scattered across formats — which is most of the physical and regulated economy.

Asset-intensive operations

Manufacturing, energy, and field service accumulate photographs, inspection reports, sensor histories, service manuals, and technician voice notes about the same equipment. The question that matters — “has this failure mode appeared on this component elsewhere in the fleet?” — requires connecting an image of a crack to a text description of a similar crack to the lot number they share. Anchored to the asset and the part lot, that is a two-hop traversal. Left in separate stores, it is a research project that happens after the recall.

Media archives and rights

A broadcast archive is the purest case: video, audio, transcripts, stills, captions, and contracts, all describing the same segments, people, and events, with rights that vary by territory and expire on different dates. The commercial question is almost always “what can we license, to whom, in which market, right now” — a governance query as much as a search query. We built archival cataloguing around exactly this shape, because rights that live in a contract PDF and content that lives in a video file are useless to each other until both attach to the same entities.

Regulated communications

Financial services and life sciences generate recorded calls, submitted documents, correspondence, and structured system records that regulators expect to be reconcilable. Answering “what was communicated to this client about this product, across every channel, before this date” means traversing from a client entity to artifacts in three modalities, with timestamps and provenance intact. Retrieval by resemblance cannot produce that answer defensibly, because the standard is completeness and the evidence is the path, not the similarity score.

What gets hard

Multimodal GraphRAG is not free, and it is worth being direct about where the difficulty concentrates.

Anchoring accuracy is the whole ballgame. Extraction quality gets the attention; resolution quality determines the outcome. A model that extracts entities at 95% accuracy but anchors them correctly only 80% of the time produces a graph where one artifact in five is attached to the wrong thing. That graph is worse than no graph, because the errors are invisible and they compound. Instrument resolution confidence from day one and quarantine aggressively.

Provenance has to survive the pipeline. By the time an answer is produced, several models have touched the evidence. Which statement came from the transcript, which from the image, which was inferred by a model, and which was asserted by a person — those distinctions have to be recorded per fact, or the audit trail is a narrative. The W3C PROV ontology exists precisely to express this kind of derivation chain, and a graph is the natural place to hold it. This is the same requirement we described in Grounding LLMs on Solid Knowledge, and it gets harder with each modality added, because each one introduces another inference step between the source and the claim.

Artifacts version, and so do their interpretations. A transcript is corrected. An image is re-analyzed by a better model and the finding changes. The underlying artifact did not change but what the system believes about it did, and both states need to be reconstructable — you may be asked why a decision was made on the basis of the earlier interpretation. Supersession, rather than overwriting, is the mechanism, and it is the same one that keeps agent memory from serving contradictory facts.

Rights and consent are per-artifact and per-region. Audio in particular carries consent constraints that text extracted from it may or may not inherit. Encoding that as policy at the data layer, rather than as filtering logic in each application, is what makes it hold as the number of consuming systems grows.

What to build first

The tempting first project is a universal multimodal index across everything the enterprise has. It is also the one most likely to be quietly shelved in month seven.

The better starting point is one entity type whose story is genuinely split across formats, and one question that is currently answered by a person opening four systems. Model that entity properly. Anchor artifacts from two modalities to it — usually documents and one of image or audio. Get resolution confidence to a level you would defend, with a real quarantine queue for the rest. Then answer the question end to end and measure whether it holds up against how the person does it today.

That scope is small enough to finish and complete enough to prove the architecture, because the hard parts — resolution, provenance, policy, supersession — all appear at two modalities. Adding the third and fourth is mostly repetition of a solved pattern. The teams that struggle are almost always the ones that added modalities before they trusted their anchoring, and ended up with a large graph nobody could vouch for.

It also compounds in the right direction. Every artifact anchored to an entity makes the next question about that entity cheaper to answer, and the unstructured data pipeline that resolves and validates those artifacts is the same one serving every other consumer of the graph. The multimodal capability ends up being a property of the data foundation rather than a separate system — which is the outcome worth aiming for.

The pattern underneath all of this is one we keep arriving at from different directions. Models are getting better at perceiving the world in every format an enterprise stores it in, and that capability keeps arriving faster than most organizations can absorb it. What does not arrive with the model is the knowledge of what your things are, how they relate, who may see them, and how you came to believe any of it. That has to be built, and once built it serves every modality — including the ones you have not adopted yet.

If you are working through what anchoring would look like across your own content, talk to our team or see how the retrieval side is put together in GraphRAG.