Skip to content

Part III: Multimodal Data Engineering

Positioning

Part III covers image-text, documents, video, audio, and cross-modal alignment. It discusses multimodal sample structures, visual and acoustic cleaning, re-captioning, document understanding, temporal slicing, cross-modal alignment, and fusion training. This part extends the text pre-training pipeline from Part II into images, layouts, timelines, and heterogeneous modality spaces.

Multimodal data introduces four additional challenges. First is alignment: text, images, audio, and video do not naturally describe the same object, so semantic, spatial, or temporal anchors are needed. Second is representation: visual and acoustic signals must pass through patches, features, tokens, or placeholders before they can enter language-model training interfaces. Third is evaluation: image clarity, OCR reliability, audio-video synchronization, and cross-modal semantic consistency are hard to judge with cheap text metrics alone. Fourth is cost: decoding, re-captioning, OCR, ASR, vector scoring, and manual sampling all raise preprocessing and pre-training validation cost.

Chapters 8 through 11 start with image-text data engineering, then move to re-captioning and document understanding, then video and audio data engineering, and finally cross-modal alignment and fusion. The main line is this: single-modality cleaning does not prove cross-modal supervision is valid.

Learning Objectives

  • Distinguish the engineering differences among image-text pairs, interleaved image-text, document screenshots, audio/video clips, and cross-modal fusion samples.
  • Master basic flows for image quality filtering, semantic alignment, re-captioning, OCR enhancement, and document structuring.
  • Understand how video slicing, ASR, diarization, audio-video synchronization, and event labels form long-temporal samples.
  • Design object-level, segment-level, and document-level cross-modal alignment data and identify common failure modes.
  • Evaluate whether a multimodal data pipeline can enter training from the perspectives of cost, compliance, sampling, and traceability.

Prerequisites

Readers should understand the methods in Part II for text pre-training sources, cleaning, deduplication, tokenization, serialization, and quality loops. Readers from computer vision, speech, or multimedia systems may focus on how image, audio, and video signals are converted into training-readable tokens, features, JSONL schemas, and metadata constraints.

Chapter Logic

Chapter 8 establishes the asset view of static image-text samples, including image cleaning, CLIP/SigLIP filtering, AnyRes, and data mixture. Chapter 9 moves into fine-grained supervision production, including re-captioning, OCR, layout structure, and document understanding samples. Chapter 10 handles the time dimension through video slicing, audio transcription, acoustic separation, and temporal alignment. Chapter 11 integrates the previous chapters into cross-modal fusion samples, with emphasis on alignment layers, hard negatives, mixture ratios, and evaluation loops.

This part connects backward to the text pre-training pipeline in Part II and forward to instruction alignment and preference data in Part IV. In other words, Part III answers whether the model can see, hear, and align different modalities; Part IV then asks how the model should act under human instructions and preferences.

Terminology Conventions

This part uses "multimodal data engineering" to refer to the collection, cleaning, re-captioning, alignment, and packaging of signals such as images, text, video, audio, and documents. Image-text pairs, document samples, audio/video clips, and cross-modal fusion samples should be described separately rather than collapsed into generic "data." In this part, "alignment" specifically means binding semantic, temporal, spatial, and structural relationships between modalities; it is not the same as the later value or preference alignment discussed in model alignment.

Part Contents

Suggested order: read Chapters 8 through 11 sequentially. Readers with VLM experience may first read Chapter 11 to understand cross-modal alignment objectives, then return to Chapters 8, 9, and 10 to locate where each alignment signal comes from.