Rikunarita-ORG/tipsv2-so400m14-dpt-merged

TIPSv2 — SO400m/14 DPT Merged

This repository provides a merged version of Google's TIPSv2 SO400m/14 backbone and SO400m/14 DPT heads in a single model.safetensors file.

The original Google release separates the vision-language backbone and DPT dense prediction heads into two repositories. This model combines both components into a unified checkpoint for easier deployment and archival.

Base Models

• Backbone: google/tipsv2-so400m14
• DPT Heads: google/tipsv2-so400m14-dpt

Original work by Google Research.

Model Overview

TIPSv2 (Text-Image Pre-training with Spatial Awareness) is a family of contrastive vision-language models that produce spatially rich image representations aligned with text embeddings.

The SO400m/14 variant contains:

Component	Parameters
Vision Encoder	412M
Text Encoder	448M
Embedding Dimension	1152

The DPT extension adds dense prediction heads for:

• Depth Estimation
• Surface Normal Prediction
• Semantic Segmentation (ADE20K, 150 classes)

Architecture

Backbone:

• ViT Vision Encoder
• 27 Vision Transformer Layers
• 14×14 Patch Size
• 1152-dimensional embeddings

Dense Prediction Heads:

• DPT Depth Head
• DPT Surface Normal Head
• DPT Semantic Segmentation Head

Training datasets used by the original Google models:

• NYU Depth V2 (depth estimation)
• NYU Depth V2 (surface normals)
• ADE20K (semantic segmentation)

Input Requirements

Images should:

• Be converted to tensors in the [0, 1] range
• Not use ImageNet normalization
• Prefer resolutions divisible by 14

Example preprocessing:

from torchvision import transforms

transform = transforms.Compose([
    transforms.Resize((448, 448)),
    transforms.ToTensor(),
])

Usage

from transformers import AutoModel

model = AutoModel.from_pretrained(
    "Rikunarita-ORG/tipsv2-so400m14-dpt-merged",
    trust_remote_code=True
)

Depth Estimation

depth = model.predict_depth(pixel_values)

Surface Normals

normals = model.predict_normals(pixel_values)

Semantic Segmentation

segmentation = model.predict_segmentation(pixel_values)

Run All Heads

outputs = model(pixel_values)

depth = outputs.depth
normals = outputs.normals
segmentation = outputs.segmentation

Merged Checkpoint Notes

This repository is not an official Google release.

The checkpoint was created by merging:

• TIPSv2 SO400m/14 backbone weights
• TIPSv2 SO400m/14 DPT head weights

into a single model.safetensors file for convenience.

No additional training, finetuning, quantization, pruning, or weight modification was performed.

License

This repository inherits the Apache 2.0 license from the original Google TIPSv2 releases.

Citation

If you use this model, please cite the original TIPSv2 paper:

@inproceedings{cao2026tipsv2,
  title     = {TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment},
  author    = {Cao, Bingyi and Chen, Koert and Maninis, Kevis-Kokitsi and Chen, Kaifeng and Karpur, Arjun and Xia, Ye and Dua, Sahil and Dabral, Tanmaya and Han, Guangxing and Han, Bohyung and Ainslie, Joshua and Bewley, Alex and Jacob, Mithun and Wagner, Rene and Ramos, Washington and Choromanski, Krzysztof and Seyedhosseini, Mojtaba and Zhou, Howard and Araujo, Andre},
  booktitle = {CVPR},
  year      = {2026}
}