CLIP4Cir
Retrieves target images using a reference image paired with text modifications via fine-tuned CLIP features.
Combine image queries with natural language text edits to search visual databases with multimodal embeddings.
CLIP4Cir is an open-source PyTorch implementation for Composed Image Retrieval (CIR) based on OpenAI's CLIP architecture. It allows developers and researchers to retrieve target images using a baseline reference image combined with descriptive text edits. It suits video research and media indexing teams building multimodal search systems.
What it is
The framework fine-tunes both CLIP encoders using contrastive loss on combined image-text queries and target images. It then trains a dedicated Combiner network from scratch to merge image and text feature vectors into a unified embedding for database lookup. Pre-trained weights for FashionIQ and CIRR datasets are provided alongside training and evaluation scripts.
Who it's for
This tool suits computer vision researchers, video media asset management engineers, and ML developers building image-and-text query pipelines. It is not designed for end users looking for a finished desktop video editor GUI or plug-and-play browser tool.
Key features
Task-Oriented CLIP Fine-Tuning
Updates both visual and textual CLIP encoders using an element-wise feature sum and contrastive loss.
Dedicated Multimodal Combiner Network
Trains a specialized fusion network to combine image and relative caption features into a single query vector.
Pre-Trained Benchmark Models
Provides ready-to-use PyTorch model checkpoints trained on the FashionIQ and CIRR datasets via GitHub releases.
Structured Dataset Evaluation Scripts
Includes scripts for validation and CIRR test submission generation alongside Comet.ml experiment logging support.
Up and running in 3 steps
Our honest verdict
What's great
- Combines visual reference images with text descriptions for precise image and frame retrieval.
- Outperforms standard CLIP baselines on benchmark datasets like FashionIQ and CIRR.
- Provides pre-trained CLIP and Combiner network weights for quick evaluation.
- Includes evaluation scripts for validation sets and test predictions.
Where it falls short
- Requires CUDA-capable GPU hardware and heavy compute resources to fine-tune encoders and train the Combiner.
- Code lacks internal speed optimizations like gradient checkpointing and feature pre-computation.
- Strict dataset directory layout requirements for training and testing workflows.
- Command-line and code library only, lacking a visual user interface.
Pricing & license
CLIP4Cir is completely free and open source under the MIT license. The primary costs come from providing your own GPU compute infrastructure and storage for large image datasets.
Open remote AI-training roles
879 open remote AI-training roles, updated November 15, 2026; most pay $20–$175 an hour. Zekai lists them free; the application goes to the employer’s platform.
Alternatives
Use it, or have us set it up
FAQ
What is Composed Image Retrieval?
Composed Image Retrieval allows you to search a visual database using a reference image alongside a text prompt specifying desired changes.
Can I run CLIP4Cir without a GPU?
While you can theoretically load PyTorch on CPU, running fine-tuning or full dataset retrieval within a practical timeframe requires a CUDA-capable GPU.
Does CLIP4Cir provide pre-trained model weights?
Yes, pre-trained CLIP and Combiner network checkpoints for FashionIQ and CIRR are hosted on the project GitHub releases page.
