Transformers documentation

关键点匹配

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.23.0).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

关键点匹配

关键点匹配用于匹配出现在两张不同图像中、属于同一物体的不同兴趣点。大多数现代关键点匹配器以图像作为输入,并输出以下内容:

  • 关键点坐标 (x,y):使用两个列表在第一张图像和第二张图像之间建立像素坐标的一一映射。第一个列表中给定索引处的关键点与第二个列表中相同索引处的关键点相匹配。
  • 匹配分数:分配给关键点匹配的分数。

在本教程中,你将使用通过 MatchAnything 框架 训练的 EfficientLoFTR 模型提取关键点匹配,并对匹配结果进行优化。该模型仅包含 16M 参数,可在 CPU 上运行。你将使用 AutoModelForKeypointMatching 类。

from transformers import AutoImageProcessor, AutoModelForKeypointMatching
import torch

processor = AutoImageProcessor.from_pretrained("zju-community/matchanything_eloftr")
model = AutoModelForKeypointMatching.from_pretrained("zju-community/matchanything_eloftr")

加载两张包含相同目标对象的图像。第二张照片拍摄于第一张之后一秒,其颜色经过编辑,并且被进一步裁剪和旋转。

Bee Bee edited
from transformers.image_utils import load_image
image1 = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg")
image2 = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee_edited.jpg")

images = [image1, image2]

我们可以将图像传递给处理器并进行推理。

inputs = processor(images, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)

我们可以对输出进行后处理。threshold 参数用于细化输出匹配中的噪声(即置信度较低的阈值)。

image_sizes = [[(image.height, image.width) for image in images]]

outputs = processor.post_process_keypoint_matching(outputs, image_sizes, threshold=0.2)
print(outputs)

以下是输出结果。

[{'keypoints0': tensor([[4514,  550],
          [4813,  683],
          [1972, 1547],
          ...
          [3916, 3408]], dtype=torch.int32),
  'keypoints1': tensor([[2280,  463],
          [2378,  613],
          [2231,  887],
          ...
          [1521, 2560]], dtype=torch.int32),
  'matching_scores': tensor([0.2189, 0.2073, 0.2414, ...
    ])}]

我们已对输出进行了裁剪,但仍有 401 个匹配项!

len(outputs[0]["keypoints0"])
# 401

我们可以使用处理器的 visualize_keypoint_matching() 方法来可视化它们。

plot_images = processor.visualize_keypoint_matching(images, outputs)
plot_images

Matched Image

可选地,您可以使用 Pipeline API 并将任务设置为 keypoint-matching。

from transformers import pipeline 

image_1 = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image_2 = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee_edited.jpg"

pipe = pipeline("keypoint-matching", model="zju-community/matchanything_eloftr")
pipe([image_1, image_2])

输出如下所示。

[{'keypoint_image_0': {'x': 2444, 'y': 2869},
  'keypoint_image_1': {'x': 837, 'y': 1500},
  'score': 0.9756593704223633},
 {'keypoint_image_0': {'x': 1248, 'y': 2819},
  'keypoint_image_1': {'x': 862, 'y': 866},
  'score': 0.9735618829727173},
 {'keypoint_image_0': {'x': 1547, 'y': 3317},
  'keypoint_image_1': {'x': 1436, 'y': 1500},
  ...
 }
]
Update on GitHub