# No code snippets available yet for this library.
# To use this model, check the repository files and the library's documentation.
# Want to help? PRs adding snippets are welcome at:
# https://github.com/huggingface/huggingface.jsblendshape + pose → MediaPipe 478 mesh PLS
Licensing scope: See the license and provenance notice before relying on this card's license metadata for pretrained-weight redistribution or commercial use. Existing valid grants are preserved.
Current default: bs_to_mesh_pls_v6.npz (Detectorv2 v2.8 blendshape space; py-feat
load_blendshape_face_mesh_model() / plot_face_mesh(blendshapes=...)).
Versions
| file | features / meshes from | held-out R² | CV | notes |
|---|---|---|---|---|
bs_to_mesh_pls_v6.npz |
Detectorv2 v2.8 | 0.739 ± 0.003 | 5-fold GroupKFold by video | current default, k = 211 |
bs_to_mesh_pls_v5.npz |
MediaPipe (MPDetector) | 0.639 ± 0.003 | 3-fold GroupKFold by video | legacy; MediaPipe's own blendshape space |
Match the model to the detector that produced your blendshapes: v6 for Detectorv2 (py-feat ≥ 2.1.3), v5 for MPDetector/MediaPipe blendshapes.
Training data (v6)
- 602,427 frames / 33,775 CelebV-HQ videos run through the py-feat
Detectorv2 v2.8 release weights (
face_multitask_v28.safetensors, md59e174bb3cd05ce8bed7ac64358a40332). Features and target meshes come from the same forward pass, so the map is self-consistent with what the detector emits. - Same frames, pose filter (|yaw| ≤ 40°, |pitch| ≤ 30°), 12-anchor GPA alignment
and top-1% anchor-residual outlier drop as
py-feat/au_to_meshv6. - Canonicalized with an Umeyama similarity on all 478 vertices onto the
au_to_meshv4 frontal mean, the same recipe asau_to_mesh_pls_v6, so AU, emotion and blendshape renders share one frame.
Method (v6)
- Training design
[52 blendshape | 3 pose | 156 pose × blendshape] = 211features; kernel PLS equivalent toPLSRegression(scale=True), with k chosen by mean held-out R² (k = 211; knee at 167). - Deployed rows = blendshape + pose (interactions vanish at pose = 0).
- Per-fold R² = [0.7449, 0.7381, 0.7357, 0.7382, 0.7391].
Input range and unsupported shapes
feature_p50 / feature_p95 / feature_p99 give the v2.8 training-distribution
percentiles. py-feat clips inputs to [0, feature_p99] by default
(clip=False to disable): the map is linear and only constrained within the
range of detector outputs it was fit on.
Seven shapes are never expressed by Detectorv2 v2.8 (p99 < 0.02) and their
coefficients are unconstrained: _neutral, cheekPuff, cheekSquintLeft, cheekSquintRight, jawForward, noseSneerLeft, noseSneerRight.
py-feat warns if they are non-zero; driving them toward 1.0 without clipping
produces a degenerate mesh.
Known limitations
- Direction checks pass for all 21 tested shapes (jawOpen opens the mouth and drops the chin; eyeBlinkLeft closes the subject-left eye; mouthSmileLeft raises the subject-left corner; etc.).
- Displacement is not strictly local. Eye shapes (blink, look, squint, wide) move the correct eye but also carry correlated lower-face and whole-face motion, because blendshapes co-vary in natural video and a linear map on absolute coordinates inherits that covariance. v5 shows the same pattern.
- Left/right mirror symmetry: median cosine 0.86 across pairs (v5: 0.82);
mouthUpperUp,mouthPressandmouthLowerDownpairs are the least symmetric.
Inference
from feat.plotting import plot_face_mesh, predict_face_mesh_from_features
plot_face_mesh(blendshapes={"jawOpen": 0.6}) # dict / Series / fex.blendshapes row
Or with numpy only:
import numpy as np
m = np.load("bs_to_mesh_pls_v6.npz")
cols = m["feature_columns"].tolist()
x = np.zeros(len(cols) + 3, dtype=np.float32) # [features | Pitch, Yaw, Roll]
x[cols.index(...)] = ...
flat = x @ m["coef"] + m["intercept"] # (1434,)
mesh = np.stack([flat[:478], flat[478:956], flat[956:]], axis=1) # axis-major [x|y|z]
File format
NPZ: coef (k+3, 1434) f32; intercept (1434,) f32; input_columns;
feature_columns (authoritative order; index by name); feature_name;
pose_columns; mean_aligned_mesh (478, 3); mean_neutral_mesh (478, 3);
reference_anchors (12, 3); anchor_indices (12,); n_components;
model_card; training_metadata (JSON); v6 adds feature_p50, feature_p95, feature_p99, unsupported_features.
License and provenance
This distribution retains its existing license: mit metadata. Py-Feat's original visualization code is MIT. The historical MIT designation for this artifact does not itself settle third-party dataset or teacher rights in its fitted parameters.
The v6 artifact uses Detectorv2 v2.8 blendshape and mesh outputs. The older v5 artifact uses MediaPipe/MPDetector blendshape space and must be assessed separately. These models use CelebV-HQ frames. The CelebV-HQ agreement restricts commercial exploitation of videos and derived data. The v2.8 detector has an ImageNet-initialized CC BY-NC 4.0 backbone checkpoint and its own dataset/teacher permission questions, summarized in the multitask notice.
Whether particular terms govern the predictions used for fitting or these PLS coefficients requires separate assessment. Review each file's training manifest and applicable agreements; do not infer either unrestricted permission or automatic transfer of every teacher restriction. Newer agreements must not be assumed to apply retroactively.
This notice clarifies scope; it does not revoke existing valid grants or create a new license for third-party material. Dataset and teacher terms do not automatically relicense every trained artifact or inference output. A software license or model-card badge alone does not establish all checkpoint redistribution or commercial-use permissions.