mexca.video.anfl

Action unit (AU) relationship-aware node feature learning (ANFL).

Implementation of the ANFL module from the paper:

Luo, C., Song, S., Xie, W., Shen, L., Gunes, H. (2022). Learning multi-dimentionsal edge feature-based AU relation graph for facial action unit recognition. arXiv. https://arxiv.org/pdf/2205.01782.pdf

Code adapted from the OpenGraphAU code base (licensed under Apache 2.0).

Module Contents

Classes

GNN

Apply a graph neural network (GNN) layer.

AUFeatureGenerator

Generate action unit (AU) features.

FacialGraphGenerator

Generate action unit (AU) activations from AU features using a facial graph.

ANFL

Apply AU relationship-aware node feature learning (ANFL).
class mexca.video.anfl.GNN(in_features: int, n_nodes: int, n_neighbors: int = 4)[source]

Apply a graph neural network (GNN) layer.

Transform action unit (AU) features using digraph connectivity. Inputs and outputs correspond to AU features.

Parameters:

  • in_features (int) – Size of each input sample.

  • n_nodes (int) – Number of nodes in the digraph.

  • n_neighbors (int, default=4) – Number of top K similar neighbors for computing graph connectivity.

Notes

See eq. 1 in the corresponding paper. Functions \({g, r}\) are linear and the nonlinear activation function \(\sigma\) is ReLU. Linear layer weights are initialized with \(N(0, \sqrt{\frac{2}{out\_features}})\). Batch norm weights are initialized as 1 and biases as 0.

class mexca.video.anfl.AUFeatureGenerator(in_features: int, out_features: int = 27)[source]

Generate action unit (AU) features.

Inputs correspond to face representations (embeddings) and outputs to AU features.

Parameters:

  • in_features (int) – Size of each input sample.

  • out_features (int, default=27) – Size of each output sample.

Notes

AU specific features are generated by individual linear and global average pooling transformations.

class mexca.video.anfl.FacialGraphGenerator(in_features: int, n_main_nodes: int = 27, n_sub_nodes: int = 14, n_neighbors: int = 4)[source]

Generate action unit (AU) activations from AU features using a facial graph.

Inputs correspond to AU features and outputs to AU activations. Main plus sub nodes represent facial AUs. Sub nodes represent left and right activations of AUs 1, 2, 4, 6, 10, 12, and 14.

Parameters:

  • in_features (int) – Size of each input sample.

  • n_main_nodes (int, default=27) – Number of main nodes in the facial graph.

  • n_sub_nodes (int, default=14) – Number of sub nodes in the facial graph.

  • n_neighbors (int, default=4) – Number of top K similar neighbors for computing graph connectivity.

Notes

First applies a graph neural network (GNN()) transformation to AU features. Transformed features are fed into similarity calculating (SC) layers for main and sub nodes as in eq. 2 of the corresponding paper. Sub node activations are calulated based on matching main node features. SC layer weights are initialized using Glorot initialization (see torch.nn.init.xavier_uniform()).

class mexca.video.anfl.ANFL(in_features: int, n_main_aus: int = 27, n_sub_aus: int = 14, n_neighbors: int = 4)[source]

Apply AU relationship-aware node feature learning (ANFL).

Transform face representations into facial action unit (AU) activations. Inputs correspond to facial representations (embeddings) and outputs to AU activations.

Parameters:

  • in_features (int) – Size of each input sample.

  • n_main_aus (int, default=27) – Number of main AUs.

  • n_sub_aus (int, default=14) – Number of sub AUs.

  • n_neighbors (int, default=4) – Number of top K similar neighbors for computing graph connectivity.

Notes

First generates AU features from face representations (see AUFeatureGenerator()) and then transforms them into activations using a facial graph (see FacialGraphGenerator()).