7th Workshop and Competition on


Affective Behavior Analysis in-the-wild (ABAW)

in conjunction with the European Conference on Computer Vision (ECCV), 2024

30 September 2024, Milan, Italy

About ABAW

The ABAW Workshop and Competition has a unique aspect of fostering cross-pollination of different disciplines, bringing together experts (from academia, industry, and government) and researchers of mobile and ubiquitous computing, computer vision and pattern recognition, artificial intelligence and machine learning, multimedia, robotics, HCI, ambient intelligence and psychology. The diversity of human behavior, the richness of multi-modal data that arises from its analysis, and the multitude of applications that demand rapid progress in this area ensure that our events provide a timely and relevant discussion and dissemination platform.

The ABAW Workshop and Competition is a continuation of the respective Workshops and Competitions held at IEEE CVPR 2024, IEEE CVPR 2023, ECCV 2022, IEEE CVPR 2022, ICCV 2021, IEEE FG 2020 (a), IEEE FG 2020 (B) and IEEE CVPR 2017 Conferences.

Organisers

Dimitrios Kollias

Queen Mary University of London, UK d.kollias@qmul.ac.uk

Stefanos Zafeiriou

Imperial College London, UK s.zafeiriou@imperial.ac.uk

Irene Kotsia

Cogitat Ltd, UK irene@cogitat.io

Abhinav Dhall

Flinders University, Australia abhinav.dhall@flinders.edu.au

Shreya Ghosh

Curtin University, Australia shreya.ghosh@curtin.edu.au

Data Chairs

Chunchang Shao,     Queen Mary University of London, UK
Guanyu Hu,                 Queen Mary University of London, UK & Xi'an Jiaotong University, China

The Workshop



Call for Papers

This Workshop will solicit contributions on the recent progress of recognition, analysis, generation-synthesis and modelling of face, body, gesture, speech, audio, text and language while embracing the most advanced systems available for such in-the-wild (i.e., in unconstrained environments) analysis, and across modalities like face to voice. In parallel, this Workshop will solicit contributions towards building fair, explainable, trustworthy and privacy-aware models that perform well on all subgroups and improve in-the-wild generalisation.

Original high-quality contributions, in terms of databases, surveys, studies, foundation models, techniques and methodologies (either uni-modal or multi-modal; uni-task or multi-task ones) are solicited on -but are not limited to- the following topics:

    facial expression (basic, compound or other) or micro-expression analysis

    facial action unit detection

    valence-arousal estimation

    physiological-based (e.g.,EEG, EDA) affect analysis

    face recognition, detection or tracking

    body recognition, detection or tracking

    gesture recognition or detection

    pose estimation or tracking

    activity recognition or tracking

    lip reading and voice understanding

    face and body characterization (e.g., behavioral understanding)

    characteristic analysis (e.g., gait, age, gender, ethnicity recognition)

    group understanding via social cues (e.g., kinship, non-blood relationships, personality)

    video, action and event understanding

    digital human modeling

    deepfake generation, detection and temporal deepfake localization

    characteristic analysis (e.g., gait, age, gender, ethnicity recognition)

    violence detection

    autonomous driving

    domain adaptation, domain generalisation, few- or zero-shot learning for the above cases

    fairness, explainability, interpretability, trustworthiness, privacy-awareness, bias mitigation and/or subgroup distribution shift analysis for the above cases

    editing, manipulation, image-to-image translation, style mixing, interpolation, inversion and semantic diffusion for all afore mentioned cases



Workshop Important Dates


Paper Submission Deadline:                                                             23:59:59 AoE (Anywhere on Earth) 29 July, 2024

Review decisions sent to authors; Notification of acceptance:       24 August, 2024

Camera ready version:                                                                       (08:00 AM UTC) 30 Aug, 2024




Submission Information

The paper format should adhere to the paper submission guidelines for main ECCV 2024 proceedings style. Please have a look at the Submission Polices here.

We welcome full long paper submissions (between 8 and 14 pages, excluding references or supplementary materials; a paper submission should be at least 8 pages long to be considered for publication). All submissions must be anonymous and conform to the ECCV 2024 standards for double-blind review.

All papers should be submitted using this CMT website.

All accepted manuscripts will be part of ECCV 2024 conference proceedings.

At the day of the workshop, oral presentations will be conducted by authors who are attending in-person.

The Competition



The Competition is a continuation of the ABAW Competition held this and last year in CVPR, the year before in ECCV and CVPR, the year before in ICCV and the year before in IEEE FG. It is split into the two below mentioned Challenges. Participants are invited to participate in at least one of these Challenges.



How to participate

In order to participate, teams will have to register. There is a maximum number of 8 participants in each team.



If you want to participate in the first Challenge (MTL Challenge) you should follow the below procedure for registration.

The lead researcher should send an email from their official address (no personal emails will be accepted) to d.kollias@qmul.ac.uk with:

i) subject "7th ABAW Competition: Team Registration";

ii) this EULA (if the team is composed of only academics) or this EULA (if the team has at least one member coming from the industry) filled in, signed and attached;

iii) the lead researcher's official academic/industrial website; the lead researcher cannot be a student (UG/PG/Ph.D.);

iv) the emails of each team member, each one in a separate line in the body of the email;

v) the team's name;

vi) the point of contact name and email address (which member of the team will be the main point of contact for future communications, data access etc)

As a reply, you will receive access to the dataset's cropped/cropped-aligned images and annotations and other important information.



If you want to participate in the second Challenge (CE Recognition) you should follow the below procedure for registration.

The lead researcher should send an email from their official address (no personal emails will be accepted) to d.kollias@qmul.ac.uk with:

i) subject "7th ABAW Competition: Team Registration";

ii) this EULA (if the team is composed of only academics) or this EULA (if the team has at least one member coming from the industry) filled in, signed and attached;

iii) the lead researcher's official academic/industrial website; the lead researcher cannot be a student (UG/PG/Ph.D.);

iv) the emails of each team member, each one in a separate line in the body of the email;

v) the team's name;

vi) the point of contact name and email address (which member of the team will be the main point of contact for future communications, data access etc)

As a reply, you will receive access to the dataset's videos and other important information.





General Information

At the end of the Challenges, each team will have to send us:

i) a link to a Github repository where their solution/source code will be stored,

ii) a link to an ArXiv paper with 2-8 pages describing their proposed methodology, data used and results.

Each team will also need to upload their test set predictions on an evaluation server (details will be circulated when the test set is released).

After that, the winner of each Challenge, along with a leaderboard, will be announced.

There will be one winner per Challenge. The top-3 performing teams of each Challenge will have to contribute paper(s) describing their approach, methodology and results to our Workshop; the accepted papers will be part of the ECCV 2024 proceedings. All other teams are also able to submit paper(s) describing their solutions and final results; the accepted papers will be part of the ECCV 2024 proceedings.

The Competition's white paper (describing the Competition, the data, the baselines and results) will be ready at a later stage and will be distributed to the participating teams.



General Rules

1) Participants can contribute to any of the 2 Challenges.

2) In order to take part in any Challenge, participants will have to register as described above.

3) Any face detector whether commercial or academic can be used in the challenge. The paper accompanying the challenge result submission should contain clear details of the detectors/libraries used.

4) The top performing teams will have to share their solution (code, model weights, executables) with the organisers upon completion of the challenge; in this way the organisers will check so as to prevent cheating or violation of rules.



Competition Important Dates


Call for participation announced, team registration begins, data available:           09 May, 2024

Team registration closed:                                                                                               22 June, 2024

Test set release:                                                                                                             12 July, 2024

Final submission deadline (Predictions, Code and ArXiv paper):                               18 July, 2024

Winners Announcement:                                                                                               24 July, 2024

Final Paper Submission Deadline:                                                                                 23:59:59 AoE (Anywhere on Earth) 29 July, 2024

Review decisions sent to authors; Notification of acceptance:                                   24 August, 2024

Camera ready version:                                                                                                   (08:00 AM UTC) 30 August, 2024

Multi-Task Learning (MTL) Challenge

Database

For this Challenge, s-Aff-Wild2 database will be used. s-Aff-Wild2 is a static version of Aff-Wild2 database; it contains selected frames from Aff-Wild2.
In total, around 221K images will be used that contain annotations in terms of valence-arousal; 6 basic expressions, plus the neutral state, plus the 'other' category (that contains affective states not included in the other categories); 12 action units, namely AU1, AU2, AU4, AU6, AU7, AU10, AU12, AU15, AU23, AU24, AU25, AU26.

Rules

The participants are allowed to use the provided s-Aff-Wild2 database and/or any publicly available or private databases; the participants are not allowed to use the (A/V) Aff-Wild2 database (images and annotations). Teams are allowed to use any -publicly or not- available pre-trained model (as long as it has not been pre-trained on Aff-Wild2). The pre-trained model can be pre-trained on any task (e.g., VA estimation, Expression Recognition, AU detection, Face Recognition). Any methodological solution will be accepted for this Challenge.

Performance Assessment

The performance measure (P) is the sum of: the mean Concordance Correlation Coefficient (CCC) of valence and arousal; the average F1 Score across all 8 expression categories; the average F1 Score across all 12 action units:

CCCarousal + CCCvalence
2
+
F1expr
8
+
F1aus
12

Baseline Results

The baseline network is a pre-trained VGGFACE (with fixed convolutional weights and with MixAugment data augmentation technique) and its performance on the validation set is:

P = 0.32

Compound Expression (CE) Recognition Challenge

Database

For this Challenge, a part of C-EXPR-DB database will be used (56 videos in total). C-EXPR-DB is audiovisual (A/V) in-the-wild database and in total consists of 400 videos of around 200K frames; each frame is annotated in terms of 12 compound expressions. For this Challenge, the following 7 compound expressions will be considered: Fearfully Surprised, Happily Surprised, Sadly Surprised, Disgustedly Surprised, Angrily Surprised, Sadly Fearful and Sadly Angry.

Goal of the Challenge and Rules

Participants will be provided with a part of C-EXPR-DB database (56 videos in total), which will be unannotated, and will be required to develop their methodologies (supervised/self-supervised, domain adaptation, zero-/few-shot learning etc) for recognising the 7 compound expressions in this unannotated part, in a per-frame basis.

Teams are allowed to use any -publicly or not- available pre-trained model and any -publicly or not- available database (that contains any annotations, e.g. VA, basic or compound expressions, AUs)

Performance Assessment

The performance measure (P) is the average F1 Score across all 7 categories:   ∑ F1/7

References

If you use the above data, you must cite all following papers (and the white paper that will be distributed at a later stage):

    D. Kollias, et. al.: "The 6th Affective Behavior Analysis in-the-wild (ABAW) Competition", 2024

    @article{kollias20246th, title={The 6th Affective Behavior Analysis in-the-wild (ABAW) Competition}, author={Kollias, Dimitrios and Tzirakis, Panagiotis and Cowen, Alan and Zafeiriou, Stefanos and Shao, Chunchang and Hu, Guanyu}, journal={arXiv preprint arXiv:2402.19344}, year={2024}}

    D. Kollias, et. al.: "ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges". IEEE CVPR, 2023

    @inproceedings{kollias2023abaw2, title={Abaw: Valence-arousal estimation, expression recognition, action unit detection \& emotional reaction intensity estimation challenges}, author={Kollias, Dimitrios and Tzirakis, Panagiotis and Baird, Alice and Cowen, Alan and Zafeiriou, Stefanos}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={5888--5897}, year={2023}}

    D. Kollias: "Multi-Label Compound Expression Recognition: C-EXPR Database & Network". IEEE CVPR, 2023

    @inproceedings{kollias2023multi, title={Multi-Label Compound Expression Recognition: C-EXPR Database \& Network}, author={Kollias, Dimitrios}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={5589--5598}, year={2023}}

    D. Kollias: "ABAW: Learning from Synthetic Data & Multi-Task Learning Challenges". ECCV, 2022

    @inproceedings{kollias2023abaw, title={ABAW: learning from synthetic data \& multi-task learning challenges}, author={Kollias, Dimitrios}, booktitle={European Conference on Computer Vision}, pages={157--172}, year={2023}, organization={Springer} }

    D. Kollias: "ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges". IEEE CVPR, 2022

    @inproceedings{kollias2022abaw, title={Abaw: Valence-arousal estimation, expression recognition, action unit detection \& multi-task learning challenges}, author={Kollias, Dimitrios}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={2328--2336}, year={2022} }

    D. Kollias, et. al.: "Analysing Affective Behavior in the second ABAW2 Competition". ICCV, 2021

    @inproceedings{kollias2021analysing, title={Analysing affective behavior in the second abaw2 competition}, author={Kollias, Dimitrios and Zafeiriou, Stefanos}, booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision}, pages={3652--3660}, year={2021}}

    D. Kollias,S. Zafeiriou: "Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework, 2021

    @article{kollias2021affect, title={Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework}, author={Kollias, Dimitrios and Zafeiriou, Stefanos}, journal={arXiv preprint arXiv:2103.15792}, year={2021}}

    D. Kollias, et. al.: "Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study", 2021

    @article{kollias2021distribution, title={Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study}, author={Kollias, Dimitrios and Sharmanska, Viktoriia and Zafeiriou, Stefanos}, journal={arXiv preprint arXiv:2105.03790}, year={2021} }

    D. Kollias, et. al.: "Analysing Affective Behavior in the First ABAW 2020 Competition". IEEE FG, 2020

    @inproceedings{kollias2020analysing, title={Analysing Affective Behavior in the First ABAW 2020 Competition}, author={Kollias, D and Schulc, A and Hajiyev, E and Zafeiriou, S}, booktitle={2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)(FG)}, pages={794--800}}

    D. Kollias, S. Zafeiriou: "Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace". BMVC, 2019

    @article{kollias2019expression, title={Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace}, author={Kollias, Dimitrios and Zafeiriou, Stefanos}, journal={arXiv preprint arXiv:1910.04855}, year={2019}}

    D. Kollias, et. al.: "Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond". International Journal of Computer Vision (IJCV), 2019

    @article{kollias2019deep, title={Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond}, author={Kollias, Dimitrios and Tzirakis, Panagiotis and Nicolaou, Mihalis A and Papaioannou, Athanasios and Zhao, Guoying and Schuller, Bj{\"o}rn and Kotsia, Irene and Zafeiriou, Stefanos}, journal={International Journal of Computer Vision}, pages={1--23}, year={2019}, publisher={Springer} }

    D. D. Kollias, et at.: "Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network", 2019

    @article{kollias2019face,title={Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network}, author={Kollias, Dimitrios and Sharmanska, Viktoriia and Zafeiriou, Stefanos}, journal={arXiv preprint arXiv:1910.11111}, year={2019}}

    S. Zafeiriou, et. al. "Aff-Wild: Valence and Arousal in-the-wild Challenge". IEEE CVPR, 2017

    @inproceedings{zafeiriou2017aff, title={Aff-wild: Valence and arousal ‘in-the-wild’challenge}, author={Zafeiriou, Stefanos and Kollias, Dimitrios and Nicolaou, Mihalis A and Papaioannou, Athanasios and Zhao, Guoying and Kotsia, Irene}, booktitle={Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on}, pages={1980--1987}, year={2017}, organization={IEEE} }