7th ABAW

Submission Site

7th Workshop and Competition on


Affective Behavior Analysis in-the-wild (ABAW)

in conjunction with the European Conference on Computer Vision (ECCV), 2024

14:00 - 18:00 CEST, 30 September 2024, Suite 5, MiCo Milano, Milan, Italy

About ABAW

The diversity of human behavior, coupled with the richness of multi-modal data emerging from its analysis, and the multitude of applications requiring rapid advancements in this domain, ensure that our Workshop serves as a timely and relevant platform for discussion and dissemination.

The Workshop will showcase novel contributions on recent progress in the recognition, analysis, generation-synthesis, and modeling of face, body, gesture, speech, audio, text, and language, while incorporating the most advanced systems available for in-the-wild analysis (i.e., in unconstrained environments) and across various modalities.

The Competition features two Challenges: one focused on Multi-Task Learning of widely used affect representations, and the other on Compound Expression Recognition. Multiple teams have participated in these Challenges and will present their results and findings.

The ABAW Workshop and Competition is distinguished by its unique ability to foster cross-pollination among different disciplines, bringing together experts from academia, industry, and government, as well as researchers in mobile and ubiquitous computing, computer vision and pattern recognition, AI and ML/DL, multimedia, robotics, HCI, ambient intelligence, and psychology.

The ABAW Workshop and Competition builds on the legacy of its predecessors held at IEEE CVPR 2024, IEEE CVPR 2023, ECCV 2022, IEEE CVPR 2022, ICCV 2021, IEEE FG 2020 (a), IEEE FG 2020 (B) and IEEE CVPR 2017 Conferences.

Organisers

Dimitrios Kollias

Queen Mary University of London, UK d.kollias@qmul.ac.uk

Stefanos Zafeiriou

Imperial College London, UK s.zafeiriou@imperial.ac.uk

Irene Kotsia

Cogitat Ltd, UK irene@cogitat.io

Abhinav Dhall

Flinders University, Australia abhinav.dhall@flinders.edu.au

Shreya Ghosh

Curtin University, Australia shreya.ghosh@curtin.edu.au

Data Chairs

Chunchang Shao,     Queen Mary University of London, UK
Guanyu Hu,                 Queen Mary University of London, UK & Xi'an Jiaotong University, China

The Workshop



Call for Papers

This Workshop will solicit contributions on the recent progress of recognition, analysis, generation-synthesis and modelling of face, body, gesture, speech, audio, text and language while embracing the most advanced systems available for such in-the-wild (i.e., in unconstrained environments) analysis, and across modalities like face to voice. In parallel, this Workshop will solicit contributions towards building fair, explainable, trustworthy and privacy-aware models that perform well on all subgroups and improve in-the-wild generalisation.

Original high-quality contributions, in terms of databases, surveys, studies, foundation models, techniques and methodologies (either uni-modal or multi-modal; uni-task or multi-task ones) are solicited on -but are not limited to- the following topics:

    facial expression (basic, compound or other) or micro-expression analysis

    facial action unit detection

    valence-arousal estimation

    physiological-based (e.g.,EEG, EDA) affect analysis

    face recognition, detection or tracking

    body recognition, detection or tracking

    gesture recognition or detection

    pose estimation or tracking

    activity recognition or tracking

    lip reading and voice understanding

    face and body characterization (e.g., behavioral understanding)

    characteristic analysis (e.g., gait, age, gender, ethnicity recognition)

    group understanding via social cues (e.g., kinship, non-blood relationships, personality)

    video, action and event understanding

    digital human modeling

    deepfake generation, detection and temporal deepfake localization

    characteristic analysis (e.g., gait, age, gender, ethnicity recognition)

    violence detection

    autonomous driving

    domain adaptation, domain generalisation, few- or zero-shot learning for the above cases

    fairness, explainability, interpretability, trustworthiness, privacy-awareness, bias mitigation and/or subgroup distribution shift analysis for the above cases

    editing, manipulation, image-to-image translation, style mixing, interpolation, inversion and semantic diffusion for all afore mentioned cases



Workshop Important Dates

(Updated)


Paper Submission Deadline:                                                             23:59:59 AoE (Anywhere on Earth) 31 July, 2024

Review decisions sent to authors; Notification of acceptance:       25 August, 2024

Camera ready version:                                                                       (08:00 AM UTC) 30 Aug, 2024




Submission Information

The paper format should adhere to the paper submission guidelines for main ECCV 2024 proceedings style. Please have a look at the Submission Polices here.

We welcome full long paper submissions (between 8 and 14 pages, excluding references or supplementary materials; a paper submission should be at least 8 pages long to be considered for publication). All submissions must be anonymous and conform to the ECCV 2024 standards for double-blind review.

All papers should be submitted using this CMT website.

All accepted manuscripts will be part of ECCV 2024 conference proceedings.

At the day of the workshop, oral presentations will be conducted by authors who are attending in-person.

The Workshop's Agenda



Keynote Speaker 1: Eric Granger



Biography

Eric Granger is the FRQS co-Chair in Artificial Intelligence and Digital Health for Health Behaviour Change, ETS Industrial Research co-Chair on Embedded Neural Networks for Intelligent Connected Buildings (Distech Inc.), and a Professor in the Department of Systems Engineering at Montreal’s École de technologie supérieure. He is also director of the LIVIA, a research laboratory focused on computer vision and artificial intelligence. His research expertise includes machine learning, pattern recognition, and computer vision, with applications in affective computing, biometrics, face recognition, medical imaging, and video analytics/surveillance. His contributions on the development of deep learning (DL) models for visual recognition has led to several collaborations with governmental and industrial partners like CBSA, Nuvoola, Ericsson, and Genetec Inc. Dr. Granger is an associate editor for Elsevier Pattern Recognition, Springer Nature Computer Science, and the EURASIP Journal on Image and Video Processing.



Keynote Speaker 2: Mohammad Soleymani



Biography

Mohammad Soleymani is a research associate professor with the USC Institute for Creative Technologies. He received his PhD in computer science from the University of Geneva in 2011. From 2012 to 2014, he was a Marie Curie fellow at Imperial College London. Prior to joining ICT, he was a research scientist at the Swiss Center for Affective Sciences, University of Geneva. His main line of research involves machine learning for emotion recognition and behavior understanding. He is a recipient of the Swiss National Science Foundation Ambizione grant and the EU Marie Curie fellowship. He has served on multiple conference organization committees and editorial roles, most notably as associate editor for the IEEE Transactions on Affective Computing (2015-2021), general chair for ICMI 2024 and ACII 2021 and technical program chair for ACM ICMI 2018 and ACII 2017. He was the president of the Association for the Advancement of Affective Computing (AAAC) (2019-2021).



Keynote Speaker 3: Shaun Canavan



Biography

Shaun Canavan received his PhD in Computer Science from Binghamton University. He is an Associate Professor in the Computer Science and Engineering Department at the University of South Florida. His research focuses broadly on Affective Computing and Human-Computer Interaction. He has over 60 publications in top conferences and journals such as CVPR, ICPR, ICMI, ACII, FG, Pattern Recognition Letters, IEEE Transactions on Visualization and Computer Graphics, and IEEE Transactions on Affective Computing. He was the program chair for Face and Gesture 2024, and the publications chair for Affective Computing and Intelligent Interaction, 2023. He is the current general chair for FG 2025, and tutorial chair for ACII 2024. He is an associate editor for Pattern Recognition and Pattern Recognition Letters. His work is and has been supported by the DOD, NSF, and Amazon. He is a senior member of the IEEE and a member of the AAAC.



The Competition



The Competition is a continuation of the ABAW Competition held this and last year in CVPR, the year before in ECCV and CVPR, the year before in ICCV and the year before in IEEE FG. It is split into the two below mentioned Challenges. Participants are invited to participate in at least one of these Challenges.



How to participate

In order to participate, teams will have to register. There is a maximum number of 8 participants in each team.



If you want to participate in the first Challenge (MTL Challenge) you should follow the below procedure for registration.

The lead researcher should send an email from their official address (no personal emails will be accepted) to d.kollias@qmul.ac.uk with:

i) subject "7th ABAW Competition: Team Registration";

ii) this EULA (if the team is composed of only academics) or this EULA (if the team has at least one member coming from the industry) filled in, signed and attached;

iii) the lead researcher's official academic/industrial website; the lead researcher cannot be a student (UG/PG/Ph.D.);

iv) the emails of each team member, each one in a separate line in the body of the email;

v) the team's name;

vi) the point of contact name and email address (which member of the team will be the main point of contact for future communications, data access etc)

As a reply, you will receive access to the dataset's cropped/cropped-aligned images and annotations and other important information.



If you want to participate in the second Challenge (CE Recognition) you should follow the below procedure for registration.

The lead researcher should send an email from their official address (no personal emails will be accepted) to d.kollias@qmul.ac.uk with:

i) subject "7th ABAW Competition: Team Registration";

ii) this EULA (if the team is composed of only academics) or this EULA (if the team has at least one member coming from the industry) filled in, signed and attached;

iii) the lead researcher's official academic/industrial website; the lead researcher cannot be a student (UG/PG/Ph.D.);

iv) the emails of each team member, each one in a separate line in the body of the email;

v) the team's name;

vi) the point of contact name and email address (which member of the team will be the main point of contact for future communications, data access etc)

As a reply, you will receive access to the dataset's videos and other important information.





General Information

At the end of the Challenges, each team will have to send us:

i) a link to a Github repository where their solution/source code will be stored,

ii) a link to an ArXiv paper with 2-8 pages describing their proposed methodology, data used and results.

Each team will also need to upload their test set predictions on an evaluation server (details will be circulated when the test set is released).

After that, the winner of each Challenge, along with a leaderboard, will be announced.

There will be one winner per Challenge. The top-3 performing teams of each Challenge will have to contribute paper(s) describing their approach, methodology and results to our Workshop; the accepted papers will be part of the ECCV 2024 proceedings. All other teams are also able to submit paper(s) describing their solutions and final results; the accepted papers will be part of the ECCV 2024 proceedings.

The Competition's white paper (describing the Competition, the data, the baselines and results) will be ready at a later stage and will be distributed to the participating teams.



General Rules

1) Participants can contribute to any of the 2 Challenges.

2) In order to take part in any Challenge, participants will have to register as described above.

3) Any face detector whether commercial or academic can be used in the challenge. The paper accompanying the challenge result submission should contain clear details of the detectors/libraries used.

4) The top performing teams will have to share their solution (code, model weights, executables) with the organisers upon completion of the challenge; in this way the organisers will check so as to prevent cheating or violation of rules.



Competition Important Dates

(Updated)


Call for participation announced, team registration begins, data available:           09 May, 2024

Team registration closed:                                                                                               22 June, 2024

Test set release:                                                                                                             12 July, 2024

Final submission deadline (Predictions, Code and ArXiv paper):                               18 July, 2024

Winners Announcement:                                                                                               26 July, 2024

Final Paper Submission Deadline:                                                                                 23:59:59 AoE (Anywhere on Earth) 31 July, 2024

Review decisions sent to authors; Notification of acceptance:                                   25 August, 2024

Camera ready version:                                                                                                   (08:00 AM UTC) 30 August, 2024

Multi-Task Learning (MTL) Challenge

Database

For this Challenge, s-Aff-Wild2 database will be used. s-Aff-Wild2 is a static version of Aff-Wild2 database; it contains selected frames from Aff-Wild2.
In total, around 221K images will be used that contain annotations in terms of valence-arousal; 6 basic expressions, plus the neutral state, plus the 'other' category (that contains affective states not included in the other categories); 12 action units, namely AU1, AU2, AU4, AU6, AU7, AU10, AU12, AU15, AU23, AU24, AU25, AU26.

Rules

The participants are allowed to use the provided s-Aff-Wild2 database and/or any publicly available or private databases; the participants are not allowed to use the (A/V) Aff-Wild2 database (images and annotations). Teams are allowed to use any -publicly or not- available pre-trained model (as long as it has not been pre-trained on Aff-Wild2). The pre-trained model can be pre-trained on any task (e.g., VA estimation, Expression Recognition, AU detection, Face Recognition). Any methodological solution will be accepted for this Challenge.

Performance Assessment

The performance measure (P) is the sum of: the mean Concordance Correlation Coefficient (CCC) of valence and arousal; the average F1 Score across all 8 expression categories; the average F1 Score across all 12 action units:

CCCarousal + CCCvalence
2
 +
F1expr
8
 +
F1aus
12

Baseline Results

The baseline network is a pre-trained VGGFACE (with fixed convolutional weights and with MixAugment data augmentation technique) and its performance on the validation set is:

P = 0.32

Compound Expression (CE) Recognition Challenge

Database

For this Challenge, a part of C-EXPR-DB database will be used (56 videos in total). C-EXPR-DB is audiovisual (A/V) in-the-wild database and in total consists of 400 videos of around 200K frames; each frame is annotated in terms of 12 compound expressions. For this Challenge, the following 7 compound expressions will be considered: Fearfully Surprised, Happily Surprised, Sadly Surprised, Disgustedly Surprised, Angrily Surprised, Sadly Fearful and Sadly Angry.

Goal of the Challenge and Rules

Participants will be provided with a part of C-EXPR-DB database (56 videos in total), which will be unannotated, and will be required to develop their methodologies (supervised/self-supervised, domain adaptation, zero-/few-shot learning etc) for recognising the 7 compound expressions in this unannotated part, in a per-frame basis.

Teams are allowed to use any -publicly or not- available pre-trained model and any -publicly or not- available database (that contains any annotations, e.g. VA, basic or compound expressions, AUs)

Performance Assessment

The performance measure (P) is the average F1 Score across all 7 categories:   ∑ F1/7

Leaderboards


Multi-Task Learning Challenge:


In total, 45 Teams participated in the Multi-Task Learning Challenge. 25 Teams submitted their results. 9 Teams made invalid (incomplete) submissions, whilst surpassing the baseline. 10 Teams scored lower than the baseline. 6 Teams scored higher than the baseline and made valid submissions.

The winner of this Challenge is team Netease Fuxi AI Lab.
The runner-up is team HSEmotion.
In the third place is team HFUT-MAC1.

Compound Expression Recognition Challenge:


In total, 30 Teams participated in the Compound Expression Recognition Challenge. 10 Teams submitted their results. 5 Teams made invalid (incomplete) submissions. 5 Teams made valid submissions.

The winner of this Challenge is team Netease Fuxi AI Lab.
The runner-up is team HSEmotion.
In the third place is team ETS-LIVIA.



The leaderboards for all Challenges can be found below:

ECCV2024_ABAW_Leaderboard

Congratulations to all teams, winning and non-winning ones! Thank you very much for participating in our Competition.
All teams are invited to submit their methodologies-papers (please see Submission Information section above). All accepted papers will be part of the ECCV 2024 proceedings.
We are looking forward to receiving your submissions!

References

If you use the above data, you must cite all following papers (and the white paper that will be distributed at a later stage):

    D. Kollias, et. al.: "7th abaw competition: Multi-task learning and compound expression recognition", 2024

    @article{kollias20247th,title={7th abaw competition: Multi-task learning and compound expression recognition},author={Kollias, Dimitrios and Zafeiriou, Stefanos and Kotsia, Irene and Dhall, Abhinav and Ghosh, Shreya and Shao, Chunchang and Hu, Guanyu},journal={arXiv preprint arXiv:2407.03835},year={2024}}

    D. Kollias, et. al.: "The 6th Affective Behavior Analysis in-the-wild (ABAW) Competition". IEEE CVPR, 2024

    @inproceedings{kollias20246th,title={The 6th affective behavior analysis in-the-wild (abaw) competition},author={Kollias, Dimitrios and Tzirakis, Panagiotis and Cowen, Alan and Zafeiriou, Stefanos and Kotsia, Irene and Baird, Alice and Gagne, Chris and Shao, Chunchang and Hu, Guanyu},booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},pages={4587--4598},year={2024}}

    D. Kollias, et. al.: "ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Emotional Reaction Intensity Estimation Challenges". IEEE CVPR, 2023

    @inproceedings{kollias2023abaw2, title={Abaw: Valence-arousal estimation, expression recognition, action unit detection \& emotional reaction intensity estimation challenges}, author={Kollias, Dimitrios and Tzirakis, Panagiotis and Baird, Alice and Cowen, Alan and Zafeiriou, Stefanos}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={5888--5897}, year={2023}}

    D. Kollias: "Multi-Label Compound Expression Recognition: C-EXPR Database & Network". IEEE CVPR, 2023

    @inproceedings{kollias2023multi, title={Multi-Label Compound Expression Recognition: C-EXPR Database \& Network}, author={Kollias, Dimitrios}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={5589--5598}, year={2023}}

    D. Kollias: "ABAW: Learning from Synthetic Data & Multi-Task Learning Challenges". ECCV, 2022

    @inproceedings{kollias2023abaw, title={ABAW: learning from synthetic data \& multi-task learning challenges}, author={Kollias, Dimitrios}, booktitle={European Conference on Computer Vision}, pages={157--172}, year={2023}, organization={Springer} }

    D. Kollias: "ABAW: Valence-Arousal Estimation, Expression Recognition, Action Unit Detection & Multi-Task Learning Challenges". IEEE CVPR, 2022

    @inproceedings{kollias2022abaw, title={Abaw: Valence-arousal estimation, expression recognition, action unit detection \& multi-task learning challenges}, author={Kollias, Dimitrios}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={2328--2336}, year={2022} }

    D. Kollias, et. al.: "Analysing Affective Behavior in the second ABAW2 Competition". ICCV, 2021

    @inproceedings{kollias2021analysing, title={Analysing affective behavior in the second abaw2 competition}, author={Kollias, Dimitrios and Zafeiriou, Stefanos}, booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision}, pages={3652--3660}, year={2021}}

    D. Kollias,S. Zafeiriou: "Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework, 2021

    @article{kollias2021affect, title={Affect Analysis in-the-wild: Valence-Arousal, Expressions, Action Units and a Unified Framework}, author={Kollias, Dimitrios and Zafeiriou, Stefanos}, journal={arXiv preprint arXiv:2103.15792}, year={2021}}

    D. Kollias, et. al.: "Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study", 2021

    @article{kollias2021distribution, title={Distribution Matching for Heterogeneous Multi-Task Learning: a Large-scale Face Study}, author={Kollias, Dimitrios and Sharmanska, Viktoriia and Zafeiriou, Stefanos}, journal={arXiv preprint arXiv:2105.03790}, year={2021} }

    D. Kollias, et. al.: "Analysing Affective Behavior in the First ABAW 2020 Competition". IEEE FG, 2020

    @inproceedings{kollias2020analysing, title={Analysing Affective Behavior in the First ABAW 2020 Competition}, author={Kollias, D and Schulc, A and Hajiyev, E and Zafeiriou, S}, booktitle={2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)(FG)}, pages={794--800}}

    D. Kollias, S. Zafeiriou: "Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace". BMVC, 2019

    @article{kollias2019expression, title={Expression, Affect, Action Unit Recognition: Aff-Wild2, Multi-Task Learning and ArcFace}, author={Kollias, Dimitrios and Zafeiriou, Stefanos}, journal={arXiv preprint arXiv:1910.04855}, year={2019}}

    D. Kollias, et. al.: "Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond". International Journal of Computer Vision (IJCV), 2019

    @article{kollias2019deep, title={Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond}, author={Kollias, Dimitrios and Tzirakis, Panagiotis and Nicolaou, Mihalis A and Papaioannou, Athanasios and Zhao, Guoying and Schuller, Bj{\"o}rn and Kotsia, Irene and Zafeiriou, Stefanos}, journal={International Journal of Computer Vision}, pages={1--23}, year={2019}, publisher={Springer} }

    D. D. Kollias, et at.: "Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network", 2019

    @article{kollias2019face,title={Face Behavior a la carte: Expressions, Affect and Action Units in a Single Network}, author={Kollias, Dimitrios and Sharmanska, Viktoriia and Zafeiriou, Stefanos}, journal={arXiv preprint arXiv:1910.11111}, year={2019}}

    S. Zafeiriou, et. al. "Aff-Wild: Valence and Arousal in-the-wild Challenge". IEEE CVPR, 2017

    @inproceedings{zafeiriou2017aff, title={Aff-wild: Valence and arousal ‘in-the-wild’challenge}, author={Zafeiriou, Stefanos and Kollias, Dimitrios and Nicolaou, Mihalis A and Papaioannou, Athanasios and Zhao, Guoying and Kotsia, Irene}, booktitle={Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on}, pages={1980--1987}, year={2017}, organization={IEEE} }