MECO: A Multimodal Dataset for Emotion and Cognitive Understanding in Older Adults

Hongbin Chen1 Jie Li1 Wei Wang1 Siyang Song2 Xiao Gu3 Jianqing Li1* Wentao Xiang1*
1 Nanjing Medical University 2 University of Exeter 3 University of Oxford
*Corresponding Author
arXiv Appendix Code Dataset
Teaser Image

Overview of MECO dataset. (a) Emotion-inducing video stimuli for older adults. (b) Data acquisition protocol, encompassing cognitive assessment, synchronized multimodal recording, and self-assessed annotations. (c) Extracted multimodal signals and corresponding feature representations. (d) Downstream tasks for emotion and cognitive prediction. (e) Demographic characteristics of the study subjects, highlighting group-specific differences.

Abstract

While affective computing has advanced considerably, multimodal emotion prediction in aging populations remains underexplored, largely due to the scarcity of dedicated datasets. Existing multimodal benchmarks predominantly target young, cognitively healthy subjects, neglecting the influence of cognitive decline on emotional expression and physiological responses. To bridge this gap, we present MECO, a Multimodal dataset for Emotion and Cognitive understanding in Older adults. MECO includes 42 participants and provides approximately 38 hours of multimodal signals, yielding 30,592 synchronized samples. To maximize ecological validity, data collection followed standardized protocols within community-based settings. The modalities cover video, audio, electroencephalography (EEG), and electrocardiography (ECG). In addition, the dataset offers comprehensive annotations of emotional and cognitive states, including self-assessed valence, arousal, six basic emotions, and Mini-Mental State Examination cognitive scores. We further establish baseline benchmarks for both emotion and cognitive prediction. MECO serves as a foundational resource for multimodal modeling of affect and cognition in aging populations, facilitating downstream applications such as personalized emotion recognition and early detection of mild cognitive impairment (MCI) in real-world settings.

42

Elderly Participants

38h

Total Duration

30,592

Sync Samples

2

Label Types
Emotion & Cognition

7

Target Tasks
5 Emotion + 2 Cognition

MODALITIES

Video

Non-intrusive

Audio

High-fidelity

EEG

2-Channel Wearable

ECG

1-Channel Sensor

BASELINE TASKS

MECO supports 7 distinct downstream baseline tasks spanning emotion and cognition.

Emotion Domain (5 Tasks)

Objective → Subjective Coarse-grained → Fine-grained

Note: Task T1 evaluates objective stimulus-induced emotion. Tasks T2–T5 evaluate subjective emotional experiences assessed via the Self-Assessment Manikin (SAM), transitioning from coarse-grained sentiment classes to fine-grained emotion categories and continuous dimensional spaces.

Task T1 (SR): Stimulus-induced Emotion Recognition

  • Target: Predict objective stimulus-induced emotions based on intended stimuli.
  • Output: Discrete labels {0: 'angry', 1: 'boredom', 2: 'happy', 3: 'neutral', 4: 'sadness', 5: 'tension'}.
  • Metric: Unweighted Average Recall (UAR), Weighted Average Recall (WAR).

Task T2 (SA): 3-class Sentiment Analysis

  • Target: Predict coarse-grained subjective sentiment.
  • Output: Discrete labels {0: 'Negative', 1: 'Neutral', 2: 'Positive'}.
  • Metric: UAR, WAR.

Task T3 (ER): 5-class Emotion Recognition

  • Target: Predict fine-grained subjective emotion categories.
  • Output: Discrete labels {0: 'Angry', 1: 'Sadness', 2: 'Tension', 3: 'Neutral', 4: 'Happiness'}.
  • Metric: UAR, WAR.

Task T4 (VR): Valence Regression

  • Target: Predict continuous subjective valence dimensional scores.
  • Output: Continuous values in [1, 9] (spanning from negative to positive).
  • Metric: Mean Absolute Error (MAE), Concordance Correlation Coefficient (CCC).

Task T5 (AR): Arousal Regression

  • Target: Predict continuous subjective arousal dimensional scores.
  • Output: Continuous values in [1, 9] (spanning from calm to excited).
  • Metric: MAE, CCC.

Cognition Domain (2 Tasks)

Coarse Screening → Fine-grained Scoring

Note: Cognition tasks progress from coarse-grained binary screening for cognitive impairment (T6) to fine-grained regression of specific cognitive scores (T7).

Task T6 (CR): Binary Cognitive Impairment Screening

  • Target: Predict coarse-grained cognitive state for Mild Cognitive Impairment (MCI) screening.
  • Output: Binary labels {0: 'HC', 1: 'MCI'}.
  • Metric: Accuracy (ACC), Macro-F1.

Task T7 (MR): MMSE Score Regression

  • Target: Predict fine-grained Mini-Mental State Examination (MMSE) scores.
  • Output: Values in [0, 30] (evaluated as a continuous regression task, though underlying target values are inherently discrete integers).
  • Metric: MAE, CCC.

EVALUATION PROTOCOLS

Personalization

Subject-Dependent

EVALUATION SPECTRUM COVERAGE

Generalization

Subject-Independent

Subject-Dependent (SD)

Personalized Modeling • Emotion Tasks Only

Designed for personalized applications (e.g., in-home elderly care systems). To preserve the temporal dynamics of elicited emotional responses, a Chronological Split is applied within each trial.

Chronological Split per Trial:

Train (First 60%)
Val (20%)
Test (20%)

Subject-Independent (SI)

Generalization • Emotion & Cognition Tasks

Designed to evaluate model generalization across unseen participants. This is strictly required for cognition screening (MCI vs. HC) to ensure valid real-world applicability.

Subject-wise 5-Fold Cross-Validation:

Train (~80%) Test (~20%)

POTENTIAL APPLICATIONS

Unlocking new frontiers in scientific discovery and clinical deployment.

Emotion-Cognition Synergy

Emotion-Cognition Synergy

Uncover the behavioral and neural links between affective states and cognitive decline.

Multimodal Foundation Models

Foundation Models

Enable self-supervised learning for large-scale affective models tuned for the elderly.

Personalized In-Home Monitoring

In-Home Monitoring

Design unobtrusive, subject-dependent systems for continuous emotional well-being tracking.

Community MCI Screening

MCI Screening

Deploy scalable, subject-independent tools for early diagnosis and intervention.

DATA ACCESS

Get started with the MECO dataset for your research.

Ethics & Privacy Statement: In compliance with the informed consent signed by the elderly participants and IRB regulations, raw data will NOT be publicly released to protect participant privacy. We provide pre-processed and aligned multimodal features for use in all downstream tasks.
Step 1

Download Encrypted Features

The extracted multimodal feature files are hosted on Baidu Netdisk in an encrypted archive. Please download the files first.

Baidu Netdisk (Pan) Extraction Code: b3yj
Step 2

Request Decryption Key

To obtain the password to unzip the dataset, please send a formal request from your institutional email to hongbinchen@stu.njmu.edu.cn. Please use the following template: 

Subject: Data Access Request for MECO Dataset

Dear MECO Dataset Team,

I am writing to request the decryption password for the MECO dataset. I agree to use the data strictly for academic research purposes and will not distribute it to any third parties without permission.

- Full Name: [Your Name]
- Institution / Affiliation: [Your University / Research Center]
- Position / Title: [e.g., Master Student, PhD Student, Professor]
- Intended Purpose of Use: [Briefly describe your research project, e.g., multimodal emotion recognition]

I confirm that I have read the ethics and privacy statement, and I commit to citing your paper if this dataset is used in my publications.

Best regards,
[Your Name]

* Applications will typically be reviewed and processed within 1-2 business days.

CITATION

If you find the MECO useful for your research, please cite our work: 

@article{chen_meco2026,
      title = {MECO: A Multimodal Dataset for Emotion and Cognitive Understanding in Older Adults},
      author = {Chen,  Hongbin and Li,  Jie and Wang,  Wei and Song,  Siyang and Gu,  Xiao and Li,  Jianqing and Xiang,  Wentao},
      journal = {arXiv preprint arXiv:2604.03050},
      doi = {10.48550/ARXIV.2604.03050},
      url = {https://arxiv.org/abs/2604.03050},
      year = {2026}
}