EROS

Personalized Emotional Intelligence in Generative AI through Symbolic Affective Reasoning

Qing Lin, Mengmi Zhang*
*Corresponding author

Paper | Poster

Abstract

Emotional intelligence enables humans to recognize emotions in images, understand their causes, reason about interventions, and generate visual modifications to achieve desired affective states. While current AI models can synthesize realistic images or perform limited content reasoning, they lack systematic emotional understanding, control, and personalization. Here, we introduce Emotion-augmented geneRatiOn System (EROS), a personalized hybrid AI that combines symbolic reasoning with deep learning. By leveraging large-scale image–emotion datasets and human psychophysics experiments, EROS derives generalizable affective rules, identifies emotion-relevant regions, and predicts context-aware modifications that preserve scene semantics. It further enables inference-time personalization through an expandable memory bank that yields interpretable emotional profiles without fine-tuning, allowing rapid and precise modulation of human affect. We demonstrate that EROS outperforms state-of-the-art large multimodal models in eliciting faster and stronger targeted emotional responses in individuals. Beyond affective computing, this framework offers a blueprint for AI systems that understand, reason about, and augment complex human cognitive states, with applications in mental health, adaptive media, and human–computer interaction.

Overview

Emotion-augmented geneRatiOn System (EROS) is an affective image editing framework that edits an input image toward a target emotional valence while preserving semantic and structural consistency.

EROS contains five modules:

1. Recognition: predicts whether an image conveys positive or negative valence.
2. Reasoning: localizes emotion-relevant regions using Grad-CAM and SAM.
3. Control: retrieves affective editing motifs from EmoTree.
4. Generation: synthesizes edited images with localized diffusion-based inpainting.
5. Personalization: adapts motif selection using user feedback through EmoMem.

This repository provides two running modes:

1. Offline EROS-NoMem
   A 5-loop non-personalized version.

2. Online Full EROS
   An interactive personalized version with user feedback.

Quick Start

Environment Setup

conda create -n EROS python=3.10
conda activate EROS 

pip install torch torchvision torchaudio
pip install git+https://github.com/huggingface/diffusers.git
pip install transformers==4.49
pip install accelerate
pip install git+https://github.com/facebookresearch/segment-anything.git
pip install grad-cam
pip install numpy==1.24.3
pip install spacy
python -m spacy download en_core_web_sm
pip install controlnet_aux
pip install 'git+https://github.com/salesforce/LAVIS.git'

pip install openai
pip install gradio==4.44.1
pip install gradio-modal==0.0.4
pip install pydantic==2.10.4
pip install pydantic-core==2.27.2

Download Required Assets

Before running EROS, please download all required assets and place them in the corresponding directories.

1. EmoTree

Download our EmoTree and place it under:

EmoTree/
└── EmoTree.json

2. Emotion Predictor

Download our pretrained emotion predictor and place it under:

Emo_Predictor_model/
└── emo_preditor.pt

3. Segment Anything Model (SAM)

Download the SAM ViT-H checkpoint from link and place it under:

sam_model/
└── sam_vit_h_4b8939.pth

Online Personalized EROS (Full Version)

Launch the interactive demo:

python app_online_demo.py

The interface allows users to:

• Upload an image
• Select a target valence
• Provide feedback ("Yes" / "No")
• Build personalized EmoMem profiles online

Offline EROS-NoMem

Run the non-personalized 5-loop version:

python EROS_noMem_top5loop.py

Input images should be placed in:

src_imgs/

Results will be saved in:

results/EROS_5loop/

EmoTree

EmoTree is a hierarchical affective knowledge structure used by EROS to retrieve interpretable editing motifs.

Use Our Pre-built EmoTree

You can directly use our pre-built EmoTree.

• Download our EmoTree and put it in EmoTree/.

Construct Your Own EmoTree

If you would like to build EmoTree from scratch using EmoSet, follow the steps below.

• Set Up the Environment
• Export your OpenAI API key:

export OPENAI_API_KEY="YOUR_API_KEY"

• Download EmoSet118k and put it in Dataset/. The dataset structure is:

Dataset/
└── EmoSet118k/
    ├── train.json
    └── image/
        ├── amusement/
        ├── awe/
        ├── contentment/
        ├── excitement/
        ├── anger/
        ├── disgust/
        ├── fear/
        └── sadness/

Step 1: Hierarchical Image Clustering

Run:

python build_tree_step1_cluster_images.py

This step extracts CLIP image features and performs hierarchical clustering.

The output will be stored in:

clusters/

Step 2: Source Concept Extraction

Run:

python build_tree_step2_extract_source_concepts.py

This step uses BLIP and GPT to extract source concepts, attributes, actions, and scenes from clustered images.

The output will be stored in:

json/src_concept/

Step 3: EmoTree Construction

Run:

python build_tree_step3_generate_emotree.py

This step generates target motifs and organizes them into the final EmoTree structure.

The output will be stored in:

json/emo_tree/emo_tree.json

EmoTree Structure

The final EmoTree consists of four levels:

Level 1:
Semantic Image Cluster

Level 2:
Source Concept

Level 3:
Target Valence

Level 4:
Target Motif
(Concept, Attribute, Action, Scene)

The resulting tree can be directly used for affective image editing and emotional motif retrieval.

EmoTree Analysis

Visualize EmoTree cluster:

Before visualizing EmoTree clusters, make sure the local cluster files have been generated. See Step 1: Hierarchical Image Clustering. The clusters should be stored in:

clusters/

Run:

python result_analysis/figure_tree_TSNE.py

Visualize EmoTree top-20 target concept:

python result_analysis/figure_tree_top20_concept.py

Visualize the global statistics of unique concepts, attributes, actions, and scenes:

python result_analysis/figure_tree_freq.py

Generate word clouds:

python result_analysis/figure_word_cloud.py

Human Experiments

Exp-EmoInteract: Personalized Interactive Affective Editing

Participants interact with EROS in real time.

An OpenAI API key is required because Exp-EmoInteract compares EROS against GPT-based affective editing during runtime.

Before launching the experiment interface, export your OpenAI API key:

export OPENAI_API_KEY="YOUR_API_KEY"

Download our test images and put them in src_imgs/.

Launch the experiment interface:

python app_inlab_exp.py

Experiment logs, participant responses, and personalization histories will be saved automatically.

Data from all participants in Exp-EmoInteract are available at the link.

For result analysis, run

python result_analysis/exp_EmoInteract_bootstrap.py

Download Pair-Out Dataset (Optional)

• If the pair-out dataset is available, Exp-EmoMotif trials will be shown during Exp-EmoInteract image generation.

This allows participants to complete motif-preference evaluations while waiting for new images to be generated.

• If the dataset is not provided, the experiment interface will still run normally. However, users must wait while images are generated online.

The pair-out dataset used in Exp-EmoMotif can be downloaded at this link.

After downloading:

unzip pair_out.zip

Exp-EmoPairwise: Comparison with State-of-the-Art Image Editing Methods

This experiment compares EROS against state-of-the-art image editing methods using pre-generated results.

We have conducted a series of Mechanical Turk experiments using the Psiturk platform, which requires JavaScript, HTML, and Python 3.8.

Experiment code can be download here.

You can follow the PsiTurk Setup Guide to configure the environment and run the experiment.

Data from all participants in Exp-EmoPairwise are available at the link.

For result analysis, run

python result_analysis/exp_EmoPairwise_bootstrap.py

Exp-EmoRegion: Emotion-Relevant Region Annotation

We collect human annotations of emotion-relevant image regions, which can be download here.

Experiment code can be download here.

You can follow the PsiTurk Setup Guide to configure the environment and run the experiment.

We evaluated the consistency between human-annotated emotion-relevant regions and those predicted by EROS.

First, extract the emotion-relevant regions predicted by EROS:

python result_analysis/exact_emo_region_EROS.py

Then, extract the human-annotated emotion-relevant regions:

python result_analysis/exact_emo_region_human.py

Finally, run the result analysis:

python result_analysis/exp_EmoRegion_bootstrap.py

Exp-EmoPrompt: Affective Rule Evaluation

This experiment evaluates whether prompts derived from EmoTree motifs convey the intended emotional valence.

Experiment code can be download here.

You can follow the PsiTurk Setup Guide to configure the environment and run the experiment.

Data from all participants in Exp-EmoPrompt are available at the link.

For result analysis, run

python result_analysis/exp_EmoPrompt_bootstrap.py

Exp-EmoRule: Evaluation on Generic and Personalized Affective Rules

This experiment evaluates both generic affective rules (EmoTree) and personalized affective profiles (EmoMem).

Experiment code can be download here.

You can follow the PsiTurk Setup Guide to configure the environment and run the experiment.

Data from all participants in Exp-EmoRule are available at the link.

For stage 1 result analysis, run

python result_analysis/exp_EmoRule_stage1_bootstrap.py

For stage 2-3 result analysis, run

python result_analysis/exp_EmoRule_stage23_bootstrap.py

Exp-EmoMotif: Cross-Scene Consistency in Motif Preference

This experiment evaluates whether motif preferences remain consistent across semantically related scenes.

The pair-out dataset used in Exp-EmoMotif can be downloaded at this link.

After downloading:

unzip pair_out.zip

To launch this experiment, run

python app_inlab_exp.py

Data from all participants in Exp-EmoMotif are available at the link.

For result analysis, run

python result_analysis/exp_EmoMotif_bootstrap.py

Evaluation

SSIM-C and L1-C

In affective image editing, models are expected to minimally modify images while achieving the target valence. To quantify both structural preservation and editing localization, we introduce two contrastive metrics:

• SSIM-C (Structural Similarity Contrast)
• L1-C (Pixel Difference Contrast)

A weight map w ∈ [0, 1] is constructed by aggregating binary masks from multiple human annotators (data collected in Exp-EmoRegion). The weight map represents the consensus of emotion-relevant regions intended for editing.

Weighted SSIM and L1 distances are computed separately for:

• Preserved regions, weighted by 1-w
• Edited regions, weighted by w

The contrastive metrics are defined as:

SSIM-C = SSIM_preserve / (SSIM_edit + ε)
L1-C = L1_preserve / (L1_edit + ε)

where ε is a small constant for numerical stability.

Interpretation:

• Higher SSIM-C indicates stronger structural preservation outside edited regions together with larger changes inside intended editing regions.
• Lower L1-C indicates smaller pixel-level deviations in preserved regions and larger modifications in targeted editing regions.

Compute SSIM-C and L1-C

Before running the script, prepare:

• Source images
• Edited images
• Human annotation masks

Modify the corresponding input paths in:

base_dir
src_img_pth
user_mask_folder

res_method_pth_dict
loop_method_pth_dict

Then run:

python result_analysis/metric_SSIMC_L1C.py

Output:

results_summary_bootstrap_ssimc_l1c.csv

Emotion Predictor Evaluation

The emotion predictor is used by EROS to estimate whether an image conveys positive or negative valence.

Before running the script, prepare source images under:

src_imgs/

Image filenames should begin with one of the eight EmoSet emotion labels:

amusement, awe, contentment, excitement, anger, disgust, fear, sadness

The pretrained checkpoint should be placed under:

Emo_Predictor_model/
└── emo_preditor.pt

To evaluate the pretrained emotion predictor, run:

python result_analysis/test_emo_predictor.py

The confusion matrix will be saved to:

plots/EmoPredictor/confusion_matrix.png

BibTeX

@article{lin2026EROS,
      title={Personalized Emotional Intelligence in Generative AI through Symbolic Affective Reasoning}, 
      author={Qing Lin and Mengmi Zhang},
      journal={arXiv preprint arXiv:2607.10678},
      year={2026}
}