Bidirectional Loop between:
Human Signals and Trustworthy LLMs:

Research Concept and Motivation

Study human-centered AI in healthcare and decision-support
from bidirectional sides: how humans align AI, and how AI augments humans.

Research Concept and Motivation: Human Signals Are Not Always Ground Truth

LLMs are aligned with humans and trained on human data, however :

Research Methods: Human Signals Across the LLM Lifecycle

Objective: Learn the right things from humans. Provide the right support to humans.

Work 1  -  Evaluation: LLM-as-a-Judge for Health Information

Research Question: whether humans and LLM-as-a-Judge can assess the information reliably?

Human trust

HIGHER under human label

Identical content is trusted more under a human-authored label.

LLM-as-a-Judge trust

HIGHER under human label

Judges reproduce the same label heuristic on the same task.

Judge decision uncertainty

LOWER under human label · rises under AI label

Logit uncertainty rises under AI labels; self-attention densifies on the label region.

Takeaways: LLM-as-a-Judge and Humans share similar processing patterns in both behavioral and mechanistic level.

[a] Sun et al. Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judge (ACL 2026) Core A*
[b] Sun et al. Understanding Trust Perception in AI-Generated Information with Behavioral and Physiological Sensing (IJHCS 2026) Top Tier JCR-1

Beyond Behaviors: Mechanistic Evaluation for LLMs and Humans

Human Gaze-Attention

Model Self-Attention

Takeaway: both humans and LLMs over-attend to heuristic cues, rather than objectively focus on content itself.

[a] Sun et al. Label Effects (ACL 2026) Core A* — behavioral results, mechanistic analysis across LLM families

Work 2  -  Alignment: How LLMs Learn the “Bad” Human Signals

Research Question: whether LLMs carry on the “bad signals” from human annotated preferences?

Takeaways: LLMs can learn the bias from the explicitly annotated human preferences.
Pre-train (next-token prediction) has the least effect, SFT comes the second, and DPO can install the bias significantly.

Work 3  -  Alignment: How Supervise Models with Domain Expertise, Not Simply Imitate

Research Question: if human signals transmit bias, what should models actually learn?
— Selected domain-specific expert strategies, not raw human imitation.

Takeaways: stepwise LLM reasoning and generation preserves structured domain expertise while suppressing risky imitation.

[c] Sun et al. Script-Strategy Aligned Generation (ACM CSCW 2025) Core A
[d] Sun et al. Eliciting MI Skill Codes in Psychotherapy with LLMs (COLING 2024) Core B
[e] Sun et al. Rethinking the Alignment of Psychotherapy Dialogue Generation (COLING 2025) Core B

Work 4  -  Adaptation: Reliable LLM’s Outputs Are Not Enough

Research Question: How LLMs can adapt to humans during the real-time interaction?

Adaptive generation and UIs based on sensed human states

Sense

Predict

Adapt

Takeaways: sensing human signals via eye-tracking and ECG/EDA can be used to model human latent states (cognitive load, trust, confidence, etc.), enabling adaptive LLM-powered support toward the user’s real-time states.

[f] Sun et al. Seeing the Reasoning (ACM CHI 2026) Core A*
[g] Sun et al. Eyes Can’t Always Tell (ACM UMAP 2026) Core A
[h] Sun et al. Interface Matters: Exploring Trust Perception in Health Information from Large Language Models via Text, Speech, and Embodiment (ACM CSCW 2024) Core A

Vision: Toward Human-Grounded, Reliable and Trustworthy LLM-powered AI Systems

Learn the right things from humans. Provide the right support to humans.

Thank you so much
for your time and patience!