Multimodal Human-Agent Interfaces for Virtual Environments

Macquarie University

November 20, 2014

Flexible Conversation Management using a BDI Agent Approach

Lawrence Cavedon (RMIT)

• Real Thing / toy / conversational agent
• BDI Plans for conversation management
• Conversational behaviours including plans
• Goal/plan tree
• Question/answers forums (answers.com, askkids.com)
• Question/Answer pairs can be used for answering user questions or for furthering conversations
• 23,800 fragments mined from the web
• Pilot evaluation
• Integration with BDI approach (provides opportunistic filler)

Conversational Agents for Training in Motivational Interviewing

Richard Lebbrandt, Martin Luerssen, David Powers (Flinders University)

• HeadX (Head based system - ECA Technologies)
• Clevertar (Full-body animated character)
• Assistive/supportive applications of ECA technology (health, disabilities)
• Motivational interviewing (creating the conditions in which the client can formulate their own plans)
• MI Theory

Language as an experience for integrated discourse and situation representation structures to support social interaction learning in virtual worlds for adults with learning disabilities

Laurianne Sitbon and Samuel Brian (QUT)

• Virtual learning envivronment
• Users with learning disabilities
• Dialog based scenarios
• Conversation: natural (not pre-programmed) and situated (based on the physical environment)
• Representtion for situated dialogue
• Ability to deal with the environment
• Episodic memory (temporally and spatially related)
• Simulating perception
• Support reasoning about space and time

Reducing Cognitive Load to improve human-agent interfaces - with a focus on the use of gesture

Nadine Marcus

UNSW

• Gestures can lead to be better learning in instructional material?
• Cognitive load theory
• Interested in relations between Long Term Memory (LTM) and Working Memory (WM)
• Animations and static graphics (Ayres & Pass, 2007a, 2007b)
• Leahy and Sweller (2011)
• Animated instructions can lead to superior learning for a human movement task when compared to equiavelent static graphics
• This may be due to our inate ability for observational learning
• Human movement is biolgically primary knowledge (Geary, 2007, 2012)
• Animations are superior for learning motor tasks unless they become very long
• Gesture studies in Mathematics
• Gesturing: Paas and Sweller (2012)
• Experiments: Adults learning Mandarin, children learning Persian
• NICTA Study: explored trust in a text chat environment (measured mouse movement)
• Gestures can facilitate knowledge by reducing load (due to primary knowledge associated with gesturing)
• However need to ensure the gestures are not redundant

The Impact of a Humanoid Robot's Nonverbal Communication and lifelike bodily movement upon decision-making and trust during a cooperative human-robot task

Chris Stanton and Kate Stevens

MARCS Institute

University of Western Sydney

• For robots to work along side humans
• Aldebaran Nao - little humanoid robot
• Measure of trust - the shell game
• Variables: task difficulty, gaze, eye tracking, breathing
• Performance measures
  • How often participants asked the robot for help
  • how often participants trusted the robot's advice
  • accuracy
• Trust, Accuracyy, Response Time, Lifelike Movements
• Robot pressure
• Gender effects? (Women don't trust a "staring" robot)
• Integroup bias and anthropomorphic movement
• Proxemics (participants don't like the robot if they are placed in the computer group, participants don't like the robot if it doesn't move)
• Future work (age, gender, posture, more natural movement, adaptive behaviour, impact of score updates)
• Teamwork (human-robot pairs)

Multimodal Communication for Human-Agent Teamwork in a Virtual Environment

Nader Hanna

Macquarie University

• Human-IVA teamwork (IVA = Intelligent Virtual Agent)
• Shared Mental Model (SMM)
• Increasing interest in heterogenous teams that include humans as an IVAs
• Human team mate to communicate their intent and to amke and identify shared understanding.
• A number of researchers found that performance improves if there is a SMM
• Shared Mental Model - perception of or knowledge about a situation or process that is shared among the the team members through communication.
• Teams that perform well hold shared mental models
• CAST = Collaborative Agent Architecture for Simulating Teamwork (Yen et. al 2003)
• SMMall = Shared Mental Model for all. (Fan et al. 2008, 2011) realizes the Hidden Markov Model (HMM)
• How can agent tell another agent something?
  • Message Passing
  • Shared Memory, Blackboard
  • Teaching
• How can an IVA tell a human something?
  • Woggles that speaks with bubbles
  • STEVE = animated virtual characters in a training field (demonstrates to a student avatar)
  • Gandalf - a solar expert (1996)
  • Greta (TELECOM ParisTech)
  • Virtual Patient (Justin) - 2007
  • Virtual Professor (Florida State)
• Does multimodal communication impact on taskwork and teamwork SMM between humans and IVA?
• Do task work and teamwork SMMs impact on human-IVA team performance?
• What are other possible factors that impact human-IVA performance?
• Virtual Scenario 1 = human and a IVA should take turn to select a region to trap an animal for scientific research
• Virtual Scenario 2 = human and IVA shold agree on selecting a pair of complementary tools to overcome a sequence of obstacles

Breakout Session

• What is hard?
• What is not so important?
• What do you need/expect others to achieve/do?
• What tools/theories exist?

1. Aiding Humans
2. Virtual Environments
3. Verbal Communication
4. Authoring, scenario, knowledge acqusition, agent learning and adaptation
5. Tools, sharing, data, reuse

Intelligent Diabetes Lifestyle Coach (aka Sunny)

Rafael Calvo

School of Electrical and Information Engineering

The University of Sydney

• Diabetes in Australia, Treatment approaches, Sunny, Future Guy
• No an agents/robotics/NLG
• Future Fellowship on "Positive Computing"
• Diabetes Type 2
• Diabetes therapy involves behaviour change (exercise, diet, medication)
• The patient needs to be motivated in the long term
• Embodied Conversational Agents (ECA) can provide an interactive and intelligent environments to talk to people with diabetes
• Using computer agents can increase the level of user engagement
• ECA vs traditional health care models
  • Occasional appointments can't maintain the user engagement
  • Privacy and time constraints can limit interventions in clinical practice
  • ECA is available 24/7
• What is the impact of different design approaches on patient's motivation?
• What design features have a significant impact on treatment outcomes (e.g. engaging digital experience, social factors)?
• Are these the same factors that have an impact on overall wellbeing (e.g. autonomy)
• When we design things we sometimes remove features
• Resilience, competence, autonomy, connectedness
• Sunny, Siri's sister (they don't get along)
• Third party avatars
• Google calendar, Facebook and Fitbit connection
• So far, 125 rules have been implemented
• Twitter: @Rafael_Calvo, Positive Computing (Book)
• The Oxford Handbook of Affective Computing

Improving Healthcare through Relational Virtual Agents

Martin Luerssen and David Powers

Clevetar CTO

• Making a virtual human
• 3D rendering, character animation, machine vision, dialogue management
• The untapped power of a smile
• Humans are primed to seek out and respond to social cues
• Emotions are fundamental to the human experience
  • Affective Computing
  • Software agents that just make you feel good
• Making a difference: anthropormophic agents been shown to
  • attract attention... and other stuff
• Virtual agents for healthcare (expensive and human intensive)
• Many health gains can be made by changing human behaviour
• Anna Cares
• Home care focus (e.g. elderly people)
• Making a useful product
• Current version does not have speech recognition
• Share your care, Helping Hand
• Leverage social network and service provider interests to provide meaningful content
• Improve long-term engagement
• Architecture and Integration
• Next step: commercial scale trial with both coordinators and clients
  • Monitor service outcomes, commercial utility, user satisfaction
• More functionality / personality / production values
• Add some 'smarts'...
  • Speech recognition
  • Dialogue management

An Empathetic Agent for Adherence to Medical Treatment Advice

Deborah Richards

• How to improve adherence to treatment advice
• There are two types of adherence
• Caring communication style
• Improving Adherence (e.g. in e-Health)
• Relational Agents
• Shared goals, shared plans to achievev those goals
• Virtual Human Toolkit
• Meet Dr Evie (eVirtual agent for Incontience and Enuresis)
• https://comp.mq.edu.au/vworlds/DrEvie/caffeine/
• Her (movie - guys fall in love with his operating system)
• People with low literacy and education skills believe these characters
• A virtual tour with Omosa

Formalising Believability and Building Believable Virtual Agents

Tomas Trescak, University of Western Sydney, t.trescak@usw.edu.au

• Motivation - virtual environments, virtual reality
• Embodied, intelligent virtual agents
• Mimicking human behaviour = believable
• "Believability" is very loosely defined
• In terms of formalisation the concept of believability resembles similarity with intelligence - hard to define and formalise
• Construct formal model of perceived believability
• Must be perceived as real in the context of its environment
• Classic works of Walt Disney
• Believable characters vs agents
• Interactivty requires AI
• Contemporary AI: engaging life-like systems, reactivity, interactive...
• Definiitions of Believable Agents:
  • life like, whose actions make sense, who allows you to suspend disbelief
  • observing sentient beings
  • giving the illusion of being controlled by a human
• Believability Features (Loyall et. all 1999)
  • Personality
  • Emotions
  • Self-Motivation
  • Change
  • Social Relationships
  • Consistency of Expression
  • Illusion of Life
• Emotional state v emotions, role of the environment, verbal behaviour, non-verbal behaviour, appearance
• Believability Formalisation (Bogdanovych et al published in ACALCI 2015)
• Motivational based goal generation (BDI)
• Personality (OCEAN)
• Emotions (OCC)
• Liveness (several models)
  • BDI, dynamic planning, obstance avoidance...
• Social relationships, virtual institutions, consistency (norms + VI), change, awareness (environmental annotations + VI)
• Case Studies
  • Uruk 3000 B.C.
  • UWS Campus, Parramatta 1770 A.D. (Pre-contact)

Cultural Change in Virtual Environments

Erik Champion

• Cultural Agents in VES
• Challenge: Develop agents that can pass on information about a past or distant culture...
• Cultural Agents
• Adobe Atmosphere
• Shadows of the Colossus: areas of vulnerability
• Cultural agent
  • Recognises, adds to or transmits physically embedded and embodied aspects of culture
  • Culture is itself heritage-biased towards past
• Public VR
  • Moved to Unity
  • Egyptian Oracle
• Special Issue on Serious Games (CFP) due in January www.digra.org
• seriousexperience@gmail.com
• Entertainment Computing, Elsevier: Special Issue on Entertainment in Serious Games and Entertaining Serious Purposes
  • 31 January 2015 (Submission Deadline)
• http://www.digra.org/cfrp-special-issue-journal-of-entertainment-computing-on-serious-games/

Generating Games from Film Resources using Smartbody and Emotive Agents

Cat Kutay

• FAtiMA
  • Ana Paiva (Portuguese University)
• Motivation
  • Living in multi-cultural society
  • Sharing technical knowledge across cultures
  • Sharing knowledge practics as a way to improve learning techniques
• Approach to teaching in games
  • Share stories
  • Witness scenarios acted out
  • Role plas with AI
  • Integrate these into games
• Sharing stories
• SMART Body
  • Stacy Marsella
  • Institute of Creative Technologies (ICT)
• Virtual Human Toolkit
• Machinima Storyteller

Narrative AI: The Challenges of Building and Intelligent Storyteller

Malcolm Ryan

• Computer storytelling
• Computer graphics encodes the physical and artistic principles of light, perspective and material in order to makes tools to empower artists
• Narrative tools - build similar tools that encode the principles of story-telling... to empower writers to make works hitherto impossible
• Fabula, Szujet and Discourse
  • Fabula: the sequence of events in the story as the happended in time
  • Szujet: The selection and ordering of events for narration
  • Discourse: The rendering of events in a medium (writing, film, game)
• Focus primarily on fabula generation
• Action, character and plot
  • Action: the sequence of events must be plausible some fictional world
  • Character: the characters should act with believable desires, emotions and intentions
  • Plot: The sequence of events should have an overarching dramatic structure
• StoryLog
  • story generation tool based on AI methods drawn from planning and knowledge representation
  • ASP Planning
  • GOLOG agent control language
  • OCC model of emotion
  • BDI model for intelligent agents
• GOLOG non-deterministic programs that have multiple paths
• BDI in GOLOG
• Intention as a first class construct in the world
• Belief models (including deception)