ESSLLI 2022 Course on Multimodal Semantics for Affordances and Actions

We (James Pustejovsky and Nikhil Krishnaswamy) are pleased to be teaching a week-long summer course from August 8-12, 2022, at the European Summer School in Logic, Language, and Information in Galway, Ireland! Our course is entitled Multimodal Semantics for Affordances and Actions, and is based on our Best Paper-winning paper from HCII 2022, of the same title. The course is part of the Language and Computation track at ESSLLI. The slides, Zoom recordings, and relevant background readings for the five lectures are below:

Lectures

• Lecture 1: Multimodal Communication
  • Here is the link to the Zoom recording of Lecture 1
• Lecture 2: Modeling Human Object Interactions
  • Here is the link to the Zoom recording of Lecture 2
• Lecture 3: Modeling Multimodal Common Ground
  • Here is the link to the Zoom recording of Lecture 3
• Lecture 4: Communicating with Multimodal Common Ground
  • Here is the link to the Zoom recording of Lecture 4 (backup link)
• Lecture 5: Reasoning with Affordances
  • Here is the link to the Zoom recording of Lecture 5 (if prompted for a password, enter "N6W17cW+")

Please let us know if you have any issues accessing the course materials.

Readings

• Pustejovsky, J. and Krishnaswamy, N. (2022). Multimodal Semantics for Affordances and Actions. In International Conference on Human-Computer Interaction (HCII). Springer.
• Krishnaswamy, N., Pickard, W., Cates, B., Blanchard, N., and Pustejovsky, J. (2022). The VoxWorld Platform for Multimodal Embodied Agents. In Language Resources and Evaluation Conference (LREC). ACL.
• Pustejovsky, J. and Krishnaswamy, N. (2021). Situated Meaning in Multimodal Dialogue: Human-Robot and Human-Computer Interactions. In Traitement Automatique des Langues: Special Issue on Dialog and Dialog Systems. Association pour le Traitement Automatique des Langues (ATALA).
• Pustejovsky, J. and Krishnaswamy, N. (2021). Embodied Human Computer Interaction. In KI - Künstliche Intelligenz: Special Issue on NLP and Semantics. Springer.
• Pustejovsky, J. (2018). From actions to events: Communicating through language and gesture. In Interaction Studies 19. John Bejamins.
• Pustejovsky, J. and Krishnaswamy, N. (2016). VoxML: A Visualization Modeling Language. In International Conference on Language Resources and Evaluation (LREC). ACL.
• Pustejovsky, J. (2013). Dynamic event structure and habitat theory. In International Conference on Generative Approaches to the Lexicon (GL). ACL.
• Pustejovsky, J. and Moszkowicz, J.L. (2011). The qualitative spatial dynamics of motion in language. In Spatial Cognition and Computation. Taylor & Francis.

Thank you to all of you who attended the course!

If you have any follow-up questions or want to stay in touch, please email us at jamesp [at] brandeis [dot] edu and nkrishna [at] colostate [dot] edu!