Multimodal Data Recognition
Research Team

Research Summary

Our goal is comprehensive environment understanding around a robot. It includes object recognition, 3D scene recognition, and human activity recognition through signal processing / pattern recognition on multimodal sensory data. Especially, we focus on recognizing unknown events / objects.

Main Research Fields
  • Computer Vision
  • Robot Vision
  • Multimodal Recognition
Keywords
  • Object Recognition
  • Activity Recognition
  • Spatio-temporal Environmental Understanding
  • Perception of Unknown Event/Object
  • Scene Graph Generation
Research theme
  • Unknown Object Recognition
  • Recognition from a Skeleton Sequence
  • Scene Change Detection
  • Human Behavior Change Detection

Yasutomo Kawanishi

Yasutomo Kawanishi

History

2006
Bachelor of Engineering, Kyoto University
2008
Master of Informatics, Kyoto University
2011
Ph.D Informatics, Kyoto University

Award

2009
Best Paper Award
2016
IEEE ITS Society Nagoya Chapter Young Researcher Award

Members

Motoharu Sonogashira
Research Scientist
Vijay John
Research Scientist
Tomohiro Fujita
Postdoctral Researcher
Daiju Kanaoka
Student Trainee
Liu Tingwei
Research Intern

Former member

Hayato Yumiya
Research Intern(2021.07-2021.08)
Masaya Mizuno
Research Intern(2021.08-2021.09)

Research results

Unknown object recognition and description

When we humans see an unknown object, we can recognize it as some kind of object even if we don't know what it is. We also describe the relationship with other objects, e.g., an unknown object is on the table and besides the laptop.

On the other hand, robots can only detect objects that their object detectors have learned about and cannot estimate the relationship with other objects. Our team is researching the topic, "object recognition including unknown objects and relationship estimation".

The recognition problem including unknown objects is called the open-set recognition problem, which has recently attracted much attention in the computer vision field. On the other hand, the problem of recognizing relations among objects and describing them in a graph structure is called scene graph generation (SGG). Our team has named the problem of describing a scene containing unknown objects in a graph structure as open-set scene graph generation (Open-set SGG).

We have formulated the problem setup, proposed experimental protocols and evaluation metrics, and proposed a baseline method of the problem.

未知の物体を含むシーンの記述

Selected Publications

  1. Vijay John, Yasutomo Kawanishi
    Audio and Video-Based Emotion Recognition Using Multimodal Transformers
    In Proceedings of the 26th International Conference on Pattern Recognition (2022).
  2. Yasutomo Kawanishi
    Label-Based Multiple Object Ensemble Tracking with Randomized Frame Dropping
    In Proceedings of the 26th International Conference on Pattern Recognition (2022).
  3. Tomohiro Fujita, Yasutomo Kawanishi
    Toward Surroundings-aware Temporal Prediction of 3D Human Skeleton Sequence
    In Proceedings of the 26th ICPR Workshop: Towards a Complete Analysis of People: From Face and Body to Clothes (2022).
  4. Motoharu Sonogashira, Masaaki Iiyama, Yasutomo Kawanishi,
    "Towards Open-Set Scene Graph Generation with Unknown Objects"
    IEEE Access, Vol.10, pp.11574-11583 (2022) ( open access )
  5. Mahmud Dwi Sulistiyo, Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Takatsugu Hirayama, Hiroshi Murase.:
    "ColAtt-Net: In Reducing the Ambiguity of Pedestrian Orientations on Attribute-aware Semantic Segmentation Task"
    IEEJ Transactions on Electronics, Information and Systems, Vol. 16, Issue 2, (2021).
  6. Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Hiroshi Murase.:
    "Ω-GAN: Object Manifold Embedding GAN for Image Generation by Disentangling Parameters into Pose and Shape Manifolds"
    In Proceedings of the International 25th International Conference on Pattern Recognition (2020).
  7. Hiroki Tatemichi, Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Hiroshi Murase.:
    "Median-shape Representation Learning for Category-level Object Pose Estimation in Cluttered Environments"
    In Proceedings of the International 25th International Conference on Pattern Recognition (2020).
  8. Saki Iwata, Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Hiroshi Murase.:
    "LFIR2Pose: Pose Estimation from an Extremely Low-Resolution FIR Image Sequence"
    In Proceedings of the International 25th International Conference on Pattern Recognition (2020).
  9. Hitoshi Nishimura, Kazuyuki Tasaka, Yasutomo Kawanishi, Hiroshi Murase.:
    "Multiple Human Tracking with Alternately Updating Trajectories and Multi-Frame Action Features"
    ITE Transactions on Media Technology and Applications, Vol. 8, No.4, pp. 269-279, (2020).
  10. Hitoshi Nishimura, Kazuyuki Tasaka, Yasutomo Kawanishi, Hiroshi Murase.:
    "Multiple Human Tracking using an Omnidirectional Camera with Local Rectification and World Coordinates Representation"
    IEICE Transactions on Information and Systems, Vol. E103-D, No. 6, pp.1745-1361, (2020).
  11. Naoki Nishida, Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Hiroshi Murase, Jun Piao.:
    "SOANets: Encoder-Decoder based Skeleton Orientation Alignment Network for White Cane User Recognition from 2D Human Skeleton Sequence"
    In Proceedings of the 15th International Conference on Computer Vision Theory and Applications, pp. 435-443, 2020.
  12. Yasutomo Kawanishi, Hiroshi Murase, Jianfeng Xu, Kazuyuki Tasaka, Hiromasa Yanagihara.:
    "Which Content is he/she Reading? --Reading Content Estimation using an Indoor Surveillance Camera--"
    In Proceedings of the 24th International Conference on Pattern Recognition, pp. 1731-1736, (2018).
  13. Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Hiroshi Murase.:
    "Trajectory Ensemble: Multiple Persons Consensus Tracking across Non-overlapping Multiple Cameras over Randomly Dropped Camera Networks"
    In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshop, pp. 56-62, (2017).
  14. Brahmastro Kresnaraman, Yasutomo Kawanishi, Daisuke Deguchi, Tomokazu Takahashi, Yoshito Mekada, Ichiro Ide, Hiroshi Murase.:
    "Human Wearable Attribute Recognition using Probability-Map-based Decomposition of Thermal Infrared Images"
    IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, Vol.E100-A Issue 3, pp.854-864, (2017).

Links

Contact Information

yasutomo.kawanishi [at] riken.jp