Multimodal Data Recognition
Research Team

Research Summary

Our goal is comprehensive environment understanding around a robot. It includes object recognition, 3D scene recognition, and human activity recognition through signal processing / pattern recognition on multimodal sensory data. Especially, we focus on recognizing unknown events / objects.

Main Research Fields
  • Computer Vision
  • Robot Vision
  • Multimodal Recognition
Keywords
  • Object Recognition
  • Activity Recognition
  • Spatio-temporal Environmental Understanding
  • Perception of Unknown Event/Object
  • Scene Graph Generation
Research theme
  • Unknown Object Recognition
  • Recognition from a Skeleton Sequence
  • Scene Change Detection
  • Human Behavior Change Detection

Yasutomo Kawanishi

Yasutomo Kawanishi

History

2006
Bachelor of Engineering, Kyoto University
2008
Master of Informatics, Kyoto University
2011
Ph.D Informatics, Kyoto University

Award

2009
Best Paper Award
2016
IEEE ITS Society Nagoya Chapter Young Researcher Award

Members

Motoharu Sonogashira
Research Scientist
Vijay John
Research Scientist
Tomohiro Fujita
Postdoctoral Researcher
Christiane Mietzsch
Research Associate
Shohei Nobuhara
Visiting Scientist
Akira Kohjin
Administrative Part-time Worker II and Student Trainee
Daiju Kanaoka
Student Trainee
Tingwei Liu
Student Trainee
Itthisak Phueaksri
Student Trainee
Diego Hernandez Rodriguez
Student Trainee
Da Huo
Student Trainee
Ziqi Li
Student Trainee

Former member

Hayato Yumiya
Research Intern (2021/07-2021/08)
Masaya Mizuno
Research Intern (2021/08-2021/09)
Thomas Reolon
Research Intern (2022/12-2023/01)
Kotaro Fujishiro
Research Intern(2023/09)
Haruto Kugo
Research Intern(2023/09)
Daijiro Suzuki
Research Intern(2023/09)
Joy Battocchio
Research Intern(2023/09-2023/10)

Research results

Unknown object recognition and description

When we humans see an unknown object, we can recognize it as some kind of object even if we don't know what it is. We also describe the relationship with other objects, e.g., an unknown object is on the table and besides the laptop.

On the other hand, robots can only detect objects that their object detectors have learned about and cannot estimate the relationship with other objects. Our team is researching the topic, "object recognition including unknown objects and relationship estimation".

The recognition problem including unknown objects is called the open-set recognition problem, which has recently attracted much attention in the computer vision field. On the other hand, the problem of recognizing relations among objects and describing them in a graph structure is called scene graph generation (SGG). Our team has named the problem of describing a scene containing unknown objects in a graph structure as open-set scene graph generation (Open-set SGG).

We have formulated the problem setup, proposed experimental protocols and evaluation metrics, and proposed a baseline method of the problem.

未知の物体を含むシーンの記述

Human pose prediction from short time observations

Observing a person's activities and predicting the person's current state and future pose a few seconds later are important for many applications, such as proactive support by robots. Our team is working on predicting a person's future poses by observing the short-term behavior of the person.

Recent development in pose estimation techniques has led to many studies on human behavior using a sequence of human skeletons. Sequences of human skeletons are often considered a graph; vertices have locations of body joints, and edges represent the connectivity of body joints. Thus, graph-convolution-based methods have been proposed. However, some of the body motions cannot be distinguished only from the skeleton sequence in the future pose estimation task. In our study, we have proposed a method to predict future motions by using additional information, such as human surroundings.

短時間の観測からの人の姿勢予測

Selected Publications

  1. Itthisak Phueaksri, Marc A. Kastner, Yasutomo Kawanishi, Takahiro Komamizu, Ichiro Ide
    “Image-Collection Summarization Using Scene-Graph Generation With External Knowledge”
    IEEE Access, vol.12, pp. 17499 - 17512, (2024) (open access)
  2. Itthisak Phueaksri, Marc A. Kastner, Yasutomo Kawanishi, Takahiro Komamizu, Ichiro Ide
    “An Approach to Generate a Caption for an Image Collection Using Scene Graph Generation”
    IEEE Access, vol.11, pp. 128245 - 128260, (2023) (open access)
  3. Daiju Kanaoka, Hakaru Tamukoh, Motoharu Sonogashira, Yasutomo Kawanishi
    “ManifoldNeRF: View-dependent Image Feature Supervision for Few-shot Neural Radiance Fields”
    In Proceedings of the 34th British Machine Vision Conference, (2023)
  4. Shu Nakamura, Yasutomo Kawanishi, Shohei Nobuhara, Ko Nishino
    “DeePoint: Visual Pointing Recognition and Direction Estimation”
    In Proceedings of the 19th International Conference on Computer Vision, (2023)
  5. Tomohiro Fujita, Yasutomo Kawanishi
    “Human Pose Prediction by Progressive Generation in Multi-scale Frequency Domain”
    In Proceedings of the 18th International Conference on Machine Vision Applications, (2023)
  6. Vijay John, Yasutomo Kawanishi
    “Combining Knowledge Distillation and Transfer Learning for Sensor Fusion in Visible and Thermal Camera-based Person Classification”
    In Proceedings of the 18th International Conference on Machine Vision Applications, (2023)
  7. Vijay John, Yasutomo Kawanishi
    “Multimodal Cascaded Framework with Metric Learning Robust to Missing Modalities for Person Classification”
    In Proceedings of the 14th ACM Multimedia Systems Conference, (2023) (open access)
  8. Vijay John, Yasutomo Kawanishi
    "Progressive Learning of a Multimodal Classifier Accounting for Different Modality Combinations"
    Sensors 2023, 23(10), 4666 (2023) (open access)
  9. Masaya Mizuno, Tomohiro Fujita, Yasutomo Kawanishi, Daisuke Deguchi, Hiroshi Murase
    "Subjective Baggage-Weight Estimation from Gait ---Can you estimate how heavy the person feels?---"
    In Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP), (2023)
  10. Hayato Yumiya, Yasutomo Kawanishi, Daisuke Deguchi, Hiroshi Murase
    "End-to-End Gaze Grounding of a Person Pictured from Behind"
    In Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP), (2023)
  11. Tomohiro Fujita, Yasutomo Kawanishi
    "Future Pose Prediction from 3D Human Skeleton Sequence with Surrounding Situation"
    Sensors 2023, 23(2), 876 (2023) (open access)
  12. Itthisak Phueaksri, Marc A. Kastner, Yasutomo Kawanishi, Takahiro Komamizu, Ichiro Ide
    "Towards Captioning an Image Collection from a Combined Scene Graph Representation Approach"
    In Proceedings of the 29th International Conference on MultiMedia Modeling (2023)
  13. Vijay John, Yasutomo Kawanishi
    "Audio-Visual Sensor Fusion Framework using Person Attributes Robust to Missing Visual Modality for Person Recognition"
    In Proceedings of the 29th International Conference on MultiMedia Modeling (2023)
  14. Jiaxin Li, Yasutomo Kawanishi, Daisuke Deguchi, Hiroshi Murase
    "A Preliminary Study on View Independent Panoptic Scene Change Detection"
    In proceedings of the 2023 International Workshop on Advanced Image Technology (2023)
  15. Vijay John, Yasutomo Kawanishi
    "A Multimodal Sensor Fusion Framework Robust to Missing Modalities for Person Recognition"
    In proceedings of the ACM Multimedia Asia 2022 (2022)
  16. Yasutomo Kawanishi, Ichiro Ide, Baidong Chu, Chihaya Matsuhira, Marc A. Kastner, Takahiro Komamizu, Daisuke Deguchi
    "Detection of Birds in a 3D Environment Referring to Audio-Visual Information"
    In Proceedings of the 18th IEEE International Conference on Advanced Video and Signal-based Surveillance (2022)
  17. Vijay John, Yasutomo Kawanishi
    "Audio and Video-Based Emotion Recognition Using Multimodal Transformers"
    In Proceedings of the 26th International Conference on Pattern Recognition (2022).
  18. Yasutomo Kawanishi
    "Label-Based Multiple Object Ensemble Tracking with Randomized Frame Dropping"
    In Proceedings of the 26th International Conference on Pattern Recognition (2022).
  19. Tomohiro Fujita, Yasutomo Kawanishi
    "Toward Surroundings-aware Temporal Prediction of 3D Human Skeleton Sequence"
    In Proceedings of the 26th ICPR Workshop: Towards a Complete Analysis of People: From Face and Body to Clothes (2022).
  20. Motoharu Sonogashira, Masaaki Iiyama, Yasutomo Kawanishi,
    "Towards Open-Set Scene Graph Generation with Unknown Objects"
    IEEE Access, Vol.10, pp.11574-11583 (2022) ( open access )
  21. Mahmud Dwi Sulistiyo, Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Takatsugu Hirayama, Hiroshi Murase.:
    "ColAtt-Net: In Reducing the Ambiguity of Pedestrian Orientations on Attribute-aware Semantic Segmentation Task"
    IEEJ Transactions on Electronics, Information and Systems, Vol. 16, Issue 2, (2021).
  22. Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Hiroshi Murase.:
    "Ω-GAN: Object Manifold Embedding GAN for Image Generation by Disentangling Parameters into Pose and Shape Manifolds"
    In Proceedings of the International 25th International Conference on Pattern Recognition (2020).
  23. Hiroki Tatemichi, Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Hiroshi Murase.:
    "Median-shape Representation Learning for Category-level Object Pose Estimation in Cluttered Environments"
    In Proceedings of the International 25th International Conference on Pattern Recognition (2020).
  24. Saki Iwata, Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Hiroshi Murase.:
    "LFIR2Pose: Pose Estimation from an Extremely Low-Resolution FIR Image Sequence"
    In Proceedings of the International 25th International Conference on Pattern Recognition (2020).
  25. Hitoshi Nishimura, Kazuyuki Tasaka, Yasutomo Kawanishi, Hiroshi Murase.:
    "Multiple Human Tracking with Alternately Updating Trajectories and Multi-Frame Action Features"
    ITE Transactions on Media Technology and Applications, Vol. 8, No.4, pp. 269-279, (2020).
  26. Hitoshi Nishimura, Kazuyuki Tasaka, Yasutomo Kawanishi, Hiroshi Murase.:
    "Multiple Human Tracking using an Omnidirectional Camera with Local Rectification and World Coordinates Representation"
    IEICE Transactions on Information and Systems, Vol. E103-D, No. 6, pp.1745-1361, (2020).
  27. Naoki Nishida, Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Hiroshi Murase, Jun Piao.:
    "SOANets: Encoder-Decoder based Skeleton Orientation Alignment Network for White Cane User Recognition from 2D Human Skeleton Sequence"
    In Proceedings of the 15th International Conference on Computer Vision Theory and Applications, pp. 435-443, 2020.
  28. Yasutomo Kawanishi, Hiroshi Murase, Jianfeng Xu, Kazuyuki Tasaka, Hiromasa Yanagihara.:
    "Which Content is he/she Reading? --Reading Content Estimation using an Indoor Surveillance Camera--"
    In Proceedings of the 24th International Conference on Pattern Recognition, pp. 1731-1736, (2018).
  29. Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Hiroshi Murase.:
    "Trajectory Ensemble: Multiple Persons Consensus Tracking across Non-overlapping Multiple Cameras over Randomly Dropped Camera Networks"
    In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshop, pp. 56-62, (2017).
  30. Brahmastro Kresnaraman, Yasutomo Kawanishi, Daisuke Deguchi, Tomokazu Takahashi, Yoshito Mekada, Ichiro Ide, Hiroshi Murase.:
    "Human Wearable Attribute Recognition using Probability-Map-based Decomposition of Thermal Infrared Images"
    IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, Vol.E100-A Issue 3, pp.854-864, (2017).

Links

Contact Information

yasutomo.kawanishi [at] riken.jp

Top