感覚データ認識研究チーム

研究概要

ロボットの周りにいる人や、周りに存在する物体に対する詳細な認識を目的とし、ロボットの周囲環境を観測した多様なセンサデータに対する信号処理・パターン認識に関する研究を行います。そのなかでも、特に未知事象の認識に注目しています．

研究分野

• コンピュータビジョン
• ロボットビジョン
• マルチモーダル認識

キーワード

• 物体認識
• 行動認識
• 時空間環境理解
• 未知事象の知覚
• シーングラフ生成

研究テーマ

• 未知物体を含む環境理解
• 骨格系列からの認識
• 環境の変化検出
• 人物のふるまいの変化検出

川西 康友

メンバー

薗頭 元春

研究員

Vijay John

研究員

Itthisak Phueaksri

特別研究員

Christiane Mietzsch

特別技術員

延原 章平

客員研究員

藤田 倫弘

客員研究員

幸壬 晃

研究パートタイマーII兼研修生

Tingwei Liu

研究パートタイマーII兼研修生

金岡 大樹

研修生

Diego Hernandez Rodriguez

研修生

Da Huo

研修生

Nguyen Trung Thanh

研修生

玉木 太耀

研修生

矢野 優雅

研修生

尾﨑 亜依里

研修生

村川 稔一

研修生

樋江井 捷

研修生

山田 史恩

研修生

Yo-hsin Fang

研修生

Hao-yu Hou

研修生

Jia-yi Chen

研修生

Yu-chen Lai

研修生

平川 颯人

研修生

Yo-Hsin Fang

研修生

過去のメンバー

Ziqi Li

研修生(2022/12-2024/3)

Joy Battocchio

インターン(2023/09-2023/10)

弓矢 隼大

インターン(2021/07-2021/08)

水野 雅也

インターン(2021/08-2021/09)

Thomas Reolon

インターン(2022/12-2023/01)

藤代 晃太朗

インターン(2023/9)

久郷 陽登

インターン(2023/9)

鈴木 大二朗

インターン(2023/9)

研究成果

未知の物体を含むシーンの記述

人間は，知らない物体を見たとき，それが何かわからなくても何らかの物体があると認識でき，また，それが机の上に乘っている，椅子の横にある等，他の物体との関係も説明できます．

一方，ロボットは物体検出器が学習した物体しか検出できません．また，他の物体との関係も推定できません．そこで，我々のチームでは，未知の物体の検出に加え，その周囲にある物体との関係まで含めて認識する手法について研究しています．

未知の物体を含む認識問題は，Open-set認識問題と呼ばれ，画像認識の分野で最近注目されています．一方で，物体同士の関係を認識する問題は，シーングラフ推定と呼ばれ，認識した物体とそれらの関係をグラフ構造で表現します．我々のチームでは，未知の物体を含むシーンをシーングラフの形式で表現する課題のことを，Open-setシーングラフ推定と名付けました．

この研究では，まず問題設定の定式化，実験プロトコル・評価指標の提案，ベースライン手法の提案をしています．

短時間の観測からの人の姿勢予測

人が何かをしている様子を観測し，人の現在の状態や数秒後の姿勢を予測することは，ロボットが人に対して適切な支援をするために必要です．我々のチームでは，人の短時間の振る舞いを観測し，その先の姿勢を推定する研究に取り組んでいます．

近年の姿勢推定技術の進歩から，姿勢推定結果の人物骨格系列を元に，人の行動を認識する研究が盛んに取り組まれています．人の骨格は関節とそれらの連結関係で表せるため，各頂点に3次元関節点座標をもつグラフとして表現されることが一般的です．しかし，骨格だけを見ていても，区別がつかない動きが存在します．本研究では，人物の周囲の情報を補助として用いることで，これらを区別し，予測精度を向上させる手法を提案しています．

主要論文

1. Motoharu Sonogashira, Masaaki Iiyama, Yasutomo Kawanishi
   “Relationship-Aware Unknown Object Detection for Open-Set Scene Graph Generation”
   IEEE Access, vol.12, pp.122513 - 122523, (2024) (open access).
2. 植田 暢大, 波部 英子, 松井 陽子, 湯口 彰重, 河野 誠也, 川西 康友, 黒橋 禎夫, 吉野 幸一郎
   “J-CRe3：実世界における参照関係解決のための日本語対話データセット”
   自然言語処理, vol. 31, no. 3, (2024) (open access).
3. Vijay John, Yasutomo Kawanishi
   “Frame-Level Latent Embedding using Weak Labels for Multi-view Action Recognition”
   IEEE International Conference on Multimedia Information Processing and Retrieval, (2024).
4. Tingwei Liu, Yasutomo Kawanishi, Takahiro Komamizu, Ichiro Ide
   “Tracking Small Birds by Detection Candidate Region Filtering and Detection History-aware Association”
   CV4Animals: Computer Vision for Animal Behavior Tracking and Modeling, In conjunction with Computer Vision and Pattern Recognition 2024, (2024).
5. Yoshimitsu Kajiwara, Wanwan Zheng, Yasutomo Kawanishi
   “Iconographic analysis of ancient roof tiles using a data science approach”
   The Indonesian Journal of Social Studies, vol. 7, no. 2, pp.41-49, (2024) (open access).
6. Trung Thanh Nguyen, Yasutomo Kawanishi, Takahiro Komamizu, Ichiro Ide
   “One-stage open-vocabulary temporal action detection leveraging temporal multi-scale and action label features”
   Proceedings of the 18th IEEE International Conference on Automatic Face and Gesture Recognition, (2024).
7. Shun Inadumi, Seiya Kawano, Akishige Yuguchi, Yasutomo Kawanishi, Koichiro Yoshino
   “A Gaze-grounded Visual Question Answering Dataset for Clarifying Ambiguous Japanese Questions”
   The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, (2024).
8. Nobuhiro Ueda, Hideko Habe, Akishige Yuguchi, Seiya Kawano, Yasutomo Kawanishi, Sadao Kurohashi, Koichiro Yoshino
   “J-CRe3: A Japanese Conversation Dataset for Real-world Reference Resolution”
   The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, (2024).
9. Yukinori Kawae, Yasutomo Kawanishi, Ichiroh Kanaya, Yoshihiro Yasumuro
   “3D Survey of the Menkaure Pyramid”
   Virtual Annual Meeting, American Research Center in Egypt, (2024).
10. Trung Thanh Nguyen, Phi Le Nguyen, Yasutomo Kawanishi, Takahiro Komamizu, Ichiro Ide
    “Zero-Shot Pill-Prescription Matching With Graph Convolutional Network and Contrastive Learning”
    IEEE Access, vol. 12, pp. 55889-55904, (2024) (open access).
11. 畑 隆聖, 出口 大輔, 平山 高嗣, 川西 康友, 村瀬 洋
    “Eye-contact Transformer: シーンコンテキストを考慮した遠方歩行者のアイコンタクト検出”
    電子情報通信学会論文誌, Vol.J107-D, No.04, pp.231-242, (2024).
12. Chihaya Matsuhira, Marc Aurel Kastner, Takahiro Komamizu, Takatsugu Hirayama, Keisuke Doman, Yasutomo Kawanishi, Ichiro Ide
    “Interpolating the Text-to-Image Correspondence Based on Phonetic and Phonological Similarities for Nonword-to-Image Generation”
    IEEE Access, vol.12, pp.41299 -41316, (2024) (open access).
13. Masaya Mizuno, Tomohiro Fujita, Yasutomo Kawanishi, Daisuke Deguchi, Hiroshi Murase
    “Subjective Baggage-Weight Estimation based on Human Walking Behavior”
    IEEE Access, Vol. 12, pp. 39390 - 39398, (2024) (open access)
14. Hiroki Tatemichi, Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Hiroshi Murase
    “Category-level Object Pose Estimation in Heavily Cluttered Scenes by Generalized Two-stage Shape Reconstructor”
    IEEE Access, vol. 12, pp. 33440-33448, (2024) (open access).
15. Naoya Kawamura, Wataru Sato, Koh Shimokawa, Tomohiro Fujita, Yasutomo Kawanishi
    “Machine learning-based interpretable modeling for subjective emotional dynamics sensing using facial EMG”
    Sensors, vol. 24, no. 5, 1536, (2024) (open access).
16. Angel Garcia Contreras, Seiya Kawano, Yasutomo Kawanishi, Yutaka Nakamura, Saito Satoru, Koichiro Yoshino
    “Examining the Impact of a Forgetful Multi-store Memory System in a Cognitive Assistive Robot”
    The 14th International Workshop on Spoken Dialogue Systems Technology, (2024).
17. Hiroto Murakami, Jialei Chen, Daisuke Deguchi, Takatsugu Hirayama, Yasutomo Kawanishi, Hiroshi Murase
    “Pedestrian's Gaze Object Detection in Traffic Scene”
    Proceedings of the 19th International Conference on Computer Vision Theory and Applications (VISAPP), (2024).
18. Itthisak Phueaksri, Marc A. Kastner, Yasutomo Kawanishi, Takahiro Komamizu, Ichiro Ide
    “Image-Collection Summarization Using Scene-Graph Generation With External Knowledge”
    IEEE Access, vol.12, pp. 17499 - 17512, (2024) (open access)
19. Itthisak Phueaksri, Marc A. Kastner, Yasutomo Kawanishi, Takahiro Komamizu, Ichiro Ide
    “An Approach to Generate a Caption for an Image Collection Using Scene Graph Generation”
    IEEE Access, vol.11, pp. 128245 - 128260, (2023) (open access)
20. Daiju Kanaoka, Hakaru Tamukoh, Motoharu Sonogashira, Yasutomo Kawanishi
    “ManifoldNeRF: View-dependent Image Feature Supervision for Few-shot Neural Radiance Fields”
    In Proceedings of the 34th British Machine Vision Conference, (2023)
21. Shu Nakamura, Yasutomo Kawanishi, Shohei Nobuhara, Ko Nishino
    “DeePoint: Visual Pointing Recognition and Direction Estimation”
    In Proceedings of the 19th International Conference on Computer Vision, (2023)
22. Tomohiro Fujita, Yasutomo Kawanishi
    “Human Pose Prediction by Progressive Generation in Multi-scale Frequency Domain”
    In Proceedings of the 18th International Conference on Machine Vision Applications, (2023)
23. Vijay John, Yasutomo Kawanishi
    “Combining Knowledge Distillation and Transfer Learning for Sensor Fusion in Visible and Thermal Camera-based Person Classification”
    In Proceedings of the 18th International Conference on Machine Vision Applications, (2023)
24. Vijay John, Yasutomo Kawanishi
    “Multimodal Cascaded Framework with Metric Learning Robust to Missing Modalities for Person Classification”
    In Proceedings of the 14th ACM Multimedia Systems Conference, (2023) (open access)
25. Vijay John, Yasutomo Kawanishi
    "Progressive Learning of a Multimodal Classifier Accounting for Different Modality Combinations"
    Sensors 2023, 23(10), 4666 (2023) (open access)
26. Masaya Mizuno, Tomohiro Fujita, Yasutomo Kawanishi, Daisuke Deguchi, Hiroshi Murase
    "Subjective Baggage-Weight Estimation from Gait ---Can you estimate how heavy the person feels?---"
    In Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP), (2023)
27. Hayato Yumiya, Yasutomo Kawanishi, Daisuke Deguchi, Hiroshi Murase
    "End-to-End Gaze Grounding of a Person Pictured from Behind"
    In Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP), (2023)
28. Tomohiro Fujita, Yasutomo Kawanishi
    "Future Pose Prediction from 3D Human Skeleton Sequence with Surrounding Situation"
    Sensors 2023, 23(2), 876 (2023) (open access)
29. Itthisak Phueaksri, Marc A. Kastner, Yasutomo Kawanishi, Takahiro Komamizu, Ichiro Ide
    "Towards Captioning an Image Collection from a Combined Scene Graph Representation Approach"
    In Proceedings of the 29th International Conference on MultiMedia Modeling (2023)
30. Vijay John, Yasutomo Kawanishi
    "Audio-Visual Sensor Fusion Framework using Person Attributes Robust to Missing Visual Modality for Person Recognition"
    In Proceedings of the 29th International Conference on MultiMedia Modeling (2023)
31. Jiaxin Li, Yasutomo Kawanishi, Daisuke Deguchi, Hiroshi Murase
    "A Preliminary Study on View Independent Panoptic Scene Change Detection"
    In proceedings of the 2023 International Workshop on Advanced Image Technology (2023)
32. Vijay John, Yasutomo Kawanishi
    "A Multimodal Sensor Fusion Framework Robust to Missing Modalities for Person Recognition"
    In proceedings of the ACM Multimedia Asia 2022 (2022)
33. Yasutomo Kawanishi, Ichiro Ide, Baidong Chu, Chihaya Matsuhira, Marc A. Kastner, Takahiro Komamizu, Daisuke Deguchi
    "Detection of Birds in a 3D Environment Referring to Audio-Visual Information"
    In Proceedings of the 18th IEEE International Conference on Advanced Video and Signal-based Surveillance (2022)
34. Vijay John, Yasutomo Kawanishi
    "Audio and Video-Based Emotion Recognition Using Multimodal Transformers"
    In Proceedings of the 26th International Conference on Pattern Recognition (2022).
35. Yasutomo Kawanishi
    "Label-Based Multiple Object Ensemble Tracking with Randomized Frame Dropping"
    In Proceedings of the 26th International Conference on Pattern Recognition (2022).
36. Tomohiro Fujita, Yasutomo Kawanishi
    "Toward Surroundings-aware Temporal Prediction of 3D Human Skeleton Sequence"
    In Proceedings of the 26th ICPR Workshop: Towards a Complete Analysis of People: From Face and Body to Clothes (2022).
37. Motoharu Sonogashira, Masaaki Iiyama, Yasutomo Kawanishi,
    "Towards Open-Set Scene Graph Generation with Unknown Objects"
    IEEE Access, Vol.10, pp.11574-11583 (2022) ( open access )
38. Mahmud Dwi Sulistiyo, Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Takatsugu Hirayama, Hiroshi Murase.:
    "ColAtt-Net: In Reducing the Ambiguity of Pedestrian Orientations on Attribute-aware Semantic Segmentation Task"
    IEEJ Transactions on Electronics, Information and Systems, Vol. 16, Issue 2, (2021).
39. Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Hiroshi Murase.:
    "Ω-GAN: Object Manifold Embedding GAN for Image Generation by Disentangling Parameters into Pose and Shape Manifolds"
    In Proceedings of the International 25th International Conference on Pattern Recognition (2020).
40. Hiroki Tatemichi, Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Hiroshi Murase.:
    "Median-shape Representation Learning for Category-level Object Pose Estimation in Cluttered Environments"
    In Proceedings of the International 25th International Conference on Pattern Recognition (2020).
41. Saki Iwata, Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Hiroshi Murase.:
    "LFIR2Pose: Pose Estimation from an Extremely Low-Resolution FIR Image Sequence"
    In Proceedings of the International 25th International Conference on Pattern Recognition (2020).
42. Hitoshi Nishimura, Kazuyuki Tasaka, Yasutomo Kawanishi, Hiroshi Murase.:
    "Multiple Human Tracking with Alternately Updating Trajectories and Multi-Frame Action Features"
    ITE Transactions on Media Technology and Applications, Vol. 8, No.4, pp. 269-279, (2020).
43. Hitoshi Nishimura, Kazuyuki Tasaka, Yasutomo Kawanishi, Hiroshi Murase.:
    "Multiple Human Tracking using an Omnidirectional Camera with Local Rectification and World Coordinates Representation"
    IEICE Transactions on Information and Systems, Vol. E103-D, No. 6, pp.1745-1361, (2020).
44. Naoki Nishida, Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Hiroshi Murase, Jun Piao.:
    "SOANets: Encoder-Decoder based Skeleton Orientation Alignment Network for White Cane User Recognition from 2D Human Skeleton Sequence"
    In Proceedings of the 15th International Conference on Computer Vision Theory and Applications, pp. 435-443, 2020.
45. Yasutomo Kawanishi, Hiroshi Murase, Jianfeng Xu, Kazuyuki Tasaka, Hiromasa Yanagihara.:
    "Which Content is he/she Reading? --Reading Content Estimation using an Indoor Surveillance Camera--"
    In Proceedings of the 24th International Conference on Pattern Recognition, pp. 1731-1736, (2018).
46. Yasutomo Kawanishi, Daisuke Deguchi, Ichiro Ide, Hiroshi Murase.:
    "Trajectory Ensemble: Multiple Persons Consensus Tracking across Non-overlapping Multiple Cameras over Randomly Dropped Camera Networks"
    In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshop, pp. 56-62, (2017).
47. Brahmastro Kresnaraman, Yasutomo Kawanishi, Daisuke Deguchi, Tomokazu Takahashi, Yoshito Mekada, Ichiro Ide, Hiroshi Murase.:
    "Human Wearable Attribute Recognition using Probability-Map-based Decomposition of Thermal Infrared Images"
    IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, Vol.E100-A Issue 3, pp.854-864, (2017).

関連リンク

川西康友
感覚データ認識研究チーム(理化学研究所)