Object Extraction模块

[cosxsinxds]

感谢您分享了项目中使用的特征文件以及训练过程，这对科研社区非常有帮助。
我特别注意到，Object Extraction 模块采用 Co-DETR 进行目标识别的设计非常巧妙。
不知是否方便公开这部分对应的代码？
如能提供，我将十分感激！

[Yisen-Feng]

Co-Detr.zip

object feature

• 利用Co-Detr进行提取，具体利用Co-Detr/get_clip_detr_feature.py

object-all

从上一步结果中提取视频中大于阈值0.6的所有obj，每帧仅提取最大值的那个，且尽量至少提取10个obj
使用该文件进行提取：Co-Detr/get_all_obj_withminnum.py

object text feature

提取物品种类的文本特征，参考get_classname_feature.ipynb

get_classname_feature.ipynb

[Yisen-Feng]

我之前应该已经把所有bug修复了，你可能需要确认一下

[cosxsinxds]

我之前应该已经把所有bug修复了，你可能需要确认一下

太有责任心了我的哥，深度学习圈都该向你学习Orz

[Yisen-Feng]

不敢当不敢当

[nsubedi11]

How do select objects that are related to the objects in the NLQ in the object selection stage? These are the "label" in the annotation files you provide. Like the following:

For NLQ: "Who did I talk to when I was washing the utensils?"
we select:
"label": [{"class_id": 303, "class_name": "cooking utensil", "score": 0.7986313700675964, "text": "utensils"}]}

I am trying to extract such annotation from NLQ for a dataset similar to Ego4D-NLQ. Specifically, how do you select cooking utensil given the NLQ?

[Yisen-Feng]

Co-Detr.zip

object feature

• 利用Co-Detr进行提取，具体利用Co-Detr/get_clip_detr_feature.py

object-all

从上一步结果中提取视频中大于阈值0.6的所有obj，每帧仅提取最大值的那个，且尽量至少提取10个obj 使用该文件进行提取：Co-Detr/get_all_obj_withminnum.py

object text feature

提取物品种类的文本特征，参考get_classname_feature.ipynb

get_classname_feature.ipynb

Please refer to this compressed file, Co-Detr/spacy/get_synonym_spacy.py is for matching nouns in text with categories in LVIS.