Fix FarSight attention missing RoPE and shell script argument mismatch

[Copilot]

Two issues: (1) FarSight decoding outputs only "." due to missing rotary position embeddings, (2) eval_chair.sh uses CHAIR evaluation arguments but eval_chair.py is an inference script with different args.

FarSight Attention Fix (farsight_patch.py)

• Added RoPE support: LLaMA requires rotary position embeddings for positional understanding. Added rotate_half() and apply_rotary_pos_emb() with defensive tensor shape handling.
• GQA support: Handle grouped-query attention by repeating KV heads when num_key_value_heads < num_heads
• Improved attention mask handling: Properly apply HuggingFace-style additive masks

# Apply rotary position embeddings (critical for LLaMA!)
if hasattr(self, 'rotary_emb'):
    cos, sin = self.rotary_emb(k, seq_len=kv_seq_len)
    q, k = apply_rotary_pos_emb(q, k, cos, sin, position_ids)

# Handle GQA
if num_key_value_heads < self.num_heads:
    num_key_value_groups = self.num_heads // num_key_value_heads
    k = k.repeat_interleave(num_key_value_groups, dim=1)
    v = v.repeat_interleave(num_key_value_groups, dim=1)

Shell Script Fix (eval_chair.sh)

• Changed from hardcoded CHAIR eval args to correct inference args matching eval_chair.py
• Uses environment variables with validation instead of placeholder paths:

IMAGE_FOLDER=/path/to/coco/val2014 QUESTION_FILE=/path/to/questions.jsonl bash eval_chair.sh

Original prompt

(1)
when using farsight setting , the model only outputs a “.”

Status: Open.
#3 In FeilongTangmonash/FarSight;· havefun12321 opened 3 weeks ago
eval_chair.py: error: unrecognized arguments: --coco_path /root/autodl-tmp/annotations --cache ./chair.pkl --cap_file ./1.jsonl --save_path ./Answers/eval-chair.json

Status: Open.

帮我解决这两个问题

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.