Skip to content

VectorFeatureExtractor

Kateryna edited this page Mar 10, 2017 · 1 revision

RelTextRank contains feature extractors to compute:

  • (i) cosine similarity over the text pair: simCOS*(T1, T2), where the input vectors are composed of word lemmas, bi-, three- an four-grams, POS-tags; similarity based on the PTK score computed for the structural representations of T1 and T2: simPTK (T1,T2) = * PTK(T1, T2), where the input trees can be both the dependency trees and/or the shallow chunk trees

  • *(ii) IR score, that is a normalized score assigned to the answer passage by an IR engine, if available.

  • *(iii) question class as a binary feature;

Then, RelTextRank includes feature extractors based on the DKPro Similarity tool for extracting:

  • (iv) longest common substring/subsequence measure;
  • (v) Jaccard similarity coefficient on 1,2,3,4-grams;
  • (vi) word containment measure (A. Z Broder. 1997. On the resemblance and containment of documents. In Compression and Complexity of Sequences 1997. Proceedings. IEEE);
  • (vii) greedy string tiling (Michael J. Wise. 1996. Yap3: improved detection of simi- larities in computer program and other texts. In Proceedings of the twenty-seventh SIGCSE technical symposium on Computer science education. ACM, SIGCSE ’96.) ;
  • (viii) ESA similarity based on Explicit Semantic Analysis (ESA) (Evgeniy Gabrilovich and Shaul Markovitch. 2007. Computing semantic relatedness using Wikipedia-based explicit semantic analysis. In Proceedings of the 20th international joint conference on Artifical intelligence (IJCAI'07), Rajeev Sangal, Harish Mehta, and R. K. Bagga (Eds.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1606-1611.);

We provide several predefined VectorFeatureExtractors: BaselineFeatures, AllFeatures and NoESAAllFeatures which incorporate features groups (i)-(iii), (i)-(viii) and (i)-(vii), respectively.

Finally, we provide a feature extractor that reads the pre-extracted features from file, FromFileVectorFeature. The full list of features can be available when exploring the contents and documentation of it.unitn.nlpir.features package.

Clone this wiki locally