Repository navigation
System module
System modules are the entry point to the pipeline, they initialize the specific structure and feature vector generation strategies Experiment and VectorFeatureExtractor and define the type (classification or reranking) and the format of output file.
Currently, we provide the system modules for generating classification (ClassTextPairconversion) and reranking training (RERTextPairConversion) files. Then, we provide system to generate the cross-validation experimental data (ClassCVTextPairConversion and CVRERTextPairConversion). Additionally, we publish a system for generating training/test data for the answer comment reranking community question answering task, CQASemevalTaskA. Finally, we provide a prediction module for classifying and reranking new text pairs with a pre-trained classifier (TextPairPrediction).
Every system must be run as follows:
java -Xmx2G -Xss512m it.unitn.nlpir.system.core.<system-class> <general-parameters> <system-specific-parameters>-
<system-class>is the name of the system modules. See the list of the system modules and their purpose here and the examples of their outputs here. -
<general-parameters>are the input parameters common for all the systems, e.g. path to the input data. You may look them up here. -
<system-specific-parameters>are the parameters particular for a specific system. For example, when you are running an experiment in cross-validation, you should specify the number of folds you wish to generate. You can see the parameters specific for each system here.
- ClassTextPairConversion. Generates the SVMLightTK train/test file to be used with the classification kernel
- ClassCVTextPairConversion. The class converts the input files with the raw texts of the text pairs, e.g questions and answers, and generates the cross-validation folds with the input training/test files to be used with the classification kernel
- CVRERTextPairConversion. The class converts the input files with the raw texts of the text pairs, e.g questions and answers, and generates the cross-validation folds with the input training/test files to be used with the reranking kernel
- RERTextPairConversion. Generates the SVMLightTK train/test file to be used with the classification kernel
- TextPairPrediction. Runs prediction on the input file using a pretrained model, writes the predictions into an external file.
Mandatory parameters are in bold.
| Parameter | Description |
| -expClassName |
[String] Fully qualified name of the Experiment class instance. This parameter controls what kind of structural representations and feature vectors you will generate. |
| -expConfigPath (OPTIONAL) |
[String] Experiment configuration path. Experiment configuration path is a properties file which contains values of the configurable parameters for the experiment class specified in the expConfigPath parameter. Currently we provide two predefined configuration files, which determine what kind of question classification to use (see the description of the question classifiers). |
| -questionsPath |
[String] Path to the questions file. The questions file format is as follows:
|
| -answersPath |
[String] Path to the answers file. The answers file format is as follows:
|
| -outputDir | [String] Output directory to write the generated files |
| -filePersistence | [String]] Path to the persistence folder where the UIMA CASes will be serialized |
| -featureExtractorClass (OPTIONAL) | [String]Preset feature extractor class (see VectorFeatureExtractor). If this parameter is not set, then the feature vector will be empty. |
| -featureCacheFileName (OPTIONAL) |
[String] If you use it.unitn.nlpir.features.presets.FromFileVectorFeature, you must specify from where to read the features vectors. The feature vector file name is tab-delimited and has three columns:
|
| -verboseResultset (OPTIONAL) | [flag] Write relevancy file in the verbose format specified by the TREC submission guidelines |
| -doNotSerializeNewAnnotations | [flag]] Do not update the annotations in the persisted files. It is false by default. This flag is needed when you a reading already persisted CASes and running some additional annotations, but do not want to persist the new annotations. False by default. |
| -allowOverwriting (OPTIONAL) | [flag] Overwrite annotations in the dynamic CASes in RAM when the pipeline is running without serializing the new annotation. False by default. This is needed for example, when you are experimenting with a new annotation engine (e.g. question classifier), and want to override the annotations that you have read from the persisted CAS files. False by default. |
When launching from the command line, prepend it.unitn.nlpir.system.core to the class name.
| System | Specific parameters |
| ClassTextPairConversion, RERTextPairConversion |
The output SVMLight-TK/relevancy files will be named svm.train/svm.train.relevancy, svm.dev/svm.dev.relevancy and svm.test/svm.relevancy correspondingly. All the files format is the same: svm.(train|dev|test) files contain one text pair representation+feature vector per line.
*.relevancy files are line-by-line aligned with the training/dev/test SVM-light-TK files and contain information about the ids of the question/answer texts used to generate the example. The format of the relevancy file is the same as the format of the -answersPath file.
|
| ClassCVTextPairConversion, CVRERTextPairConversion |
Note: Both ClassCVTextPairConversion and CVRERTextPairConversion generate cross-validation folds in the folder specified by the -outputDir parameter, with i-th fold placed into the fold${i} subfolder.
|
| TextPairPrediction |
|
A reranking training example consists in two text pairs (e.g. two question/answer passage pairs):
<(QUESTION,ANSWERPASSAGE1), (QUESTION,ANSWERPASSAGE2)>.
The example is positive if ANSWERPASSAGE1 is a correct answer for QUESTION, while ANSWERPASSAGE2 is not; and negative in the other case.
A line in the training reranking file looks as follows:
<label> |BT| RelationalQuestionRepresentation(QUESTION,ANSWERPASSAGE1) |BT| RelationalAnswerRepresentation(QUESTION,ANSWERPASSAGE1) |BT| RelationalQuestionRepresentation(QUESTION,ANSWERPASSAGE2) |BT| RelationalAnswerRepresentation(QUESTION,ANSWERPASSAGE2) |ET| FeatureVector(QUESTION,ANSWERPASSAGE1) |BV| FeatureVector(QUESTION,ANSWERPASSAGE2) |EV|
Here:
-
<label>is the label of the reranking example -
RelationalQuestionRepresentation(Q,A),RelationalAnswerRepresentation(Q,A)are the structural representations of Q and A enriched with the relational links between Q and A, built by the Experiment module. -
FeatureVector(Q,A)is the feature vector representation for the (Q,A) pair extracted by the VectorFeatureExtractor module.
A reranking test example is generated for each (QUESTION,ANSWERPASSAGE) separately (see Libin Shen and Aravind K. Joshi. 2003. An SVM based voting algorithm with application to parse reranking . In Proceedings of the Seventh Conference on Natural Language Learning, CONLL ’03, pages 9–16, Edmonton, Canada. for the theoretical grounding).
A line in svm.text generated for a specific (QUESTION,ANSWERPASSAGE) pair looks as follows:
<label> |BT| RelationalQuestionRepresentation(QUESTION,ANSWERPASSAGE) |BT| RelationalAnswerRepresentation(QUESTION,ANSWERPASSAGE) |BT| |BT| |ET| FeatureVector(QUESTION,ANSWERPASSAGE) |BV| |EV|
See the snippets of the output files produced by RERTextPairConversion, CVRERTextPairConversion below.
svm.train:
+1 |BT| (ROOT (S (NP (WP (who))) (VP (VBD (write))) (REL-FOCUS-HUM-NP (REL-JJ (white)) (REL-NNS (christmas))))) |BT| (ROOT (S (REL-NP (REL-NNP (white)) (REL-NNP (christmas))) (VP (VBZ (be))) (NP (DT (a)) (NNP (irving)) (NNP (berlin)) (NN (song))) (VP (VBG (reminisce))) (PP (IN (about))) (REL-NP (DT (a)) (JJ (old-fashioned)) (REL-NNP (christmas)) (NN (setting))))) |BT| (ROOT (S (NP (WP (who))) (VP (VBD (write))) (REL-FOCUS-HUM-NP (JJ (white)) (NNS (christmas))))) |BT| (ROOT (S (NP (NN (thumb))))) |ET| 11:0.040723148118768406 12:0.035805743701971635 14:0.3769849180313604 15:0.24137760593933996 23:0.2266699724914541 24:0.1186155202249612 25:0.21677757910180362 |BV| 14:0.4223655372965166 15:0.3252503222354572 23:0.34754056908445496 24:0.22089655011641934 25:0.19674178075871707 |EV|-1 |BT| (ROOT (S (NP (WP (who))) (REL-VP (REL-VBD (write))) (REL-FOCUS-HUM-NP (JJ (white)) (NNS (christmas))))) |BT| (ROOT (S (NP (PRP (i))) (ADVP (RB (just))) (REL-VP (REL-VBD (write))) (NP (DT (the)) (JJS (best)) (NN (song))) (NP (PRP (i))) (REL-VP (VBP (have)) (RB (ever)) (REL-VBN (write))) (VP (NN (heck))) (NP (PRP (i))) (ADVP (RB (just))) (REL-VP (REL-VBD (write))) (NP (DT (the)) (JJS (best)) (NN (song))) (NP (WDT (that))) (VP (NN (anybody))) (NP (POS ('s))) (REL-ADJP (RB (ever)) (REL-VBN (write))))) |BT| (ROOT (S (NP (WP (who))) (VP (VBD (write))) (REL-FOCUS-HUM-NP (REL-JJ (white)) (REL-NNS (christmas))))) |BT| (ROOT (S (REL-NP (REL-NNP (white)) (REL-NNP (christmas))) (VP (VBZ (be))) (NP (DT (a)) (NNP (irving)) (NNP (berlin)) (NN (song))) (VP (VBG (reminisce))) (PP (IN (about))) (REL-NP (DT (a)) (JJ (old-fashioned)) (REL-NNP (christmas)) (NN (setting))))) |ET| 1:0.2721655269759087 2:0.15848577035006547 3:0.3113995776646092 4:0.12022261795565764 5:0.25819888974716115 11:0.05780733130160799 12:0.049690399499995326 14:0.4931315404293993 15:0.3556784825189464 16:0.2721655269759087 17:0.4815434123430769 18:0.15848577035006547 19:0.3113995776646092 20:0.12022261795565764 21:0.25819888974716115 23:0.28162697492488 24:0.28229955332295686 25:0.4150440944984307 |BV| 11:0.040723148118768406 12:0.035805743701971635 14:0.3769849180313604 15:0.24137760593933996 23:0.2266699724914541 24:0.1186155202249612 25:0.21677757910180362 |EV|
svm.train.res: This file is line-by-line aligned with the svm train file and contains information about the labels of the questions and the two answer passages used to generate a specific example.
Q1024 Q1024-D987-1 Q1024-D987-5 0 0 +1Q1024 Q1024-D987-6 Q1024-D987-1 0 0 -1
svm.test:
+1 |BT| (ROOT (S (REL-FOCUS-DATE-NP (WDT (what)) (REL-NN (day))) (VP (VBZ (be)) (VBN (st.))) (REL-NP (REL-NNS (patrick)) (REL-NN (day))))) |BT| (ROOT (S (REL-FOCUS-DATE-NP (NNP (saint)) (REL-NNP (patric k))) (REL-FOCUS-DATE-NP (POS ('s)) (REL-NN (day))) (NP (DT (the)) (NNP (feast))) (PP (IN (of))) (REL-NP (NNP (saint)) (REL-NNP (patrick))) (REL-FOCUS-DATE-NP (DT (the)) (REL-NN (day))) (PP (IN (of))) (N P (DT (the)) (NNP (festival))) (PP (IN (of))) (REL-NP (REL-NNP (patrick))) (NP (-RRB- (-rrb-))) (VP (VBZ (be))) (NP (DT (a)) (JJ (cultural)) (CC (and)) (JJ (religious)) (NN (holiday))) (VP (VBD (celebra te))) (PP (IN (on))) (REL-FOCUS-DATE-NP (CD (17)) (NNP (march))))) |BT| |BT| |ET| 1:0.2383656473113981 2:0.1469952580094637 3:0.20333142711013957 4:0.11070186069251192 5:0.15974461276617433 11:0.1206537 9433217113 12:0.10320936930842799 14:0.45299095184255256 15:0.3373343264825431 16:0.2383656473113981 17:0.3202563076101743 18:0.1469952580094637 19:0.20333142711013957 20:0.11070186069251192 21:0.159744 61276617433 23:0.2093978522088741 24:0.1876009908353872 25:0.25021782979147805 |BV| |EV|
svm.relevancy/svm.dev.relevancy This file is line-by-line aligned with the svm.test or svm.dev file (in case of svm.dev.relevancy) and contains the ids of the question and answer passage employed to generate the given SVMLight-TK file line.
Q331 Q331-D330-0 0 1.00000 true Saint Patrick's Day or the Feast of Saint Patrick (, "the Day of the Festival of Patrick") is a cultural and religious holiday celebrated on 17 March.
Train, development test files are generated using the same strategy.
A line in the output SVMLight-TK file generated for the (QUESTION,ANSWERPASSAGE) pair looks as follows:
<label> |BT| RelationalQuestionRepresentation(QUESTION,ANSWERPASSAGE) |BT| RelationalAnswerRepresentation(QUESTION,ANSWERPASSAGE) |ET| FeatureVector(QUESTION,ANSWERPASSAGE) |EV|
- svm.train:
+1 |BT| (ROOT (S (NP (WP (who))) (VP (VBD (write))) (REL-FOCUS-HUM-NP (REL-JJ (white)) (REL-NNS (christmas))))) |BT| (ROOT (S (REL-NP (REL-NNP (white)) (REL-NNP (christmas))) (VP (VBZ (be))) (NP (DT (a)) (NNP (irving)) (NNP (berlin)) (NN (song))) (VP (VBG (reminisce))) (PP (IN (about))) (REL-NP (DT (a)) (JJ (old-fashioned)) (REL-NNP (christmas)) (NN (setting))))) |ET| 11:0.040723148118768406 12:0.035805743701971635 14:0.3769849180313604 15:0.24137760593933996 23:0.2266699724914541 24:0.1186155202249612 25:0.21677757910180362 |EV| - svm.train.relevancy:
Q1024 Q1024-D987-1 0 1.00000 true White Christmas is an Irving Berlin song reminiscing about an old-fashioned Christmas setting .
The output scores are the distances to the svm decision hyperplane. It is line-by-line aligned with the input file specified by the -answersPath parameter. The answers with the higher scores should be ranked higher wrt the other answers.