Interface to Custom Problem Selectors

As described in Selection Algorithms, TutorShop supports three types of custom problem selectors. For all three types the requests to the selection program and the expected responses are JSON objects with the following top level keys:

Request:

• skills: a list of objects representing the current state for knowledge components associated with the student and tracked by the problem set; the most important property of each skill object is p_known, the most-recently-calculated probability that the student has mastered the skill;
• variables: a list of objects representing extended student model variables associated with the student, as the values most-recently-generated by detectors running in the tutors;
• dynamic_model: an object, initially empty, representing a student model extension, which can be used by the custom selector to store key/value pairs between runs;
• problem_set: an object representing the current problem set that a problem selection is requested for;
• problems: a list of problem objects representing selection candidates;
• problem_skills: a list of objects representing the skills associated with each problem in the problems array;
• condition: a string representing the study condition associated with the assignment that runs the problem set, if any, or an empty string if there is no study condition associated with the assignment;
• course: an object representing the course that the problem set is part of.

Response:

• problem_name: a string specifying the selected problem name; if the returned string is missing or empty, TutorShop will take it to mean that the custom selector has no problem to select, and it will end the problem set;
• dynamic_model: an object representing the updated student model extension, which will be passed back to the custom selector on the next call.

A sample request and response are shown below. First, the JSON request from TutorShop to the custom problem selector:

{
  "skills":[
    {
      "skill_id":414,"p_initial":1.0,"p_known":1.0,"initial_opportunity_count":3,"opportunity_count":12,"history":3555,
      "user":{
        "school_id":14,"standard_id":101,"user_role_id":8,"first_name":"student","last_name":"0","middle_name":"",
        "username":"stu0","email_address":"","date_of_birth":"1998-08-23","status":"Active"
      }
    },
    {
      "skill_id":412,"p_initial":0.93,"p_known":0.93,"initial_opportunity_count":1,"opportunity_count":4,"history":13,
      "user":{
        "school_id":14,"standard_id":101,"user_role_id":8,"first_name":"student","last_name":"0","middle_name":"",
        "username":"stu0","email_address":"","date_of_birth":"1998-08-23","status":"Active"
      }
    }
  ],
  "variables":[],
  "dynamic_model":{"+=":0.01,"for":0.02},
  "problem_set": {
    "package_id":264,"name":"For Loop Test3","label":"For Loop Test3","description":"For Loop Test3","picture":null,
    "selection_algorithm":"remote_selection","selection_command":"http://pawscomp2.sis.pitt.edu/bn/GetNextProblem",
    "max_count":1000,"max_repeat":1,"initial_sequence":0,"team_size":1,"status":"Active"
  },
  "problems":[
    {
      "package_id":264,"name":"foraddasm_s1","label":"foraddasm_s1","description":"For with Add Assignment",
      "problem_file":"foraddasm_itgt_s1.brd","student_interface":"foraddasm_s1.html","student_interface_class":null,
      "interface_type":"html","tutor_flag":"tutor","selection_features":"","completed_count":0
    },
    {
      "package_id":264,"name":"foraddasm_s2","label":"foraddasm_s2","description":"For with Add Assignment",
      "problem_file":"foraddasm_itgt_s2.brd","student_interface":"foraddasm_s2.html","student_interface_class":null,
      "interface_type":"html","tutor_flag":"tutor","selection_features":"","completed_count":0
    },
    {
      "package_id":264,"name":"foraddasm_s3","label":"foraddasm_s3","description":"foraddasm_s3",
      "problem_file":"foraddasm_itgt_s3.brd","student_interface":"foraddasm_s3.html","student_interface_class":null,
      "interface_type":"html","tutor_flag":"tutor","selection_features":"","completed_count":0
    }
  ],
  "condition":""
}

And second the expected response from the custom problem selector to the TutorShop. It includes (a) what problem to show the student next and (b) what dynamic model data to save and return in the next request to the custom problem selector:

{
  "problem_name":"foraddasm_s2",
  "dynamic_model":{
    "+=":0.02,"for":0.03
  }
}

Request and response details

Skills array

A skill, also called a knowledge component, represents the student's mastery of a concept or procedure in the problem. Skills are typically assigned to individual problem steps; a step can be associated with any number of different skills, and a skill can be associated with any number of steps, across problems, problem sets and packages. The mastery is expressed as a probability.

The array includes an element for each skill whose value might have been recalculated in the most recent problem set. Each array element includes a skill identifier and a number p_known, the most-recently-calculated probability that the student has mastered the skill. The initial_probability property is the value of that probability at the beginning of the problem set.

The history property's value is an integer which encodes a bit vector recording the student's performance on that skill. The bit vector is revised only on a student's first attempt on a new step associated with that skill; second and later attempts on the step will not change the bit vector. A 1 bit records a correct answer, a 0 bit records a hint request or incorrect answer. The latest result is in the vector's least significant bit (LSB). Each time the tutor revises the history, it shifts prior results leftward by 1 bit position. The opportunity_count property is incremented once each time a new bit is appended to the vector: hence the difference between the count at the end of a problem and the count at the end of the prior problem is the number of updates in the most recent problem. (N.B.: the initial_opportunity_count property is different; it changes only at the start of a problem set, not on each problem).

For example, using the 2nd skill in the sample request above,

{ "skill_id":412,"p_initial":0.93,"p_known":0.93,"initial_opportunity_count":1,"opportunity_count":4,"history":13, ...}

the history value, decimal 13, is 1101 in binary, meaning that among the student's last 4 opportunities at any step to learn or demonstrate skill 412, the first attempts at that step were scored as follows (right-to-left--that is, latest listed first):

bit position	opportunity at step using this skill	bit value	score
LSB	most recent opportunity	1	correct on 1st attempt at step
LSB << 1	next most recent opportunity	0	incorrect or hint on 1st attempt
LSB << 2	2 opportunities ago	1	correct on 1st attempt
LSB << 3	3 opportunities ago	1	correct on 1st attempt