Learning from Comparisons

We consider learning from comparisons, i.e., learning from a dataset containing pairs of samples and labels indicating their relative order. For example, a medical expert presented with a pair of patient files can order the pair w.r.t. the relative severity of a disease. Comparisons are often less noisy than class labels: human labelers disagreeing when generating class judgments often exhibit reduced variability when asked to compare pairs of samples instead. Comparisons are also more informative,  as they capture both inter and intra-class relationships; the latter are not revealed via class labels alone. We discuss inference algorithms in this setting, as well as means to identify which comparison labels to collect.