Abstract:
Quadratic Weighted Kappa (QWK) is a statistic
to measure the agreement between two annotators. QWK has
been widely used as the evaluation measure for various medical
imaging problems, where, the class labels have a natural ordering,
e.g., no Diabetic Retinopathy (DR), mild DR, and severe DR. The
easiest way to treat the classification problem with these ordinal
labels is to consider the problem as a multiclass classification
problem and apply the Cross Entropy (CE) loss. However, when
applying CE loss the order of the classes becomes meaningless,
i.e., the loss will be same if a healthy image is classified into
mild DR or severe DR. At the same time, the QWK score will
be severely affected if a healthy image is classified into severe
DR than mild DR. The most appropriate way to get a better
classification score is to optimize the evaluation measure itself. i.e.,
directly optimize the QWK statistics. However, this optimization
may hinder the learning, and may lead to sub-optimal solutions,
and therefore, may give lower performance than expected. On
the other hand, Ordinal Regression (OR) based approaches also
can be used for such problems. The main focus of this work is to
investigate which loss function (CE loss, QWK loss or OR loss) is
the most appropriate one to the Convolutional Neural Networkbased
ordinal classification problems, where, QWK is used as
the evaluation measure. Experiments on two public datasets,
Diabetic Retinopathy and Prostate Cancer, with two different
network architectures suggest that directly optimizing QWK is
the better choice when small networks are used. On the other
hand, we found that for large networks OR based loss function
gives better performance.
