How kappa is calculated
Kappa compares the observed agreement p_o with the agreement expected by chance p_e, given how often each rater uses each category: κ = (p_o − p_e) / (1 − p_e) (Cohen, 1960). Weighted kappa gives partial credit to near misses on an ordered scale. Linear weights are 1 − |i − j|/(k − 1) and quadratic weights 1 − (i − j)²/(k − 1)² (Cohen, 1968). The confidence interval uses the large-sample standard error of Fleiss, Cohen and Everitt (1969), and the z test uses their standard error under the hypothesis that κ = 0.
Kappa for wearable sleep staging
Put one 30-second epoch per row with the device's stage and the polysomnography stage, and set the category order (for example W, N1, N2, N3, R). Sleep stages aren't a simple ordered scale, since REM doesn't sit between light and deep sleep, so unweighted kappa is the safer default. Use weights only with an order you can defend. Report the confusion matrix too: per-stage sensitivity shows where a device confuses light and deep sleep. Public datasets for this are listed in the wearable sleep-staging datasets guide, and published device results are on the sleep accuracy page.
Interpreting kappa
Landis and Koch (1977) proposed bands: below 0 poor, 0–0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial and 0.81–1 almost perfect. They described the cut-offs as arbitrary, so report the value with its confidence interval and the confusion matrix rather than the label alone.
Validated against R and Python
Unweighted and weighted kappa, z statistics and variances match R's irr::kappa2 and psych::cohen.kappa on the Fleiss (1971) diagnosis data and the vision-grade data distributed with irr. Kappa matches scikit-learn's cohen_kappa_score for unweighted, linear and quadratic weights.