Noun

Nominal dichotomous yes/no data: Krippendorff alpha inter-rater reliability

• Report rights infringement
• published: 03 Mar 2017
• views: 865

How to use a statistical test (Krippendorff alpha) to check the reliability of a variable with nominal/dichotomous data. (Windows PC & SPSS.) Reference: Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures 1(1), 77-89. Step 1. Search for and download the Kalpha macro. I use the keywords: download, macro, kalpha, spss to find it on the internet. Sometimes you will find the file, which is named “kalpha.sps”. And sometimes you will find a compressed, zipped file called “kalpha.zip”. Either way, just download this to your own computer. Step 2. If it is the uncompressed file “kalpha.sps”, you just leave it as it is. If it is the zip file, which is a compressed folder, you must extract it. Right-click on it, and extract. Remember where you save it because you need it later. Step 3. Start SPSS and open a blank, empty page. Step 4. File, Open, Syntax Step 5. Search your computer, select and open the file “kalpha.sps”. The macro. As you can see, it is just a script. Do not change anything. Just leave it. Step 6. Execute this macro by Run, All. The window that pops up is normal. Just a confirmation that all is OK. You can close it and the script, if you want. Step 7. Open your data. Make sure that you data is correct. Numeric type and nominal level of measure. Step 8. It is time to run the statistical test. Normally, you select a test from a menu. But, since this test is not include inside SPSS, we need to execute it ourselves. File, New, Syntax and type: kalpha judges = obs1 obs2 obs3 obs4 obs5/level = 1/detail = 0/boot = 10000 I will explain what it means, later. Then: Run, All This is how I would write my report: Method The Krippendorff’s alpha test was used (Hayes & Krippendorff, 2007) to estimate the inter-coder reliability, and these alpha (α) values are reported in the results below. Results The results show that the inter-coder reliability was relatively high (α = 0. 7831), i.e. that the five coders did agree. Discussion (Here, you interpret and discuss plausible reasons why you think that the coders did (or did not) agree. It is beyond the scope of this example, where I only focus on the statistical test.) References Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures 1(1), 77-89.