WEBVTT Kind: captions; language: en-us NOTE Treffsikkerhet: 81% (HØY) 00:00:00.100 --> 00:00:10.200 And there we are. So we are going to talk about cutoff points quickly and that is what we did 00:00:10.200 --> 00:00:16.100 also last time. But again, I added some slides. So we talked about sensitivity and specificity and 00:00:16.100 --> 00:00:24.600 the discussion was where to cut them off. What is a good cutoff point, 70, 60? It is an arbitrary cutoff point, 00:00:24.600 --> 00:00:31.100 means it's not black and white, you can debate about it. Usually people use 70, little bit 00:00:31.100 --> 00:00:32.849 higher sensitivity than specifity. NOTE Treffsikkerhet: 79% (HØY) 00:00:32.849 --> 00:00:40.300 But again, it depends on what are the characteristics of the disease, 00:00:40.300 --> 00:00:47.800 on resources and all of that. Now this is that we said, okay, this is our problem. So we've got a group 00:00:47.800 --> 00:00:54.100 of healthy people not having a disorder. In this case, a language disorder. And then we've got on 00:00:54.100 --> 00:01:01.800 the other hand, we've got a group with the language disorder. Somewhere on this continuum, we say, NOTE Treffsikkerhet: 78% (HØY) 00:01:01.800 --> 00:01:07.800 somewhere you need to decide, I'm gonna use 00:01:07.800 --> 00:01:15.449 this cutoff, that means if you score lower than this particular point I consider you to be healthy, 00:01:15.449 --> 00:01:22.600 not having a condition. If you are above that score by this side, you are having a condition. 00:01:22.600 --> 00:01:28.800 Now this overlap is the problem, if the two groups were totally separated, life would be fantastic, 00:01:28.800 --> 00:01:31.550 we would not have a problem. But we do have got a problem. NOTE Treffsikkerhet: 91% (HØY) 00:01:31.550 --> 00:01:38.200 This is the problem. So, if that is your cutoff, then these are the false positives, persons without 00:01:38.200 --> 00:01:43.800 the disease and a positive test, but they do not have the disease. The other problem I have false 00:01:43.800 --> 00:01:50.900 negatives persons with the disease, they are actually part of the red distribution, but you now 00:01:50.900 --> 00:01:56.350 consider them as having a negative test. So, where do you cut them off? NOTE Treffsikkerhet: 79% (HØY) 00:01:56.350 --> 00:02:02.700 If you also saw that if you move that cut off, so if this is my first cut off then these are my 00:02:02.700 --> 00:02:13.000 false positives, but if I were to move that line to left, 00:02:13.000 --> 00:02:18.500 then you can see that the number of false positives are changing. So the conclusion is different cutoffs 00:02:18.500 --> 00:02:23.900 will yield different sensitivities and specificities. I hope that is clear. So somewhere 00:02:23.900 --> 00:02:26.399 you need to decide that's what I want as cutoff NOTE Treffsikkerhet: 82% (HØY) 00:02:26.399 --> 00:02:35.100 but you can of course debate that. Now sometimes you say it's more important to have 00:02:35.100 --> 00:02:41.500 a high sensitivity or high specificity. So let's have a look at cut off, what 00:02:41.500 --> 00:02:48.200 happens. And this is all the repeat still. So we said okay you have on the left we've got a low cutoff, 00:02:48.200 --> 00:02:53.200 these are people with the disease and not a disease or condition. Here is a 00:02:53.200 --> 00:02:56.450 high cut off and if you change that cutoff NOTE Treffsikkerhet: 89% (HØY) 00:02:56.450 --> 00:03:02.700 of course, these numbers in the cells change, the cross tabs will look different. Meaning, 00:03:02.700 --> 00:03:11.300 you can see here we've got 17, those are true positives, and here we've 00:03:11.300 --> 00:03:18.300 got only five. So if you move that up, things will change. We saw that. But also different cutoff 00:03:18.300 --> 00:03:25.900 points again yield different sensitivities and specificities. The cutoff determines, how many subjects 00:03:26.550 --> 00:03:31.700 will be considered as having the disease. Because if I move this up and down, you've got different 00:03:31.700 --> 00:03:38.700 people that you say, well, you identify them as a disease. And as you can see this one has got the 00:03:38.700 --> 00:03:44.500 sensitivity and specificity and here it's the other way around. Now, you also see that very likely, 00:03:44.500 --> 00:03:51.200 the true cutoff should be somewhere in the middle because you don't want a specificity that low 00:03:51.200 --> 00:03:56.450 lbut you also don't want to sensitivity that low. So some somewhere in between is NOTE Treffsikkerhet: 84% (HØY) 00:03:56.450 --> 00:04:04.550 probably the better choice. I probably speak too fast. Okay, now, still the same pictures. 00:04:04.550 --> 00:04:12.200 If we talk about a cut-off that identifies more true negatives, you will also have more false 00:04:12.200 --> 00:04:18.200 false negatives. So if you've got true negatives here, they are more than there but 00:04:18.200 --> 00:04:26.500 then you also have got more false negatives. If you've got a cut off of that identifies NOTE Treffsikkerhet: 84% (HØY) 00:04:26.500 --> 00:04:34.450 more true positives, then in this case you will also have more 00:04:34.450 --> 00:04:40.900 false positives on the other side. So that is inherent. If you move one up, the other one goes up as well. 00:04:40.900 --> 00:04:48.700 So cut-offs have on both sides an effect. Okay, this is new. I'm going to give 00:04:48.700 --> 00:04:56.450 you another example. Just to talk about cutoff points. So this is about the E10. NOTE Treffsikkerhet: 73% (MEDIUM) 00:04:56.450 --> 00:05:02.000 The E10 is a bad measure but still we're going to have a look at it because it's an example to just 00:05:02.000 --> 00:05:07.500 that you can see how that work. So the E10 has got poor psychometric properties. So you need to 00:05:07.500 --> 00:05:13.900 throw it away. But many, many speech pathologist use it, but we've got now several papers telling 00:05:13.900 --> 00:05:18.900 don't use this, but still people use it. We will talk about 00:05:18.900 --> 00:05:26.200 psychometrics next lecture, poor psychometrics is a No-No. So let's have a look at cutoff point. 00:05:26.450 --> 00:05:33.500 So the E10 is 10 items. We are talking about swallowing problems and I know it's not swallowing problems is not 00:05:33.500 --> 00:05:38.600 special needs but I had this example and I thought this is probably useful. So just forget about 00:05:38.600 --> 00:05:44.100 swallowing problems, but it is a questionnaire to identify that. Now this one is also a little bit 00:05:44.100 --> 00:05:49.800 problematic because it's got actually Functional Health status and this is actually quality of life. 00:05:49.800 --> 00:05:55.700 So you've got kind of symptoms versus how does it make you feel, the problem if you combine NOTE Treffsikkerhet: 78% (HØY) 00:05:56.400 --> 00:06:03.800 different concepts or domains, then you don't know which one is actually the problem. But okay, the E10, 00:06:03.800 --> 00:06:11.500 this is it. It is a self-report on swallowing problems or dysphasia, you've got 10 items, 00:06:11.500 --> 00:06:20.050 each is being scored on a five-point scale, 0 to 4, which means a total score of 40. 00:06:20.050 --> 00:06:25.900 Okay, and we are now going to have a look at what is the sensitivity and specificity for aspiration. 00:06:26.450 --> 00:06:31.450 Now the gold standard, you know if you got a screen you need to have a gold standard to create the crosstabs. 00:06:31.450 --> 00:06:37.700 The gold standard is passed in in this case PAS (FEES). But I'll show you in a minute what it is, 00:06:37.700 --> 00:06:44.900 but there are different cutoffs and the cut-offs mean the risk of swallowing problems or a risk 00:06:44.900 --> 00:06:51.900 of material going into your lungs... But you get the idea, We've got a questionnaire, we want to know 00:06:51.900 --> 00:06:56.600 what the sensitivity and specificity and there are different cutoffs. NOTE Treffsikkerhet: 91% (HØY) 00:06:56.600 --> 00:07:02.000 Now, just to give you a little bit background information about the gold standard in swallowing, 00:07:02.000 --> 00:07:08.950 this is the gold standard in swallowing. This is a endoscopy. NOTE Treffsikkerhet: 75% (MEDIUM) 00:07:08.950 --> 00:07:22.500 And you can see blue dye going straight into the trachea, into your lungs. That's not okay, that's a problem. 00:07:22.500 --> 00:07:28.600 So this is endoscopy, fiber-optic endoscopic evaluation of swallowing. The other one is 00:07:28.600 --> 00:07:35.700 video fluoroscopic swallowing study and that is the VFSS. And what you can see here is a baby 00:07:35.700 --> 00:07:39.299 taking a bottle. Now, you don't need toknow, it's just for the fun of it. NOTE Treffsikkerhet: 84% (HØY) 00:07:39.299 --> 00:07:45.350 That's the the gold standard. So, with the gold standard, we identify whether there is 00:07:45.350 --> 00:07:51.400 aspiration or not. And aspiration is in anything goes down here to the lungs. I will talk you 00:07:51.400 --> 00:07:58.600 through this table. On the left we've got the total score. So these are all patients or these are 00:07:58.600 --> 00:08:03.400 total scores. And then on the right, you can see the total number of patients having that particular 00:08:03.400 --> 00:08:09.400 score. In the middle we've got two groups my screen identifies as NOTE Treffsikkerhet: 89% (HØY) 00:08:09.400 --> 00:08:15.800 yes or no aspiration. And aspiration means as much as anything goes straight into your lungs. Which 00:08:15.800 --> 00:08:21.800 is not good. Okay. So now let's have a look. So, these are my scores, it is still 40, 00:08:21.800 --> 00:08:27.000 but apparently there were no higher scores. These are, by the way, a real patient data and these are 00:08:27.000 --> 00:08:33.100 the total number of, so, there are seven patients having a, a score of four, all of these patients 00:08:33.100 --> 00:08:37.200 were having swallowing problems. At least they were at risk of swallowing problems. And these are the totals. NOTE Treffsikkerhet: 79% (HØY) 00:08:40.549 --> 00:08:47.300 Now of course you need to know if you use a screen, sometimes you can use the screen for 00:08:47.300 --> 00:08:52.400 swallowing screen for anything going in your lungs. So those are related concept that you will every time see. 00:08:52.400 --> 00:08:57.600 If you change this, what you're looking for, you get of course different numbers. And that is 00:08:57.600 --> 00:09:04.200 also if you work with kids with ADHD, if you want to cutoff like for a certain behavior that is 00:09:04.200 --> 00:09:11.550 present, of course you get different numbers, different scores. But again, we now take aspiration. NOTE Treffsikkerhet: 87% (HØY) 00:09:11.550 --> 00:09:17.550 And this is them how cross tabs should look like, this is my gold standard, my reference test, 00:09:17.550 --> 00:09:25.400 aspirational or no aspiration and this is my cut off. If it would have been 2 or higher or below two. 00:09:25.400 --> 00:09:27.300 This is the other cut off. NOTE Treffsikkerhet: 83% (HØY) 00:09:27.300 --> 00:09:33.700 Now, my question to you... And I'm going to stop recording now. 00:09:42.200 --> 00:10:14.100 There we are. *fixing share screen* 00:10:14.100 --> 00:10:20.800 So this was the question, like, okay, how do we take it from here? Now the first thing you're 00:10:20.800 --> 00:10:24.900 going to do is... Let's focus on the one on the left. NOTE Treffsikkerhet: 86% (HØY) 00:10:24.900 --> 00:10:33.200 Cutoff, as you can see, is 2. So this is cut off 2, so larger 2... Larger means 2 or higher. 00:10:33.200 --> 00:10:39.800 You probably say 2 or higher. So this would be below my cutoff. 00:10:39.800 --> 00:10:46.700 Then if I try to complete cells and this is the most easy bit, these three down 00:10:46.700 --> 00:10:54.900 here, fit exactly there. Because this is the total number of my whole group. NOTE Treffsikkerhet: 84% (HØY) 00:10:54.900 --> 00:11:01.200 And again, this is my gold standard. So, whatever this is the condition is 45 and no condition is 66. 00:11:01.200 --> 00:11:09.500 So just plug that into your your cross tabs. Then the next step would be like, okay, we've 00:11:09.500 --> 00:11:18.100 got nobody who's aspirating if you've got a score below 2, meaning 0 or 1, there is no one 00:11:18.100 --> 00:11:25.750 aspirating there. And we've got 2 green one, 2 true negatives there. NOTE Treffsikkerhet: 88% (HØY) 00:11:25.750 --> 00:11:34.500 Then we've got the true positives. So those are the ones that are aspirating and I count 00:11:34.500 --> 00:11:40.600 all of that and I can calculate that. By the way if I know this is 45 00:11:40.600 --> 00:11:45.700 and that is 0, I also know that this should be 45 but I can also understand and that's 00:11:45.700 --> 00:11:53.100 why you didn't have all the ratings but you knew the 45. So this red cell goes in there. That means 00:11:53.100 --> 00:11:56.350 that this yellow bit is my 64 false positives. Treffsikkerhet: 90% (HØY) 00:11:58.600 --> 00:12:04.300 Once you've got and then actually you can also complete those two, because this is just a 00:12:04.300 --> 00:12:13.600 sum of it. So if I use this as a cut-off, if I've got my cross-tabs, then I can simply use the 00:12:13.600 --> 00:12:22.800 formula to calculate sensitivity and specificity. That's the easiest bit. The tricky bit is get that 00:12:22.800 --> 00:12:24.400 cross tabs done. NOTE Treffsikkerhet: 83% (HØY) 00:12:24.400 --> 00:12:27.099 Is this clear for everybody? NOTE Treffsikkerhet: 82% (HØY) 00:12:27.099 --> 00:12:35.200 Yeah, okay. Now that means if you go to the other one, that's exactly the same. There's no rocket 00:12:35.200 --> 00:12:42.800 science there. So now we've got this was the first one, the first cutoff. 00:12:42.800 --> 00:12:50.050 This is the second one. That is a cutoff of 3 or higher. 00:12:50.050 --> 00:12:56.750 So I replaced that one bit and then that was on the right. NOTE Treffsikkerhet: 76% (HØY) 00:12:56.750 --> 00:13:02.700 These numbers you can determine them exactly the same way as I did for the left one. Yeah, so 00:13:02.700 --> 00:13:07.900 if you understand the first one you can try for the one on the right. But again, you look at how 00:13:07.900 --> 00:13:14.950 many are here. So zero false negatives at this one. I've got four true negatives, that is that bit. 00:13:14.950 --> 00:13:22.100 And you calculate sensitivity and specificity. Now just by the look of it, 00:13:22.100 --> 00:13:26.800 you can see that is bad. That is horrible. NOTE Treffsikkerhet: 85% (HØY) 00:13:26.800 --> 00:13:31.900 Yeah it's sensitive but look at specificity, that's a nightmare. You don't want this one, you throw it away. 00:13:31.900 --> 00:13:39.200 Is it clear to everybody, how to calculate that? If I don't hear 00:13:39.200 --> 00:13:44.800 anything or see I just continue. But this is something that I hope you understand and you 00:13:44.800 --> 00:13:50.500 are able to follow. So let's move on. So again, how to draw them? And we talked about that 00:13:50.500 --> 00:13:57.200 last time, we said, well if it is expensive or invasive, NOTE Treffsikkerhet: 84% (HØY) 00:13:57.200 --> 00:14:03.500 and that would be when you want a cutoff with a high specificity. 00:14:03.500 --> 00:14:11.300 And specificity is how well does the screening test for absence. And if we the penalty for missing a 00:14:11.300 --> 00:14:17.400 case is high, for instance you die or it's contagious, it's pretty bad. Then you want one with a 00:14:17.400 --> 00:14:23.900 high sensitivity and that means sensitivity is how well does the screening test for presence of disease. 00:14:25.200 --> 00:14:33.600 So just keep those two in mind. Now, Roc curve. 00:14:33.600 --> 00:14:40.400 What is the ROC curve? It's short for receiver operating characteristics. 00:14:40.400 --> 00:14:46.700 I didn't make it, that's the term and it is an evaluation classifier performance. 00:14:46.700 --> 00:14:53.200 It's splits between at risk or not at risk. Something like that. So it categorize, it classifies. 00:14:53.200 --> 00:14:55.050 There's always a trade-off between NOTE Treffsikkerhet: 82% (HØY) 00:14:55.050 --> 00:15:01.000 sensitivity and specificity. We saw that. If you've got very high sensitivity your specificity 00:15:01.000 --> 00:15:07.700 maybe a bit lower. If you change the cutoff then you balance them more. That's got to do with 00:15:07.700 --> 00:15:13.400 the ROC curve as well. So that trade-off is visible in this graph. 00:15:13.400 --> 00:15:23.500 We talk about it a bit later. On the y-axis, this is sensitivity. On the x-axis, we've got hundred specificity, NOTE Treffsikkerhet: 81% (HØY) 00:15:25.050 --> 00:15:30.300 it is what it is. Now, how do you get that Roc curve? I'm going to give you a simple 00:15:30.300 --> 00:15:37.700 example from last time. So I've got a threshold, one, two, three, four and suppose these are my 00:15:37.700 --> 00:15:44.900 sensitivity and specificity that I calculated for that particular threshold. Remember, that's 00:15:44.900 --> 00:15:50.800 exactly what we just did in the example. Now, if I've got specificity 00:15:50.800 --> 00:15:54.950 it's easy to calculate hundred minus specificity. NOTE Treffsikkerhet: 85% (HØY) 00:15:54.950 --> 00:16:01.300 Yeah, hundred minus hundred is zero and etc, that's not 00:16:01.300 --> 00:16:11.100 rocket science. Here's my beautiful Roc curve again. Now if I plot 0 0, so here you can zero zero, 00:16:11.100 --> 00:16:22.400 that's the first one. The 50 25, that's that point. The third one is 75 50 00:16:22.400 --> 00:16:25.050 and forth is a 100 and is 0. NOTE Treffsikkerhet: 79% (HØY) 00:16:25.050 --> 00:16:32.550 Then I get this beautiful curve and that's called a Roc curve. Why do you want to 00:16:32.550 --> 00:16:41.300 have a Roc curve? I'll explain a bit more. This is what a Roc curve does. Let's 00:16:41.300 --> 00:16:49.100 have a look at the two extremes, if you find a Roc curve like that for your screen it's useless. 00:16:49.100 --> 00:16:55.250 Throw it away. Do not use it. It's random. It doesn't mean anything. NOTE Treffsikkerhet: 90% (HØY) 00:16:55.250 --> 00:17:03.200 If your Roc curve was that, really getting into the corner, fantastic, looking good. I want that one. 00:17:03.200 --> 00:17:11.900 The bigger the area under the curve, the better my Roc curve is. Here again, this is the 00:17:11.900 --> 00:17:19.400 random classifier, doesn't mean a thing, not helping. But the more that curve goes into this, 00:17:19.400 --> 00:17:25.099 this is perfect, you never have got a perfect one, but the more it gets to that corner, the better it is. NOTE Treffsikkerhet: 91% (HØY) 00:17:25.099 --> 00:17:32.300 And the area under the curve, also short AUC, it's the same thing. 00:17:32.300 --> 00:17:39.400 Just to make life a little bit more complex, sensitivity is also called true 00:17:39.400 --> 00:17:46.600 positive rate and one minus specificity is called false positive rate. So you can see both, it's 00:17:46.600 --> 00:17:53.800 identical. Okay. So this is how you just look at it. Now, if you've got two screens, and these are 00:17:53.800 --> 00:17:55.150 the Roc curves, NOTE Treffsikkerhet: 84% (HØY) 00:17:55.150 --> 00:18:00.900 then I immediately would take the pinky one because that is much more towards the curve. So that's 00:18:00.900 --> 00:18:06.400 how you look at it. You want the better one. This is moderate. That's bit better. Yeah, so 00:18:06.400 --> 00:18:11.650 that's how you should look at. And this is just chance, doesn't mean anything and here you can see 00:18:11.650 --> 00:18:18.900 false positive rate and this is TPF, true positive rate, again it's exactly the same. Now this was 00:18:18.900 --> 00:18:23.900 from last time, we said okay let's have a look at that one. If you look at that, if you see that Roc curve, 00:18:25.250 --> 00:18:33.100 you know random doesn't mean a thing. If you see this one, then you think well it's pretty good, almost in 00:18:33.100 --> 00:18:43.400 the corner. That means that one is a bit poor, it's rather low. And okay, that's moderate, that's 00:18:43.400 --> 00:18:47.200 reasonable. So, that's how you look at these Roc curves. That's just 00:18:47.200 --> 00:18:55.000 interpretation. Now, what's behind all of this is if I've got a random classifier. NOTE Treffsikkerhet: 91% (HØY) 00:18:55.000 --> 00:19:01.900 Meaning that line is just in the middle. Meaning my area under the curve is 0.5 because the total 00:19:01.900 --> 00:19:08.000 area is 1, if the whole thing you call 1, if it's there it's a 0.5, but that means there's 00:19:08.000 --> 00:19:13.100 a total overlap between these two. So you've got two different distributions, condition and not 00:19:13.100 --> 00:19:19.800 condition. But using my screen, it's just totally overlapping. It's absolutely not classifying. So 00:19:19.800 --> 00:19:24.950 you've got one distribution actually instead of two separate. If I go to the moderate one, NOTE Treffsikkerhet: 71% (MEDIUM) 00:19:24.950 --> 00:19:30.400 there is still quite a bit of overlap. And you have got a fantastic one, 00:19:30.400 --> 00:19:37.900 there it is, you can hardly see it but it is in the corner, then it's really distinguishing. 00:19:37.900 --> 00:19:43.300 There is just a very little bit overlap. So that is what behind the Roc curves. NOTE Treffsikkerhet: 81% (HØY) 00:19:43.300 --> 00:19:51.800 Okay, one more time. And you haven't got all these slides yet, I know. So this is about you 00:19:51.800 --> 00:19:57.300 move around your cutoff and that has got different consequences on true negatives, false positives, 00:19:57.300 --> 00:20:05.300 all these things in these cells change. If the number in my cross tab changes, ignore the 0, that 00:20:05.300 --> 00:20:09.800 doesn't mean anything. But if it changes then sensitivity and 00:20:09.800 --> 00:20:13.350 specificity change. I see a question, Melissa. NOTE Treffsikkerhet: 91% (HØY) 00:20:13.350 --> 00:20:19.100 Melissa: You keep saying about moving that little bar. Why would you move the bar? 00:20:19.100 --> 00:20:25.600 Renee: Because you would move it for 00:20:25.600 --> 00:20:31.500 instance to determine What is the best balance between sensitivity and specificity. So what is the 00:20:31.500 --> 00:20:39.400 best cutoff to distinguish between children ADHD or not. To do that you will try different 00:20:39.400 --> 00:20:43.450 cutoffs, you determine sensitivity specificity, actually making a NOTE Treffsikkerhet: 85% (HØY) 00:20:43.450 --> 00:20:49.100 yourself a Roc curve and then you're going to do the balance. You're going to say, well, do I like a 00:20:49.100 --> 00:20:56.000 higher sensitivity or... That's what you're going to do. Melissa: So that's a like based on the 00:20:56.000 --> 00:21:03.200 researchers interpretation. Renee: And I will get back to that on the 00:21:03.200 --> 00:21:09.200 next slide then you can see them. But you do get if I move around my cut off, 00:21:09.200 --> 00:21:13.500 my sensitivity and specificity changes. Now, that means if NOTE Treffsikkerhet: 74% (MEDIUM) 00:21:13.500 --> 00:21:19.400 I've put that in my Roc curve and I move this thing around, 00:21:19.400 --> 00:21:21.750 then this one will move around. NOTE Treffsikkerhet: 88% (HØY) 00:21:21.750 --> 00:21:29.000 Okay, it's the same thing because the pair sensitivity specificity or one minus specific, it changes. 00:21:29.000 --> 00:21:37.900 It's exactly the same. Going to this slide just as an example. So I've got no idea what 00:21:37.900 --> 00:21:43.500 they're doing here but what I do see is they've got different cut-offs, and they distinguish 00:21:43.500 --> 00:21:49.600 between not having the disease or condition versus having the condition. And then for all these four 00:21:49.600 --> 00:21:52.100 point, three, four, Etc. They plot NOTE Treffsikkerhet: 86% (HØY) 00:21:52.100 --> 00:22:00.550 it into a into a Roc curve and then you can decide where do you want that cut off, 00:22:00.550 --> 00:22:07.000 ignore the percentages but this is how you create a Roc curve. Now, to give it one more time. We've 00:22:07.000 --> 00:22:15.100 got a roc curve here. And if you, for instance, look at same Roc curve, this point that is 00:22:15.100 --> 00:22:21.100 we've got the ROC curve. And there you can see how that divides the two groups. If I go to the left 00:22:21.100 --> 00:22:22.050 there you can see what happens. NOTE Treffsikkerhet: 73% (MEDIUM) 00:22:22.050 --> 00:22:33.150 So if I move this this cutoff, I move over the ROC curve. That's what happens. NOTE Treffsikkerhet: 73% (MEDIUM) 00:22:51.850 --> 00:23:00.000 So this is that line at random, that doesn't make any 00:23:00.000 --> 00:23:08.500 sense. And that is the roc curve for a screen X, whatever. And behind is 0.85, which is pretty 00:23:08.500 --> 00:23:16.700 good. Area under the curve of 0.85 means 85% in this square, 85 percent is below that curve. 00:23:16.700 --> 00:23:22.050 The higher that is the better your screen functions, and this is what NOTE Treffsikkerhet: 67% (MEDIUM) 00:23:22.050 --> 00:23:28.600 happens if you move over that roc curve, you can see that this threshold moves around and your 00:23:28.600 --> 00:23:39.100 sensitivity and specificity change. Okay, here we go again. So that is my true positive rate or 00:23:39.100 --> 00:23:44.900 sensitivity, is always the same thing, and there is my false positives. Now this thing is called The 00:23:44.900 --> 00:23:50.500 Roc curve. That is my line of random chance. NOTE Treffsikkerhet: 74% (MEDIUM) 00:23:50.500 --> 00:23:57.750 That is my moderately valid diagnostic test, whatever it is. And then I've got these for 00:23:57.750 --> 00:24:06.500 extremes, these four bullets. Now, what is that? If I take this one I see a sensitivity of 0 0, 00:24:06.500 --> 00:24:15.900 that means sensitivity of 0, specificity of 1 because this is 1 minus specificity and if that should 00:24:15.900 --> 00:24:21.050 be 0, then 1 minus specificity must be one. Is that clear for everybody? NOTE Treffsikkerhet: 81% (HØY) 00:24:21.050 --> 00:24:27.700 So this is 0 0, and that means sensitivity 0, specificity 1. If we go to the next one there, 00:24:27.700 --> 00:24:37.400 I've got a sensitivity of one and specificity is 0, because 1 minus 0 is 1, 00:24:37.400 --> 00:24:44.400 that's how you work with x axis and y axis. Then if I go to 00:24:44.400 --> 00:24:50.650 that one here my sensitivity is 0, NOTE Treffsikkerhet: 79% (HØY) 00:24:50.650 --> 00:24:59.550 specificity is 0. And that is sensitivity is 1 and specificity is 1, fantastic. 00:24:59.550 --> 00:25:06.900 Meaning, this is fantastic. This is just a nightmare, 0-0, can't be any worse. But in real life it's of course 00:25:06.900 --> 00:25:14.700 somewhere in between. But this is how you read the extremes of your Roc curve. Now, which cut off to 00:25:14.700 --> 00:25:20.600 use? Because we know if we change the cut-offs, you move the roc curve, NOTE Treffsikkerhet: 80% (HØY) 00:25:20.600 --> 00:25:27.800 and the sensitivity and specificity, the pairs are slightly changing every time. 00:25:27.800 --> 00:25:33.900 Now, you probably want to be somewhere there. But you need to think about this would be talked about before, 00:25:33.900 --> 00:25:41.800 if sensitivity is presence of disease. So how deadly and contagious is it? Specificity, how 00:25:41.800 --> 00:25:47.850 well does it tests for absence of disease in case of an expensive screen or expensive follow-up 00:25:47.850 --> 00:25:50.550 research, but NOTE Treffsikkerhet: 84% (HØY) 00:25:50.550 --> 00:25:56.650 this will determine where you exactly want to have your cut off. NOTE Treffsikkerhet: 88% (HØY) 00:25:56.650 --> 00:26:00.050 I hope this is a little bit more clear than last time. NOTE Treffsikkerhet: 91% (HØY) 00:26:00.050 --> 00:26:03.100 Any questions so far? NOTE Treffsikkerhet: 91% (HØY) 00:26:07.000 --> 00:26:14.500 I hear the church bells. I'm going to give you a break. I'm gonna stop sharing. 00:26:19.100 --> 00:26:26.300 Okay. So this was actually where we ended up last time. So I would like to continue now with a 00:26:26.300 --> 00:26:33.300 little bit more information because there's more than just sensitivity and specificity. So you will 00:26:33.300 --> 00:26:39.100 also encounter terms like predictive value and you've got positive and negative predictive 00:26:39.100 --> 00:26:45.300 value, and it is about the proportions of positive and negative results in statistics and 00:26:45.300 --> 00:26:50.000 diagnostic tests that are true positive and true negatives. Well now it's NOTE Treffsikkerhet: 89% (HØY) 00:26:50.000 --> 00:26:56.250 easier to see the definitions. Here we are. So we've got positive and negative predictive values. 00:26:56.250 --> 00:27:03.800 You can see it's about proportions. And this is again, the formula. 00:27:03.800 --> 00:27:12.200 So this is just an example of poor predictive value. These are my formulas. And if 00:27:12.200 --> 00:27:19.500 I were to use this cross tabs and I want to determine the positive predictive value, 00:27:20.000 --> 00:27:26.650 then you just complete this one and you can see it's 80, over hundred eighty because of course the 00:27:26.650 --> 00:27:34.400 true positives and false positives, true and false is the same as that cell. So 80 over a hundred 00:27:34.400 --> 00:27:39.800 and eighty. That is my positive predictive value. Meaning, if I look at the negative predictive 00:27:39.800 --> 00:27:48.750 value I've got 800 over 820. You see that is that definition. So it's nothing else but that. 00:27:48.750 --> 00:27:50.050 Now, what does that mean? NOTE Treffsikkerhet: 91% (HØY) 00:27:50.050 --> 00:27:57.200 Means that in a positive test, that is 44 percent chance of actually having the disease. Again, 00:27:57.200 --> 00:28:03.700 we're talking about screening at risk, so it's not for sure that you've got it. But if it's positive, 00:28:03.700 --> 00:28:12.100 there is 44 percent chance that you have the disease. However, if negative test, 98% chance of actually not 00:28:12.100 --> 00:28:20.000 having the disease. So if you get a negative result, it's pretty sure that you don't have it. NOTE Treffsikkerhet: 76% (HØY) 00:28:20.000 --> 00:28:25.200 But if you got a positive result, there's over 50 percent that you still don't have it. So this 40 percent 00:28:25.200 --> 00:28:31.800 chance of having it meaning 56 percent chance of not having it. So this is very nice. That is a 00:28:31.800 --> 00:28:36.000 little bit of problem, 44. Okay, predictive value. NOTE Treffsikkerhet: 87% (HØY) 00:28:36.000 --> 00:28:45.900 Now predictive value and sensitivity, if the look at sensitivity and if we look at predictive 00:28:45.900 --> 00:28:51.800 positive predictive values because they seem to be same but they're not. So sensitivity is the 00:28:51.800 --> 00:28:59.600 true positives divided by that sum, whereas predictive value is true positives but 00:28:59.600 --> 00:29:04.950 then divided over that sum. So it is slightly different and if you look at the definitions they are 00:29:04.950 --> 00:29:05.850 kind of similar, NOTE Treffsikkerhet: 82% (HØY) 00:29:05.850 --> 00:29:17.700 but not the same. So why would we need both? Because sensitivity is about the proportion of 00:29:17.700 --> 00:29:23.600 reference test positive, subjects who test positives and that with the screen, and that is important 00:29:23.600 --> 00:29:30.000 for the physician, for the clinician, the one who's performing the test. However, for the 00:29:30.000 --> 00:29:35.900 patient, it's more important. Well, should a panic or not? Do I have it or not? Positive predictive NOTE Treffsikkerhet: 84% (HØY) 00:29:35.900 --> 00:29:41.400 value is the proportion of positive results in statistics and diagnostic tests that are true positive 00:29:41.400 --> 00:29:46.600 results. So that's for me as a patient, I want to know if I have got the condition or not, that's the 00:29:46.600 --> 00:29:51.450 difference between sensitivity and positive predictive value. NOTE Treffsikkerhet: 91% (HØY) 00:29:51.450 --> 00:29:58.500 Now, there's also such a thing as likelihood ratios, and I know you need to read it again and think 00:29:58.500 --> 00:30:04.700 about what it is and what it means, but just hang in there. So we've got positive and we've got 00:30:04.700 --> 00:30:10.200 negative likelihood ratios. And if you look at the definition, positive is the probability of a 00:30:10.200 --> 00:30:15.400 person who has the disease testing positive divided by the probability of a person who does not have 00:30:15.400 --> 00:30:21.750 the disease Etc. Again, you can see here how that looks like in formula, NOTE Treffsikkerhet: 76% (HØY) 00:30:21.750 --> 00:30:28.300 the positive one and how the negative one likelihood ratio is formulated. NOTE Treffsikkerhet: 85% (HØY) 00:30:28.300 --> 00:30:37.500 Now, what does it tell me? Likelihood ratio, LR tells me how likely a patient has a disease or 00:30:37.500 --> 00:30:44.100 condition. The higher the ratio the more likely they have the disease or condition. A low ratio 00:30:44.100 --> 00:30:51.300 means they very likely do not have the disease. And likelihood ratios can range from 00:30:51.300 --> 00:30:57.800 0 to endless. The higher the value though, the more likely that you've got the diesase. 00:30:58.699 --> 00:31:07.400 Rule of thumb, how to interpret a likelihood ratio, 0 to 1 means decreased evidence 00:31:07.400 --> 00:31:16.000 for disease. Values closer to zero have higher degrees and probability of disease. One is no 00:31:16.000 --> 00:31:22.300 diagnostic value and above one is an increased evidence of the disease, the farther away from one 00:31:22.300 --> 00:31:28.650 the more chance of having that disease. So you use cutoff. NOTE Treffsikkerhet: 74% (MEDIUM) 00:31:28.650 --> 00:31:37.700 Below one decrase, above one increased evidence. Now let's have a look at an example. So, if I tell you a 00:31:37.700 --> 00:31:45.500 positive likelihood ratio of 9.2, that means this result, whatever it is, is nine point two times 00:31:45.500 --> 00:31:55.500 more likely to happen. A negative one, if it is a 0.1, then this result is ten times 00:31:55.500 --> 00:31:58.699 less likely to happen in a NOTE Treffsikkerhet: 88% (HØY) 00:31:58.699 --> 00:32:04.300 condition than it would in a patient without the condition. Long sentences. Yeah, this is likelihood ratios, 00:32:04.300 --> 00:32:07.500 we talk about probabilities. NOTE Treffsikkerhet: 73% (MEDIUM) 00:32:07.600 --> 00:32:16.699 Now if we're trying to do this qualitatively, then relatively high likelihood ratio of 10 or greater 00:32:16.699 --> 00:32:21.800 will result in large and significant increase of the probability of the disease given a positive 00:32:21.800 --> 00:32:29.000 test. Then we've got a likelihood of five to lower, Etc. So, this is how to kind of get a 00:32:29.000 --> 00:32:36.100 qualitative sense of what on Earth, does a likelihood raio mean. Well, if you've got a likelihood 00:32:36.100 --> 00:32:38.300 ratio of about one that NOTE Treffsikkerhet: 87% (HØY) 00:32:38.300 --> 00:32:43.300 is it's not capable of changing the process, probably either up or down instead of tests, 00:32:43.300 --> 00:32:51.050 it's nothing, it doesn't give you any further detail. But if it would be low, close to zero, 00:32:51.050 --> 00:33:00.300 significantly decrease the probability of the disease, Etc. Okay, hang in there. Some rough 00:33:00.300 --> 00:33:06.400 estimates of likelihood ratios. This is what you can do. So, if you've got likelihood ratio of 00:33:07.900 --> 00:33:14.050 increasing probability of disease and presence of a positive test, positive likelihood ratio. 00:33:14.050 --> 00:33:25.699 2, probability by 15%. 5 by 30%. 10 by 45 percent. Increase in probability of the disease 00:33:25.699 --> 00:33:32.500 when you've got this positive test. Yes, so the higher the more likely you've got that 00:33:32.500 --> 00:33:38.300 disease. If I look at negative likelihood ratio, that is a decreased NOTE Treffsikkerhet: 76% (HØY) 00:33:38.300 --> 00:33:45.400 probability of a disease in presence of a negative test. 00:33:45.400 --> 00:33:54.500 0.5 is decreased probability of the Disease by 15. 0.2 by 30. 0.1 by 45. These are only rough 00:33:54.500 --> 00:34:03.200 estimates to give you kind of a qualitative idea of what do these numbers mean. So estimates do 00:34:03.200 --> 00:34:08.250 not work very well in very low or very high probability, then it's not really useful NOTE Treffsikkerhet: 82% (HØY) 00:34:08.250 --> 00:34:16.000 to have these likelihood ratios anymore. I give you an example. I've got a test, a water 00:34:16.000 --> 00:34:22.100 swallow test. We've got it and 00:34:22.100 --> 00:34:29.199 we want to interpret the data that I'll give you. So this again, we've got my famous VFS, 00:34:29.199 --> 00:34:35.800 you've seen that before. We're looking at whether anything goes into the lung and I use a 00:34:35.800 --> 00:34:38.150 TOR-BSST. Now, TOR-BSST is NOTE Treffsikkerhet: 73% (MEDIUM) 00:34:38.150 --> 00:34:42.800 a water swallow test, first you check whether there are any abnormal obvious abnormalities. Whether 00:34:42.800 --> 00:34:47.699 someone is alert or not, then you give him some spoons of water and if someone coughs or anything 00:34:47.699 --> 00:34:53.900 like it, that means risk if swallowing problems because you don't cough if I give you a spoon 00:34:53.900 --> 00:34:59.900 of water. Okay, that's the TOR-BSST, doesn't matter. But this is my cross tabs that I get out of it. 00:34:59.900 --> 00:35:05.900 So, I've got a number of patients. This is my gold standard. This is the condition, yes or no 00:35:05.900 --> 00:35:08.250 swallowing problems, and this is my water swallow. NOTE Treffsikkerhet: 74% (MEDIUM) 00:35:08.250 --> 00:35:15.450 This is the results of if I were to calculate all my diagnostic performance using the 00:35:15.450 --> 00:35:21.900 formulas from the previous slides. Now what does that mean? And that's what we're going to talk 00:35:21.900 --> 00:35:29.450 about now. There is again my crosstabs and these are my 00:35:29.450 --> 00:35:37.400 definitions and the formulas Etc. So we do know how to calculate it. But what does it mean? 00:35:37.400 --> 00:35:38.250 So if we've got a sensitivity NOTE Treffsikkerhet: 82% (HØY) 00:35:38.250 --> 00:35:46.900 of 80% and specificity of 68 percent. That means 80% of reference test positive disease 00:35:46.900 --> 00:35:53.100 subjects who test positive or 80% of all these subjects are identified by screening and you miss 00:35:53.100 --> 00:36:01.600 20%. But 80% is pretty good, t identifies 80% of everybody who has the disease. That's not too bad. 00:36:01.600 --> 00:36:08.149 Specificity of 68%, that is 68% of healthy subjects are NOTE Treffsikkerhet: 85% (HØY) 00:36:08.149 --> 00:36:18.100 identified as being healthy, but 38 are miss identified. You identified as having the disease, 00:36:18.100 --> 00:36:22.100 one moment, you can have a look at the slide. 00:36:34.200 --> 00:36:43.800 Okay, so I hope that was clear. Now the next data we had had to do with predictive values. 00:36:43.800 --> 00:36:48.500 To make life a little bit more easy, we've got here the definitions, we've got the formulas. 00:36:48.500 --> 00:36:54.800 If this is the output and this is based on that, so if you put all these numbers in you 00:36:54.800 --> 00:37:02.000 should get these. Now, these are 50% and 89.5%. Now, what does that mean? 00:37:02.000 --> 00:37:03.300 A positive predictive 50% NOTE Treffsikkerhet: 91% (HØY) 00:37:03.300 --> 00:37:13.000 means if you've got a positive test, 50% chance of actually having the disease. Negative 00:37:13.000 --> 00:37:18.800 predictive value, if you've got the negative one, you've got 89.5% chance of 00:37:18.800 --> 00:37:24.800 actually not having the disease. So this is pretty good. But if you've got a positive chance, then 00:37:24.800 --> 00:37:32.800 about 50%, they don't have actually swallowing problems. That's how you read it. Hang in. 00:37:32.800 --> 00:37:33.450 Likelihood ratios. NOTE Treffsikkerhet: 82% (HØY) 00:37:33.450 --> 00:37:41.800 We had a likelihood ratios of 2.5 and 0.3 negative likelihood ratios, again definitions Etc. 00:37:41.800 --> 00:37:50.500 But what on Earth does it mean? Now we said, okay you can use this to interpret. 00:37:50.500 --> 00:37:56.300 So if I look at the 2.5 that is about here, likelihood ratios, 00:37:56.300 --> 00:38:02.950 decreased probability, is small amount. 0.3 , that is actually it will NOTE Treffsikkerhet: 89% (HØY) 00:38:02.950 --> 00:38:09.300 probably significantly decrease the probability of a disease given a negative test. Again these 00:38:09.300 --> 00:38:14.800 numbers, try to fit them in a little bit and try to interact, that's how you can interpret them. NOTE Treffsikkerhet: 78% (HØY) 00:38:14.800 --> 00:38:22.000 Now, this was pretty easy, prevalence, sensitivity, and specificity. We talked about predictive 00:38:22.000 --> 00:38:32.000 values and likelihood ratios. And that is, as far as I go with diagnostic performance. Except for, 00:38:32.000 --> 00:38:39.500 I want to talk about error, accuracy and precision just briefly, but we've got plenty of time. 00:38:39.500 --> 00:38:45.100 I give you the formula. So formulas, everything is based NOTE Treffsikkerhet: 85% (HØY) 00:38:45.100 --> 00:38:53.600 on the cross tabs, so if this is the formula, this is how it looks like. So we've got the false 00:38:53.600 --> 00:39:00.600 positives and false negatives. So any observations that are misclassified, the false ones, and you 00:39:00.600 --> 00:39:07.100 divide them by the total, that gives you a ratio and you call that error, which makes sense. All the 00:39:07.100 --> 00:39:13.450 ones that are in the wrong cells. Remember green is good and red is bad, so that the red ones, 00:39:13.450 --> 00:39:14.950 that's the problem. But you want to know the percentage, NOTE Treffsikkerhet: 88% (HØY) 00:39:14.950 --> 00:39:19.800 so you divide it by the total sum, that is error. NOTE Treffsikkerhet: 91% (HØY) 00:39:19.900 --> 00:39:27.000 Accuracy is a little bit different but it makes also I think more sense. 00:39:27.000 --> 00:39:33.700 How precise is it? What are the correctly identify one? So now we're looking at the green ones and we 00:39:33.700 --> 00:39:41.300 divide those by the total population. So we had error, that is the the misdiagnosed ones. 00:39:41.300 --> 00:39:49.250 The green ones are the correctly classified ones. And that is actually the same as 1 minus error. NOTE Treffsikkerhet: 91% (HØY) 00:39:49.250 --> 00:39:58.000 It's the same. Okay, then the third one and the last one is presition. The proportion of true 00:39:58.000 --> 00:40:05.600 positives to all positive predictions. And that is if you look at it, the true ones compared to all 00:40:05.600 --> 00:40:12.400 positive predictions and that is how precise is it. That's how you call that. And that is the same 00:40:12.400 --> 00:40:19.100 as positive predictive value. So, positive predictive value is precision. If you look at 00:40:19.100 --> 00:40:19.600 this one, NOTE Treffsikkerhet: 78% (HØY) 00:40:19.600 --> 00:40:25.700 error, accuracy and precisition, you can see that is the same definition as my positive predictive 00:40:25.700 --> 00:40:32.100 value. And again remember that accuracy is just 1 minus error. So if you've got error or accuracy, 00:40:32.100 --> 00:40:41.900 you can calculate the other one. Here is again a test I got, the same one. I'm doing the 00:40:41.900 --> 00:40:47.400 dsyphagia and no dsyphagia one, swallowing problems and the screen. And I want to determine error, 00:40:47.400 --> 00:40:49.850 accuracy and precision. NOTE Treffsikkerhet: 87% (HØY) 00:40:49.850 --> 00:41:02.550 Now if I use my formulas, then I just fill them in. You just hang in. 00:41:02.550 --> 00:41:11.100 Then in the end, you get error 30%, or 29, accuracy 71 and precision is 50. I thought I could ask 00:41:11.100 --> 00:41:17.300 you to calculate... But is this actually clear? Would you have been able to calculate this? NOTE Treffsikkerhet: 91% (HØY) 00:41:18.100 --> 00:41:20.800 Anybody? NOTE Treffsikkerhet: 91% (HØY) 00:41:21.000 --> 00:41:26.000 Melissa: Maybe after a couple of tries. 00:41:26.000 --> 00:41:29.900 Renee: Okay, how about...