WEBVTT Kind: captions; language: en-us 00:00:00.000 --> 00:00:04.100 Okay, here we are. NOTE Treffsikkerhet: 80% (HØY) 00:00:04.300 --> 00:00:11.500 Okay, I'm going to give you examples. This one for example we're going to do one by one. 00:00:11.500 --> 00:00:17.300 So let's have a look. Correlations between data on two different social inclusion measures in 00:00:17.300 --> 00:00:24.200 School-aged children. I'll give you the answer to not to make every time the same people talk. But if 00:00:24.200 --> 00:00:31.200 you see, correlations, and you've got measures of the same construct, we are talking about 00:00:31.200 --> 00:00:35.000 convergent validity and that is hypothesis testing. NOTE Treffsikkerhet: 90% (HØY) 00:00:35.000 --> 00:00:41.100 You need to interrupt me if you're lost. So whatever the topic is, it's the same topic that 00:00:41.100 --> 00:00:46.400 two measures are measuring and that would be convergent validity. But the name of the psychometric 00:00:46.400 --> 00:00:55.000 property is hypothesis testing. I've got another one, repeated measurements within one week in a 00:00:55.000 --> 00:01:01.800 group of adolescence with chronic voice problems. Now have a look, what should immediately ring a bell 00:01:01.800 --> 00:01:04.900 is repeated measurements, 00:01:04.900 --> 00:01:12.500 that is reliability and it's the bit test-retest. So you repeat within one week. Remember if it's too 00:01:12.500 --> 00:01:17.900 short, you've got your memory that you can use. If it's too late, after two weeks usually, then you 00:01:17.900 --> 00:01:25.300 can have actually learning effects Etc. But this is just a repeat to check how my measure is doing. 00:01:25.300 --> 00:01:32.000 We are talking about test-retest. I expected the group of adolescence have not changed and that 00:01:32.000 --> 00:01:34.750 was also confirmed by chronical problems. NOTE Treffsikkerhet: 89% (HØY) 00:01:34.750 --> 00:01:43.900 So, the group is stable. I do two measurements in a stable group. Meaning I expect the data to 00:01:43.900 --> 00:01:53.400 correlate very high and that is test-retest reliability. Another one, factor analysis on raw 00:01:53.400 --> 00:02:02.000 self-report data. The word that needs to trigger is factor analysis, because factor analysis is 00:02:02.000 --> 00:02:04.550 being used to identify different NOTE Treffsikkerhet: 87% (HØY) 00:02:04.550 --> 00:02:12.300 factors, different sub scores. So you want to know how is the structure of that measure, were talking 00:02:12.300 --> 00:02:15.300 about structural validity. NOTE Treffsikkerhet: 84% (HØY) 00:02:15.700 --> 00:02:25.400 Melissa: I'm sorry. Quick question about the hypothesis testing. You said you want to 00:02:25.400 --> 00:02:34.000 measure it how A is correlated to B For example, is that the same as the internal 00:02:34.000 --> 00:02:39.600 consistency when you're measuring? I feel like there is an overlap or maybe I 00:02:39.600 --> 00:02:44.550 misunderstood. Renee: I understand- So hypothesis testing if I NOTE Treffsikkerhet: 86% (HØY) 00:02:44.550 --> 00:02:50.600 want to use two different measures and I want to compare two different measures. So you've 00:02:50.600 --> 00:02:56.700 developed a measure and so did I and they both are measuring the same concept and I'm interested how do 00:02:56.700 --> 00:03:06.100 they correlate. Internal consistency is I've got one measure and 00:03:06.100 --> 00:03:13.250 I want to know how do the items correlate within the measure. That is internal consistency. NOTE Treffsikkerhet: 87% (HØY) 00:03:13.250 --> 00:03:21.900 Okay, so convergent validity or hypothesis testing, I have different measures. Internal consistency 00:03:21.900 --> 00:03:27.700 is related to that one measure I have got. It could be there are sub skills in that measure, but we're 00:03:27.700 --> 00:03:38.400 talking about one single measure. And that brings me to cronbach's alpha 00:03:38.400 --> 00:03:42.649 for items of a functional health status questionnaire. NOTE Treffsikkerhet: 91% (HØY) 00:03:42.649 --> 00:03:50.500 What should trigger you now is cronbach's alpha, because that was a measure of inter correlatedness 00:03:50.500 --> 00:04:00.200 between the items and that is internal consistency. Okay, then I've got this one. Someone is 00:04:00.200 --> 00:04:09.500 describing in his article how he/she developed a new questionnaire. Bingo, that is content validity. 00:04:09.500 --> 00:04:13.100 The person could say well we did a delfi study, NOTE Treffsikkerhet: 91% (HØY) 00:04:13.100 --> 00:04:21.000 we invited external experts, they gave us the items, we ask patients for feedback... 00:04:21.000 --> 00:04:31.450 That is all development of a measure. You call that content validity. Okay, then this one, 00:04:31.450 --> 00:04:38.500 associations. It's another word for correlations, between the original and the shortened form or 00:04:38.500 --> 00:04:43.050 version of early literacy measure. I hope you remember that NOTE Treffsikkerhet: 84% (HØY) 00:04:43.050 --> 00:04:50.150 if you compare the original measure with the shortened version of the original measure in that case, 00:04:50.150 --> 00:04:54.100 that's a special case of Criterion validity. NOTE Treffsikkerhet: 91% (HØY) 00:04:54.500 --> 00:05:03.800 I think this is almost the last one, data on pre and post play intervention in shy kids. What needs 00:05:03.800 --> 00:05:12.000 to trigger you is pre and post. We are talking responsiveness. Yep. So, you use it to measure an 00:05:12.000 --> 00:05:19.100 intervention output before and after the intervention, that means we're looking at responsiveness. 00:05:19.100 --> 00:05:21.800 And then this one, NOTE Treffsikkerhet: 91% (HØY) 00:05:21.800 --> 00:05:30.000 you translate something in Norwegian... Bingo! Cross-cultural validity. So if we look at the whole, 00:05:30.000 --> 00:05:36.950 this is the same slide again, correlations between different measures but measuring the same concept is 00:05:36.950 --> 00:05:43.300 hypothesis testing. Repeated measurements within a week, group has not changed, reliability test-retest. 00:05:43.300 --> 00:05:49.900 Factor analysis, bingo, that is structural validity. How many groups, how many factors are 00:05:49.900 --> 00:05:54.300 there with subscales within your scale, within your measure? NOTE Treffsikkerhet: 86% (HØY) 00:05:54.300 --> 00:05:59.100 Cronbach's Alpha is used for internal consistency. You describe how you 00:05:59.100 --> 00:06:05.500 developed your measure. that's content validity. Association, correlation between original and 00:06:05.500 --> 00:06:13.400 shortened versions is a particular form is Criterion validity. Pre-post bloody blah, they talking 00:06:13.400 --> 00:06:22.150 responsiveness. And the last one, you see a translation going on, cross-cultural validity. And that is the framework. 00:06:22.150 --> 00:06:28.100 Here, we've got the whole framework. Now, I'm going to stop for now. 00:06:33.400 --> 00:06:42.750 Okay, and we are back. So we've seen this one by now, the nine psychometric properties, the three domains. 00:06:42.750 --> 00:06:49.100 And if you talk about a measure that is valid, reliable and responsive, you are talking about all 00:06:49.100 --> 00:06:54.500 nine segment properties if relevant. For instance, if the measure is never translated, of course 00:06:54.500 --> 00:07:01.300 the cross-cultural validity may not be interesting or if there's not a gold standard and there's not 00:07:01.300 --> 00:07:02.900 a shorter version, then NOTE Treffsikkerhet: 83% (HØY) 00:07:02.900 --> 00:07:08.200 criterion validity is out. But otherwise all psychometric properties should be taken into 00:07:08.200 --> 00:07:14.500 consideration. So when you select a measure, you need to have a look at all measure, all 00:07:14.500 --> 00:07:21.700 psychometrics. It's not that if one segment property is poor, you can't compensate that. If content 00:07:21.700 --> 00:07:28.100 validity is poor, there's no way compensating for that. Or if measure is not reliable, then that's 00:07:28.100 --> 00:07:32.850 the fact and then you've got a measure that's unreliable, So you may not want that measure. NOTE Treffsikkerhet: 85% (HØY) 00:07:32.850 --> 00:07:40.400 In general, do not use measures with proven poor psychometric properties, because you 00:07:40.400 --> 00:07:48.200 don't know how to interpret your outcome data. 00:07:48.800 --> 00:08:00.000 If you've got a measure with known psychometric properties that is the best one. That is the one you want, or maybe 00:08:00.000 --> 00:08:02.850 with unknown, but the worst thing you can do is NOTE Treffsikkerhet: 89% (HØY) 00:08:02.850 --> 00:08:08.300 you know it's a bad measure and you still keep using it. Measure the poor psychometric 00:08:08.300 --> 00:08:13.799 properties, you cannot rely on them. That's the message actually. So again, we've got the nine 00:08:13.799 --> 00:08:20.750 psychometric properties. So not sure that's clear. So I'm going to give you a few more examples, but 00:08:20.750 --> 00:08:27.800 this way now we're going to do little bit different. So these are just different articles. Now you 00:08:27.800 --> 00:08:32.799 look at the title and based on the title you can already see what psychometric property is NOTE Treffsikkerhet: 90% (HØY) 00:08:32.799 --> 00:08:40.400 being addressed in this article. Now, I think that I normally I do that in a group discussion, but I 00:08:40.400 --> 00:08:46.200 actually decided now on the spot that I will just give you the answer. You interrupt if you are lost- 00:08:46.200 --> 00:08:52.900 If I see here, anxiety symptoms and functional impairment and you want to know the correlation, we are 00:08:52.900 --> 00:09:01.300 talking about hypothesis testing. I hope that is clear. Otherwise this your time to response. 00:09:01.300 --> 00:09:02.900 Another one, psychometric properties NOTE Treffsikkerhet: 85% (HØY) 00:09:02.900 --> 00:09:10.300 and factor analysis of a short form of the multi-dimensional measure of emotional abuse. 00:09:10.300 --> 00:09:18.500 What should trigger you is the word factor analysis. We are talking about structural validity. NOTE Treffsikkerhet: 82% (HØY) 00:09:19.100 --> 00:09:25.600 Good practices in normal childbirth, reliability analysis of your instrument, cronbach's alpha. 00:09:25.600 --> 00:09:33.100 Remember that this is cronbach's alpha internal consistency. And that is part of the domain 00:09:33.100 --> 00:09:39.600 reliability. Remember that the domain reliability included internal 00:09:39.600 --> 00:09:48.350 consistency. I'm talking about a child abuse screening tool and it's about NOTE Treffsikkerhet: 91% (HØY) 00:09:48.350 --> 00:09:55.600 instrument development and Multinational pilot testing. You read the term instrument development and we 00:09:55.600 --> 00:10:02.300 know that for sure they talk about content validity. Are you still with me? NOTE Treffsikkerhet: 91% (HØY) 00:10:03.200 --> 00:10:12.600 I hope so. Otherwise just just stop me. Another title, recognition of 00:10:12.600 --> 00:10:16.900 depression, by internists in primary care, 00:10:16.900 --> 00:10:24.200 a comparison of internists and gold standard, psychiatric assesments. 00:10:24.200 --> 00:10:33.250 Now, if you look at that, gold standard, so what you want to know is what this NOTE Treffsikkerhet: 90% (HØY) 00:10:33.250 --> 00:10:40.700 clinician doing compared to Gold Standard assessments and that is 00:10:40.700 --> 00:10:46.900 Criterion validity- That is how we Define criteria validity. Here we're looking at an 00:10:46.900 --> 00:10:53.900 intervention, proposed measurement. We're talking about responsiveness. NOTE Treffsikkerhet: 89% (HØY) 00:10:54.700 --> 00:11:02.400 Here we're talking about Vella validation of Spanish language translation, Etc. You see 00:11:02.400 --> 00:11:07.900 translation and you know, you're talking about cross cultural validity. NOTE Treffsikkerhet: 91% (HØY) 00:11:10.100 --> 00:11:14.900 It was too quick, but this is what we're doing. You see, all these things like minimal important 00:11:14.900 --> 00:11:21.900 change, Smalls detectable change, remember the pictures with high and low resolution, we are talking 00:11:21.900 --> 00:11:29.900 about measurement error. The source of convergence between measures of apathy and depression to 00:11:29.900 --> 00:11:37.599 measures being compared, again, hypothesis testing, convergent validity. NOTE Treffsikkerhet: 82% (HØY) 00:11:37.599 --> 00:11:45.800 Now, that is more tricky one, principal component analysis of bloody blah scales, identification of 00:11:45.800 --> 00:11:50.900 sub scales, this should give it a little bit away. Now, principal component analysis is something 00:11:50.900 --> 00:11:56.400 bit similar to factor analysis, but also you see they are talking about 00:11:56.400 --> 00:12:04.150 identifying subscales. That is the structure of your measure, that is structural validity. NOTE Treffsikkerhet: 90% (HØY) 00:12:04.150 --> 00:12:11.500 Okay, measurement of body temperature to prevent bloody blah, repeated measurement is 00:12:11.500 --> 00:12:16.900 necessary... That could be test-retest. We're not quite sure because they don't really say what 00:12:16.900 --> 00:12:23.700 they're doing, but it seems, but is very likely that that is test-retest. Now, that was 00:12:23.700 --> 00:12:28.600 the last one of examples, but I hope that you get a little bit, the picture of how to do this 00:12:28.600 --> 00:12:33.500 because this is actually what I would like you to understand of Cosmin, of the Framework, NOTE Treffsikkerhet: 72% (MEDIUM) 00:12:33.500 --> 00:12:38.800 that you understand, there are nine psychometric properties and how to recognize what's being 00:12:38.800 --> 00:12:44.800 described in the article. Any questions so far, or should I just continue? NOTE Treffsikkerhet: 88% (HØY) 00:12:45.300 --> 00:12:49.200 Then I just continue. Okay. NOTE Treffsikkerhet: 81% (HØY) 00:12:49.200 --> 00:12:54.200 Now there is something as Cosmin checklist. So, the Cosmin was the framework and then we 00:12:54.200 --> 00:12:59.900 had all these examples and I would like you to introduce the cosmin checklist, 00:12:59.900 --> 00:13:07.750 which is kind of a critical appraisal tool for psychometric studies. So, it is a checklist. It is 00:13:07.750 --> 00:13:13.700 for the methodological quality of studies that describe psychometric properties of a measure. 00:13:13.700 --> 00:13:19.550 So statics describes, the Criterion validity of a measure, that is a psychometric NOTE Treffsikkerhet: 85% (HØY) 00:13:19.550 --> 00:13:26.349 property, that is a psychometric study therefore and you would like to address, evaluate the 00:13:26.349 --> 00:13:32.800 methodological quality of that study using the cosmin when checklist. It is in fact a Cat, critical 00:13:32.800 --> 00:13:38.600 appraisal tool. So it's got three to thirty eight items per measurement property. You rate them on 00:13:38.600 --> 00:13:45.600 a 4-point scale and the items refer to quality of study design, whether used to write statistics, 00:13:45.600 --> 00:13:49.500 Etc, but it is nothing but a critical appraisal tool, just NOTE Treffsikkerhet: 82% (HØY) 00:13:49.500 --> 00:13:57.600 a little bit of complex one. So there are different ways of dealing with it. So Kaufman says, well, if 00:13:57.600 --> 00:14:03.100 any item is wrong, the whole thing is wrong. Well, there are other authors as some including 00:14:03.100 --> 00:14:09.500 our group and we say, well we make it as a percentage, we say well if one out of ten item is bad 00:14:09.500 --> 00:14:14.800 we are not going to say the whole measure is bad or the whole psychometric property is bad. We say one 00:14:14.800 --> 00:14:19.550 out of ten is bad and we make that into a percentage score. You can NOTE Treffsikkerhet: 74% (MEDIUM) 00:14:19.550 --> 00:14:25.300 argue that but otherwise all measures are being banished. You won't have many left. 00:14:25.300 --> 00:14:32.300 So, this is just an example of cosmin when checklist in the looking at internal consistency. As 00:14:32.300 --> 00:14:40.000 you can see, there are all these items and they are about the design, the statistics, and 00:14:40.000 --> 00:14:45.600 any other item and any other topic left. Just before you start rating, you always need to 00:14:45.600 --> 00:14:49.550 confirm that it NOTE Treffsikkerhet: 85% (HØY) 00:14:49.550 --> 00:14:55.200 based on reflective model, it actually means is it the same underlying construct. Is that what 00:14:55.200 --> 00:15:02.800 we are doing? So you don't want to use a cosmin checklist for a measure that is actually consisting of 00:15:02.800 --> 00:15:09.500 several measures. Meaning not just subscales, but it's different measures about maybe 00:15:09.500 --> 00:15:15.400 different concepts, different constructs. NOTE Treffsikkerhet: 86% (HØY) 00:15:15.400 --> 00:15:21.500 Okay, then we go to back to our Cosmin checklist. This is again internal 00:15:21.500 --> 00:15:28.400 consistency. And these were the items that you need to address and you've got items like this for 00:15:28.400 --> 00:15:34.200 each psychometric property. So every psychometric property has got a different number of items that 00:15:34.200 --> 00:15:40.200 also with different content. Again, this is internal consistency. Just we stay there. And the first 00:15:40.200 --> 00:15:45.349 item is was an internal consistency statistic, that could be cronbach's alpha for instance, NOTE Treffsikkerhet: 80% (HØY) 00:15:45.349 --> 00:15:51.550 used or calculated for each uni dimensional scale or sub skill separately. So if you've got a 00:15:51.550 --> 00:15:57.900 measure with three subscales, you need to use cronbach's Alpha 3 times, each subscale. Now, you can 00:15:57.900 --> 00:16:03.800 score very good, adequate, doubtful, and sometimes it's grey so you can't measure that one, here 00:16:03.800 --> 00:16:09.050 you can do very good, doubtful and in adequate and a is not applicable, but you can't score that one. 00:16:09.050 --> 00:16:15.450 So overall you've got like these are the response options that you can usually tick NOTE Treffsikkerhet: 88% (HØY) 00:16:15.450 --> 00:16:23.400 so, you go through all these items per psychometric properties. And then you rate them 00:16:23.400 --> 00:16:30.700 and that means you'll get yourself a score. Now. This is the, the whole box. You can see the 00:16:30.700 --> 00:16:36.900 overview, you score them. And if you sum them up, you've got the rating for a psychometric 00:16:36.900 --> 00:16:42.500 properties. So, you do each item, but then together, that is your psychometric property, in this case 00:16:42.500 --> 00:16:45.300 internal consistency. NOTE Treffsikkerhet: 72% (MEDIUM) 00:16:45.300 --> 00:16:50.800 You do similar things for each psychometric property and that is a Cat, a critical appraisal tool 00:16:50.800 --> 00:16:56.600 for psychometric studies. And it depends on, you look at the study, you look at what 00:16:56.600 --> 00:17:01.800 psychometric property is being addressed and then you take that part of the cosmin checklist to see 00:17:01.800 --> 00:17:08.300 whether that article is doing a good job or not. Well, maybe that is a bit confusing, you will have 00:17:08.300 --> 00:17:15.500 way more feedback, literature on how this exactly works. I just want you to NOTE Treffsikkerhet: 90% (HØY) 00:17:15.500 --> 00:17:22.500 have a little bit of an idea of how you address the psychometric quality, the methodological 00:17:22.500 --> 00:17:30.650 quality of studies, describing psychometrics. Now, you also have got things like psychometric reviews 00:17:30.650 --> 00:17:34.000 and that is what I would like briefly also to discuss about. NOTE Treffsikkerhet: 86% (HØY) 00:17:34.000 --> 00:17:40.000 So you've seen this before, this is the level of evidence and we said well, you've got at the bottom 00:17:40.000 --> 00:17:45.500 the case series and on top you've got clinical guidelines and the higher in the hierarchy, the 00:17:45.500 --> 00:17:52.400 better your study design, the more severity your data have, the more evidence-based prove 00:17:52.400 --> 00:17:58.200 it actually represents. And one of those is Reviews, they're pretty high in hierarchy. Now, we 00:17:58.200 --> 00:18:04.449 talked about systematic reviews, about how you describe your, how you get your data, NOTE Treffsikkerhet: 84% (HØY) 00:18:04.449 --> 00:18:11.300 your search strategies, we've done that in the database. And if you do a systematic 00:18:11.300 --> 00:18:17.400 review, you need to use Prisma and the Prisma statement will tell you how to perform and how to 00:18:17.400 --> 00:18:22.800 report on a systematic review. It's always a good start because for instance if you look at 00:18:22.800 --> 00:18:31.200 special needs education and we use some of these. This is free text, you get a number 00:18:31.200 --> 00:18:34.350 of hits, but if we look at for instance in this dyslexia NOTE Treffsikkerhet: 78% (HØY) 00:18:34.350 --> 00:18:40.400 and writing we've got way more hits. So there's no way you can read all of them. Now, the 00:18:40.400 --> 00:18:47.500 different type of Reviews, there are reviews that only look at therapy effects of a 00:18:47.500 --> 00:18:54.000 specific intervention. There are reviews about strategies and interventions, then they summarize 00:18:54.000 --> 00:19:00.200 anything that's out there in a certain area like dyslexia You've got guidelines that are 00:19:00.200 --> 00:19:04.400 published, you've got diagnostic reviews, and you've got psychometric reviews. NOTE Treffsikkerhet: 88% (HØY) 00:19:04.400 --> 00:19:10.400 And this is an example of psychometric properties of measures of social inclusion. We've done 00:19:10.400 --> 00:19:18.100 another one on the psychometric review on school, connectedness measures how children experience 00:19:18.100 --> 00:19:26.600 whether they are part of, they belong to the educational institute, that they feel fit and all that. 00:19:26.600 --> 00:19:34.300 We do that. If you do second preview like that, you are back to Cosmin. Cosmin has a NOTE Treffsikkerhet: 83% (HØY) 00:19:34.300 --> 00:19:40.900 database of systematic reviews using psychometric reviews. So including all these outcome 00:19:40.900 --> 00:19:47.850 measures in different areas. So, you can go to the website, database.cosmin.nl 00:19:47.850 --> 00:19:55.800 and any psychometric review using Cosmin has been listed there. You can look for different 00:19:55.800 --> 00:20:01.500 systematic reviews. You can look for different aims and construct of interests, but the overarching 00:20:01.500 --> 00:20:05.200 effect it is Health, but it's also education. NOTE Treffsikkerhet: 73% (MEDIUM) 00:20:05.200 --> 00:20:13.100 And you report according to Prismam because normally what you do is you perform a 00:20:13.100 --> 00:20:19.700 systematic literature search. You evaluate the methodological quality of studies that is using your 00:20:19.700 --> 00:20:25.200 Cat, in this case, your Cosmin risk of bias checklist. 00:20:25.200 --> 00:20:32.050 And then you evaluate the psychometric properties of the measure. So that together is a psychometric review. 00:20:32.050 --> 00:20:34.500 This, is it at a glance. NOTE Treffsikkerhet: 75% (MEDIUM) 00:20:34.500 --> 00:20:41.150 Now, reviews like that are brought together at the Cosmin website. NOTE Treffsikkerhet: 91% (HØY) 00:20:41.150 --> 00:20:46.900 Now if you do a psychometric review, you first start with Prisma and you need to have your 00:20:46.900 --> 00:20:52.100 Cosmin... nearby. Those are the two frameworks you use. One is how to perform your review. 00:20:52.100 --> 00:20:58.800 The other one is what to do is psychometrics and you combine the two. Now, evaluation of measurement, 00:20:58.800 --> 00:21:05.400 you do that in three steps. We need the Cosmin checklist, we need quality 00:21:05.400 --> 00:21:11.500 criteria for good measurement properties, and we need to have an overall quality score. How do we do that? 00:21:12.300 --> 00:21:17.500 So the first step is relatively easy. So you've done your systematic review, you've got your 00:21:17.500 --> 00:21:23.100 studies. And now you want to know, this is exactly with any other review, systematic review. 00:21:23.100 --> 00:21:29.600 You want to know about the study methodology. How good is it? Was there bias that they mess up? What did 00:21:29.600 --> 00:21:36.300 they do? And you use the cosmin risk of bias checklist as the checklist, that is your Cat. Step number one. 00:21:37.500 --> 00:21:44.500 Step number two, there need to be criteria for good measurement property. Meaning how high should 00:21:44.500 --> 00:21:49.900 for instance cronbach's alpha should be. What is good enough? So you need to have criteria that you can 00:21:49.900 --> 00:21:55.700 tick off. Like, okay, they did the Chronbach's Alpha , that is here, so that would be about stats from 00:21:55.700 --> 00:22:03.100 Cronbach's Alpha. Yes, that is good. But then you go to criteria. What is the value for that Alpha? 00:22:03.100 --> 00:22:07.949 What should it be? If it's too low, it's still a bad measure. That's the second step. NOTE Treffsikkerhet: 78% (HØY) 00:22:07.949 --> 00:22:14.350 The third step is you're going to summarize all the studies that you found on a particular measure. 00:22:14.350 --> 00:22:20.800 Let's move on. Step number one is you determine in the article you need to find 00:22:20.800 --> 00:22:26.900 what psychometric properties are addressed and you need to get to your Cosmin 00:22:26.900 --> 00:22:34.600 checklist and tick the boxes that are in that article actually being described. The second one is 00:22:34.600 --> 00:22:37.650 you need to get all the data out of the Articles, the Chronbach's, NOTE Treffsikkerhet: 79% (HØY) 00:22:37.650 --> 00:22:44.300 the ICC's, all the stats and everything you need and you need to check whether that's good 00:22:44.300 --> 00:22:51.200 enough and there are predefined criteria where you can compare with. If you've 00:22:51.200 --> 00:22:57.600 done that, you put it all in overview tables. And then you want to combine all these studies. 00:22:57.600 --> 00:23:03.500 If you've got five studies on one measure, you want to combine the data so you can come up with an 00:23:03.500 --> 00:23:08.000 overall quality assessment and overall NOTE Treffsikkerhet: 74% (MEDIUM) 00:23:08.000 --> 00:23:15.000 decision, that's a good, robust measure or not. So you want to do some pooling or summirazing. 00:23:15.000 --> 00:23:20.200 Now we're not going to do 00:23:20.200 --> 00:23:26.700 this in detail. But step number one, is your cat. Your Cosmin risk of bias checklist. The second 00:23:26.700 --> 00:23:35.900 one is compare the data from your articles, compare the stats with predefined criteria, and that is 00:23:35.900 --> 00:23:37.650 you can evaluate NOTE Treffsikkerhet: 74% (MEDIUM) 00:23:37.650 --> 00:23:43.700 that particular psychometric property of that one measure and this is where you compare all 00:23:43.700 --> 00:23:48.900 the studies on one particular measure. You compare then all the measures then you say 00:23:48.900 --> 00:23:54.600 this is actually the very best form or maybe there's no evidence on it. This is in short 00:23:54.600 --> 00:24:00.500 psychometric review. Now, I'll give you some examples. So this is an example, internal consistency 00:24:00.500 --> 00:24:07.750 again, we are doing measure from checklist, and what we're doing is, every item NOTE Treffsikkerhet: 87% (HØY) 00:24:07.750 --> 00:24:14.200 is rated on these cells. For instance, here. 00:24:14.200 --> 00:24:18.500 You can see your chronbach's alpha but there are some other as well. You tick them off and you do 00:24:18.500 --> 00:24:27.900 your job. Now. How does that look like if I've got here an example, I've got here in measure and 00:24:27.900 --> 00:24:34.000 I've got a number of studies that discuss that measure they discuss different psychometric 00:24:34.000 --> 00:24:37.650 properties. You can see them all here passing by. It's not all NOTE Treffsikkerhet: 87% (HØY) 00:24:37.650 --> 00:24:42.500 complete. Now, let's zoom out a little bit. This is the overview. I know you can hardly see it, but 00:24:42.500 --> 00:24:48.800 you've got here different measures. It was a long list and the AP2. There are five studies 00:24:48.800 --> 00:24:55.700 addressing AP2, and 2 studies addressing that Etc. So, you just list them. Now, based for each 00:24:55.700 --> 00:25:00.100 article you can say this first article, it dresses structural validity, internal 00:25:00.100 --> 00:25:06.200 consistency and hypothesis testing. And by accident, none of these articles address cross-cultural 00:25:06.200 --> 00:25:07.650 validity, reliability or NOTE Treffsikkerhet: 69% (MEDIUM) 00:25:07.650 --> 00:25:14.300 criterion validity. That is just by accident. Because if you see, look here, it only addresses 00:25:14.300 --> 00:25:22.300 hypothesis testing. So you get overviews like that. So there are your studies per measure and this 00:25:22.300 --> 00:25:25.800 is the data that you retrieved from the literature. NOTE Treffsikkerhet: 89% (HØY) 00:25:25.800 --> 00:25:32.500 And that is missing. Gray is missing. There's a lot of data are actually missing. Now if we zoom 00:25:32.500 --> 00:25:39.700 in again, then you can see here for instance internal consistency and you need to combine that. You want 00:25:39.700 --> 00:25:48.000 to know for API to based on the literature we've got these data. So three is a very good, one says 00:25:48.000 --> 00:25:53.300 adequate and one is not reported. So in general, it's actually pretty good for that internal 00:25:53.300 --> 00:25:56.550 consistency. Now, there are all these for NOTE Treffsikkerhet: 89% (HØY) 00:25:56.550 --> 00:26:01.700 quality criteria that you can compare with. Like, this is for instance, again, the internal 00:26:01.700 --> 00:26:08.300 consistency 0.7 , we've seen that one before, sometimes criteria are not met, low evidence, 00:26:08.300 --> 00:26:12.900 it's below the 0.7 possibly, this is below the 0.7 and sometimes it's just not 00:26:12.900 --> 00:26:20.050 clear. So you rate the overall evidence then Plus or that or that. Going back here. 00:26:20.050 --> 00:26:26.199 Those questions, we had this already, and we want to combine these data. NOTE Treffsikkerhet: 81% (HØY) 00:26:26.199 --> 00:26:31.800 So that one needs to be combined. And you say, well, if you look at this, for instance, into the 00:26:31.800 --> 00:26:37.600 consistency, we've done now, we compared with the criteria and then all of a sudden, what you can 00:26:37.600 --> 00:26:43.700 see, is the study methodology was okay, but the data they give us, for instance cronbach's alpha 00:26:43.700 --> 00:26:50.700 is actually doubtful and one is not reported. So the overall outcome on internal consistency 00:26:50.700 --> 00:26:56.150 for the API2 to is actually doubtful. We don't know indeterminant. We don't know. NOTE Treffsikkerhet: 75% (MEDIUM) 00:26:56.150 --> 00:26:59.150 It is insufficient data to say yes or no. NOTE Treffsikkerhet: 82% (HØY) 00:26:59.150 --> 00:27:05.100 So that you understand the difference between one is your cat, is to study methodology good? And 00:27:05.100 --> 00:27:11.900 this one has got to do with do you meet the criteria. If the study quality is poor by the way, you're not 00:27:11.900 --> 00:27:16.100 even interested in the article anymore because there's bias probably in it. NOTE Treffsikkerhet: 91% (HØY) 00:27:16.100 --> 00:27:24.700 Now, then the third step, that was at the bottom. That is that one and that is 00:27:24.700 --> 00:27:29.100 you need to pull all the data evidence you've got. And that's where I stopped, because I don't think 00:27:29.100 --> 00:27:33.400 you're going to perform a systematic review at this stage. But as you can see, is quite 00:27:33.400 --> 00:27:40.600 complex. We've got a lot of literature on it. But what I want you to understand is what is the cosmin 00:27:40.600 --> 00:27:46.200 framework? Why do you need a checklist? Well, that is for NOTE Treffsikkerhet: 82% (HØY) 00:27:46.200 --> 00:27:54.500 determine the quality, methodological quality of studies that address psychometrics. And if you do, 00:27:54.500 --> 00:28:01.900 psychometric review, what you do is you retrieve using Prisma guidelines, retrieve any data, any 00:28:01.900 --> 00:28:08.400 psychometic data per measure or for all measures in a certain area. You first do your psychometric review, then 00:28:08.400 --> 00:28:13.800 you've got your systematic review. Then you retrieve all the articlesa and in each article you're going to 00:28:13.800 --> 00:28:16.050 have a look at what NOTE Treffsikkerhet: 77% (HØY) 00:28:16.050 --> 00:28:23.600 psychometric properties are addressed. Use your Cosmin Cat for to address the methodological quality. If 00:28:23.600 --> 00:28:31.200 it's not poor, you include the measure. You go to the next step, you compare the data on the psychometric 00:28:31.200 --> 00:28:38.500 properties with predefined criteria. And the third step, you try to combine any studies you found on 00:28:38.500 --> 00:28:43.800 a particular measured so you can do an overall statement. Like well, there were three studies that 00:28:43.800 --> 00:28:46.300 said it's fantastic and one said it's not that good, NOTE Treffsikkerhet: 87% (HØY) 00:28:46.300 --> 00:28:53.000 it's probably a pretty good measure still. Okay, that is 00:28:53.000 --> 00:29:01.200 psychometric review. That's all very nice. But how do we select a screen or an assessment? How do we 00:29:01.200 --> 00:29:08.100 do that? Okay. Now, we talked about beforehand. There's a difference between screening and 00:29:08.100 --> 00:29:15.300 assessment and what's important is what do you want to screen? Who are you going to screen or assess? 00:29:15.300 --> 00:29:16.050 And how do you do NOTE Treffsikkerhet: 83% (HØY) 00:29:16.050 --> 00:29:23.100 deal with quality assessment? Now if we go to the screen and I say, okay, the screen and we 00:29:23.100 --> 00:29:29.500 want to know the what and the who, then you need to decide on what is the construct you want to 00:29:29.500 --> 00:29:36.700 measure. I've got to talk reading. I'm going to do pragmatics, are you interested in intellectual 00:29:36.700 --> 00:29:42.400 disabilities... The second one is what is your target population? So do you want to measure a 00:29:42.400 --> 00:29:46.100 children with ASD or children with physical disabilities? NOTE Treffsikkerhet: 91% (HØY) 00:29:46.100 --> 00:29:52.600 So, those are the two things you need to decide before you start thinking about a screen. Now, if I 00:29:52.600 --> 00:29:58.850 talk about quality assessment, then what we're going to do is because that would be the next step. 00:29:58.850 --> 00:30:05.050 You'll first need to think about study methodology and remember there are critical appraisal tools, 00:30:05.050 --> 00:30:10.600 the cats. And one example, it's QUADRAS, if we talk about screens. So you need to know 00:30:10.600 --> 00:30:16.000 that the study describing and screen is sound. NOTE Treffsikkerhet: 76% (HØY) 00:30:16.000 --> 00:30:23.200 Its robust. It's got no bias. It's got no big problems. But if you look at the screening tool, then 00:30:23.200 --> 00:30:30.000 we've got things like as we discuss diagnostic performance. That one should be okay, and remember 00:30:30.000 --> 00:30:35.700 that it's actually Criterion validity if you use the terms in line with Cosmin because if you do the 00:30:35.700 --> 00:30:42.300 diagnostic performance, if you check that one, remember the crosstabs, your screen was compared to a 00:30:42.300 --> 00:30:46.100 reference test, to the gold standard. Actually this whole NOTE Treffsikkerhet: 76% (HØY) 00:30:46.100 --> 00:30:52.900 thing of diagnosing performance is Criterion validity if you talk about screens. NOTE Treffsikkerhet: 89% (HØY) 00:30:52.900 --> 00:30:58.500 Then there's such a thing as content validity. It needs to be valid. It needs indeed to screen for 00:30:58.500 --> 00:31:04.700 the construct you are interested in. It needs also to have good reliability. In addition 00:31:04.700 --> 00:31:10.000 to diagnostic performance you want intra-rater, inter-rater and test-retest reliability, should be 00:31:10.000 --> 00:31:16.600 okay. And, of course, it should be feasible. If it takes an hour to screen someone, and you've got a 00:31:16.600 --> 00:31:21.900 war with a hundred people and it changes every week, that's impossible. So it needs to be easy to be 00:31:21.900 --> 00:31:23.449 administered. NOTE Treffsikkerhet: 91% (HØY) 00:31:23.449 --> 00:31:29.800 Non-Invasive. Well is thank God in social sciences usually we don't put shots in the kids. So it's 00:31:29.800 --> 00:31:35.500 not invasive. Anyhow, it should not require a lot of training. It should be freely 00:31:35.500 --> 00:31:42.200 available to everybody. Etc. So, this helps when you select a screen. Constructed 00:31:42.200 --> 00:31:48.500 target population is you to decide, study methodology is your Cat and this has got the bit below has 00:31:48.500 --> 00:31:53.350 hgot to do with the screen itself, whether you want to implement it or not. NOTE Treffsikkerhet: 91% (HØY) 00:31:53.350 --> 00:31:58.400 Diagnostic performance, constant validity, reliability and feasibility. NOTE Treffsikkerhet: 90% (HØY) 00:31:59.700 --> 00:32:08.650 Okay. Now we're going to the next one, assessment, because screen is only to identify those at risk, 00:32:08.650 --> 00:32:14.300 but assessments is the follow-up and that's a different story. They're still overlap. If you want 00:32:14.300 --> 00:32:20.150 to, if you are interested in assessment same things need to happen. You need to know your construct, 00:32:20.150 --> 00:32:26.900 your target population. But also the respondent who is going to complete your assessment with child, 00:32:26.900 --> 00:32:27.750 the parent, you the clinician, NOTE Treffsikkerhet: 90% (HØY) 00:32:27.750 --> 00:32:34.400 observer... who is going to use the assessment, who's going to complete it. Of course, a 00:32:34.400 --> 00:32:40.700 child, then you've got issues of comprehensibility, should have a totally different form for a paraent. 00:32:40.700 --> 00:32:47.800 So this is what you decide when we go to the assessment. That's the next step. 00:32:47.800 --> 00:32:54.650 Then you look first at your study methodology. That is a Cat, your Cosmin risk of bias checklist. 00:32:54.650 --> 00:32:57.800 And when you look at your assessment, we are talking NOTE Treffsikkerhet: 89% (HØY) 00:32:57.800 --> 00:33:04.300 about measurement properties. Well, that is the whole story of today. So the three domains, validity 00:33:04.300 --> 00:33:12.100 reliability and responsiveness with all the nine psychometric properties if applicable. Remember if you 00:33:12.100 --> 00:33:19.500 don't have a gold standard and no short version, then Criterion validity is out. If you have no 00:33:19.500 --> 00:33:25.700 translated version than cross-culture validity is very likely not in. But the other ones should 00:33:25.700 --> 00:33:27.750 always be addressed and the NOTE Treffsikkerhet: 89% (HØY) 00:33:27.750 --> 00:33:35.100 last one is again feasibility. And again, like screen it should be easy to administer. The length 00:33:35.100 --> 00:33:42.300 is important. Some assessment take really a long time and you don't always have resources. Like you 00:33:42.300 --> 00:33:48.800 don't have the time or you simply do not have the educationaist or the clinicians available to 00:33:48.800 --> 00:33:55.400 spend so much time. All of that is together helps you deciding about wWhat assessment do you use. 00:33:55.400 --> 00:33:59.750 And that is actually all the whole story of today. NOTE Treffsikkerhet: 77% (HØY) 00:33:59.750 --> 00:34:03.990 Now, let's stop here because I speed it up in the second bit, I can tell.