Validity of CBTIs Essay Example for Free

Validity of CBTIs Essay Opinions regarding the trustworthiness of CBTIs vary considerably. On one extreme are those who argue that the practitioner can have a great deal of confidence in most commercially available test scoring and interpretation programs, since the testing industry usually maintains high standards for computerized assessment. At the other extreme are those who argue that for many programs there are few, if any, published reports of empirically demonstrated validity. Obviously, the validity of any CBTI system depends on the validity of the test for which it is developed: Discussion of this constraint imposed by the general status of psychological assessment lies beyond the scope of this article. Assuming validity of the measure itself, the validity of the CBTI depends on how closely the developer of the system conforms to actuarial findings for the instrument. A CBTI system is actuarial only if its interpretive output is wholly determined by statistical rules linking test scores to non test criteria. In reality, even the best CBTI systems are not fully actuarial, and many present no evidence of relying on actuarial findings to any extent. Moreland lamented that many CBTI systems are developed by individuals with no special qualifications who have never published a scholarly article on either the test in question or on the CBTI. (Forgionne 2005) Scholar noted several factors limiting validity of CBTIs. Among these are (a) unreliability of measures (including both the test itself as well as non test criteria on which test narratives may be based), (b) unwillingness to permit unclassified cases in situations of insufficient or ambiguous data, (c) inadequate attention to base rates and poorly analyzed cutting scores, (d) prediction of non test behaviors beyond those warranted by specific test findings, and (e) generalization across testing situations and populations without regard for potential moderators. Additional sources of error in CBTIs include miscommunication between the interpretation author and the programmer, as well as errors in the automated-interpretive algorithm itself. It bears noting that, with the exception of programming errors, these same shortcomings plague clinician-generated reports to an equal or greater degree. Thus, potential threats to the validity of CBTIs noted here should not be interpreted as favoring non computerized reports. Identifying potential sources of error in a CBTI system remains difficult. Consumers would be better positioned to evaluate specific components of CBTIs if interpretive rules were made available, although these are rarely presented because of proprietary concerns of the developers. Often only portions of CBTI systems have been validated. for example, restricted samples used in validity studies may activate only a subset of narrative statements in the interpretive library. Moreover, CBTIs are often rated for accuracy at the global level. rarely have discrete elements at the sentence or even paragraph level been examined empirically In addition, most CBTIs are closed systems that preclude the user from modifying or replacing either the decision rules or corresponding interpretations for specific components based on local norms, characteristics of the respondent known to moderate test-criterion relationships, or other considerations. Specific strategies for examining the validity of CBTIs vary but generally can be grouped into three approaches ranging from least to most restrictive: (a) customer satisfaction studies, (b) expert opinion modeling, and (c) external criterion studies. (Steiger 1998) Customer Satisfaction Studies In customer satisfaction studies, consumers rate one or more elements of the CBTI (eg, single ratings of the overall report vs. multiple ratings of discrete narrative elements) along one or more dimensions (eg, accuracy, clarity, internal consistency, omission of relevant information, and inclusion of trivial or misleading information). Because such studies do not rely on independent criterion data regarding the test respondent and therefore are less difficult and less costly to conduct, customer satisfaction studies of CBTI validity are more common than other types of CBTI validity research (although even customer satisfaction studies are infrequent relative to the number of commercial CBTI systems available). Several methodological considerations germane to CBTI customer satisfaction studies, among these were the need for (a) a relatively large, representative sample of users or potential users of that interpretive system. (b) Adequate sampling of test respondents both with respect to common socio demographic moderators, as well as the behavioral domains addressed by the test. And (c) specific ratings of discrete narrative elements along multiple dimensions such as those noted above. Even more critical than these are the need in customer satisfaction studies to control for various rater response sets, Considerable attention has been given in the literature to the impact of global, non differentiating descriptors or Barnum statements on ratings of perceived accuracy. As noted, various studies have shown that consumers assign greater accuracy to narratives with a higher saturation of nonspecific statements, particularly when these descriptions are attributed to persons of authority or expertise. Given this tendency, findings from the international study of computer-based MMPI-2 reports might be regarded as less than compelling, a limitation also suggested. (Martinsons 1999)